I finished my model merging experiment day.🤗I would love your thoughts on this.
text
I finished my model merging experiment day.🤗I would love your thoughts on this.
new_line
new_line
What did I do? I merged Mistral Instruct 0.1 and 0.2 models using different merging techniques:
text
What did I do? I merged Mistral Instruct 0.1 and 0.2 models using different merging techniques:
new_line
- SLERP: linear interpolation (most popular method)
text
- SLERP: linear interpolation (most popular method)
new_line
- MoE: replace some forward layers with MoE layers; using a random gate for now
text
- MoE: replace some forward layers with MoE layers; using a random gate for now
new_line
- Frankenmerge: also known as passthrough, but that isn't very cool. It concatenates some specified layers ending in different numbers of params. In my case, I went from 7B to 9B.
text
- Frankenmerge: also known as passthrough, but that isn't very cool. It concatenates some specified layers ending in different numbers of params. In my case, I went from 7B to 9B.
new_line
new_line
Note: merging is not building an ensemble of models. You can read more about merging techniques at 
text
Note: merging is not building an ensemble of models. You can read more about merging techniques at 
https://huggingface.co/blog/mlabonne/merge-models
https://huggingface.co/blog/mlabonne/merge-models
link
new_line
new_line
Results
text
Results
new_line
I built the 3 models using mergekit (running in an HF Space) - took less than an hour to do the three) 
text
I built the 3 models using mergekit (running in an HF Space) - took less than an hour to do the three) 
https://huggingface.co/collections/osanseviero/mistral-instruct-merges-659ebf35ca0781acdb86bb0a
osanseviero/mistral-instruct-merges-659ebf35ca0781acdb86bb0a
collection
resource
https://huggingface.co/collections/osanseviero/mistral-instruct-merges-659ebf35ca0781acdb86bb0a
new_line
new_line
I'm doing a quick check with the OpenLLM Leaderboard. 
text
I'm doing a quick check with the OpenLLM Leaderboard. 
new_line
🚨The OpenLLM Leaderboard is more suitable for pre-trained models than instruct models, but I still thought it would be interesting to look at the insights🚨
text
🚨The OpenLLM Leaderboard is more suitable for pre-trained models than instruct models, but I still thought it would be interesting to look at the insights🚨
new_line
new_line
You can look at the attached image. Some interesting things
text
You can look at the attached image. Some interesting things
new_line
- All three models performed somewhere between 0.1 and 0.2 - congrats to the 140 people who got it right in 
text
- All three models performed somewhere between 0.1 and 0.2 - congrats to the 140 people who got it right in 
https://twitter.com/osanseviero/status/1745071548866736171
https://twitter.com/osanseviero/status/1745071548866736171
link
new_line
- Frankenmerge terribly sucked with GSM8K. It seems that adding some Mistral 0.1 layers actually degraded the performance a lot  - this is worse than even 0.1!
text
- Frankenmerge terribly sucked with GSM8K. It seems that adding some Mistral 0.1 layers actually degraded the performance a lot  - this is worse than even 0.1!
new_line
- Otherwise, frankenmerge was decent across HellaSwag, MMLU, and specially TruthfulQA
text
- Otherwise, frankenmerge was decent across HellaSwag, MMLU, and specially TruthfulQA
new_line
- MoE is using random gating, so I expected something right in between 0.1 and 0.2, which was the case
text
- MoE is using random gating, so I expected something right in between 0.1 and 0.2, which was the case
new_line
new_line
What do I do with this?
text
What do I do with this?
new_line
Not sure tbh! I think doing proper MT bench evals would be nice. I also think all of us should give a nice GH star to mergekit because it's awesome. I would love to have the time to do end-to-end ablation studies, but cool new things are coming up. Let me know if you have any thoughts in the results
text
Not sure tbh! I think doing proper MT bench evals would be nice. I also think all of us should give a nice GH star to mergekit because it's awesome. I would love to have the time to do end-to-end ablation studies, but cool new things are coming up. Let me know if you have any thoughts in the results
I finished my model merging experiment day.🤗I would love your thoughts on this.

What did I do? I merged Mistral Instruct 0.1 and 0.2 models using different merging techniques:
- SLERP: linear interpolation (most popular method)
- MoE: replace some forward layers with MoE layers; using a random gate for now
- Frankenmerge: also known as passthrough, but that isn't very cool. It concatenates some specified layers ending in different numbers of params. In my case, I went from 7B to 9B.

Note: merging is not building an ensemble of models. You can read more about merging techniques at https://huggingface.co/blog/mlabonne/merge-models

Results
I built the 3 models using mergekit (running in an HF Space) - took less than an hour to do the three) https://huggingface.co/collections/osanseviero/mistral-instruct-merges-659ebf35ca0781acdb86bb0a

I'm doing a quick check with the OpenLLM Leaderboard. 
🚨The OpenLLM Leaderboard is more suitable for pre-trained models than instruct models, but I still thought it would be interesting to look at the insights🚨

You can look at the attached image. Some interesting things
- All three models performed somewhere between 0.1 and 0.2 - congrats to the 140 people who got it right in https://twitter.com/osanseviero/status/1745071548866736171
- Frankenmerge terribly sucked with GSM8K. It seems that adding some Mistral 0.1 layers actually degraded the performance a lot  - this is worse than even 0.1!
- Otherwise, frankenmerge was decent across HellaSwag, MMLU, and specially TruthfulQA
- MoE is using random gating, so I expected something right in between 0.1 and 0.2, which was the case

What do I do with this?
Not sure tbh! I think doing proper MT bench evals would be nice. I also think all of us should give a nice GH star to mergekit because it's awesome. I would love to have the time to do end-to-end ablation studies, but cool new things are coming up. Let me know if you have any thoughts in the results
❤️
davanstrien merve mlabonne mrm8488 abhibisht89 gblazex eitanturok detakarang dvilasuero victor julien-c kgourgou thomwolf mwitiderrick not-lain pcuenq marcsun13 clem chansung radames lysandre KnutJaegersberg toshas Mohammadreza m-ric ernestp56 wenqiglantz bhadresh-savani Dlbk nouamanetazi samusenps Csplk Warung jiandong chrissarmstrong sbrandeis Haleshot
🤯
davanstrien pprp dvilasuero julien-c not-lain mrfakename clem chansung lysandre KnutJaegersberg DrishtiSharma nouamanetazi macadeliccc
🤗
mrm8488 julien-c not-lain clem chansung Haleshot
6032802e1f993496bc14d9e3
https://cdn-avatars.huggingface.co/v1/production/uploads/6032802e1f993496bc14d9e3/w6hr-DEQot4VVkoyRIBiy.png
Omar Sanseviero
osanseviero
user
61b8e2ba285851687028d395
https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/Rq3xWG7mJ3aCRoBsq340h.jpeg
Maxime Labonne
mlabonne
user
5f7fbd813e94f16a85448745
https://cdn-avatars.huggingface.co/v1/production/uploads/1649681653581-5f7fbd813e94f16a85448745.jpeg
Sayak Paul
sayakpaul
user
5dd96eb166059660ed1ee413
https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/NQtzmrDdbG0H8qkZvRyGk.jpeg
Julien Chaumond
julien-c
user
63f140e52f7c0152e87223f6
/avatars/d6db37999b3099a10787e8fc044becc5.svg
J.D.
duxans
user
60284efc4dabd1865d402f1a
/avatars/ca5f5066924ccb4e7f02471c01e49478.svg
Aiswarya Sankar
aiswaryasankar
user