Remember stacking in ensemble ML? 🤔
text
Remember stacking in ensemble ML? 🤔
new_line
new_line
What happens if you do the reverse of that but with LLMs? 🤯
text
What happens if you do the reverse of that but with LLMs? 🤯
new_line
new_line
Basically, MoE created by merging multiple models (instead of being pre-trained like Mixtral)? 🧠
text
Basically, MoE created by merging multiple models (instead of being pre-trained like Mixtral)? 🧠
new_line
new_line
Frankenstein MoE! (not an official name) 🧟‍♂️
text
Frankenstein MoE! (not an official name) 🧟‍♂️
new_line
new_line
That's the new Kraken architecture! 🐙
text
That's the new Kraken architecture! 🐙
new_line
new_line
It uses a sequence classification model to route inputs to the most suitable language model based on the input's characteristics. 🚦
text
It uses a sequence classification model to route inputs to the most suitable language model based on the input's characteristics. 🚦
new_line
new_line
Yup, multiple full-fledged LLMs are loaded into memory, and then a classification layer decides who gets to generate an output! 🎰
text
Yup, multiple full-fledged LLMs are loaded into memory, and then a classification layer decides who gets to generate an output! 🎰
new_line
new_line
Tell me you have too many GPUs without telling me you have too many GPUs! 🖥️🔥
text
Tell me you have too many GPUs without telling me you have too many GPUs! 🖥️🔥
new_line
new_line
Jokes aside, extremely fascinating research but I don't understand why this can't just be a big model with multiple LORA adapters, that can be decided on the fly? 🤷‍♂️
text
Jokes aside, extremely fascinating research but I don't understand why this can't just be a big model with multiple LORA adapters, that can be decided on the fly? 🤷‍♂️
new_line
new_line
Model: 
text
Model: 
https://huggingface.co/cognitivecomputations/Kraken
https://huggingface.co/cognitivecomputations/Kraken
link
new_line
Github: 
text
Github: 
https://github.com/cognitivecomputations/kraken
https://github.com/cognitivecomputations/kraken
link
new_line
Remember stacking in ensemble ML? 🤔

What happens if you do the reverse of that but with LLMs? 🤯

Basically, MoE created by merging multiple models (instead of being pre-trained like Mixtral)? 🧠

Frankenstein MoE! (not an official name) 🧟‍♂️

That's the new Kraken architecture! 🐙

It uses a sequence classification model to route inputs to the most suitable language model based on the input's characteristics. 🚦

Yup, multiple full-fledged LLMs are loaded into memory, and then a classification layer decides who gets to generate an output! 🎰

Tell me you have too many GPUs without telling me you have too many GPUs! 🖥️🔥

Jokes aside, extremely fascinating research but I don't understand why this can't just be a big model with multiple LORA adapters, that can be decided on the fly? 🤷‍♂️

Model: https://huggingface.co/cognitivecomputations/Kraken
Github: https://github.com/cognitivecomputations/kraken

🔥
NickyNicky
64da9e4bdafcf7595956abcf
https://cdn-avatars.huggingface.co/v1/production/uploads/64da9e4bdafcf7595956abcf/Wf8o2zTUEghptxk-ao_43.jpeg
Alex Sherstinsky
alexsherstinsky
user
635efe2b398ff343c4fa209b
/avatars/53ebfcab852efd849a848a26dc65751c.svg
elsatch
elsatch
user