Effective Distillation to Hybrid xLSTM Architectures
There have been numerous attempts to distill quadratic attention-based large language models (LLMs) into sub-quadratic linearized architectures. However, despite extensive research, such distilled models often fail to match the performance of their teacher LLMs on various downstream tasks. We set out the goal of lossless distillation, which we define in terms of tolerance-corrected Win-and-Tie rates between student and teacher on sets of tasks. To this end, we introduce an effective distillation pipeline for xLSTM-based students. We propose an additional merging stage, where individually linearized experts are combined into a single model. We show the effectiveness of this pipeline by distilling base and instruction-tuned models from the Llama, Qwen, and Olmo families. In many settings, our xLSTM-based students recover most of the teacher's performance, and even exceed it on some downstream tasks. Our contributions are an important step towards more energy-efficient and cost-effective replacements for transformer-based LLMs.
69b8fd7a6b3f36c249fbc567
Lukas Hauzenberger
claimed_verified
2026-03-17T09:53:28.478Z
6041ff7ff84ebe399f1c85ea
/avatars/a5e2306f3cd27e0ea1c30eeb81f870fa.svg
Lukas
sirluk
user
sirluk
69b8fd7a6b3f36c249fbc568
Niklas Schmidinger
69b8fd7a6b3f36c249fbc569
Thomas Schmied
69b8fd7a6b3f36c249fbc56a
Anamaria-Roberta Hartl
claimed_verified
2026-03-17T09:53:26.379Z
64d35b95508a6313e319778a
https://cdn-avatars.huggingface.co/v1/production/uploads/64d35b95508a6313e319778a/_epY7-7H4rALNPXh9uZ7h.jpeg
Anamaria-Roberta Hartl
anamariarobertap
user
anamariarobertap
69b8fd7a6b3f36c249fbc56b
David Stap
claimed_verified
2026-03-17T09:53:30.075Z
6359028d6ca91ddc7401b757
/avatars/c45c7c3e8c25eb18cd41afce1d5e9d54.svg
David Stap
davidstap
user
davidstap
69b8fd7a6b3f36c249fbc56c
Pieter-Jan Hoedt
claimed_verified
2026-03-18T08:46:11.605Z
6964d14d0bcee4247528788c
/avatars/31e59dc0b86aa35f09e8fcc1bb3c84e9.svg
Pieter-Jan Hoedt
hoedt
user
hoedt
69b8fd7a6b3f36c249fbc56d
Maximilian Beck
69b8fd7a6b3f36c249fbc56e
Sebastian Böck
69b8fd7a6b3f36c249fbc56f
Günter Klambauer
claimed_verified
2026-03-17T17:34:40.967Z
65a56378482c457f35b1f14b
/avatars/0f95f4e159864a37754d096fad0ecfbb.svg
gk
gklambauer
user
gklambauer
69b8fd7a6b3f36c249fbc570
Sepp Hochreiter
Effective Distillation to Hybrid xLSTM Architectures
There have been numerous attempts to distill quadratic attention-based large language models (LLMs) into sub-quadratic linearized architectures. However, despite extensive research, such distilled models often fail to match the performance of their teacher LLMs on various downstream tasks. We set out the goal of lossless distillation, which we define in terms of tolerance-corrected Win-and-Tie rates between student and teacher on sets of tasks. To this end, we introduce an effective distillation pipeline for xLSTM-based students. We propose an additional merging stage, where individually linearized experts are combined into a single model. We show the effectiveness of this pipeline by distilling base and instruction-tuned models from the Llama, Qwen, and Olmo families. In many settings, our xLSTM-based students recover most of the teacher's performance, and even exceed it on some downstream tasks. Our contributions are an important step towards more energy-efficient and cost-effective replacements for transformer-based LLMs.