2Mamba2Furious: Linear in Complexity, Competitive in Accuracy
Linear attention transformers have become a strong alternative to softmax attention due to their efficiency. However, linear attention tends to be less expressive and results in reduced accuracy compared to softmax attention. To bridge the accuracy gap between softmax attention and linear attention, we manipulate Mamba-2, a very strong linear attention variant. We first simplify Mamba-2 down to its most fundamental and important components, evaluating which specific choices make it most accurate. From this simplified Mamba variant (Mamba-2S), we improve the A-mask and increase the order of the hidden state, resulting in a method, which we call 2Mamba, that is nearly as accurate as softmax attention, yet much more memory efficient for long context lengths. We also investigate elements to Mamba-2 that help surpass softmax attention accuracy. Code is provided for all our experiments
6997d08a7a658569d5a10187
Gabriel Mongaras
admin_assigned
2026-02-20T09:10:43.401Z
62a1280b88bfb47fc40fe75b
https://cdn-avatars.huggingface.co/v1/production/uploads/62a1280b88bfb47fc40fe75b/u6teJWcB6BWdD04G7g6uy.png
Gabriel Mongaras
gmongaras
user
gmongaras
6997d08a7a658569d5a10188
Eric C. Larson
2Mamba2Furious: Linear in Complexity, Competitive in Accuracy
Linear attention transformers have become a strong alternative to softmax attention due to their efficiency. However, linear attention tends to be less expressive and results in reduced accuracy compared to softmax attention. To bridge the accuracy gap between softmax attention and linear attention, we manipulate Mamba-2, a very strong linear attention variant. We first simplify Mamba-2 down to its most fundamental and important components, evaluating which specific choices make it most accurate. From this simplified Mamba variant (Mamba-2S), we improve the A-mask and increase the order of the hidden state, resulting in a method, which we call 2Mamba, that is nearly as accurate as softmax attention, yet much more memory efficient for long context lengths. We also investigate elements to Mamba-2 that help surpass softmax attention accuracy. Code is provided for all our experiments
Researchers enhance linear attention by simplifying Mamba-2 and improving its architectural components to achieve near-softmax accuracy while maintaining memory efficiency for long sequences.
linear attention softmax attention Mamba-2 hidden state A-mask 2Mamba