Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models
This work explores the challenge of building ``Machines that Can Remember'', framing long-term memory as the problem of efficient ultra-long context modeling. We argue that this requires three key properties: sparsity, random-access flexibility, and length generalization. To address ultra-long-context modeling, we leverage Hierarchical Sparse Attention (HSA), a novel attention mechanism that satisfies all three properties. We integrate HSA into Transformers to build HSA-UltraLong, which is an 8B-parameter MoE model trained on over 8 trillion tokens and is rigorously evaluated on different tasks with in-domain and out-of-domain context lengths to demonstrate its capability in handling ultra-long contexts. Results show that our model performs comparably to full-attention baselines on in-domain lengths while achieving over 90\% accuracy on most in-context retrieval tasks with contexts up to 16M. This report outlines our experimental insights and open problems, contributing a foundation for future research in ultra-long context modeling.
692d0cae4397b1ec214f67d1
Xiang Hu
692d0cae4397b1ec214f67d2
Zhanchao Zhou
claimed_verified
2025-12-01T09:11:04.203Z
671b3d82101c0d9f78749256
/avatars/902c1006280c6552885b0b946cd2bb16.svg
Zhanchao Zhou
Zcchill
user
Zcchill
692d0cae4397b1ec214f67d3
Ruiqi Liang
692d0cae4397b1ec214f67d4
Zehuan Li
692d0cae4397b1ec214f67d5
Wei Wu
692d0cae4397b1ec214f67d6
Jianguo Li
claimed_verified
2025-12-25T20:53:50.772Z
6539fc3fe279d2fc80c88902
/avatars/432ac2287a7f7d04c2c7e505070a3331.svg
Jianguo Li
jglee2046
user
jglee2046
Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models
This work explores the challenge of building ``Machines that Can Remember'', framing long-term memory as the problem of efficient ultra-long context modeling. We argue that this requires three key properties: sparsity, random-access flexibility, and length generalization. To address ultra-long-context modeling, we leverage Hierarchical Sparse Attention (HSA), a novel attention mechanism that satisfies all three properties. We integrate HSA into Transformers to build HSA-UltraLong, which is an 8B-parameter MoE model trained on over 8 trillion tokens and is rigorously evaluated on different tasks with in-domain and out-of-domain context lengths to demonstrate its capability in handling ultra-long contexts. Results show that our model performs comparably to full-attention baselines on in-domain lengths while achieving over 90\% accuracy on most in-context retrieval tasks with contexts up to 16M. This report outlines our experimental insights and open problems, contributing a foundation for future research in ultra-long context modeling.
Hierarchical Sparse Attention (HSA) is integrated into Transformers to efficiently handle ultra-long contexts, achieving high accuracy on in-context retrieval tasks.
Hierarchical Sparse Attention (HSA) Transformers ultra-long context modeling sparsity random-access flexibility length generalization MoE model