MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens
Long-term memory is a cornerstone of human intelligence. Enabling AI to process lifetime-scale information remains a long-standing pursuit in
  the field. Due to the constraints of full-attention architectures, the effective context length of large language models (LLMs) is typically
  limited to 1M tokens. Existing approaches, such as hybrid linear attention, fixed-size memory states (e.g., RNNs), and external storage
  methods like RAG or agent systems, attempt to extend this limit. However, they often suffer from severe precision degradation and rapidly
  increasing latency as context length grows, an inability to dynamically modify memory content, or a lack of end-to-end optimization. These
  bottlenecks impede complex scenarios like large-corpus summarization, Digital Twins, and long-history agent reasoning, while limiting memory
  capacity and slowing inference. We present Memory Sparse Attention (MSA), an end-to-end trainable, efficient, and massively scalable memory
  model framework. Through core innovations including scalable sparse attention and document-wise RoPE, MSA achieves linear complexity in both
  training and inference while maintaining exceptional stability, exhibiting less than 9% degradation when scaling from 16K to 100M tokens.
  Furthermore, KV cache compression, combined with Memory Parallel, enables 100M-token inference on 2xA800 GPUs. We also propose Memory
  Interleaving to facilitate complex multi-hop reasoning across scattered memory segments. MSA significantly surpasses frontier LLMs,
  state-of-the-art RAG systems, and leading memory agents in long-context benchmarks. These results demonstrate that by decoupling memory
  capacity from reasoning, MSA provides a scalable foundation to endow general-purpose models with intrinsic, lifetime-scale memory.
69c492354ae7eef815b9d955
Yu Chen
69c492354ae7eef815b9d956
Runkai Chen
69c492354ae7eef815b9d957
Sheng Yi
69c492354ae7eef815b9d958
Xinda Zhao
69c492354ae7eef815b9d959
Xiaohong Li
69c492354ae7eef815b9d95a
Jianjin Zhang
claimed_verified
2026-03-27T11:38:07.712Z
67ba8f9607be4ba2bd051f91
/avatars/2fa41be6f51fd3d94545a94f23d22fc8.svg
Jianjin Zhang
Virgilllll
user
Virgilllll
69c492354ae7eef815b9d95b
Jun Sun
69c492354ae7eef815b9d95c
Chuanrui Hu
69c492354ae7eef815b9d95d
Yunyun Han
69c492354ae7eef815b9d95e
Lidong Bing
69c492354ae7eef815b9d95f
Yafeng Deng
69c492354ae7eef815b9d960
Tianqiao Chen
MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens
Long-term memory is a cornerstone of human intelligence. Enabling AI to process lifetime-scale information remains a long-standing pursuit in
  the field. Due to the constraints of full-attention architectures, the effective context length of large language models (LLMs) is typically
  limited to 1M tokens. Existing approaches, such as hybrid linear attention, fixed-size memory states (e.g., RNNs), and external storage
  methods like RAG or agent systems, attempt to extend this limit. However, they often suffer from severe precision degradation and rapidly
  increasing latency as context length grows, an inability to dynamically modify memory content, or a lack of end-to-end optimization. These
  bottlenecks impede complex scenarios like large-corpus summarization, Digital Twins, and long-history agent reasoning, while limiting memory
  capacity and slowing inference. We present Memory Sparse Attention (MSA), an end-to-end trainable, efficient, and massively scalable memory
  model framework. Through core innovations including scalable sparse attention and document-wise RoPE, MSA achieves linear complexity in both
  training and inference while maintaining exceptional stability, exhibiting less than 9% degradation when scaling from 16K to 100M tokens.
  Furthermore, KV cache compression, combined with Memory Parallel, enables 100M-token inference on 2xA800 GPUs. We also propose Memory
  Interleaving to facilitate complex multi-hop reasoning across scattered memory segments. MSA significantly surpasses frontier LLMs,
  state-of-the-art RAG systems, and leading memory agents in long-context benchmarks. These results demonstrate that by decoupling memory
  capacity from reasoning, MSA provides a scalable foundation to endow general-purpose models with intrinsic, lifetime-scale memory.
Memory Sparse Attention (MSA) enables large language models to process extremely long contexts with linear complexity and high efficiency through innovations like sparse attention and document-wise RoPE.
long-term memory large language models attention mechanisms sparse attention RoPE KV cache compression Memory Parallel Memory Interleaving long-context benchmarks