MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling
The evolution of large language models (LLMs) towards applications with ultra-long contexts faces challenges posed by the high computational and memory costs of the Transformer architecture. While existing sparse and linear attention mechanisms attempt to mitigate these issues, they typically involve a trade-off between memory efficiency and model performance. This paper introduces MiniCPM-SALA, a 9B-parameter hybrid architecture that integrates the high-fidelity long-context modeling of sparse attention (InfLLM-V2) with the global efficiency of linear attention (Lightning Attention). By employing a layer selection algorithm to integrate these mechanisms in a 1:3 ratio and utilizing a hybrid positional encoding (HyPE), the model maintains efficiency and performance for long-context tasks. Furthermore, we introduce a cost-effective continual training framework that transforms pre-trained Transformer-based models into hybrid models, which reduces training costs by approximately 75% compared to training from scratch. Extensive experiments show that MiniCPM-SALA maintains general capabilities comparable to full-attention models while offering improved efficiency. On a single NVIDIA A6000D GPU, the model achieves up to 3.5x the inference speed of the full-attention model at the sequence length of 256K tokens and supports context lengths of up to 1M tokens, a scale where traditional full-attention 8B models fail because of memory constraints.
698e93e8cace060ff123acbe
MiniCPM Team
698e93e8cace060ff123acbf
Wenhao An
698e93e8cace060ff123acc0
Yingfa Chen
698e93e8cace060ff123acc1
Yewei Fang
698e93e8cace060ff123acc2
Jiayi Li
698e93e8cace060ff123acc3
Xin Li
698e93e8cace060ff123acc4
Yaohui Li
698e93e8cace060ff123acc5
Yishan Li
698e93e8cace060ff123acc6
Yuxuan Li
698e93e8cace060ff123acc7
Biyuan Lin
698e93e8cace060ff123acc8
Chuan Liu
698e93e8cace060ff123acc9
Hezi Liu
698e93e8cace060ff123acca
Siyuan Liu
698e93e8cace060ff123accb
Hongya Lyu
698e93e8cace060ff123accc
Yinxu Pan
698e93e8cace060ff123accd
Shixin Ren
698e93e8cace060ff123acce
Xingyu Shen
698e93e8cace060ff123accf
Zhou Su
698e93e8cace060ff123acd0
Haojun Sun
698e93e8cace060ff123acd1
Yangang Sun
698e93e8cace060ff123acd2
Zhen Leng Thai
698e93e8cace060ff123acd3
Xin Tian
698e93e8cace060ff123acd4
Rui Wang
698e93e8cace060ff123acd5
Xiaorong Wang
698e93e8cace060ff123acd6
Yudong Wang
698e93e8cace060ff123acd7
Bo Wu
698e93e8cace060ff123acd8
Xiaoyue Xu
698e93e8cace060ff123acd9
Dong Xu
698e93e8cace060ff123acda
Shuaikang Xue
698e93e8cace060ff123acdb
Jiawei Yang
698e93e8cace060ff123acdc
Bowen Zhang
698e93e8cace060ff123acdd
Jinqian Zhang
698e93e8cace060ff123acde
Letian Zhang
698e93e8cace060ff123acdf
Shengnan Zhang
698e93e8cace060ff123ace0
Xinyu Zhang
698e93e8cace060ff123ace1
Xinyuan Zhang
698e93e8cace060ff123ace2
Zhu Zhang
698e93e8cace060ff123ace3
Hengyu Zhao
698e93e8cace060ff123ace4
Jiacheng Zhao
698e93e8cace060ff123ace5
Jie Zhou
698e93e8cace060ff123ace6
Zihan Zhou
698e93e8cace060ff123ace7
Shuo Wang
698e93e8cace060ff123ace8
Chaojun Xiao
698e93e8cace060ff123ace9
Xu Han
698e93e8cace060ff123acea
Zhiyuan Liu
698e93e8cace060ff123aceb
Maosong Sun
MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling
The evolution of large language models (LLMs) towards applications with ultra-long contexts faces challenges posed by the high computational and memory costs of the Transformer architecture. While existing sparse and linear attention mechanisms attempt to mitigate these issues, they typically involve a trade-off between memory efficiency and model performance. This paper introduces MiniCPM-SALA, a 9B-parameter hybrid architecture that integrates the high-fidelity long-context modeling of sparse attention (InfLLM-V2) with the global efficiency of linear attention (Lightning Attention). By employing a layer selection algorithm to integrate these mechanisms in a 1:3 ratio and utilizing a hybrid positional encoding (HyPE), the model maintains efficiency and performance for long-context tasks. Furthermore, we introduce a cost-effective continual training framework that transforms pre-trained Transformer-based models into hybrid models, which reduces training costs by approximately 75% compared to training from scratch. Extensive experiments show that MiniCPM-SALA maintains general capabilities comparable to full-attention models while offering improved efficiency. On a single NVIDIA A6000D GPU, the model achieves up to 3.5x the inference speed of the full-attention model at the sequence length of 256K tokens and supports context lengths of up to 1M tokens, a scale where traditional full-attention 8B models fail because of memory constraints.
MiniCPM-SALA combines sparse and linear attention mechanisms in a hybrid architecture to enable efficient processing of ultra-long contexts while maintaining model performance and reducing training costs.
large language models Transformer architecture sparse attention linear attention hybrid architecture layer selection algorithm hybrid positional encoding continual training framework inference speed sequence length token context