HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
This work introduces Hybrid Sparse Attention (HySparse), a new architecture that interleaves each full attention layer with several sparse attention layers. While conceptually simple, HySparse strategically derives each sparse layer's token selection and KV caches directly from the preceding full attention layer. This architecture resolves two fundamental limitations of prior sparse attention methods. First, conventional approaches typically rely on additional proxies to predict token importance, introducing extra complexity and potentially suboptimal performance. In contrast, HySparse uses the full attention layer as a precise oracle to identify important tokens. Second, existing sparse attention designs often reduce computation without saving KV cache. HySparse enables sparse attention layers to reuse the full attention KV cache, thereby reducing both computation and memory. We evaluate HySparse on both 7B dense and 80B MoE models. Across all settings, HySparse consistently outperforms both full attention and hybrid SWA baselines. Notably, in the 80B MoE model with 49 total layers, only 5 layers employ full attention, yet HySparse achieves substantial performance gains while reducing KV cache storage by nearly 10x.
6984201be34659da7e1f4e3c
Yizhao Gao
admin_assigned
2026-02-13T09:28:57.762Z
661c96f48921f03a9dae04c3
/avatars/c486a45ffea7d1a8c72bd8512014b07e.svg
Yizhao Gao
LongMountain
user
LongMountain
6984201be34659da7e1f4e3d
Jianyu Wei
6984201be34659da7e1f4e3e
Qihao Zhang
6984201be34659da7e1f4e3f
Yu Cheng
6984201be34659da7e1f4e40
Shimao Chen
6984201be34659da7e1f4e41
Zhengju Tang
6984201be34659da7e1f4e42
Zihan Jiang
6984201be34659da7e1f4e43
Yifan Song
6984201be34659da7e1f4e44
Hailin Zhang
6984201be34659da7e1f4e45
Liang Zhao
6984201be34659da7e1f4e46
Bo Yang
6984201be34659da7e1f4e47
Gang Wang
6984201be34659da7e1f4e48
Shijie Cao
6984201be34659da7e1f4e49
Fuli Luo
HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
This work introduces Hybrid Sparse Attention (HySparse), a new architecture that interleaves each full attention layer with several sparse attention layers. While conceptually simple, HySparse strategically derives each sparse layer's token selection and KV caches directly from the preceding full attention layer. This architecture resolves two fundamental limitations of prior sparse attention methods. First, conventional approaches typically rely on additional proxies to predict token importance, introducing extra complexity and potentially suboptimal performance. In contrast, HySparse uses the full attention layer as a precise oracle to identify important tokens. Second, existing sparse attention designs often reduce computation without saving KV cache. HySparse enables sparse attention layers to reuse the full attention KV cache, thereby reducing both computation and memory. We evaluate HySparse on both 7B dense and 80B MoE models. Across all settings, HySparse consistently outperforms both full attention and hybrid SWA baselines. Notably, in the 80B MoE model with 49 total layers, only 5 layers employ full attention, yet HySparse achieves substantial performance gains while reducing KV cache storage by nearly 10x.
Hybrid Sparse Attention architecture interleaves full and sparse attention layers, using full attention output to guide sparse layer token selection and cache reuse for improved efficiency and performance.
Hybrid Sparse Attention sparse attention layers full attention layer token selection KV caches attention mechanism MoE models computational efficiency memory reduction