SeerAttention-R: Sparse Attention Adaptation for Long Reasoning
We introduce SeerAttention-R, a sparse attention framework specifically
tailored for the long decoding of reasoning models. Extended from
SeerAttention, SeerAttention-R retains the design of learning attention
sparsity through a self-distilled gating mechanism, while removing query
pooling to accommodate auto-regressive decoding. With a lightweight plug-in
gating, SeerAttention-R is flexible and can be easily integrated into existing
pretrained model without modifying the original parameters. We demonstrate that
SeerAttention-R, trained on just 0.4B tokens, maintains near-lossless reasoning
accuracy with 4K token budget in AIME benchmark under large sparse attention
block sizes (64/128). Using TileLang, we develop a highly optimized sparse
decoding kernel that achieves near-theoretical speedups of up to 9x over
FlashAttention-3 on H100 GPU at 90% sparsity. Code is available at:
https://github.com/microsoft/SeerAttention.
684a39599b38e1e5a33a6822
Yizhao Gao
684a39599b38e1e5a33a6823
Shuming Guo
684a39599b38e1e5a33a6824
Shijie Cao
684a39599b38e1e5a33a6825
Yuqing Xia
684a39599b38e1e5a33a6826
Yu Cheng
684a39599b38e1e5a33a6827
Lei Wang
684a39599b38e1e5a33a6828
Lingxiao Ma
684a39599b38e1e5a33a6829
Yutao Sun
684a39599b38e1e5a33a682a
Tianzhu Ye
claimed_verified
2025-11-17T10:33:33.971Z
68ef6325918a07690443c922
/avatars/6200c248f89afca2bab5b62428f3df0d.svg
Tianzhu Ye
6eo
user
6eo
684a39599b38e1e5a33a682b
Li Dong
claimed_verified
2025-08-27T07:12:13.245Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
unilm
user
unilm
684a39599b38e1e5a33a682c
Hayden Kwok-Hay So
684a39599b38e1e5a33a682d
Yu Hua
684a39599b38e1e5a33a682e
Ting Cao
684a39599b38e1e5a33a682f
Fan Yang
684a39599b38e1e5a33a6830
Mao Yang
SeerAttention-R: Sparse Attention Adaptation for Long Reasoning
We introduce SeerAttention-R, a sparse attention framework specifically
tailored for the long decoding of reasoning models. Extended from
SeerAttention, SeerAttention-R retains the design of learning attention
sparsity through a self-distilled gating mechanism, while removing query
pooling to accommodate auto-regressive decoding. With a lightweight plug-in
gating, SeerAttention-R is flexible and can be easily integrated into existing
pretrained model without modifying the original parameters. We demonstrate that
SeerAttention-R, trained on just 0.4B tokens, maintains near-lossless reasoning
accuracy with 4K token budget in AIME benchmark under large sparse attention
block sizes (64/128). Using TileLang, we develop a highly optimized sparse
decoding kernel that achieves near-theoretical speedups of up to 9x over
FlashAttention-3 on H100 GPU at 90% sparsity. Code is available at:
https://github.com/microsoft/SeerAttention.
SeerAttention-R is a sparse attention framework for reasoning models that maintains high accuracy and achieves significant speedups through optimized sparse decoding kernels.
sparse attention reasoning models self-distilled gating mechanism query pooling lightweight plug-in gating AIME benchmark TileLang sparse decoding kernel FlashAttention-3 H100 GPU