HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
Token-level sparse attention mechanisms, exemplified by DeepSeek Sparse Attention (DSA), achieve fine-grained key selection by scoring every historical token for each query using a lightweight indexer, and then computing attention only over the selected subset. While the downstream sparse attention scales efficiently, the indexer still scans the entire prefix for every query, introducing an O(L^2) per-layer bottleneck that becomes prohibitive as context length grows. We propose HISA (Hierarchical Indexed Sparse Attention), a drop-in replacement for the indexer that transforms the search process from a flat token scan into a two-stage hierarchical procedure. First, a block-level coarse filter scores pooled block representatives to prune irrelevant regions. Then, a token-level refinement applies the original indexer only within the remaining candidate blocks. HISA preserves the exact token-level top-k sparsity pattern required by the downstream Sparse MLA operator and requires no additional training. On kernel-level benchmarks, HISA achieves a 2times speedup at 32K context length and 4times at 128K. On Needle-in-a-Haystack and LongBench, we directly replace the indexer in DeepSeek-V3.2 with HISA, without any fine-tuning. HISA closely matches the original DSA in quality while significantly outperforming block-sparse baselines. Moreover, the token selection sets produced by HISA and the original DSA exhibit a mean IoU greater than 99%, indicating that the efficiency gains come with virtually no impact on selection fidelity.
69cb3f9622f9e9dce3fe1ff3
Yufei Xu
claimed_verified
2026-04-02T08:17:42.751Z
66389aa942fbbd291b73b7c3
/avatars/2770e3839df607ad2a17cd69e0cc6f6e.svg
Yufei Xu
YufeiX
user
YufeiX
69cb3f9622f9e9dce3fe1ff4
Fanxu Meng
claimed_verified
2026-04-01T12:46:34.499Z
643f55d4ec817b766686438a
/avatars/0feb460432c92ab9ada0d417a7a38f6a.svg
mengfanxu
fxmeng
user
fxmeng
69cb3f9622f9e9dce3fe1ff5
Fan Jiang
69cb3f9622f9e9dce3fe1ff6
Yuxuan Wang
claimed_verified
2026-04-01T12:46:31.656Z
67d5459c43f1a23c222a9224
https://cdn-avatars.huggingface.co/v1/production/uploads/67d5459c43f1a23c222a9224/A2nJkYD2eGmM5HS5LgjKN.jpeg
Billy Wang
billyisavailable
user
billyisavailable
69cb3f9622f9e9dce3fe1ff7
Ruijie Zhou
claimed_verified
2026-04-02T08:17:40.916Z
6662f3a345a5b23e246b94ba
/avatars/a47dfdc5b6e1e5bf81d33506c8c0e43d.svg
rj z
psv26
user
psv26
69cb3f9622f9e9dce3fe1ff8
Jiexi Wu
claimed_verified
2026-04-02T08:17:38.814Z
691fd60d6101ec36927d56fa
/avatars/9c5c0fe06bb5285e3f05e2811616208a.svg
Jason Wu
jasonjxwu
user
jasonjxwu
69cb3f9622f9e9dce3fe1ff9
Zhixin Pan
claimed_verified
2026-04-03T09:09:05.013Z
69cf366ebfc22869bacd4bf7
/avatars/223a82d682dbb78d80fa0813494d3d27.svg
Zhixin Pan
pzx1050
user
pzx1050
69cb3f9622f9e9dce3fe1ffa
Zhaohui Wang
69cb3f9622f9e9dce3fe1ffb
Xiaojuan Tang
69cb3f9622f9e9dce3fe1ffc
Wenjie Pei
69cb3f9622f9e9dce3fe1ffd
Tongxuan Liu
69cb3f9622f9e9dce3fe1ffe
Di yin
claimed_verified
2026-03-31T09:25:23.596Z
63fc75f9b9db84750cea9c5c
/avatars/2c5bf9685e0cfc4b5785a4a86c34e0db.svg
DI YIN
DIYIN
user
DIYIN
69cb3f9622f9e9dce3fe1fff
Xing Sun
claimed_verified
2026-04-03T09:09:06.894Z
647401e50da364bd0d002f2a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vPuPn7EV092mLBOM2YZXd.png
XING SUN
tedsun
user
tedsun
69cb3f9622f9e9dce3fe2000
Muhan Zhang
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
Token-level sparse attention mechanisms, exemplified by DeepSeek Sparse Attention (DSA), achieve fine-grained key selection by scoring every historical token for each query using a lightweight indexer, and then computing attention only over the selected subset. While the downstream sparse attention scales efficiently, the indexer still scans the entire prefix for every query, introducing an O(L^2) per-layer bottleneck that becomes prohibitive as context length grows. We propose HISA (Hierarchical Indexed Sparse Attention), a drop-in replacement for the indexer that transforms the search process from a flat token scan into a two-stage hierarchical procedure. First, a block-level coarse filter scores pooled block representatives to prune irrelevant regions. Then, a token-level refinement applies the original indexer only within the remaining candidate blocks. HISA preserves the exact token-level top-k sparsity pattern required by the downstream Sparse MLA operator and requires no additional training. On kernel-level benchmarks, HISA achieves a 2times speedup at 32K context length and 4times at 128K. On Needle-in-a-Haystack and LongBench, we directly replace the indexer in DeepSeek-V3.2 with HISA, without any fine-tuning. HISA closely matches the original DSA in quality while significantly outperforming block-sparse baselines. Moreover, the token selection sets produced by HISA and the original DSA exhibit a mean IoU greater than 99%, indicating that the efficiency gains come with virtually no impact on selection fidelity.
HISA improves sparse attention efficiency by replacing the traditional indexer with a hierarchical approach that reduces computational complexity from O(L²) to sub-quadratic scaling while maintaining selection fidelity.
sparse attention mechanisms DeepSeek Sparse Attention token-level sparsity indexer attention computation hierarchical indexing block-level filtering token-level refinement Sparse MLA operator context length kernel-level benchmarks Needle-in-a-Haystack LongBench mean IoU