Prism: Spectral-Aware Block-Sparse Attention
Block-sparse attention is promising for accelerating long-context LLM pre-filling, yet identifying relevant blocks efficiently remains a bottleneck. Existing methods typically employ coarse-grained attention as a proxy for block importance estimation, but often resort to expensive token-level searching or scoring, resulting in significant selection overhead. In this work, we trace the inaccuracy of standard coarse-grained attention via mean pooling to a theoretical root cause: the interaction between mean pooling and Rotary Positional Embeddings (RoPE). We prove that mean pooling acts as a low-pass filter that induces destructive interference in high-frequency dimensions, effectively creating a "blind spot" for local positional information (e.g., slash patterns). To address this, we introduce Prism, a training-free spectral-aware approach that decomposes block selection into high-frequency and low-frequency branches. By applying energy-based temperature calibration, Prism restores the attenuated positional signals directly from pooled representations, enabling block importance estimation using purely block-level operations, thereby improving efficiency. Extensive evaluations confirm that Prism maintains accuracy parity with full attention while delivering up to 5.1times speedup.
698bedd06052d3bed9630a13
Xinghao Wang
claimed_verified
2026-02-11T11:15:09.483Z
6191f22d08a57f265f7f5266
/avatars/215164e7b6d025a3c32555ff541cdd62.svg
XinghaoWang
Singhoo
user
Singhoo
698bedd06052d3bed9630a14
Pengyu Wang
698bedd06052d3bed9630a15
Xiaoran Liu
claimed_verified
2026-02-11T11:15:02.192Z
64f033ef82c6eea604c4da8b
/avatars/51b93fea7fd68b4274ee03701245dcca.svg
Xiaoran Liu (SII)
SII-xrliu
user
SII-xrliu
698bedd06052d3bed9630a16
Fangxu Liu
698bedd06052d3bed9630a17
Jason Chu
698bedd06052d3bed9630a18
Kai Song
698bedd06052d3bed9630a19
Xipeng Qiu
Prism: Spectral-Aware Block-Sparse Attention
Block-sparse attention is promising for accelerating long-context LLM pre-filling, yet identifying relevant blocks efficiently remains a bottleneck. Existing methods typically employ coarse-grained attention as a proxy for block importance estimation, but often resort to expensive token-level searching or scoring, resulting in significant selection overhead. In this work, we trace the inaccuracy of standard coarse-grained attention via mean pooling to a theoretical root cause: the interaction between mean pooling and Rotary Positional Embeddings (RoPE). We prove that mean pooling acts as a low-pass filter that induces destructive interference in high-frequency dimensions, effectively creating a "blind spot" for local positional information (e.g., slash patterns). To address this, we introduce Prism, a training-free spectral-aware approach that decomposes block selection into high-frequency and low-frequency branches. By applying energy-based temperature calibration, Prism restores the attenuated positional signals directly from pooled representations, enabling block importance estimation using purely block-level operations, thereby improving efficiency. Extensive evaluations confirm that Prism maintains accuracy parity with full attention while delivering up to 5.1times speedup.
Prism addresses inefficiencies in block-sparse attention for long-context LLM pre-filling by using a spectral-aware approach that improves block selection accuracy through energy-based temperature calibration.
block-sparse attention long-context LLM pre-filling coarse-grained attention Rotary Positional Embeddings RoPE mean pooling low-pass filter destructive interference spectral-aware approach energy-based temperature calibration block selection attention mechanisms