FASA: Frequency-aware Sparse Attention
The deployment of Large Language Models (LLMs) faces a critical bottleneck when handling lengthy inputs: the prohibitive memory footprint of the Key Value (KV) cache. To address this bottleneck, the token pruning paradigm leverages attention sparsity to selectively retain a small, critical subset of tokens. However, existing approaches fall short, with static methods risking irreversible information loss and dynamic strategies employing heuristics that insufficiently capture the query-dependent nature of token importance. We propose FASA, a novel framework that achieves query-aware token eviction by dynamically predicting token importance. FASA stems from a novel insight into RoPE: the discovery of functional sparsity at the frequency-chunk (FC) level. Our key finding is that a small, identifiable subset of "dominant" FCs consistently exhibits high contextual agreement with the full attention head. This provides a robust and computationally free proxy for identifying salient tokens. %making them a powerful and efficient proxy for token importance. Building on this insight, FASA first identifies a critical set of tokens using dominant FCs, and then performs focused attention computation solely on this pruned subset. % Since accessing only a small fraction of the KV cache, FASA drastically lowers memory bandwidth requirements and computational cost. Across a spectrum of long-context tasks, from sequence modeling to complex CoT reasoning, FASA consistently outperforms all token-eviction baselines and achieves near-oracle accuracy, demonstrating remarkable robustness even under constraint budgets. Notably, on LongBench-V1, FASA reaches nearly 100\% of full-KV performance when only keeping 256 tokens, and achieves 2.56times speedup using just 18.9\% of the cache on AIME24.
698406a2e34659da7e1f4d65
Yifei Wang
698406a2e34659da7e1f4d66
Yueqi Wang
698406a2e34659da7e1f4d67
Zhenrui Yue
698406a2e34659da7e1f4d68
Huimin Zeng
698406a2e34659da7e1f4d69
Yong Wang
698406a2e34659da7e1f4d6a
Ismini Lourentzou
claimed_verified
2026-02-12T20:28:53.722Z
678ac3b31cbaa0b4bc295885
/avatars/1244fc1b305c9c6383df9bb5e4707347.svg
Ismini Lourentzou
isminoula
user
isminoula
698406a2e34659da7e1f4d6b
Zhengzhong Tu
698406a2e34659da7e1f4d6c
Xiangxiang Chu
698406a2e34659da7e1f4d6d
Julian McAuley
FASA: Frequency-aware Sparse Attention
The deployment of Large Language Models (LLMs) faces a critical bottleneck when handling lengthy inputs: the prohibitive memory footprint of the Key Value (KV) cache. To address this bottleneck, the token pruning paradigm leverages attention sparsity to selectively retain a small, critical subset of tokens. However, existing approaches fall short, with static methods risking irreversible information loss and dynamic strategies employing heuristics that insufficiently capture the query-dependent nature of token importance. We propose FASA, a novel framework that achieves query-aware token eviction by dynamically predicting token importance. FASA stems from a novel insight into RoPE: the discovery of functional sparsity at the frequency-chunk (FC) level. Our key finding is that a small, identifiable subset of "dominant" FCs consistently exhibits high contextual agreement with the full attention head. This provides a robust and computationally free proxy for identifying salient tokens. %making them a powerful and efficient proxy for token importance. Building on this insight, FASA first identifies a critical set of tokens using dominant FCs, and then performs focused attention computation solely on this pruned subset. % Since accessing only a small fraction of the KV cache, FASA drastically lowers memory bandwidth requirements and computational cost. Across a spectrum of long-context tasks, from sequence modeling to complex CoT reasoning, FASA consistently outperforms all token-eviction baselines and achieves near-oracle accuracy, demonstrating remarkable robustness even under constraint budgets. Notably, on LongBench-V1, FASA reaches nearly 100\% of full-KV performance when only keeping 256 tokens, and achieves 2.56times speedup using just 18.9\% of the cache on AIME24.
FASA is a novel framework that uses query-aware token eviction and functional sparsity in RoPE to reduce KV cache memory usage while maintaining high performance in long-context LLM tasks.
Large Language Models Key Value cache token pruning attention sparsity query-dependent token importance RoPE functional sparsity frequency-chunk level dominant frequency-chunks token eviction attention computation long-context tasks sequence modeling CoT reasoning token-eviction baselines KV cache memory footprint computational cost LongBench-V1 AIME24