LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation
Transformer-based large language models (LLMs) rely on key-value (KV) caching to avoid redundant computation during autoregressive inference. While this mechanism greatly improves efficiency, the cache size grows linearly with the input sequence length, quickly becoming a bottleneck for long-context tasks. Existing solutions mitigate this problem by evicting prompt KV that are deemed unimportant, guided by estimated importance scores. Notably, a recent line of work proposes to improve eviction quality by "glimpsing into the future", in which a draft generator produces a surrogate future response approximating the target model's true response, and this surrogate is subsequently used to estimate the importance of cached KV more accurately. However, these approaches rely on computationally expensive draft generation, which introduces substantial prefilling overhead and limits their practicality in real-world deployment. To address this challenge, we propose LookaheadKV, a lightweight eviction framework that leverages the strength of surrogate future response without requiring explicit draft generation. LookaheadKV augments transformer layers with parameter-efficient modules trained to predict true importance scores with high accuracy. Our design ensures negligible runtime overhead comparable to existing inexpensive heuristics, while achieving accuracy superior to more costly approximation methods. Extensive experiments on long-context understanding benchmarks, across a wide range of models, demonstrate that our method not only outperforms recent competitive baselines in various long-context understanding tasks, but also reduces the eviction cost by up to 14.5x, leading to significantly faster time-to-first-token. Our code is available at https://github.com/SamsungLabs/LookaheadKV.
69b7662d91a28162b233fbfd
Jinwoo Ahn
69b7662d91a28162b233fbfe
Ingyu Seong
admin_assigned
2026-03-16T11:10:59.255Z
61c835dfccae6d3b4a6c0037
/avatars/3191ad8aef8688c1ea869d4ac3243529.svg
Ingyu Seong
ingyu
user
ingyu
69b7662d91a28162b233fbff
Akhil Kedia
69b7662d91a28162b233fc00
Junhan Kim
admin_assigned
2026-03-16T11:11:15.265Z
69280e706379309c8abd574f
/avatars/3a7ca41b2f8cb305593789e615edaf37.svg
Junhan Kim
kjuni
user
kjuni
69b7662d91a28162b233fc01
Hyemi Jang
69b7662d91a28162b233fc02
Kangwook Lee
69b7662d91a28162b233fc03
Yongkweon Jeon
LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation
Transformer-based large language models (LLMs) rely on key-value (KV) caching to avoid redundant computation during autoregressive inference. While this mechanism greatly improves efficiency, the cache size grows linearly with the input sequence length, quickly becoming a bottleneck for long-context tasks. Existing solutions mitigate this problem by evicting prompt KV that are deemed unimportant, guided by estimated importance scores. Notably, a recent line of work proposes to improve eviction quality by "glimpsing into the future", in which a draft generator produces a surrogate future response approximating the target model's true response, and this surrogate is subsequently used to estimate the importance of cached KV more accurately. However, these approaches rely on computationally expensive draft generation, which introduces substantial prefilling overhead and limits their practicality in real-world deployment. To address this challenge, we propose LookaheadKV, a lightweight eviction framework that leverages the strength of surrogate future response without requiring explicit draft generation. LookaheadKV augments transformer layers with parameter-efficient modules trained to predict true importance scores with high accuracy. Our design ensures negligible runtime overhead comparable to existing inexpensive heuristics, while achieving accuracy superior to more costly approximation methods. Extensive experiments on long-context understanding benchmarks, across a wide range of models, demonstrate that our method not only outperforms recent competitive baselines in various long-context understanding tasks, but also reduces the eviction cost by up to 14.5x, leading to significantly faster time-to-first-token. Our code is available at https://github.com/SamsungLabs/LookaheadKV.
LookaheadKV is a lightweight framework that improves KV cache eviction in transformers by using parameter-efficient modules to predict importance scores without draft generation, achieving faster inference with reduced eviction costs.
key-value caching autoregressive inference transformer layers parameter-efficient modules surrogate future response eviction quality lookaheadKV