Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
Efficient key-value (KV) cache management is crucial for the practical deployment of large language models (LLMs), yet existing compression techniques often incur a trade-off between performance degradation and computational overhead. We propose a novel gating-based KV cache eviction method for frozen-weight LLMs that achieves high compression ratios with negligible computational cost. Our approach introduces lightweight sink-attention gating modules to identify and retain critical KV pairs, and integrates seamlessly into both the prefill and decoding stages. The proposed gate training algorithm relies on forward passes of an LLM, avoiding expensive backpropagation, while achieving strong task generalization through a task-agnostic reconstruction objective. Extensive experiments across the Qwen2.5-1M, Qwen3, and Gemma3 families show that our method maintains near-lossless performance while evicting up to 70% of the KV cache. The results are consistent across a wide range of tasks, including long-context understanding, code comprehension, and mathematical reasoning, demonstrating the generality of our approach.
69796de4df44b75fa47e4711
Jang-Hyun Kim
69796de4df44b75fa47e4712
Dongyoon Han
claimed_verified
2026-02-17T15:50:06.632Z
660f8cc1a61244f3df3d4426
/avatars/45d59766122bb3482f6dd7f9d98aa87a.svg
Dongyoon Han
calintz
user
calintz
69796de4df44b75fa47e4713
Sangdoo Yun
Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
Efficient key-value (KV) cache management is crucial for the practical deployment of large language models (LLMs), yet existing compression techniques often incur a trade-off between performance degradation and computational overhead. We propose a novel gating-based KV cache eviction method for frozen-weight LLMs that achieves high compression ratios with negligible computational cost. Our approach introduces lightweight sink-attention gating modules to identify and retain critical KV pairs, and integrates seamlessly into both the prefill and decoding stages. The proposed gate training algorithm relies on forward passes of an LLM, avoiding expensive backpropagation, while achieving strong task generalization through a task-agnostic reconstruction objective. Extensive experiments across the Qwen2.5-1M, Qwen3, and Gemma3 families show that our method maintains near-lossless performance while evicting up to 70% of the KV cache. The results are consistent across a wide range of tasks, including long-context understanding, code comprehension, and mathematical reasoning, demonstrating the generality of our approach.
A novel gating-based key-value cache eviction method for frozen-weight large language models achieves high compression ratios with minimal computational overhead while maintaining near-lossless performance across diverse tasks.
key-value cache KV cache eviction frozen-weight LLMs sink-attention gating modules forward passes task-agnostic reconstruction objective long-context understanding code comprehension mathematical reasoning