FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration
While large language models (LLMs) excel at handling long-context sequences, they require substantial prefill computation and key-value (KV) cache, which can heavily burden computational efficiency and memory usage in both prefill and decoding stages. Recent works that compress KV caches with prefill acceleration reduce this cost but inadvertently tie the prefill compute reduction to the decoding KV budget. This coupling arises from overlooking the layer-dependent variation of critical context, often leading to accuracy degradation. To address this issue, we introduce FastKV, a KV cache compression framework designed to reduce latency in both prefill and decoding by leveraging the stabilization of token importance in later layers. FastKV performs full-context computation until a Token-Selective Propagation (TSP) layer, which forwards only the most informative tokens to subsequent layers. From these propagated tokens, FastKV independently selects salient KV entries for caching, thereby decoupling KV budget from the prefill compute reduction based on the TSP decision. This independent control of the TSP rate and KV retention rate enables flexible optimization of efficiency and accuracy. Experimental results show that FastKV achieves speedups of up to 1.82times in prefill and 2.87times in decoding compared to the full-context baseline, while matching the accuracy of the baselines that only accelerate the decoding stage. Our code is available at https://github.com/dongwonjo/FastKV.
67a1a75f6aa8429da4945eeb
Dongwon Jo
claimed_verified
2025-02-04T09:39:16.125Z
639ffbc6beb95d698de9640d
/avatars/7ef1aaadd5b378d00e17dc548e42cb7e.svg
Dongwon Jo
dongwonjo
user
dongwonjo
67a1a75f6aa8429da4945eec
Jiwon Song
claimed_verified
2025-02-04T09:39:14.253Z
662672eaebdfec5cfdf1d034
https://cdn-avatars.huggingface.co/v1/production/uploads/662672eaebdfec5cfdf1d034/RhsKly3KvbtPkDuVnEdWb.jpeg
Jiwon Song
jiwonsong
user
jiwonsong
67a1a75f6aa8429da4945eed
Yulhwa Kim
admin_assigned
2025-02-04T17:35:45.706Z
6566ddb96af53c602f80b1e2
/avatars/403c8e486115920e50867b6462ddfd99.svg
Yulhwa Kim
YulhwaKim
user
YulhwaKim
67a1a75f6aa8429da4945eee
Jae-Joon Kim
FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration
While large language models (LLMs) excel at handling long-context sequences, they require substantial prefill computation and key-value (KV) cache, which can heavily burden computational efficiency and memory usage in both prefill and decoding stages. Recent works that compress KV caches with prefill acceleration reduce this cost but inadvertently tie the prefill compute reduction to the decoding KV budget. This coupling arises from overlooking the layer-dependent variation of critical context, often leading to accuracy degradation. To address this issue, we introduce FastKV, a KV cache compression framework designed to reduce latency in both prefill and decoding by leveraging the stabilization of token importance in later layers. FastKV performs full-context computation until a Token-Selective Propagation (TSP) layer, which forwards only the most informative tokens to subsequent layers. From these propagated tokens, FastKV independently selects salient KV entries for caching, thereby decoupling KV budget from the prefill compute reduction based on the TSP decision. This independent control of the TSP rate and KV retention rate enables flexible optimization of efficiency and accuracy. Experimental results show that FastKV achieves speedups of up to 1.82times in prefill and 2.87times in decoding compared to the full-context baseline, while matching the accuracy of the baselines that only accelerate the decoding stage. Our code is available at https://github.com/dongwonjo/FastKV.
FastKV is a KV cache compression framework that reduces prefill and decoding latency by decoupling context computation from cache budget through token-selective propagation and independent key-value entry selection.
large language models key-value cache prefill computation decoding stage KV cache compression token importance Token-Selective Propagation layer-dependent variation context compression computational efficiency