You Need an Encoder for Native Position-Independent Caching
The Key-Value (KV) cache of Large Language Models (LLMs) is prefix-based, making it highly inefficient for processing contexts retrieved in arbitrary order. Position-Independent Caching (PIC) has been proposed to enable KV reuse without positional constraints; however, existing approaches often incur substantial accuracy degradation, limiting their practical adoption. To address this issue, we propose native PIC by reintroducing the encoder to prevalent decoder-only LLMs and explicitly training it to support PIC. We further develop COMB, a PIC-aware caching system that integrates seamlessly with existing inference frameworks. Experimental results show that COMB reduces Time-to-First-Token (TTFT) by 51-94% and increases throughput by 3times with comparable accuracy. Furthermore, the quality improvement when using DeepSeek-V2-Lite-Chat demonstrates the applicability of COMB to other types of decoder-only LLMs. Our code is available at https://github.com/shijuzhao/Comb.
6983319b9084cb4f0ecb5989
Shiju Zhao
claimed_verified
2026-02-04T12:26:17.510Z
69832f2c95375aec9d779304
/avatars/3705813d1c2ebc89af987ad28981a484.svg
Shiju Zhao
shijuzhao
user
shijuzhao
6983319b9084cb4f0ecb598a
Junhao Hu
6983319b9084cb4f0ecb598b
Jiaqi Zheng
6983319b9084cb4f0ecb598c
Guihai Chen
You Need an Encoder for Native Position-Independent Caching
The Key-Value (KV) cache of Large Language Models (LLMs) is prefix-based, making it highly inefficient for processing contexts retrieved in arbitrary order. Position-Independent Caching (PIC) has been proposed to enable KV reuse without positional constraints; however, existing approaches often incur substantial accuracy degradation, limiting their practical adoption. To address this issue, we propose native PIC by reintroducing the encoder to prevalent decoder-only LLMs and explicitly training it to support PIC. We further develop COMB, a PIC-aware caching system that integrates seamlessly with existing inference frameworks. Experimental results show that COMB reduces Time-to-First-Token (TTFT) by 51-94% and increases throughput by 3times with comparable accuracy. Furthermore, the quality improvement when using DeepSeek-V2-Lite-Chat demonstrates the applicability of COMB to other types of decoder-only LLMs. Our code is available at https://github.com/shijuzhao/Comb.
Native position-independent caching enhances LLM inference efficiency by reintroducing encoders and developing a caching system that reduces latency while maintaining accuracy.
Key-Value cache Large Language Models position-independent caching decoder-only LLMs encoder caching system Time-to-First-Token throughput