Detecting Overflow in Compressed Token Representations for Retrieval-Augmented Generation
Efficient long-context processing remains a crucial challenge for contemporary large language models (LLMs), especially in resource-constrained environments. Soft compression architectures promise to extend effective context length by replacing long token sequences with smaller sets of learned compressed tokens. Yet, the limits of compressibility -- and when compression begins to erase task-relevant content -- remain underexplored. In this paper, we define token overflow as a regime in which compressed representations no longer contain sufficient information to answer a given query, and propose a methodology to characterize and detect it. In the xRAG soft-compression setting, we find that query-agnostic saturation statistics reliably separate compressed from uncompressed token representations, providing a practical tool for identifying compressed tokens but showing limited overflow detection capability. Lightweight probing classifiers over both query and context xRAG representations detect overflow with 0.72 AUC-ROC on average on HotpotQA, SQuADv2, and TriviaQA datasets, demonstrating that incorporating query information improves detection performance. These results advance from query-independent diagnostics to query-aware detectors, enabling low-cost pre-LLM gating to mitigate compression-induced errors.
69947bbbd2ea89ac106cf997
Julia Belikova
69947bbbd2ea89ac106cf998
Danila Rozhevskii
69947bbbd2ea89ac106cf999
Dennis Svirin
claimed_verified
2026-02-17T15:08:12.027Z
693b43e05908e6c3a08ddf38
/avatars/7b56943f8a5e130f1a2a3b7331cfb892.svg
Dennis
wexumin
user
wexumin
69947bbbd2ea89ac106cf99a
Konstantin Polev
69947bbbd2ea89ac106cf99b
Alexander Panchenko
claimed_verified
2026-02-19T09:53:44.030Z
605473729d7c1d4d81b7e52b
https://cdn-avatars.huggingface.co/v1/production/uploads/1662046050710-605473729d7c1d4d81b7e52b.jpeg
Alexander Panchenko
apanc
user
apanc
Detecting Overflow in Compressed Token Representations for Retrieval-Augmented Generation
Efficient long-context processing remains a crucial challenge for contemporary large language models (LLMs), especially in resource-constrained environments. Soft compression architectures promise to extend effective context length by replacing long token sequences with smaller sets of learned compressed tokens. Yet, the limits of compressibility -- and when compression begins to erase task-relevant content -- remain underexplored. In this paper, we define token overflow as a regime in which compressed representations no longer contain sufficient information to answer a given query, and propose a methodology to characterize and detect it. In the xRAG soft-compression setting, we find that query-agnostic saturation statistics reliably separate compressed from uncompressed token representations, providing a practical tool for identifying compressed tokens but showing limited overflow detection capability. Lightweight probing classifiers over both query and context xRAG representations detect overflow with 0.72 AUC-ROC on average on HotpotQA, SQuADv2, and TriviaQA datasets, demonstrating that incorporating query information improves detection performance. These results advance from query-independent diagnostics to query-aware detectors, enabling low-cost pre-LLM gating to mitigate compression-induced errors.
Soft compression architectures for long-context LLMs use query-aware probing classifiers to detect token overflow and mitigate compression-induced errors.
soft compression architectures long-context processing large language models compressed tokens token overflow query-agnostic saturation statistics xRAG probing classifiers AUC-ROC HotpotQA SQuADv2 TriviaQA