Context Compression via Explicit Information Transmission
Long-context inference with Large Language Models (LLMs) is costly due to quadratic attention and growing key-value caches, motivating context compression. In this work, we study soft context compression, where a long context is condensed into a small set of continuous representations. Existing methods typically re-purpose the LLM itself as a trainable compressor, relying on layer-by-layer self-attention to iteratively aggregate information. We argue that this paradigm suffers from two structural limitations: (i) progressive representation overwriting across layers (ii) uncoordinated allocation of compression capacity across tokens. We propose ComprExIT (Context Compression via Explicit Information Transmission), a lightweight framework that formulates soft compression into a new paradigm: explicit information transmission over frozen LLM hidden states. This decouples compression from the model's internal self-attention dynamics. ComprExIT performs (i) depth-wise transmission to selectively transmit multi-layer information into token anchors, mitigating progressive overwriting, and (ii) width-wise transmission to aggregate anchors into a small number of slots via a globally optimized transmission plan, ensuring coordinated allocation of information. Across six question-answering benchmarks, ComprExIT consistently outperforms state-of-the-art context compression methods while introducing only ~1% additional parameters, demonstrating that explicit and coordinated information transmission enables more effective and robust long-context compression.
698a23e11b2dc6b37d61ae04
Jiangnan Ye
claimed_verified
2026-02-09T21:06:43.177Z
6356b7c580f8cb3ab777d1e1
/avatars/e815a2e2d496c698c41ce6f75c22840c.svg
Jiangnan Ye
Jiang-nan
user
Jiang-nan
698a23e11b2dc6b37d61ae05
Hanqi Yan
698a23e11b2dc6b37d61ae06
Zhenyi Shen
claimed_verified
2026-02-10T09:07:33.465Z
641b31a4ec5b871c0bcd6932
/avatars/bad42279ad99918b0846053f2fa95ac8.svg
Zhenyi Shen
zen-E
user
zen-E
698a23e11b2dc6b37d61ae07
Heng Chang
claimed_verified
2026-03-23T11:01:22.037Z
64c3698f007b906a75a14943
/avatars/da31d842225cc6074ecd904a82caf931.svg
Heng Chang
hchang95
user
hchang95
698a23e11b2dc6b37d61ae08
Ye Mao
698a23e11b2dc6b37d61ae09
Yulan He
Context Compression via Explicit Information Transmission
Long-context inference with Large Language Models (LLMs) is costly due to quadratic attention and growing key-value caches, motivating context compression. In this work, we study soft context compression, where a long context is condensed into a small set of continuous representations. Existing methods typically re-purpose the LLM itself as a trainable compressor, relying on layer-by-layer self-attention to iteratively aggregate information. We argue that this paradigm suffers from two structural limitations: (i) progressive representation overwriting across layers (ii) uncoordinated allocation of compression capacity across tokens. We propose ComprExIT (Context Compression via Explicit Information Transmission), a lightweight framework that formulates soft compression into a new paradigm: explicit information transmission over frozen LLM hidden states. This decouples compression from the model's internal self-attention dynamics. ComprExIT performs (i) depth-wise transmission to selectively transmit multi-layer information into token anchors, mitigating progressive overwriting, and (ii) width-wise transmission to aggregate anchors into a small number of slots via a globally optimized transmission plan, ensuring coordinated allocation of information. Across six question-answering benchmarks, ComprExIT consistently outperforms state-of-the-art context compression methods while introducing only ~1% additional parameters, demonstrating that explicit and coordinated information transmission enables more effective and robust long-context compression.
ComprExIT introduces a novel approach to long-context inference in LLMs by using explicit information transmission over frozen hidden states, improving compression efficiency through depth-wise and width-wise transmission mechanisms.
soft context compression LLMs quadratic attention key-value caches layer-by-layer self-attention information transmission frozen LLM hidden states depth-wise transmission width-wise transmission token anchors compression capacity