ARC-Encoder: learning compressed text representations for large language
  models
Recent techniques such as retrieval-augmented generation or chain-of-thought
reasoning have led to longer contexts and increased inference costs. Context
compression techniques can reduce these costs, but the most effective
approaches require fine-tuning the target model or even modifying its
architecture. This can degrade its general abilities when not used for this
specific purpose. Here we explore an alternative approach: an encoder that
compresses the context into continuous representations which replace token
embeddings in decoder LLMs. First, we perform a systematic study of training
strategies and architecture choices for the encoder. Our findings led to the
design of an Adaptable text Representations Compressor, named ARC-Encoder,
which outputs x-times fewer continuous representations (typically
x!in!{4,8}) than text tokens. We evaluate ARC-Encoder across a variety
of LLM usage scenarios, ranging from in-context learning to context window
extension, on both instruct and base decoders. Results show that ARC-Encoder
achieves state-of-the-art performance on several benchmarks while improving
computational efficiency at inference. Finally, we demonstrate that our models
can be adapted to multiple decoders simultaneously, allowing a single encoder
to generalize across different decoder LLMs. This makes ARC-Encoder a flexible
and efficient solution for portable encoders that work seamlessly with multiple
LLMs. We release a training code at https://github.com/kyutai-labs/ARC-Encoder
, fine-tuning dataset and pretrained models are available at
https://huggingface.co/collections/kyutai/arc-encoders-68ee18787301407d60a57047 .
68fb287f1ff1070cf1666d1f
Hippolyte Pilchen
claimed_verified
2025-10-24T15:59:40.720Z
66ea9f89a43597a36208be6c
/avatars/7024721892d8171923a8d4dced143d09.svg
Hippolyte Pilchen
HippolyteP
user
HippolyteP
68fb287f1ff1070cf1666d20
Edouard Grave
68fb287f1ff1070cf1666d21
Patrick Pérez
ARC-Encoder: learning compressed text representations for large language
  models
Recent techniques such as retrieval-augmented generation or chain-of-thought
reasoning have led to longer contexts and increased inference costs. Context
compression techniques can reduce these costs, but the most effective
approaches require fine-tuning the target model or even modifying its
architecture. This can degrade its general abilities when not used for this
specific purpose. Here we explore an alternative approach: an encoder that
compresses the context into continuous representations which replace token
embeddings in decoder LLMs. First, we perform a systematic study of training
strategies and architecture choices for the encoder. Our findings led to the
design of an Adaptable text Representations Compressor, named ARC-Encoder,
which outputs x-times fewer continuous representations (typically
x!in!{4,8}) than text tokens. We evaluate ARC-Encoder across a variety
of LLM usage scenarios, ranging from in-context learning to context window
extension, on both instruct and base decoders. Results show that ARC-Encoder
achieves state-of-the-art performance on several benchmarks while improving
computational efficiency at inference. Finally, we demonstrate that our models
can be adapted to multiple decoders simultaneously, allowing a single encoder
to generalize across different decoder LLMs. This makes ARC-Encoder a flexible
and efficient solution for portable encoders that work seamlessly with multiple
LLMs. We release a training code at https://github.com/kyutai-labs/ARC-Encoder
, fine-tuning dataset and pretrained models are available at
https://huggingface.co/collections/kyutai/arc-encoders-68ee18787301407d60a57047 .
An ARC-Encoder compresses context into continuous representations for LLMs, improving inference efficiency and performance across various scenarios.
retrieval-augmented generation chain-of-thought reasoning context compression token embeddings decoder LLMs Adaptable text Representations Compressor ARC-Encoder in-context learning context window extension instruct decoders base decoders pretrained models