Proxy Compression for Language Modeling
Modern language models are trained almost exclusively on token sequences produced by a fixed tokenizer, an external lossless compressor often over UTF-8 byte sequences, thereby coupling the model to that compressor. This work introduces proxy compression, an alternative training scheme that preserves the efficiency benefits of compressed inputs while providing an end-to-end, raw-byte interface at inference time. During training, one language model is jointly trained on raw byte sequences and compressed views generated by external compressors; through the process, the model learns to internally align compressed sequences and raw bytes. This alignment enables strong transfer between the two formats, even when training predominantly on compressed inputs which are discarded at inference. Extensive experiments on code language modeling demonstrate that proxy compression substantially improves training efficiency and significantly outperforms pure byte-level baselines given fixed compute budgets. As model scale increases, these gains become more pronounced, and proxy-trained models eventually match or rival tokenizer approaches, all while operating solely on raw bytes and retaining the inherent robustness of byte-level modeling.
69846d265de9acb9debfb1cd
Lin Zheng
69846d265de9acb9debfb1ce
Xinyu Li
69846d265de9acb9debfb1cf
Qian Liu
69846d265de9acb9debfb1d0
Xiachong Feng
claimed_verified
2026-04-14T08:18:25.988Z
6776ae0c91b4c75dac91249c
https://cdn-avatars.huggingface.co/v1/production/uploads/6776ae0c91b4c75dac91249c/uJk3ZnRrzjPCcBNjmrWLI.png
Oran Feng
xiachongfeng
user
xiachongfeng
69846d265de9acb9debfb1d1
Lingpeng Kong
Proxy Compression for Language Modeling
Modern language models are trained almost exclusively on token sequences produced by a fixed tokenizer, an external lossless compressor often over UTF-8 byte sequences, thereby coupling the model to that compressor. This work introduces proxy compression, an alternative training scheme that preserves the efficiency benefits of compressed inputs while providing an end-to-end, raw-byte interface at inference time. During training, one language model is jointly trained on raw byte sequences and compressed views generated by external compressors; through the process, the model learns to internally align compressed sequences and raw bytes. This alignment enables strong transfer between the two formats, even when training predominantly on compressed inputs which are discarded at inference. Extensive experiments on code language modeling demonstrate that proxy compression substantially improves training efficiency and significantly outperforms pure byte-level baselines given fixed compute budgets. As model scale increases, these gains become more pronounced, and proxy-trained models eventually match or rival tokenizer approaches, all while operating solely on raw bytes and retaining the inherent robustness of byte-level modeling.
Proxy compression trains language models on both raw byte sequences and compressed views, enabling efficient training with end-to-end raw-byte inference while maintaining model robustness.
language models token sequences tokenizer UTF-8 byte sequences proxy compression compressed inputs raw byte sequences external compressors joint training internal alignment transfer learning code language modeling training efficiency inference time model scale byte-level modeling