ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer
Modern language models still rely on fixed, pre-defined subword tokenizations. Once a tokenizer is trained, the LM can only operate at this fixed level of granularity, which often leads to brittle and counterintuitive behaviors even in otherwise strong reasoning models. We introduce ByteFlow Net, a new hierarchical architecture that removes tokenizers entirely and instead enables models to learn their own segmentation of raw byte streams into semantically meaningful units. ByteFlow Net performs compression-driven segmentation based on the coding rate of latent representations, yielding adaptive boundaries while preserving a static computation graph via Top-K selection. Unlike prior self-tokenizing methods that depend on brittle heuristics with human-designed inductive biases, ByteFlow Net adapts its internal representation granularity to the input itself. Experiments demonstrate that this compression-based chunking strategy yields substantial performance gains, with ByteFlow Net outperforming both BPE-based Transformers and previous byte-level architectures. These results suggest that end-to-end, tokenizer-free modeling is not only feasible but also more effective, opening a path toward more adaptive and information-grounded language models.
69b093731f7902d47f6df896
Chunyuan Deng
claimed_verified
2026-03-11T08:44:55.244Z
634b9914dcf125e4da02498b
/avatars/c3eff53d6dd0aa8990f6e54a51a4a510.svg
Chunyuan Deng
CharlesDDDD
user
CharlesDDDD
69b093731f7902d47f6df897
Sanket Lokegaonkar
69b093731f7902d47f6df898
Colin Lockard
69b093731f7902d47f6df899
Besnik Fetahu
69b093731f7902d47f6df89a
Nasser Zalmout
69b093731f7902d47f6df89b
Xian Li
ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer
Modern language models still rely on fixed, pre-defined subword tokenizations. Once a tokenizer is trained, the LM can only operate at this fixed level of granularity, which often leads to brittle and counterintuitive behaviors even in otherwise strong reasoning models. We introduce ByteFlow Net, a new hierarchical architecture that removes tokenizers entirely and instead enables models to learn their own segmentation of raw byte streams into semantically meaningful units. ByteFlow Net performs compression-driven segmentation based on the coding rate of latent representations, yielding adaptive boundaries while preserving a static computation graph via Top-K selection. Unlike prior self-tokenizing methods that depend on brittle heuristics with human-designed inductive biases, ByteFlow Net adapts its internal representation granularity to the input itself. Experiments demonstrate that this compression-based chunking strategy yields substantial performance gains, with ByteFlow Net outperforming both BPE-based Transformers and previous byte-level architectures. These results suggest that end-to-end, tokenizer-free modeling is not only feasible but also more effective, opening a path toward more adaptive and information-grounded language models.
ByteFlow Net presents a tokenizer-free hierarchical architecture that enables language models to learn adaptive segmentation of raw byte streams through compression-driven methods while maintaining a static computation graph.
byte-level architectures self-tokenizing methods compression-driven segmentation latent representations Top-K selection tokenizer-free modeling hierarchical architecture raw byte streams semantically meaningful units