Bolmo: Byteifying the Next Generation of Language Models
We introduce Bolmo, the first family of competitive fully open byte-level language models (LMs) at the 1B and 7B parameter scales. In contrast to prior research on byte-level LMs, which focuses predominantly on training from scratch, we train Bolmo by byteifying existing subword-level LMs. Byteification enables overcoming the limitations of subword tokenization - such as insufficient character understanding and efficiency constraints due to the fixed subword vocabulary - while performing at the level of leading subword-level LMs. Bolmo is specifically designed for byteification: our architecture resolves a mismatch between the expressivity of prior byte-level architectures and subword-level LMs, which makes it possible to employ an effective exact distillation objective between Bolmo and the source subword model. This allows for converting a subword-level LM to a byte-level LM by investing less than 1\% of a typical pretraining token budget. Bolmo substantially outperforms all prior byte-level LMs of comparable size, and outperforms the source subword-level LMs on character understanding and, in some cases, coding, while coming close to matching the original LMs' performance on other tasks. Furthermore, we show that Bolmo can achieve inference speeds competitive with subword-level LMs by training with higher token compression ratios, and can be cheaply and effectively post-trained by leveraging the existing ecosystem around the source subword-level LM. Our results finally make byte-level LMs a practical choice competitive with subword-level LMs across a wide set of use cases.
69449224fbf17e708e185ee9
Benjamin Minixhofer
claimed_verified
2025-12-25T20:52:47.333Z
5fbe9404dd6e953c5a521e13
https://cdn-avatars.huggingface.co/v1/production/uploads/1648291574509-5fbe9404dd6e953c5a521e13.jpeg
Benjamin Minixhofer
benjamin
user
benjamin
69449224fbf17e708e185eea
Tyler Murray
69449224fbf17e708e185eeb
Tomasz Limisiewicz
69449224fbf17e708e185eec
Anna Korhonen
69449224fbf17e708e185eed
Luke Zettlemoyer
69449224fbf17e708e185eee
Noah A. Smith
69449224fbf17e708e185eef
Edoardo M. Ponti
69449224fbf17e708e185ef0
Luca Soldaini
claimed_verified
2026-02-04T19:04:39.240Z
5f04d8c45d08220171a0ad32
https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/B0szblazqtHQql_-pWOHW.jpeg
Luca Soldaini
soldni
user
soldni
69449224fbf17e708e185ef1
Valentin Hofmann
Bolmo: Byteifying the Next Generation of Language Models
We introduce Bolmo, the first family of competitive fully open byte-level language models (LMs) at the 1B and 7B parameter scales. In contrast to prior research on byte-level LMs, which focuses predominantly on training from scratch, we train Bolmo by byteifying existing subword-level LMs. Byteification enables overcoming the limitations of subword tokenization - such as insufficient character understanding and efficiency constraints due to the fixed subword vocabulary - while performing at the level of leading subword-level LMs. Bolmo is specifically designed for byteification: our architecture resolves a mismatch between the expressivity of prior byte-level architectures and subword-level LMs, which makes it possible to employ an effective exact distillation objective between Bolmo and the source subword model. This allows for converting a subword-level LM to a byte-level LM by investing less than 1\% of a typical pretraining token budget. Bolmo substantially outperforms all prior byte-level LMs of comparable size, and outperforms the source subword-level LMs on character understanding and, in some cases, coding, while coming close to matching the original LMs' performance on other tasks. Furthermore, we show that Bolmo can achieve inference speeds competitive with subword-level LMs by training with higher token compression ratios, and can be cheaply and effectively post-trained by leveraging the existing ecosystem around the source subword-level LM. Our results finally make byte-level LMs a practical choice competitive with subword-level LMs across a wide set of use cases.
Bolmo, a family of competitive byte-level language models, is trained by converting existing subword-level models, overcoming character understanding and efficiency limitations while achieving performance comparable to subword models.
byte-level language models subword-level LMs byteification character understanding exact distillation objective post-training token budget token compression ratios