TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs). Despite the importance of tokenization, its role in LM performance and behavior is poorly understood due to the challenge of measuring the impact of tokenization in isolation. To address this need, we present TokSuite, a collection of models and a benchmark that supports research into tokenization's influence on LMs. Specifically, we train fourteen models that use different tokenizers but are otherwise identical using the same architecture, dataset, training budget, and initialization. Additionally, we curate and release a new benchmark that specifically measures model performance subject to real-world perturbations that are likely to influence tokenization. Together, TokSuite allows robust decoupling of the influence of a model's tokenizer, supporting a series of novel findings that elucidate the respective benefits and shortcomings of a wide range of popular tokenizers.
694cd450746a34b55dd54396
Gül Sena Altıntaş
claimed_verified
2025-12-25T20:43:20.164Z
6462472fd1ccd517f46737a4
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/sqHi525isv9wsDXxHBbUa.png
Gül Sena Altıntaş
gsaltintas
user
gsaltintas
694cd450746a34b55dd54397
Malikeh Ehghaghi
claimed_verified
2025-12-25T20:43:26.290Z
63cacbc502ee13c2af9d6759
https://cdn-avatars.huggingface.co/v1/production/uploads/63cacbc502ee13c2af9d6759/dQCSTb6IOPIQDK-DJmIgs.jpeg
Malikeh Ehghaghi
Malikeh1375
user
Malikeh1375
694cd450746a34b55dd54398
Brian Lester
694cd450746a34b55dd54399
Fengyuan Liu
694cd450746a34b55dd5439a
Wanru Zhao
694cd450746a34b55dd5439b
Marco Ciccone
claimed_verified
2025-12-25T20:43:17.328Z
662fbe32246247eeadd0e1f4
https://cdn-avatars.huggingface.co/v1/production/uploads/662fbe32246247eeadd0e1f4/h0bsU30qrzYjC9e4d6BdD.jpeg
Marco Ciccone
mciccone
user
mciccone
694cd450746a34b55dd5439c
Colin Raffel
claimed_verified
2025-12-25T20:43:23.045Z
6079c29765b9d0165cb18392
https://cdn-avatars.huggingface.co/v1/production/uploads/1618592397610-noauth.jpeg
Colin Raffel
craffel
user
craffel
TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs). Despite the importance of tokenization, its role in LM performance and behavior is poorly understood due to the challenge of measuring the impact of tokenization in isolation. To address this need, we present TokSuite, a collection of models and a benchmark that supports research into tokenization's influence on LMs. Specifically, we train fourteen models that use different tokenizers but are otherwise identical using the same architecture, dataset, training budget, and initialization. Additionally, we curate and release a new benchmark that specifically measures model performance subject to real-world perturbations that are likely to influence tokenization. Together, TokSuite allows robust decoupling of the influence of a model's tokenizer, supporting a series of novel findings that elucidate the respective benefits and shortcomings of a wide range of popular tokenizers.
TokSuite enables systematic研究 of tokenization effects on language model performance by providing identical models with different tokenizers and a benchmark for real-world perturbations.
tokenizers language models tokenization TokSuite benchmark real-world perturbations