Does tokenizing numbers into single digits outperform three-digit or BPE tokenization for arithmetic tasks? We explore various tokenization methods in our upcoming blog (releasing next week 👀)!
text
Does tokenizing numbers into single digits outperform three-digit or BPE tokenization for arithmetic tasks? We explore various tokenization methods in our upcoming blog (releasing next week 👀)!
new_line
new_line
🔹 Bringing objectivity to comparisons
text
🔹 Bringing objectivity to comparisons
new_line
new_line
Existing comparisons of number tokenization methods often ignore the difference in models’ compute budgets: larger tokenizer vocabularies naturally lead to more parameters, which produces less objective comparisons of model performances due to more “learning” being done by these bigger models.
text
Existing comparisons of number tokenization methods often ignore the difference in models’ compute budgets: larger tokenizer vocabularies naturally lead to more parameters, which produces less objective comparisons of model performances due to more “learning” being done by these bigger models.
new_line
new_line
We addressed this by keeping architectures consistent but adjusting the number of hidden layers to produce roughly equal parameter counts.
text
We addressed this by keeping architectures consistent but adjusting the number of hidden layers to produce roughly equal parameter counts.
new_line
new_line
🔹 Key results
text
🔹 Key results
new_line
new_line
We trained models on the same data mix and evaluated their performance on various arithmetic tasks (digits, operations, floats vs. ints):
text
We trained models on the same data mix and evaluated their performance on various arithmetic tasks (digits, operations, floats vs. ints):
new_line
new_line
- When splitting evals based on operators, single-digit tokenization consistently outperformed other methods.
text
- When splitting evals based on operators, single-digit tokenization consistently outperformed other methods.
new_line
- Right-to-left tokenization (which I covered in a previous post) matched or exceeded left-to-right approaches in all tasks.
text
- Right-to-left tokenization (which I covered in a previous post) matched or exceeded left-to-right approaches in all tasks.
new_line
new_line
All in all, single-digit tokenization is best compared to other methods, and similar to our previous post’s finding, R2L works better than L2R tokenization, although not as significant as the gap between single-digit and the rest!
text
All in all, single-digit tokenization is best compared to other methods, and similar to our previous post’s finding, R2L works better than L2R tokenization, although not as significant as the gap between single-digit and the rest!
new_line
new_line
The wait is almost over 🤗, the full report is coming next week - stay tuned!
text
The wait is almost over 🤗, the full report is coming next week - stay tuned!
new_line
Does tokenizing numbers into single digits outperform three-digit or BPE tokenization for arithmetic tasks? We explore various tokenization methods in our upcoming blog (releasing next week 👀)!

🔹 Bringing objectivity to comparisons

Existing comparisons of number tokenization methods often ignore the difference in models’ compute budgets: larger tokenizer vocabularies naturally lead to more parameters, which produces less objective comparisons of model performances due to more “learning” being done by these bigger models.

We addressed this by keeping architectures consistent but adjusting the number of hidden layers to produce roughly equal parameter counts.

🔹 Key results

We trained models on the same data mix and evaluated their performance on various arithmetic tasks (digits, operations, floats vs. ints):

- When splitting evals based on operators, single-digit tokenization consistently outperformed other methods.
- Right-to-left tokenization (which I covered in a previous post) matched or exceeded left-to-right approaches in all tasks.

All in all, single-digit tokenization is best compared to other methods, and similar to our previous post’s finding, R2L works better than L2R tokenization, although not as significant as the gap between single-digit and the rest!

The wait is almost over 🤗, the full report is coming next week - stay tuned!

👀
John6666