rost-tokenizer
Collection
rost tokenizers: the 32,768-token bilingual tokenizer and the Romanian diacritics variants • 4 items • Updated
rost-tok-ro-normalized
Romanian BPE tokenizer, 32,768 vocab, trained on cedilla-normalized text.
Measured fertility
| tokenizer | tokens/word on Romanian |
|---|---|
| English-only (control) | 2.8421 |
| Romanian, raw (mixed encodings) | 1.5660 |
| Romanian, normalized | 1.5571 |
| Romanian, diacritics stripped | 1.5190 — but 2.0083 on correct Romanian |
A Romanian tokenizer is worth 1.8× over an English one. Stripping
diacritics looks 2.5% better on de-diacriticised text and is 29% worse on
correct Romanian, because it byte-falls-back on every ș ț ă â î.
- Downloads last month
- 9
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support