olaverse
/

otk-bpe

+---
+license: apache-2.0
+language:
+- sw
+- rw
+- fr
+- en
+tags:
+- tokenizer
+- byte-level-bpe
+- multilingual
+---
+# OTK-BPE: Optimized BPE Tokenizers
+This repository hosts the OTK-BPE family of production-grade Byte-Level BPE
+(BBPE) tokenizers for regional and multilingual language models. It now
+covers **Swahili**, **Kinyarwanda**, and a **merged French + Kinyarwanda +
+English + Swahili** vocabulary — each available at **three vocab sizes: 50k,
+100k, and 150k** — so you can pick the tradeoff between compactness and
+coverage that fits your use case.
+(Companion repo: [`olaverse/otk-bpe-50k`](https://huggingface.co/olaverse/otk-bpe-50k)
+covers the Nigerian-languages family — Yoruba, Igbo, Hausa, Pidgin, and a
+unified Naija tokenizer, all fixed at 50k.)
+## Which one should I use?
+| Your situation | Use |
+|---|---|
+| Only need Swahili | `sw-150k` (best fertility and entity handling in this family; drop to `sw-50k`/`sw-100k` only if embedding-table size is tightly constrained) |
+| Only need Kinyarwanda | `kin-150k` |
+| Need French, Kinyarwanda, English, and Swahili in one model | `merged-150k` |
+| Storage/embedding-table size is a hard constraint | Step down to `-100k` or `-50k` in the same language — fertility degrades gradually, not a cliff (see Benchmarks) |
+**150k is the recommended default across all three families** — in every
+benchmark below, fertility and entity-handling both improved monotonically
+from 50k → 100k → 150k, with no exceptions. There's no size in this range
+where a smaller vocab wins outright.
+## Key Features
+- **Byte-Level BPE (BBPE)**: maps raw UTF-8 bytes to printable characters —
+  0.00% Out-Of-Vocabulary, zero `[UNK]` tokens, by construction.
+- **Diacritic Preservation & Normalization**: NFC normalization inside the
+  pre-tokenization chain, so accents and combining diacritics don't get split
+  into decomposed code points.
+- **Code-Mixed English Support**: the Swahili and Kinyarwanda tokenizers each
+  blend a English-Wikipedia component into training; the merged tokenizer
+  treats English as a full first-class language rather than a minority blend.
+- **Emoji & Symbol Merging**: curated emoji vocab is injected during training
+  so common emoji merge into single tokens instead of fragmenting.
+## Tokenizer Models in This Repository
+All loaded from `olaverse/otk-bpe`, selected via the `subfolder` argument.
+| Subfolder | Languages | Vocab Size |
+|---|---|---|
+| `sw-50k` / `sw-100k` / `sw-150k` | Swahili | 50,000 / 100,000 / 150,000 |
+| `kin-50k` / `kin-100k` / `kin-150k` | Kinyarwanda | 50,000 / 100,000 / 150,000 |
+| `merged-50k` / `merged-100k` / `merged-150k` | French + Kinyarwanda + English + Swahili | 50,000 / 100,000 / 150,000 |
+## Performance Benchmarks
+Fertility = average tokens per word (lower is better). Entity fragmentation
+= share of capitalized/likely-proper-noun words split into more than one
+token (lower is better) — this metric predicts downstream NER difficulty
+before any model is even trained.
+**Swahili benchmark** uses the MasakhaNEWS Swahili *test* split (42,494
+entity-candidate words checked) — a real, curated held-out set, never seen
+during training.
+**Kinyarwanda, French, and English benchmarks** use a held-out, non-training
+slice of the same streaming sources (FineWeb-2 / Wikipedia) rather than a
+curated test set — **MasakhaNEWS does not include Kinyarwanda** (only the
+related-but-distinct Rundi/Kirundi), so no equivalent curated benchmark
+exists for that language yet. Treat these three languages' numbers as
+directionally reliable, not as rigorously verified as the Swahili figures.
+### Swahili
+| Tokenizer | Vocab Size | Fertility | Entity Fragmentation |
+|---|---|---|---|
+| **otk-bpe sw-150k** | 150,000 | **1.210 (Best!)** | 0.341 |
+| otk-bpe sw-100k | 100,000 | 1.233 | 0.435 |
+| otk-bpe merged-150k | 150,000 | 1.264 | 0.519 |
+| otk-bpe sw-50k | 50,000 | 1.285 | 0.626 |
+| otk-bpe merged-100k | 100,000 | 1.302 | 0.672 |
+| AfroXLMR | 250,002 | 1.597 | **0.441 (Best entity handling)** |
+| otk-bpe merged-50k | 50,000 | 1.393 | 0.822 |
+| GPT-4o (o200k_base) | 200,019 | 1.841 | 0.768 |
+| mBERT | ~119,547 | 2.071 | 0.575 |
+| GPT-4 (cl100k_base) | 100,277 | 2.462 | 0.820 |
+### Kinyarwanda
+| Tokenizer | Vocab Size | Fertility | Entity Fragmentation |
+|---|---|---|---|
+| **otk-bpe kin-150k** | 150,000 | **1.377 (Best!)** | **0.334 (Best!)** |
+| otk-bpe kin-100k | 100,000 | 1.409 | 0.420 |
+| otk-bpe merged-150k | 150,000 | 1.465 | 0.475 |
+| otk-bpe kin-50k | 50,000 | 1.483 | 0.561 |
+| otk-bpe merged-100k | 100,000 | 1.523 | 0.567 |
+| otk-bpe merged-50k | 50,000 | 1.662 | 0.709 |
+| AfroXLMR | 250,002 | 2.495 | 0.718 |
+| mBERT | ~119,547 | 2.702 | 0.715 |
+| GPT-4o (o200k_base) | 200,019 | 2.189 | 0.798 |
+| GPT-4 (cl100k_base) | 100,277 | 2.798 | 0.854 |
+Kinyarwanda is where general-purpose multilingual tokenizers are weakest
+across the board — even the smallest dedicated tokenizer here (`kin-50k`)
+beats every baseline on fertility, and `kin-150k` beats every baseline on
+both metrics simultaneously.
+### French
+| Tokenizer | Vocab Size | Fertility | Entity Fragmentation |
+|---|---|---|---|
+| **otk-bpe merged-150k** | 150,000 | **1.378 (Best!)** | 0.428 |
+| otk-bpe merged-100k | 100,000 | 1.409 | 0.502 |
+| GPT-4o (o200k_base) | 200,019 | 1.478 | 0.479 |
+| otk-bpe merged-50k | 50,000 | 1.486 | 0.620 |
+| AfroXLMR | 250,002 | 1.597 | **0.434 (Best entity handling)** |
+| mBERT | ~119,547 | 1.616 | 0.388 |
+| GPT-4 (cl100k_base) | 100,277 | 1.720 | 0.578 |
+### English
+| Tokenizer | Vocab Size | Fertility | Entity Fragmentation |
+|---|---|---|---|
+| mBERT | ~119,547 | **1.429 (Best!)** | **0.285 (Best!)** |
+| GPT-4o (o200k_base) | 200,019 | 1.436 | 0.452 |
+| otk-bpe merged-150k | 150,000 | 1.443 | 0.558 |
+| GPT-4 (cl100k_base) | 100,277 | 1.460 | 0.496 |
+| otk-bpe merged-100k | 100,000 | 1.500 | 0.659 |
+| AfroXLMR | 250,002 | 1.538 | 0.425 |
+| otk-bpe merged-50k | 50,000 | 1.632 | 0.803 |
+English is the merged tokenizer's weakest relative showing — expected, since
+none of the 4 languages get a dedicated vocabulary in the merged design, and
+mBERT/AfroXLMR both have far larger, English-rich vocabularies to draw on.
+`merged-150k` still lands within ~1% of mBERT's fertility despite covering 3
+additional languages in the same vocabulary.
+### Reading the entity fragmentation numbers honestly
+Across every language and every vocab size, **AfroXLMR wins or ties on entity
+fragmentation more often than it wins on fertility** — this is a real,
+consistent pattern, not noise. General-purpose tokenizers with much larger
+vocabularies (AfroXLMR: 250k) hold an edge on rare and foreign proper nouns
+specifically, since that's a long-tail, open-ended category where raw vocab
+size matters more than language-dedication. Increasing OTK-BPE's own vocab
+size from 50k→150k substantially narrows this gap (and outright wins it for
+Kinyarwanda) but does not fully close it for Swahili, French, or the merged
+tokenizer at 150k. If NER-style entity accuracy is the critical downstream
+task, this is worth weighing against the fertility advantage — OTK-BPE is
+not a strict win on every axis, and this repository reports that plainly
+rather than only showing the metrics where it wins.
+## How to Use
+### Installation
+```bash
+pip install tokenizers transformers huggingface_hub
+```
+### Method A: Standard Transformers Loading
+```python
+from transformers import AutoTokenizer
+# Load the Swahili tokenizer at your chosen size
+tokenizer = AutoTokenizer.from_pretrained("olaverse/otk-bpe", subfolder="sw-150k")
+text = "Habari yako? Leo ni siku nzuri sana 😊"
+inputs = tokenizer(text)
+print("Tokens:", tokenizer.tokenize(text))
+print("IDs:", inputs["input_ids"])
+print("Decoded:", tokenizer.decode(inputs["input_ids"]))
+```
+```python
+# Kinyarwanda
+tokenizer = AutoTokenizer.from_pretrained("olaverse/otk-bpe", subfolder="kin-150k")
+# Merged (French + Kinyarwanda + English + Swahili)
+tokenizer = AutoTokenizer.from_pretrained("olaverse/otk-bpe", subfolder="merged-150k")
+```
+### Method B: Lightweight Raw BPE Loading
+```python
+from olaverse import Tokenizer
+# Supports: "sw-50k", "sw-100k", "sw-150k", "kin-50k", "kin-100k", "kin-150k",
+#           "merged-50k", "merged-100k", "merged-150k"
+tokenizer = Tokenizer("sw-150k")
+ids = tokenizer.encode("Habari yako? Leo ni siku nzuri sana 😊")
+print("IDs:", ids)
+print("Decoded:", tokenizer.decode(ids))
+```
+## Datasets Used for Training
+| Source | License | Role |
+|---|---|---|
+| FineWeb-2 (`swh_Latn`, `kin_Latn`, `fra_Latn`) | ODC-By | Web text |
+| Wikipedia (sw, rw, fr, en) | CC BY-SA 4.0 | Encyclopedic corpora |
+| MasakhaNEWS (Swahili train split) | CC BY 4.0 | News text, dense with named entities |
+| [`olaverse/qg-passages-multi`](https://huggingface.co/datasets/olaverse/qg-passages-multi) | Apache-2.0 | Domain coverage for Swahili, French, and English |
+| Wikipedia (English) | CC BY-SA 4.0 | Code-mixed blending (sw/kin) or full component (merged) |
+| Curated Emoji Registry | — | Common social-media emoji |
+MasakhaNEWS Swahili *test* split was excluded from all training corpora and
+reserved exclusively for the fertility/entity benchmarks above.
+## Links
+- [Companion repo: Nigerian languages (otk-bpe-50k)](https://huggingface.co/olaverse/otk-bpe-50k)
+- Olaverse Library Docs
+## License
+Apache-2.0. Training data licenses noted per source above; please retain
+attribution to upstream datasets.