Update README.md

README.md

@@ -83,15 +83,15 @@ directionally reliable, not as rigorously verified as the Swahili figures.
 
 | Tokenizer | Vocab Size | Fertility | Entity Fragmentation |
 |---|---|---|---|
-| **otk-bpe sw-150k** | 150,000 | **1.210 (Best!)** | 0.341 |
+| **otk-bpe sw-150k** | 150,000 | **1.210 (Best!)** | **0.341 (Best!)** |
 | otk-bpe sw-100k | 100,000 | 1.233 | 0.435 |
+| AfroXLMR | 250,002 | 1.597 | 0.441 |
 | otk-bpe merged-150k | 150,000 | 1.264 | 0.519 |
 | otk-bpe sw-50k | 50,000 | 1.285 | 0.626 |
 | otk-bpe merged-100k | 100,000 | 1.302 | 0.672 |
-| AfroXLMR | 250,002 | 1.597 | **0.441 (Best entity handling)** |
+| mBERT | ~119,547 | 2.071 | 0.575 |
 | otk-bpe merged-50k | 50,000 | 1.393 | 0.822 |
 | GPT-4o (o200k_base) | 200,019 | 1.841 | 0.768 |
-| mBERT | ~119,547 | 2.071 | 0.575 |
 | GPT-4 (cl100k_base) | 100,277 | 2.462 | 0.820 |
 
 ### Kinyarwanda
@@ -122,8 +122,8 @@ both metrics simultaneously.
 | otk-bpe merged-100k | 100,000 | 1.409 | 0.502 |
 | GPT-4o (o200k_base) | 200,019 | 1.478 | 0.479 |
 | otk-bpe merged-50k | 50,000 | 1.486 | 0.620 |
-| AfroXLMR | 250,002 | 1.597 | **0.434 (Best entity handling)** |
-| mBERT | ~119,547 | 1.616 | 0.388 |
+| AfroXLMR | 250,002 | 1.597 | 0.434 |
+| mBERT | ~119,547 | 1.616 | **0.388 (Best!)** |
 | GPT-4 (cl100k_base) | 100,277 | 1.720 | 0.578 |
 
 ### English
@@ -146,18 +146,28 @@ additional languages in the same vocabulary.
 
 ### Reading the entity fragmentation numbers honestly
 
-Across every language and every vocab size, **AfroXLMR wins or ties on entity
-fragmentation more often than it wins on fertility** — this is a real,
-consistent pattern, not noise. General-purpose tokenizers with much larger
-vocabularies (AfroXLMR: 250k) hold an edge on rare and foreign proper nouns
-specifically, since that's a long-tail, open-ended category where raw vocab
-size matters more than language-dedication. Increasing OTK-BPE's own vocab
-size from 50k→150k substantially narrows this gap (and outright wins it for
-Kinyarwanda) but does not fully close it for Swahili, French, or the merged
-tokenizer at 150k. If NER-style entity accuracy is the critical downstream
-task, this is worth weighing against the fertility advantage — OTK-BPE is
-not a strict win on every axis, and this repository reports that plainly
-rather than only showing the metrics where it wins.
+At 150k vocab, OTK-BPE tokenizers win entity fragmentation **outright**, not
+just fertility, for Swahili and Kinyarwanda — `sw-150k` and `kin-150k` both
+have the lowest fragmentation rate of every tokenizer tested, dedicated or
+general-purpose. The vocab-size increase from 50k to 150k didn't just narrow
+the gap to larger multilingual tokenizers on this metric, it closed and then
+reversed it for these two languages specifically.
+
+The exceptions are French and English, where **mBERT** — not AfroXLMR — has
+the best entity handling (0.388 and 0.285 respectively), edging out even the
+merged tokenizer. This makes sense: mBERT's vocabulary, while smaller than
+AfroXLMR's, was built with heavy exposure to major European-language
+Wikipedia text including large volumes of proper nouns, and English/French
+are exactly where that shows. The merged tokenizer's own English and French
+entity handling (0.558 and 0.428) is competitive but not best-in-class for
+those two languages specifically — a real, honest tradeoff of covering 4
+languages in one shared vocabulary rather than dedicating full budget to
+either.
+
+**Net read**: don't assume general-purpose tokenizers automatically win on
+entity handling just because they have larger vocabularies — that held at
+the smaller 50k vocab size tested earlier in this project, but stopped
+holding once vocab size reached 150k for the languages that needed it most.
 
 ## How to Use