--- license: mit language: - ces # ISO 639-3 code or "und" if not identifiable tags: - tokenizer - bpe - flexitok - fineweb2 --- # Byte-Level BPE Tokenizer: ces_Latn (16K) A **Byte-Level BPE** tokenizer trained on **ces_Latn** data from Fineweb-2-HQ. ## Training Details | Parameter | Value | |-----------|-------| | Algorithm | Byte-Level BPE | | Language | `ces_Latn` | | Target Vocab Size | 16,000 | | Final Vocab Size | 16,000 | | Pre-tokenizer | gpt4 | | Number handling | individual | | Contraction handling | True | | Normalizer | NFC | | Special Tokens | ``, ``, ``, `` | | Training Shards | 2 | ## Usage ```python from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("flexitok/-bpe_ces_Latn_16000") tokens = tokenizer.encode("Hello, world!") ``` ## Files - `tokenizer.json` — Full HuggingFace tokenizer - `vocab.json` — Vocabulary mapping - `merges.txt` — BPE merge rules ## Sample Encoding | Text | Tokens | Token IDs | |------|--------|-----------| | `Hello, world! 12345 This is a test. こんにちは` | `H, el, lo, ,, Ġw, orld, !, Ġ, 12, 3, 45, ĠT, his, Ġis, Ġa, Ġtest, ., Ġ, ãģ, ĵ` | `43, 585, 262, 15, 935, 7574, 4, 176, 1235, 22, 4127, 343, 6419, 2744, 224, 2408, 17, 176, 9167, 196` |