|
Download README.md from flexitok/bpe_ces_Latn_16000: direct link, hf CLI and curl.
- Browser
- Download file 1.28 kB
-
https://huggingface.co/flexitok/bpe_ces_Latn_16000/resolve/main/README.md
- Command line
-
hf download hf://flexitok/bpe_ces_Latn_16000/README.md
-
curl -L -o README.md https://huggingface.co/flexitok/bpe_ces_Latn_16000/resolve/main/README.md
1.28 kB
metadata
license: mit
language:
- ces
tags:
- tokenizer
- bpe
- flexitok
- fineweb2
Byte-Level BPE Tokenizer: ces_Latn (16K)
A Byte-Level BPE tokenizer trained on ces_Latn data from Fineweb-2-HQ.
Training Details
| Parameter | Value |
|---|---|
| Algorithm | Byte-Level BPE |
| Language | ces_Latn |
| Target Vocab Size | 16,000 |
| Final Vocab Size | 16,000 |
| Pre-tokenizer | gpt4 |
| Number handling | individual |
| Contraction handling | True |
| Normalizer | NFC |
| Special Tokens | <s>, </s>, <pad>, <unk> |
| Training Shards | 2 |
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("flexitok/-bpe_ces_Latn_16000")
tokens = tokenizer.encode("Hello, world!")
Files
tokenizer.json— Full HuggingFace tokenizervocab.json— Vocabulary mappingmerges.txt— BPE merge rules
Sample Encoding
| Text | Tokens | Token IDs |
|---|---|---|
Hello, world! 12345 This is a test. こんにちは |
H, el, lo, ,, Ġw, orld, !, Ġ, 12, 3, 45, ĠT, his, Ġis, Ġa, Ġtest, ., Ġ, ãģ, ĵ |
43, 585, 262, 15, 935, 7574, 4, 176, 1235, 22, 4127, 343, 6419, 2744, 224, 2408, 17, 176, 9167, 196 |