bpe_ces_Latn_16000 / README.md
gsaltintas's picture
Upload folder using huggingface_hub
2ebeb5f verified
|
Raw History Blame Contribute Delete
1.28 kB
metadata
license: mit
language:
  - ces
tags:
  - tokenizer
  - bpe
  - flexitok
  - fineweb2

Byte-Level BPE Tokenizer: ces_Latn (16K)

A Byte-Level BPE tokenizer trained on ces_Latn data from Fineweb-2-HQ.

Training Details

Parameter Value
Algorithm Byte-Level BPE
Language ces_Latn
Target Vocab Size 16,000
Final Vocab Size 16,000
Pre-tokenizer gpt4
Number handling individual
Contraction handling True
Normalizer NFC
Special Tokens <s>, </s>, <pad>, <unk>
Training Shards 2

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("flexitok/-bpe_ces_Latn_16000")
tokens = tokenizer.encode("Hello, world!")

Files

  • tokenizer.json — Full HuggingFace tokenizer
  • vocab.json — Vocabulary mapping
  • merges.txt — BPE merge rules

Sample Encoding

Text Tokens Token IDs
Hello, world! 12345 This is a test. こんにちは H, el, lo, ,, Ġw, orld, !, Ġ, 12, 3, 45, ĠT, his, Ġis, Ġa, Ġtest, ., Ġ, ãģ, ĵ 43, 585, 262, 15, 935, 7574, 4, 176, 1235, 22, 4127, 343, 6419, 2744, 224, 2408, 17, 176, 9167, 196