SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizers
The quality of subword tokenization is critical for Large Language Models, yet evaluating tokenizers for morphologically rich Uralic languages is hampered by the lack of clean morpheme lexicons.
  We introduce SampoNLP, a corpus-free toolkit for morphological lexicon creation using MDL-inspired Self-Referential Atomicity Scoring, which filters composite forms through internal structural cues - suited for low-resource settings.
  Using the high-purity lexicons generated by SampoNLP for Finnish, Hungarian, and Estonian, we conduct a systematic evaluation of BPE tokenizers across a range of vocabulary sizes (8k-256k). We propose a unified metric, the Integrated Performance Score (IPS), to navigate the trade-off between morpheme coverage and over-splitting. By analyzing the IPS curves, we identify the "elbow points" of diminishing returns and provide the first empirically grounded recommendations for optimal vocabulary sizes (k) in these languages. Our study not only offers practical guidance but also quantitatively demonstrates the limitations of standard BPE for highly agglutinative languages. The SampoNLP library and all generated resources are made publicly available: https://github.com/AragonerUA/SampoNLP
696908c58854afb90e2fc9b5
Iaroslav Chelombitko
claimed_verified
2026-01-16T14:37:09.005Z
65030ef13e61bc0192252999
/avatars/c1f46bbffc36765ad1bba4bb1cbe3612.svg
Iaroslav
Aragoner
user
Aragoner
696908c58854afb90e2fc9b6
Ekaterina Chelombitko
696908c58854afb90e2fc9b7
Aleksey Komissarov
SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizers
The quality of subword tokenization is critical for Large Language Models, yet evaluating tokenizers for morphologically rich Uralic languages is hampered by the lack of clean morpheme lexicons.
  We introduce SampoNLP, a corpus-free toolkit for morphological lexicon creation using MDL-inspired Self-Referential Atomicity Scoring, which filters composite forms through internal structural cues - suited for low-resource settings.
  Using the high-purity lexicons generated by SampoNLP for Finnish, Hungarian, and Estonian, we conduct a systematic evaluation of BPE tokenizers across a range of vocabulary sizes (8k-256k). We propose a unified metric, the Integrated Performance Score (IPS), to navigate the trade-off between morpheme coverage and over-splitting. By analyzing the IPS curves, we identify the "elbow points" of diminishing returns and provide the first empirically grounded recommendations for optimal vocabulary sizes (k) in these languages. Our study not only offers practical guidance but also quantitatively demonstrates the limitations of standard BPE for highly agglutinative languages. The SampoNLP library and all generated resources are made publicly available: https://github.com/AragonerUA/SampoNLP
A corpus-free toolkit for morphological lexicon creation using MDL-inspired scoring enables systematic evaluation of BPE tokenizers for morphologically rich Uralic languages, establishing optimal vocabulary sizes through integrated performance metrics.
morphological lexicon creation MDL-inspired Self-Referential Atomicity Scoring BPE tokenizers Integrated Performance Score elbow points morpheme coverage over-splitting agglutinative languages