roman2khmer

Romanised Khmer → Khmer script. Many people type Khmer in Latin letters when they chat, with no fixed spelling. This model writes such text back in Khmer script, choosing each word with its neighbours in mind.

input	output
bong mok nham bay nov	បងមកញ៉ាំបាយនៅ
knhom tov sala rean	ខ្ញុំទៅសាលារៀន
ថ្ងៃនេះ Sokha mok pteah	ថ្ងៃនេះ សុខាមកផ្ទះ
oun tov psar jea muy mak	អូនទៅផ្សារជាមួយម៉ាក់

Research preview (v0.1.0). Quality was measured on LLM-written chat messages and on one native annotator's typing; there is no test set of real chat from many writers yet. See Evaluation and Limitations.

How it works

Three small parts, all run with NumPy on a CPU:

1. Word model (translit.npz, 10 MB): a character-level Transformer encoder–decoder (2.6M parameters, width 192, 3 + 3 layers, 4 heads), trained from scratch. It proposes 8 Khmer spellings per Latin word (beam search).
2. Recheck (reverse.npz, 10 MB): the same architecture trained in the other direction (Khmer → Latin). Each candidate's score gains 1.0 × log P(typed Latin | candidate), so candidates the typed letters do not support lose ground (a noisy-channel rescoring).
3. Sentence step (khmer_bigram.npz, 7 MB): a Khmer word-bigram model counted from web text chooses one candidate per word for the whole message (dynamic programming), and may write two neighbouring Latin words as one Khmer word (pon man → ប៉ុន្មាន). Khmer already in the message is kept and used as context.

Not a Transformers checkpoint: use the roman2khmer package (the wheel in this repository).

Quick start

python -m pip install -U huggingface_hub                          # provides the `hf` command
hf download ric912/roman2khmer --revision 67eacafe5bb9592622ebcdd6d64695eb3b6b69f6 --local-dir roman2khmer-model
python -m pip install roman2khmer-model/roman2khmer-0.1.0-py3-none-any.whl   # installs roman2khmer + numpy only

from roman2khmer import Converter

c = Converter.from_dir("roman2khmer-model")   # load once (under a second, about 300 MB RAM)
c.convert("pa ban mok vinh hz")               # 'ប៉ាបានមកវិញហើយ'
c.candidates("mok", 3)                        # [('មក', -1.06), ('ម៉ក', -2.60), ('មុខ', -2.72)]
Converter.from_dir("roman2khmer-model", gamma=0)   # without the recheck (faster)

Speed: about 0.9 s per chat message (2 to 15 words) whose words are all new, 0.4 s without the recheck, measured on a laptop CPU (Intel i7-8750H, NumPy, one process); repeated words are cached.

Evaluation

We have not evaluated this release on a public, multi-writer real-chat benchmark. We report:

Measure	Result
Character error rate on 450 LLM-written chat messages (spaces removed; references LLM-written, unreviewed). Development messages: they were used to choose decoding settings and this release's training seed, so the figure is optimistic.	8.95%
Exact messages, same set	36.9%
Fresh word test: 148 less common words typed by one native annotator (179 spellings), each converted alone; mean over words of the share of each word's spellings written exactly	58.2%
Recheck on vs off, earlier word model, 165 messages scored once before any release choice	CER 10.56% → 9.37%; blinded preference 17 to 7 (5 same, 11 both bad)

Details, intervals and the release decision are in technical-report.pdf.

Training data

Word pairs (romanised spelling, Khmer word): the public Chhunneng romanisation pairs (MIT), name and loanword pairs mined from Khmer Wikipedia (CC BY-SA 4.0, Khmer Wikipedia contributors) and FineWeb-2 (ODC-By 1.0), romanised spellings of common Khmer words generated with Gemini and GPT and filtered by fixed rules, and the author's own typed and checked spellings. A side task during training used Google's Khmer pronunciation lexicon (CC BY 4.0). khmer_bigram.npz holds word and word-pair statistics derived from FineWeb-2 (no text). Sources, versions and notices: NOTICE.md.

Excluded: spellings generated only by Claude (Anthropic) and LLM drafts whose generating model was not recorded. Spellings that Gemini or GPT also produced are kept. Claude-written messages were used to choose three decoding settings and to evaluate, not to train. This exclusion is the author's conservative choice, not a stated legal requirement; averaged over three retrainings it cost little on human-typed words and about one point of error rate on the LLM-written messages (the released retraining is not worse there).

Limitations

• One annotator. All human judgments and typed test words come from one native speaker.
• No real-chat test set. Message results use LLM-written messages and LLM-written references.
• Context errors remain, e.g. ព្រុស "bark" vs the more common ព្រោះ "because"; chat abbreviations that stand for whole words or names (fb, BKK); English words are not always written out in Khmer (ok).
• Non-words can still appear: about 2% of output words on the development messages are outside the Khmer word list and the word-pair vocabulary (being outside a list does not prove a word is wrong).
• Exact match undercounts acceptable spellings.

Licence

Model files (translit.npz, reverse.npz, khmer_bigram.npz): the author releases their rights in these files under CC BY-SA 4.0, a conservative choice given Wikipedia-derived training data. If you share them or an adaptation, give attribution (this repository and NOTICE.md), indicate changes, and share adaptations under the same licence. This licence covers the author's rights only; the upstream sources keep their own terms. Code (the wheel): MIT.

Files

File	What
translit.npz, reverse.npz, khmer_bigram.npz	the three model files (load the folder with Converter.from_dir)
manifest.json	sha256 of each model file and the decoding settings
roman2khmer-0.1.0-py3-none-any.whl	the Python package (MIT)
technical-report.pdf	the technical report
NOTICE.md, LICENSE	data sources and licences; CC BY-SA 4.0 text reference