Where Does Translation Emerge? 48 Arabic Models from 0.5M to 50M Parameters

TL;DR — I pretrained two ladders of Arabic language models from scratch, 0.5M to 50M parameters, and finetuned all of them into bidirectional translators across five language pairs. Under a fixed 5B-token budget, translation is absent at 1M parameters and working at 5M. Below that threshold the models don't produce noise — they produce fluent, grammatical dialect that has nothing to do with the input. Fluency arrives before meaning.

All 48 models are public and Apache-2.0: Emhotob — Arabic Translation Scaling Study

The question

Most scaling work measures how smoothly quality improves. I wanted the opposite end: the point where a capability is simply not there, and the point just after where it is. For translation between Modern Standard Arabic and its dialects, where is that line?

To answer it you need models small enough to fail. So I trained down to half a million parameters.

Two ladders, and why the token budget matters

I pretrained two ladders from scratch on Arabic web text (kaust-generative-ai/fineweb-edu-ar). They differ in one policy, and it turns out to be the most important variable in the whole study:

v1 holds tokens-per-parameter roughly constant at ~100 — every model gets a budget proportional to its size. v2 holds the total constant at ~5B tokens for every model, from the 0.5M up to the 25M.

That makes the v2 ladder the cleaner emergence experiment: the data is identical at every rung, so anything that changes is attributable to parameter count alone.

Every model then gets an identical finetuning recipe — ChatML, prompt-masked cross-entropy (loss only on the assistant turn), 3 epochs, effective batch 64, LR 3e-4 cosine, bf16, max length 1024, best checkpoint on eval_loss. Evaluation is deterministic greedy decoding over 3,000 held-out pairs, seed 42. No per-model tuning anywhere.

Emergence, at a fixed 5B tokens

MSA ↔ Egyptian Arabic, BLEU, v2 ladder — same data at every rung:

The jump happens between 1M and 5M. Below it the score is indistinguishable from zero; above it the models translate. More data does not rescue the small end — the 0.5M model saw 5B tokens, roughly 100× what the v1 0.5M model saw, and still scores 0.00. Below the threshold, the bottleneck is capacity, not data.

Fluency arrives before meaning

The interesting part isn't that small models fail. It's how.

Here is a sub-threshold model asked to translate I want to go to the market to buy some fresh vegetables:

أنا عايز أقولك على حاجة
("I want to tell you something")

And We are planning to visit the pyramids next weekend:

احنا بنعمل كل حاجة
("We do everything")

These are fluent, grammatical, idiomatic Egyptian Arabic. The register is right. The dialect is right. The output is simply unrelated to the input.

The model has learned the shape of its target language, and learned to produce it on demand, without learning that the output should be conditioned on the source sentence. Language modelling is solved well before translation is.

This is worth dwelling on, because it's a failure mode that survives a casual eyeball check. A model that emits confident, well-formed text in the target language looks like it works. Any evaluation that doesn't compare against a reference will not catch it.

Full results

BLEU, greedy decoding, 3,000 held-out pairs, both directions (a→b / b→a):

Three remaining 25M v2 pairs are still training. The 50M rung exists only in v1 (20B tokens).

English↔MSA scores far above every dialect pair at the same size. MSA is standardised; the dialects are not, so a single reference covers a much larger share of valid outputs. Cross-pair BLEU comparisons mostly measure orthographic variance, not translation quality.

More tokens or more parameters?

Because v2 gave every model 5B tokens and v1 scaled the budget with size, the v2-over-v1 gap at each rung is a direct read on what extra data buys — and it shrinks exactly as the two budgets converge:

At 5M parameters, a 10× token budget is worth about 7 BLEU. At 25M, doubling it is worth 0.2. The v2 ladder isn't a better recipe — it's the same recipe with a different data allocation, and the return on that allocation falls off sharply once a model is past the emergence threshold.

In-domain scores flatter you

Every number above is in-domain: a held-out split of the same corpus the models trained on. I also scored the English→Egyptian models on an independent benchmark — 319 pairs written by a different annotator, with different orthographic conventions.

Roughly half, across the board. Both numbers are honest; they answer different questions.

Note the 25M row: v2 wins in-domain (19.72 vs 19.01) and loses out-of-domain (8.24 vs 9.91). It's the only rung where the ladders cross, and it's also the rung where v2's token advantage is smallest. A plausible reading is that at a 2× budget the extra data is going into fitting the training distribution more tightly rather than generalising — but with 319 evaluation rows, a 1.7 BLEU gap is close to noise. I'd call it a lead worth chasing, not a result.

It would have been invisible with only the in-domain axis, where v2 wins at every rung.

BLEU is the wrong headline for dialectal Arabic

Egyptian Arabic has no standardised orthography, and the benchmark has one reference per sentence. So correct translations get penalised for word choice:

Both are good Egyptian. Only one matches. Character-level chrF degrades far more gracefully here and tracks perceived quality better; I'd report it alongside BLEU for any dialectal work, and lead with it when the target has no orthographic standard.

Reproducing this

Everything is public and Apache-2.0 — 11 pretrained bases and 37 translation finetunes:

The full collection

Each card carries its own eval numbers, the exact prompt format, and an inference snippet. Two implementation notes that will save you time:

• These tokenizers have no chat template. ChatML is constructed manually, and inference must prepend BOS — training used add_bos=True, so skipping it puts the model off-distribution and scores collapse.
• Greedy decoding throughout. An earlier sweep with repetition_penalty=1.2 cost 5–12 BLEU across every model, so plain greedy is the honest default here.

Limitations

Small research models — 0.5M to 50M parameters. They are for studying scaling behaviour, not for production translation. Everything below ~5M is degenerate by design, and it's published because the failure is the finding. The 50M rung exists only in v1, on a 20B-token budget, so it is not directly comparable to the fixed-budget v2 ladder. Single-reference BLEU on dialects under-credits valid paraphrase throughout.

---

Models: Emhotob collection. Questions and corrections welcome — particularly from anyone who has measured an emergence threshold on a different language pair.