ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining,
  Finetuning, and Decoding the Curse of Multilinguality
Scaling laws research has focused overwhelmingly on English -- yet the most
prominent AI models explicitly serve billions of international users. In this
work, we undertake the largest multilingual scaling laws study to date,
totaling 774 multilingual training experiments, spanning 10M-8B model
parameters, 400+ training languages and 48 evaluation languages. We introduce
the Adaptive Transfer Scaling Law (ATLAS) for both monolingual and multilingual
pretraining, which outperforms existing scaling laws' out-of-sample
generalization often by more than 0.3 R^2. Our analyses of the experiments shed
light on multilingual learning dynamics, transfer properties between languages,
and the curse of multilinguality. First, we derive a cross-lingual transfer
matrix, empirically measuring mutual benefit scores between 38 x 38=1444
language pairs. Second, we derive a language-agnostic scaling law that reveals
how to optimally scale model size and data when adding languages without
sacrificing performance. Third, we identify the computational crossover points
for when to pretrain from scratch versus finetune from multilingual
checkpoints. We hope these findings provide the scientific foundation for
democratizing scaling laws across languages, and enable practitioners to
efficiently scale models -- beyond English-first AI.
69018d27646208eac0d1f4d1
Shayne Longpre
69018d27646208eac0d1f4d2
Sneha Kudugunta
69018d27646208eac0d1f4d3
Niklas Muennighoff
69018d27646208eac0d1f4d4
I-Hung Hsu
69018d27646208eac0d1f4d5
Isaac Caswell
69018d27646208eac0d1f4d6
Alex Pentland
69018d27646208eac0d1f4d7
Sercan Arik
69018d27646208eac0d1f4d8
Chen-Yu Lee
69018d27646208eac0d1f4d9
Sayna Ebrahimi
ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining,
  Finetuning, and Decoding the Curse of Multilinguality
Scaling laws research has focused overwhelmingly on English -- yet the most
prominent AI models explicitly serve billions of international users. In this
work, we undertake the largest multilingual scaling laws study to date,
totaling 774 multilingual training experiments, spanning 10M-8B model
parameters, 400+ training languages and 48 evaluation languages. We introduce
the Adaptive Transfer Scaling Law (ATLAS) for both monolingual and multilingual
pretraining, which outperforms existing scaling laws' out-of-sample
generalization often by more than 0.3 R^2. Our analyses of the experiments shed
light on multilingual learning dynamics, transfer properties between languages,
and the curse of multilinguality. First, we derive a cross-lingual transfer
matrix, empirically measuring mutual benefit scores between 38 x 38=1444
language pairs. Second, we derive a language-agnostic scaling law that reveals
how to optimally scale model size and data when adding languages without
sacrificing performance. Third, we identify the computational crossover points
for when to pretrain from scratch versus finetune from multilingual
checkpoints. We hope these findings provide the scientific foundation for
democratizing scaling laws across languages, and enable practitioners to
efficiently scale models -- beyond English-first AI.
The study introduces ATLAS, a multilingual scaling law that improves out-of-sample generalization and provides insights into cross-lingual transfer, optimal scaling, and computational crossover points for model training.
Adaptive Transfer Scaling Law ATLAS multilingual scaling laws cross-lingual transfer matrix language-agnostic scaling law computational crossover points pretraining finetuning multilingual checkpoints