Diffusion Language Models are Super Data Learners
Under strictly controlled pre-training settings, we observe a Crossover: when
unique data is limited, diffusion language models (DLMs) consistently surpass
autoregressive (AR) models by training for more epochs. The crossover shifts
later with more or higher-quality data, earlier with larger models, and
persists across dense and sparse architectures. We attribute the gains to three
compounding factors: (1) any-order modeling, (2) super-dense compute from
iterative bidirectional denoising, and (3) built-in Monte Carlo augmentation;
input or parameter noise improves AR under data constraint but cannot close the
gap. At scale, a 1.7B DLM trained with a ~1.5T-token compute budget on 10B
unique Python tokens overtakes an AR coder trained with strictly matched
settings. In addition, a 1B-parameter DLM achieves > 56% accuracy on HellaSwag
and > 33% on MMLU using only 1B tokens, without any special tricks, just by
repeating standard pre-training data. We also show that rising validation
cross-entropy does not imply degraded downstream performance in this regime.
690c1cdc60494e4fa76756a9
Jinjie Ni
claimed_verified
2025-11-06T11:47:00.935Z
62a7362fd1e7a011fd4e31a7
https://cdn-avatars.huggingface.co/v1/production/uploads/62a7362fd1e7a011fd4e31a7/ZY_mwH-sI0o05SHpLqwc7.png
Jinjie Ni
jinjieni
user
jinjieni
690c1cdc60494e4fa76756aa
Qian Liu
claimed_verified
2025-11-07T10:31:23.866Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
SivilTaram
user
SivilTaram
690c1cdc60494e4fa76756ab
Longxu Dou
claimed_verified
2025-11-18T14:09:50.728Z
6214e4ee1e35c843d42d1f88
https://cdn-avatars.huggingface.co/v1/production/uploads/6214e4ee1e35c843d42d1f88/fj-9wuIdPhvogh3BrcXTB.jpeg
Longxu Dou
dreamerdeo
user
dreamerdeo
690c1cdc60494e4fa76756ac
Chao Du
690c1cdc60494e4fa76756ad
Zili Wang
690c1cdc60494e4fa76756ae
Hang Yan
690c1cdc60494e4fa76756af
Tianyu Pang
690c1cdc60494e4fa76756b0
Michael Qizhe Shieh
Diffusion Language Models are Super Data Learners
Under strictly controlled pre-training settings, we observe a Crossover: when
unique data is limited, diffusion language models (DLMs) consistently surpass
autoregressive (AR) models by training for more epochs. The crossover shifts
later with more or higher-quality data, earlier with larger models, and
persists across dense and sparse architectures. We attribute the gains to three
compounding factors: (1) any-order modeling, (2) super-dense compute from
iterative bidirectional denoising, and (3) built-in Monte Carlo augmentation;
input or parameter noise improves AR under data constraint but cannot close the
gap. At scale, a 1.7B DLM trained with a ~1.5T-token compute budget on 10B
unique Python tokens overtakes an AR coder trained with strictly matched
settings. In addition, a 1B-parameter DLM achieves > 56% accuracy on HellaSwag
and > 33% on MMLU using only 1B tokens, without any special tricks, just by
repeating standard pre-training data. We also show that rising validation
cross-entropy does not imply degraded downstream performance in this regime.
Diffusion language models outperform autoregressive models in low-data settings due to any-order modeling, iterative bidirectional denoising, and Monte Carlo augmentation, and maintain advantages even at scale.
diffusion language models autoregressive models any-order modeling iterative bidirectional denoising Monte Carlo augmentation HellaSwag MMLU