Large Language Diffusion Models
Autoregressive models (ARMs) are widely regarded as the cornerstone of large
language models (LLMs). We challenge this notion by introducing LLaDA, a
diffusion model trained from scratch under the pre-training and supervised
fine-tuning (SFT) paradigm. LLaDA models distributions through a forward data
masking process and a reverse process, parameterized by a vanilla Transformer
to predict masked tokens. By optimizing a likelihood bound, it provides a
principled generative approach for probabilistic inference. Across extensive
benchmarks, LLaDA demonstrates strong scalability, outperforming our
self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong
LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive
instruction-following abilities in case studies such as multi-turn dialogue.
Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal
poem completion task. Our findings establish diffusion models as a viable and
promising alternative to ARMs, challenging the assumption that key LLM
capabilities discussed above are inherently tied to ARMs.
67b2c31125f77e5fc242f4f8
Shen Nie
admin_assigned
2025-02-28T12:48:04.969Z
640dd9a5fdeaae13908208a7
/avatars/61f8f1d5f6ef1c4af1f47285e9cc0217.svg
nieshen
nieshen
user
nieshen
67b2c31125f77e5fc242f4f9
Fengqi Zhu
67b2c31125f77e5fc242f4fa
Zebin You
admin_assigned
2025-02-28T12:47:21.498Z
624f909eac5dd186b01ac3f5
/avatars/0aafdb1cbb492fda52a0303031cc6c14.svg
Zebin You
yyyou
user
yyyou
67b2c31125f77e5fc242f4fb
Xiaolu Zhang
admin_assigned
2025-02-28T12:47:27.956Z
67513d6d3b8586521cda5d76
/avatars/0f95cc5c23a0a1da289aa785bd33b616.svg
Xiaolu  Zhang
xiaolu0714
user
xiaolu0714
67b2c31125f77e5fc242f4fc
Jingyang Ou
admin_assigned
2025-02-19T15:51:46.024Z
656949b71d7c2ca7b7aae5f2
/avatars/e7b23e260eb348cc26b849aaa601a503.svg
Jingyang Ou
JingyangOu
user
JingyangOu
67b2c31125f77e5fc242f4fd
Jun Hu
67b2c31125f77e5fc242f4fe
Jun Zhou
67b2c31125f77e5fc242f4ff
Yankai Lin
admin_assigned
2025-02-19T15:51:39.766Z
657a651e1433ea7d44de6397
/avatars/ccfc76f94595a38ff4a80f77c911eabf.svg
Yankai Lin
lyk423
user
lyk423
67b2c31125f77e5fc242f500
Ji-Rong Wen
admin_assigned
2025-02-19T15:51:33.428Z
64b8c89052b7353d8c6a1013
/avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
Ji-Rong Wen
jrwen
user
jrwen
67b2c31125f77e5fc242f501
Chongxuan Li
admin_assigned
2025-02-19T15:51:26.668Z
64c07b488e2612254361153b
/avatars/ade0f783cc4c2d3e73f402637f595471.svg
chongxuan li
zhenxuan00
user
zhenxuan00
Large Language Diffusion Models
Autoregressive models (ARMs) are widely regarded as the cornerstone of large
language models (LLMs). We challenge this notion by introducing LLaDA, a
diffusion model trained from scratch under the pre-training and supervised
fine-tuning (SFT) paradigm. LLaDA models distributions through a forward data
masking process and a reverse process, parameterized by a vanilla Transformer
to predict masked tokens. By optimizing a likelihood bound, it provides a
principled generative approach for probabilistic inference. Across extensive
benchmarks, LLaDA demonstrates strong scalability, outperforming our
self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong
LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive
instruction-following abilities in case studies such as multi-turn dialogue.
Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal
poem completion task. Our findings establish diffusion models as a viable and
promising alternative to ARMs, challenging the assumption that key LLM
capabilities discussed above are inherently tied to ARMs.
LLaDA, a diffusion model trained from scratch, outperforms autoregressive models in benchmarks and demonstrates strong instruction-following capabilities, challenging the dominance of ARMs in LLMs.
autoregressive models LLaDA diffusion model pre-training supervised fine-tuning vanilla Transformer likelihood bound probabilistic inference in-context learning instruction-following reversal curse LLaMA3 GPT-4o reversal poem completion