Scaling Behavior of Discrete Diffusion Language Models
Modern LLM pre-training consumes vast amounts of compute and training data, making the scaling behavior, or scaling laws, of different models a key distinguishing factor. Discrete diffusion language models (DLMs) have been proposed as an alternative to autoregressive language models (ALMs). However, their scaling behavior has not yet been fully explored, with prior work suggesting that they require more data and compute to match the performance of ALMs.
  We study the scaling behavior of DLMs on different noise types by smoothly interpolating between masked and uniform diffusion while paying close attention to crucial hyperparameters such as batch size and learning rate. Our experiments reveal that the scaling behavior of DLMs strongly depends on the noise type and is considerably different from ALMs. While all noise types converge to similar loss values in compute-bound scaling, we find that uniform diffusion requires more parameters and less data for compute-efficient training compared to masked diffusion, making them a promising candidate in data-bound settings. We scale our uniform diffusion model up to 10B parameters trained for 10^{22} FLOPs, confirming the predicted scaling behavior and making it the largest publicly known uniform diffusion model to date.
693fce1bf516c69324681321
Dimitri von Rütte
claimed_verified
2025-12-15T13:11:10.509Z
640501091a3babee78e41e99
https://cdn-avatars.huggingface.co/v1/production/uploads/1678365850657-640501091a3babee78e41e99.jpeg
Dimitri von Rütte
dvruette
user
dvruette
693fce1bf516c69324681322
Janis Fluri
693fce1bf516c69324681323
Omead Pooladzandi
693fce1bf516c69324681324
Bernhard Schölkopf
693fce1bf516c69324681325
Thomas Hofmann
693fce1bf516c69324681326
Antonio Orvieto
Scaling Behavior of Discrete Diffusion Language Models
Modern LLM pre-training consumes vast amounts of compute and training data, making the scaling behavior, or scaling laws, of different models a key distinguishing factor. Discrete diffusion language models (DLMs) have been proposed as an alternative to autoregressive language models (ALMs). However, their scaling behavior has not yet been fully explored, with prior work suggesting that they require more data and compute to match the performance of ALMs.
  We study the scaling behavior of DLMs on different noise types by smoothly interpolating between masked and uniform diffusion while paying close attention to crucial hyperparameters such as batch size and learning rate. Our experiments reveal that the scaling behavior of DLMs strongly depends on the noise type and is considerably different from ALMs. While all noise types converge to similar loss values in compute-bound scaling, we find that uniform diffusion requires more parameters and less data for compute-efficient training compared to masked diffusion, making them a promising candidate in data-bound settings. We scale our uniform diffusion model up to 10B parameters trained for 10^{22} FLOPs, confirming the predicted scaling behavior and making it the largest publicly known uniform diffusion model to date.
Discrete diffusion language models exhibit different scaling behaviors depending on noise types, with uniform diffusion requiring fewer parameters and less data for efficient training compared to masked diffusion.
discrete diffusion language models autoregressive language models scaling laws noise types masked diffusion uniform diffusion compute-bound scaling data-bound settings FLOPs parameter-efficient training