Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed
Diffusion language models (dLMs) have emerged as a promising paradigm that enables parallel, non-autoregressive generation, but their learning efficiency lags behind that of autoregressive (AR) language models when trained from scratch. To this end, we study AR-to-dLM conversion to transform pretrained AR models into efficient dLMs that excel in speed while preserving AR models' task accuracy. We achieve this by identifying limitations in the attention patterns and objectives of existing AR-to-dLM methods and then proposing principles and methodologies for more effective AR-to-dLM conversion. Specifically, we first systematically compare different attention patterns and find that maintaining pretrained AR weight distributions is critical for effective AR-to-dLM conversion. As such, we introduce a continuous pretraining scheme with a block-wise attention pattern, which remains causal across blocks while enabling bidirectional modeling within each block. We find that this approach can better preserve pretrained AR models' weight distributions than fully bidirectional modeling, in addition to its known benefit of enabling KV caching, and leads to a win-win in accuracy and efficiency. Second, to mitigate the training-test gap in mask token distributions (uniform vs. highly left-to-right), we propose a position-dependent token masking strategy that assigns higher masking probabilities to later tokens during training to better mimic test-time behavior. Leveraging this framework, we conduct extensive studies of dLMs' attention patterns, training dynamics, and other design choices, providing actionable insights into scalable AR-to-dLM conversion. These studies lead to the Efficient-DLM family, which outperforms state-of-the-art AR models and dLMs, e.g., our Efficient-DLM 8B achieves +5.4%/+2.7% higher accuracy with 4.5x/2.7x higher throughput compared to Dream 7B and Qwen3 4B, respectively.
694220d65d5b2dc105274849
Yonggan Fu
694220d65d5b2dc10527484a
Lexington Whalen
694220d65d5b2dc10527484b
Zhifan Ye
694220d65d5b2dc10527484c
Xin Dong
claimed_verified
2026-01-12T10:37:14.763Z
6092aaac134b77647f624bce
https://cdn-avatars.huggingface.co/v1/production/uploads/6092aaac134b77647f624bce/HGc1cJo7iOFJMKpZWrYJ0.jpeg
Xin Dong
SimonX
user
SimonX
694220d65d5b2dc10527484d
Shizhe Diao
694220d65d5b2dc10527484e
Jingyu Liu
claimed_verified
2026-03-03T10:45:10.368Z
6278fbf13503c4f835c7ff70
/avatars/e19960973132ccb0fc0b60a76328b5d8.svg
Jingyu Liu
Jingyu6
user
Jingyu6
694220d65d5b2dc10527484f
Chengyue Wu
694220d65d5b2dc105274850
Hao Zhang
694220d65d5b2dc105274851
Enze Xie
694220d65d5b2dc105274852
Song Han
694220d65d5b2dc105274853
Maksim Khadkevich
694220d65d5b2dc105274854
Jan Kautz
694220d65d5b2dc105274855
Yingyan Celine Lin
694220d65d5b2dc105274856
Pavlo Molchanov
claimed_verified
2026-01-22T08:47:48.044Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
pmolchanov
user
pmolchanov
Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed
Diffusion language models (dLMs) have emerged as a promising paradigm that enables parallel, non-autoregressive generation, but their learning efficiency lags behind that of autoregressive (AR) language models when trained from scratch. To this end, we study AR-to-dLM conversion to transform pretrained AR models into efficient dLMs that excel in speed while preserving AR models' task accuracy. We achieve this by identifying limitations in the attention patterns and objectives of existing AR-to-dLM methods and then proposing principles and methodologies for more effective AR-to-dLM conversion. Specifically, we first systematically compare different attention patterns and find that maintaining pretrained AR weight distributions is critical for effective AR-to-dLM conversion. As such, we introduce a continuous pretraining scheme with a block-wise attention pattern, which remains causal across blocks while enabling bidirectional modeling within each block. We find that this approach can better preserve pretrained AR models' weight distributions than fully bidirectional modeling, in addition to its known benefit of enabling KV caching, and leads to a win-win in accuracy and efficiency. Second, to mitigate the training-test gap in mask token distributions (uniform vs. highly left-to-right), we propose a position-dependent token masking strategy that assigns higher masking probabilities to later tokens during training to better mimic test-time behavior. Leveraging this framework, we conduct extensive studies of dLMs' attention patterns, training dynamics, and other design choices, providing actionable insights into scalable AR-to-dLM conversion. These studies lead to the Efficient-DLM family, which outperforms state-of-the-art AR models and dLMs, e.g., our Efficient-DLM 8B achieves +5.4%/+2.7% higher accuracy with 4.5x/2.7x higher throughput compared to Dream 7B and Qwen3 4B, respectively.
AR-to-dLM conversion enhances diffusion language models' efficiency and speed while maintaining task accuracy through refined attention patterns and token masking strategies.
diffusion language models autoregressive language models attention patterns pretrained AR models continuous pretraining scheme block-wise attention KV caching position-dependent token masking Efficient-DLM family Dream 7B Qwen3 4B