Learning Unmasking Policies for Diffusion Language Models
Diffusion (Large) Language Models (dLLMs) now match the downstream performance of their autoregressive counterparts on many tasks, while holding the promise of being more efficient during inference. One particularly successful variant is masked discrete diffusion, in which a buffer filled with special mask tokens is progressively replaced with tokens sampled from the model's vocabulary. Efficiency can be gained by unmasking several tokens in parallel, but doing too many at once risks degrading the generation quality. Thus, one critical design aspect of dLLMs is the sampling procedure that selects, at each step of the diffusion process, which tokens to replace. Indeed, recent work has found that heuristic strategies such as confidence thresholding lead to both higher quality and token throughput compared to random unmasking. However, such heuristics have downsides: they require manual tuning, and we observe that their performance degrades with larger buffer sizes. In this work, we instead propose to train sampling procedures using reinforcement learning. Specifically, we formalize masked diffusion sampling as a Markov decision process in which the dLLM serves as the environment, and propose a lightweight policy architecture based on a single-layer transformer that maps dLLM token confidences to unmasking decisions. Our experiments show that these trained policies match the performance of state-of-the-art heuristics when combined with semi-autoregressive generation, while outperforming them in the full diffusion setting. We also examine the transferability of these policies, finding that they can generalize to new underlying dLLMs and longer sequence lengths. However, we also observe that their performance degrades when applied to out-of-domain data, and that fine-grained tuning of the accuracy-efficiency trade-off can be challenging with our approach.
693a36d474fced5bf9c323f2
Metod Jazbec
admin_assigned
2025-12-11T10:51:43.631Z
6662f75e338c960a6048728c
/avatars/9d01f63fa58fab25d4476f77064f457f.svg
Metod Jazbec
mjazbec
user
mjazbec
693a36d474fced5bf9c323f3
Theo X. Olausson
admin_assigned
2025-12-11T10:51:49.683Z
6489e543b9e9258ba06d73de
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Xjs0N338heFNkRkvHg9hL.jpeg
Theo X. Olausson
theoxo
user
theoxo
693a36d474fced5bf9c323f4
Louis Béthune
admin_assigned
2025-12-11T10:51:55.984Z
662a8d7faa7781f8ea521f16
/avatars/c05f9a7885b89c8818d52b98b52ecc7b.svg
Louis Bethune
AlgueRythme
user
AlgueRythme
693a36d474fced5bf9c323f5
Pierre Ablin
admin_assigned
2025-12-11T10:52:01.637Z
68779eeb7ef0f82c9b75b1a8
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/peQ2TU6FiNPJgNU30UQop.png
Pierre Ablin
pierreablin
user
pierreablin
693a36d474fced5bf9c323f6
Michael Kirchhof
admin_assigned
2025-12-11T10:52:07.133Z
6698ff8866813fa1b885f052
/avatars/49f3d379fa6ee134bb97f3fcd9447def.svg
Michael Kirchhof
michaelkir
user
michaelkir
693a36d474fced5bf9c323f7
Joao Monterio
693a36d474fced5bf9c323f8
Victor Turrisi
693a36d474fced5bf9c323f9
Jason Ramapuram
693a36d474fced5bf9c323fa
Marco Cuturi
admin_assigned
2025-12-11T10:52:24.596Z
65d88dfda5269668b735a39b
/avatars/cf46727be075a51ab894efbe906ee25a.svg
marco cuturi
marcocuturi
user
marcocuturi
Learning Unmasking Policies for Diffusion Language Models
Diffusion (Large) Language Models (dLLMs) now match the downstream performance of their autoregressive counterparts on many tasks, while holding the promise of being more efficient during inference. One particularly successful variant is masked discrete diffusion, in which a buffer filled with special mask tokens is progressively replaced with tokens sampled from the model's vocabulary. Efficiency can be gained by unmasking several tokens in parallel, but doing too many at once risks degrading the generation quality. Thus, one critical design aspect of dLLMs is the sampling procedure that selects, at each step of the diffusion process, which tokens to replace. Indeed, recent work has found that heuristic strategies such as confidence thresholding lead to both higher quality and token throughput compared to random unmasking. However, such heuristics have downsides: they require manual tuning, and we observe that their performance degrades with larger buffer sizes. In this work, we instead propose to train sampling procedures using reinforcement learning. Specifically, we formalize masked diffusion sampling as a Markov decision process in which the dLLM serves as the environment, and propose a lightweight policy architecture based on a single-layer transformer that maps dLLM token confidences to unmasking decisions. Our experiments show that these trained policies match the performance of state-of-the-art heuristics when combined with semi-autoregressive generation, while outperforming them in the full diffusion setting. We also examine the transferability of these policies, finding that they can generalize to new underlying dLLMs and longer sequence lengths. However, we also observe that their performance degrades when applied to out-of-domain data, and that fine-grained tuning of the accuracy-efficiency trade-off can be challenging with our approach.
Reinforcement learning is used to train sampling procedures for masked discrete diffusion language models, improving token throughput and quality compared to heuristic strategies.
diffusion (large) language models masked discrete diffusion buffer mask tokens sampling procedure confidence thresholding reinforcement learning Markov decision process single-layer transformer semi-autoregressive generation full diffusion setting transferability out-of-domain data accuracy-efficiency trade-off