Beyond Hard Masks: Progressive Token Evolution for Diffusion Language Models
Diffusion Language Models (DLMs) offer a promising alternative for language modeling by enabling parallel decoding through iterative refinement. However, most DLMs rely on hard binary masking and discrete token assignments, which hinder the revision of early decisions and underutilize intermediate probabilistic representations. In this paper, we propose EvoToken-DLM, a novel diffusion-based language modeling approach that replaces hard binary masks with evolving soft token distributions. EvoToken-DLM enables a progressive transition from masked states to discrete outputs, supporting revisable decoding. To effectively support this evolution, we introduce continuous trajectory supervision, which aligns training objectives with iterative probabilistic updates. Extensive experiments across multiple benchmarks show that EvoToken-DLM consistently achieves superior performance, outperforming strong diffusion-based and masked DLM baselines. Project webpage: https://aim-uofa.github.io/EvoTokenDLM.
6965e693fc8c4ecc02c7fa43
Linhao Zhong
claimed_verified
2026-01-13T08:23:27.270Z
644238f6fcbe90d73b319ea6
/avatars/06dcede0ea77344de7dded917706bcb2.svg
zhongzero
zhongzero
user
zhongzero
6965e693fc8c4ecc02c7fa44
Linyu Wu
6965e693fc8c4ecc02c7fa45
Bozhen Fang
6965e693fc8c4ecc02c7fa46
Tianjian Feng
6965e693fc8c4ecc02c7fa47
Chenchen Jing
6965e693fc8c4ecc02c7fa48
Wen Wang
6965e693fc8c4ecc02c7fa49
Jiaheng Zhang
6965e693fc8c4ecc02c7fa4a
Hao Chen
6965e693fc8c4ecc02c7fa4b
Chunhua Shen
Beyond Hard Masks: Progressive Token Evolution for Diffusion Language Models
Diffusion Language Models (DLMs) offer a promising alternative for language modeling by enabling parallel decoding through iterative refinement. However, most DLMs rely on hard binary masking and discrete token assignments, which hinder the revision of early decisions and underutilize intermediate probabilistic representations. In this paper, we propose EvoToken-DLM, a novel diffusion-based language modeling approach that replaces hard binary masks with evolving soft token distributions. EvoToken-DLM enables a progressive transition from masked states to discrete outputs, supporting revisable decoding. To effectively support this evolution, we introduce continuous trajectory supervision, which aligns training objectives with iterative probabilistic updates. Extensive experiments across multiple benchmarks show that EvoToken-DLM consistently achieves superior performance, outperforming strong diffusion-based and masked DLM baselines. Project webpage: https://aim-uofa.github.io/EvoTokenDLM.
EvoToken-DLM introduces a diffusion-based language modeling approach that uses soft token distributions and continuous trajectory supervision to enable revisable decoding and outperforms existing baselines.
diffusion language models hard binary masking discrete token assignments soft token distributions iterative refinement progressive transition continuous trajectory supervision revisable decoding