Residual Context Diffusion Language Models
Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to purely autoregressive language models because they can decode multiple tokens in parallel. However, state-of-the-art block-wise dLLMs rely on a "remasking" mechanism that decodes only the most confident tokens and discards the rest, effectively wasting computation. We demonstrate that recycling computation from the discarded tokens is beneficial, as these tokens retain contextual information useful for subsequent decoding iterations. In light of this, we propose Residual Context Diffusion (RCD), a module that converts these discarded token representations into contextual residuals and injects them back for the next denoising step. RCD uses a decoupled two-stage training pipeline to bypass the memory bottlenecks associated with backpropagation. We validate our method on both long CoT reasoning (SDAR) and short CoT instruction following (LLaDA) models. We demonstrate that a standard dLLM can be efficiently converted to the RCD paradigm with merely ~1 billion tokens. RCD consistently improves frontier dLLMs by 5-10 points in accuracy with minimal extra computation overhead across a wide range of benchmarks. Notably, on the most challenging AIME tasks, RCD nearly doubles baseline accuracy and attains up to 4-5x fewer denoising steps at equivalent accuracy levels.
698018846676f93322706587
Yuezhou Hu
claimed_verified
2026-02-02T16:53:04.233Z
67dc66fe55c24fc4f981a4ab
/avatars/7bd900ade802d99db7c562ad6c2f6661.svg
Yuezhou Hu
yuezhouhu
user
yuezhouhu
698018846676f93322706588
Harman Singh
698018846676f93322706589
Monishwaran Maheswaran
698018846676f9332270658a
Haocheng Xi
698018846676f9332270658b
Coleman Hooper
698018846676f9332270658c
Jintao Zhang
claimed_verified
2026-02-05T10:57:09.830Z
66c0a08bac74db25de8427ec
https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg
Jintao Zhang
jt-zhang
user
jt-zhang
698018846676f9332270658d
Aditya Tomar
claimed_verified
2026-02-05T10:57:07.939Z
65fe49de871b36bf84c0ba05
/avatars/0fe082518fb9ea40e23414c83ee5043e.svg
Aditya Tomar
adityastomar
user
adityastomar
698018846676f9332270658e
Michael W. Mahoney
698018846676f9332270658f
Sewon Min
698018846676f93322706590
Mehrdad Farajtabar
698018846676f93322706591
Kurt Keutzer
698018846676f93322706592
Amir Gholami
698018846676f93322706593
Chenfeng Xu
Residual Context Diffusion Language Models
Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to purely autoregressive language models because they can decode multiple tokens in parallel. However, state-of-the-art block-wise dLLMs rely on a "remasking" mechanism that decodes only the most confident tokens and discards the rest, effectively wasting computation. We demonstrate that recycling computation from the discarded tokens is beneficial, as these tokens retain contextual information useful for subsequent decoding iterations. In light of this, we propose Residual Context Diffusion (RCD), a module that converts these discarded token representations into contextual residuals and injects them back for the next denoising step. RCD uses a decoupled two-stage training pipeline to bypass the memory bottlenecks associated with backpropagation. We validate our method on both long CoT reasoning (SDAR) and short CoT instruction following (LLaDA) models. We demonstrate that a standard dLLM can be efficiently converted to the RCD paradigm with merely ~1 billion tokens. RCD consistently improves frontier dLLMs by 5-10 points in accuracy with minimal extra computation overhead across a wide range of benchmarks. Notably, on the most challenging AIME tasks, RCD nearly doubles baseline accuracy and attains up to 4-5x fewer denoising steps at equivalent accuracy levels.
Residual Context Diffusion (RCD) enhances diffusion large language models by recycling discarded token information through contextual residuals, improving accuracy with minimal computational overhead.
diffusion large language models autoregressive language models remasking mechanism token representations contextual residuals denoising steps decoupled two-stage training backpropagation long CoT reasoning short CoT instruction following AIME tasks