MemDLM: Memory-Enhanced DLM Training
Diffusion Language Models (DLMs) offer attractive advantages over Auto-Regressive (AR) models, such as full-attention parallel decoding and flexible generation. However, they suffer from a notable train-inference mismatch: DLMs are trained with a static, single-step masked prediction objective, but deployed through a multi-step progressive denoising trajectory. We propose MemDLM (Memory-Enhanced DLM), which narrows this gap by embedding a simulated denoising process into training via Bi-level Optimization. An inner loop updates a set of fast weights, forming a Parametric Memory that captures the local trajectory experience of each sample, while an outer loop updates the base model conditioned on this memory. By offloading memorization pressure from token representations to parameters, MemDLM yields faster convergence and lower training loss. Moreover, the inner loop can be re-enabled at inference time as an adaptation step, yielding additional gains on long-context understanding. We find that, when activated at inference time, this Parametric Memory acts as an emergent in-weight retrieval mechanism, helping MemDLM further reduce token-level attention bottlenecks on challenging Needle-in-a-Haystack retrieval tasks. Code: https://github.com/JarvisPei/MemDLM.
69c205c4c105f45cb0481ed4
Zehua Pei
claimed_verified
2026-03-24T09:21:44.618Z
6527c063e86758eb6ca800a1
/avatars/9091be87eea518209c1de9eebfa663c0.svg
JarvisPei
JarvisPei
user
JarvisPei
69c205c4c105f45cb0481ed5
Hui-Ling Zhen
69c205c4c105f45cb0481ed6
Weizhe Lin
69c205c4c105f45cb0481ed7
Sinno Jialin Pan
69c205c4c105f45cb0481ed8
Yunhe Wang
69c205c4c105f45cb0481ed9
Mingxuan Yuan
69c205c4c105f45cb0481eda
Bei Yu
MemDLM: Memory-Enhanced DLM Training
Diffusion Language Models (DLMs) offer attractive advantages over Auto-Regressive (AR) models, such as full-attention parallel decoding and flexible generation. However, they suffer from a notable train-inference mismatch: DLMs are trained with a static, single-step masked prediction objective, but deployed through a multi-step progressive denoising trajectory. We propose MemDLM (Memory-Enhanced DLM), which narrows this gap by embedding a simulated denoising process into training via Bi-level Optimization. An inner loop updates a set of fast weights, forming a Parametric Memory that captures the local trajectory experience of each sample, while an outer loop updates the base model conditioned on this memory. By offloading memorization pressure from token representations to parameters, MemDLM yields faster convergence and lower training loss. Moreover, the inner loop can be re-enabled at inference time as an adaptation step, yielding additional gains on long-context understanding. We find that, when activated at inference time, this Parametric Memory acts as an emergent in-weight retrieval mechanism, helping MemDLM further reduce token-level attention bottlenecks on challenging Needle-in-a-Haystack retrieval tasks. Code: https://github.com/JarvisPei/MemDLM.
MemDLM addresses the train-inference mismatch in diffusion language models by incorporating a bi-level optimization framework with parametric memory that enhances both training efficiency and inference performance through adaptive denoising processes.
Diffusion Language Models Auto-Regressive models train-inference mismatch masked prediction objective multi-step progressive denoising Bi-level Optimization Parametric Memory token-level attention bottlenecks Needle-in-a-Haystack retrieval