Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models
Masked Discrete Diffusion Models (MDMs) have achieved strong performance across a wide range of multimodal tasks, including image understanding, generation, and editing. However, their inference speed remains suboptimal due to the need to repeatedly process redundant masked tokens at every sampling step. In this work, we propose Sparse-LaViDa, a novel modeling framework that dynamically truncates unnecessary masked tokens at each inference step to accelerate MDM sampling. To preserve generation quality, we introduce specialized register tokens that serve as compact representations for the truncated tokens. Furthermore, to ensure consistency between training and inference, we design a specialized attention mask that faithfully matches the truncated sampling procedure during training. Built upon the state-of-the-art unified MDM LaViDa-O, Sparse-LaViDa achieves up to a 2x speedup across diverse tasks including text-to-image generation, image editing, and mathematical reasoning, while maintaining generation quality.
694238f45d5b2dc1052748e8
Shufan Li
694238f45d5b2dc1052748e9
Jiuxiang Gu
694238f45d5b2dc1052748ea
Kangning Liu
694238f45d5b2dc1052748eb
Zhe Lin
694238f45d5b2dc1052748ec
Zijun Wei
694238f45d5b2dc1052748ed
Aditya Grover
694238f45d5b2dc1052748ee
Jason Kuen
Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models
Masked Discrete Diffusion Models (MDMs) have achieved strong performance across a wide range of multimodal tasks, including image understanding, generation, and editing. However, their inference speed remains suboptimal due to the need to repeatedly process redundant masked tokens at every sampling step. In this work, we propose Sparse-LaViDa, a novel modeling framework that dynamically truncates unnecessary masked tokens at each inference step to accelerate MDM sampling. To preserve generation quality, we introduce specialized register tokens that serve as compact representations for the truncated tokens. Furthermore, to ensure consistency between training and inference, we design a specialized attention mask that faithfully matches the truncated sampling procedure during training. Built upon the state-of-the-art unified MDM LaViDa-O, Sparse-LaViDa achieves up to a 2x speedup across diverse tasks including text-to-image generation, image editing, and mathematical reasoning, while maintaining generation quality.
Sparse-LaViDa accelerates Masked Discrete Diffusion Models by dynamically truncating masked tokens during inference, maintaining quality and achieving up to a 2x speedup across various tasks.
Masked Discrete Diffusion Models MDMs Sparse-LaViDa register tokens attention mask LaViDa-O text-to-image generation image editing mathematical reasoning