The Design Space of Tri-Modal Masked Diffusion Models
Discrete diffusion models have emerged as strong alternatives to autoregressive language models, with recent work initializing and fine-tuning a base unimodal model for bimodal generation. Diverging from previous approaches, we introduce the first tri-modal masked diffusion model pretrained from scratch on text, image-text, and audio-text data. We systematically analyze multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects, and we provide optimized inference sampling defaults. Our batch-size analysis yields a novel stochastic differential equation (SDE)-based reparameterization that eliminates the need for tuning the optimal batch size as reported in recent work. This reparameterization decouples the physical batch size, often chosen based on compute constraints (GPU saturation, FLOP efficiency, wall-clock time), from the logical batch size, chosen to balance gradient variance during stochastic optimization. Finally, we pretrain a preliminary 3B-parameter tri-modal model on 6.4T tokens, demonstrating the capabilities of a unified design and achieving strong results in text generation, text-to-image tasks, and text-to-speech tasks. Our work represents the largest-scale systematic open study of multimodal discrete diffusion models conducted to date, providing insights into scaling behaviors across multiple modalities.
699fbf82ebfce7fbcca91ba3
Louis Bethune
699fbf82ebfce7fbcca91ba4
Victor Turrisi
699fbf82ebfce7fbcca91ba5
Bruno Kacper Mlodozeniec
699fbf82ebfce7fbcca91ba6
Pau Rodriguez Lopez
699fbf82ebfce7fbcca91ba7
Lokesh Boominathan
699fbf82ebfce7fbcca91ba8
Nikhil Bhendawade
699fbf82ebfce7fbcca91ba9
Amitis Shidani
699fbf82ebfce7fbcca91baa
Joris Pelemans
699fbf82ebfce7fbcca91bab
Theo X. Olausson
699fbf82ebfce7fbcca91bac
Devon Hjelm
699fbf82ebfce7fbcca91bad
Paul Dixon
699fbf82ebfce7fbcca91bae
Joao Monteiro
699fbf82ebfce7fbcca91baf
Pierre Ablin
699fbf82ebfce7fbcca91bb0
Vishnu Banna
699fbf82ebfce7fbcca91bb1
Arno Blaas
699fbf82ebfce7fbcca91bb2
Nick Henderson
699fbf82ebfce7fbcca91bb3
Kari Noriy
699fbf82ebfce7fbcca91bb4
Dan Busbridge
699fbf82ebfce7fbcca91bb5
Josh Susskind
699fbf82ebfce7fbcca91bb6
Marco Cuturi
699fbf82ebfce7fbcca91bb7
Irina Belousova
699fbf82ebfce7fbcca91bb8
Luca Zappella
699fbf82ebfce7fbcca91bb9
Russ Webb
699fbf82ebfce7fbcca91bba
Jason Ramapuram
claimed_verified
2026-03-03T10:43:11.214Z
64b88ad0d2176850e0b79d9c
/avatars/32edd35db291be19bf09d4aaba58fdbd.svg
Jason Ramapuram
svd0
user
svd0
The Design Space of Tri-Modal Masked Diffusion Models
Discrete diffusion models have emerged as strong alternatives to autoregressive language models, with recent work initializing and fine-tuning a base unimodal model for bimodal generation. Diverging from previous approaches, we introduce the first tri-modal masked diffusion model pretrained from scratch on text, image-text, and audio-text data. We systematically analyze multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects, and we provide optimized inference sampling defaults. Our batch-size analysis yields a novel stochastic differential equation (SDE)-based reparameterization that eliminates the need for tuning the optimal batch size as reported in recent work. This reparameterization decouples the physical batch size, often chosen based on compute constraints (GPU saturation, FLOP efficiency, wall-clock time), from the logical batch size, chosen to balance gradient variance during stochastic optimization. Finally, we pretrain a preliminary 3B-parameter tri-modal model on 6.4T tokens, demonstrating the capabilities of a unified design and achieving strong results in text generation, text-to-image tasks, and text-to-speech tasks. Our work represents the largest-scale systematic open study of multimodal discrete diffusion models conducted to date, providing insights into scaling behaviors across multiple modalities.
A large-scale study of tri-modal discrete diffusion models demonstrates improved performance across text, image, and speech generation tasks through systematic analysis of scaling laws and optimized inference methods.
discrete diffusion models autoregressive language models unimodal model bimodal generation tri-modal masked diffusion model multimodal scaling laws modality mixing ratios noise schedules stochastic differential equation SDE-based reparameterization batch-size effects gradient variance stochastic optimization text-to-image text-to-speech