Beyond Language Modeling: An Exploration of Multimodal Pretraining
The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.
69a7ac062833ff1122ff64c4
Shengbang Tong
69a7ac062833ff1122ff64c5
David Fan
69a7ac062833ff1122ff64c6
John Nguyen
69a7ac062833ff1122ff64c7
Ellis Brown
claimed_verified
2026-03-04T09:22:33.268Z
626dc5105f7327906f0b2a4e
https://cdn-avatars.huggingface.co/v1/production/uploads/626dc5105f7327906f0b2a4e/QCSzuwYqsv8ozRnusVb-F.jpeg
Ellis Brown
ellisbrown
user
ellisbrown
69a7ac062833ff1122ff64c8
Gaoyue Zhou
69a7ac062833ff1122ff64c9
Shengyi Qian
claimed_verified
2026-03-05T09:59:31.125Z
6445dc7e219dc6142a6a6861
https://cdn-avatars.huggingface.co/v1/production/uploads/6445dc7e219dc6142a6a6861/Ak_YV8V1NZpkF5AQnJVlJ.png
Shengyi Qian
shengyi-qian
user
shengyi-qian
69a7ac062833ff1122ff64ca
Boyang Zheng
69a7ac062833ff1122ff64cb
Théophane Vallaeys
69a7ac062833ff1122ff64cc
Junlin Han
69a7ac062833ff1122ff64cd
Rob Fergus
69a7ac062833ff1122ff64ce
Naila Murray
69a7ac062833ff1122ff64cf
Marjan Ghazvininejad
69a7ac062833ff1122ff64d0
Mike Lewis
69a7ac062833ff1122ff64d1
Nicolas Ballas
69a7ac062833ff1122ff64d2
Amir Bar
69a7ac062833ff1122ff64d3
Michael Rabbat
69a7ac062833ff1122ff64d4
Jakob Verbeek
69a7ac062833ff1122ff64d5
Luke Zettlemoyer
69a7ac062833ff1122ff64d6
Koustuv Sinha
69a7ac062833ff1122ff64d7
Yann LeCun
69a7ac062833ff1122ff64d8
Saining Xie
Beyond Language Modeling: An Exploration of Multimodal Pretraining
The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.
Controlled multimodal pretraining experiments reveal key insights about unified visual representations, data complementarity, world modeling emergence, and efficient scaling through mixture-of-experts architectures.
multimodal models Transfusion framework next-token prediction diffusion representation Autoencoder visual understanding visual generation world modeling Mixture-of-Experts scaling laws IsoFLOP analysis data complementarity modality specialization