Exploring MLLM-Diffusion Information Transfer with MetaCanvas
Multimodal learning has rapidly advanced visual understanding, largely via multimodal large language models (MLLMs) that use powerful LLMs as cognitive cores. In visual generation, however, these powerful core models are typically reduced to global text encoders for diffusion models, leaving most of their reasoning and planning ability unused. This creates a gap: current multimodal LLMs can parse complex layouts, attributes, and knowledge-intensive scenes, yet struggle to generate images or videos with equally precise and structured control. We propose MetaCanvas, a lightweight framework that lets MLLMs reason and plan directly in spatial and spatiotemporal latent spaces and interface tightly with diffusion generators. We empirically implement MetaCanvas on three different diffusion backbones and evaluate it across six tasks, including text-to-image generation, text/image-to-video generation, image/video editing, and in-context video generation, each requiring precise layouts, robust attribute binding, and reasoning-intensive control. MetaCanvas consistently outperforms global-conditioning baselines, suggesting that treating MLLMs as latent-space planners is a promising direction for narrowing the gap between multimodal understanding and generation.
693f7a62f516c6932468121f
Han Lin
693f7a62f516c69324681220
Xichen Pan
693f7a62f516c69324681221
Ziqi Huang
claimed_verified
2026-01-13T08:24:56.362Z
60efe7fa0d920bc7805cada5
https://cdn-avatars.huggingface.co/v1/production/uploads/60efe7fa0d920bc7805cada5/2LBrJBjSCOP5ilZIpWLHl.png
Ziqi Huang
Ziqi
user
Ziqi
693f7a62f516c69324681222
Ji Hou
693f7a62f516c69324681223
Jialiang Wang
693f7a62f516c69324681224
Weifeng Chen
693f7a62f516c69324681225
Zecheng He
693f7a62f516c69324681226
Felix Juefei-Xu
claimed_verified
2026-01-02T15:40:47.978Z
6417cf37dce1e4c0229f17b1
https://cdn-avatars.huggingface.co/v1/production/uploads/6417cf37dce1e4c0229f17b1/7h-ZCB5f4wif7TsnF-B1M.jpeg
Felix Xu
katanaxu
user
katanaxu
693f7a62f516c69324681227
Junzhe Sun
claimed_verified
2026-02-19T09:54:16.653Z
697563dbce43f259ee32d7ed
/avatars/ac626eb596216c6e87f13fc52ba3fa11.svg
Junzhe Sun
junzhesun
user
junzhesun
693f7a62f516c69324681228
Zhipeng Fan
693f7a62f516c69324681229
Ali Thabet
693f7a62f516c6932468122a
Mohit Bansal
693f7a62f516c6932468122b
Chu Wang
Exploring MLLM-Diffusion Information Transfer with MetaCanvas
Multimodal learning has rapidly advanced visual understanding, largely via multimodal large language models (MLLMs) that use powerful LLMs as cognitive cores. In visual generation, however, these powerful core models are typically reduced to global text encoders for diffusion models, leaving most of their reasoning and planning ability unused. This creates a gap: current multimodal LLMs can parse complex layouts, attributes, and knowledge-intensive scenes, yet struggle to generate images or videos with equally precise and structured control. We propose MetaCanvas, a lightweight framework that lets MLLMs reason and plan directly in spatial and spatiotemporal latent spaces and interface tightly with diffusion generators. We empirically implement MetaCanvas on three different diffusion backbones and evaluate it across six tasks, including text-to-image generation, text/image-to-video generation, image/video editing, and in-context video generation, each requiring precise layouts, robust attribute binding, and reasoning-intensive control. MetaCanvas consistently outperforms global-conditioning baselines, suggesting that treating MLLMs as latent-space planners is a promising direction for narrowing the gap between multimodal understanding and generation.
Multimodal large language models can be enhanced for visual generation by enabling direct reasoning and planning in spatial and spatiotemporal latent spaces rather than just serving as text encoders for diffusion models.
multimodal large language models diffusion models spatial latent spaces spatiotemporal latent spaces latent-space planners text-to-image generation text/image-to-video generation image/video editing in-context video generation