LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model
Unified models (UMs) hold promise for their ability to understand and generate content across heterogeneous modalities. Compared to merely generating visual content, the use of UMs for interleaved cross-modal reasoning is more promising and valuable, e.g., for solving understanding problems that require dense visual thinking, improving visual generation through self-reflection, or modeling visual dynamics of the physical world guided by stepwise action interventions. However, existing UMs necessitate pixel decoding as a bridge due to their disjoint visual representations for understanding and generation, which is both ineffective and inefficient. In this paper, we introduce LatentUM, a novel unified model that represents all modalities within a shared semantic latent space, eliminating the need for pixel-space mediation between visual understanding and generation. This design naturally enables flexible interleaved cross-modal reasoning and generation. Beyond improved computational efficiency, the shared representation substantially alleviates codec bias and strengthens cross-modal alignment, allowing LatentUM to achieve state-of-the-art performance on the Visual Spatial Planning benchmark, push the limits of visual generation through self-reflection, and support world modeling by predicting future visual states within the shared semantic latent space.
69cf3956e02fdd68b91b7a93
Jiachun Jin
claimed_verified
2026-04-03T09:07:44.022Z
66260b48d0a96a5db0de9722
/avatars/2901980e13b37af52cdb13bc5f8a6de3.svg
Jiachun Jin
orres
user
orres
69cf3956e02fdd68b91b7a94
Zetong Zhou
claimed_verified
2026-04-07T08:45:36.757Z
6697e7e55ef2828a1ff371c3
https://cdn-avatars.huggingface.co/v1/production/uploads/6697e7e55ef2828a1ff371c3/U7-_BtDtSsrf02LIdUTN8.jpeg
Zetong Zhou
Frywind
user
Frywind
69cf3956e02fdd68b91b7a95
Xiao Yang
69cf3956e02fdd68b91b7a96
Hao Zhang
69cf3956e02fdd68b91b7a97
Pengfei Liu
69cf3956e02fdd68b91b7a98
Jun Zhu
69cf3956e02fdd68b91b7a99
Zhijie Deng
LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model
Unified models (UMs) hold promise for their ability to understand and generate content across heterogeneous modalities. Compared to merely generating visual content, the use of UMs for interleaved cross-modal reasoning is more promising and valuable, e.g., for solving understanding problems that require dense visual thinking, improving visual generation through self-reflection, or modeling visual dynamics of the physical world guided by stepwise action interventions. However, existing UMs necessitate pixel decoding as a bridge due to their disjoint visual representations for understanding and generation, which is both ineffective and inefficient. In this paper, we introduce LatentUM, a novel unified model that represents all modalities within a shared semantic latent space, eliminating the need for pixel-space mediation between visual understanding and generation. This design naturally enables flexible interleaved cross-modal reasoning and generation. Beyond improved computational efficiency, the shared representation substantially alleviates codec bias and strengthens cross-modal alignment, allowing LatentUM to achieve state-of-the-art performance on the Visual Spatial Planning benchmark, push the limits of visual generation through self-reflection, and support world modeling by predicting future visual states within the shared semantic latent space.
LatentUM is a unified model that represents all modalities in a shared semantic latent space, enabling efficient cross-modal reasoning and generation without pixel-space mediation.
unified models visual understanding visual generation semantic latent space cross-modal reasoning codec bias cross-modal alignment visual spatial planning world modeling visual dynamics