Architecture Decoupling Is Not All You Need For Unified Multimodal Model
Unified multimodal models for image generation and understanding represent a significant step toward AGI and have attracted widespread attention from researchers. The main challenge of this task lies in the difficulty in establishing an optimal training paradigm due to inherent conflicting targets in understanding and generation tasks. To alleviate these conflicts and pursue higher performance, many researchers adopt varying degrees of model decoupling (e.g., Double image encoders, MOE/MOT architecture, or frozen MLLM). However, excessive model decoupling can lead to the loss of interleave generation ability, undermining the original intent of unified models. In this work, we aim to explore how to mitigate task conflicts without resorting to model decoupling. Firstly, we analyze why decoupling alleviates conflicts by studying the cross-modal attention behavior of models. We observe that model decoupling essentially drives models toward task-specific multimodal interaction patterns, as seen in Qwen-VL and HunyuanImage, and that the more thorough the decoupling, the more consistent the behavior becomes. Motivated by this observation, we propose Attention Interaction Alignment (AIA) loss, which explicitly learns Task-Specific multimodal interaction patterns during training. To demonstrate the generalizability of our AIA loss, we apply it to Emu3 and Janus-Pro during SFT and post-training stage respectively. Without bells and whistles, AIA not only refines cross-modal attention patterns, but also boosts both generation and understanding performance.
692cff484397b1ec214f676e
Dian Zheng
claimed_verified
2025-12-01T16:24:48.518Z
67e60ae6ac37824273d74389
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/YvPKZ_0gyJnvNwM1zK3JS.png
Dian Zheng
zhengli1013
user
zhengli1013
692cff484397b1ec214f676f
Manyuan Zhang
692cff484397b1ec214f6770
Hongyu Li
claimed_verified
2025-12-08T08:35:50.015Z
66fd43e80cde4879f9aeca01
/avatars/7bc9afa5e023e00820333e8d18dc4bc5.svg
Hongyu Li
appletea2333
user
appletea2333
692cff484397b1ec214f6771
Kai Zou
692cff484397b1ec214f6772
Hongbo Liu
692cff484397b1ec214f6773
Ziyu Guo
692cff484397b1ec214f6774
Kaituo Feng
admin_assigned
2025-12-01T10:13:27.799Z
67079840a9bcb7459b8d2a46
/avatars/32466863c5554f20cb2775b138832ac3.svg
Kaituo Feng
KaituoFeng
user
KaituoFeng
692cff484397b1ec214f6775
Yexin Liu
admin_assigned
2025-12-01T10:14:03.791Z
6497ff395b5d43c1c77d2a11
/avatars/e3704701ed3ea0c1c165001521f40086.svg
Yexin Liu
AIPeanutman
user
AIPeanutman
692cff484397b1ec214f6776
Ying Luo
692cff484397b1ec214f6777
Yan Feng
692cff484397b1ec214f6778
Peng Pei
692cff484397b1ec214f6779
Xunliang Cai
692cff484397b1ec214f677a
Hongsheng Li
Architecture Decoupling Is Not All You Need For Unified Multimodal Model
Unified multimodal models for image generation and understanding represent a significant step toward AGI and have attracted widespread attention from researchers. The main challenge of this task lies in the difficulty in establishing an optimal training paradigm due to inherent conflicting targets in understanding and generation tasks. To alleviate these conflicts and pursue higher performance, many researchers adopt varying degrees of model decoupling (e.g., Double image encoders, MOE/MOT architecture, or frozen MLLM). However, excessive model decoupling can lead to the loss of interleave generation ability, undermining the original intent of unified models. In this work, we aim to explore how to mitigate task conflicts without resorting to model decoupling. Firstly, we analyze why decoupling alleviates conflicts by studying the cross-modal attention behavior of models. We observe that model decoupling essentially drives models toward task-specific multimodal interaction patterns, as seen in Qwen-VL and HunyuanImage, and that the more thorough the decoupling, the more consistent the behavior becomes. Motivated by this observation, we propose Attention Interaction Alignment (AIA) loss, which explicitly learns Task-Specific multimodal interaction patterns during training. To demonstrate the generalizability of our AIA loss, we apply it to Emu3 and Janus-Pro during SFT and post-training stage respectively. Without bells and whistles, AIA not only refines cross-modal attention patterns, but also boosts both generation and understanding performance.
The proposed Attention Interaction Alignment (AIA) loss improves cross-modal attention and performance in unified multimodal models for image generation and understanding without decoupling.
multimodal models image generation understanding task-specific multimodal interaction cross-modal attention Attention Interaction Alignment (AIA) loss SFT post-training stage Qwen-VL HunyuanImage Emu3 Janus-Pro