LightBagel: A Light-weighted, Double Fusion Framework for Unified
  Multimodal Understanding and Generation
Unified multimodal models have recently shown remarkable gains in both
capability and versatility, yet most leading systems are still trained from
scratch and require substantial computational resources. In this paper, we show
that competitive performance can be obtained far more efficiently by
strategically fusing publicly available models specialized for either
generation or understanding. Our key design is to retain the original blocks
while additionally interleaving multimodal self-attention blocks throughout the
networks. This double fusion mechanism (1) effectively enables rich multi-modal
fusion while largely preserving the original strengths of the base models, and
(2) catalyzes synergistic fusion of high-level semantic representations from
the understanding encoder with low-level spatial signals from the generation
encoder. By training with only ~ 35B tokens, this approach achieves strong
results across multiple benchmarks: 0.91 on GenEval for compositional
text-to-image generation, 82.16 on DPG-Bench for complex text-to-image
generation, 6.06 on GEditBench, and 3.77 on ImgEdit-Bench for image editing. By
fully releasing the entire suite of code, model weights, and datasets, we hope
to support future research on unified multimodal modeling.
6900334b22d452aac6dd4324
Zeyu Wang
6900334b22d452aac6dd4325
Zilong Chen
6900334b22d452aac6dd4326
Chenhui Gou
6900334b22d452aac6dd4327
Feng Li
6900334b22d452aac6dd4328
Chaorui Deng
6900334b22d452aac6dd4329
Deyao Zhu
6900334b22d452aac6dd432a
Kunchang Li
6900334b22d452aac6dd432b
Weihao Yu
6900334b22d452aac6dd432c
Haoqin Tu
6900334b22d452aac6dd432d
Haoqi Fan
6900334b22d452aac6dd432e
Cihang Xie
LightBagel: A Light-weighted, Double Fusion Framework for Unified
  Multimodal Understanding and Generation
Unified multimodal models have recently shown remarkable gains in both
capability and versatility, yet most leading systems are still trained from
scratch and require substantial computational resources. In this paper, we show
that competitive performance can be obtained far more efficiently by
strategically fusing publicly available models specialized for either
generation or understanding. Our key design is to retain the original blocks
while additionally interleaving multimodal self-attention blocks throughout the
networks. This double fusion mechanism (1) effectively enables rich multi-modal
fusion while largely preserving the original strengths of the base models, and
(2) catalyzes synergistic fusion of high-level semantic representations from
the understanding encoder with low-level spatial signals from the generation
encoder. By training with only ~ 35B tokens, this approach achieves strong
results across multiple benchmarks: 0.91 on GenEval for compositional
text-to-image generation, 82.16 on DPG-Bench for complex text-to-image
generation, 6.06 on GEditBench, and 3.77 on ImgEdit-Bench for image editing. By
fully releasing the entire suite of code, model weights, and datasets, we hope
to support future research on unified multimodal modeling.
A unified multimodal model achieves competitive performance with efficient fusion of generation and understanding models, using interleaved multimodal self-attention blocks and minimal training resources.
multimodal models self-attention blocks compositional text-to-image generation complex text-to-image generation image editing