UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings
The remarkable success of multimodal large language models (MLLMs) has driven
advances in multimodal embeddings, yet existing models remain inherently
discriminative, limiting their ability to benefit from reasoning-driven
generation paradigm. In this work, we pioneer the exploration of generative
embeddings, unifying embedding tasks within a generative paradigm. We propose
UME-R1, a universal multimodal embedding framework consisting of a two-stage
training strategy: a cold-start supervised fine-tuning equips the model with
reasoning capabilities and enables it to generate both discriminative and
generative embeddings; a subsequent reinforcement learning enhances reasoning
and further optimizes generative embedding quality. This pioneering work
reveals four key insights: 1) generative embeddings unlock substantial
performance gains over conventional discriminative embeddings by leveraging the
powerful generative reasoning capabilities of MLLMs; 2) discriminative and
generative embeddings are complementary, whose combined oracle performance far
exceeding that of either alone; 3) RL can effectively enhance generative
embeddings, establishing a scalable optimization paradigm.; 4) repeated
sampling at inference boosts downstream task coverage (pass@k), highlighting
the inference-time scalability potential of generative embeddings. Evaluated on
the MMEB-V2 benchmark across 78 tasks spanning video, image, and visual
documents, UME-R1 significantly outperforms conventional discriminative
embedding models and offers a foundation for more interpretable,
reasoning-driven generative multimodal embeddings. Our code, models, and
datasets will be publicly available at https://github.com/XMUDeepLIT/UME-R1.
69097f4d812eca10f9cc6373
Zhibin Lan
69097f4d812eca10f9cc6374
Liqiang Niu
69097f4d812eca10f9cc6375
Fandong Meng
69097f4d812eca10f9cc6376
Jie Zhou
69097f4d812eca10f9cc6377
Jinsong Su
UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings
The remarkable success of multimodal large language models (MLLMs) has driven
advances in multimodal embeddings, yet existing models remain inherently
discriminative, limiting their ability to benefit from reasoning-driven
generation paradigm. In this work, we pioneer the exploration of generative
embeddings, unifying embedding tasks within a generative paradigm. We propose
UME-R1, a universal multimodal embedding framework consisting of a two-stage
training strategy: a cold-start supervised fine-tuning equips the model with
reasoning capabilities and enables it to generate both discriminative and
generative embeddings; a subsequent reinforcement learning enhances reasoning
and further optimizes generative embedding quality. This pioneering work
reveals four key insights: 1) generative embeddings unlock substantial
performance gains over conventional discriminative embeddings by leveraging the
powerful generative reasoning capabilities of MLLMs; 2) discriminative and
generative embeddings are complementary, whose combined oracle performance far
exceeding that of either alone; 3) RL can effectively enhance generative
embeddings, establishing a scalable optimization paradigm.; 4) repeated
sampling at inference boosts downstream task coverage (pass@k), highlighting
the inference-time scalability potential of generative embeddings. Evaluated on
the MMEB-V2 benchmark across 78 tasks spanning video, image, and visual
documents, UME-R1 significantly outperforms conventional discriminative
embedding models and offers a foundation for more interpretable,
reasoning-driven generative multimodal embeddings. Our code, models, and
datasets will be publicly available at https://github.com/XMUDeepLIT/UME-R1.
UME-R1, a generative multimodal embedding framework, enhances performance through reasoning-driven generation and reinforcement learning, outperforming conventional discriminative models.
multimodal large language models multimodal embeddings generative embeddings discriminative embeddings two-stage training strategy cold-start supervised fine-tuning reinforcement learning MMEB-V2 benchmark pass@k inference-time scalability