MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Control
MLLMs have been successfully applied to multimodal embedding tasks, yet their generative reasoning capabilities remain underutilized. Directly incorporating chain-of-thought reasoning into embedding learning introduces two fundamental challenges. First, structural misalignment between instance-level reasoning and pairwise contrastive supervision may lead to shortcut behavior, where the model merely learns the superficial format of reasoning. Second, reasoning is not universally beneficial for embedding tasks. Enforcing reasoning for all inputs may introduce unnecessary computation and latency, and can even obscure salient semantic signals for simple cases. To address these issues, we propose MMEmb-R1, an adaptive reasoning-based multimodal embedding framework. We formulate reasoning as a latent variable and introduce pair-aware reasoning selection that employs counterfactual intervention to identify reasoning paths beneficial for query-target alignment. Furthermore, we adopt reinforcement learning to selectively invoke reasoning only when necessary. Experiments on the MMEB-V2 benchmark demonstrate that our model achieves a score of 71.2 with only 4B parameters, establishing a new state-of-the-art while significantly reducing reasoning overhead and inference latency.
69d5bd46b9b236b710ae810d
Yuchi Wang
claimed_verified
2026-04-13T19:24:33.169Z
622842e296588dd1a2594746
/avatars/b96d0b49e4cff25d83fefc67b7cd1076.svg
wangyuchi
YuchiWang
user
YuchiWang
69d5bd46b9b236b710ae810e
Haiyang Yu
69d5bd46b9b236b710ae810f
Weikang Bian
69d5bd46b9b236b710ae8110
Jiefeng Long
69d5bd46b9b236b710ae8111
Xiao Liang
69d5bd46b9b236b710ae8112
Chao Feng
69d5bd46b9b236b710ae8113
Hongsheng Li
MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Control
MLLMs have been successfully applied to multimodal embedding tasks, yet their generative reasoning capabilities remain underutilized. Directly incorporating chain-of-thought reasoning into embedding learning introduces two fundamental challenges. First, structural misalignment between instance-level reasoning and pairwise contrastive supervision may lead to shortcut behavior, where the model merely learns the superficial format of reasoning. Second, reasoning is not universally beneficial for embedding tasks. Enforcing reasoning for all inputs may introduce unnecessary computation and latency, and can even obscure salient semantic signals for simple cases. To address these issues, we propose MMEmb-R1, an adaptive reasoning-based multimodal embedding framework. We formulate reasoning as a latent variable and introduce pair-aware reasoning selection that employs counterfactual intervention to identify reasoning paths beneficial for query-target alignment. Furthermore, we adopt reinforcement learning to selectively invoke reasoning only when necessary. Experiments on the MMEB-V2 benchmark demonstrate that our model achieves a score of 71.2 with only 4B parameters, establishing a new state-of-the-art while significantly reducing reasoning overhead and inference latency.
An adaptive multimodal embedding framework that selectively applies reasoning through latent variables and reinforcement learning to improve efficiency and performance on benchmark tasks.
multimodal embedding chain-of-thought reasoning structural misalignment pairwise contrastive supervision shortcut behavior latent variable counterfactual intervention reinforcement learning adaptive reasoning MMEB-V2 benchmark