Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models
Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions. Prior approaches to bridge this gap are largely limited by oversimplified isotropic assumptions, hindering their application in large-scale scenarios. In this paper, we address these limitations by precisely characterizing the geometric shape of the modality gap and leveraging it for efficient model scaling. First, we propose the Fixed-frame Modality Gap Theory, which decomposes the modality gap within a frozen reference frame into stable biases and anisotropic residuals. Guided by this precise modeling, we introduce ReAlign, a training-free modality alignment strategy. Utilizing statistics from massive unpaired data, ReAlign aligns text representation into the image representation distribution via a three-step process comprising Anchor, Trace, and Centroid Alignment, thereby explicitly rectifying geometric misalignment. Building on ReAlign, we propose ReVision, a scalable training paradigm for Multimodal Large Language Models (MLLMs). ReVision integrates ReAlign into the pretraining stage, enabling the model to learn the distribution of visual representations from unpaired text before visual instruction tuning, without the need for large-scale, high-quality image-text pairs. Our framework demonstrates that statistically aligned unpaired data can effectively substitute for expensive image-text pairs, offering a robust path for the efficient scaling of MLLMs.
698a98541b2dc6b37d61af09
Xiaomin Yu
claimed_verified
2026-02-10T09:07:26.323Z
64084fa192033c150738e4f2
/avatars/dfff2216eb235c635e5abe6fda3084f0.svg
Yu_xm
Yu2020
user
Yu2020
698a98541b2dc6b37d61af0a
Yi Xin
698a98541b2dc6b37d61af0b
Wenjie Zhang
698a98541b2dc6b37d61af0c
Chonghan Liu
698a98541b2dc6b37d61af0d
Hanzhen Zhao
698a98541b2dc6b37d61af0e
Xiaoxing Hu
698a98541b2dc6b37d61af0f
Xinlei Yu
698a98541b2dc6b37d61af10
Ziyue Qiao
698a98541b2dc6b37d61af11
Hao Tang
698a98541b2dc6b37d61af12
Xue Yang
698a98541b2dc6b37d61af13
Xiaobin Hu
698a98541b2dc6b37d61af14
Chengwei Qin
698a98541b2dc6b37d61af15
Hui Xiong
698a98541b2dc6b37d61af16
Yu Qiao
698a98541b2dc6b37d61af17
Shuicheng Yan
Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models
Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions. Prior approaches to bridge this gap are largely limited by oversimplified isotropic assumptions, hindering their application in large-scale scenarios. In this paper, we address these limitations by precisely characterizing the geometric shape of the modality gap and leveraging it for efficient model scaling. First, we propose the Fixed-frame Modality Gap Theory, which decomposes the modality gap within a frozen reference frame into stable biases and anisotropic residuals. Guided by this precise modeling, we introduce ReAlign, a training-free modality alignment strategy. Utilizing statistics from massive unpaired data, ReAlign aligns text representation into the image representation distribution via a three-step process comprising Anchor, Trace, and Centroid Alignment, thereby explicitly rectifying geometric misalignment. Building on ReAlign, we propose ReVision, a scalable training paradigm for Multimodal Large Language Models (MLLMs). ReVision integrates ReAlign into the pretraining stage, enabling the model to learn the distribution of visual representations from unpaired text before visual instruction tuning, without the need for large-scale, high-quality image-text pairs. Our framework demonstrates that statistically aligned unpaired data can effectively substitute for expensive image-text pairs, offering a robust path for the efficient scaling of MLLMs.
Researchers address the modality gap in multimodal learning by proposing a fixed-frame theory and a training-free alignment method that enables efficient scaling of multimodal models using unpaired data.
multimodal contrastive learning modality gap geometric anomaly isotropic assumptions Fixed-frame Modality Gap Theory ReAlign Anchor Alignment Trace Alignment Centroid Alignment ReVision Multimodal Large Language Models unpaired data visual representation distribution pretraining stage visual instruction tuning