Revisiting Multimodal Positional Encoding in Vision-Language Models
Multimodal position encoding is essential for vision-language models, yet
there has been little systematic investigation into multimodal position
encoding. We conduct a comprehensive analysis of multimodal Rotary Positional
Embedding (RoPE) by examining its two core components: position design and
frequency allocation. Through extensive experiments, we identify three key
guidelines: positional coherence, full frequency utilization, and preservation
of textual priors-ensuring unambiguous layout, rich representation, and
faithful transfer from the pre-trained LLM. Based on these insights, we propose
Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), two simple and
plug-and-play variants that require no architectural changes. Our methods
consistently outperform existing approaches across diverse benchmarks, with
significant improvements in both general and fine-grained multimodal
understanding. Code will be avaliable at
https://github.com/JJJYmmm/Multimodal-RoPEs.
6902d73072739622ee92a856
Jie Huang
claimed_verified
2025-10-30T14:27:21.611Z
6686b9f2978a8880573681e9
/avatars/965caeedb1ba847894c6b0e10025986e.svg
Jie Huang
JJJYmmm
user
JJJYmmm
6902d73072739622ee92a857
Xuejing Liu
claimed_verified
2025-10-30T14:27:16.199Z
6486c50978ab6bf101afc29f
/avatars/cd101a2c5188b48a1874f20756eb8f51.svg
XuejingLiu
GingL
user
GingL
6902d73072739622ee92a858
Sibo Song
6902d73072739622ee92a859
Ruibing Hou
6902d73072739622ee92a85a
Hong Chang
6902d73072739622ee92a85b
Junyang Lin
6902d73072739622ee92a85c
Shuai Bai
Revisiting Multimodal Positional Encoding in Vision-Language Models
Multimodal position encoding is essential for vision-language models, yet
there has been little systematic investigation into multimodal position
encoding. We conduct a comprehensive analysis of multimodal Rotary Positional
Embedding (RoPE) by examining its two core components: position design and
frequency allocation. Through extensive experiments, we identify three key
guidelines: positional coherence, full frequency utilization, and preservation
of textual priors-ensuring unambiguous layout, rich representation, and
faithful transfer from the pre-trained LLM. Based on these insights, we propose
Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), two simple and
plug-and-play variants that require no architectural changes. Our methods
consistently outperform existing approaches across diverse benchmarks, with
significant improvements in both general and fine-grained multimodal
understanding. Code will be avaliable at
https://github.com/JJJYmmm/Multimodal-RoPEs.
A comprehensive analysis of multimodal Rotary Positional Embedding (RoPE) leads to the proposal of Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), which improve multimodal understanding in vision-language models.
multimodal position encoding Rotary Positional Embedding (RoPE) position design frequency allocation positional coherence full frequency utilization preservation of textual priors Multi-Head RoPE (MHRoPE) MRoPE-Interleave (MRoPE-I) multimodal understanding