Instruction Anchors: Dissecting the Causal Dynamics of Modality Arbitration
Modality following serves as the capacity of multimodal large language models (MLLMs) to selectively utilize multimodal contexts based on user instructions. It is fundamental to ensuring safety and reliability in real-world deployments. However, the underlying mechanisms governing this decision-making process remain poorly understood. In this paper, we investigate its working mechanism through an information flow lens. Our findings reveal that instruction tokens function as structural anchors for modality arbitration: Shallow attention layers perform non-selective information transfer, routing multimodal cues to these anchors as a latent buffer; Modality competition is resolved within deep attention layers guided by the instruction intent, while MLP layers exhibit semantic inertia, acting as an adversarial force. Furthermore, we identify a sparse set of specialized attention heads that drive this arbitration. Causal interventions demonstrate that manipulating a mere 5% of these critical heads can decrease the modality-following ratio by 60% through blocking, or increase it by 60% through targeted amplification of failed samples. Our work provides a substantial step toward model transparency and offers a principled framework for the orchestration of multimodal information in MLLMs.
6983275f9084cb4f0ecb596f
Yu Zhang
6983275f9084cb4f0ecb5970
Mufan Xu
6983275f9084cb4f0ecb5971
Xuefeng Bai
6983275f9084cb4f0ecb5972
Kehai chen
6983275f9084cb4f0ecb5973
Pengfei Zhang
6983275f9084cb4f0ecb5974
Yang Xiang
6983275f9084cb4f0ecb5975
Min Zhang
Instruction Anchors: Dissecting the Causal Dynamics of Modality Arbitration
Modality following serves as the capacity of multimodal large language models (MLLMs) to selectively utilize multimodal contexts based on user instructions. It is fundamental to ensuring safety and reliability in real-world deployments. However, the underlying mechanisms governing this decision-making process remain poorly understood. In this paper, we investigate its working mechanism through an information flow lens. Our findings reveal that instruction tokens function as structural anchors for modality arbitration: Shallow attention layers perform non-selective information transfer, routing multimodal cues to these anchors as a latent buffer; Modality competition is resolved within deep attention layers guided by the instruction intent, while MLP layers exhibit semantic inertia, acting as an adversarial force. Furthermore, we identify a sparse set of specialized attention heads that drive this arbitration. Causal interventions demonstrate that manipulating a mere 5% of these critical heads can decrease the modality-following ratio by 60% through blocking, or increase it by 60% through targeted amplification of failed samples. Our work provides a substantial step toward model transparency and offers a principled framework for the orchestration of multimodal information in MLLMs.
Research reveals that instruction tokens act as structural anchors in multimodal large language models, with shallow layers performing non-selective information transfer and deep layers resolving modality competition guided by instruction intent.
multimodal large language models instruction tokens modality arbitration attention layers multimodal cues semantic inertia attention heads causal interventions information flow lens