CASA: Cross-Attention via Self-Attention for Efficient Vision-Language Fusion
Vision-language models (VLMs) are commonly trained by inserting image tokens from a pretrained vision encoder into the textual stream of a language model. This allows text and image information to fully attend to one another within the model, but becomes extremely costly for high-resolution images, long conversations, or streaming videos, both in memory and compute. VLMs leveraging cross-attention are an efficient alternative to token insertion but exhibit a clear performance gap, in particular on tasks involving fine-grained visual details. We find that a key to improving such models is to also enable local text-to-text interaction in the dedicated cross-attention layers. Building on this, we propose CASA, Cross-Attention via Self-Attention, a simple and efficient paradigm which substantially reduces the gap with full token insertion on common image understanding benchmarks, while enjoying the same scalability as cross-attention models when applied to long-context multimodal tasks such as streaming video captioning. For samples and code, please see our project page at https://kyutai.org/casa .
694a4603335742716e9323bc
Moritz Böhle
694a4603335742716e9323bd
Amélie Royer
claimed_verified
2025-12-25T20:47:43.045Z
66b1ded53ddfb745eddfd072
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/JD85tZKReU8CWS-QDWnTU.png
Amelie Royer
ameroyer
user
ameroyer
694a4603335742716e9323be
Juliette Marrie
694a4603335742716e9323bf
Edouard Grave
694a4603335742716e9323c0
Patrick Pérez
CASA: Cross-Attention via Self-Attention for Efficient Vision-Language Fusion
Vision-language models (VLMs) are commonly trained by inserting image tokens from a pretrained vision encoder into the textual stream of a language model. This allows text and image information to fully attend to one another within the model, but becomes extremely costly for high-resolution images, long conversations, or streaming videos, both in memory and compute. VLMs leveraging cross-attention are an efficient alternative to token insertion but exhibit a clear performance gap, in particular on tasks involving fine-grained visual details. We find that a key to improving such models is to also enable local text-to-text interaction in the dedicated cross-attention layers. Building on this, we propose CASA, Cross-Attention via Self-Attention, a simple and efficient paradigm which substantially reduces the gap with full token insertion on common image understanding benchmarks, while enjoying the same scalability as cross-attention models when applied to long-context multimodal tasks such as streaming video captioning. For samples and code, please see our project page at https://kyutai.org/casa .
CASA, a cross-attention method enhanced with self-attention, improves vision-language models' performance on detailed visual tasks while maintaining scalability for long-context multimodal applications.
vision-language models VLMs image tokens pretrained vision encoder textual stream cross-attention local text-to-text interaction CASA Cross-Attention via Self-Attention image understanding benchmarks streaming video captioning