Steerable Visual Representations
Pretrained Vision Transformers (ViTs) such as DINOv2 and MAE provide generic image features that can be applied to a variety of downstream tasks such as retrieval, classification, and segmentation. However, such representations tend to focus on the most salient visual cues in the image, with no way to direct them toward less prominent concepts of interest. In contrast, Multimodal LLMs can be guided with textual prompts, but the resulting representations tend to be language-centric and lose their effectiveness for generic visual tasks. To address this, we introduce Steerable Visual Representations, a new class of visual representations, whose global and local features can be steered with natural language. While most vision-language models (e.g., CLIP) fuse text with visual features after encoding (late fusion), we inject text directly into the layers of the visual encoder (early fusion) via lightweight cross-attention. We introduce benchmarks for measuring representational steerability, and demonstrate that our steerable visual features can focus on any desired objects in an image while preserving the underlying representation quality. Our method also matches or outperforms dedicated approaches on anomaly detection and personalized object discrimination, exhibiting zero-shot generalization to out-of-distribution tasks.
69cf9739e02fdd68b91b7c2f
Jona Ruthardt
claimed_verified
2026-04-07T08:44:51.969Z
6447d49e30fa4ecb85dfd04e
/avatars/ae2fd52587420ea452874085e1494578.svg
Jona Ruthardt
JonaRuthardt
user
JonaRuthardt
69cf9739e02fdd68b91b7c30
Manu Gaur
claimed_verified
2026-04-07T08:44:48.097Z
66f686bbad4fe83c91862a11
/avatars/647ea8d6379568f6cf25f48015f725b9.svg
Manu Gaur
manu-gaur
user
manu-gaur
69cf9739e02fdd68b91b7c31
Deva Ramanan
69cf9739e02fdd68b91b7c32
Makarand Tapaswi
69cf9739e02fdd68b91b7c33
Yuki M. Asano
Steerable Visual Representations
Pretrained Vision Transformers (ViTs) such as DINOv2 and MAE provide generic image features that can be applied to a variety of downstream tasks such as retrieval, classification, and segmentation. However, such representations tend to focus on the most salient visual cues in the image, with no way to direct them toward less prominent concepts of interest. In contrast, Multimodal LLMs can be guided with textual prompts, but the resulting representations tend to be language-centric and lose their effectiveness for generic visual tasks. To address this, we introduce Steerable Visual Representations, a new class of visual representations, whose global and local features can be steered with natural language. While most vision-language models (e.g., CLIP) fuse text with visual features after encoding (late fusion), we inject text directly into the layers of the visual encoder (early fusion) via lightweight cross-attention. We introduce benchmarks for measuring representational steerability, and demonstrate that our steerable visual features can focus on any desired objects in an image while preserving the underlying representation quality. Our method also matches or outperforms dedicated approaches on anomaly detection and personalized object discrimination, exhibiting zero-shot generalization to out-of-distribution tasks.
Steerable Visual Representations enable language-guided focus on specific image elements while maintaining representation quality through early fusion of text and visual features.
Vision Transformers DINOv2 MAE multimodal LLMs visual representations cross-attention late fusion early fusion anomaly detection personalized object discrimination zero-shot generalization