A Mixed Diet Makes DINO An Omnivorous Vision Encoder
Pre-trained vision encoders like DINOv2 have demonstrated exceptional performance on unimodal tasks. However, we observe that their feature representations are poorly aligned across different modalities. For instance, the feature embedding for an RGB image and its corresponding depth map of the same scene exhibit a cosine similarity that is nearly identical to that of two random, unrelated images. To address this, we propose the Omnivorous Vision Encoder, a novel framework that learns a modality-agnostic feature space. We train the encoder with a dual objective: first, to maximize the feature alignment between different modalities of the same scene; and second, a distillation objective that anchors the learned representations to the output of a fully frozen teacher such as DINOv2. The resulting student encoder becomes "omnivorous" by producing a consistent, powerful embedding for a given scene, regardless of the input modality (RGB, Depth, Segmentation, etc.). This approach enables robust cross-modal understanding while retaining the discriminative semantics of the original foundation model.
69b3cf632ad6c5d4cf486c80
Rishabh Kabra
claimed_verified
2026-03-17T17:34:49.715Z
69b2f8502934912b0f4fb597
/avatars/852cf0f9e30de97ccb0186bf03454840.svg
Rishabh Kabra
rkabra
user
rkabra
69b3cf632ad6c5d4cf486c81
Maks Ovsjanikov
69b3cf632ad6c5d4cf486c82
Drew A. Hudson
69b3cf632ad6c5d4cf486c83
Ye Xia
69b3cf632ad6c5d4cf486c84
Skanda Koppula
69b3cf632ad6c5d4cf486c85
Andre Araujo
69b3cf632ad6c5d4cf486c86
Joao Carreira
69b3cf632ad6c5d4cf486c87
Niloy J. Mitra
A Mixed Diet Makes DINO An Omnivorous Vision Encoder
Pre-trained vision encoders like DINOv2 have demonstrated exceptional performance on unimodal tasks. However, we observe that their feature representations are poorly aligned across different modalities. For instance, the feature embedding for an RGB image and its corresponding depth map of the same scene exhibit a cosine similarity that is nearly identical to that of two random, unrelated images. To address this, we propose the Omnivorous Vision Encoder, a novel framework that learns a modality-agnostic feature space. We train the encoder with a dual objective: first, to maximize the feature alignment between different modalities of the same scene; and second, a distillation objective that anchors the learned representations to the output of a fully frozen teacher such as DINOv2. The resulting student encoder becomes "omnivorous" by producing a consistent, powerful embedding for a given scene, regardless of the input modality (RGB, Depth, Segmentation, etc.). This approach enables robust cross-modal understanding while retaining the discriminative semantics of the original foundation model.
A novel vision encoder framework is presented that learns modality-agnostic feature representations by aligning multi-modal inputs while preserving semantic distinctions from a frozen teacher model.
vision encoders DINOv2 feature representations modality-agnostic feature alignment distillation objective teacher-student training cross-modal understanding discriminative semantics