VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a
  Unified Concept Set
The alignment of vision-language representations endows current
Vision-Language Models (VLMs) with strong multi-modal reasoning capabilities.
However, the interpretability of the alignment component remains uninvestigated
due to the difficulty in mapping the semantics of multi-modal representations
into a unified concept set. To address this problem, we propose VL-SAE, a
sparse autoencoder that encodes vision-language representations into its hidden
activations. Each neuron in its hidden layer correlates to a concept
represented by semantically similar images and texts, thereby interpreting
these representations with a unified concept set. To establish the
neuron-concept correlation, we encourage semantically similar representations
to exhibit consistent neuron activations during self-supervised training.
First, to measure the semantic similarity of multi-modal representations, we
perform their alignment in an explicit form based on cosine similarity. Second,
we construct the VL-SAE with a distance-based encoder and two modality-specific
decoders to ensure the activation consistency of semantically similar
representations. Experiments across multiple VLMs (e.g., CLIP, LLaVA)
demonstrate the superior capability of VL-SAE in interpreting and enhancing the
vision-language alignment. For interpretation, the alignment between vision and
language representations can be understood by comparing their semantics with
concepts. For enhancement, the alignment can be strengthened by aligning
vision-language representations at the concept level, contributing to
performance improvements in downstream tasks, including zero-shot image
classification and hallucination elimination. Codes are available at
https://github.com/ssfgunner/VL-SAE.
6900561a22d452aac6dd4439
Shufan Shen
claimed_verified
2025-10-28T15:35:14.639Z
660229531737e5cd4a6e7948
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/_S1dkCnhREr1XchhdPEGM.png
Shufan Shen
shufanshen
user
shufanshen
6900561a22d452aac6dd443a
Junshu Sun
6900561a22d452aac6dd443b
Qingming Huang
6900561a22d452aac6dd443c
Shuhui Wang
VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a
  Unified Concept Set
The alignment of vision-language representations endows current
Vision-Language Models (VLMs) with strong multi-modal reasoning capabilities.
However, the interpretability of the alignment component remains uninvestigated
due to the difficulty in mapping the semantics of multi-modal representations
into a unified concept set. To address this problem, we propose VL-SAE, a
sparse autoencoder that encodes vision-language representations into its hidden
activations. Each neuron in its hidden layer correlates to a concept
represented by semantically similar images and texts, thereby interpreting
these representations with a unified concept set. To establish the
neuron-concept correlation, we encourage semantically similar representations
to exhibit consistent neuron activations during self-supervised training.
First, to measure the semantic similarity of multi-modal representations, we
perform their alignment in an explicit form based on cosine similarity. Second,
we construct the VL-SAE with a distance-based encoder and two modality-specific
decoders to ensure the activation consistency of semantically similar
representations. Experiments across multiple VLMs (e.g., CLIP, LLaVA)
demonstrate the superior capability of VL-SAE in interpreting and enhancing the
vision-language alignment. For interpretation, the alignment between vision and
language representations can be understood by comparing their semantics with
concepts. For enhancement, the alignment can be strengthened by aligning
vision-language representations at the concept level, contributing to
performance improvements in downstream tasks, including zero-shot image
classification and hallucination elimination. Codes are available at
https://github.com/ssfgunner/VL-SAE.
VL-SAE, a sparse autoencoder, enhances vision-language alignment by correlating neurons to unified concepts, improving interpretability and performance in tasks like zero-shot image classification and hallucination elimination.
sparse autoencoder vision-language representations multi-modal reasoning neuron-concept correlation cosine similarity distance-based encoder modality-specific decoders zero-shot image classification hallucination elimination