SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models
Models that bridge vision and language, such as CLIP, are key components of multimodal AI, yet their large-scale, uncurated training data introduce severe social and spurious biases. Existing post-hoc debiasing methods often operate directly in the dense CLIP embedding space, where bias and task-relevant information are highly entangled. This entanglement limits their ability to remove bias without degrading semantic fidelity. In this work, we propose Sparse Embedding Modulation (SEM), a post-hoc, zero-shot debiasing framework that operates in a Sparse Autoencoder (SAE) latent space. By decomposing CLIP text embeddings into disentangled features, SEM identifies and modulates bias-relevant neurons while preserving query-relevant ones. This enables more precise, non-linear interventions. Across four benchmark datasets and two CLIP backbones, SEM achieves substantial fairness gains in retrieval and zero-shot classification. Our results demonstrate that sparse latent representations provide an effective foundation for post-hoc debiasing of vision-language models.
69c25826995349774064f531
Quentin Guimard
claimed_verified
2026-03-24T09:24:15.408Z
6361986b108cc69e68810b28
/avatars/564a2003efdbb3dd3b06253a54aa9642.svg
Quentin Guimard
mardgui
user
mardgui
69c25826995349774064f532
Federico Bartsch
69c25826995349774064f533
Simone Caldarella
69c25826995349774064f534
Rahaf Aljundi
69c25826995349774064f535
Elisa Ricci
69c25826995349774064f536
Massimiliano Mancini
SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models
Models that bridge vision and language, such as CLIP, are key components of multimodal AI, yet their large-scale, uncurated training data introduce severe social and spurious biases. Existing post-hoc debiasing methods often operate directly in the dense CLIP embedding space, where bias and task-relevant information are highly entangled. This entanglement limits their ability to remove bias without degrading semantic fidelity. In this work, we propose Sparse Embedding Modulation (SEM), a post-hoc, zero-shot debiasing framework that operates in a Sparse Autoencoder (SAE) latent space. By decomposing CLIP text embeddings into disentangled features, SEM identifies and modulates bias-relevant neurons while preserving query-relevant ones. This enables more precise, non-linear interventions. Across four benchmark datasets and two CLIP backbones, SEM achieves substantial fairness gains in retrieval and zero-shot classification. Our results demonstrate that sparse latent representations provide an effective foundation for post-hoc debiasing of vision-language models.
Sparse Embedding Modulation (SEM) addresses bias in vision-language models by operating in sparse autoencoder latent space to selectively modulate bias-relevant neurons while preserving semantic information.
CLIP sparse autoencoder sparse embedding modulation post-hoc debiasing latent space bias-relevant neurons query-relevant neurons vision-language models sparse latent representations