ExpAlign: Expectation-Guided Vision-Language Alignment for Open-Vocabulary Grounding
Open-vocabulary grounding requires accurate vision-language alignment under weak supervision, yet existing methods either rely on global sentence embeddings that lack fine-grained expressiveness or introduce token-level alignment with explicit supervision or heavy cross-attention designs. We propose ExpAlign, a theoretically grounded vision-language alignment framework built on a principled multiple instance learning formulation. ExpAlign introduces an Expectation Alignment Head that performs attention-based soft MIL pooling over token-region similarities, enabling implicit token and instance selection without additional annotations. To further stabilize alignment learning, we develop an energy-based multi-scale consistency regularization scheme, including a Top-K multi-positive contrastive objective and a Geometry-Aware Consistency Objective derived from a Lagrangian-constrained free-energy minimization. Extensive experiments show that ExpAlign consistently improves open-vocabulary detection and zero-shot instance segmentation, particularly on long-tail categories. Most notably, it achieves 36.2 AP_r on the LVIS minival split, outperforming other state-of-the-art methods at comparable model scale, while remaining lightweight and inference-efficient.
6980c7ba020e48d648c5d3fa
Junyi Hu
6980c7ba020e48d648c5d3fb
Tian Bai
6980c7ba020e48d648c5d3fc
Fengyi Wu
claimed_verified
2026-02-03T10:08:29.612Z
6894778dadd77868b89d809f
/avatars/44eff49c5d43745712e2cf2c68967ced.svg
Fengyi Wu
fengyiwu
user
fengyiwu
6980c7ba020e48d648c5d3fd
Wenyan Li
6980c7ba020e48d648c5d3fe
Zhenming Peng
6980c7ba020e48d648c5d3ff
Yi Zhang
ExpAlign: Expectation-Guided Vision-Language Alignment for Open-Vocabulary Grounding
Open-vocabulary grounding requires accurate vision-language alignment under weak supervision, yet existing methods either rely on global sentence embeddings that lack fine-grained expressiveness or introduce token-level alignment with explicit supervision or heavy cross-attention designs. We propose ExpAlign, a theoretically grounded vision-language alignment framework built on a principled multiple instance learning formulation. ExpAlign introduces an Expectation Alignment Head that performs attention-based soft MIL pooling over token-region similarities, enabling implicit token and instance selection without additional annotations. To further stabilize alignment learning, we develop an energy-based multi-scale consistency regularization scheme, including a Top-K multi-positive contrastive objective and a Geometry-Aware Consistency Objective derived from a Lagrangian-constrained free-energy minimization. Extensive experiments show that ExpAlign consistently improves open-vocabulary detection and zero-shot instance segmentation, particularly on long-tail categories. Most notably, it achieves 36.2 AP_r on the LVIS minival split, outperforming other state-of-the-art methods at comparable model scale, while remaining lightweight and inference-efficient.
ExpAlign presents a vision-language alignment framework using multiple instance learning and attention-based pooling to improve open-vocabulary detection and zero-shot instance segmentation without additional annotations.
vision-language alignment multiple instance learning Expectation Alignment Head soft MIL pooling token-region similarities energy-based multi-scale consistency regularization Top-K multi-positive contrastive objective Geometry-Aware Consistency Objective Lagrangian-constrained free-energy minimization open-vocabulary detection zero-shot instance segmentation LVIS minival split