Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ExpAlign: Expectation-Guided Vision-Language Alignment for Open-Vocabulary Grounding

Junyi Hu, Tian Bai, Fengyi Wu, Wenyan Li, Zhenming Peng, Yi Zhang

TL;DR

ExpAlign introduces an Expectation Alignment Head (EAH) to perform token-level, prompt-conditioned spatial alignment via soft MIL pooling, addressing the limitations of global sentence embeddings for open-vocabulary grounding. It augments this with a Geometry-Aware Consistency Objective (GACO) and a multi-positive InfoNCE loss to stabilize multi-scale alignment and enforce semantic and geometric coherence. Across LVIS, ODinW, and referring-expression datasets, ExpAlign delivers strong zero-shot detection and segmentation while maintaining a lightweight, inference-efficient design, with notable gains on long-tail categories. The work provides a principled framework that bridges token-level reasoning and weak supervision without heavy cross-attention, advancing robust, open-world vision-language grounding.

Abstract

Open-vocabulary grounding requires accurate vision-language alignment under weak supervision, yet existing methods either rely on global sentence embeddings that lack fine-grained expressiveness or introduce token-level alignment with explicit supervision or heavy cross-attention designs. We propose ExpAlign, a theoretically grounded vision-language alignment framework built on a principled multiple instance learning formulation. ExpAlign introduces an Expectation Alignment Head that performs attention-based soft MIL pooling over token-region similarities, enabling implicit token and instance selection without additional annotations. To further stabilize alignment learning, we develop an energy-based multi-scale consistency regularization scheme, including a Top-K multi-positive contrastive objective and a Geometry-Aware Consistency Objective derived from a Lagrangian-constrained free-energy minimization. Extensive experiments show that ExpAlign consistently improves open-vocabulary detection and zero-shot instance segmentation, particularly on long-tail categories. Most notably, it achieves 36.2 AP$_r$ on the LVIS minival split, outperforming other state-of-the-art methods at comparable model scale, while remaining lightweight and inference-efficient.

ExpAlign: Expectation-Guided Vision-Language Alignment for Open-Vocabulary Grounding

TL;DR

ExpAlign introduces an Expectation Alignment Head (EAH) to perform token-level, prompt-conditioned spatial alignment via soft MIL pooling, addressing the limitations of global sentence embeddings for open-vocabulary grounding. It augments this with a Geometry-Aware Consistency Objective (GACO) and a multi-positive InfoNCE loss to stabilize multi-scale alignment and enforce semantic and geometric coherence. Across LVIS, ODinW, and referring-expression datasets, ExpAlign delivers strong zero-shot detection and segmentation while maintaining a lightweight, inference-efficient design, with notable gains on long-tail categories. The work provides a principled framework that bridges token-level reasoning and weak supervision without heavy cross-attention, advancing robust, open-world vision-language grounding.

Abstract

Open-vocabulary grounding requires accurate vision-language alignment under weak supervision, yet existing methods either rely on global sentence embeddings that lack fine-grained expressiveness or introduce token-level alignment with explicit supervision or heavy cross-attention designs. We propose ExpAlign, a theoretically grounded vision-language alignment framework built on a principled multiple instance learning formulation. ExpAlign introduces an Expectation Alignment Head that performs attention-based soft MIL pooling over token-region similarities, enabling implicit token and instance selection without additional annotations. To further stabilize alignment learning, we develop an energy-based multi-scale consistency regularization scheme, including a Top-K multi-positive contrastive objective and a Geometry-Aware Consistency Objective derived from a Lagrangian-constrained free-energy minimization. Extensive experiments show that ExpAlign consistently improves open-vocabulary detection and zero-shot instance segmentation, particularly on long-tail categories. Most notably, it achieves 36.2 AP on the LVIS minival split, outperforming other state-of-the-art methods at comparable model scale, while remaining lightweight and inference-efficient.
Paper Structure (28 sections, 1 theorem, 35 equations, 6 figures, 9 tables, 1 algorithm)

This paper contains 28 sections, 1 theorem, 35 equations, 6 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Sentence-level vision-language Alignment.
  4. Token-level and Phrase-level Alignment.
  5. Alignment Regularization and Objectives.
  6. RL-Inspired Regularization in VLMs
  7. Method
  8. Overview
  9. Expectation Alignment Head
  10. Consistency Regularization Module
  11. Full Training Objective
  12. Connection to Multiple Instance Learning
  13. Experiment
  14. Implementation Details
  15. Zero-shot Detection and Segmentation Performance
  16. ...and 13 more sections

Key Result

Theorem 2.4

Under Assumptions B1--B3, consider the variational free-energy functional Then:

Figures (6)

  • Figure 1: Overview of the proposed ExpAlign framework. Top: the overall pipeline, where prompt-conditioned Expectation Alignment Maps (EAMs) are computed at multiple feature scales and injected into visual features for open-vocabulary grounding and segmentation. Bottom-left: the Expectation Alignment Head, which aggregates token-level vision-language similarities into prompt-specific spatial alignment maps via expectation-based token weighting. Bottom-right: the Consistency Regularization Module, which applies semantic and geometric constraints to regularize the alignment maps. Best viewed in color.
  • Figure 2: Expectation alignment map calculation diagram. Spatial alignment maps are first computed for individual text tokens. All maps are then aggregated with their importance weight (displayed below each map) to form a prompt-conditioned expectation alignment map.
  • Figure 3: Qualitative examples of detection and segmentation results. (a) prompts: laptop, cellphone, watch, cup, mouse, long arm desk lamp, pen, mouse pad, touchpad, screen, keyboard. (b) prompts: paper cutting, cabinet, exit sign. (c) prompts: capybara, monkey on the back of capybara. (d) prompts: person wearing helmet, pliers, gloves, goggles. Zoom in for better visual effect.
  • Figure 4: EAM heatmaps for positive prompt sailor uniform and negative prompt black sailor uniform. Background-dominant activations indicate effective suppression of unseen negative prompts.
  • Figure 5: (a) prompts: girl, sailor uniform, the right loafer, bow-knot, knee-high socks, pleated skirt. (b) prompts: snowboard, ski goggles, gondola lift, gloves, bunny ears headband. (c) prompts: minion, ballons, a kid in white shirt, woman wear sunglasses. (d) prompts: black cat, panda toy, round panda toy. Zoom in for better visual effect.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 2.4: Variational optimality and induced Gibbs form
  • proof