Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GTMA: Dynamic Representation Optimization for OOD Vision-Language Models

Jensen Zhang, Ningyuan Liu, Keze Wang

TL;DR

The paper identifies a fundamental bottleneck in Vision-Language Models for open-world use: modal asymmetry, where a fixed text vocabulary prevents synthesis of new anchors for OOD concepts. It introduces GTMA, a test-time framework that dynamically optimizes a continuous pseudo-word embedding to align with an OOD visual anchor, via the GRPO algorithm and semantic regularization to stay on a plausible semantic manifold. Through experiments on the VISTA-Beyond benchmark, GTMA yields notable gains in zero-shot and few-shot OOD accuracy while maintaining in-distribution performance, and ablations confirm the necessity of pseudo-word optimization and other components. The work also provides the VISTA-Beyond benchmark to rigorously evaluate OOD generalization in VLMs and highlights a promising direction toward robust open-world vision-language understanding.

Abstract

Vision-language models (VLMs) struggle in open-world applications, where out-of-distribution (OOD) concepts often trigger cross-modal alignment collapse and severely degrade zero-shot performance. We identify the root cause as modal asymmetry: while the visual encoder can extract discriminative features from unseen images, the text encoder is constrained by a fixed discrete vocabulary and cannot synthesize new semantic anchors. Existing approaches such as CoOp or LoRA provide only partial remedies, as they remain confined to the pre-trained semantic space. To overcome this bottleneck, we propose dynamic representation optimization, realized through the Guided Target-Matching Adaptation (GTMA) framework. At inference time, GTMA constructs a continuous pseudo-word embedding that best aligns with an OOD image's visual anchor, effectively bypassing vocabulary limitations. The optimization is driven by an adaptive gradient-based representation policy optimization algorithm, which incorporates semantic regularization to preserve plausibility and compatibility with the model's prior knowledge. Experiments on ImageNet-R and the VISTA-Beyond benchmark demonstrate that GTMA improves zero-shot and few-shot OOD accuracy by up to 15-20 percent over the base VLM while maintaining performance on in-distribution concepts. Ablation studies further confirm the necessity of pseudo-word optimization.

GTMA: Dynamic Representation Optimization for OOD Vision-Language Models

TL;DR

The paper identifies a fundamental bottleneck in Vision-Language Models for open-world use: modal asymmetry, where a fixed text vocabulary prevents synthesis of new anchors for OOD concepts. It introduces GTMA, a test-time framework that dynamically optimizes a continuous pseudo-word embedding to align with an OOD visual anchor, via the GRPO algorithm and semantic regularization to stay on a plausible semantic manifold. Through experiments on the VISTA-Beyond benchmark, GTMA yields notable gains in zero-shot and few-shot OOD accuracy while maintaining in-distribution performance, and ablations confirm the necessity of pseudo-word optimization and other components. The work also provides the VISTA-Beyond benchmark to rigorously evaluate OOD generalization in VLMs and highlights a promising direction toward robust open-world vision-language understanding.

Abstract

Vision-language models (VLMs) struggle in open-world applications, where out-of-distribution (OOD) concepts often trigger cross-modal alignment collapse and severely degrade zero-shot performance. We identify the root cause as modal asymmetry: while the visual encoder can extract discriminative features from unseen images, the text encoder is constrained by a fixed discrete vocabulary and cannot synthesize new semantic anchors. Existing approaches such as CoOp or LoRA provide only partial remedies, as they remain confined to the pre-trained semantic space. To overcome this bottleneck, we propose dynamic representation optimization, realized through the Guided Target-Matching Adaptation (GTMA) framework. At inference time, GTMA constructs a continuous pseudo-word embedding that best aligns with an OOD image's visual anchor, effectively bypassing vocabulary limitations. The optimization is driven by an adaptive gradient-based representation policy optimization algorithm, which incorporates semantic regularization to preserve plausibility and compatibility with the model's prior knowledge. Experiments on ImageNet-R and the VISTA-Beyond benchmark demonstrate that GTMA improves zero-shot and few-shot OOD accuracy by up to 15-20 percent over the base VLM while maintaining performance on in-distribution concepts. Ablation studies further confirm the necessity of pseudo-word optimization.

Paper Structure

This paper contains 10 sections, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Formulation
  5. Experiments
  6. Experimental Settings
  7. Few-Shot Learning
  8. Zero-Shot Learning
  9. Ablation Study
  10. Conclusion

Figures (2)

  • Figure 1: GTMA inference workflow: for an OOD visual anchor $\mathbf{c}_v$, iteratively optimize a pseudo-word embedding $z$ under visual guidance and semantic regularization, synthesizing textual representations for novel concept recognition.
  • Figure 2: Accuracy scales positively with the number of shots. Across all settings, GTMA maintains a consistent accuracy advantage over baselines due to its dynamic representation optimization.