VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models

TL;DR

VisCoP addresses the sharp domain shift observed when applying Vision-Language Models to new visual domains by freezing the vision encoder and introducing a compact set of learnable visual probes that interact with intermediate VE features. Through layer-wise cross-attention, these probes extract domain-specific cues without perturbing pretrained representations, enabling robust cross-view, cross-modal, and cross-task adaptation while mitigating catastrophic forgetting. Empirical results across egocentric vs exocentric View, RGB vs depth, and human understanding vs robot control demonstrate substantial target-domain gains with minimal loss to source-domain capabilities, and ablations highlight the importance of dense, multi-layer probing. The work suggests a practical path for domain adaptation in multimodal models with strong real-world applicability and yields open-source code and data for follow-up research.

Abstract

Large Vision-Language Models (VLMs) excel at general visual reasoning tasks but exhibit sharp performance degradation when applied to novel domains with substantial distribution shifts from pretraining data. Existing domain adaptation approaches finetune different VLM components, but this often results in limited domain-specific feature learning or catastrophic forgetting of prior capabilities. To address these issues, we introduce Vision Contextualized Probing (VisCoP), which augments the VLM's vision encoder with a compact set of learnable visual probes. These probes enable efficient domain-specific adaptation with minimal modification to pretrained parameters. We evaluate VisCoP across three challenging domain adaptation settings-cross-view (exocentric to egocentric), cross-modal (RGB to depth), and cross-task (human understanding to robot control). Experiments show that VisCoP consistently outperforms existing adaptation strategies, achieving superior performance on target domains while effectively retaining source-domain knowledge.

Figures (14)

• Figure 1: Domain adaptation performance of different adaptation strategies.VisCoP achieves superior target domain performance while better retaining source domain knowledge compared to other strategies.
• Figure 2: Architecture of our proposed VisCoP. Learnable visual probes are conditioned on intermediate representations of a frozen vision encoder through vision-probe cross-attention, which extracts domain-specific features that may have otherwise been discarded by the frozen encoder.
• Figure 3: Ablation on the number of visual probes in VisCoP.
• Figure 4: Ablation on the positions of interaction modules in VisCoP.
• Figure 5: Ablation on alternative designs of VisCoP. VE annotations: VP (visual probes and probe connector with no interaction modules), Last-4 (train only the last 4 vision encoder layers), QFormer Style. (interaction module is placed only at the last layer of the VE).