T2T-VICL: Unlocking the Boundaries of Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs

TL;DR

The paper addresses the boundary of visual in-context learning when the visual prompts and query images come from different tasks. It proposes T2T-VICL, a collaborative pipeline that leverages a large VLM to generate implicit cross-task descriptions, and a small VLM to learn a scalable prompt generation mechanism, enabling cross-task VICL without fine-tuning. A novel cross-task implicit-description dataset is created, and a score-based reasoning framework using VIEScore guides inference, complemented by traditional metrics like PSNR and SSIM. The approach achieves top-tier results across nine cross-task scenarios and strong performance in additional cases, demonstrating robust cross-task transfer and generalization while highlighting the value of semantic coherence over pixel-perfect fidelity for cross-task reasoning in VLMs.

Abstract

In large language models (LLM), in-context learning (ICL) refers to performing new tasks by conditioning on small demonstrations provided in the input context. Recent advances in visual in-context learning (VICL) demonstrate promising capabilities for solving downstream tasks by unified vision-language models (VLMs). When the visual prompt and the target images originate from different visual tasks, can VLMs still enable VICL? In the paper, we propose a fully collaborative pipeline, i.e. T2T-VICL, for VLMs to investigate the potential of cross-task VICL. Fundamentally, we design a mechanism to generate and select text prompts that best implicitly describe the differences between two distinct low-level vision tasks, and construct the first cross-task VICL dataset. Building upon this, we propose a novel inference framework that combines perceptual score-based reasoning with traditional evaluation metrics to perform cross-task VICL. Our approach achieves top-tier results across nine cross-task scenarios and second-tier performance in ten additional scenarios, unlocking the boundaries of cross-task VICL within VLMs.

Figures (4)

• Figure 1: Illustration of Cross-Task Visual In-Context Learning
• Figure 2: Overview of the Proposed Pipeline and its Workflow
• Figure 3: Representative Examples of Cross-Task In-Context Learning in Nine Pairs
• Figure 4: Representative Examples of Cross-Task In-Context Learning from the Ten Pairs