Enhancing Visual In-Context Learning by Multi-Faceted Fusion

TL;DR

This work tackles Visual In-Context Learning (VICL) limitations by moving beyond single-prompt retrieval and naive multi-prompt fusion. It introduces a three-branch contextual representation framework via Multiple Prompt Group Selection (MPGS) and a hierarchical, multi-branch decoder, MULTI-VQGAN, to jointly interpret diverse guidance signals. The method yields superior cross-task performance in segmentation, detection, and colorization, with strong cross-domain generalization (e.g., COCO-5^i to Pascal-5^i) and robustness to context variation. These results highlight the value of collaborative, structured fusion over simplistic prompt aggregation for unlocking VICL’s full reasoning potential.

Abstract

Visual In-Context Learning (VICL) has emerged as a powerful paradigm, enabling models to perform novel visual tasks by learning from in-context examples. The dominant "retrieve-then-prompt" approach typically relies on selecting the single best visual prompt, a practice that often discards valuable contextual information from other suitable candidates. While recent work has explored fusing the top-K prompts into a single, enhanced representation, this still simply collapses multiple rich signals into one, limiting the model's reasoning capability. We argue that a more multi-faceted, collaborative fusion is required to unlock the full potential of these diverse contexts. To address this limitation, we introduce a novel framework that moves beyond single-prompt fusion towards an multi-combination collaborative fusion. Instead of collapsing multiple prompts into one, our method generates three contextual representation branches, each formed by integrating information from different combinations of top-quality prompts. These complementary guidance signals are then fed into proposed MULTI-VQGAN architecture, which is designed to jointly interpret and utilize collaborative information from multiple sources. Extensive experiments on diverse tasks, including foreground segmentation, single-object detection, and image colorization, highlight its strong cross-task generalization, effective contextual fusion, and ability to produce more robust and accurate predictions than existing methods.

Figures (8)

• Figure 1: Overview and performance of our proposed framework.(I) and (II) illustrate the core pipeline, contrasting our collaborative fusion with MULTI-VQGAN (c) against the baseline MAE-VQGAN (a) and the CONDENSER (b). (III) and (IV) showcase the framework's superior performance through qualitative and quantitative comparisons on different tasks against existing methods.
• Figure 2: Pipeline of our framework in contrast to existing methods.(I) The single prompt method uses only the most similar prompt to guide a MAE-VQGAN. (II) The CONDENSER method trains a module to fuse multiple prompts into one, which then guides a frozen MAE-VQGAN. (III) Our proposed MULTI-VQGAN architecture. The support pairs are grouped by MPGS, then processed by the frozen prompt generator, and the resulting multiple fused prompts are used to train the MULTI-VQGAN for a more robust, hierarchical fusion.
• Figure 3: The architecture of MULTI-VQGAN, which performs hierarchical feature fusion with a multi-branch encoder.
• Figure 4: The architecture of the FUSE module.
• Figure 5: Ablation study on the key hyperparameters of the MULTI-VQGAN framework.