Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning

TL;DR

The paper addresses the limitation of single-prompt selection in Visual In-Context Learning by introducing prompt condensation, where multiple prompts collaborate through the Condenser plugin to provide rich, high-resolution context. The Condenser performs end-to-end, patch-wise cross-attention across candidate prompts to produce a condensed prompt, optimized with two learning objectives: token prediction for query labels and a pre-alignment term to align condensed prompts with the query. Experiments on foreground segmentation, object detection, and image colorization show Condenser outperforms state-of-the-art baselines, with better scalability as the number of prompts increases and enhanced efficiency compared to ensemble approaches. This approach demonstrates a practical pathway to improve VICL generalization and efficiency, enabling more robust use of diverse contextual cues in vision tasks.

Abstract

Visual In-Context Learning (VICL) enables adaptively solving vision tasks by leveraging pixel demonstrations, mimicking human-like task completion through analogy. Prompt selection is critical in VICL, but current methods assume the existence of a single "ideal" prompt in a pool of candidates, which in practice may not hold true. Multiple suitable prompts may exist, but individually they often fall short, leading to difficulties in selection and the exclusion of useful context. To address this, we propose a new perspective: prompt condensation. Rather than relying on a single prompt, candidate prompts collaborate to efficiently integrate informative contexts without sacrificing resolution. We devise Condenser, a lightweight external plugin that compresses relevant fine-grained context across multiple prompts. Optimized end-to-end with the backbone, Condenser ensures accurate integration of contextual cues. Experiments demonstrate Condenser outperforms state-of-the-arts across benchmark tasks, showing superior context compression, scalability with more prompts, and enhanced computational efficiency compared to ensemble methods, positioning it as a highly competitive solution for VICL. Code is open-sourced at https://github.com/gimpong/CVPR25-Condenser.

Figures (5)

• Figure 1: (a) The impainting framework bar2022visual is a typical prototype for Visual In-Context Learning (VICL), where a fixed-size input canvas is divided into regions. The prompt image and its label occupy the top half, while the query image is placed in the bottom left. The vision backbone is tasked with recovering the bottom right region, which represents the query label, by reasoning within the pixel context. Prompt selection plays a crucial role in constructing the input and significantly impacts model predictions. (b) Most existing VICL methods view prompt selection as a competitive process, where a single prompt is chosen, often discarding useful alternatives. Meanwhile, the top-1 prompt may not be ideal for the query image. (c) We introduce the concept of collaborative condensation, where multiple raw prompts are combined to leverage their individual strengths into a single condensed prompt. The proposed Condenser is a lightweight, flexible, and efficient plugin to the vision backbone, effective across a range of VICL tasks.
• Figure 2: Prompt condensation with Condenser for VICL. Given a query image $I_q$ from the training set, we first retrieve a set of candidate prompts $\mathcal{S}_q$ from the prompt database $\mathcal{P}$. Then, we assemble each prompt from $\mathcal{S}_q$ with the labeled query pair $P_q=(I_q,L_q)$ and construct $K$ contextualized labeled samples, namely $\tilde{X}_{c_1},\cdots,\tilde{X}_{c_K}$. We encode each of them with a VQGAN encoder VQGAN_2021$h(\cdot)$ and obtain a serial of query label tokens, $\bm{T}_{q_{c_1}}^L,\cdots,\bm{T}_{q_{c_K}}^L$ as signals. Next, we transform and condense raw candidate prompts $\mathcal{S}_q$ into an integrated prompt, $(\bm{F}_{c*}^I,\bm{F}_{c*}^L)$, using the proposed Condenser. The patches of condensed prompt and the query image form the contextualized unlabeled sample $\bm{F}_{c*}^X$, which is subsequently processed by the vision backbone $f(\cdot)$ to predict masked tokens $\hat{\bm{T}}_{q_{c_*}}^L$. Finally, we incorporate token prediction and pre-alignment between the condensed prompt and the labeled query to guide optimization.
• Figure 3: MAE-VQGAN bar2022visual neatly combines MAE he2022masked and VQGAN VQGAN_2021 for inpainting. MAE subsequently acts as the backbone in VICL. (a) For pre-training, the MAE is trained to predict the masked tokens. (b) For inference, the VQGAN decoder decodes pixel results from MAE's predicted tokens.
• Figure 4: Condenser design. The cross-prompt attention performs patch-wise cross-attention to aggregate informative context among the $K$ prompts. LN: layer normalization. P: permutation.
• Figure 5: Impact of the prompt number $K$ on foreground segmentation, single object detection.