Beyond Single Prompts: Synergistic Fusion and Arrangement for VICL

TL;DR

This work tackles two limitations of Visual In-Context Learning: dependence on a single, most-similar prompt and the neglect of prompt arrangements. It introduces an end-to-end VICL framework with an adaptive Fusion Module to fuse multiple prompts, arrangement-specific lightweight MLPs to encode layout priors, and a bidirectional query–support fine-tuning loop to reinforce collaboration between fusion and inpainting. The approach yields state-of-the-art or near-state-of-the-art results across foreground segmentation, single-object detection, and image colorization, with strong cross-domain generalization (e.g., COCO-5^i to Pascal-5^i). Overall, the method demonstrates robust, scalable VICL capable of leveraging diverse prompts and layouts for improved contextual understanding and reconstruction.

Abstract

Vision In-Context Learning (VICL) enables inpainting models to quickly adapt to new visual tasks from only a few prompts. However, existing methods suffer from two key issues: (1) selecting only the most similar prompt discards complementary cues from other high-quality prompts; and (2) failing to exploit the structured information implied by different prompt arrangements. We propose an end-to-end VICL framework to overcome these limitations. Firstly, an adaptive Fusion Module aggregates critical patterns and annotations from multiple prompts to form more precise contextual prompts. Secondly, we introduce arrangement-specific lightweight MLPs to decouple layout priors from the core model, while minimally affecting the overall model. In addition, an bidirectional fine-tuning mechanism swaps the roles of query and prompt, encouraging the model to reconstruct the original prompt from fused context and thus enhancing collaboration between the fusion module and the inpainting model. Experiments on foreground segmentation, single-object detection, and image colorization demonstrate superior results and strong cross-task generalization of our method.

Figures (6)

• Figure 1: (I) The standard VICL pipeline, which essentially performs image inpainting guided by support prompts. (II) One critical factor of VICL, prompt selection. While more similar prompts yield better results, our fused prompt surpasses even the most similar one. (III) Another critical factor, prompt arrangement. Varying layouts influence the model inpainting in distinct ways. (IV) Our proposed joint fine-tuning strategy. (V, VI) Quantitative and qualitative results demonstrating our model's state-of-the-art performance and generalization.
• Figure 2: MAE-VQGAN Overview.(a) Training: ViT-MAE predicts tokens for masked patches, supervised by a frozen VQGAN encoder. (b) Inference: The model generates tokens from a masked input, which are decoded by VQGAN.
• Figure 3: Three-stage training pipeline.(I) Preliminary Fusion Training: The Fusion Module is first trained independently to establish a strong fusion capability. (II) Arrangement-Specific MLP Training: Lightweight MLPs are trained for various geometric layouts to efficiently identify the optimal arrangements. (III) Joint Fine-tuning: Finally, the whole model is fine-tuned on the optimal arrangements using a support–query swapping strategy to enhance overall performance and generalization.
• Figure 4: Ablation studies on key hyperparameters and module generalization.
• Figure 5: Visualization across tasks, showing progressive improvements from adding the Fusion Module, the MLP, and joint fine-tuning.