Exploring Effective Factors for Improving Visual In-Context Learning

TL;DR

This paper investigates how to improve visual in-context learning by isolating two factors—prompt selection and prompt fusion—that govern performance. It introduces Prompt-SelF, a framework that uses pixel-level retrieval to select prompts, employs multiple fusion arrangements to activate diverse knowledge in large-scale vision models, and ensembles their predictions. Through extensive experiments on single-object segmentation and detection, Prompt-SelF achieves strong gains, including outperforming OSLSM in 1-shot segmentation and demonstrating robustness to domain shifts. The work highlights the untapped potential of prompt design in computer vision and provides actionable strategies for leveraging large pretrained models in context-based tasks.

Abstract

The In-Context Learning (ICL) is to understand a new task via a few demonstrations (aka. prompt) and predict new inputs without tuning the models. While it has been widely studied in NLP, it is still a relatively new area of research in computer vision. To reveal the factors influencing the performance of visual in-context learning, this paper shows that prompt selection and prompt fusion are two major factors that have a direct impact on the inference performance of visual context learning. Prompt selection is the process of identifying the most appropriate prompt or example to help the model understand new tasks. This is important because providing the model with relevant prompts can help it learn more effectively and efficiently. Prompt fusion involves combining knowledge from different positions within the large-scale visual model. By doing this, the model can leverage the diverse knowledge stored in different parts of the model to improve its performance on new tasks. Based these findings, we propose a simple framework prompt-SelF for visual in-context learning. Specifically, we first use the pixel-level retrieval method to select a suitable prompt, and then use different prompt fusion methods to activate all the knowledge stored in the large-scale model, and finally ensemble the prediction results obtained from different prompt fusion methods to obtain the final prediction results. And we conduct extensive experiments on single-object segmentation and detection tasks to demonstrate the effectiveness of prompt-SelF. Remarkably, the prompt-SelF has outperformed OSLSM based meta-learning in 1-shot segmentation for the first time. This indicated the great potential of visual in-context learning. The source code and models will be available at https://github.com/syp2ysy/prompt-SelF.

Figures (12)

• Figure 1: A framework for visual in-context learning. It involves the following steps. (a) Select the prompt pair corresponding to the task from the database based on the query image. (b) Fusion the query image and the selected prompt image to create a new image. (c) Feed new images into a pre-trained large-scale vision model to get predictions for query images. This paradigm enables the visual model to learn in-context by utilizing the knowledge captured from the prompt image to enhance the accuracy of the prediction for the demand image.
• Figure 2: (a) The selection of different prompts can affect the results of the same query image, therefore, choosing a high-quality prompt can have a positive impact on visual in-context learning. (b) Different prompt fusion methods activate knowledge at different positions in the large-scale model, leading to varying degrees of impact on visual in-context learning. Note that the red box in the figure is the prediction result, and the blue box is the prompt pair.
• Figure 3: Visualization results of different prompt fusion methods. A1-A8 respectively correspond to Arrangement1-Arrangement8 in Table \ref{['tab:diff_position']}, where the blue box is the prompt pair, and the red box is the prediction result. The best results are show in bold.
• Figure 4: Ablation study of thresholds in ensemble strategy. The denominator and numerator of the threshold indicate the complete count of arrangements and the minimum count needed to establish the final outcome, correspondingly.
• Figure 5: Visualization results of more prompts and ensemble. Increasing the number of prompts can significantly increase performance. Note that the red box in the figure is the prediction result, and the blue box is the prompt pair.