Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Incomplete In-context Learning

Wenqiang Wang, Yangshijie Zhang

TL;DR

The paper addresses incomplete retrieval databases in vision-language in-context learning (VICL) by formalizing Incomplete In-context Learning (IICL) and introducing IJIP, a two-stage framework. The Iterative Judgment Stage reframes an $m$-class problem into $m$ binary subproblems using retrieved demonstrations, and the Integrated Prediction Stage consolidates these binary judgments with the input to produce a final label, even when data for some classes is missing. IJIP demonstrates strong performance across two LVLMs and datasets, achieving up to 93.9% accuracy under missing-label conditions and maintaining competitive results even with only a single label available; it also extends to complete VICL, prompt learning, and text modalities. The work provides a practical, versatile approach for robust in-context learning under data incompleteness, with implications for real-world deployment and cross-domain adaptation.

Abstract

Large vision language models (LVLMs) achieve remarkable performance through Vision In-context Learning (VICL), a process that depends significantly on demonstrations retrieved from an extensive collection of annotated examples (retrieval database). Existing studies often assume that the retrieval database contains annotated examples for all labels. However, in real-world scenarios, delays in database updates or incomplete data annotation may result in the retrieval database containing labeled samples for only a subset of classes. We refer to this phenomenon as an \textbf{incomplete retrieval database} and define the in-context learning under this condition as \textbf{Incomplete In-context Learning (IICL)}. To address this challenge, we propose \textbf{Iterative Judgments and Integrated Prediction (IJIP)}, a two-stage framework designed to mitigate the limitations of IICL. The Iterative Judgments Stage reformulates an \(\boldsymbol{m}\)-class classification problem into a series of \(\boldsymbol{m}\) binary classification tasks, effectively converting the IICL setting into a standard VICL scenario. The Integrated Prediction Stage further refines the classification process by leveraging both the input image and the predictions from the Iterative Judgments Stage to enhance overall classification accuracy. IJIP demonstrates considerable performance across two LVLMs and two datasets under three distinct conditions of label incompleteness, achieving the highest accuracy of 93.9\%. Notably, even in scenarios where labels are fully available, IJIP still achieves the best performance of all six baselines. Furthermore, IJIP can be directly applied to \textbf{Prompt Learning} and is adaptable to the \textbf{text domain}.

Incomplete In-context Learning

TL;DR

The paper addresses incomplete retrieval databases in vision-language in-context learning (VICL) by formalizing Incomplete In-context Learning (IICL) and introducing IJIP, a two-stage framework. The Iterative Judgment Stage reframes an -class problem into binary subproblems using retrieved demonstrations, and the Integrated Prediction Stage consolidates these binary judgments with the input to produce a final label, even when data for some classes is missing. IJIP demonstrates strong performance across two LVLMs and datasets, achieving up to 93.9% accuracy under missing-label conditions and maintaining competitive results even with only a single label available; it also extends to complete VICL, prompt learning, and text modalities. The work provides a practical, versatile approach for robust in-context learning under data incompleteness, with implications for real-world deployment and cross-domain adaptation.

Abstract

Large vision language models (LVLMs) achieve remarkable performance through Vision In-context Learning (VICL), a process that depends significantly on demonstrations retrieved from an extensive collection of annotated examples (retrieval database). Existing studies often assume that the retrieval database contains annotated examples for all labels. However, in real-world scenarios, delays in database updates or incomplete data annotation may result in the retrieval database containing labeled samples for only a subset of classes. We refer to this phenomenon as an \textbf{incomplete retrieval database} and define the in-context learning under this condition as \textbf{Incomplete In-context Learning (IICL)}. To address this challenge, we propose \textbf{Iterative Judgments and Integrated Prediction (IJIP)}, a two-stage framework designed to mitigate the limitations of IICL. The Iterative Judgments Stage reformulates an -class classification problem into a series of binary classification tasks, effectively converting the IICL setting into a standard VICL scenario. The Integrated Prediction Stage further refines the classification process by leveraging both the input image and the predictions from the Iterative Judgments Stage to enhance overall classification accuracy. IJIP demonstrates considerable performance across two LVLMs and two datasets under three distinct conditions of label incompleteness, achieving the highest accuracy of 93.9\%. Notably, even in scenarios where labels are fully available, IJIP still achieves the best performance of all six baselines. Furthermore, IJIP can be directly applied to \textbf{Prompt Learning} and is adaptable to the \textbf{text domain}.
Paper Structure (30 sections, 13 equations, 5 figures, 8 tables)

This paper contains 30 sections, 13 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Incomplete In-context Learning
  3. Problem Formulation
  4. Empirical Study of IICL
  5. Observation and Motivation
  6. Method
  7. Iterative Judgment Stage
  8. Analysis of Iterative Judgments Stage
  9. Transforming Incomplete In-context Learning into In-context Learning.
  10. Reduction in Classification Complexity: $m$ Binary Classification Tasks vs. a Single $m$-Class Classification Task.
  11. Integrated Prediction Stage
  12. Analysis of Integrated Prediction Stage
  13. Experiments
  14. Experimental Setup
  15. Main Results
  16. ...and 15 more sections

Figures (5)

  • Figure 1: The overview of incomplete retrieval database, IICL, and sequence of binary classifications.
  • Figure 2: IJIP utilizes the CLIP model to vectorize images and retrieves the top $k$ most similar images from the incomplete retrieval database as demonstrations based on their similarity to the input image. In the Iterative Judgment Stage, IJIP first queries the LVLMs to classify the input image using the retrieved demonstrations. Specifically, IJIP identifies labels receiving positive responses as candidate classes. If no labels receive positive judgments, IJIP requests the LVLMs to perform a full-label classification. Conversely, if multiple labels receive positive judgments, IJIP initiates a second inquiry limited to these positively judged labels to refine the decision. Therefore, IJIP queries LVLMs at most twice and at least once, depending on the initial judgments.
  • Figure 3: Subfigure (a) shows the empirical study of IICL with different missing label numbers. Subfigure (b) shows the empirical study of IICL with different VICL methods. Subfigure (c) shows the empirical study of IICL with different demonstration numbers.
  • Figure 4: The accuracies(%)↑ of different demonstration numbers. Each experiment is conducted three times, and the average result is reported.
  • Figure 5: The effectiveness of IJIP in improving zero-shot prompt learning performance. Each experiment is conducted three times, and the average result is reported.

Theorems & Definitions (1)

  • Definition 2.1: Incomplete In-context Learning (IICL)