Large Multimodal Models as General In-Context Classifiers

TL;DR

CIRCLE is proposed, a simple training-free method that assigns pseudo-labels to in-context examples, iteratively refining them with the available context itself, establishing a robust baseline for open-world classification and highlighting the potential of LMMs to serve as unified classifiers, and a flexible alternative to specialized models.

Abstract

Which multimodal model should we use for classification? Previous studies suggest that the answer lies in CLIP-like contrastive Vision-Language Models (VLMs), due to their remarkable performance in zero-shot classification. In contrast, Large Multimodal Models (LMM) are more suitable for complex tasks. In this work, we argue that this answer overlooks an important capability of LMMs: in-context learning. We benchmark state-of-the-art LMMs on diverse datasets for closed-world classification and find that, although their zero-shot performance is lower than CLIP's, LMMs with a few in-context examples can match or even surpass contrastive VLMs with cache-based adapters, their "in-context" equivalent. We extend this analysis to the open-world setting, where the generative nature of LMMs makes them more suitable for the task. In this challenging scenario, LMMs struggle whenever provided with imperfect context information. To address this issue, we propose CIRCLE, a simple training-free method that assigns pseudo-labels to in-context examples, iteratively refining them with the available context itself. Through extensive experiments, we show that CIRCLE establishes a robust baseline for open-world classification, surpassing VLM counterparts and highlighting the potential of LMMs to serve as unified classifiers, and a flexible alternative to specialized models.

Figures (13)

• Figure 1: The role of context in classification. CLIP-like models outperform Large Multimodal Models (LMMs) in closed-world classification. However, we show that context dramatically unlocks LMM performance in closed-world classification while allowing them to surpass VLMs on open-world classification.
• Figure 2: Sample efficiency. We visualize the relative improvement of a $k$-shot context w.r.t. the corresponding zero-shot model. For contrastive VLMs (dashed lines), we use Tip-Adapter zhang2021tip. For LMMs (solid lines), a simple Vanilla ICL setup. We report both the Qwen family (top row) and the Phi series (bottom row). LMMs benefit much more from additional context than VLMs on most datasets, with peaks of up to $>+50\%$ (e.g., Qwen2-VL-7B on Stanford Cars, Phi3.5-V on Flowers102). In contrast, CLIP-ViT-B/32 peaks at $\approx +25\%$.
• Figure 3: Concept similarity issues.bCS can be misleading, for instance by favoring comprehensive yet ungrounded lists of candidates (64 for Vanilla and CIRCLE). mCS instead rewards more coherent and precise answers (50 for Vanilla, 59 for CIRCLE).
• Figure 4: Qualitative results. We visualize some qualitative results on examples from UCF101, FGVC Aircraft, and Food101. Underline indicates a correct prediction of the ground truth.
• Figure 5: Purple denotes our default configuration, applied to Qwen2-VL 7B wang2024qwen2. We report semantic similarity (SS), Concept Similarity (bCS), Median Concept Similarity (mCS), and Llama Inclusion (LI). (a) Ablation on context size. Results are reported for Pseudo-labeling (PL) and CIRCLE, with 4, 8, and 16 shots. (b) Ablation on CIRCLE rounds. We report results comparing Pseudo-labeling against increasing numbers of CIRCLE refinement rounds. See the Supp. Mat. for an extended version of these results.