On Large Multimodal Models as Open-World Image Classifiers

TL;DR

This work tackles open-world image classification with Large Multimodal Models (LMMs) by formalizing the OW task and establishing a four-m metrics evaluation protocol. It conducts a large-scale study of 13 LMMs across 10 benchmarks spanning prototypical to very fine-grained classes, comparing against open-world baselines and CLIP-based methods. Findings show LMMs outperform some contrastive baselines in OW but still lag behind closed-world models on several semantic metrics, with performance highly sensitive to granularity and prompting. The authors demonstrate that tailoring prompts and incorporating test-time reasoning can mitigate certain errors, and they provide an extensive evaluation suite to guide future improvements in open-world vision-language classification.

Abstract

Traditional image classification requires a predefined list of semantic categories. In contrast, Large Multimodal Models (LMMs) can sidestep this requirement by classifying images directly using natural language (e.g., answering the prompt "What is the main object in the image?"). Despite this remarkable capability, most existing studies on LMM classification performance are surprisingly limited in scope, often assuming a closed-world setting with a predefined set of categories. In this work, we address this gap by thoroughly evaluating LMM classification performance in a truly open-world setting. We first formalize the task and introduce an evaluation protocol, defining various metrics to assess the alignment between predicted and ground truth classes. We then evaluate 13 models across 10 benchmarks, encompassing prototypical, non-prototypical, fine-grained, and very fine-grained classes, demonstrating the challenges LMMs face in this task. Further analyses based on the proposed metrics reveal the types of errors LMMs make, highlighting challenges related to granularity and fine-grained capabilities, showing how tailored prompting and reasoning can alleviate them.

Figures (13)

• Figure 1: We extensively test 13 Large Multimodal Models (LMMs) for Open-World (OW) classification on 10 datasets using four evaluation metrics. We show that LMMs outperform contrastive-based approaches in OW (CaSED conti2023vocabulary, and CLIP radford2021clip with image-to-text retrieval) but still lag behind closed-world models with fixed categories (CLIP radford2021clip, dashed line).
• Figure 2: Per-image examples of model predictions. Bold indicates the ground truth class names. For visualization purposes, we show only the predictions with the highest/lowest concept similarity. Blue and red indicate positive and negative Llama inclusion values.
• Figure 3: Per-class examples of model predictions. Bold indicates the ground truth class names. On the x-axis we report the average LI, and on the y-axis the average CS. For visualization purposes, we show the most frequent concepts predicted for each quadrant.
• Figure 4: Types of model predictions per dataset groups. Blue indicates correct and specific and correct but generic predictions, red indicates wrong but specific and wrong and generic mistakes.
• Figure 5: Percentage of correct and specific predictions shared between models. Higher values indicate higher agreement.