Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities

TL;DR

The work systematically re-evaluates image classification in Multimodal Large Language Models, showing that recent MLLMs can rival or surpass CLIP-style models on a broad set of benchmarks. By conducting targeted ablations across network architecture, training data, and training recipe, it attributes gains primarily to enhanced LLM conceptual knowledge and richer exposure to target concepts through diverse data. The study introduces a 26-option MCQ evaluation framework and demonstrates that stronger LLMs (e.g., Qwen2) and domain-specific data (e.g., SI-3.2M) yield significant accuracy gains, including substantial improvements on hard classes. These findings offer practical guidance for designing and evaluating MLLMs for image classification and highlight the importance of data diversity and advanced language models in transfer of conceptual knowledge.

Abstract

With the rapid advancement of Multimodal Large Language Models (MLLMs), a variety of benchmarks have been introduced to evaluate their capabilities. While most evaluations have focused on complex tasks such as scientific comprehension and visual reasoning, little attention has been given to assessing their fundamental image classification abilities. In this paper, we address this gap by thoroughly revisiting the MLLMs with an in-depth analysis of image classification. Specifically, building on established datasets, we examine a broad spectrum of scenarios, from general classification tasks (e.g., ImageNet, ObjectNet) to more fine-grained categories such as bird and food classification. Our findings reveal that the most recent MLLMs can match or even outperform CLIP-style vision-language models on several datasets, challenging the previous assumption that MLLMs are bad at image classification \cite{VLMClassifier}. To understand the factors driving this improvement, we conduct an in-depth analysis of the network architecture, data selection, and training recipe used in public MLLMs. Our results attribute this success to advancements in language models and the diversity of training data sources. Based on these observations, we further analyze and attribute the potential reasons to conceptual knowledge transfer and enhanced exposure of target concepts, respectively. We hope our findings will offer valuable insights for future research on MLLMs and their evaluation in image classification tasks.

Figures (2)

• Figure 1: Specific and overall comparisons of image classification. (a) LLaVA-OV llava-ov handles well on bad cases, such as "black-footed ferret" recognition, than previous LLaVA-1.5 llava. (b) Recent proposed MLLMs obtain comparable or even better classification results on ObjectNet barbu2019objectnet dataset than SigLIP siglip.
• Figure 2: Performance comparison of public MLLMs on various classification and MLLM benchmarks. Here, we compare the established LLaVA-1.5 llava, Phi-3-Vision phi3vision, recent LLaVA-OV llava-ov and Qwen2-VL qwen2vl on a total of ten conventional image datasets (detailed in Table \ref{['tab:data_classification']}), such as ImageNet deng2009imagenet and ObjectNet barbu2019objectnet for general classification evaluation, as well as CUB200 cub and Food101 bossard2014food for fine-grained scenarios. We also report the results on ten well-established MLLM benchmarks, covering evaluation with four common categories tong2024cambrian, including general, knowledge, chart & OCR, and vision-centric. Best view in color.