Rethinking VLMs and LLMs for Image Classification

TL;DR

A lightweight fix involving a relatively small LLM that efficiently routes visual tasks to the most suitable model for the task is proposed, which surpasses or matches the accuracy of state-of-the-art alternatives, including GPT-4V and HuggingGPT, while improving cost-effectiveness.

Abstract

Visual Language Models (VLMs) are now increasingly being merged with Large Language Models (LLMs) to enable new capabilities, particularly in terms of improved interactivity and open-ended responsiveness. While these are remarkable capabilities, the contribution of LLMs to enhancing the longstanding key problem of classifying an image among a set of choices remains unclear. Through extensive experiments involving seven models, ten visual understanding datasets, and multiple prompt variations per dataset, we find that, for object and scene recognition, VLMs that do not leverage LLMs can achieve better performance than VLMs that do. Yet at the same time, leveraging LLMs can improve performance on tasks requiring reasoning and outside knowledge. In response to these challenges, we propose a pragmatic solution: a lightweight fix involving a relatively small LLM that efficiently routes visual tasks to the most suitable model for the task. The LLM router undergoes training using a dataset constructed from more than 2.5 million examples of pairs of visual task and model accuracy. Our results reveal that this lightweight fix surpasses or matches the accuracy of state-of-the-art alternatives, including GPT-4V and HuggingGPT, while improving cost-effectiveness.

Figures (9)

• Figure 1: VLMs achieve higher performance on object and scene recognition tasks than VLM+LLMs with the same vision encoder. Q: prompt provided to the models. RO: response options provided to the models. Left: on tasks involving visual reasoning and outside knowledge the VLM+LLM (Flamingo) outperforms the VLM (CLIP) with the same vision encoder. Right: on object and scene recognition the VLM is superior to the VLM+LLM. Note that Flamingo only uses CLIP's vision encoder, but since this vision encoder is pre-trained with the full CLIP architecture, in the figure we show CLIP as part of Flamingo.
• Figure 2: Input prompts affect model performance but mostly do not affect winning model type. a) Model performance for each model and prompt combination (markers) per dataset (y-axis). The accuracy varies across prompts, in particular for VLM+LLM models in recognition tasks (yellow markers in the first three rows). b) Proportion of times a VLM or VLM+LLM wins, across all prompts. Most datasets (y-axis) show a proportion of 1 for either VLM or VLM+LLM, indicating the winning model type is not affected by the specific prompt.
• Figure 3: Our router approach where an LLM selects a suitable VLM/VLM+LLM to obtain high accuracy. The user input is first provided to the LLM router (1), which uses this information to select a model. The chosen model (2) is then provided with the user input (3) and generates a prediction (4).
• Figure 4: Heat maps of the LLM router's model selection distribution (left) and the distribution of models in the training data (right).
• Figure 5: Example samples from the datasets used to evalute VLMs and VLM+LLMs.