You May Speak Freely: Improving the Fine-Grained Visual Recognition Capabilities of Multimodal Large Language Models with Answer Extraction

TL;DR

This work tackles the challenge of evaluating free-form outputs from multimodal large language models in fine-grained visual classification (FGVC). It introduces nlg2choice, a two-stage pipeline that first elicits an open-ended answer and then uses text-only constrained decoding to extract a final class, supplemented by an early-stopping mechanism to speed retrieval. Across seven FGVC datasets, nlg2choice yields consistent improvements in both zero-shot classification and retrieval, and proves robust to prompt variations without requiring additional training. The study also analyzes the extraction process, showing high out-of-the-box extraction performance for several models and highlighting the nature of remaining errors, with resources released to support further research.

Abstract

Despite the renewed interest in zero-shot visual classification due to the rise of Multimodal Large Language Models (MLLMs), the problem of evaluating free-form responses of auto-regressive models remains a persistent challenge. Most existing works focus on language-only tasks or don't consider Multiple Choice Questions (MCQs) beyond 5-way options, both of which are critical capabilities to solve tasks in Fine-Grained Visual Classification (FGVC) where choice counts are in the hundreds to thousands and the choices are highly related. Furthermore, in this highly multi-way MCQ setting it is not clear how to extend LLM choice extraction to retrieval-based problems, where computing probabilities over the choice set is computationally costly. In this work we investigate nlg2choice, a simple two-stage method which first asks the MLLM an open-ended question for the task with minimal constraints, then uses text-only constrained decoding to predict the most likely choice. In retrieval settings, we compute the probability of the constrained response taking that choice with an early stopping method to significantly improve throughput. Our results show improvement over a suite of seven fine-grained visual datasets when evaluating in terms of classification and retrieval, and show that this performance holds over the various ways that users of LLMs can implement tasks in natural language.

Figures (5)

• Figure 1: Proposed evaluation setup of nlg2choice. Whereas previous approaches for MLLM classification use methods like probability sequences or constrained decoding, nlg2choice first prompts the MLLM for a natural language response, then performs text-only constrained decoding on the text to extract classes. Accuracy is computed as an average over multiple rewritings of prompts as a robustness measure. Correct MLLM responses are shown in green whereas incorrect responses are red.
• Figure 2: Seemingly innocuous instruction changes greatly affect MLLM performance. When testing on the gull subset of CUB200cub_dataset, replacing one word can change the predictions of a model (left). When rewriting an instruction in a semantically equivalent way, performance varies substantially for Qwen-2.5VL (right). Each distribution comprises 15 semantically equivalent ways of writing the same instruction. These cases highlight the need to test LLMs over various equivalent inputs to ensure robustness.
• Figure 3: Difference in accuracies on the question-level between nlg2choice and constrained decoding on CUB200. Each distribution is made up of the performance difference when switching to nlg2choice over 15 questions. nlg2choice uses an open-ended prompt whereas constrained decoding uses an instruction to increase brevity ("... Answer with species only.").
• Figure 4: Breakdown of the small labeled set of natural language responses. "Other" contains two main categories, "Refused to answer" and "No discernable information." Examples of these are given in \ref{['sec:appendix:answer_extraction']}.
• Figure 5: Labeling interface for efficacy of the answer extraction stage of nlg2choice. Labelers are asked to highlight the name predicted in free-form responses from various models and whether the name occurs easily within the dataset choice list. Above shows an interesting example: for the Chipping Sparrow on the right, the model freely responded with "Painted Bunting" and answer extraction made an additional mistake by predicting "Scarlet Tanager."