Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Automatic VQA Evaluation Using Large Language Models

Oscar Mañas, Benno Krojer, Aishwarya Agrawal

TL;DR

This work tackles the limited alignment of VQA Accuracy with human judgments in open-ended and OOD VQA. It introduces LAVE, an LLM-assisted evaluation metric that scores candidate answers against reference answers using in-context learning and provides rationales, mapped to a 0–1 score. Across multiple models and benchmarks, LAVE demonstrates superior correlation with human judgments compared to strong baselines, with thorough ablations showing robustness to design choices and clear guidance on practical deployment. The study also analyzes VQA Accuracy failure modes and shows LAVE can recover many missed correct answers, offering a more reliable and interpretable gauge of progress in vision-language QA.

Abstract

8 years after the visual question answering (VQA) task was proposed, accuracy remains the primary metric for automatic evaluation. VQA Accuracy has been effective so far in the IID evaluation setting. However, our community is undergoing a shift towards open-ended generative models and OOD evaluation. In this new paradigm, the existing VQA Accuracy metric is overly stringent and underestimates the performance of VQA systems. Thus, there is a need to develop more robust automatic VQA metrics that serve as a proxy for human judgment. In this work, we propose to leverage the in-context learning capabilities of instruction-tuned large language models (LLMs) to build a better VQA metric. We formulate VQA evaluation as an answer-rating task where the LLM is instructed to score the accuracy of a candidate answer given a set of reference answers. We demonstrate the proposed metric better correlates with human judgment compared to existing metrics across several VQA models and benchmarks. We hope wide adoption of our metric will contribute to better estimating the research progress on the VQA task. We plan to release the evaluation code and collected human judgments.

Improving Automatic VQA Evaluation Using Large Language Models

TL;DR

This work tackles the limited alignment of VQA Accuracy with human judgments in open-ended and OOD VQA. It introduces LAVE, an LLM-assisted evaluation metric that scores candidate answers against reference answers using in-context learning and provides rationales, mapped to a 0–1 score. Across multiple models and benchmarks, LAVE demonstrates superior correlation with human judgments compared to strong baselines, with thorough ablations showing robustness to design choices and clear guidance on practical deployment. The study also analyzes VQA Accuracy failure modes and shows LAVE can recover many missed correct answers, offering a more reliable and interpretable gauge of progress in vision-language QA.

Abstract

8 years after the visual question answering (VQA) task was proposed, accuracy remains the primary metric for automatic evaluation. VQA Accuracy has been effective so far in the IID evaluation setting. However, our community is undergoing a shift towards open-ended generative models and OOD evaluation. In this new paradigm, the existing VQA Accuracy metric is overly stringent and underestimates the performance of VQA systems. Thus, there is a need to develop more robust automatic VQA metrics that serve as a proxy for human judgment. In this work, we propose to leverage the in-context learning capabilities of instruction-tuned large language models (LLMs) to build a better VQA metric. We formulate VQA evaluation as an answer-rating task where the LLM is instructed to score the accuracy of a candidate answer given a set of reference answers. We demonstrate the proposed metric better correlates with human judgment compared to existing metrics across several VQA models and benchmarks. We hope wide adoption of our metric will contribute to better estimating the research progress on the VQA task. We plan to release the evaluation code and collected human judgments.
Paper Structure (33 sections, 6 figures, 9 tables)

This paper contains 33 sections, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Metrics for VQA
  4. Metrics for GenQA
  5. Using LLMs as evaluators
  6. Analysis of VQA Accuracy Failure Modes
  7. Method
  8. Choosing a Large Language Model
  9. Prompt for VQA Evaluation
  10. Scoring Function
  11. Experiments
  12. Experimental Setup
  13. VQA models and benchmarks
  14. Baselines
  15. Implementation details
  16. ...and 18 more sections

Figures (6)

  • Figure 1: Existing VQA metrics and other strong baselines tend to miss out on correct answers generated by VQA models. Our proposed metric, LAVE, is more aligned with human judgment and provides a rationale for its rating, making it also more interpretable.
  • Figure 2: Example of a VQA Accuracy failure mode from the multiple answers category (Tab. \ref{['tab:failure_modes']}). Q: What are the sticks for? A: balance, pushing, skating, ...
  • Figure 3: VQA evaluation with an LLM via in-context learning.
  • Figure 4: Average score of VQA evaluation metrics for cases where VQA Accuracy misses correct candidate answers, broken down by failure mode category.
  • Figure 5: Amazon Mechanical Turk interface
  • ...and 1 more figures