Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions

Ting-Yao Hsu, Chieh-Yang Huang, Ryan Rossi, Sungchul Kim, C. Lee Giles, Ting-Hao K. Huang

TL;DR

This work tackles the challenging problem of evaluating scientific figure captions without references or domain-expert human labor by using pre-trained LLMs, notably GPT-4, as zero-shot evaluators. It introduces SciCap-Eval, a 3,600-caption benchmark derived from 600 arXiv figures with expert and undergraduate human judgments, and demonstrates that GPT-4's ratings correlate strongly with Ph.D. rankings (Kendall ≈ $0.401$). The study shows GPT-4 outperforms undergraduates and other models in assessing helpfulness, and it can effectively identify low-quality captions to inform data cleaning. These findings offer a scalable, cost-effective approach to evaluating and improving caption-generation systems in scientific contexts, with potential extensions to personalization and factuality checks.

Abstract

There is growing interest in systems that generate captions for scientific figures. However, assessing these systems output poses a significant challenge. Human evaluation requires academic expertise and is costly, while automatic evaluation depends on often low-quality author-written captions. This paper investigates using large language models (LLMs) as a cost-effective, reference-free method for evaluating figure captions. We first constructed SCICAP-EVAL, a human evaluation dataset that contains human judgments for 3,600 scientific figure captions, both original and machine-made, for 600 arXiv figures. We then prompted LLMs like GPT-4 and GPT-3 to score (1-6) each caption based on its potential to aid reader understanding, given relevant context such as figure-mentioning paragraphs. Results show that GPT-4, used as a zero-shot evaluator, outperformed all other models and even surpassed assessments made by Computer Science and Informatics undergraduates, achieving a Kendall correlation score of 0.401 with Ph.D. students rankings

GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions

TL;DR

This work tackles the challenging problem of evaluating scientific figure captions without references or domain-expert human labor by using pre-trained LLMs, notably GPT-4, as zero-shot evaluators. It introduces SciCap-Eval, a 3,600-caption benchmark derived from 600 arXiv figures with expert and undergraduate human judgments, and demonstrates that GPT-4's ratings correlate strongly with Ph.D. rankings (Kendall ≈ ). The study shows GPT-4 outperforms undergraduates and other models in assessing helpfulness, and it can effectively identify low-quality captions to inform data cleaning. These findings offer a scalable, cost-effective approach to evaluating and improving caption-generation systems in scientific contexts, with potential extensions to personalization and factuality checks.

Abstract

There is growing interest in systems that generate captions for scientific figures. However, assessing these systems output poses a significant challenge. Human evaluation requires academic expertise and is costly, while automatic evaluation depends on often low-quality author-written captions. This paper investigates using large language models (LLMs) as a cost-effective, reference-free method for evaluating figure captions. We first constructed SCICAP-EVAL, a human evaluation dataset that contains human judgments for 3,600 scientific figure captions, both original and machine-made, for 600 arXiv figures. We then prompted LLMs like GPT-4 and GPT-3 to score (1-6) each caption based on its potential to aid reader understanding, given relevant context such as figure-mentioning paragraphs. Results show that GPT-4, used as a zero-shot evaluator, outperformed all other models and even surpassed assessments made by Computer Science and Informatics undergraduates, achieving a Kendall correlation score of 0.401 with Ph.D. students rankings
Paper Structure (28 sections, 3 figures, 4 tables)

This paper contains 28 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. SciCap-Eval Dataset
  4. Human assessments by Ph.D. students on a six-item ranking task.
  5. Human assessments by undergraduate students on a three-class labeling task.
  6. Methods
  7. Zero-Shot and Few-Shot Prompting with LLMs.
  8. Chain-of-Thought Prompting wei2022chain (QA, QA-YN).
  9. Classifiers learned from SciCap-Eval data.
  10. Experimental Results
  11. Experimental setups.
  12. GPT-4 prompting achieved the highest correlations with Ph.D. students' assessments.
  13. GPT-4 is better at spotting bad examples than selecting good examples.
  14. Discussion
  15. Do we need paragraphs that mention figures for this approach?
  16. ...and 13 more sections

Figures (3)

  • Figure 1: A cost-effective, reference-free evaluation of scientific figure captions using LLMs. We prompted LLMs to score (1-6) a caption based on its potential to aid reader understanding, given relevant context such as figure-mentioning paragraphs.
  • Figure 2: The interface used by the Ph.D. students to rank the captions.
  • Figure 3: The interface used by the undergraduate students to rate the captions.