Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GPT-5 Model Corrected GPT-4V's Chart Reading Errors, Not Prompting

Kaichun Yang, Jian Chen

TL;DR

This study evaluates how LLM architecture versus prompting influences chart-reading performance by testing GPT-5, GPT-4o, and GPT-4V on 107 CHART-6 questions under three prompting regimes. It finds that GPT-5 substantially outperforms the other models, while prompting variations—including chart-descriptions—provide only modest or inconsistent gains. Using bootstrap CIs and mixed-effects models, the analysis attributes the bulk of accuracy differences to model capability rather than prompting strategy. The findings imply that architectural advances drive chart-understanding performance, with limited immediate benefits from prompt engineering alone, guiding future evaluation and deployment of visualization-focused LLMs.

Abstract

We present a quantitative evaluation to understand the effect of zero-shot large-language model (LLMs) and prompting uses on chart reading tasks. We asked LLMs to answer 107 visualization questions to compare inference accuracies between the agentic GPT-5 and multimodal GPT-4V, for difficult image instances, where GPT-4V failed to produce correct answers. Our results show that model architecture dominates the inference accuracy: GPT5 largely improved accuracy, while prompt variants yielded only small effects. Pre-registration of this work is available here: https://osf.io/u78td/?view_only=6b075584311f48e991c39335c840ded3; the Google Drive materials are here:https://drive.google.com/file/d/1ll8WWZDf7cCNcfNWrLViWt8GwDNSvVrp/view.

GPT-5 Model Corrected GPT-4V's Chart Reading Errors, Not Prompting

TL;DR

This study evaluates how LLM architecture versus prompting influences chart-reading performance by testing GPT-5, GPT-4o, and GPT-4V on 107 CHART-6 questions under three prompting regimes. It finds that GPT-5 substantially outperforms the other models, while prompting variations—including chart-descriptions—provide only modest or inconsistent gains. Using bootstrap CIs and mixed-effects models, the analysis attributes the bulk of accuracy differences to model capability rather than prompting strategy. The findings imply that architectural advances drive chart-understanding performance, with limited immediate benefits from prompt engineering alone, guiding future evaluation and deployment of visualization-focused LLMs.

Abstract

We present a quantitative evaluation to understand the effect of zero-shot large-language model (LLMs) and prompting uses on chart reading tasks. We asked LLMs to answer 107 visualization questions to compare inference accuracies between the agentic GPT-5 and multimodal GPT-4V, for difficult image instances, where GPT-4V failed to produce correct answers. Our results show that model architecture dominates the inference accuracy: GPT5 largely improved accuracy, while prompt variants yielded only small effects. Pre-registration of this work is available here: https://osf.io/u78td/?view_only=6b075584311f48e991c39335c840ded3; the Google Drive materials are here:https://drive.google.com/file/d/1ll8WWZDf7cCNcfNWrLViWt8GwDNSvVrp/view.

Paper Structure

This paper contains 24 sections, 1 equation, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Quantifying Observers by Visualization Literacy
  4. Quantifying LLM's Prompt Design
  5. Methodology
  6. Dataset and Sampling
  7. Independent variables
  8. LLMs
  9. Prompting Conditions
  10. Generation of Graph Description
  11. Experimental Setup and Data Analysis
  12. Results
  13. Statistical analysis
  14. Confidence intervals.
  15. Hypothesis tests / regression.
  16. ...and 9 more sections

Figures (5)

  • Figure 1: Selected image samples in our experiment.
  • Figure 2: Performance comparison results.
  • Figure 3: GEE estimated coefficients (log-odds) with 95% confidence intervals for model and prompt effects across multiple-choice datasets(GGR, VLAT, CALVI).
  • Figure 4: LMM fixed-effect coefficients with 95% confidence intervals for model and prompt effects across continuous-response datasets (HOLF and HOLF-Multi).
  • Figure :