ChatBCG: Can AI Read Your Slide Deck?
Nikita Singh, Rob Balian, Lukas Martinelli
TL;DR
This paper investigates the end-to-end ability of multimodal vision-enabled models GPT-4o and Gemini Flash-1.5 to read and extract data from slide-deck charts, separating labeled charts (with explicit data points) from unlabeled charts (requiring axis-based inference). Using a dataset of roughly 31 charts and three reading-focused question types, the authors measure match rates for labeled data and MAE/MAPE for unlabeled estimations. Results show high but imperfect performance on labeled charts (84% for GPT-4o and 86% for Gemini) and substantial estimation error on unlabeled charts (MAPE around 55% for both), with only about 7–8 charts read perfectly end-to-end in a typical deck of ~30 charts. The findings suggest that despite advanced vision capabilities, these models are not yet reliable for high-precision, business-deck analysis without human oversight, a conclusion supported by detailed appendix data of model responses.
Abstract
Multimodal models like GPT4o and Gemini Flash are exceptional at inference and summarization tasks, which approach human-level in performance. However, we find that these models underperform compared to humans when asked to do very specific 'reading and estimation' tasks, particularly in the context of visual charts in business decks. This paper evaluates the accuracy of GPT 4o and Gemini Flash-1.5 in answering straightforward questions about data on labeled charts (where data is clearly annotated on the graphs), and unlabeled charts (where data is not clearly annotated and has to be inferred from the X and Y axis). We conclude that these models aren't currently capable of reading a deck accurately end-to-end if it contains any complex or unlabeled charts. Even if a user created a deck of only labeled charts, the model would only be able to read 7-8 out of 15 labeled charts perfectly end-to-end. For full list of slide deck figures visit https://www.repromptai.com/chat_bcg