Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Tasks

João Palmeiro, Diogo Duarte, Rita Costa, Pedro Bizarro

TL;DR

BIY addresses the lack of scatterplot-specific benchmarks by releasing a large synthetic dataset and a task-based evaluation framework for clustering and outlier detection in scatterplots. It demonstrates that current LLMs can count clusters and outliers with high accuracy using few-shot prompting, but localization tasks are still unreliable. Chart design has a modest impact on performance, suggesting focusing on prompting strategies and model choice. The work lays the groundwork for scalable benchmarking and outlines plans to extend data generators, designs, prompting strategies, and open-model evaluations, with future goals including chart captioning for accessible alt-text.

Abstract

AI models are increasingly used for data analysis and visualization, yet benchmarks rarely address scatterplot-specific tasks, limiting insight into performance. To address this gap for one of the most common chart types, we introduce a synthetic, annotated dataset of over 18,000 scatterplots from six data generators and 17 chart designs, and a benchmark based on it. We evaluate proprietary models from OpenAI and Google using N-shot prompting on five distinct tasks derived from annotations of cluster bounding boxes, their center coordinates, and outlier coordinates. OpenAI models and Gemini 2.5 Flash, especially when prompted with examples, are viable options for counting clusters and, in Flash's case, outliers (90%+ Accuracy). However, the results for localization-related tasks are unsatisfactory: Precision and Recall are near or below 50%, except for Flash in outlier identification (65.01%). Furthermore, the impact of chart design on performance appears to be a secondary factor, but it is advisable to avoid scatterplots with wide aspect ratios (16:9 and 21:9) or those colored randomly. Supplementary materials are available at https://github.com/feedzai/biy-paper.

Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Tasks

TL;DR

BIY addresses the lack of scatterplot-specific benchmarks by releasing a large synthetic dataset and a task-based evaluation framework for clustering and outlier detection in scatterplots. It demonstrates that current LLMs can count clusters and outliers with high accuracy using few-shot prompting, but localization tasks are still unreliable. Chart design has a modest impact on performance, suggesting focusing on prompting strategies and model choice. The work lays the groundwork for scalable benchmarking and outlines plans to extend data generators, designs, prompting strategies, and open-model evaluations, with future goals including chart captioning for accessible alt-text.

Abstract

AI models are increasingly used for data analysis and visualization, yet benchmarks rarely address scatterplot-specific tasks, limiting insight into performance. To address this gap for one of the most common chart types, we introduce a synthetic, annotated dataset of over 18,000 scatterplots from six data generators and 17 chart designs, and a benchmark based on it. We evaluate proprietary models from OpenAI and Google using N-shot prompting on five distinct tasks derived from annotations of cluster bounding boxes, their center coordinates, and outlier coordinates. OpenAI models and Gemini 2.5 Flash, especially when prompted with examples, are viable options for counting clusters and, in Flash's case, outliers (90%+ Accuracy). However, the results for localization-related tasks are unsatisfactory: Precision and Recall are near or below 50%, except for Flash in outlier identification (65.01%). Furthermore, the impact of chart design on performance appears to be a secondary factor, but it is advisable to avoid scatterplots with wide aspect ratios (16:9 and 21:9) or those colored randomly. Supplementary materials are available at https://github.com/feedzai/biy-paper.

Paper Structure

This paper contains 13 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Datasets
  4. Benchmarks and Experiments
  5. Benchmarking Scatterplot-Related Tasks
  6. Dataset
  7. Benchmark
  8. Results
  9. Counting
  10. Detection and Identification
  11. Chart Design Performance
  12. Considerations
  13. Conclusion and Future Work

Figures (2)

  • Figure 1: Accuracy for each model and prompting strategy. The results for cluster counting [height=2ex]figures/cluster_counting.svg (top) using few-shot prompting are particularly promising for several models. On the other hand, Flash excels at the outlier counting [height=2ex]figures/outlier_counting.svg task (bottom) when one-shot (86.67%) and few-shot (90.49%) prompted.
  • Figure 2: Recall for each model and prompting strategy. None surpass 25% Recall in the cluster detection [height=2ex]figures/cluster_detection.svg (top) and identification [height=2ex]figures/cluster_identification.svg (middle) tasks. Recall is also low for the outlier identification [height=2ex]figures/outlier_identification.svg task (bottom), although Flash, when few-shot prompted, seems promising (65.01%). Lowering the IoU to 0.5 considerably changes the results for some models, but Recall remains low (o3, when few-shot prompted, reaches a Recall of 50.26%). Doubling the distance threshold to 20px does not change the conclusions.