ChartEdit: How Far Are MLLMs From Automating Chart Analysis? Evaluating MLLMs' Capability via Chart Editing

TL;DR

ChartEdit presents the first high-quality, diverse benchmark for chart editing, pairing 233 real-world charts with 1,405 editing instructions and reference edited code. It evaluates 10 multimodal LLMs across code- and chart-level tasks, revealing that while large models can generate edits that partially reproduce references, precise, instruction-following edits remain challenging. The study shows a substantial performance gap between proprietary models (notably GPT-4o) and open-source or chart-domain models, especially for complex edits without code input. By providing a public dataset and a rigorous evaluation framework, ChartEdit aims to drive further advances in automated chart editing for academic and data-driven workflows.

Abstract

Although multimodal large language models (MLLMs) show promise in generating chart rendering code, editing charts via code presents a greater challenge. This task demands MLLMs to integrate chart understanding and reasoning capacities, which are labor-intensive. While many MLLMs claim such editing capabilities, current evaluations rely on limited case studies, highlighting the urgent need for a comprehensive evaluation framework. In this work, we propose \textsc{ChartEdit}, a novel benchmark designed for chart editing tasks, featuring $1405$ diverse editing instructions applied to $233$ real-world charts, each manually annotated and validated for accuracy. Utilizing \textsc{ChartEdit}, we evaluate the performance of 10 mainstream MLLMs across two types of experiments at both the code and chart levels. The results suggest that large-scale models can generate code to produce images that partially match the reference images. However, their ability to generate accurate edits according to the instructions remains limited. The state-of-the-art (SOTA) model achieves a score of only $59.96$, highlighting significant challenges in precise modification. In contrast, small-scale models, including chart-domain models, struggle both with following editing instructions and generating overall chart images, underscoring the need for further development in this area. Code is available at https://github.com/xxlllz/ChartEdit.

Figures (13)

• Figure 1: Overall pipeline. The inputs are a chart, editing instruction w/ or w/o code. The MLLMs are instructed to generate the edited code. The final evaluation is constructed at both the code level and chart level.
• Figure 2: The pipeline for constructing the ChartEdit evaluation dataset begins with filtering and crawling ArXiv papers based on keywords found in their comments. After that, we remove irrelevant files by filtering out specific suffixes. We then use an MLLM to distinguish and filter out non-chart images, scoring the remaining images. These images are further screened based on these scores and reviewed by human evaluators. Also, Code annotations are manually written by human evaluators. The editing instructions and reference edited code are constructed utilizing two strategies: one based on LLM and the other manually written. Finally, all the <Chart, Instruction, Code> triplets are reviewed and modified by human evaluators to enhance correspondence and accuracy.
• Figure 3: The number and specific proportions of different types of editing instructions in our ChartEdit evaluation benchmark.
• Figure 4: The dimension reduction of the editing instructions in ChartEdit with various colors represents different editing types. We choose the Sentence-BERT reimers2019sentence as the embedding model.
• Figure 5: The result comparisons between GPT-4o and InternVL2.5-78B in direct and zero-shot Cot prompting setting. In most cases, the effect of CoT prompting shows a negligible improvement over direct prompting.