SIMPLOT: Enhancing Chart Question Answering by Distilling Essentials

TL;DR

SIMPLOT tackles chart reasoning by distilling charts to essential information for reliable chart-to-table extraction, then leveraging a Large Multimodal Model with a chart-specific prompt for reasoning. It introduces a two-phase training scheme (teacher on simple charts; student maps original charts into that space) plus a preprocessing pipeline with row-column rendering and a Human-oriented chart instruction prompt. Empirical results on ChartQA and PlotQA show SIMPLOT surpasses prior methods in both extraction fidelity and QA accuracy, with ablations confirming the value of each component and prompt. The approach is model-agnostic and extends to related tasks like OpenCQA, demonstrating practical impact for robust, scalable chart reasoning; limitations include OOD generalization and reliance on prompt design, with code made publicly available.

Abstract

Recently, interpreting complex charts with logical reasoning has emerged as challenges due to the development of vision-language models. A prior state-of-the-art (SOTA) model has presented an end-to-end method that leverages the vision-language model to convert charts into table format utilizing Large Language Model (LLM) for reasoning. However, unlike natural images, charts contain a mix of essential and irrelevant information required for chart reasoning, and we discover that this characteristic can lower the performance of chart-to-table extraction. In this paper, we introduce SIMPLOT, a method designed to extract only the elements necessary for chart reasoning. The proposed method involves two steps: 1) training to mimic a simple plot that contains only the essential information from a complex chart for table extraction, followed by 2) performing reasoning based on the table. Our model enables accurate chart reasoning without the need for additional annotations or datasets, and its effectiveness is demonstrated through various experiments. Furthermore, we propose a novel prompt mimicking how human interpret charts for more accurate reasoning. Our source code is available at https://github.com/sangwu99/Simplot.

Figures (19)

• Figure 1: An example of chart-to-table extraction (from (a) to (b)) by Deplot liu2022deplot on ChartQA.
• Figure 2: Overall framework of Simplot. Upper box presents the preprocessing stage, which involves generating a simple positive sample containing only essential information from the original chart, as well as a negative sample, along with row and column rendering. Lower left box illustrates the training stage including two phases. In Phase 1 of the training stage, a teacher encoder and a table decoder are trained using a simple chart, and in Phase 2, a student encoder is trained with the original chart, while being distilled the knowledge of the teacher encoder on how to generate a table from a simple chart. Lower right box illustrates the inference stage, where an LMM receives the original chart and the extracted table along with prompts for reasoning.
• Figure 3: Accuracy of chart-to-table extraction using the original (in green) and simple charts (in red).
• Figure 4: A snippet of the proposed prompt.
• Figure 5: Improvements in performance observed when Simplot is applied across diverse models.