Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA

TL;DR

This work lever-age Large Language Models, which have shown to have strong reasoning ability, as an automatic data anno-tator that generates question-answer annotations for chart images that significantly enhance the chart VQA models, achieving the state-of-the-art accuracy on the ChartQA and PlotQA datasets.

Abstract

Understanding data visualizations like charts and plots requires reasoning about both visual elements and numerics. Although strong in extractive questions, current chart visual question answering (chart VQA) models suffer on complex reasoning questions. In this work, we address the lack of reasoning ability by data augmentation. We leverage Large Language Models (LLMs), which have shown to have strong reasoning ability, as an automatic data annotator that generates question-answer annotations for chart images. The key innovation in our method lies in the Synthesize Step-by-Step strategy: our LLM-based data generator learns to decompose the complex question into step-by-step sub-questions (rationales), which are then used to derive the final answer using external tools, i.e. Python. This step-wise generation procedure is trained on synthetic data generated using a template-based QA generation pipeline. Experimental results highlight the significance of the proposed step-by-step generation. By training with the LLM-augmented data (LAMENDA), we significantly enhance the chart VQA models, achieving the state-of-the-art accuracy on the ChartQA and PlotQA datasets. In particular, our approach improves the accuracy of the previous state-of-the-art approach from 38% to 54% on the human-written questions in the ChartQA dataset, which needs strong reasoning. We hope our work underscores the potential of synthetic data and encourages further exploration of data augmentation using LLMs for reasoning-heavy tasks.

Figures (6)

• Figure 1: Existing chart VQA models struggle with complex reasoning questions. We attribute this to limited reasoning questions in existing datasets and address it by data augmentation. We fine-tune an LLM-based data generator that automatically generates question-answer annotations given a chart image. Our key innovation is Synthesize Step-by-Step, which breaks complex questions down into easy steps that could be solved using external tools. We use templates to train the LLM. Training with LLM-augmented data, LaMenDa, greatly enhances the chart VQA models.
• Figure 2: Method overview. (a) The architecture of the LLM-based question-answer generator. Image features, projected by a linear layer concatenated with the predicted data table and a prompt, are fed into LLM for QA generation. (b) Straight-forward generation, where questions with answers are generated in a straightforward way. (c) Synthesizing step-by-step, which breaks the question down into rationale programs and derives the answer by executing the program.
• Figure 3: Example of template-based question generation. Given a manually created template with defined program, by querying the image SVG annotations, templatic questions and answers can be generated, with rationales.
• Figure 4: Analysis of model predictions before and after training with LaMenDa. In I (left), ChartQA val questions are divided into four categories according to the model prediction correctness (before/after). II (right) further decomposes each category into different question types. LaMenDa improves the model prediction, and the improvement is most significant on reasoning questions.
• Figure 5: Examples of generated QA, 3 good examples and 1 failure case. Rationales are only shown for 2 examples for brevity.