ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarization

TL;DR

ChartThinker tackles chart summarization by integrating a context-enhanced chain-of-thought (CoT) reasoning paradigm with retrieval-augmented context. It introduces Chart-Sum-QA, a large-scale dataset of chart-caption pairs and instruction-question data, and a ChartThinker architecture featuring a CLIP-based image encoder, a chart parsing module, a retrieval library, and an Idefics-based CoT generator. Through extensive automatic and human evaluations, ChartThinker outperforms eight state-of-the-art baselines across seven metrics, demonstrating improved factual matching and reasoning. The work provides public data and code, and establishes a practical framework for robust, interpretable chart narration that can benefit visualization analysis and downstream decision-making.

Abstract

Data visualization serves as a critical means for presenting data and mining its valuable insights. The task of chart summarization, through natural language processing techniques, facilitates in-depth data analysis of charts. However, there still are notable deficiencies in terms of visual-language matching and reasoning ability for existing approaches. To address these limitations, this study constructs a large-scale dataset of comprehensive chart-caption pairs and fine-tuning instructions on each chart. Thanks to the broad coverage of various topics and visual styles within this dataset, better matching degree can be achieved from the view of training data. Moreover, we propose an innovative chart summarization method, ChartThinker, which synthesizes deep analysis based on chains of thought and strategies of context retrieval, aiming to improve the logical coherence and accuracy of the generated summaries. Built upon the curated datasets, our trained model consistently exhibits superior performance in chart summarization tasks, surpassing 8 state-of-the-art models over 7 evaluation metrics. Our dataset and codes are publicly accessible.

Figures (20)

• Figure 1: Comparison with large visual-language models in chart summarization (LLaMA-Adapter-v2 gao2023llama, MiniGPT-4 zhu2023minigpt). There are two types of errors that occur during the generation process: Insufficient matching degree (inconsistency between the generated results and the chart content, such as content omission or fabricated content), and reasoning errors (inconsistency between the inferred meaning and the intended message of the chart).
• Figure 2: Overview of ChartThinker. The encoded input chart and prompt are simultaneously fed into the Context-Enhanced CoT Generator. This module generates thought chains, and for each thought generated, the model retrieves the top-k image-text pairs from the chart library that best align with the thought, serving as contextual learning examples. Subsequently, the corresponding output for each thought is generated. Finally, all the outputs are consolidated to derive the final chart description.
• Figure 3: The workflow of the chart analysis module. The input is the chart, the output of OCR is all of the textual and numerical information, and the output of Deplot is a table containing text and corresponding numerical data. The final integrated output is divided into two parts: text-number pair and other text.
• Figure 4: The CoT Generation Process: For a given chart, the Context-Enhanced generator produces thoughts at each step. These thoughts help the model determine proper actions and generate conclusive statements. Finally, the conclusions from each step are integrated to yield the output answer.
• Figure 5: This is a bar chart. The chart shows the number of foreign students newly enrolled in associate, bachelor's, master's, or doctorate degree programs in different states in the United States. California has the highest number of foreign students enrolled in a degree program, with 59,801 students enrolled. New York follows with 41,040 students, while Texas has 26,701 students. Massachusetts has 19,901 students, Pennsylvania has 17,801 students, Illinois has 16,801 students, Florida has 15,701 students, Ohio has 10,801 students, Michigan has 9,801 students, and Washington has 9,801 students.