Advancing Chart Question Answering with Robust Chart Component Recognition

TL;DR

This work addresses ChartQA by introducing ChartFormer, a unified end-to-end model for accurate chart component recognition across diverse chart types, and QDChart, a multimodal QA system thatGrounds question-relevant chart features through a novel Question-guided Deformable Co-Attention (QDCAt) fusion mechanism. ChartFormer achieves state-of-the-art performance in chart component recognition, while QDChart delivers substantial improvements on ChartQA by grounding visual chart information to questions via a question-conditioned sampling and deformable attention pipeline. The authors validate their approach on an automatically annotated ExcelChart400K dataset for recognition and the ChartQA dataset for QA, reporting improvements of $3.2\%$ in $mAP$ and $15.4\%$ in accuracy, respectively, demonstrating robust performance in detailed visual data interpretation. The work highlights the value of end-to-end chart understanding and question-guided multimodal fusion for practical applications across data visualization tasks.

Abstract

Chart comprehension presents significant challenges for machine learning models due to the diverse and intricate shapes of charts. Existing multimodal methods often overlook these visual features or fail to integrate them effectively for chart question answering (ChartQA). To address this, we introduce Chartformer, a unified framework that enhances chart component recognition by accurately identifying and classifying components such as bars, lines, pies, titles, legends, and axes. Additionally, we propose a novel Question-guided Deformable Co-Attention (QDCAt) mechanism, which fuses chart features encoded by Chartformer with the given question, leveraging the question's guidance to ground the correct answer. Extensive experiments demonstrate that the proposed approaches significantly outperform baseline models in chart component recognition and ChartQA tasks, achieving improvements of 3.2% in mAP and 15.4% in accuracy, respectively. These results underscore the robustness of our solution for detailed visual data interpretation across various applications.

Figures (10)

• Figure 1: An example of Chart Question Answering.
• Figure 2: Model architecture of ChartFormer for chart component recognition
• Figure 3: The QDChart model structure
• Figure 4: Question-guided offset network
• Figure 5: Qualitative examples on question-guided deformed points