ChartKG: A Knowledge-Graph-Based Representation for Chart Images

TL;DR

This paper proposes ChartKG, a novel knowledge graph (KG) based representation for chart images, which can model the visual elements in a chart image and semantic relations among them including visual encodings and visual insights in a unified manner.

Abstract

Chart images, such as bar charts, pie charts, and line charts, are explosively produced due to the wide usage of data visualizations. Accordingly, knowledge mining from chart images is becoming increasingly important, which can benefit downstream tasks like chart retrieval and knowledge graph completion. However, existing methods for chart knowledge mining mainly focus on converting chart images into raw data and often ignore their visual encodings and semantic meanings, which can result in information loss for many downstream tasks. In this paper, we propose ChartKG, a novel knowledge graph (KG) based representation for chart images, which can model the visual elements in a chart image and semantic relations among them including visual encodings and visual insights in a unified manner. Further, we develop a general framework to convert chart images to the proposed KG-based representation. It integrates a series of image processing techniques to identify visual elements and relations, e.g., CNNs to classify charts, yolov5 and optical character recognition to parse charts, and rule-based methods to construct graphs. We present four cases to illustrate how our knowledge-graph-based representation can model the detailed visual elements and semantic relations in charts, and further demonstrate how our approach can benefit downstream applications such as semantic-aware chart retrieval and chart question answering. We also conduct quantitative evaluations to assess the two fundamental building blocks of our chart-to-KG framework, i.e., object recognition and optical character recognition. The results provide support for the usefulness and effectiveness of ChartKG.

Figures (6)

• Figure 1: An overview of the framework to convert chart images to the proposed knowledge graph representations. We first use CNNs to detect input images for chart classification. Then, object recognition and Optical Character Recognition (OCR) are introduced to parsing the charts. A rule-based method is developed to construct the final knowledge graphs for chart images.
• Figure 2: An overview of the relationship types and their corresponding specific relationships. The VE, VEPV, DV, DVV and VI respectively denote visual element, visual element property value, data variable, data variable value and visual insight.
• Figure 3: Examples of knowledge graph representation for four types of charts. (a) Bar Chart; (b) Line Chart; (c) Pie Chart; (d) Scatter Plot.
• Figure 4: Example results of ChartKG-powered chart retrieval. (a) illustrates the outcomes of data variable retrieval with the keywords that meet the query requirements highlighted. (b) depicts the results of combining data variables and visual insights retrieval with the associated entities and relationships displayed below the chart.
• Figure 5: This figure illustrates the results of question answering for specific charts based on three question templates proposed in our work.