Improved Iterative Refinement for Chart-to-Code Generation via Structured Instruction

Abstract

Recently, multimodal large language models (MLLMs) have attracted increasing research attention due to their powerful visual understanding capabilities. While they have achieved impressive results on various vision tasks, their performance on chart-to-code generation remains suboptimal. This task requires MLLMs to generate executable code that can reproduce a given chart, demanding not only precise visual understanding but also accurate translation of visual elements into structured code. Directly prompting MLLMs to perform this complex task often yields unsatisfactory results. To address this challenge, we propose {ChartIR}, an iterative refinement method based on structured instruction. First, we distinguish two tasks: visual understanding and code translation. To accomplish the visual understanding component, we design two types of structured instructions: description and difference. The description instruction captures the visual elements of the reference chart, while the difference instruction characterizes the discrepancies between the reference chart and the generated chart. These instructions effectively transform visual features into language representations, thereby facilitating the subsequent code translation process. Second, we decompose the overall chart generation pipeline into two stages: initial code generation and iterative refinement, enabling progressive enhancement of the final output. Experimental results show that, compared to other method, our method achieves superior performance on both the open-source model Qwen2-VL and the closed-source model GPT-4o.

Figures (6)

• Figure 1: Overview of the ChartIR. The method consists of two steps: 1) Initial Code Generation: First, a description is generated based on the reference chart. Then, using both the description and the reference chart, an initial code is produced. 2) Iterative Refinement: In this stage, first comparing the chart generated from the initial code with the reference chart and produces a description of the difference. Based on this difference, the reference chart, and the description of the reference chart, a new code is generated and then get a new chart. This process is repeated iteratively. The iteration continues until convergence is achieved.
• Figure 2: A case study based on ChartIR.The initial chart had missing textual information. The first refinement recovered the text but caused a color error, while the second refinement fixed the color error without new issues, resulting in a chart with better overall performance than the initial version.
• Figure 3: Visual comparison of chart generation results across different methods (Example 1).
• Figure 4: Visual comparison of chart generation results across different methods (Example 2).
• Figure 5: Visual comparison of chart generation results across different methods (Example 3).