MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoning

TL;DR

MathCoder-VL introduces a dedicated image-to-code data engine (FigCodifier) and a large ImgCode-8.6M dataset to enforce precise cross-modal alignment between mathematical visuals and their renderable code. A two-stage training pipeline—image-to-code mid-training followed by multimodal math instruction fine-tuning on MM-MathInstruct-3M—yields open-source state-of-the-art performance across multiple math benchmarks, with particular strength in geometry problem solving. The approach demonstrates that leveraging code-generated images and synthetic multimodal data substantially enhances multimodal mathematical reasoning and offers scalable data-generation workflows for domain-specific LMMs. The work provides a practical, open-source pathway for advancing multimodal math reasoning and highlights effective strategies for data curation, synthesis, and fine-tuning specialized for mathematical tasks.

Abstract

Natural language image-caption datasets, widely used for training Large Multimodal Models, mainly focus on natural scenarios and overlook the intricate details of mathematical figures that are critical for problem-solving, hindering the advancement of current LMMs in multimodal mathematical reasoning. To this end, we propose leveraging code as supervision for cross-modal alignment, since code inherently encodes all information needed to generate corresponding figures, establishing a precise connection between the two modalities. Specifically, we co-develop our image-to-code model and dataset with model-in-the-loop approach, resulting in an image-to-code model, FigCodifier and ImgCode-8.6M dataset, the largest image-code dataset to date. Furthermore, we utilize FigCodifier to synthesize novel mathematical figures and then construct MM-MathInstruct-3M, a high-quality multimodal math instruction fine-tuning dataset. Finally, we present MathCoder-VL, trained with ImgCode-8.6M for cross-modal alignment and subsequently fine-tuned on MM-MathInstruct-3M for multimodal math problem solving. Our model achieves a new open-source SOTA across all six metrics. Notably, it surpasses GPT-4o and Claude 3.5 Sonnet in the geometry problem-solving subset of MathVista, achieving improvements of 8.9% and 9.2%. The dataset and models will be released at https://github.com/mathllm/MathCoder.

Figures (15)

• Figure 1: (a) Natural language captions often struggle to convey all details in a image and guarantee correctness. (b) Our approach uses image-translated $\mathbf{Code}$ and code-generated $\mathbf{Image^C}$ to create $\mathbf{\langle Image^C, Code \rangle}$ pairs. Since the $\mathbf{Image^{C}}$ is rendered from the $\mathbf{Code}$, the cross-modal alignment is always accurate and contains all the details. Below are four examples of new figures synthesized based on $\mathbf{Image^{Raw}}$.
• Figure 2: (a) The iterative training pipeline of our image-to-code model. We use DaTikZ-119K as seed data to train our first image-to-code model. We start by collecting 3 million math-related images and ultimately synthesize 8.6 million image-code pairs. Our final image-to-code model, FigCodifier, is based on InternVL2-8B chen2024internvl, with all model parameters being fully learnable. (b) The pipeline for generating new math problems with diverse new images. Using the final model from (a), we convert raw images into code and leverage Qwen models to generate new questions and step-by-step solutions based on the newly synthesized images.
• Figure 3: Sample questions paired with newly synthesized images, as generated in Figure \ref{['fig:data_pipeline']} (b).
• Figure 4: Two training stages of MathCoder-VL.
• Figure 5: The pipeline for processing the K12 math problem-solving dataset.