Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning

Xiaotian Han, Yiren Jian, Xuefeng Hu, Haogeng Liu, Yiqi Wang, Qihang Fan, Yuang Ai, Huaibo Huang, Ran He, Zhenheng Yang, Quanzeng You

TL;DR

The paper tackles the lack of open, large-scale multimodal datasets for mathematical reasoning by constructing InfiMM-WebMath-40B from CommonCrawl, featuring interleaved text and image data with rigorous filtering and deduplication. It details a three-stage training pipeline—modality alignment, continue pre-training, and instruction fine-tuning—using a SigLip-based visual encoder and Perceiver Resampler to train InfiMM-Math on DeepSeek-Coder backbones. Empirical results show strong text-only gains at 40B tokens, competitive performance on GSM8K/MMLU, and state-of-the-art multimodal results on MathVerse and We-Math for open models. The work provides a publicly available dataset and demonstrates open-source potential for advanced multimodal mathematical reasoning, with future directions including improved math-focused vision encoders and reinforcement-learning-based reasoning enhancements.

Abstract

Pre-training on large-scale, high-quality datasets is crucial for enhancing the reasoning capabilities of Large Language Models (LLMs), especially in specialized domains such as mathematics. Despite the recognized importance, the Multimodal LLMs (MLLMs) field currently lacks a comprehensive open-source pre-training dataset specifically designed for mathematical reasoning. To address this gap, we introduce InfiMM-WebMath-40B, a high-quality dataset of interleaved image-text documents. It comprises 24 million web pages, 85 million associated image URLs, and 40 billion text tokens, all meticulously extracted and filtered from CommonCrawl. We provide a detailed overview of our data collection and processing pipeline. To demonstrate the robustness of InfiMM-WebMath-40B, we conducted evaluations in both text-only and multimodal settings. Our evaluations on text-only benchmarks show that, despite utilizing only 40 billion tokens, our dataset significantly enhances the performance of our 1.3B model, delivering results comparable to DeepSeekMath-1.3B, which uses 120 billion tokens for the same model size. Nevertheless, with the introduction of our multi-modal math pre-training dataset, our models set a new state-of-the-art among open-source models on multi-modal math benchmarks such as MathVerse and We-Math. We release our data at https://huggingface.co/datasets/Infi-MM/InfiMM-WebMath-40B.

InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning

TL;DR

The paper tackles the lack of open, large-scale multimodal datasets for mathematical reasoning by constructing InfiMM-WebMath-40B from CommonCrawl, featuring interleaved text and image data with rigorous filtering and deduplication. It details a three-stage training pipeline—modality alignment, continue pre-training, and instruction fine-tuning—using a SigLip-based visual encoder and Perceiver Resampler to train InfiMM-Math on DeepSeek-Coder backbones. Empirical results show strong text-only gains at 40B tokens, competitive performance on GSM8K/MMLU, and state-of-the-art multimodal results on MathVerse and We-Math for open models. The work provides a publicly available dataset and demonstrates open-source potential for advanced multimodal mathematical reasoning, with future directions including improved math-focused vision encoders and reinforcement-learning-based reasoning enhancements.

Abstract

Pre-training on large-scale, high-quality datasets is crucial for enhancing the reasoning capabilities of Large Language Models (LLMs), especially in specialized domains such as mathematics. Despite the recognized importance, the Multimodal LLMs (MLLMs) field currently lacks a comprehensive open-source pre-training dataset specifically designed for mathematical reasoning. To address this gap, we introduce InfiMM-WebMath-40B, a high-quality dataset of interleaved image-text documents. It comprises 24 million web pages, 85 million associated image URLs, and 40 billion text tokens, all meticulously extracted and filtered from CommonCrawl. We provide a detailed overview of our data collection and processing pipeline. To demonstrate the robustness of InfiMM-WebMath-40B, we conducted evaluations in both text-only and multimodal settings. Our evaluations on text-only benchmarks show that, despite utilizing only 40 billion tokens, our dataset significantly enhances the performance of our 1.3B model, delivering results comparable to DeepSeekMath-1.3B, which uses 120 billion tokens for the same model size. Nevertheless, with the introduction of our multi-modal math pre-training dataset, our models set a new state-of-the-art among open-source models on multi-modal math benchmarks such as MathVerse and We-Math. We release our data at https://huggingface.co/datasets/Infi-MM/InfiMM-WebMath-40B.
Paper Structure (27 sections, 2 figures, 7 tables)

This paper contains 27 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM for Math
  4. MLLM for Math
  5. Dataset Construction
  6. Text-only Data Curation Pipeline
  7. Text Extraction and Language Filtering
  8. Mathematical Content Extraction
  9. High-Recall Filtering for Mathematical Content
  10. Deduplication
  11. Rule-based Filtering
  12. High-Precision Filtering for Mathematical Content
  13. Evaluation on Text-Only Filtering
  14. Multimodal Data Construction
  15. Experiments
  16. ...and 12 more sections

Figures (2)

  • Figure 1: InfiMM-WebMath-40B data curation pipeline.
  • Figure 2: A comparative illustration of extraction results from a Wikipedia webpage using Trafilatura and our enhanced version of Resiliparse, highlighting the successful retrieval of mathematical equations and image URLs.