Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Training

Mengzhao Jia, Zhihan Zhang, Wenhao Yu, Fangkai Jiao, Meng Jiang

TL;DR

Open-source MLLMs struggle with multimodal mathematical reasoning due to weak visual comprehension. VCAR introduces a two-step training pipeline that first strengthens visual understanding through image description generation and then enhances reasoning with description-assisted rationales, using separate LoRA adapters. The method achieves significant improvements on MathVista and MathVerse, especially for visually demanding problems, and gains are observed across different base models. Ablation studies confirm the necessity of both components and the proposed training order, underscoring the importance of visual-centric supervision for multimodal math.

Abstract

Open-source multimodal large language models (MLLMs) excel in various tasks involving textual and visual inputs but still struggle with complex multimodal mathematical reasoning, lagging behind proprietary models like GPT-4V(ision) and Gemini-Pro. Although fine-tuning with intermediate steps (i.e., rationales) elicits some mathematical reasoning skills, the resulting models still fall short in visual comprehension due to inadequate visual-centric supervision, which leads to inaccurate interpretation of math figures. To address this issue, we propose a two-step training pipeline VCAR, which emphasizes the Visual Comprehension training in Addition to mathematical Reasoning learning. It first improves the visual comprehension ability of MLLMs through the visual description generation task, followed by another training step on generating rationales with the assistance of descriptions. Experimental results on two popular benchmarks demonstrate that VCAR substantially outperforms baseline methods solely relying on rationale supervision, especially on problems with high visual demands.

Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Training

TL;DR

Open-source MLLMs struggle with multimodal mathematical reasoning due to weak visual comprehension. VCAR introduces a two-step training pipeline that first strengthens visual understanding through image description generation and then enhances reasoning with description-assisted rationales, using separate LoRA adapters. The method achieves significant improvements on MathVista and MathVerse, especially for visually demanding problems, and gains are observed across different base models. Ablation studies confirm the necessity of both components and the proposed training order, underscoring the importance of visual-centric supervision for multimodal math.

Abstract

Open-source multimodal large language models (MLLMs) excel in various tasks involving textual and visual inputs but still struggle with complex multimodal mathematical reasoning, lagging behind proprietary models like GPT-4V(ision) and Gemini-Pro. Although fine-tuning with intermediate steps (i.e., rationales) elicits some mathematical reasoning skills, the resulting models still fall short in visual comprehension due to inadequate visual-centric supervision, which leads to inaccurate interpretation of math figures. To address this issue, we propose a two-step training pipeline VCAR, which emphasizes the Visual Comprehension training in Addition to mathematical Reasoning learning. It first improves the visual comprehension ability of MLLMs through the visual description generation task, followed by another training step on generating rationales with the assistance of descriptions. Experimental results on two popular benchmarks demonstrate that VCAR substantially outperforms baseline methods solely relying on rationale supervision, especially on problems with high visual demands.
Paper Structure (24 sections, 3 equations, 9 figures, 5 tables)

This paper contains 24 sections, 3 equations, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Models
  4. Multimodal Reasoning
  5. Method
  6. Problem Formulation
  7. Supervision Collection
  8. Two-step Training Pipeline
  9. Experiments
  10. Setup
  11. Main Results
  12. Ablation Study
  13. Case Study
  14. Analysis
  15. Using GPT-4V to Collect Data
  16. ...and 9 more sections

Figures (9)

  • Figure 1: VCAR trains the model with an additional visual comprehension task in addition to the mathematical reasoning training, avoiding errors caused by inaccurate visual understanding. In contrast, the baseline method solely trained on rationales fails to correctly answer the question. It demonstrates the need for specified visual comprehension training.
  • Figure 2: Illustration of the proposed method. VCAR consists of two components, namely supervision collection and two-step training. The inference pipeline of VCAR is also demonstrated.
  • Figure 3: Inference results of VCAR and CoT-GT baseline on an example from MathVista. Correct and incorrect image descriptive expressions are highlighted in green and red, respectively.
  • Figure 4: Performance comparisons of various variants on two benchmarks. We present the averaged accuracy. Detailed category-specific results are at Appendix \ref{['apdx:ana']}.
  • Figure :
  • ...and 4 more figures