Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision-Language Models Can Self-Improve Reasoning via Reflection

Kanzhi Cheng, Yantao Li, Fangzhi Xu, Jianbing Zhang, Hao Zhou, Yang Liu

TL;DR

This work tackles the scarcity and noisiness of multimodal chain-of-thought data by introducing R3V, a self-training framework that bootstraps positive and negative CoT rationales from vision-language data and learns from mistakes through reflection. It adds self-refine and self-select losses and a test-time selection mechanism to iteratively improve Vision-Language Reasoning without manual CoT annotations. Across six diverse benchmarks, R3V yields substantial relative gains over GPT-distilled baselines and outperforms prior self-training methods, with robust gains even in out-of-distribution settings due to test-time reasoning. The findings demonstrate that learning from self-generated CoT and performing reflection at inference can significantly boost multimodal reasoning, offering a scalable path to stronger open-source MLLMs.

Abstract

Chain-of-thought (CoT) has proven to improve the reasoning capability of large language models (LLMs). However, due to the complexity of multimodal scenarios and the difficulty in collecting high-quality CoT data, CoT reasoning in multimodal LLMs has been largely overlooked. To this end, we propose a simple yet effective self-training framework, R3V, which iteratively enhances the model's Vision-language Reasoning by Reflecting on CoT Rationales. Our framework consists of two interleaved parts: (1) iteratively bootstrapping positive and negative solutions for reasoning datasets, and (2) reflection on rationale for learning from mistakes. Specifically, we introduce the self-refine and self-select losses, enabling the model to refine flawed rationale and derive the correct answer by comparing rationale candidates. Experiments on a wide range of vision-language tasks show that R3V consistently improves multimodal LLM reasoning, achieving a relative improvement of 23 to 60 percent over GPT-distilled baselines. Additionally, our approach supports self-reflection on generated solutions, further boosting performance through test-time computation.

Vision-Language Models Can Self-Improve Reasoning via Reflection

TL;DR

This work tackles the scarcity and noisiness of multimodal chain-of-thought data by introducing R3V, a self-training framework that bootstraps positive and negative CoT rationales from vision-language data and learns from mistakes through reflection. It adds self-refine and self-select losses and a test-time selection mechanism to iteratively improve Vision-Language Reasoning without manual CoT annotations. Across six diverse benchmarks, R3V yields substantial relative gains over GPT-distilled baselines and outperforms prior self-training methods, with robust gains even in out-of-distribution settings due to test-time reasoning. The findings demonstrate that learning from self-generated CoT and performing reflection at inference can significantly boost multimodal reasoning, offering a scalable path to stronger open-source MLLMs.

Abstract

Chain-of-thought (CoT) has proven to improve the reasoning capability of large language models (LLMs). However, due to the complexity of multimodal scenarios and the difficulty in collecting high-quality CoT data, CoT reasoning in multimodal LLMs has been largely overlooked. To this end, we propose a simple yet effective self-training framework, R3V, which iteratively enhances the model's Vision-language Reasoning by Reflecting on CoT Rationales. Our framework consists of two interleaved parts: (1) iteratively bootstrapping positive and negative solutions for reasoning datasets, and (2) reflection on rationale for learning from mistakes. Specifically, we introduce the self-refine and self-select losses, enabling the model to refine flawed rationale and derive the correct answer by comparing rationale candidates. Experiments on a wide range of vision-language tasks show that R3V consistently improves multimodal LLM reasoning, achieving a relative improvement of 23 to 60 percent over GPT-distilled baselines. Additionally, our approach supports self-reflection on generated solutions, further boosting performance through test-time computation.

Paper Structure

This paper contains 35 sections, 9 equations, 12 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminaries
  5. r3v: Reflection on Rationales
  6. Self-Refine
  7. Self-Select
  8. Test-Time Selection
  9. Experiments
  10. Datasets
  11. Experimental Setings
  12. Main Results
  13. Self-training effectively converts MLLMs from weak to strong.
  14. R3V further enhances self-training efficiency by learning from mistakes.
  15. Out-of-Distribution Evaluation
  16. ...and 20 more sections

Figures (12)

  • Figure 1: Results of Qwen-VL on TabMWP, a visual mathematical reasoning dataset. Qwen-VL exhibits weak zero-shot CoT reasoning performance, while our R3V iteratively self-improves, surpassing the GPT-distilled baseline by a large margin.
  • Figure 2: Overview of our multimodal self-training framework of R3V. It boosts vision-language reasoning by iteratively reflecting on bootstrapped CoT rationales and enables self-reflection through test-time computing.
  • Figure 3: Comparison of the iterative self-training process between R3V and STaR on Qwen-VL across four benchmarks. Full results are provided in \ref{['app:evo']}. R3V demonstrates higher efficiency in evolution and superior final performance.
  • Figure 4: Performance comparison of different test-time methods. Our test-time selection is robust and effective, consistently outperforming Test@1 and majority voting.
  • Figure 5: Comparison of scalability between test-time selection and majority voting.
  • ...and 7 more figures