Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Tasks

TL;DR

This work addresses the reliability gap in Vision-Language Models by examining intrinsic self-correction during inference and proposing a learning-based path to improve performance through self-generated data.The authors introduce Self-Correction Learning (SCL), which uses Direct Preference Optimization (DPO) on a self-curated SelfCorSet, constructed from intrinsic self-correction outputs, to teach VLMs to generate correct responses directly.Across eight multimodal MCQ benchmarks, SCL demonstrates competitive gains over existing preference-tuning methods, with stronger results as training data increases, and reveals that purely iterative self-correction during inference is often unstable.The work highlights that self-correction should be coupled with learning to meaningfully enhance reasoning capabilities, and it provides a data-efficient framework (SelfCorSet + LoRA + DPO) for self-improvement without external feedback.

Abstract

While Vision-Language Models (VLMs) have shown remarkable abilities in visual and language reasoning tasks, they invariably generate flawed responses. Self-correction that instructs models to refine their outputs presents a promising solution to this issue. Previous studies have mainly concentrated on Large Language Models (LLMs), while the self-correction abilities of VLMs, particularly concerning both visual and linguistic information, remain largely unexamined. This study investigates the self-correction capabilities of VLMs during both inference and fine-tuning stages. We introduce a Self-Correction Learning (SCL) approach that enables VLMs to learn from their self-generated self-correction data through Direct Preference Optimization (DPO) without relying on external feedback, facilitating self-improvement. Specifically, we collect preferred and disfavored samples based on the correctness of initial and refined responses, which are obtained by two-turn self-correction with VLMs during the inference stage. Experimental results demonstrate that although VLMs struggle to self-correct effectively during iterative inference without additional fine-tuning and external feedback, they can enhance their performance and avoid previous mistakes through preference fine-tuning when their self-generated self-correction data are categorized into preferred and disfavored samples. This study emphasizes that self-correction is not merely a refinement process; rather, it should enhance the reasoning abilities of models through additional training, enabling them to generate high-quality responses directly without further refinement.

Figures (6)

• Figure 1: Comparison of inference-based and training-based self-correction. Inference-based methods aim to refine an initial response over $K$ iterations, keeping model parameters fixed. Training-based methods focus on training the model to directly generate high-quality initial responses without iterative refinement.
• Figure 2: SCL begins with intrinsic self-correction applied to the VLM, generating four types of self-correction samples. Correct responses from Type 2 and incorrect responses from Type 3 samples are designated as preferences and disfavors, respectively, to construct the SelfCorSet preference dataset. The VLM then undergoes DPO on SelfCorSet for self-improvement.
• Figure 3: Two examples of intrinsic self-correction processes generated by InternLM-XComposer-2-7B.
• Figure 4: Distribution of self-correction examples of MiniCPM-Llama3-V2.5 and InternLM-XComposer-2-7B under VP-1 on ScienceQA.
• Figure 5: LLaVA-V1.5-7B successfully answers the question after learning from its self-correction samples.