Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ARES: Alternating Reinforcement Learning and Supervised Fine-Tuning for Enhanced Multi-Modal Chain-of-Thought Reasoning Through Diverse AI Feedback

Ju-Seung Byun, Jiyun Chun, Jihyung Kil, Andrew Perrault

TL;DR

A two-stage algorithm ARES that Alternates REinforcement Learning and Supervised Fine-Tuning and asks the Teacher to score how much each sentence contributes to solving the problem in a Chain-of-Thought (CoT) allows us to consider individual valuable segments, providing more granular rewards for the RL procedure.

Abstract

Large Multimodal Models (LMMs) excel at comprehending human instructions and demonstrate remarkable results across a broad spectrum of tasks. Reinforcement Learning from Human Feedback (RLHF) and AI Feedback (RLAIF) further refine LLMs by aligning them with specific preferences. These methods primarily use ranking-based feedback for entire generations. With advanced AI models (Teacher), such as GPT-4 and Claude 3 Opus, we can request various types of detailed feedback that are expensive for humans to provide. We propose a two-stage algorithm ARES that Alternates REinforcement Learning (RL) and Supervised Fine-Tuning (SFT). First, we request the Teacher to score how much each sentence contributes to solving the problem in a Chain-of-Thought (CoT). This sentence-level feedback allows us to consider individual valuable segments, providing more granular rewards for the RL procedure. Second, we ask the Teacher to correct the wrong reasoning after the RL stage. The RL procedure requires massive efforts for hyperparameter tuning and often generates errors like repetitive words and incomplete sentences. With the correction feedback, we stabilize the RL fine-tuned model through SFT. We conduct experiments on multi-model dataset ScienceQA and A-OKVQA to demonstrate the effectiveness of our proposal. ARES rationale reasoning achieves around 70% win rate against baseline models judged by GPT-4o. Additionally, we observe that the improved rationale reasoning leads to a 2.5% increase in inference answer accuracy on average for the multi-modal datasets.

ARES: Alternating Reinforcement Learning and Supervised Fine-Tuning for Enhanced Multi-Modal Chain-of-Thought Reasoning Through Diverse AI Feedback

TL;DR

A two-stage algorithm ARES that Alternates REinforcement Learning and Supervised Fine-Tuning and asks the Teacher to score how much each sentence contributes to solving the problem in a Chain-of-Thought (CoT) allows us to consider individual valuable segments, providing more granular rewards for the RL procedure.

Abstract

Large Multimodal Models (LMMs) excel at comprehending human instructions and demonstrate remarkable results across a broad spectrum of tasks. Reinforcement Learning from Human Feedback (RLHF) and AI Feedback (RLAIF) further refine LLMs by aligning them with specific preferences. These methods primarily use ranking-based feedback for entire generations. With advanced AI models (Teacher), such as GPT-4 and Claude 3 Opus, we can request various types of detailed feedback that are expensive for humans to provide. We propose a two-stage algorithm ARES that Alternates REinforcement Learning (RL) and Supervised Fine-Tuning (SFT). First, we request the Teacher to score how much each sentence contributes to solving the problem in a Chain-of-Thought (CoT). This sentence-level feedback allows us to consider individual valuable segments, providing more granular rewards for the RL procedure. Second, we ask the Teacher to correct the wrong reasoning after the RL stage. The RL procedure requires massive efforts for hyperparameter tuning and often generates errors like repetitive words and incomplete sentences. With the correction feedback, we stabilize the RL fine-tuned model through SFT. We conduct experiments on multi-model dataset ScienceQA and A-OKVQA to demonstrate the effectiveness of our proposal. ARES rationale reasoning achieves around 70% win rate against baseline models judged by GPT-4o. Additionally, we observe that the improved rationale reasoning leads to a 2.5% increase in inference answer accuracy on average for the multi-modal datasets.
Paper Structure (25 sections, 3 equations, 4 figures, 17 tables)

This paper contains 25 sections, 3 equations, 4 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Preliminaries
  4. Reinforcement Learning
  5. Correction: Supervised Fine-Tuning
  6. Algorithm Details
  7. Experimental Setup
  8. Experimental Results
  9. Emergent Behavior Through RL
  10. Guide RL with Correction
  11. Rationale Reasoning Comparison
  12. Inference Accuracy
  13. Related Work
  14. Conclusion
  15. Prompts
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Overview of ARES: ARES alternates RL and SFT. The Teacher provides scores (rewards) for each sentence for RL. Blue indicates the advantages of RL, and red indicates potential degeneration. ARES corrects the issues through the Teacher's correction feedback.
  • Figure 2: ARES pipeline: For a given model generating rationale reasoning, we request an AI model's sentence-level scores (ranging from $0.0$ to $1.0$). The closer the score is to $1.0$, the more it helps solve the problem. We proceed with the RL stage using these sentence-level scores. After RL, the training model may produce incorrect parts (colored in red), so we enhance the training model by requesting correction feedback from the AI model (colored in blue) and then proceed with the supervised fine-tuning stage.
  • Figure 3: Comparison of rationales: dataset, baseline, correction feedback for baseline, and ARES.
  • Figure 4: MM-CoT zhang2023multimodal uses two same size separate models for reasoning and inference. We replace the inference model with a LoRA adapter (only $1/10$ the weights) added to the rationale model.