Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflection

Kyungjae Lee, Dasol Hwang, Sunghyun Park, Youngsoo Jang, Moontae Lee

TL;DR

A novel framework: Reinforcement Learning from Reflective Feedback (RLRF), which leverages fine-grained feedback based on detailed criteria to improve the core capabilities of LLMs beyond superficial surface-level adjustment is proposed.

Abstract

Despite the promise of RLHF in aligning LLMs with human preferences, it often leads to superficial alignment, prioritizing stylistic changes over improving downstream performance of LLMs. Underspecified preferences could obscure directions to align the models. Lacking exploration restricts identification of desirable outputs to improve the models. To overcome these challenges, we propose a novel framework: Reinforcement Learning from Reflective Feedback (RLRF), which leverages fine-grained feedback based on detailed criteria to improve the core capabilities of LLMs. RLRF employs a self-reflection mechanism to systematically explore and refine LLM responses, then fine-tuning the models via a RL algorithm along with promising responses. Our experiments across Just-Eval, Factuality, and Mathematical Reasoning demonstrate the efficacy and transformative potential of RLRF beyond superficial surface-level adjustment.

Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflection

TL;DR

A novel framework: Reinforcement Learning from Reflective Feedback (RLRF), which leverages fine-grained feedback based on detailed criteria to improve the core capabilities of LLMs beyond superficial surface-level adjustment is proposed.

Abstract

Despite the promise of RLHF in aligning LLMs with human preferences, it often leads to superficial alignment, prioritizing stylistic changes over improving downstream performance of LLMs. Underspecified preferences could obscure directions to align the models. Lacking exploration restricts identification of desirable outputs to improve the models. To overcome these challenges, we propose a novel framework: Reinforcement Learning from Reflective Feedback (RLRF), which leverages fine-grained feedback based on detailed criteria to improve the core capabilities of LLMs. RLRF employs a self-reflection mechanism to systematically explore and refine LLM responses, then fine-tuning the models via a RL algorithm along with promising responses. Our experiments across Just-Eval, Factuality, and Mathematical Reasoning demonstrate the efficacy and transformative potential of RLRF beyond superficial surface-level adjustment.
Paper Structure (33 sections, 6 equations, 5 figures, 16 tables)

This paper contains 33 sections, 6 equations, 5 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Preference-based RLHF
  4. Challenges in Improving the Capabilities of LLM via preference-based RLHF
  5. RL from Reflective Feedback (RLRF)
  6. Fine-Grained Feedback Model
  7. Fine-Grained Self-Reflection
  8. RL Fine-tuning
  9. Iterative Training
  10. Experiment
  11. Experimental Setup
  12. Training Dataset
  13. Base Model and Hyper-parameters
  14. Baselines for Comparison
  15. Evaluation Benchmarks
  16. ...and 18 more sections

Figures (5)

  • Figure 1: An overview of our proposed Reinforcement Learning from Reflective Feedback (RLRF).
  • Figure 2: On Math reasoning (GSM8K). We observed that reward or feedback scores indicate the correctness of responses. (Left) Reward Scores, (Center) Feedback Scores with Reference, (Right) Feedback Scores without Reference.
  • Figure 3: On the factuality task. (Left) Reward Scores, (Center) Feedback Scores with Reference, (Right) Feedback Scores without Reference.
  • Figure 4: Results on different numbers of samples in each stage: Generating responses, feedbacks, or refined responses. The y-axis is the total scores on Just-Eval.
  • Figure 5: Average Token Lengths of several models.