Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReZero: Enhancing LLM search ability by trying one-more-time

Alan Dao, Thinh Le

TL;DR

The paper addresses the vulnerability of Retrieval-Augmented Generation to poor initial search queries by punishing premature abandonment. It proposes ReZero, a reinforcement learning framework using Group Relative Policy Optimization to explicitly reward the act of retrying a search after an initial failure, thereby promoting persistence in information gathering. On the Apollo-3 dataset, ReZero achieves a peak accuracy of $46.88\%$ compared to a $25.00\%$ baseline, demonstrating that encouraging retries can significantly improve knowledge retrieval robustness. Limitations include domain confinement to a single dataset and potential instability in long RL training runs, with future work aiming at generalization across domains, stabilization of training, and integration with complementary RAG techniques.

Abstract

Retrieval-Augmented Generation (RAG) improves Large Language Model (LLM) performance on knowledge-intensive tasks but depends heavily on initial search query quality. Current methods, often using Reinforcement Learning (RL), typically focus on query formulation or reasoning over results, without explicitly encouraging persistence after a failed search. We introduce ReZero (Retry-Zero), a novel RL framework that directly rewards the act of retrying a search query following an initial unsuccessful attempt. This incentivizes the LLM to explore alternative queries rather than prematurely halting. ReZero demonstrates significant improvement, achieving 46.88% accuracy compared to a 25% baseline. By rewarding persistence, ReZero enhances LLM robustness in complex information-seeking scenarios where initial queries may prove insufficient.

ReZero: Enhancing LLM search ability by trying one-more-time

TL;DR

The paper addresses the vulnerability of Retrieval-Augmented Generation to poor initial search queries by punishing premature abandonment. It proposes ReZero, a reinforcement learning framework using Group Relative Policy Optimization to explicitly reward the act of retrying a search after an initial failure, thereby promoting persistence in information gathering. On the Apollo-3 dataset, ReZero achieves a peak accuracy of compared to a baseline, demonstrating that encouraging retries can significantly improve knowledge retrieval robustness. Limitations include domain confinement to a single dataset and potential instability in long RL training runs, with future work aiming at generalization across domains, stabilization of training, and integration with complementary RAG techniques.

Abstract

Retrieval-Augmented Generation (RAG) improves Large Language Model (LLM) performance on knowledge-intensive tasks but depends heavily on initial search query quality. Current methods, often using Reinforcement Learning (RL), typically focus on query formulation or reasoning over results, without explicitly encouraging persistence after a failed search. We introduce ReZero (Retry-Zero), a novel RL framework that directly rewards the act of retrying a search query following an initial unsuccessful attempt. This incentivizes the LLM to explore alternative queries rather than prematurely halting. ReZero demonstrates significant improvement, achieving 46.88% accuracy compared to a 25% baseline. By rewarding persistence, ReZero enhances LLM robustness in complex information-seeking scenarios where initial queries may prove insufficient.

Paper Structure

This paper contains 13 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Retrieval-Augmented Generation (RAG)
  4. Learning and Search for Reasoning in RAG
  5. Reinforcement Learning for LLM Alignment and Reasoning
  6. Methodology
  7. Overview
  8. Reinforcement Learning Framework
  9. Reward Functions
  10. Training Process
  11. Experiments and Results
  12. Discussion
  13. Conclusion

Figures (2)

  • Figure 1: Receives a reward signal for retrying after failure.
  • Figure 2: Comparison of evaluation accuracy between the ReZero model (incorporating the reward_retry component) and the Baseline model (lacking the retry incentive) over 1000 training steps on the held-out Apollo 3 dataset chunks. Peak accuracies are highlighted.