Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Verifiable Accuracy and Abstention Rewards in Curriculum RL to Alleviate Lost-in-Conversation

Ming Li

TL;DR

This work tackles the Lost-in-Conversation phenomenon in multi-turn LLMs by introducing RLAAR, a curriculum-based reinforcement learning framework with verifiable accuracy and abstention rewards. By employing on-policy multi-turn rollouts, a mixed reward structure, and a competence-gated curriculum, the method teaches models to balance solving with informed abstention. Empirical results on LiC benchmarks show significant LiC reduction (from about 62.6% to 75.1%) and calibrated abstention (from about 33.5% to 73.4%), demonstrating improved reliability and trustworthiness in multi-turn interactions. The approach provides a practical recipe for building multi-turn, trustworthy LLMs, with potential applicability beyond math/code tasks to broader knowledge-intensive dialogues.

Abstract

Large Language Models demonstrate strong capabilities in single-turn instruction following but suffer from Lost-in-Conversation (LiC), a degradation in performance as information is revealed progressively in multi-turn settings. Motivated by the current progress on Reinforcement Learning with Verifiable Rewards (RLVR), we propose Curriculum Reinforcement Learning with Verifiable Accuracy and Abstention Rewards (RLAAR), a framework that encourages models not only to generate correct answers, but also to judge the solvability of questions in the multi-turn conversation setting. Our approach employs a competence-gated curriculum that incrementally increases dialogue difficulty (in terms of instruction shards), stabilizing training while promoting reliability. Using multi-turn, on-policy rollouts and a mixed-reward system, RLAAR teaches models to balance problem-solving with informed abstention, reducing premature answering behaviors that cause LiC. Evaluated on LiC benchmarks, RLAAR significantly mitigates LiC performance decay (62.6% to 75.1%) and improves calibrated abstention rates (33.5% to 73.4%). Together, these results provide a practical recipe for building multi-turn reliable and trustworthy LLMs.

Verifiable Accuracy and Abstention Rewards in Curriculum RL to Alleviate Lost-in-Conversation

TL;DR

This work tackles the Lost-in-Conversation phenomenon in multi-turn LLMs by introducing RLAAR, a curriculum-based reinforcement learning framework with verifiable accuracy and abstention rewards. By employing on-policy multi-turn rollouts, a mixed reward structure, and a competence-gated curriculum, the method teaches models to balance solving with informed abstention. Empirical results on LiC benchmarks show significant LiC reduction (from about 62.6% to 75.1%) and calibrated abstention (from about 33.5% to 73.4%), demonstrating improved reliability and trustworthiness in multi-turn interactions. The approach provides a practical recipe for building multi-turn, trustworthy LLMs, with potential applicability beyond math/code tasks to broader knowledge-intensive dialogues.

Abstract

Large Language Models demonstrate strong capabilities in single-turn instruction following but suffer from Lost-in-Conversation (LiC), a degradation in performance as information is revealed progressively in multi-turn settings. Motivated by the current progress on Reinforcement Learning with Verifiable Rewards (RLVR), we propose Curriculum Reinforcement Learning with Verifiable Accuracy and Abstention Rewards (RLAAR), a framework that encourages models not only to generate correct answers, but also to judge the solvability of questions in the multi-turn conversation setting. Our approach employs a competence-gated curriculum that incrementally increases dialogue difficulty (in terms of instruction shards), stabilizing training while promoting reliability. Using multi-turn, on-policy rollouts and a mixed-reward system, RLAAR teaches models to balance problem-solving with informed abstention, reducing premature answering behaviors that cause LiC. Evaluated on LiC benchmarks, RLAAR significantly mitigates LiC performance decay (62.6% to 75.1%) and improves calibrated abstention rates (33.5% to 73.4%). Together, these results provide a practical recipe for building multi-turn reliable and trustworthy LLMs.
Paper Structure (32 sections, 3 equations, 1 figure, 3 tables)

This paper contains 32 sections, 3 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multi-turn Conversation
  4. Response Abstention
  5. RL in LLMs
  6. Methodology
  7. Problem Formulation
  8. Curriculum RL with Verifiable Accuracy and Abstention Rewards
  9. Multi-turn Rollout
  10. Solvable-Single.
  11. Solvable-Multi.
  12. Unsolvable-Multi.
  13. Mixed Rewards
  14. Accuracy Reward ($r_{\text{acc}}$).
  15. Abstention Reward ($r_{\text{abs}}$).
  16. ...and 17 more sections

Figures (1)

  • Figure 1: The common single-turn interactive setting with LLMs (upper) and the multi-turn setting proposed in "Lost in Conversation" (LiC). The multi-turn setting simulates the real-world scenario where the user gradually reveals the requirements of the task through multiple turns of dialogue with LLMs. laban2025llms shows even the state-of-the-art models encounter severe performance declines. The left panel shows the situation of most of the current LLMs, which try to solve the problem prematurely without checking the solvability of the question. The right panel shows the desired behavior of the model, where we novelly reveal the connections between LiC and models' abstention capability, which critically checks the solvability of the question and then solves the problem.