Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-turn Training with Basic Human Feedback Helps Little on LLM Reasoning

Qiang Liu, Wuganjing Song, Zhenzhou Lin, Feifan Chen, Qiaolong Cai, Chen Li, Yongduo Sui

TL;DR

The paper investigates whether multi-turn training with basic human feedback improves LLM reasoning, focusing on complete-information tasks such as GSM8K. It defines three multi-turn training strategies (UACR, ULCR, UADR) and evaluates them against single-turn training using $K$-turn and Pass@$K$ assessments with GRPO-based RL. The main finding is that single-turn training generalizes well to both single- and multi-turn inference, while multi-turn training provides little to no benefit and can degrade single-turn reasoning. A key contribution is arguing that for complete-information tasks, robust single-turn training is preferable, and multi-turn feedback offers limited practical gains.

Abstract

The reasoning capabilities of Large Language Models (LLMs) are typically developed through the single-turn reinforcement learning, whereas real-world applications often involve multi-turn interactions with human feedback, leading to a potential mismatch between training and deployment conditions. In this work, we study whether multi-turn training with human feedback is necessary for reasoning tasks. We compare conventional single-turn training with three multi-turn strategies and reach contrary conclusions to previous research. We find that models trained in a single-turn setting generalize effectively to both single- and multi-turn evaluations, while models trained with multi-turn strategies exhibit a significant degradation in single-turn reasoning performance. These results suggest that for tasks with complete information, robust single-turn training remains more effective and reliable, as multi-turn training with basic feedback provides limited benefits and can even degrade reasoning capabilities.

Multi-turn Training with Basic Human Feedback Helps Little on LLM Reasoning

TL;DR

The paper investigates whether multi-turn training with basic human feedback improves LLM reasoning, focusing on complete-information tasks such as GSM8K. It defines three multi-turn training strategies (UACR, ULCR, UADR) and evaluates them against single-turn training using -turn and Pass@ assessments with GRPO-based RL. The main finding is that single-turn training generalizes well to both single- and multi-turn inference, while multi-turn training provides little to no benefit and can degrade single-turn reasoning. A key contribution is arguing that for complete-information tasks, robust single-turn training is preferable, and multi-turn feedback offers limited practical gains.

Abstract

The reasoning capabilities of Large Language Models (LLMs) are typically developed through the single-turn reinforcement learning, whereas real-world applications often involve multi-turn interactions with human feedback, leading to a potential mismatch between training and deployment conditions. In this work, we study whether multi-turn training with human feedback is necessary for reasoning tasks. We compare conventional single-turn training with three multi-turn strategies and reach contrary conclusions to previous research. We find that models trained in a single-turn setting generalize effectively to both single- and multi-turn evaluations, while models trained with multi-turn strategies exhibit a significant degradation in single-turn reasoning performance. These results suggest that for tasks with complete information, robust single-turn training remains more effective and reliable, as multi-turn training with basic feedback provides limited benefits and can even degrade reasoning capabilities.
Paper Structure (10 sections, 6 equations, 4 figures)

This paper contains 10 sections, 6 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Reasoning ability enhancement with reinforcement learning
  3. Multi-turn reasoning
  4. Single-turn and multi-turn reasoning ability
  5. Experimental results
  6. UACR and ULCR: Single-turn Performance Gap between Single- and Multi-turn Training
  7. UADR: Reward the Model Solving with Fewer Turns
  8. Compare among three strategies
  9. Conclusions
  10. Complete and incomplete information: a rough categorization of tasks

Figures (4)

  • Figure 1: UACR experimental results: both single- and multi- turn trained models are evaluated with Pass@$8$ and $8$-turn inferences. Multi-turn trained model shows an inferior performance on single-turn inference and no obvious performance advantage on multi-turn inference
  • Figure 2: ULCR experimental results: similar to that of UACR
  • Figure 3: UADR experimental results: similar to that of UACR, but there is an improvement of the multi-turn trained model performing on single-turn inference
  • Figure 4: An overview of the three training strategies by averaging over steps