ICPO: Illocution-Calibrated Policy Optimization for Multi-Turn Conversation

TL;DR

The paper tackles the lost-in-conversation problem in multi-turn LLMs, showing that standard RLVR encourages overconfident, direct answers when prompts are ambiguous. It introduces Illocution-Calibrated Policy Optimization (ICPO), which augments training with underspecified prompts and rewards models based on illocutionary intent to promote humility and clarification. Empirical results demonstrate about a 75% improvement in multi-turn reasoning with ICPO while maintaining single-turn performance, along with higher output diversity and entropy compared to baseline RLVR and entropy-regularized variants. The approach offers a practical path toward more robust, collaborative conversational AI capable of adaptively seeking clarification and avoiding premature conclusions.

Abstract

Large Language Models (LLMs) in multi-turn conversations often suffer from a ``lost-in-conversation'' phenomenon, where they struggle to recover from early incorrect assumptions, particularly when users provide ambiguous initial instructions. We find that standard post-training techniques like Reinforcement Learning with Verifiable Rewards (RLVR) exacerbate this issue by rewarding confident, direct answers, thereby inducing overconfidence and discouraging the model from seeking clarification. To address this, we propose Illocution-Calibrated Policy Optimization (ICPO), a novel training framework that sensitizes the model to instruction ambiguity. ICPO augments the training corpus with underspecified prompts and conditions the reward signal on the user's illocutionary intent, rewarding the model for expressing uncertainty or asking for clarification when faced with ambiguity. Experiments demonstrate that ICPO fosters appropriate humility, yielding a substantial average improvement of 75\% in multi-turn conversation, while preserving robust performance on single-turn benchmarks. Our work presents a practical path toward more robust and collaborative conversational AI that can better navigate the nuances of human interaction.

Figures (5)

• Figure 1: Average distribution of response type during multi-turn conversations starting with ambiguous instructions, in which the instruction conditions gradually become complete as the conversation turns progress.
• Figure 2: A breakdown of the averaged multi-turn performance of the Qwen2.5-Instruct series, categorized by how early the LLM make its first answer attempt in the conversation.
• Figure 3: Overview of ICPO. Unlike standard RLVR which rewards only final outcomes, ICPO conditions the reward signal on the user's illocutionary intent. By incorporating underspecified prompts via Scenario Simulation and rewarding calibrated humility through Illocutionary Judgment, the model learns to avoid overconfident hallucinations.
• Figure 4: Entropy comparison between Standard RLVR and ICPO during training.
• Figure 5: Conversation example, in which the user starts the conversation with a problem that has insufficient conditions and cannot be solved.