Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ClarifyMT-Bench: Benchmarking and Improving Multi-Turn Clarification for Conversational Large Language Models

Sichun Luo, Yi Huang, Mukai Li, Shichang Meng, Fengyuan Liu, Zefa Hu, Junlan Feng, Qi Liu

TL;DR

ClarifyMT-Bench introduces a principled, multi-turn clarification benchmark for open-domain LLMs based on a five-dimensional ambiguity taxonomy and six user personas, yielding 6,120 dialogues to study ask–answer decisions under noisy interactions. The work reveals a consistent under-clarification bias and degradation of performance with dialogue depth across ten LLMs. To address this, the authors propose ClarifyAgent, a perception–forecasting–tracking–planning framework that adds user-persona inference to guide clarification, achieving robust improvements over baselines. Together, these contributions establish a reproducible platform for evaluating and improving how LLMs handle ambiguity in realistic, multi-turn conversations, with implications for safer and more reliable interactive AI.

Abstract

Large language models (LLMs) are increasingly deployed as conversational assistants in open-domain, multi-turn settings, where users often provide incomplete or ambiguous information. However, existing LLM-focused clarification benchmarks primarily assume single-turn interactions or cooperative users, limiting their ability to evaluate clarification behavior in realistic settings. We introduce \textbf{ClarifyMT-Bench}, a benchmark for multi-turn clarification grounded in a five-dimensional ambiguity taxonomy and a set of six behaviorally diverse simulated user personas. Through a hybrid LLM-human pipeline, we construct 6,120 multi-turn dialogues capturing diverse ambiguity sources and interaction patterns. Evaluating ten representative LLMs uncovers a consistent under-clarification bias: LLMs tend to answer prematurely, and performance degrades as dialogue depth increases. To mitigate this, we propose \textbf{ClarifyAgent}, an agentic approach that decomposes clarification into perception, forecasting, tracking, and planning, substantially improving robustness across ambiguity conditions. ClarifyMT-Bench establishes a reproducible foundation for studying when LLMs should ask, when they should answer, and how to navigate ambiguity in real-world human-LLM interactions.

ClarifyMT-Bench: Benchmarking and Improving Multi-Turn Clarification for Conversational Large Language Models

TL;DR

ClarifyMT-Bench introduces a principled, multi-turn clarification benchmark for open-domain LLMs based on a five-dimensional ambiguity taxonomy and six user personas, yielding 6,120 dialogues to study ask–answer decisions under noisy interactions. The work reveals a consistent under-clarification bias and degradation of performance with dialogue depth across ten LLMs. To address this, the authors propose ClarifyAgent, a perception–forecasting–tracking–planning framework that adds user-persona inference to guide clarification, achieving robust improvements over baselines. Together, these contributions establish a reproducible platform for evaluating and improving how LLMs handle ambiguity in realistic, multi-turn conversations, with implications for safer and more reliable interactive AI.

Abstract

Large language models (LLMs) are increasingly deployed as conversational assistants in open-domain, multi-turn settings, where users often provide incomplete or ambiguous information. However, existing LLM-focused clarification benchmarks primarily assume single-turn interactions or cooperative users, limiting their ability to evaluate clarification behavior in realistic settings. We introduce \textbf{ClarifyMT-Bench}, a benchmark for multi-turn clarification grounded in a five-dimensional ambiguity taxonomy and a set of six behaviorally diverse simulated user personas. Through a hybrid LLM-human pipeline, we construct 6,120 multi-turn dialogues capturing diverse ambiguity sources and interaction patterns. Evaluating ten representative LLMs uncovers a consistent under-clarification bias: LLMs tend to answer prematurely, and performance degrades as dialogue depth increases. To mitigate this, we propose \textbf{ClarifyAgent}, an agentic approach that decomposes clarification into perception, forecasting, tracking, and planning, substantially improving robustness across ambiguity conditions. ClarifyMT-Bench establishes a reproducible foundation for studying when LLMs should ask, when they should answer, and how to navigate ambiguity in real-world human-LLM interactions.
Paper Structure (21 sections, 7 equations, 9 figures, 8 tables)

This paper contains 21 sections, 7 equations, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Taxonomy and Task
  4. Ambiguity Taxonomy in the Era of LLMs
  5. Task Formulation
  6. Dataset Construction
  7. Single-Turn Dialogue Generation
  8. Multi-Turn Dialogue Generation
  9. Data Statistics
  10. Evaluation
  11. Evaluation Setup
  12. Main Results
  13. Task 1: Multi-Turn Ambiguous Query Clarification
  14. Task 2: Clarifying Question Quality Evaluation
  15. ClarifyAgent: An Agentic Method for Multi-Turn Clarification
  16. ...and 6 more sections

Figures (9)

  • Figure 1: Illustration of clarification in user–LLM interactions. The upper example shows an under-clarified response that fails to capture user intent, while the lower example demonstrates effective clarification through follow-up questions, leading to user satisfaction. The user response may contain contradictory or vague information, labeled in red and gray respectively.
  • Figure 2: The pipeline of dataset construction.
  • Figure 3: Clarifying question quality for each ambiguity subtype evaluated by LLM-as-a-Judge.
  • Figure 4: Clarifying question quality evaluated by human and LLM-as-a-Judge.
  • Figure 5: Pipeline of ClarifyAgent.
  • ...and 4 more figures