Clarifying Ambiguities: on the Role of Ambiguity Types in Prompting Methods for Clarification Generation

TL;DR

This work tackles ambiguity in information retrieval by integrating ambiguity types into prompt-based clarifications. It introduces Ambiguity Type-Chain of Thought (AT-CoT), an action-based Ambiguity Type Taxonomy with Semantic, Generalize, and Specify categories, and couples this taxonomy with CoT prompting to generate clarifying questions. Evaluations on CG and IR tasks across diverse datasets show that AT-CoT consistently outperforms standard, AT-standard, and CoT prompting, with notable gains in CG quality and IR effectiveness, including robust performance across interaction modes. The study reveals a strong link between clarification quality and downstream retrieval, while acknowledging limitations in AT granularity and model-scale generalization, suggesting future work on finer-grained taxonomies and multi-model analyses.

Abstract

In information retrieval (IR), providing appropriate clarifications to better understand users' information needs is crucial for building a proactive search-oriented dialogue system. Due to the strong in-context learning ability of large language models (LLMs), recent studies investigate prompting methods to generate clarifications using few-shot or Chain of Thought (CoT) prompts. However, vanilla CoT prompting does not distinguish the characteristics of different information needs, making it difficult to understand how LLMs resolve ambiguities in user queries. In this work, we focus on the concept of ambiguity for clarification, seeking to model and integrate ambiguities in the clarification process. To this end, we comprehensively study the impact of prompting schemes based on reasoning and ambiguity for clarification. The idea is to enhance the reasoning abilities of LLMs by limiting CoT to predict first ambiguity types that can be interpreted as instructions to clarify, then correspondingly generate clarifications. We name this new prompting scheme Ambiguity Type-Chain of Thought (AT-CoT). Experiments are conducted on various datasets containing human-annotated clarifying questions to compare AT-CoT with multiple baselines. We also perform user simulations to implicitly measure the quality of generated clarifications under various IR scenarios.

Figures (3)

• Figure 1: Illustration of AT-CoT: Unlike vanilla CoT, LLM-generated reasoning is limited to predict possible ambiguity types.
• Figure 2: Ambiguity type taxonomies for analytical purposes in previous work.
• Figure 3: Illustration of the two clarification interaction scenarios in our user simulation: select and respond.