Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zero-Shot and Efficient Clarification Need Prediction in Conversational Search

Lili Lu, Chuan Meng, Federico Ravenda, Mohammad Aliannejadi, Fabio Crestani

TL;DR

Problem: Clarification-need prediction (CNP) in conversational search suffers from limited training data and high latency when using large language models. Approach: Zef-CNP generates synthetic specific and ambiguous queries in zero-shot using TIQ-CoT and CoQu, then fine-tunes an efficient CNP model that operates without retrieval or LLM at inference. Contributions: (1) a fully zero-shot CNP framework, (2) TIQ-CoT prompting strategy, (3) CoQu counterfactual data generation, and (4) empirical results on ClariQ and AmbigNQ showing superior effectiveness and efficiency; data and code released as CoQu-CS. Impact: enables scalable, low-latency CNP for mixed-initiative conversational search across domains.

Abstract

Clarification need prediction (CNP) is a key task in conversational search, aiming to predict whether to ask a clarifying question or give an answer to the current user query. However, current research on CNP suffers from the issues of limited CNP training data and low efficiency. In this paper, we propose a zero-shot and efficient CNP framework (Zef-CNP), in which we first prompt large language models (LLMs) in a zero-shot manner to generate two sets of synthetic queries: ambiguous and specific (unambiguous) queries. We then use the generated queries to train efficient CNP models. Zef-CNP eliminates the need for human-annotated clarification-need labels during training and avoids the use of LLMs with high query latency at query time. To further improve the generation quality of synthetic queries, we devise a topic-, information-need-, and query-aware chain-of-thought (CoT) prompting strategy (TIQ-CoT). Moreover, we enhance TIQ-CoT with counterfactual query generation (CoQu), which guides LLMs first to generate a specific/ambiguous query and then sequentially generate its corresponding ambiguous/specific query. Experimental results show that Zef-CNP achieves superior CNP effectiveness and efficiency compared with zero- and few-shot LLM-based CNP predictors.

Zero-Shot and Efficient Clarification Need Prediction in Conversational Search

TL;DR

Problem: Clarification-need prediction (CNP) in conversational search suffers from limited training data and high latency when using large language models. Approach: Zef-CNP generates synthetic specific and ambiguous queries in zero-shot using TIQ-CoT and CoQu, then fine-tunes an efficient CNP model that operates without retrieval or LLM at inference. Contributions: (1) a fully zero-shot CNP framework, (2) TIQ-CoT prompting strategy, (3) CoQu counterfactual data generation, and (4) empirical results on ClariQ and AmbigNQ showing superior effectiveness and efficiency; data and code released as CoQu-CS. Impact: enables scalable, low-latency CNP for mixed-initiative conversational search across domains.

Abstract

Clarification need prediction (CNP) is a key task in conversational search, aiming to predict whether to ask a clarifying question or give an answer to the current user query. However, current research on CNP suffers from the issues of limited CNP training data and low efficiency. In this paper, we propose a zero-shot and efficient CNP framework (Zef-CNP), in which we first prompt large language models (LLMs) in a zero-shot manner to generate two sets of synthetic queries: ambiguous and specific (unambiguous) queries. We then use the generated queries to train efficient CNP models. Zef-CNP eliminates the need for human-annotated clarification-need labels during training and avoids the use of LLMs with high query latency at query time. To further improve the generation quality of synthetic queries, we devise a topic-, information-need-, and query-aware chain-of-thought (CoT) prompting strategy (TIQ-CoT). Moreover, we enhance TIQ-CoT with counterfactual query generation (CoQu), which guides LLMs first to generate a specific/ambiguous query and then sequentially generate its corresponding ambiguous/specific query. Experimental results show that Zef-CNP achieves superior CNP effectiveness and efficiency compared with zero- and few-shot LLM-based CNP predictors.

Paper Structure

This paper contains 16 sections, 3 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Task definition
  4. Methodology
  5. Zero-shot specific/ambiguous query generation with TIQ-CoT and CoQu
  6. TIQ-CoT.
  7. CoQu.
  8. Fine-tuning an efficient CNP model
  9. Applying the fine-tuned efficient CNP model for inference
  10. Experiments
  11. Results and analysis
  12. Impact of the choice of LLM on synthetic data generation
  13. Analysis of TIQ-CoT and CoQu
  14. Impact of synthetic data scale on CNP quality
  15. Room for improvement
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Our proposed zero-shot and efficient CNP framework (Zef-CNP). Please be aware that we implement TIQ-CoT and CoQu in the same prompt (see in Fig. \ref{['fig:prompt']}).
  • Figure 2: A topic-, information-need-, and query-aware CoT prompting strategy (TIQ-CoT) enhanced with counterfactual query generation (CoQu).
  • Figure 3: Prompt for LLM as clarification-need predictors.
  • Figure 4: CNP results of Zef-CNP in terms of weighted F1 w.r.t. the impact of different scales of generated data by Llama-3.1-8B-Instruct and GPT-4o-mini, on ClariQ and AmbigNQ.