Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models

Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, Tat-Seng Chua

TL;DR

CLAMBER introduces a taxonomy-based benchmark to evaluate LLMs on identifying and clarifying ambiguous information needs. It structures ambiguity into Epistemic Misalignment, Linguistic Ambiguity, and Aleatoric Output across eight subcategories and provides ~12K high-quality data for comprehensive evaluation. Across five LLMs and four prompting schemes, results show current models struggle with ambiguity identification and clarifying questions, with CoT and few-shot prompting sometimes inducing overconfidence, especially in smaller models, and semantic/referential ambiguities proving particularly hard. The work offers guidance for developing proactive, trustworthy LLMs and releases the dataset to support ongoing research.

Abstract

Large language models (LLMs) are increasingly used to meet user information needs, but their effectiveness in dealing with user queries that contain various types of ambiguity remains unknown, ultimately risking user trust and satisfaction. To this end, we introduce CLAMBER, a benchmark for evaluating LLMs using a well-organized taxonomy. Building upon the taxonomy, we construct ~12K high-quality data to assess the strengths, weaknesses, and potential risks of various off-the-shelf LLMs. Our findings indicate the limited practical utility of current LLMs in identifying and clarifying ambiguous user queries, even enhanced by chain-of-thought (CoT) and few-shot prompting. These techniques may result in overconfidence in LLMs and yield only marginal enhancements in identifying ambiguity. Furthermore, current LLMs fall short in generating high-quality clarifying questions due to a lack of conflict resolution and inaccurate utilization of inherent knowledge. In this paper, CLAMBER presents a guidance and promotes further research on proactive and trustworthy LLMs. Our dataset is available at https://github.com/zt991211/CLAMBER

CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models

TL;DR

CLAMBER introduces a taxonomy-based benchmark to evaluate LLMs on identifying and clarifying ambiguous information needs. It structures ambiguity into Epistemic Misalignment, Linguistic Ambiguity, and Aleatoric Output across eight subcategories and provides ~12K high-quality data for comprehensive evaluation. Across five LLMs and four prompting schemes, results show current models struggle with ambiguity identification and clarifying questions, with CoT and few-shot prompting sometimes inducing overconfidence, especially in smaller models, and semantic/referential ambiguities proving particularly hard. The work offers guidance for developing proactive, trustworthy LLMs and releases the dataset to support ongoing research.

Abstract

Large language models (LLMs) are increasingly used to meet user information needs, but their effectiveness in dealing with user queries that contain various types of ambiguity remains unknown, ultimately risking user trust and satisfaction. To this end, we introduce CLAMBER, a benchmark for evaluating LLMs using a well-organized taxonomy. Building upon the taxonomy, we construct ~12K high-quality data to assess the strengths, weaknesses, and potential risks of various off-the-shelf LLMs. Our findings indicate the limited practical utility of current LLMs in identifying and clarifying ambiguous user queries, even enhanced by chain-of-thought (CoT) and few-shot prompting. These techniques may result in overconfidence in LLMs and yield only marginal enhancements in identifying ambiguity. Furthermore, current LLMs fall short in generating high-quality clarifying questions due to a lack of conflict resolution and inaccurate utilization of inherent knowledge. In this paper, CLAMBER presents a guidance and promotes further research on proactive and trustworthy LLMs. Our dataset is available at https://github.com/zt991211/CLAMBER
Paper Structure (28 sections, 3 figures, 20 tables)

This paper contains 28 sections, 3 figures, 20 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. CLAMBER Benchmark
  4. Epistemic Misalignment (EM)
  5. Linguistic Ambiguity (LA)
  6. Aleatoric Output (AO)
  7. Validation and Revision
  8. Experimental Design
  9. Task 1: Identifying Ambiguity
  10. Overall Evaluation
  11. Fine-Grained Evaluation
  12. Task 2: Asking Clarifying Questions
  13. Overall Evaluation
  14. Fine-Grained Evaluation
  15. Conclusion
  16. ...and 13 more sections

Figures (3)

  • Figure 1: Investigation on the identification accuracy when handling ambiguous (i.e, Acc@1) versus unambiguous queries (i.e, Acc@0). We report the results under Zero-shot w/o CoT setting. Small-scale LLMs tend to classify most queries as ambiguous.
  • Figure 2: Performance of ChatGPT enhanced with multiple examples. We ensure a variety of categories in the examples and maintain an equal balance of ambiguous and unambiguous instances.
  • Figure 3: The statistics of error analysis. ChatGPT is unable to recognize their knowledge gap for the inadequacies in asking the effective clarifying questions.