Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

I Could've Asked That: Reformulating Unanswerable Questions

Wenting Zhao, Ge Gao, Claire Cardie, Alexander M. Rush

TL;DR

This work tackles reformulating unanswerable questions in document-grounded QA by introducing CouldAsk, a benchmark that combines existing datasets with new cross-domain data to study both detection of unanswerability and ground-truth reformulation into answerable, document-grounded queries. The authors propose a reference-free evaluation framework using edit distance, entity overlap, and cosine similarity, plus a trained classifier to judge answerability, and they release the benchmark and code. Empirical results reveal substantial gaps in current models: reformulation success remains low (GPT-4 average 26.21%), with many failures arising from mere rephrasing or global edits that are difficult to implement. The work highlights domain-related challenges, analyzes prompting strategies, and provides metrics that better align with human judgments, offering a path toward more user-centric, grounded QA systems.

Abstract

When seeking information from unfamiliar documents, users frequently pose questions that cannot be answered by the documents. While existing large language models (LLMs) identify these unanswerable questions, they do not assist users in reformulating their questions, thereby reducing their overall utility. We curate CouldAsk, an evaluation benchmark composed of existing and new datasets for document-grounded question answering, specifically designed to study reformulating unanswerable questions. We evaluate state-of-the-art open-source and proprietary LLMs on CouldAsk. The results demonstrate the limited capabilities of these models in reformulating questions. Specifically, GPT-4 and Llama2-7B successfully reformulate questions only 26% and 12% of the time, respectively. Error analysis shows that 62% of the unsuccessful reformulations stem from the models merely rephrasing the questions or even generating identical questions. We publicly release the benchmark and the code to reproduce the experiments.

I Could've Asked That: Reformulating Unanswerable Questions

TL;DR

This work tackles reformulating unanswerable questions in document-grounded QA by introducing CouldAsk, a benchmark that combines existing datasets with new cross-domain data to study both detection of unanswerability and ground-truth reformulation into answerable, document-grounded queries. The authors propose a reference-free evaluation framework using edit distance, entity overlap, and cosine similarity, plus a trained classifier to judge answerability, and they release the benchmark and code. Empirical results reveal substantial gaps in current models: reformulation success remains low (GPT-4 average 26.21%), with many failures arising from mere rephrasing or global edits that are difficult to implement. The work highlights domain-related challenges, analyzes prompting strategies, and provides metrics that better align with human judgments, offering a path toward more user-centric, grounded QA systems.

Abstract

When seeking information from unfamiliar documents, users frequently pose questions that cannot be answered by the documents. While existing large language models (LLMs) identify these unanswerable questions, they do not assist users in reformulating their questions, thereby reducing their overall utility. We curate CouldAsk, an evaluation benchmark composed of existing and new datasets for document-grounded question answering, specifically designed to study reformulating unanswerable questions. We evaluate state-of-the-art open-source and proprietary LLMs on CouldAsk. The results demonstrate the limited capabilities of these models in reformulating questions. Specifically, GPT-4 and Llama2-7B successfully reformulate questions only 26% and 12% of the time, respectively. Error analysis shows that 62% of the unsuccessful reformulations stem from the models merely rephrasing the questions or even generating identical questions. We publicly release the benchmark and the code to reproduce the experiments.
Paper Structure (31 sections, 5 figures, 15 tables)

This paper contains 31 sections, 5 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Task: Question Reformulation
  4. The CouldAsk Benchmark
  5. Datasets
  6. BBC, Reddit, and Yelp.
  7. SQuADv2, QA$^2$, and BanditQA.
  8. Evaluating Reformulation
  9. Experimental Setup
  10. Models.
  11. Comparisons.
  12. Results
  13. Detecting Unanswerable Questions
  14. Reformulating Questions
  15. Analysis
  16. ...and 16 more sections

Figures (5)

  • Figure 1: An example of an LLM suggesting an alternative relevant question the user could have asked whose answers can be found in the document, besides only informing users with the presupposition errors.
  • Figure 2: Example outputs produced by GPT-4. Via prompting, the model detects unanswerable questions then reformulates the questions with a second prompt.
  • Figure 3: The relation between the percentage of reformulations answerable by the documents and the type of minimum spans in the original questions. q span are the examples where the minimum span is the full question, while short span includes all other instances.
  • Figure 4: Qualitative analysis on the rule-based heuristic approach to hack our proposed metric, success rates.
  • Figure 5: Annotation guideline for crowdworkers to annotate unanswerable questions.