Which questions should I answer? Salience Prediction of Inquisitive Questions

TL;DR

This work tackles the problem of predicting which inquisitive questions raised during reading are most salient for understanding a text. It introduces QSalience, a dataset of 1,766 (context, question) pairs with human salience judgments, and shows that salient questions are more likely to be answered later in the article and correlate with improved summarization quality. The authors propose QSalience, an instruction-tuned predictor that outperforms strong GPT-4 baselines and achieves meaningful agreement with human annotators on salience. A pilot use-case demonstrates that summaries addressing more salient questions align with higher human judgments, suggesting practical value for improving long-form summarization and reader comprehension in discourse-heavy text. The work connects discourse-theoretic notions (potential questions, QUDs) with modern NLP to enable better question generation, selection, and downstream text expansion tasks.

Abstract

Inquisitive questions -- open-ended, curiosity-driven questions people ask as they read -- are an integral part of discourse processing (Kehler and Rohde, 2017; Onea, 2016) and comprehension (Prince, 2004). Recent work in NLP has taken advantage of question generation capabilities of LLMs to enhance a wide range of applications. But the space of inquisitive questions is vast: many questions can be evoked from a given context. So which of those should be prioritized to find answers? Linguistic theories, unfortunately, have not yet provided an answer to this question. This paper presents QSALIENCE, a salience predictor of inquisitive questions. QSALIENCE is instruction-tuned over our dataset of linguist-annotated salience scores of 1,766 (context, question) pairs. A question scores high on salience if answering it would greatly enhance the understanding of the text (Van Rooy, 2003). We show that highly salient questions are empirically more likely to be answered in the same article, bridging potential questions (Onea, 2016) with Questions Under Discussion (Roberts, 2012). We further validate our findings by showing that answering salient questions is an indicator of summarization quality in news.

Figures (6)

• Figure 1: Examples of inquisitive questions and their annotated salience (with rationales). Each question is evoked by an anchor sentence (shown in the same highlight color). Whether the question is answered is shown on the right. Q1 is taken from human-annotated QUDs in DCQA ko-etal-2022-discourse; Q2-5 are GPT-4 generated questions.
• Figure 2: This example illustrates the expanded summarization task and how question salience is used in evaluation. The summaries are generated by expanding on a short TL;DR of an article. We show the inquisitive questions generated from the TL;DR and their salience scores. A better summary (left) answers more salient questions than the worse summary (right), where only one medium-salient question is answered.
• Figure 3: GPT-4-turbo zero-shot vanilla (left), GPT-4-turbo few-shot vanilla (right)
• Figure 4: Mistral-7B-Instruct (left), Llama-2-7B-chat (right)
• Figure 5: Flan-t5-base (left), TinyLlama-1.1B-chat (right)