Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering with Multi-Granularity Answers

Gal Yona, Roee Aharoni, Mor Geva

TL;DR

This work targets the knowledge evaluation gap in open-domain QA by introducing GRANOLA QA, which evaluates predictions with both accuracy and informativeness against multi-granularity gold answers. It provides a practical methodology to enrich datasets to GRANOLA style, demonstrated by Granola-EQ built from EntityQuestions with WikiData guidance. The paper then introduces DRAG, a decoding strategy that aggregates multiple model outputs to align answer granularity with the model’s uncertainty, significantly boosting GRANOLA accuracy (≈ up to 20 points) and informativeness, especially for rare entities. Overall, GRANOLA reveals that standard evaluation substantially underestimates a model’s factual knowledge and that granularity-aware evaluation and decoding can yield more reliable, informative QA outputs with practical implications for deploying LMs in factual tasks.

Abstract

Factual questions typically can be answered correctly at different levels of granularity. For example, both ``August 4, 1961'' and ``1961'' are correct answers to the question ``When was Barack Obama born?''. Standard question answering (QA) evaluation protocols, however, do not explicitly take this into account and compare a predicted answer against answers of a single granularity level. In this work, we propose GRANOLA QA, a novel evaluation setting where a predicted answer is evaluated in terms of accuracy and informativeness against a set of multi-granularity answers. We present a simple methodology for enriching existing datasets with multi-granularity answers, and create GRANOLA-EQ, a multi-granularity version of the EntityQuestions dataset. We evaluate a range of decoding methods on GRANOLA-EQ, including a new algorithm, called Decoding with Response Aggregation (DRAG), that is geared towards aligning the response granularity with the model's uncertainty. Our experiments show that large language models with standard decoding tend to generate specific answers, which are often incorrect. In contrast, when evaluated on multi-granularity answers, DRAG yields a nearly 20 point increase in accuracy on average, which further increases for rare entities. Overall, this reveals that standard evaluation and decoding schemes may significantly underestimate the knowledge encapsulated in LMs.

Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering with Multi-Granularity Answers

TL;DR

This work targets the knowledge evaluation gap in open-domain QA by introducing GRANOLA QA, which evaluates predictions with both accuracy and informativeness against multi-granularity gold answers. It provides a practical methodology to enrich datasets to GRANOLA style, demonstrated by Granola-EQ built from EntityQuestions with WikiData guidance. The paper then introduces DRAG, a decoding strategy that aggregates multiple model outputs to align answer granularity with the model’s uncertainty, significantly boosting GRANOLA accuracy (≈ up to 20 points) and informativeness, especially for rare entities. Overall, GRANOLA reveals that standard evaluation substantially underestimates a model’s factual knowledge and that granularity-aware evaluation and decoding can yield more reliable, informative QA outputs with practical implications for deploying LMs in factual tasks.

Abstract

Factual questions typically can be answered correctly at different levels of granularity. For example, both ``August 4, 1961'' and ``1961'' are correct answers to the question ``When was Barack Obama born?''. Standard question answering (QA) evaluation protocols, however, do not explicitly take this into account and compare a predicted answer against answers of a single granularity level. In this work, we propose GRANOLA QA, a novel evaluation setting where a predicted answer is evaluated in terms of accuracy and informativeness against a set of multi-granularity answers. We present a simple methodology for enriching existing datasets with multi-granularity answers, and create GRANOLA-EQ, a multi-granularity version of the EntityQuestions dataset. We evaluate a range of decoding methods on GRANOLA-EQ, including a new algorithm, called Decoding with Response Aggregation (DRAG), that is geared towards aligning the response granularity with the model's uncertainty. Our experiments show that large language models with standard decoding tend to generate specific answers, which are often incorrect. In contrast, when evaluated on multi-granularity answers, DRAG yields a nearly 20 point increase in accuracy on average, which further increases for rare entities. Overall, this reveals that standard evaluation and decoding schemes may significantly underestimate the knowledge encapsulated in LMs.
Paper Structure (36 sections, 12 figures, 9 tables)

This paper contains 36 sections, 12 figures, 9 tables.

Table of Contents

  1. Introduction
  2. GRANOLA Question Answering
  3. Problem Setting
  4. Evaluation
  5. Enriching QA Samples with Multi-Granularity Answers
  6. Automatic Answer Generation
  7. GRANOLA EntityQuestions
  8. Data Quality
  9. Decoding with Response Aggregation
  10. Choice of hyperparameters
  11. DRAG vs existing decoding strategies
  12. Experiments
  13. Experimental Setting
  14. Baselines
  15. Evaluation
  16. ...and 21 more sections

Figures (12)

  • Figure 1: Top: GRANOLA QA evaluation with multi-granularity answers. Middle: Decoding with Response Aggregation (DRAG) outputs a (potentially coarser) response by aggregating several responses of the model. Bottom: Accuracy gain from evaluating using multi-granularity answers for several decoding strategies. DRAG reveals a significant knowledge evaluation gap.
  • Figure 2: Our procedure for adding multi-granularity answers to given QA pairs.
  • Figure 3: An illustration of multi-granularity answers as entity abstractions. Given an answer entity, we use an external KG to generate coarser answers from its properties (turquoise) in addition to the original answer (purple). Notably, not all KG properties are equally good candidates for multi-granular answers (red).
  • Figure 4: Decoding with Response Aggregation (DRAG). We implement ResponseAgg by instructing an LLM to output what $r_1, \dots, r_N$ have in common, or IDK if they do not share meaningful properties.
  • Figure 5: Standard accuracy vs. GRANOLA accuracy for the different models we evaluate.
  • ...and 7 more figures

Theorems & Definitions (1)

  • Definition 1: GRANOLA Evaluation