Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowing More About Questions Can Help: Improving Calibration in Question Answering

Shujian Zhang, Chengyue Gong, Eunsol Choi

TL;DR

This work addresses the problem of calibrating QA systems at the per-example level, going beyond raw model confidence to predict the likelihood that a given answer is correct. It introduces a simple, post-hoc calibrator that leverages an input-example embedding derived from the base QA model and enhances it with back-translation paraphrase augmentation, trained via XGBoost. Across reading comprehension and open retrieval QA, and under in-domain, adversarial, and out-of-domain conditions, the proposed method yields consistent calibration gains of approximately 5–10 percentage points and also provides modest improvements when used as an answer reranker. The approach is data-efficient, model-agnostic, and easily transferable to other tasks with rich output spaces, offering practical benefits for selective answering and error handling in real-world QA systems.

Abstract

We study calibration in question answering, estimating whether model correctly predicts answer for each question. Unlike prior work which mainly rely on the model's confidence score, our calibrator incorporates information about the input example (e.g., question and the evidence context). Together with data augmentation via back translation, our simple approach achieves 5-10% gains in calibration accuracy on reading comprehension benchmarks. Furthermore, we present the first calibration study in the open retrieval setting, comparing the calibration accuracy of retrieval-based span prediction models and answer generation models. Here again, our approach shows consistent gains over calibrators relying on the model confidence. Our simple and efficient calibrator can be easily adapted to many tasks and model architectures, showing robust gains in all settings.

Knowing More About Questions Can Help: Improving Calibration in Question Answering

TL;DR

This work addresses the problem of calibrating QA systems at the per-example level, going beyond raw model confidence to predict the likelihood that a given answer is correct. It introduces a simple, post-hoc calibrator that leverages an input-example embedding derived from the base QA model and enhances it with back-translation paraphrase augmentation, trained via XGBoost. Across reading comprehension and open retrieval QA, and under in-domain, adversarial, and out-of-domain conditions, the proposed method yields consistent calibration gains of approximately 5–10 percentage points and also provides modest improvements when used as an answer reranker. The approach is data-efficient, model-agnostic, and easily transferable to other tasks with rich output spaces, offering practical benefits for selective answering and error handling in real-world QA systems.

Abstract

We study calibration in question answering, estimating whether model correctly predicts answer for each question. Unlike prior work which mainly rely on the model's confidence score, our calibrator incorporates information about the input example (e.g., question and the evidence context). Together with data augmentation via back translation, our simple approach achieves 5-10% gains in calibration accuracy on reading comprehension benchmarks. Furthermore, we present the first calibration study in the open retrieval setting, comparing the calibration accuracy of retrieval-based span prediction models and answer generation models. Here again, our approach shows consistent gains over calibrators relying on the model confidence. Our simple and efficient calibrator can be easily adapted to many tasks and model architectures, showing robust gains in all settings.

Paper Structure

This paper contains 43 sections, 6 equations, 1 figure, 12 tables.

Table of Contents

  1. Introduction
  2. Problem Definition
  3. Metric: Calibrator performance
  4. Accuracy:
  5. AUROC:
  6. Metric: End task performance
  7. Selective QA (coverage at fixed accuracy):
  8. Open Retrieval QA (top-N accuracy):
  9. Methods
  10. Calibrator Architecture
  11. Input Example Embedding Feature From Base QA Model
  12. Base QA Model
  13. Data Augmentation Via Paraphrasing
  14. Experimental Settings
  15. Data
  16. ...and 28 more sections

Figures (1)

  • Figure 1: A visualization for the input example embedding from HotPotQA and SQuAD datasets. We denote the data domain by markers with different shapes and denote the correctness with different colors. The X-axis and Y-axis denote the first and second dimensions extracted by linear discriminant analysis, respectively.