Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Robust Extractive Question Answering Models: Rethinking the Training Methodology

Son Quoc Tran, Matt Kretchmar

TL;DR

This paper proposes a novel training method to improve the robustness of Extractive Question Answering models, which includes a novel loss function for the EQA problem and challenges an implicit assumption present in numerous EQA datasets.

Abstract

This paper proposes a novel training method to improve the robustness of Extractive Question Answering (EQA) models. Previous research has shown that existing models, when trained on EQA datasets that include unanswerable questions, demonstrate a significant lack of robustness against distribution shifts and adversarial attacks. Despite this, the inclusion of unanswerable questions in EQA training datasets is essential for ensuring real-world reliability. Our proposed training method includes a novel loss function for the EQA problem and challenges an implicit assumption present in numerous EQA datasets. Models trained with our method maintain in-domain performance while achieving a notable improvement on out-of-domain datasets. This results in an overall F1 score improvement of 5.7 across all testing sets. Furthermore, our models exhibit significantly enhanced robustness against two types of adversarial attacks, with a performance decrease of only about a third compared to the default models.

Towards Robust Extractive Question Answering Models: Rethinking the Training Methodology

TL;DR

This paper proposes a novel training method to improve the robustness of Extractive Question Answering models, which includes a novel loss function for the EQA problem and challenges an implicit assumption present in numerous EQA datasets.

Abstract

This paper proposes a novel training method to improve the robustness of Extractive Question Answering (EQA) models. Previous research has shown that existing models, when trained on EQA datasets that include unanswerable questions, demonstrate a significant lack of robustness against distribution shifts and adversarial attacks. Despite this, the inclusion of unanswerable questions in EQA training datasets is essential for ensuring real-world reliability. Our proposed training method includes a novel loss function for the EQA problem and challenges an implicit assumption present in numerous EQA datasets. Models trained with our method maintain in-domain performance while achieving a notable improvement on out-of-domain datasets. This results in an overall F1 score improvement of 5.7 across all testing sets. Furthermore, our models exhibit significantly enhanced robustness against two types of adversarial attacks, with a performance decrease of only about a third compared to the default models.
Paper Structure (29 sections, 13 equations, 1 figure, 9 tables)

This paper contains 29 sections, 13 equations, 1 figure, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Models and Tasks
  4. Models
  5. Extractive Question Answering
  6. Datasets
  7. Adversarial Attacks
  8. Robustness Evaluation
  9. Algorithms for Attack Construction
  10. AddOneSent Attacks
  11. Negation Attacks
  12. Extractive Question Answering Loss Functions
  13. Default Loss Function
  14. Our Loss Function
  15. Inference Pipeline
  16. ...and 14 more sections

Figures (1)

  • Figure 1: The training dynamics of RoBERTa models trained using the Devlin method versus our proposed method on SQuAD 2.0. We analyze the performance gap on unanswerable questions between SQuAD 2.0 and SQuAD AGent across three training epochs. The error bars represent the standard deviations of five runs.