Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Attention-Guided Answer Distillation for Machine Reading Comprehension

Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, Ming Zhou

TL;DR

Ensemble-based MRC systems achieve high accuracy but are slow and vulnerable to adversarial attacks. The authors propose distilling an ensemble into a single, faster student model and introduce vanilla KD plus two novel distillation variants with joint training to transfer knowledge. The student nearly matches the ensemble on SQuAD with only a 0.4% F1 drop while being roughly 12x faster, and it outperforms the teacher on adversarial SQuAD and NarrativeQA Bleu-1; ablation studies show each distillation component contributes to robustness and efficiency. This work demonstrates that targeted distillation of ensemble knowledge, including attention and answer boundaries, can yield practical, robust MRC systems.

Abstract

Despite that current reading comprehension systems have achieved significant advancements, their promising performances are often obtained at the cost of making an ensemble of numerous models. Besides, existing approaches are also vulnerable to adversarial attacks. This paper tackles these problems by leveraging knowledge distillation, which aims to transfer knowledge from an ensemble model to a single model. We first demonstrate that vanilla knowledge distillation applied to answer span prediction is effective for reading comprehension systems. We then propose two novel approaches that not only penalize the prediction on confusing answers but also guide the training with alignment information distilled from the ensemble. Experiments show that our best student model has only a slight drop of 0.4% F1 on the SQuAD test set compared to the ensemble teacher, while running 12x faster during inference. It even outperforms the teacher on adversarial SQuAD datasets and NarrativeQA benchmark.

Attention-Guided Answer Distillation for Machine Reading Comprehension

TL;DR

Ensemble-based MRC systems achieve high accuracy but are slow and vulnerable to adversarial attacks. The authors propose distilling an ensemble into a single, faster student model and introduce vanilla KD plus two novel distillation variants with joint training to transfer knowledge. The student nearly matches the ensemble on SQuAD with only a 0.4% F1 drop while being roughly 12x faster, and it outperforms the teacher on adversarial SQuAD and NarrativeQA Bleu-1; ablation studies show each distillation component contributes to robustness and efficiency. This work demonstrates that targeted distillation of ensemble knowledge, including attention and answer boundaries, can yield practical, robust MRC systems.

Abstract

Despite that current reading comprehension systems have achieved significant advancements, their promising performances are often obtained at the cost of making an ensemble of numerous models. Besides, existing approaches are also vulnerable to adversarial attacks. This paper tackles these problems by leveraging knowledge distillation, which aims to transfer knowledge from an ensemble model to a single model. We first demonstrate that vanilla knowledge distillation applied to answer span prediction is effective for reading comprehension systems. We then propose two novel approaches that not only penalize the prediction on confusing answers but also guide the training with alignment information distilled from the ensemble. Experiments show that our best student model has only a slight drop of 0.4% F1 on the SQuAD test set compared to the ensemble teacher, while running 12x faster during inference. It even outperforms the teacher on adversarial SQuAD datasets and NarrativeQA benchmark.

Paper Structure

This paper contains 18 sections, 10 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Machine Reading Comprehension
  4. Knowledge Distillation
  5. Attention-guided Answer Distillation
  6. Vanilla Knowledge Distillation
  7. Answer Distillation
  8. Attention Distillation
  9. Joint Training
  10. Experiments
  11. Datasets and Metrics
  12. Implementation Details
  13. Main Results
  14. Speedup over Ensemble Model
  15. Ablation Study
  16. ...and 3 more sections

Figures (4)

  • Figure 1: An illustration of confusing answer and biased distillation in machine reading comprehension.
  • Figure 2: Overview of our approaches. In vanilla knowledge distillation (green), the cross entropy is minimized between the student/teacher distributions of answer positions. In answer distillation (green), the student is trained to penalize the most confusing answer distilled from the teacher. In attention distillation (yellow), mean-squared error is minimized between the student/teacher attention distributions. Darker color denotes higher probability.
  • Figure 3: A case study between the base model and the distilled model.
  • Figure 4: