Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems

Yoshitomo Matsubara, Luca Soldaini, Eric Lind, Alessandro Moschitti

TL;DR

This work tackles Answer Sentence Selection (AS2) efficiency by distilling an ensemble of heterogeneous transformers into a single Cerberus model with a shared encoder and multiple ranking heads. Each head is trained to mimic a different teacher (RoBERTa Large, ELECTRA Large, ALBERT XXLarge) to preserve ensemble diversity, and inference averages the head outputs. On ASNQ, IAS2, and WikiQA, Cerberus matches or surpasses large, slow ensembles while using about 64% fewer parameters and 60% less latency. The work demonstrates that multiple heads and heterogeneous teachers are key to restoring ensemble performance in a compact model, enabling practical deployment in real-world QA systems.

Abstract

Large transformer models can highly improve Answer Sentence Selection (AS2) tasks, but their high computational costs prevent their use in many real-world applications. In this paper, we explore the following research question: How can we make the AS2 models more accurate without significantly increasing their model complexity? To address the question, we propose a Multiple Heads Student architecture (named CERBERUS), an efficient neural network designed to distill an ensemble of large transformers into a single smaller model. CERBERUS consists of two components: a stack of transformer layers that is used to encode inputs, and a set of ranking heads; unlike traditional distillation technique, each of them is trained by distilling a different large transformer architecture in a way that preserves the diversity of the ensemble members. The resulting model captures the knowledge of heterogeneous transformer models by using just a few extra parameters. We show the effectiveness of CERBERUS on three English datasets for AS2; our proposed approach outperforms all single-model distillations we consider, rivaling the state-of-the-art large AS2 models that have 2.7x more parameters and run 2.5x slower. Code for our model is available at https://github.com/amazon-research/wqa-cerberus

Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems

TL;DR

This work tackles Answer Sentence Selection (AS2) efficiency by distilling an ensemble of heterogeneous transformers into a single Cerberus model with a shared encoder and multiple ranking heads. Each head is trained to mimic a different teacher (RoBERTa Large, ELECTRA Large, ALBERT XXLarge) to preserve ensemble diversity, and inference averages the head outputs. On ASNQ, IAS2, and WikiQA, Cerberus matches or surpasses large, slow ensembles while using about 64% fewer parameters and 60% less latency. The work demonstrates that multiple heads and heterogeneous teachers are key to restoring ensemble performance in a compact model, enabling practical deployment in real-world QA systems.

Abstract

Large transformer models can highly improve Answer Sentence Selection (AS2) tasks, but their high computational costs prevent their use in many real-world applications. In this paper, we explore the following research question: How can we make the AS2 models more accurate without significantly increasing their model complexity? To address the question, we propose a Multiple Heads Student architecture (named CERBERUS), an efficient neural network designed to distill an ensemble of large transformers into a single smaller model. CERBERUS consists of two components: a stack of transformer layers that is used to encode inputs, and a set of ranking heads; unlike traditional distillation technique, each of them is trained by distilling a different large transformer architecture in a way that preserves the diversity of the ensemble members. The resulting model captures the knowledge of heterogeneous transformer models by using just a few extra parameters. We show the effectiveness of CERBERUS on three English datasets for AS2; our proposed approach outperforms all single-model distillations we consider, rivaling the state-of-the-art large AS2 models that have 2.7x more parameters and run 2.5x slower. Code for our model is available at https://github.com/amazon-research/wqa-cerberus
Paper Structure (27 sections, 4 equations, 3 figures, 5 tables)

This paper contains 27 sections, 4 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Answer Sentence Selection (AS2)
  4. Single Model Distillation
  5. Ensemble Distillation
  6. Multi-head Transformers
  7. Methodology
  8. Training Transformer Models for Answer Sentence Selection (AS2)
  9. Distilled Models and Ensembles
  10. Cerberus: Multiple-Heads Student
  11. Experimental Setup
  12. Datasets
  13. Evaluation Metrics
  14. Results
  15. Answer Sentence Selection Performance
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Cerberus model for answer sentence selection. The model consists of a shared encoder body and multiple ranking heads. Cerberus independently scores up to hundreds candidate answers $a_i$ for question $q$; The one with highest likelihood is selected as answer.
  • Figure 2: Detailed overview of Cerberus model that consists of a shared encoder body of $b$ transformer layers, followed by $k$ ranking heads of $h$ layers each; we use notation $\mathit{B}_b\ k\textit{H}_{h}$ to identify a Cerberus configuration. All heads are jointly trained, but each head learns from a unique teacher model; at inference time, predictions from heads are combined by a pooler layer.
  • Figure 3: Agreement between heads and their teacher model in Cerberus. It is obtained by diving the number of correct candidates each head and teacher agree on by the total number of correct answer for each head.