Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TWEAC: Transformer with Extendable QA Agent Classifiers

Gregor Geigle, Nils Reimers, Andreas Rücklé, Iryna Gurevych

TL;DR

The paper tackles the challenge of broad-question answering by routing queries to a curated suite of specialized QA agents in a meta-QA framework. It compares similarity-based retrieval and a transformer-based Extendable Agent Classifier (TWEAC) for selecting suitable agents, demonstrating strong performance and sample efficiency even with hundreds of agents. TWEAC benefits from per-agent classification heads that allow scalable extension, and the authors propose half-and-half sampling to enable rapid integration of new agents without full retraining. The study provides two evaluation setups (QA-Tasks and Many-Agents), analyzes scalability and extension strategies, and offers insights into error patterns due to overlapping agent domains, with code and data available for replication.

Abstract

Question answering systems should help users to access knowledge on a broad range of topics and to answer a wide array of different questions. Most systems fall short of this expectation as they are only specialized in one particular setting, e.g., answering factual questions with Wikipedia data. To overcome this limitation, we propose composing multiple QA agents within a meta-QA system. We argue that there exist a wide range of specialized QA agents in literature. Thus, we address the central research question of how to effectively and efficiently identify suitable QA agents for any given question. We study both supervised and unsupervised approaches to address this challenge, showing that TWEAC -- Transformer with Extendable Agent Classifiers -- achieves the best performance overall with 94% accuracy. We provide extensive insights on the scalability of TWEAC, demonstrating that it scales robustly to over 100 QA agents with each providing just 1000 examples of questions they can answer. Our code and data is available: https://github.com/UKPLab/TWEAC-qa-agent-selection

TWEAC: Transformer with Extendable QA Agent Classifiers

TL;DR

The paper tackles the challenge of broad-question answering by routing queries to a curated suite of specialized QA agents in a meta-QA framework. It compares similarity-based retrieval and a transformer-based Extendable Agent Classifier (TWEAC) for selecting suitable agents, demonstrating strong performance and sample efficiency even with hundreds of agents. TWEAC benefits from per-agent classification heads that allow scalable extension, and the authors propose half-and-half sampling to enable rapid integration of new agents without full retraining. The study provides two evaluation setups (QA-Tasks and Many-Agents), analyzes scalability and extension strategies, and offers insights into error patterns due to overlapping agent domains, with code and data available for replication.

Abstract

Question answering systems should help users to access knowledge on a broad range of topics and to answer a wide array of different questions. Most systems fall short of this expectation as they are only specialized in one particular setting, e.g., answering factual questions with Wikipedia data. To overcome this limitation, we propose composing multiple QA agents within a meta-QA system. We argue that there exist a wide range of specialized QA agents in literature. Thus, we address the central research question of how to effectively and efficiently identify suitable QA agents for any given question. We study both supervised and unsupervised approaches to address this challenge, showing that TWEAC -- Transformer with Extendable Agent Classifiers -- achieves the best performance overall with 94% accuracy. We provide extensive insights on the scalability of TWEAC, demonstrating that it scales robustly to over 100 QA agents with each providing just 1000 examples of questions they can answer. Our code and data is available: https://github.com/UKPLab/TWEAC-qa-agent-selection

Paper Structure

This paper contains 28 sections, 2 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. QA Agent Identification
  4. Task Definition
  5. Similarity-based Models
  6. Transformer with Extendable Agent Classifiers (TWEAC)
  7. Data and Evaluation
  8. QA Tasks
  9. Large-Scale QA Agent Identification
  10. Experiments
  11. Experimental Setup
  12. Models and Hyperparameters.
  13. Sample Efficiency.
  14. Scalability.
  15. Results
  16. ...and 13 more sections

Figures (5)

  • Figure 1: The meta-QA system receives a broad range of questions and needs to decide which QA agent is best suited for answering a question.
  • Figure 2: Visualization of TWEAC with $N$ agents
  • Figure 3: Accuracy of our models trained on QA-Tasks with increasing number of training examples per agent.
  • Figure 4: Accuracy of models trained with an increasing number of agents from Many-Agents
  • Figure 5: Accuracy of Iterative Extension with Many-Agents. Solid line starts with 10 agents and dashed line starts with 50 agents. Dots show performance of full training .