MIX : a Multi-task Learning Approach to Solve Open-Domain Question Answering

TL;DR

Open-domain QA faces scalability and cross-document answer comparison challenges. MIX proposes a simple, efficient solution by stacking a BM25 retriever, a RoBERTa-based Scorer, and an Extractor, with a multitask variant that shares the backbone to parallelize re-ranking and reading. It achieves state-of-the-art results on squad-open with RoBERTa-base and demonstrates that shorter paragraph granularity (100 tokens) improves performance. The approach offers practical benefits for search engines and points to future improvements using larger models such as RoBERTa-large.

Abstract

This paper introduces MIX, a multi-task deep learning approach to solve open-ended question-answering. First, we design our system as a multi-stage pipeline of 3 building blocks: a BM25-based Retriever to reduce the search space, a RoBERTa-based Scorer, and an Extractor to rank retrieved paragraphs and extract relevant text spans, respectively. Eventually, we further improve the computational efficiency of our system to deal with the scalability challenge: thanks to multi-task learning, we parallelize the close tasks solved by the Scorer and the Extractor. Our system is on par with state-of-the-art performances on the squad-open benchmark while being simpler conceptually.

Figures (7)

• Figure 1: The proposed pipeline for the open-domain QA task resolution. The search space is first reduced in this preliminary pipeline thanks to the Retriever. Secondly, the ranking of paragraphs retrieved is refined thanks to the Scorer. Eventually, the Extractor allows highlighting the most relevant text spans in each paragraph.
• Figure 2: The open-domain QA setting.
• Figure 3: The proposed scoring model. An example consists of a question and several candidate paragraphs. For each paragraph, the classification token embedding (CLS) is built to represent the given paragraph according to the question. Eventually relevance scores are computed from these vectors and normalized by the $softmax$ function.
• Figure 4: The proposed QA model. The question and the paragraph are processed simultaneously by the language model to build question-aware paragraph representations, and then 2 dense layers allow us to find the beginning and the end of the answer.
• Figure 5: The proposed multi-task pipeline. As before, the search space is first reduced thanks to the Retriever, then the most relevant spans of text are extracted while the paragraphs are re-ranked. We can see that the question is processed only twice, once by the Retriever and once by the multi-task model instead of three times in the previous pipeline.