EfficientQA : a RoBERTa Based Phrase-Indexed Question-Answering System

TL;DR

EfficientQA tackles the open-domain QA scalability challenge by separating encoding and retrieval: it uses agnostic extraction to offline-index a limited set of candidate answer spans and trains a Siamese RoBERTa-based model to map questions and candidates into a shared dense vector space. The system then retrieves the nearest candidate via simple vector search, reducing online computation and enabling indexing of multiple documents. Empirically, EfficientQA achieves state-of-the-art results on the PIQA benchmark, beating prior approaches like DENSPI while using a lighter model (RoBERTa-base) and a smaller candidate set. Across SQuAD v1.1 and FQuAD, the method demonstrates competitive performance and highlights the potential of dense representations for scalable open-domain QA, suggesting future work toward full-corpus indexing and faster retrieval.

Abstract

State-of-the-art extractive question-answering models achieve superhuman performances on the SQuAD benchmark. Yet, they are unreasonably heavy and need expensive GPU computing to answer questions in a reasonable time. Thus, they cannot be used in the open-domain question-answering paradigm for real-world queries on hundreds of thousands of documents. In this paper, we explore the possibility of transferring the natural language understanding of language models into dense vectors representing questions and answer candidates to make question-answering compatible with a simple nearest neighbor search task. This new model, which we call EfficientQA, takes advantage of the pair of sequences kind of input of BERT-based models to build meaningful, dense representations of candidate answers. These latter are extracted from the context in a question-agnostic fashion. Our model achieves state-of-the-art results in Phrase-Indexed Question Answering (PIQA), beating the previous state-of-art by 1.3 points in exact-match and 1.4 points in f1-score. These results show that dense vectors can embed rich semantic representations of sequences, although these were built from language models not originally trained for the use case. Thus, to build more resource-efficient NLP systems in the future, training language models better adapted to build dense representations of phrases is one of the possibilities.

Figures (8)

• Figure 1: The PIQA challenge from seo2018phraseindexed
• Figure 2: Agnostic extraction of answer candidates
• Figure 3: Agnostic extraction of answer candidates with beam search. Paragraph tokens are provided to the language models to produce their embeddings, then a first dense layer allows to identify the $s$ most likely start positions of candidates. The embeddings of the paragraph's tokens are concatenated to each start position, and a second dense layer identifies the $e$ most likely end positions associated with each start position. We end up with $s \times e$ possible spans.
• Figure 4: Answer candidate dense vectors. The paragraph's tokens and the candidate's tokens separated by special SEP token are provided to the language model. The final embeddings are provided to an additional dense layer and averaged to produce the candidate dense representation.
• Figure 5: Question dense vectors. The question's tokens are passed to the same language model as the candidates', and the final embeddings are passed to the same dense layer. Eventually, the vectors are averaged to produce the question-dense representation.