Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model

TL;DR

The paper interrogates whether retriever-augmented language models can reliably perform reasoning over retrieved statements. It shows that simple similarity-based retrieval often misses essential statements, and that language models still struggle to reason even with ideal retrievers; this degradation worsens when retrievers are imperfect. Larger language models yield gains, but reach only partial improvements, with substantial room for advancement. Multihop retrieve-and-read offers some benefits for GPT-3.5 but does not generalize to all models, underscoring model-dependent dynamics and the need for improved retrievers or reasoning capabilities.

Abstract

Augmenting pretrained language models with retrievers has shown promise in effectively solving common NLP problems, such as language modeling and question answering. In this paper, we evaluate the strengths and weaknesses of popular retriever-augmented language models, namely kNN-LM, REALM, DPR + FiD, Contriever + ATLAS, and Contriever + Flan-T5, in reasoning over retrieved statements across different tasks. Our findings indicate that the simple similarity metric employed by retrievers is insufficient for retrieving all the necessary statements for reasoning. Additionally, the language models do not exhibit strong reasoning even when provided with only the required statements. Furthermore, when combined with imperfect retrievers, the performance of the language models becomes even worse, e.g., Flan-T5's performance drops by 28.6% when retrieving 5 statements using Contriever. While larger language models improve performance, there is still a substantial room for enhancement. Our further analysis indicates that multihop retrieve-and-read is promising for large language models like GPT-3.5, but does not generalize to other language models like Flan-T5-xxl.

Figures (17)

• Figure 1: Example of retriever and language model failures when reasoning is needed. The correct and incorrect retrievals are highlighted in green and red, respectively. This example demonstrates that the retrievers' similarity metric is insufficient for retrieving required statements, and language models cannot perform reasoning over the retrieved statements perfectly.
• Figure 2: The architecture of retrieve-then-read retriever-augmented language models. The language model predicts the answer using the query and the retriever's selected statements.
• Figure 3: Retrievers' accuracy on EB-Hard test set in LM and QA tasks. Results show that retrievers do not select required statements properly, as the best retriever, Contriever, achieves only a 47% accuracy in QA task.
• Figure 4: Retreivers' recall score on EB-Hard test set in LM and QA based on the number of retrieved statements ($\mathbf{k}$). Contriever is shown to be superior among the studied retrievers. Results further indicate that $k$NN-LM does not cover 100% of the gold statements when $k=100$ ($k$NN-LM's recall is $\approx 97\%$).
• Figure 5: Token overlap F1 score of language models on EB-Hard QA test set. The dotted and solid lines refer to experiments given the single oracle statement and gold statements (when reasoning is required), respectively. Results illustrate that language models perform worse when answering the question requires reasoning.