Similarity is Not All You Need: Endowing Retrieval Augmented Generation with Multi Layered Thoughts

TL;DR

This work tackles knowledge-update and hallucination challenges in large language models by enhancing retrieval-augmented generation with MetRag, a multi-layered thoughts framework. It integrates a similarity-based retriever with a supervision-driven utility model, guided by LLM feedback, and adds a task-adaptive summarizer to produce compact, task-relevant external knowledge. The final knowledge-augmented generation step leverages multi-layered signals to generate accurate answers, achieving state-of-the-art performance on four knowledge-intensive public datasets in a zero-shot setting. The approach demonstrates that combining similarity with utility supervision and compact Summarization yields robust retrieval quality, reduces information overload, and improves instruction-following and factuality in downstream tasks. These results have practical implications for deploying RAG systems in dynamic domains where knowledge must be updated efficiently and hallucination risks must be mitigated, with potential extensions to ultra-long contexts and more diverse tasks. The method relies on formulas such as $\mathcal{L}_\textit{U}$ for utility-alignment, $f(q,d)$ gating, and $r_\phi$-based alignment, all encapsulated within a DPO-inspired reward framework to steer summaries toward end-task success.

Abstract

In recent years, large language models (LLMs) have made remarkable achievements in various domains. However, the untimeliness and cost of knowledge updates coupled with hallucination issues of LLMs have curtailed their applications in knowledge intensive tasks, where retrieval augmented generation (RAG) can be of help. Nevertheless, existing retrieval augmented models typically use similarity as a bridge between queries and documents and follow a retrieve then read procedure. In this work, we argue that similarity is not always the panacea and totally relying on similarity would sometimes degrade the performance of retrieval augmented generation. To this end, we propose MetRag, a Multi layEred Thoughts enhanced Retrieval Augmented Generation framework. To begin with, beyond existing similarity oriented thought, we embrace a small scale utility model that draws supervision from an LLM for utility oriented thought and further come up with a smarter model by comprehensively combining the similarity and utility oriented thoughts. Furthermore, given the fact that the retrieved document set tends to be huge and using them in isolation makes it difficult to capture the commonalities and characteristics among them, we propose to make an LLM as a task adaptive summarizer to endow retrieval augmented generation with compactness-oriented thought. Finally, with multi layered thoughts from the precedent stages, an LLM is called for knowledge augmented generation. Extensive experiments on knowledge-intensive tasks have demonstrated the superiority of MetRag.

Figures (4)

• Figure 1: A toy example illustrating the difference between similarity and utility, where the score of similarity model is given by BGE. Can we reunite the virtues of both worlds and come up with a better model?
• Figure 2: The proposed MetRag framework, where we endow retrieval-augmented generation with multi-layered thoughts from Stage I and II, and utilize the derived knowledge in Stage III for answer generation.
• Figure 3: Ablation study.
• Figure 4: The influence of passage window size.