Meta-prompting Optimized Retrieval-augmented Generation

TL;DR

This paper tackles the challenge of noisy and dispersed retrieved content in Retrieval-Augmented Generation (RAG) by introducing a refinement step powered by meta-prompting optimization. A transformation-LLM refines the retrieved content, guided by an optimizer-LLM that iteratively generates and scores refinement instructions within a meta-prompt, retaining the top-performing instruction to shape the final prompt. Evaluated on StrategyQA multi-hop questions with Llama-2-70b variants, the refined-content approach achieves about a 8.5-point absolute improvement over plain RAG (34.69% vs 26.12%), corresponding to a >30% relative gain. The results demonstrate that content refinement via meta-prompting is a effective, complementary enhancement to RAG, with potential for integration with existing RAG techniques and directions for scaling to larger models and broader tasks.

Abstract

Retrieval-augmented generation resorts to content retrieved from external sources in order to leverage the performance of large language models in downstream tasks. The excessive volume of retrieved content, the possible dispersion of its parts, or their out of focus range may happen nevertheless to eventually have a detrimental rather than an incremental effect. To mitigate this issue and improve retrieval-augmented generation, we propose a method to refine the retrieved content before it is included in the prompt by resorting to meta-prompting optimization. Put to empirical test with the demanding multi-hop question answering task from the StrategyQA dataset, the evaluation results indicate that this method outperforms a similar retrieval-augmented system but without this method by over 30%.