An Analysis of Hyper-Parameter Optimization Methods for Retrieval Augmented Generation

TL;DR

This work addresses the costly problem of tuning Retrieval-Augmented Generation configurations by benchmarking five HPO methods across five diverse datasets within a 162-configuration search space. It demonstrates that effective improvements can be achieved with around 10 evaluated configurations, and that greedy, model-first optimization outperforms sequential, pipeline-order strategies. The study also reveals that the optimization objective strongly shapes the best RAG configuration and that development-set sampling can dramatically reduce HPO costs, making practical deployment feasible. Open resources, including full grid-search results and a new enterprise WatsonxQA dataset, are released to support reproducibility and future explorations into larger or multi-modal RAG pipelines.

Abstract

Optimizing Retrieval-Augmented Generation (RAG) configurations for specific tasks is a complex and resource-intensive challenge. Motivated by this challenge, frameworks for RAG hyper-parameter optimization (HPO) have recently emerged, yet their effectiveness has not been rigorously benchmarked. To fill this gap, we present a comprehensive study involving five HPO algorithms over five datasets from diverse domains, including a newly curated real-world product documentation dataset. Our study explores the largest RAG HPO search space to date that includes full grid-search evaluations, and uses three evaluation metrics as optimization targets. Analysis of the results shows that RAG HPO can be done efficiently, either greedily or with random search, and that it significantly boosts RAG performance for all datasets. For greedy HPO approaches, we show that optimizing model selection first is preferable to the common practice of following the RAG pipeline order during optimization.

Figures (13)

• Figure 1: We study hyper parameter optimization over a RAG pipeline with $5$ parameters. The explored search space includes $162$ RAG configurations formed from combinations of the depicted hyper parameters.
• Figure 2: The distribution of configurations across bins for the normalized LLMaaJ-AC metric on the development sets. Most datasets have a few top-performing configurations.
• Figure 3: Per-iteration performance of all HPO algorithms on the test sets of five datasets, optimizing answer correctness computed with an LLMaaJ metric (a) and a lexical metric (b). The dashed black lines show the best achievable performance. The red lines are the performance of the best configuration chosen with development set evaluation, on the test set.
• Figure 4: The effect of chosen optimization metric on the generative model within the best RAG configuration. Shown is the maximal answer correctness score per dataset and model (the highest of the $54$ configurations in which the model appears).
• Figure 5: Per-iteration performance on the test sets of the two largest datasets, for HPO algorithms optimized using the full development data (solid lines) or its sample (dotted). The dashed black lines show the best achievable test performance. The solid (dashed) red lines are the performance of the best configuration chosen by (sampled) development set evaluation.