Optimizing Retrieval-Augmented Generation: Analysis of Hyperparameter Impact on Performance and Efficiency

TL;DR

This paper investigates how hyperparameters shape both the performance and efficiency of retrieval-augmented generation (RAG) systems. It introduces a unified, multi-metric evaluation framework and conducts a systematic grid search over vector stores (Chroma vs Faiss), chunking (naive vs semantic), re-ranking, and temperature settings, using six quality/efficiency metrics. Key findings show a clear speed–accuracy trade-off: Faiss offers higher retrieval quality at the cost of latency, while Chroma enables faster responses; naive chunking generally outperforms semantic chunking in both quality and speed, and re-ranking provides meaningful gains with substantial runtime costs. The study demonstrates that, with corrective retrieval (CRAG), the advantages of top configurations persist under iterative evidence gathering, achieving near-perfect context precision in some settings. These insights offer practical guidance for deploying RAG systems in domains requiring up-to-date, trustworthy outputs and help tune the balance between latency and accuracy across applications.

Abstract

Large language models achieve high task performance yet often hallucinate or rely on outdated knowledge. Retrieval-augmented generation (RAG) addresses these gaps by coupling generation with external search. We analyse how hyperparameters influence speed and quality in RAG systems, covering Chroma and Faiss vector stores, chunking policies, cross-encoder re-ranking, and temperature, and we evaluate six metrics: faithfulness, answer correctness, answer relevancy, context precision, context recall, and answer similarity. Chroma processes queries 13% faster, whereas Faiss yields higher retrieval precision, revealing a clear speed-accuracy trade-off. Naive fixed-length chunking with small windows and minimal overlap outperforms semantic segmentation while remaining the quickest option. Re-ranking provides modest gains in retrieval quality yet increases runtime by roughly a factor of 5, so its usefulness depends on latency constraints. These results help practitioners balance computational cost and accuracy when tuning RAG systems for transparent, up-to-date responses. Finally, we re-evaluate the top configurations with a corrective RAG workflow and show that their advantages persist when the model can iteratively request additional evidence. We obtain a near-perfect context precision (99%), which demonstrates that RAG systems can achieve extremely high retrieval accuracy with the right combination of hyperparameters, with significant implications for applications where retrieval quality directly impacts downstream task performance, such as clinical decision support in healthcare.

Figures (5)

• Figure 1: Systematic evaluation framework for hyperparameter optimization in RAG systems
• Figure 2: Score Distributions by Vector Store Type
• Figure 3: Execution Time Distribution by Vectorstore Type
• Figure 4: Score Distributions With and Without Reranking
• Figure 5: Parameter Impact on Execution Time