L-RAG: Balancing Context and Retrieval with Entropy-Based Lazy Loading

TL;DR

RAG systems incur high latency due to always-on retrieval, motivating an uncertainty-driven lazy approach. L-RAG introduces a two-pass, entropy-gated framework with a summary-first context and on-demand chunk retrieval, enabling a tunable balance between accuracy and efficiency via threshold $\tau$. Empirical results on SQuAD 2.0 show that L-RAG at $\tau=0.5$ matches Standard RAG accuracy with 8% fewer retrievals, while $\tau=1.0$ yields 76.0% accuracy with 26% fewer retrievals and notable latency savings; entropy distributions confirm that higher entropy correlates with incorrect predictions, validating the gating signal. The method is training-free, model-agnostic, and practical for production deployment, offering a configurable knob to adapt RAG pipelines to varying throughput and latency requirements.

Abstract

Retrieval-Augmented Generation (RAG) has emerged as the predominant paradigm for grounding Large Language Model outputs in factual knowledge, effectively mitigating hallucinations. However, conventional RAG systems operate under a "retrieve-always" assumption, querying vector databases for every input regardless of query complexity. This static approach incurs substantial computational overhead and inference latency, particularly problematic for high-throughput production deployments. We introduce L-RAG (Lazy Retrieval-Augmented Generation), an adaptive framework that implements hierarchical context management through entropy-based gating. L-RAG employs a two-tier architecture: queries are first processed with a compact document summary, and expensive chunk retrieval is triggered only when the model's predictive entropy exceeds a calibrated threshold, signaling genuine uncertainty. Through experiments on SQuAD 2.0 (N=500) using the Phi-2 model, we demonstrate that L-RAG provides a tunable accuracy-efficiency trade-off: at a conservative threshold (tau=0.5), L-RAG achieves 78.2% accuracy, matching Standard RAG (77.8%), with 8% retrieval reduction; at a balanced threshold (tau=1.0), retrieval reduction increases to 26% with modest accuracy trade-off (76.0%). Latency analysis shows that L-RAG saves 80-210ms per query when retrieval latency exceeds 500ms. Analysis of entropy distributions reveals statistically significant separation (p < 0.001) between correct predictions (H=1.72) and errors (H=2.20), validating entropy as a reliable uncertainty signal. L-RAG offers a practical, training-free approach toward more efficient RAG deployment, providing system architects with a configurable knob to balance accuracy and throughput requirements.

Figures (6)

• Figure 1: Efficiency vs. Accuracy Trade-off. L-RAG configurations (green markers) explore the accuracy-efficiency trade-off space. At $\tau=1.0$, L-RAG achieves accuracy comparable to Standard RAG while reducing retrieval operations by 26%. Strong RAG achieves highest accuracy (79.8%) but requires 100% retrieval.
• Figure 2: Performance Comparison. L-RAG ($\tau=0.5$) achieves 78.2% accuracy, comparable to Standard RAG (77.8%) while requiring 8% fewer retrieval operations. Strong RAG achieves highest accuracy (79.8%) with always-retrieve strategy.
• Figure 3: Database Load Reduction. L-RAG reduces the retrieval burden compared to always-retrieve baselines. At the balanced threshold ($\tau=1.0$), retrieval operations decrease by 26%, directly translating to infrastructure cost savings. More aggressive thresholds achieve up to 46% reduction.
• Figure 4: Impact of Entropy Threshold $\tau$. The graph illustrates the trade-off between accuracy (blue) and retrieval rate (red). The threshold $\tau$ allows practitioners to select their preferred operating point based on accuracy-efficiency requirements.
• Figure 5: Impact of Retrieval Latency on System Performance. L-RAG's latency savings scale with external retrieval latency. At typical cloud latencies (500ms), L-RAG ($\tau=1.0$) saves 80ms per query. The break-even point for $\tau=1.0$ is 192ms---above this threshold, L-RAG provides net latency reduction.