On the Effectiveness of Proposed Techniques to Reduce Energy Consumption in RAG Systems: A Controlled Experiment

TL;DR

The paper investigates how to reduce energy consumption in retrieval-augmented generation (RAG) systems through a controlled experiment. It evaluates five practical techniques (threshold tuning, lightweight reranking, embedding-size reduction, vector-indexing, and caching) in a production-like ChatRAG system using the CRAG dataset, across nine configurations and 200+ hours. Key findings show that increasing similarity thresholds (0.78 or 0.88) and reducing embedding sizes can substantially cut energy and latency without harming accuracy, while vector indexing yields the largest energy savings but can hurt accuracy; a combination of T1-0.78 and T3-384 achieves strong energy and latency gains with minimal accuracy impact. The study provides actionable guidance for building sustainable RAG applications and motivates further research into energy-aware design trade-offs.

Abstract

The rising energy demands of machine learning (ML), e.g., implemented in popular variants like retrieval-augmented generation (RAG) systems, have raised significant concerns about their environmental sustainability. While previous research has proposed green tactics for ML-enabled systems, their empirical evaluation within RAG systems remains largely unexplored. This study presents a controlled experiment investigating five practical techniques aimed at reducing energy consumption in RAG systems. Using a production-like RAG system developed at our collaboration partner, the Software Improvement Group, we evaluated the impact of these techniques on energy consumption, latency, and accuracy. Through a total of 9 configurations spanning over 200 hours of trials using the CRAG dataset, we reveal that techniques such as increasing similarity retrieval thresholds, reducing embedding sizes, applying vector indexing, and using a BM25S reranker can significantly reduce energy usage, up to 60% in some cases. However, several techniques also led to unacceptable accuracy decreases, e.g., by up to 30% for the indexing strategies. Notably, finding an optimal retrieval threshold and reducing embedding size substantially reduced energy consumption and latency with no loss in accuracy, making these two techniques truly energy-efficient. We present the first comprehensive, empirical study on energy-efficient design techniques for RAG systems, providing guidance for developers and researchers aiming to build sustainable RAG applications.

Figures (8)

• Figure 1: ChatRAG Software Architecture and Workflow
• Figure 2: Experiment Environment and Workflow
• Figure 3: Energy Consumption Experiment Results
• Figure 4: GPU Contribution to Total Energy Consumption.
• Figure 5: Latency Experiment Results