EACO-RAG: Towards Distributed Tiered LLM Deployment using Edge-Assisted and Collaborative RAG with Adaptive Knowledge Update

TL;DR

This paper tackles the inefficiency and rigidity of centralized RAG systems by introducing EACO-RAG, an edge-assisted and collaborative RAG framework that distributes knowledge updates across edge nodes and uses a hierarchical gating mechanism to dynamically choose between local, edge, and cloud retrieval/generation. The approach couples GraphRAG-based structured retrieval with a contextual multi-armed bandit and Safe Online Bayesian Optimization to balance accuracy, delay, and cost under real-time QoS constraints. Through a dual-layer prototype and extensive experiments on Wiki QA and Harry Potter QA datasets, EACO-RAG demonstrates near cloud-GraphRAG accuracy while achieving large cost reductions (up to 84.6% under relaxed delays and 65.3–81.2% in other settings). The work presents a scalable step toward distributed, edge-cloud intelligent systems, with implications for industrial deployments requiring low latency, data privacy, and cost efficiency.

Abstract

Large language models (LLMs) have demonstrated impressive capabilities in language tasks, but they require high computing power and rely on static knowledge. To overcome these limitations, Retrieval-Augmented Generation (RAG) incorporates up-to-date external information into LLMs without extensive fine-tuning. Meanwhile, small language models (SLMs) deployed on edge devices offer efficiency and low latency but often struggle with complex reasoning tasks. Unfortunately, current RAG approaches are predominantly based on centralized databases and have not been adapted to address the distinct constraints associated with deploying SLMs in edge environments. To bridge this gap, we propose Edge-Assisted and Collaborative RAG (EACO-RAG), a lightweight framework that leverages distributed edge nodes for adaptive knowledge updates and retrieval. EACO-RAG also employs a hierarchical collaborative gating mechanism to dynamically select among local, edge-assisted, and cloud-based strategies, with a carefully designed algorithm based on Safe Online Bayesian Optimization to maximize the potential performance enhancements. Experimental results demonstrate that EACO-RAG matches the accuracy of cloud-based knowledge graph RAG systems while reducing total costs by up to 84.6% under relaxed delay constraints and by 65.3% under stricter delay requirements. This work represents our initial effort toward achieving a distributed and scalable tiered LLM deployments, with EACO-RAG serving as a promising first step in unlocking the full potential of hybrid edge-cloud intelligence.

Figures (4)

• Figure 1: EACO-RAG's adaptive retrieval. Each edge maintains a dynamic local dataset of popular topics. The collaborative gating mechanism selects retrieval sources from local, edge, or cloud datasets to adapt to evolving user interests and knowledge distributions. Black, blue, and red arrows denote local, edge-assisted, and cloud communications, respectively.
• Figure 2: Performance trade-offs in LLM-only applications using Qwen2.5 qwen2, evaluated on the TriviaQA dataset joshi2017triviaqa. Left: Model size vs. inference cost, showing the relationship between LLM parameters and TFLOPs. Right: Model size vs. accuracy and delay, illustrating the impact of LLM parameter on accuracy and generation latency.
• Figure 3: Workflow of the EACO-RAG system design.
• Figure 4: Accuracy comparison under different hyperparameter settings in the ablation study. Left: shows how varying the local adaptive update trigger interval influences accuracy. Right: illustrates the effect of different chunk sizes in the edge dataset.