Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VoiceAgentRAG: Solving the RAG Latency Bottleneck in Real-Time Voice Agents Using Dual-Agent Architectures

Jielin Qiu, Jianguo Zhang, Zixiang Chen, Liangwei Yang, Ming Zhu, Juntao Tan, Haolin Chen, Wenting Zhao, Rithesh Murthy, Roshan Ram, Akshara Prabhakar, Shelby Heinecke, Caiming Xiong, Silvio Savarese, Huan Wang

TL;DR

VoiceAgentRAG is presented, an open-source dual-agent memory router that decouples retrieval from response generation and pre-fetches relevant document chunks into a FAISS-backed semantic cache.

Abstract

We present VoiceAgentRAG, an open-source dual-agent memory router that decouples retrieval from response generation. A background Slow Thinker agent continuously monitors the conversation stream, predicts likely follow-up topics using an LLM, and pre-fetches relevant document chunks into a FAISS-backed semantic cache. A foreground Fast Talker agent reads only from this sub-millisecond cache, bypassing the vector database entirely on cache hits.

VoiceAgentRAG: Solving the RAG Latency Bottleneck in Real-Time Voice Agents Using Dual-Agent Architectures

TL;DR

VoiceAgentRAG is presented, an open-source dual-agent memory router that decouples retrieval from response generation and pre-fetches relevant document chunks into a FAISS-backed semantic cache.

Abstract

We present VoiceAgentRAG, an open-source dual-agent memory router that decouples retrieval from response generation. A background Slow Thinker agent continuously monitors the conversation stream, predicts likely follow-up topics using an LLM, and pre-fetches relevant document chunks into a FAISS-backed semantic cache. A foreground Fast Talker agent reads only from this sub-millisecond cache, bypassing the vector database entirely on cache hits.
Paper Structure (31 sections, 1 figure, 6 tables)

This paper contains 31 sections, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Architecture Overview
  4. Semantic Cache
  5. Slow Thinker: Predictive Prefetching
  6. Fast Talker: Cache-First Response
  7. Conversation Stream
  8. Experimental Setup
  9. Knowledge Base
  10. Vector Store
  11. LLM
  12. Conversation Scenarios
  13. Evaluation Protocol
  14. Results
  15. Overall Performance
  16. ...and 16 more sections

Figures (1)

  • Figure 1: Architecture of VoiceAgentRAG. The Slow Thinker (left) continuously monitors the conversation stream, predicts follow-up topics, retrieves from the vector store, and populates the semantic cache. The Fast Talker (right) checks the cache first ($<$1ms), bypassing the vector store on hits. On misses, it falls back to direct retrieval (dashed red) and caches the results for future queries.