Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

StreamingRAG: Real-time Contextual Retrieval and Generation Framework

Murugan Sankaradas, Ravi K. Rajendran, Srimat T. Chakradhar

TL;DR

StreamingRAG tackles real-time extraction from multi-modal streaming data by constructing a temporal knowledge graph of scene-object-entity relations using lightweight models. The framework combines a dynamic priority-based extraction, a frame Scheduler, a Constraint Resolver, and complementary pipelines (extraction, knowledge, stream processing, and Lambda) to enable real-time, context-aware retrieval. It reports 5-6x throughput improvements and 2-3x reductions in resource use, while preserving temporal contextual accuracy in ITS anomaly detection. The approach enables timely decision-making across domains like ITS and remote sensing, providing interactive querying capabilities that adapt to evolving events.

Abstract

Extracting real-time insights from multi-modal data streams from various domains such as healthcare, intelligent transportation, and satellite remote sensing remains a challenge. High computational demands and limited knowledge scope restrict the applicability of Multi-Modal Large Language Models (MM-LLMs) on these data streams. Traditional Retrieval-Augmented Generation (RAG) systems address knowledge limitations of these models, but suffer from slow preprocessing, making them unsuitable for real-time analysis. We propose StreamingRAG, a novel RAG framework designed for streaming data. StreamingRAG constructs evolving knowledge graphs capturing scene-object-entity relationships in real-time. The knowledge graph achieves temporal-aware scene representations using MM-LLMs and enables timely responses for specific events or user queries. StreamingRAG addresses limitations in existing methods, achieving significant improvements in real-time analysis (5-6x faster throughput), contextual accuracy (through a temporal knowledge graph), and reduced resource consumption (using lightweight models by 2-3x).

StreamingRAG: Real-time Contextual Retrieval and Generation Framework

TL;DR

StreamingRAG tackles real-time extraction from multi-modal streaming data by constructing a temporal knowledge graph of scene-object-entity relations using lightweight models. The framework combines a dynamic priority-based extraction, a frame Scheduler, a Constraint Resolver, and complementary pipelines (extraction, knowledge, stream processing, and Lambda) to enable real-time, context-aware retrieval. It reports 5-6x throughput improvements and 2-3x reductions in resource use, while preserving temporal contextual accuracy in ITS anomaly detection. The approach enables timely decision-making across domains like ITS and remote sensing, providing interactive querying capabilities that adapt to evolving events.

Abstract

Extracting real-time insights from multi-modal data streams from various domains such as healthcare, intelligent transportation, and satellite remote sensing remains a challenge. High computational demands and limited knowledge scope restrict the applicability of Multi-Modal Large Language Models (MM-LLMs) on these data streams. Traditional Retrieval-Augmented Generation (RAG) systems address knowledge limitations of these models, but suffer from slow preprocessing, making them unsuitable for real-time analysis. We propose StreamingRAG, a novel RAG framework designed for streaming data. StreamingRAG constructs evolving knowledge graphs capturing scene-object-entity relationships in real-time. The knowledge graph achieves temporal-aware scene representations using MM-LLMs and enables timely responses for specific events or user queries. StreamingRAG addresses limitations in existing methods, achieving significant improvements in real-time analysis (5-6x faster throughput), contextual accuracy (through a temporal knowledge graph), and reduced resource consumption (using lightweight models by 2-3x).
Paper Structure (27 sections, 5 figures, 3 tables)

This paper contains 27 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Limitations of baseline architecture
  4. Our approach
  5. Streaming RAG
  6. Overview
  7. Extraction Pipeline
  8. Frame Scheduler:
  9. Constraint Resolver:
  10. Inference Engine:
  11. Knowledge Pipeline
  12. Knowledge Base:
  13. Knowledge Graph:
  14. Knowledge Builder:
  15. Temporal Context Identifier:
  16. ...and 12 more sections

Figures (5)

  • Figure 1: Baseline architecture
  • Figure 2: Latency vs output token size of ShareGPT4V sharegpt4v-chen-2023. Red shaded area indicates descriptive answers, using larger output token size which incrases latency. Green shared area generates less descriptive output meeting real-time constraints using smaller output token size. Need to ask descriptive questions to model so that spatio-temporal information across frames is extracted.
  • Figure 3: StreamingRAG approach
  • Figure 4: StreamingRAG: System Architecture
  • Figure 5: Throughput