Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LiveVectorLake: A Real-Time Versioned Knowledge Base Architecture for Streaming Vector Updates and Temporal Retrieval

Tarun Prajapati

TL;DR

LiveVectorLake tackles the challenge of maintaining both real-time, semantically rich knowledge updates and complete version history for RAG systems. It achieves this with a five-layer architecture that combines chunk-level content-addressable change detection, selective embedding of modified content, and a dual-tier storage strategy (hot Milvus vector indices and cold Delta Lake history) coupled with a temporal query engine for point-in-time retrieval. The key contributions are automatic chunk-level CDC via SHA-256, independent optimization of hot and cold storage with ACID cross-tier consistency, and dual-mode temporal queries that prevent leakage while preserving performance. Preliminary evaluation on a 100-document corpus shows 10–15% re-processing during updates, sub-100 ms latency for current queries, sub-2 s latency for temporal queries, and substantial hot-tier storage savings, indicating practical viability for production RAG deployments with regulatory and audit requirements.

Abstract

Modern Retrieval-Augmented Generation (RAG) systems struggle with a fundamental architectural tension: vector indices are optimized for query latency but poorly handle continuous knowledge updates, while data lakes excel at versioning but introduce query latency penalties. We introduce LiveVectorLake, a dual-tier temporal knowledge base architecture that enables real-time semantic search on current knowledge while maintaining complete version history for compliance, auditability, and point-in-time retrieval. The system introduces three core architectural contributions: (1) Content-addressable chunk-level synchronization using SHA-256 hashing for deterministic change detection without external state tracking; (2) Dual-tier storage separating hot-tier vector indices (Milvus with HNSW) from cold-tier columnar versioning (Delta Lake with Parquet), optimizing query latency and storage cost independently; (3) Temporal query routing enabling point-in-time knowledge retrieval via delta-versioning with ACID consistency across tiers. Evaluation on a 100-document corpus versioned across five time points demonstrates: (i) 10-15% re-processing of content during updates compared to 100% for full re-indexing; (ii) sub-100ms retrieval latency on current knowledge; (iii) sub-2s latency for temporal queries across version history; and (iv) storage cost optimization through hot/cold tier separation (only current chunks in expensive vector indices). The approach enables production RAG deployments requiring simultaneous optimization for query performance, update efficiency, and regulatory compliance. Code and resources: [https://github.com/praj-tarun/LiveVectorLake]

LiveVectorLake: A Real-Time Versioned Knowledge Base Architecture for Streaming Vector Updates and Temporal Retrieval

TL;DR

LiveVectorLake tackles the challenge of maintaining both real-time, semantically rich knowledge updates and complete version history for RAG systems. It achieves this with a five-layer architecture that combines chunk-level content-addressable change detection, selective embedding of modified content, and a dual-tier storage strategy (hot Milvus vector indices and cold Delta Lake history) coupled with a temporal query engine for point-in-time retrieval. The key contributions are automatic chunk-level CDC via SHA-256, independent optimization of hot and cold storage with ACID cross-tier consistency, and dual-mode temporal queries that prevent leakage while preserving performance. Preliminary evaluation on a 100-document corpus shows 10–15% re-processing during updates, sub-100 ms latency for current queries, sub-2 s latency for temporal queries, and substantial hot-tier storage savings, indicating practical viability for production RAG deployments with regulatory and audit requirements.

Abstract

Modern Retrieval-Augmented Generation (RAG) systems struggle with a fundamental architectural tension: vector indices are optimized for query latency but poorly handle continuous knowledge updates, while data lakes excel at versioning but introduce query latency penalties. We introduce LiveVectorLake, a dual-tier temporal knowledge base architecture that enables real-time semantic search on current knowledge while maintaining complete version history for compliance, auditability, and point-in-time retrieval. The system introduces three core architectural contributions: (1) Content-addressable chunk-level synchronization using SHA-256 hashing for deterministic change detection without external state tracking; (2) Dual-tier storage separating hot-tier vector indices (Milvus with HNSW) from cold-tier columnar versioning (Delta Lake with Parquet), optimizing query latency and storage cost independently; (3) Temporal query routing enabling point-in-time knowledge retrieval via delta-versioning with ACID consistency across tiers. Evaluation on a 100-document corpus versioned across five time points demonstrates: (i) 10-15% re-processing of content during updates compared to 100% for full re-indexing; (ii) sub-100ms retrieval latency on current knowledge; (iii) sub-2s latency for temporal queries across version history; and (iv) storage cost optimization through hot/cold tier separation (only current chunks in expensive vector indices). The approach enables production RAG deployments requiring simultaneous optimization for query performance, update efficiency, and regulatory compliance. Code and resources: [https://github.com/praj-tarun/LiveVectorLake]
Paper Structure (43 sections, 1 equation, 1 figure, 3 tables)

This paper contains 43 sections, 1 equation, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. The LiveVectorLake Concept
  3. Core Contributions
  4. Evaluation Summary
  5. Related Work
  6. Retrieval-Augmented Generation Systems
  7. Change Data Capture and Streaming
  8. Temporal Databases and Data Lakehouses
  9. Vector Databases and Scaling RAG
  10. Research Gap
  11. System Architecture
  12. Layer 1: Change Detection and Ingestion
  13. Semantic Chunking
  14. Content-Addressable Hashing
  15. Change Detection Logic
  16. ...and 28 more sections

Figures (1)

  • Figure 1: LiveVectorLake system architecture showing CDC-based ingestion, dual-tier storage (hot: Milvus for active chunks, cold: Delta Lake for complete history), and temporal query routing.