GPU-Resident Inverted File Index for Streaming Vector Databases

TL;DR

This work identifies CPU–GPU roundtrips as the fundamental bottleneck blocking streaming vector databases from using GPU-based IVF indices. It introduces SIVF, a GPU-native architecture that replaces rigid contiguous memory with a slab-based allocator, a validity bitmap for in-place lazy eviction, and a GPU-resident address translation table, enabling lock-free insertion and $O(1)$ deletion. Across SIFT1M and GIST1M, SIVF achieves up to $13{,}300\times$ faster deletions, $36\times$ to $105\times$ higher ingestion throughput, and $161\times$ to $266\times$ improvements in end-to-end streaming latency with negligible memory overhead ($<0.8\%$). These results demonstrate that moving memory management to the GPU transforms static vector search into a dynamic, real-time streaming engine with broad practical impact for live recommendations, RAG systems, and streaming knowledge bases.

Abstract

Vector search has emerged as the computational backbone of modern AI infrastructure, powering critical systems ranging from Vector Databases to Retrieval-Augmented Generation (RAG). While the GPU-accelerated Inverted File (IVF) index acts as one of the most widely used techniques for these large-scale workloads due to its memory efficiency, its traditional architecture remains fundamentally static. Existing designs rely on rigid and contiguous memory layouts that lack native support for in-place mutation, creating a severe bottleneck for streaming scenarios. In applications requiring real-time knowledge updates, such as live recommendation engines or dynamic RAG systems, maintaining index freshness necessitates expensive CPU-GPU roundtrips that cause system latency to spike from milliseconds to seconds. In this paper, we propose SIVF (Streaming Inverted File), a new GPU-native architecture designed to empower vector databases with high-velocity data ingestion and deletion capabilities. SIVF replaces the static memory layout with a slab-based allocation system and a validity bitmap, enabling lock-free and in-place mutation directly in VRAM. We further introduce a GPU-resident address translation table (ATT) to resolve the overhead of locating vectors, providing $O(1)$ access to physical storage slots. We evaluate SIVF against the industry-standard GPU IVF implementation on the SIFT1M and GIST1M datasets. Microbenchmarks demonstrate that SIVF reduces deletion latency by up to $13,300\times$ (from 11.8 seconds to 0.89 ms on GIST1M) and improves ingestion throughput by $36\times$ to $105\times$. In end-to-end sliding window scenarios, SIVF eliminates system freezes and achieves a $161\times$ to $266\times$ speedup with single-digit millisecond latency. Notably, this performance incurs negligible storage penalty, maintaining less than 0.8\% memory overhead compared to static indices.

Figures (11)

• Figure 1: The asymmetry between insertion and deletion latency on GPU indices.
• Figure 2: Performance evaluation of vector ingestion.
• Figure 3: Performance evaluation of vector search.
• Figure 4: Performance comparison of vector deletion between Faiss IVF (Baseline) and SIVF.
• Figure 5: Ingestion throughput.