Jasper: ANNS Quantized for Speed, Built for Change on GPU

TL;DR

Jasper tackles the challenge of updatable, GPU-accelerated ANNS by integrating a GPU-native Vamana graph with lock-free batch-parallel construction and RaBitQ quantization. The three core innovations are batch-parallel construction for streaming updates, a GPU-optimized beam search kernel, and a quantization approach that reduces memory movement up to $8\times$ while maintaining accuracy. Across five datasets, Jasper achieves up to $1.93\times$ higher query throughput than CAGRA and up to $19$–$131\times$ faster queries than the prior GPU Vamana baseline, with peak insertion throughput of 674K insertions/s. The results demonstrate that reducing memory traffic and exploiting GPU-friendly parallelism yield substantial gains for large-scale, updatable ANNS on GPUs.

Abstract

Approximate nearest neighbor search (ANNS) is a core problem in machine learning and information retrieval applications. GPUs offer a promising path to high-performance ANNS: they provide massive parallelism for distance computations, are readily available, and can co-locate with downstream applications. Despite these advantages, current GPU-accelerated ANNS systems face three key limitations. First, real-world applications operate on evolving datasets that require fast batch updates, yet most GPU indices must be rebuilt from scratch when new data arrives. Second, high-dimensional vectors strain memory bandwidth, but current GPU systems lack efficient quantization techniques that reduce data movement without introducing costly random memory accesses. Third, the data-dependent memory accesses inherent to greedy search make overlapping compute and memory difficult, leading to reduced performance. We present Jasper, a GPU-native ANNS system with both high query throughput and updatability. Jasper builds on the Vamana graph index and overcomes existing bottlenecks via three contributions: (1) a CUDA batch-parallel construction algorithm that enables lock-free streaming insertions, (2) a GPU-efficient implementation of RaBitQ quantization that reduces memory footprint up to 8x without the random access penalties, and (3) an optimized greedy search kernel that increases compute utilization, resulting in better latency hiding and higher throughput. Our evaluation across five datasets shows that Jasper achieves up to 1.93x higher query throughput than CAGRA and achieves up to 80% peak utilization as measured by the roofline model. Jasper's construction scales efficiently and constructs indices an average of 2.4x faster than CAGRA while providing updatability that CAGRA lacks. Compared to BANG, the previous fastest GPU Vamana implementation, Jasper delivers 19-131x faster queries.

Figures (12)

• Figure 1: GPU memory hierarchy and roofline model.
• Figure 2: Batch graph construction pipeline in Jasper.
• Figure 3: ANNS query pipeline in Jasper.
• Figure 4: Comparison of the two different load strategies tested in Jasper: tiled loads load one object per thread in the tile at a time, whereas the chunked load strategy loads a fixed amount of data (16 bytes) per thread and has each thread work on all objects loaded in that chunk.
• Figure 5: During distance calculation, RaBitQ loads query vector and its metadata from shared memory, and it loads quantized data vector and its metadata from global memory.