FaTRQ: Tiered Residual Quantization for LLM Vector Search in Far-Memory-Aware ANNS Systems

TL;DR

FaTRQ addresses the pervasive bottleneck in ANNS refinement by moving the second-pass distance computation into far memory and performing progressive, residual-based refinement on tiered data. By decomposing the L2 distance and encoding residuals as multiplication-free, ternary codes, FaTRQ enables streaming refinements that prune candidates early and avoid reconstructing full vectors. A lightweight offline calibration aligns the estimator with recall boundaries, while a CXL Type-2 accelerator demonstrates feasible hardware integration. The result is up to 9× throughput improvement and 2.4× storage efficiency gains over state-of-the-art GPU pipelines, highlighting the practical potential of far-memory–aware refinement for large-scale RAG systems.

Abstract

Approximate Nearest-Neighbor Search (ANNS) is a key technique in retrieval-augmented generation (RAG), enabling rapid identification of the most relevant high-dimensional embeddings from massive vector databases. Modern ANNS engines accelerate this process using prebuilt indexes and store compressed vector-quantized representations in fast memory. However, they still rely on a costly second-pass refinement stage that reads full-precision vectors from slower storage like SSDs. For modern text and multimodal embeddings, these reads now dominate the latency of the entire query. We propose FaTRQ, a far-memory-aware refinement system using tiered memory that eliminates the need to fetch full vectors from storage. It introduces a progressive distance estimator that refines coarse scores using compact residuals streamed from far memory. Refinement stops early once a candidate is provably outside the top-k. To support this, we propose tiered residual quantization, which encodes residuals as ternary values stored efficiently in far memory. A custom accelerator is deployed in a CXL Type-2 device to perform low-latency refinement locally. Together, FaTRQ improves the storage efficiency by 2.4$\times$ and improves the throughput by up to 9$ \times$ than SOTA GPU ANNS system.

Figures (8)

• Figure 1: ANNS is the key component of the RAG pipeline.
• Figure 2: Runtime breakdown of IVF-refinement ANNS system
• Figure 3: Memory-tiered layout of the FaTRQ framework
• Figure 4: Visualization of the residual vector and query vector
• Figure 5: System architecture of the FaTRQ-augmented ANNS pipeline.