CUBO: Self-Contained Retrieval-Augmented Generation on Consumer Laptops 10 GB Corpora, 16 GB RAM, Single-Device Deployment
Paolo Astrino
TL;DR
CUBO tackles the challenge of self-contained, privacy-preserving retrieval-augmented generation on consumer laptops with only 16 GB RAM. It combines streaming ingestion, a tiered hot/cold FAISS/indexing stack, and quantization-aware routing to deliver competitive retrieval performance within a hard 15.5 GB RAM ceiling, demonstrating stable end-to-end latency and memory usage on real BEIR and UltraDomain benchmarks. Key innovations include 8-bit IVFPQ with memory-mapped cold storage, a 500K-vector hot index, and Reciprocal Rank Fusion with domain-robust defaults, supplemented by offline QAR calibration and laptop-mode auto-detection for zero-config operation. The work provides a practical path to deploy privacy-preserving, air-gapped RAG systems on commodity hardware, with thorough ablations, memory profiling, and BEIR-based evaluations that illuminate the trade-offs between resource constraints and retrieval quality in real-world professional archives.
Abstract
Organizations handling sensitive documents face a tension: cloud-based AI risks GDPR violations, while local systems typically require 18-32 GB RAM. This paper presents CUBO, a systems-oriented RAG platform for consumer laptops with 16 GB shared memory. CUBO's novelty lies in engineering integration of streaming ingestion (O(1) buffer overhead), tiered hybrid retrieval, and hardware-aware orchestration that enables competitive Recall@10 (0.48-0.97 across BEIR domains) within a hard 15.5 GB RAM ceiling. The 37,000-line codebase achieves retrieval latencies of 185 ms (p50) on C1,300 laptops while maintaining data minimization through local-only processing aligned with GDPR Art. 5(1)(c). Evaluation on BEIR benchmarks validates practical deployability for small-to-medium professional archives. The codebase is publicly available at https://github.com/PaoloAstrino/CUBO.