SwiftEmbed: Ultra-Fast Text Embeddings via Static Token Lookup for Real-Time Applications

TL;DR

This work tackles the latency bottleneck of transformer-based text embeddings by introducing a static token lookup approach that bypasses transformer inference. By combining pre-computed token representations with SIMD-optimized pooling and zero-copy serialization in Rust, the method achieves sub-millisecond latency and high throughput (≈50k RPS) while retaining a substantial portion of contextual model quality (≈89% of Sentence-BERT). Extensive evaluation across MTEB and domain tasks demonstrates strong performance on duplicate detection and semantic similarity, with predictable, linear scaling and a compact 32 MB model footprint. The results enable real-time embedding applications in high-density deployments, edge computing, and rapid retrieval tasks, albeit with trade-offs in multilingual transfer and some domain-specific accuracy. Future directions include hybrid contextualization, adaptive quantization, and cross-modal static embeddings to further narrow the gap to contextual models while preserving low latency.

Abstract

We present a static token lookup methodology for text embedding generation that achieves 1.12 ms p50 latency for single text embeddings while maintaining 60.6 MTEB average score across 8 representative tasks, corresponding to 89% of contextual model quality. The Rust implementation delivers 50,000 requests per second throughput through static embedding lookup, optimized mean pooling, and zero-copy IEEE754 binary serialization. Evaluation demonstrates exceptional duplicate detection performance (90.1% AP), strong semantic similarity (76.1% Spearman correlation), and domain-specific performance ranging from 75% to 131% of baseline across specialized domains. The system enables real-time embedding applications where sub-5ms latency is critical.