Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond the Geometric Curse: High-Dimensional N-Gram Hashing for Dense Retrieval

Sangeet Sharma

TL;DR

The paper tackles the fundamental bottleneck in dense retrieval: the limited capacity imposed by fixed, learned embedding dimensions. It introduces NUMEN, a training-free pipeline that maps text into extremely high-dimensional vectors via deterministic character n-gram hashing, eliminating the learned embedding layer. Empirically, NUMEN achieves 93.90% Recall@100 on the LIMIT benchmark at 32k dimensions, surpassing the BM25 sparse baseline and outperforming contemporary dense models, suggesting that increasing geometric space—not architectural complexity—is key. The approach enables scalable, training-free dense retrieval with compatibility for reranking and retrieval-augmented generation, and provides a practical path toward verifiable grounding in NLP systems.

Abstract

Why do even the most powerful 7B-parameter embedding models struggle with simple retrieval tasks that the decades old BM25 handles with ease? Recent theory suggests that this happens because of a dimensionality bottleneck. This occurs when we force infinite linguistic nuances into small, fixed-length learned vectors. We developed NUMEN to break this bottleneck by removing the learning process entirely. Instead of training heavy layers to map text to a constrained space, NUMEN uses deterministic character hashing to project language directly onto high-dimensional vectors. This approach requires no training, supports an unlimited vocabulary, and allows the geometric capacity scale as needed. On the LIMIT benchmark, NUMEN achieves 93.90 % Recall@100 at 32,768 dimensions. This makes it the first dense retrieval model to officially surpass the sparse BM25 baseline 93.6 %. Our findings show that the real problem in dense retrieval isn't the architecture, but the embedding layer itself. The solution isn't necessarily smarter training, but simply providing more room to breathe.

Beyond the Geometric Curse: High-Dimensional N-Gram Hashing for Dense Retrieval

TL;DR

The paper tackles the fundamental bottleneck in dense retrieval: the limited capacity imposed by fixed, learned embedding dimensions. It introduces NUMEN, a training-free pipeline that maps text into extremely high-dimensional vectors via deterministic character n-gram hashing, eliminating the learned embedding layer. Empirically, NUMEN achieves 93.90% Recall@100 on the LIMIT benchmark at 32k dimensions, surpassing the BM25 sparse baseline and outperforming contemporary dense models, suggesting that increasing geometric space—not architectural complexity—is key. The approach enables scalable, training-free dense retrieval with compatibility for reranking and retrieval-augmented generation, and provides a practical path toward verifiable grounding in NLP systems.

Abstract

Why do even the most powerful 7B-parameter embedding models struggle with simple retrieval tasks that the decades old BM25 handles with ease? Recent theory suggests that this happens because of a dimensionality bottleneck. This occurs when we force infinite linguistic nuances into small, fixed-length learned vectors. We developed NUMEN to break this bottleneck by removing the learning process entirely. Instead of training heavy layers to map text to a constrained space, NUMEN uses deterministic character hashing to project language directly onto high-dimensional vectors. This approach requires no training, supports an unlimited vocabulary, and allows the geometric capacity scale as needed. On the LIMIT benchmark, NUMEN achieves 93.90 % Recall@100 at 32,768 dimensions. This makes it the first dense retrieval model to officially surpass the sparse BM25 baseline 93.6 %. Our findings show that the real problem in dense retrieval isn't the architecture, but the embedding layer itself. The solution isn't necessarily smarter training, but simply providing more room to breathe.
Paper Structure (39 sections, 9 equations, 1 figure, 3 tables, 1 algorithm)

This paper contains 39 sections, 9 equations, 1 figure, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Dense vs. Sparse Retrieval
  4. The LIMIT Benchmark
  5. Methodology
  6. Architecture Overview
  7. Character N-Gram Extraction
  8. Feature Hashing
  9. Weighted Aggregation
  10. Log-Saturation and Normalization
  11. Retrieval
  12. Algorithm
  13. Experiments
  14. Setup
  15. Baselines
  16. ...and 24 more sections

Figures (1)

  • Figure 1: Performance Analysis. Left: Numen recall metrics scale logarithmically with dimension. Right: Numen (32k) outperforms BM25 and SOTA dense retrievers at Recall@100.