$\mathbb{R}^{2k}$ is Theoretically Large Enough for Embedding-based Top-$k$ Retrieval
Zihao Wang, Hang Yin, Lihui Liu, Hanghang Tong, Yangqiu Song, Ginny Wong, Simon See
TL;DR
This work formalizes Minimal Embeddable Dimension (MED) as the smallest embedding dimension required to perfectly retrieve top-$k$ subset memberships, relating MED to VC-dimension through $k$-shattering. It derives tight, geometry-based bounds for standard scoring families: $k-1 \le \textsc{MED}(m,k;\mathcal{F}) \le 2k$ for $\mathcal{F}_{\rm linear}$ and analogous bounds for $\mathcal{F}_{\cos}$ and $\mathcal{F}_{\ell_{2}}$, showing $\Theta(k)$ scaling independent of $m$. The centroid variant MED-C admits upper bounds of $O(k^2\log m)$ and, in numerical simulations, exhibits a logarithmic dependence on $m$, suggesting learnability, not geometry, as the main bottleneck for embedding-based retrieval. The results imply that embedding-based top-$k$ retrieval can be feasible in relatively low dimensions, guiding algorithm design toward learnability improvements and efficient centroid-based schemes rather than increasing ambient dimension alone.
Abstract
This paper studies the minimal dimension required to embed subset memberships ($m$ elements and ${m\choose k}$ subsets of at most $k$ elements) into vector spaces, denoted as Minimal Embeddable Dimension (MED). The tight bounds of MED are derived theoretically and supported empirically for various notions of "distances" or "similarities," including the $\ell_2$ metric, inner product, and cosine similarity. In addition, we conduct numerical simulation in a more achievable setting, where the ${m\choose k}$ subset embeddings are chosen as the centroid of the embeddings of the contained elements. Our simulation easily realizes a logarithmic dependency between the MED and the number of elements to embed. These findings imply that embedding-based retrieval limitations stem primarily from learnability challenges, not geometric constraints, guiding future algorithm design.