Optimization Can Learn Johnson Lindenstrauss Embeddings

TL;DR

This paper asks whether Johnson-Lindenstrauss (JL) embeddings can be learned via optimization rather than randomization. Direct optimization over projection matrices is plagued by a non-convex landscape with bad local minima; the authors instead optimize over Gaussian solution samplers $(M,\sigma^2)$ where $A\sim N(M,\sigma^2)$, and iteratively reduce variance in a diffusion-inspired fashion. They define a relaxed objective $f$ and a regularized objective $g=f+\sigma^2/2$, showing that any $\rho$-second-order stationary point of $g$ has vanishing variance and yields a JL-compliant deterministic matrix $M$; a Hessian-descent algorithm then converges in polynomial time to such a point. Theoretical results are complemented by simulations demonstrating near-optimal distortion as the method converges to a deterministic JL embedding, outperforming standard Gaussian constructions. The approach provides a principled derandomization pathway with potential applicability to other optimization-on-samplers problems beyond JL embeddings.

Abstract

Embeddings play a pivotal role across various disciplines, offering compact representations of complex data structures. Randomized methods like Johnson-Lindenstrauss (JL) provide state-of-the-art and essentially unimprovable theoretical guarantees for achieving such representations. These guarantees are worst-case and in particular, neither the analysis, nor the algorithm, takes into account any potential structural information of the data. The natural question is: must we randomize? Could we instead use an optimization-based approach, working directly with the data? A first answer is no: as we show, the distance-preserving objective of JL has a non-convex landscape over the space of projection matrices, with many bad stationary points. But this is not the final answer. We present a novel method motivated by diffusion models, that circumvents this fundamental challenge: rather than performing optimization directly over the space of projection matrices, we use optimization over the larger space of random solution samplers, gradually reducing the variance of the sampler. We show that by moving through this larger space, our objective converges to a deterministic (zero variance) solution, avoiding bad stationary points. This method can also be seen as an optimization-based derandomization approach and is an idea and method that we believe can be applied to many other problems.

Figures (1)

• Figure 1: Plot of the distortion obtained through optimization over $5000$ iterations vs the average distortion using a random Gaussian matrix (left plot), and the progression of variance over the same number of iterations (right plot). To calculate the distortions' progression with our method, we sample from the updated mean matrix and variance at each iteration and compute the distortion. We remark that the distortion plotted is a proxy for our objective in Equation \ref{['eq:regularizedobjectivefunction']}. We observe that our optimization-based approach converges to a deterministic solution sampler. By using the mean matrix $\bm{M}$, we achieve nearly optimal distortion, where $|\bm{M}x| \approx |x|$.

Theorems & Definitions (21)

• Theorem : Informal version of Theorem \ref{['thm:counterexample']}
• Theorem : Informal Version of Theorem \ref{['thm:qualitative']}
• Theorem : Informal Version of Theorem \ref{['thm:quantitative']}
• Definition 1: Johnson-Lindenstrauss guarantee
• Lemma 1: Distributional Johnson-Lindenstrauss Lemma
• Definition 2
• Definition 3
• Theorem 1
• Lemma 2
• Lemma 3