On Design Choices in Similarity-Preserving Sparse Randomized Embeddings

TL;DR

The paper investigates practical design choices in FlyHash, an Expand & Sparsify-inspired similarity-preserving embedding built from sparse binary random projections and a $k$-WTA sparsification. It demonstrates that preprocessing, the distribution and density of the projection matrix, and the choice between binary and block-sparse nonlinearities dramatically affect MAP for similarity search on standard datasets. Key findings include that mean-centering with normalization often improves ranking (especially for Ang/Ang similarity), that hypergeometric projection can outperform binomial under dense data conditions, and that block sparse codes can reduce bit-count while offering competitive performance when bits are matched. The work highlights hardware- and algorithm-enabled trade-offs and suggests avenues for learned projections, data-adaptive expansion, and applications to classification tasks.

Abstract

Expand & Sparsify is a principle that is observed in anatomically similar neural circuits found in the mushroom body (insects) and the cerebellum (mammals). Sensory data are projected randomly to much higher-dimensionality (expand part) where only few the most strongly excited neurons are activated (sparsify part). This principle has been leveraged to design a FlyHash algorithm that forms similarity-preserving sparse embeddings, which have been found useful for such tasks as novelty detection, pattern recognition, and similarity search. Despite its simplicity, FlyHash has a number of design choices to be set such as preprocessing of the input data, choice of sparsifying activation function, and formation of the random projection matrix. In this paper, we explore the effect of these choices on the performance of similarity search with FlyHash embeddings. We find that the right combination of design choices can lead to drastic difference in the search performance.

Figures (4)

• Figure 1: The effect of data preprocessing on the MAP obtained from the embeddings. Plots depict the dependency between MAP and the number of nonzero components $k$ in an embedding; $D$ was set to $20k$. Plots correspond to unique combinations of a dataset (columns) and an embedding variant (rows). The values are averaged over $10$ random initializations of $\mathbf{M}$.
• Figure 2: The effect of the distribution of the values in the RP matrix on the performance of the embedding. The plot depicts the dependency between MAP and the number of nonzero components $k$ in a randomized embedding; $D$ was set to be $20k$ (i.e., as $D$ grows, the density of $\mathbf{z}$ is fixed to $0.05$). The ranking of both input vectors and their embeddings was done by Euclidean distances. Plots correspond to unique combinations of a dataset (columns) and an embedding variant (rows). The datasets were processed with the original preprocessing, Eqs. (\ref{['eq:norm:abs']})-(\ref{['eq:norm:mean']}). The MAP values are averaged over $10$ random initializations of $\mathbf{M}$.
• Figure 3: The effect of the block sparse non-linearity on the performance of the embeddings. Each plot depicts MAP against the number of 1-components $k$ in an embedding; $D$ was set to be $20k$. The ranking of both input vectors and their embeddings was done by Euclidean distances. Plots correspond to unique combinations of a dataset (columns) and type of preprocessing (rows). The reported values are averaged over $10$ random initializations of $\mathbf{M}$.
• Figure 4: The effect of the "sequential" processing of randomized embeddings (dash-dotted and dotted lines). Gray lines (solid and dashed) are the corresponding results from Fig. \ref{['fig:sparse:block']} and used as the reference. A plot depicts the dependency between MAP and the number of processed blocks of a randomized embedding with $k=256$; the block size was fixed to $20$ so $D$ was set to be $5,120$. The ranking between both input vectors and their embeddings was by Euclidean distances. Plots correspond to unique combinations of a dataset (columns) and type of preprocessing (rows). The reported values are averaged over $10$ random initializations of $\mathbf{M}$.