Robust Nearest Neighbour Retrieval Using Targeted Manifold Manipulation

TL;DR

This work tackles nearest-neighbour retrieval in high-dimensional deep feature spaces by removing the reliance on hand-picked feature layers and distance metrics. It introduces Targeted Manifold Manipulation-NN (TMM-NN), a query-specific backdoor trigger that creates a local distortion around the query, enabling neighbours to be ranked by dummy-class activation rather than by geometric proximity. The approach is supported by a formal theoretical analysis, including a robustness radius, Margin Implies Ranking Stability, Margin Bound, and a key Theorem showing larger robustness for the trigger-based method, along with an OOD data self-retrieval bound. Empirically, TMM-NN demonstrates improved noise robustness and semantic alignment across MNIST, SVHN, GTSRB, and CIFAR-10, with additional validation via LVLM-based semantic judgments and architecture-agnostic performance (including ViT). These results suggest a practical, scalable alternative for robust NN retrieval that bypasses manual feature-layer tuning while preserving semantic integrity.

Abstract

Nearest-neighbour retrieval is central to classification and explainable-AI pipelines, but current practice relies on hand-tuning feature layers and distance metrics. We propose Targeted Manifold Manipulation-Nearest Neighbour (TMM-NN), which reconceptualises retrieval by assessing how readily each sample can be nudged into a designated region of the feature manifold; neighbourhoods are defined by a sample's responsiveness to a targeted perturbation rather than absolute geometric distance. TMM-NN implements this through a lightweight, query-specific trigger patch. The patch is added to the query image, and the network is weakly ``backdoored'' so that any input with the patch is steered toward a dummy class. Images similar to the query need only a slight shift and are classified as the dummy class with high probability, while dissimilar ones are less affected. By ranking candidates by this confidence, TMM-NN retrieves the most semantically related neighbours. Robustness analysis and benchmark experiments confirm this trigger-based ranking outperforms traditional metrics under noise and across diverse tasks.

Figures (8)

• Figure 1: An example query point from both the MNIST and GTSRB datasets and the nearest neighbour retrieved by Cosine based similarity measure, $L_{2}$-norm distance measure and Our proposed method.
• Figure 2: Binary classifier with two classes $C_{1}$ and $C_{2}$. A query point is situated in the $C_{1}$ class region. Once the sample is attached with trigger it produces a local optima which helps to capture the nearest neighbours.
• Figure 3: Performance comparison of retrieval robustness under two perturbations---brightness variation and Gaussian noise---across four datasets (CIFAR-10, MNIST, SVHN, and GTSRB). In both scenarios, our proposed method consistently outperforms the baselines ($L_{2}$-distance and cosine similarity (CS)), demonstrating superior robustness to input distortions.
• Figure 4: Illustration of query samples ( $x_{q}$) along with their top-3 nearest neighbours retrieved using two baseline methods ($L_{2}$-distance and cosine similarity (CS)) and our proposed approach.
• Figure 5: GPT-4o response regarding similarity between the query image and retrieved nearest neighbours.