Prototype Perturbation for Relaxing Alignment Constraints in Backward-Compatible Learning

TL;DR

This work tackles the challenge of updating retrieval models without costly backfilling by relaxing strict alignment to the old feature space. It introduces prototype perturbation, forming a pseudo-old space from perturbed class prototypes, and presents two implementations, NDPP and ODPP, that leverage both old and new prototypes during training. Empirical results on GLDv2 and In-shop show improved discriminative performance and cross-model compatibility compared to state-of-the-art BCL methods, including in sequential update scenarios. The approach offers a practical pathway to faster, more scalable model updates for large-scale retrieval systems, with future work aimed at reducing hyperparameter dependence and further adaptive perturbation strategies.

Abstract

The traditional paradigm to update retrieval models requires re-computing the embeddings of the gallery data, a time-consuming and computationally intensive process known as backfilling. To circumvent backfilling, Backward-Compatible Learning (BCL) has been widely explored, which aims to train a new model compatible with the old one. Many previous works focus on effectively aligning the embeddings of the new model with those of the old one to enhance the backward-compatibility. Nevertheless, such strong alignment constraints would compromise the discriminative ability of the new model, particularly when different classes are closely clustered and hard to distinguish in the old feature space. To address this issue, we propose to relax the constraints by introducing perturbations to the old feature prototypes. This allows us to align the new feature space with a pseudo-old feature space defined by these perturbed prototypes, thereby preserving the discriminative ability of the new model in backward-compatible learning. We have developed two approaches for calculating the perturbations: Neighbor-Driven Prototype Perturbation (NDPP) and Optimization-Driven Prototype Perturbation (ODPP). Particularly, they take into account the feature distributions of not only the old but also the new models to obtain proper perturbations along with new model updating. Extensive experiments on the landmark and commodity datasets demonstrate that our approaches perform favorably against state-of-the-art BCL algorithms.

Figures (6)

• Figure 1: Two different paradigms for updating the embedding model for the retrieval system. (a) The new model is trained independently and used to re-extract the embeddings of all gallery data, i.e., backfilling, which is time-consuming and computationally intensive. (b) With backward-compatible learning, the query embedding is directly comparable with the existing gallery embeddings without backfilling.
• Figure 2: Distribution of the embeddings and prototypes of 4 classes from GLDv2 gldv2, which is visualized with t-SNE t-SNE. (a) Classes 782 & 1414 are nearly indistinguishable in the old feature space. (b) The new model is trained with rigorous alignment constraints to the old feature space, producing new embeddings strictly surrounding the old prototypes. As a result, the new embeddings of these classes are clustered closely and hard to distinguish.
• Figure 3: Illustration of neighbor-driven prototype perturbation. (a) Perturbations are calculated based on the neighboring old prototypes, and new embeddings are constrained to align with the perturbed old prototypes. (b) Perturbations are updated based on the neighboring new prototypes, and new embeddings are constrained to align with the secondary perturbed old prototypes.
• Figure 4: Experimental results of our proposed NDPP with varying scale factor ($\alpha_1$ and $\alpha_2$) and varying neighbor number ($K$) on the RParis dataset.
• Figure 5: Experimental results of our proposed ODPP with varying threshold ($\theta_{old}$ and $\theta_{new}$) and varying loss weight ($\gamma$) on the RParis dataset.