Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacements

TL;DR

This paper shows the strengths and critical issues of stationary representations in the case in which a model undergoing sequential fine-tuning is asynchronously replaced by downloading a better-performing model pretrained elsewhere.

Abstract

Learning compatible representations enables the interchangeable use of semantic features as models are updated over time. This is particularly relevant in search and retrieval systems where it is crucial to avoid reprocessing of the gallery images with the updated model. While recent research has shown promising empirical evidence, there is still a lack of comprehensive theoretical understanding about learning compatible representations. In this paper, we demonstrate that the stationary representations learned by the $d$-Simplex fixed classifier optimally approximate compatibility representation according to the two inequality constraints of its formal definition. This not only establishes a solid foundation for future works in this line of research but also presents implications that can be exploited in practical learning scenarios. An exemplary application is the now-standard practice of downloading and fine-tuning new pre-trained models. Specifically, we show the strengths and critical issues of stationary representations in the case in which a model undergoing sequential fine-tuning is asynchronously replaced by downloading a better-performing model pre-trained elsewhere. Such a representation enables seamless delivery of retrieval service (i.e., no reprocessing of gallery images) and offers improved performance without operational disruptions during model replacement. Code available at: https://github.com/miccunifi/iamcl2r.

Figures (10)

• Figure 1: Improved Asynchronous Model Compatible Lifelong Learning Representation (IAM-CL$^2$R pronounced "I am clear"). In the process of lifelong learning, a model is sequentially fine-tuned and asynchronously replaced with improved third-party models that are pre-trained externally. Stationary representations ensure seamless retrieval services and better performance, without the need to reprocess gallery images.
• Figure 2: Key concepts and relationships underlying Theorem \ref{['theo:compatibility']}. Distances in feature space of two distinct samples within their hyperballs before and after model update, with the update process represented by a dotted arrow. (a): Distances between samples $\mathbf{x}_i$ and $\mathbf{x}_j$ of the same class $y$ before (red) and after (cyan) model update. (b): Distances between samples $\mathbf{x}_i$ of class $y_i$ and $\mathbf{x}_j$ of class $y_j$, before (red) and after (cyan) model update. Compatibility is verified by computing the expected lengths of the segments and verifying if they satisfy the inequalities of the compatibility definition. A transparently colored instance shows counter-intuitive distance behavior. Expectation reveals the underlying pattern of approximation.
• Figure 3: Training loss of a $d$-Simplex fixed classifier during a model update. Values are the cross-entropy loss of Eq. \ref{['eq:loss_ce_simplex']} (red line) and the loss of Eq. \ref{['eq:total_loss']} (blue line). Models are trained on MNIST.
• Figure 4: Average multi-model Accuracy ($AA_{t}$) evaluated across 31 tasks using CIFAR100R/10, showing: (a) model replacements at tasks 11 and 21 (indicated by yellow markers); (b) no model replacement.
• Figure 5: Compatibility Matrices for $d$-Simplex-HOC, CVS, BCT-ER, and $d$-Simplex-FD on CIFAR100R/10 across $7$ tasks. Model replacements at tasks $3$ and $5$ are highlighted in bold. Entries failing to meet compatibility criteria as defined in shen2020towards are marked with a light-red background.

Theorems & Definitions (5)

• Definition 1: Compatibility
• Theorem 1: Stationarity $\implies$ Compatibility
• Lemma 1
• Theorem 1: Stationarity $\implies$ Compatibility
• Corollary 1: Infeasibility