Feature Representation Transferring to Lightweight Models via Perception Coherence

TL;DR

This work introduces perception coherence, a probabilistic, ranking-based criterion for transferring feature representations from a large teacher to a lightweight student model, enabling cross-space transfers with potentially different feature dimensions. It replaces exact geometry preservation with preservation of relative dissimilarity orderings using a differentiable soft ranking loss, and provides a probabilistic framework linking local/global coherence to convergence guarantees in mini-batch estimations. Theoretically, the paper establishes convergence rates for the mini-batch estimator and derives bounds on local and global rank preservation as the student approximates the teacher. Empirically, it demonstrates competitive or superior performance to KD-based baselines on retrieval and classification benchmarks, and analyzes the impact of batch size and student capacity on coherence and downstream metrics. Overall, perception coherence offers a flexible, topology-informed approach to knowledge transfer that is robust to dimension mismatch and variational data densities, with practical implications for efficient deployment of lightweight models.

Abstract

In this paper, we propose a method for transferring feature representation to lightweight student models from larger teacher models. We mathematically define a new notion called \textit{perception coherence}. Based on this notion, we propose a loss function, which takes into account the dissimilarities between data points in feature space through their ranking. At a high level, by minimizing this loss function, the student model learns to mimic how the teacher model \textit{perceives} inputs. More precisely, our method is motivated by the fact that the representational capacity of the student model is weaker than the teacher model. Hence, we aim to develop a new method allowing for a better relaxation. This means that, the student model does not need to preserve the absolute geometry of the teacher one, while preserving global coherence through dissimilarity ranking. Importantly, while rankings are defined only on finite sets, our notion of \textit{perception coherence} extends them into a probabilistic form. This formulation depends on the input distribution and applies to general dissimilarity metrics. Our theoretical insights provide a probabilistic perspective on the process of feature representation transfer. Our experiments results show that our method outperforms or achieves on-par performance compared to strong baseline methods for representation transferring.

Figures (9)

• Figure 1: Illustration of feature representation transfer. A non-labeled transfer dataset is passed through both teacher and student models to obtain their respective feature representations. The transfer process consists in training the student model to somehow capture the input–input relationships encoded in the teacher model’s feature space.
• Figure 2: General scheme of our method. In each mini-batch, we use each input as the reference point to compute the dissimilarity to the others (compared set). Then, using loss function in Eq. (\ref{['eq:main_loss']}), the student model is trained to respect the perception coherence. Notice that $f_1$ and $f_2$ can be the whole model (when using the output layer) or a part of the model (when using an intermediate layer).
• Figure 3: Illustration of dissimilarity ranking (in feature space) in the case of 4 points. Each of all the data points is considered as the reference point, and the dissimilarities to other data points are ranked to capture the cumulative distribution of dissimilarity (in feature space).
• Figure 4: Transferring process on $2D$ datasets. Top row represents the teacher configuration (fixed), bottom row represents the evolution of the student configuration along the transfer process (from left to right). Each gray line associates $x^2_i$ with $x^1_i$ (for the same $i$). We observe that the learned configuration preserves a global structural coherence without preserving completely the geometry.
• Figure 5: Transferring process from $3D$ to $2D$ data.

Theorems & Definitions (35)

• Definition 3.1: Absolute perception coherence
• Remark 3.1
• Remark 3.2: Relation between cumulative function and ranking
• Definition 3.2: Perception coherence level
• Remark 3.3
• Proposition 3.1
• Proof 3.1
• Definition 3.3: Local $\alpha$-perception coherence
• Definition 3.4: Global $\alpha$-perception coherence
• Definition 4.1: Empirical Estimator