Mitigating Negative Flips via Margin Preserving Training

TL;DR

The paper addresses negative flips that occur when updating image classifiers as new classes are added, noting that margin shrinkage exacerbates this issue. It introduces Margin Preserving Training (MPT), which combines a margin-calibrated softmax loss with a positive logit bias for new classes and a double-source focal distillation from both the old model and a fully trained new model to balance learning across old and new classes. The approach achieves consistent reductions in Negative Flip Rate while maintaining or improving overall accuracy on CIFAR100 and ImageNet1K, with qualitative analyses confirming margin preservation. The method provides a practical upgrade mechanism that mitigates prediction churn without resorting to expensive ensembles, albeit with some computational overhead and hyperparameter tuning.

Abstract

Minimizing inconsistencies across successive versions of an AI system is as crucial as reducing the overall error. In image classification, such inconsistencies manifest as negative flips, where an updated model misclassifies test samples that were previously classified correctly. This issue becomes increasingly pronounced as the number of training classes grows over time, since adding new categories reduces the margin of each class and may introduce conflicting patterns that undermine their learning process, thereby degrading performance on the original subset. To mitigate negative flips, we propose a novel approach that preserves the margins of the original model while learning an improved one. Our method encourages a larger relative margin between the previously learned and newly introduced classes by introducing an explicit margin-calibration term on the logits. However, overly constraining the logit margin for the new classes can significantly degrade their accuracy compared to a new independently trained model. To address this, we integrate a double-source focal distillation loss with the previous model and a new independently trained model, learning an appropriate decision margin from both old and new data, even under a logit margin calibration. Extensive experiments on image classification benchmarks demonstrate that our approach consistently reduces the negative flip rate with high overall accuracy.

Figures (4)

• Figure 1: Illustration of possible two prediction changes following a model update. Each row shows the predictions of the old model (left) and the new model (right) for a given image, showing two possible cases: (1) a Positive Flip, where the old model was wrong and the new model is correct---the desirable outcome after a model update; (2) a Negative Flip, where the old model was correct but the new model is wrong, which can lead to unexpected system behaviors. In this paper, we focus on reducing Negative Flips in a classification task.
• Figure 2: Schematic overview of the proposed Margin Preserving Training (MPT) approach. A model $\phi$ is trained with the margin-calibrated loss $\mathcal{L}_\Delta$ while receiving double-source focal distillation from two reference models, $\phi_\text{old}$ and $\phi_\text{new}$ (both optimized with cross-entropy). This combination mitigates margin underestimation for new classes, reduces negative flips between model updates, and preserves decision margins for old classes. The gray region marks areas near decision boundaries where negative flips typically occur. Margins are illustrated in a 2D embedding space for clarity, although MPT operates on logit-space margins over all classes.
• Figure 3: Comparison of embedding spaces for ResNet-18 models (embedding size = 2) on the CIFAR10 test set under various update strategies. Notably, when the model is trained directly on multiple classes ("No Treatment"), the inter-class margins are narrower compared to the original model. Our proposed Margin Preservation Training (MPT) maintains the margins of the original model, thereby significantly reducing the rate of negative flips without compromising the error rate across all classes.
• Figure 4: Effect of different margin value $k$ of Equation \ref{['eq:shift']} on CIFAR100 for both LM and KL distances in the focal distillation objectives. Bars show the error rate (ER), and lines represent the negative flip rate (NFR).