ELODI: Ensemble Logit Difference Inhibition for Positive-Congruent Training

TL;DR

This work tackles the challenge of updating classifiers with minimal negative flips while preserving accuracy. It analyzes how logit-displacement in ensemble predictions underpins negative flips and demonstrates that homogeneous ensembles shrink this displacement, enabling ensemble-level performance without heavier inference costs. The authors introduce Ensemble Logit Difference Inhibition (Elodi), which distills ensemble variance into a single model via a top-K logit-focused distillation loss (LDI) combined with cross-entropy, achieving near-ensemble NFR reductions and improved ER in multiple settings. Through extensive experiments on ImageNet, iNaturalist, and text data, Elodi proves effective across data-growth, multi-update chains, and architecture changes, offering a practical, scalable path to positive-congruent training. The approach significantly mitigates cross-model incompatibilities in production systems, with broad implications for safely updating deployed models.

Abstract

Negative flips are errors introduced in a classification system when a legacy model is updated. Existing methods to reduce the negative flip rate (NFR) either do so at the expense of overall accuracy by forcing a new model to imitate the old models, or use ensembles, which multiply inference cost prohibitively. We analyze the role of ensembles in reducing NFR and observe that they remove negative flips that are typically not close to the decision boundary, but often exhibit large deviations in the distance among their logits. Based on the observation, we present a method, called Ensemble Logit Difference Inhibition (ELODI), to train a classification system that achieves paragon performance in both error rate and NFR, at the inference cost of a single model. The method distills a homogeneous ensemble to a single student model which is used to update the classification system. ELODI also introduces a generalized distillation objective, Logit Difference Inhibition (LDI), which only penalizes the logit difference of a subset of classes with the highest logit values. On multiple image classification benchmarks, model updates with ELODI demonstrate superior accuracy retention and NFR reduction.

Figures (12)

• Figure 1: The overview of the proposed Ensemble Logit Difference Inhibition (Elodi) method and its effectiveness compared to baseline methods. When updating an old model to a new one, we aim to minimize the negative flip rate (NFR), namely the ratio of instances that are misclassified by the new model but correctly classified by the old one over all samples, and the error rate (ER) simultaneously. (a) In Elodi, for both the old and new models that we wish to deploy, we first train $m$ replicas of the same architectures on the same training data with different network initializations to form a deep ensemble. Next, we train both the old and new models to deploy using the Logit Difference Inhibition (LDI) loss with respect to the ensemble of $m$ models of their same architecture. The result is a new model that achieves a significantly reduced NFR compared to the old model. (b) We look at an example where the old model is a ResNet-18 and the new model is a ResNet-50 and present the scatter plot of the ResNet-50's ER and its NFR w.r.t the ResNet-18. The more left and lower, the better. Elodi improves both ER and NFR than baseline methods. The numbers can be found in \ref{['tab:main_result']}. Particularly, Elodi is close to the ensemble paragon, without the prohibitive computation cost of ensembles.
• Figure 2: ER/NFR w.r.t ensemble size. Our hypothesis explains the behaviors of model updates under different ensemble settings. Please refer to the second last paragraph in \ref{['sec:probe:landscape']}.
• Figure 3: Visualization of a 2-class example. (a-c) Two-class logits of two single models and/or ensembles. ▲ and $\bullet$ refer to the ground-truth classes, while red and green data points refer to old and new model's logits. Magenta arrow, blue arrow, and gray arrow link negative flip, positive flip, and consistent (either both correct or both wrong) prediction pairs. All dots with black borders depict the same image. (a) The output logits from two single models. We observe frequent occurrences of crossing the decision boundary, leading to either positive or negative flipping. Notably, test samples flip even if not close to the boundary, illustrated by magenta long arrows. (b) The output logits from two 3-model ensembles. We see fewer samples that are far from the boundary flip, illustrated by magenta shorter arrows. (c) The output logits from two 3-model ensembles and their members in the same plot. We observe that individual members' logits, illustrated in lighter circles, center around the mode that is illustrated in darker circles. (d) Estimated probability mass function (PMF) of logit displacement between two single models or ensembles. The $x,y$-axes denote the two classes' logit displacement. The heatmap value denotes the estimated probability density. The ensemble's co-variance is significantly smaller than the single model. The figure is best viewed in color.
• Figure 4: Rank of negative flips' wrong prediction in the old model. The wrongly predicted class of the negative flip samples in the new model also ranks high in the old model, motivating us to focus on those flipping-susceptible classes. The experiments are done on ImageNet-1K and $C=1000$.
• Figure 5: $\ell_2$ norm histogram of logit displacement between two random ensembles. The bin size is $0.5$. We also plot the simulated PMFs: solid lines for $\ell_2$ norm of a simulated normal distribution $\mathcal{N}\left(\Delta{\bm{\mu}}, ({\bm{\Sigma}}_1+{\bm{\Sigma}}_2)\right)$ whose parameters are estimated from all available single models; dashed lines for those of extrapolated distribution $\mathcal{N}\left(\Delta{\bm{\mu}}, \frac{1}{m}{\bm{\Sigma}}'\right)$. Consistency between the histograms and PMFs supports our hypotheses in \ref{['sec:probe:landscape']}. (a) denotes the case of a homogeneous ResNet-18 ensemble and another ResNet-18 homogeneous ensemble. In this case, we have $\Delta{\bm{\mu}}={\bm{0}}, {\bm{\Sigma}}'=2{\bm{\Sigma}}_{1,2}$. (b) denotes the case of a homogeneous ResNet-18 ensemble and a ResNet-50 homogeneous ensemble. In this case, we have $\Delta{\bm{\mu}}\neq{\bm{0}}, {\bm{\Sigma}}'={\bm{\Sigma}}_1 + {\bm{\Sigma}}_2$.