Adaptive Discriminative Regularization for Visual Classification

TL;DR

The paper tackles the challenge that real-world visual data violate the i.i.d. assumption, with semantically similar classes and high intra-class variance hindering standard cross-entropy optimization. It introduces Adaptive Discriminative Regularization (ADR), which calibrates each sample's top likelihood using semantically neighboring classes and imposes an adaptive exponential penalty whose gradient scales with prediction uncertainty, enabling stronger learning early in training and stabilization as predictions become confident. ADR is formulated in a general way with a simple, implementable solution that adds the term ${\cal L}_d$ to the usual cross-entropy loss as ${\cal L} = {\cal L}_{ce} + \gamma {\cal L}_d$, using TopK-based sufficient statistics and an entropy-based uncertainty measure ${\phi}$ to drive the adaptive gradients. Empirically, ADR yields consistent, non-trivial improvements across large-scale and fine-grained image classification (ImageNet-1K, Flowers-102, CIFAR-10), face verification, FER, action recognition, and unsupervised segmentation, while showing robustness to noisy labels and long-tailed distributions and compatibility with CNN, transformer, and MLP backbones. The approach offers a practical, broadly applicable regularization technique that complements existing discriminative objectives and can enhance real-world visual recognition systems.

Abstract

How to improve discriminative feature learning is central in classification. Existing works address this problem by explicitly increasing inter-class separability and intra-class similarity, whether by constructing positive and negative pairs for contrastive learning or posing tighter class separating margins. These methods do not exploit the similarity between different classes as they adhere to i.i.d. assumption in data. In this paper, we embrace the real-world data distribution setting that some classes share semantic overlaps due to their similar appearances or concepts. Regarding this hypothesis, we propose a novel regularization to improve discriminative learning. We first calibrate the estimated highest likelihood of one sample based on its semantically neighboring classes, then encourage the overall likelihood predictions to be deterministic by imposing an adaptive exponential penalty. As the gradient of the proposed method is roughly proportional to the uncertainty of the predicted likelihoods, we name it adaptive discriminative regularization (ADR), trained along with a standard cross entropy loss in classification. Extensive experiments demonstrate that it can yield consistent and non-trivial performance improvements in a variety of visual classification tasks (over 10 benchmarks). Furthermore, we find it is robust to long-tailed and noisy label data distribution. Its flexible design enables its compatibility with mainstream classification architectures and losses.

Figures (8)

• Figure 1: Different predicted logit distributions are plotted in (a). Different data distributions are drawn in (b) and (c). When collecting data in a wild, one often expects the distance between clusters to be equal i.e., (c) an ideal category distribution. But the most cases, we collected the dataset can be represented as (b), in which the clusters vary in variance and inter-class distance (various sizes of circles indicate different variances). The hard samples at the edge of the clusters contribute significantly to the decision surface (see Forgetting), while they are more likely to make ambiguous predictions as the blue curve drawn in (a).
• Figure 2: The conceptual illustration of how training a supervised model with our ADR in a toy experiment (details see Sec. \ref{['Discussion']}). Minibatch images are fed into the backbone and softmax to obtain the predicted logits ${\tilde{\cal Y}}$. At different training phases, we capture the predicted logits, normalize them according to their confidence, and plot the change curve of ADR (the gray shaded area indicates the possible range of Top-1 logits). The cross-entropy loss pushes the optimization direction (the blue arrow) to the positive side at the beginning, while our ADR enlarges the separability between classes and makes the optimization fall to one side further (the green arrow).
• Figure 3: Visualization of classifier layer's features. The first row comes from a training set, the second row is from a validation set. To be convincing, we plotted the entire sample. The first five columns represent the features extracted from the model which is trained by CE loss, CE w/ ADR, CE w/ LS, CE w/ LS w/ ADR, and CE w/ Entropy loss respectively. The Expected Calibration Error (ECE) on the CIFAR-10 validation set is reported (below) too. Lower is better.
• Figure 4: Visualization of the function curves. The discriminative function is shown below and its derivative function is shown above. (a) Variance-base functions. (b) Entropy-base functions (c) and (d) Motivation of our introduced ADR: (c) exponential-variance based functions, (d) exponential-entropy based functions in Eq. \ref{['entropy_phi']}.
• Figure 5: The classification performance of ADR and the entropy loss, varying $\gamma,\lambda$ for ADR (w. optimal $\tau$) and the entropy loss respectively.