Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ACCon: Angle-Compensated Contrastive Regularizer for Deep Regression

Botao Zhao, Xiaoyang Qu, Zuheng Kang, Junqing Peng, Jing Xiao, Jianzong Wang

TL;DR

This work addresses the challenge of learning distance-aware representations in deep regression by introducing ACCon, an angle-compensated contrastive regularizer that encodes label-distance information into the embedding space. Built on supervised contrastive learning, ACCon adjusts negative similarities through an angle that reflects label differences, guiding representations toward a semi-hyperspherical arrangement. The authors provide theoretical bounds showing convergence of learned angles to the label-distance target and demonstrate superior data efficiency and robustness to imbalance across age estimation and semantic textual similarity tasks. The approach is plug-and-play with existing contrastive frameworks, achieving state-of-the-art or competitive performance while requiring fewer labeled examples in several settings, indicating strong practical impact for regression under data scarcity and long-tail regimes.

Abstract

In deep regression, capturing the relationship among continuous labels in feature space is a fundamental challenge that has attracted increasing interest. Addressing this issue can prevent models from converging to suboptimal solutions across various regression tasks, leading to improved performance, especially for imbalanced regression and under limited sample sizes. However, existing approaches often rely on order-aware representation learning or distance-based weighting. In this paper, we hypothesize a linear negative correlation between label distances and representation similarities in regression tasks. To implement this, we propose an angle-compensated contrastive regularizer for deep regression, which adjusts the cosine distance between anchor and negative samples within the contrastive learning framework. Our method offers a plug-and-play compatible solution that extends most existing contrastive learning methods for regression tasks. Extensive experiments and theoretical analysis demonstrate that our proposed angle-compensated contrastive regularizer not only achieves competitive regression performance but also excels in data efficiency and effectiveness on imbalanced datasets.

ACCon: Angle-Compensated Contrastive Regularizer for Deep Regression

TL;DR

This work addresses the challenge of learning distance-aware representations in deep regression by introducing ACCon, an angle-compensated contrastive regularizer that encodes label-distance information into the embedding space. Built on supervised contrastive learning, ACCon adjusts negative similarities through an angle that reflects label differences, guiding representations toward a semi-hyperspherical arrangement. The authors provide theoretical bounds showing convergence of learned angles to the label-distance target and demonstrate superior data efficiency and robustness to imbalance across age estimation and semantic textual similarity tasks. The approach is plug-and-play with existing contrastive frameworks, achieving state-of-the-art or competitive performance while requiring fewer labeled examples in several settings, indicating strong practical impact for regression under data scarcity and long-tail regimes.

Abstract

In deep regression, capturing the relationship among continuous labels in feature space is a fundamental challenge that has attracted increasing interest. Addressing this issue can prevent models from converging to suboptimal solutions across various regression tasks, leading to improved performance, especially for imbalanced regression and under limited sample sizes. However, existing approaches often rely on order-aware representation learning or distance-based weighting. In this paper, we hypothesize a linear negative correlation between label distances and representation similarities in regression tasks. To implement this, we propose an angle-compensated contrastive regularizer for deep regression, which adjusts the cosine distance between anchor and negative samples within the contrastive learning framework. Our method offers a plug-and-play compatible solution that extends most existing contrastive learning methods for regression tasks. Extensive experiments and theoretical analysis demonstrate that our proposed angle-compensated contrastive regularizer not only achieves competitive regression performance but also excels in data efficiency and effectiveness on imbalanced datasets.
Paper Structure (22 sections, 1 theorem, 8 equations, 5 figures, 2 tables, 1 algorithm)

This paper contains 22 sections, 1 theorem, 8 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deep Regression
  4. Contrastive Learning
  5. Methodology
  6. Problem Setting
  7. Angle-Compensated Supervised Contrastive Loss
  8. Theoretical Insights
  9. Deep Regression Framework
  10. Experiments and Results
  11. Experiments Setup
  12. Datasets:
  13. Metrics:
  14. Baselines
  15. Basic Performance
  16. ...and 7 more sections

Key Result

Theorem 1

. $\mathrm{L}^{*} := \frac{1}{4N^{2}} \sum_{i=0}^{2N} \sum\limits_{m \in \mathcal{N}(i)} \cos\left(\tilde{\theta}_{i,m}\right)/\tau - \frac{\mathrm{log N / \tau}}{2N}$ is a lower bound of $\mathcal{L}_{\mathrm{ACCon}}$, i.e, $\mathcal{L}_{\mathrm{ACCon}} > \mathrm{L}^{*}$.

Figures (5)

  • Figure 1: The subfigure (A) illustrates two types of representation learning for regression: order-aware and distance-aware. The subfigure (B) depicts the supervised contrastive learning for gender classification, where all samples from the same class are treated as positives and contrasted against negatives. The subfigure (C) presents the proposed angle-compensated supervised contrastive learning approach for age regression, which projects the input onto a semi-hypersphere while preserving the label relationship information.
  • Figure 2: The frameworks of the deep regression with angle-compensated supervised contrastive.
  • Figure 3: The visualization and quantitation analysis of feature representations. The subfigure (a) is the t-SNE visualization of feature space on the AgeDB-natural test dataset. The subfigure (b) depicts the joint distribution of $\cos\left(\theta_{i,j}\right)$ and the distance of labels $\left|y_{i}-y_{j}\right|/100$ on the AgeDB-natural test dataset using kernel density estimation.
  • Figure 4: The performance comparison on AgeDB-Natural, when reducing the training dataset.
  • Figure 5: The cosine similarity changes as the label distance varies between the anchor and contrast samples in the AgeDB-Natural test dataset. The stars represent the anchors, while the plot depicts the mean cosine similarity between each anchor and all its contrastive samples. The shaded region indicates the standard deviation of these cosine similarities.

Theorems & Definitions (1)

  • Theorem 1: Lower bound of $\mathcal{L}_{\mathrm{ACCon}}$