C2AL: Cohort-Contrastive Auxiliary Learning for Large-scale Recommendation Systems

TL;DR

The paper tackles representation bias in large-scale recommender systems caused by heterogeneous user cohorts under a single global objective. It introduces Cohort-Contrastive Auxiliary Learning (C2AL), a two-stage framework that first identifies highly contrasting head and tail cohorts and then trains auxiliary losses tied to those cohorts, regularizing the shared encoder without affecting inference. Through gradient analysis, C2AL is shown to reshape the FM-based attention layer to produce a denser, more diverse set of feature interactions, improving generalization across cohorts. Empirically, C2AL yields consistent Normalized Entropy reductions across six production models and delivers meaningful gains for minority cohorts, demonstrating practical impact for production-scale ads systems.

Abstract

Training large-scale recommendation models under a single global objective implicitly assumes homogeneity across user populations. However, real-world data are composites of heterogeneous cohorts with distinct conditional distributions. As models increase in scale and complexity and as more data is used for training, they become dominated by central distribution patterns, neglecting head and tail regions. This imbalance limits the model's learning ability and can result in inactive attention weights or dead neurons. In this paper, we reveal how the attention mechanism can play a key role in factorization machines for shared embedding selection, and propose to address this challenge by analyzing the substructures in the dataset and exposing those with strong distributional contrast through auxiliary learning. Unlike previous research, which heuristically applies weighted labels or multi-task heads to mitigate such biases, we leverage partially conflicting auxiliary labels to regularize the shared representation. This approach customizes the learning process of attention layers to preserve mutual information with minority cohorts while improving global performance. We evaluated C2AL on massive production datasets with billions of data points each for six SOTA models. Experiments show that the factorization machine is able to capture fine-grained user-ad interactions using the proposed method, achieving up to a 0.16% reduction in normalized entropy overall and delivering gains exceeding 0.30% on targeted minority cohorts.

Figures (6)

• Figure 1: The Deep and Hierarchical Ensemble Network (DHEN) and its internal interaction layer.
• Figure 2: Comparison of attention weight sparsity between two representative production models. The baseline model demonstrates pronounced sparsity in attention weights, indicating that feature selection is primarily driven by majority cohorts. Consequently, features relevant to minority cohorts are frequently underrepresented or ignored.
• Figure 3: Probability density functions of the baseline Model A's (optimize for click objective on Instagram) predictions for the head and tail cohorts along a selected semantic axis, illustrating distributional divergence.
• Figure 4: Weight distributions across various network layers. Left: two layers preceding the attention MLP. Right: attention weights (top) and post-attention weights (bottom).
• Figure 5: Evolution of Attention Weights Throughout Training