Comparing Contrastive and Triplet Loss: Variance Analysis and Optimization Behavior
Donghuo Zeng
TL;DR
The paper compares contrastive loss $\mathcal{L}_{\text{con}}$ and triplet loss $\mathcal{L}_{\text{tri}}$ as margin-based objectives in deep metric learning, focusing on embedding variance and optimization dynamics. It analyzes intra-class variance $\sigma_{\mathrm{intra}}^2$ and inter-class variance $\sigma_{\mathrm{inter}}^2$ and introduces greediness metrics such as loss-decay rate, active ratio, and gradient norm to quantify updates. Empirically, triplet loss preserves greater intra-class variance and achieves stronger, targeted updates, yielding superior retrieval across MNIST, CIFAR-10, CUB-200, and CARS196, while contrastive loss converges faster but tends to compact intra-class structure and underperform in retrieval. The findings suggest using triplet loss for detail retention and hard-sample emphasis, contrastive loss for smoother, broad-based embedding refinement, and motivate future work on hybrid/adaptive-margin strategies.
Abstract
Contrastive loss and triplet loss are widely used objectives in deep metric learning, yet their effects on representation quality remain insufficiently understood. We present a theoretical and empirical comparison of these losses, focusing on intra- and inter-class variance and optimization behavior (e.g., greedy updates). Through task-specific experiments with consistent settings on synthetic data and real datasets-MNIST, CIFAR-10-it is shown that triplet loss preserves greater variance within and across classes, supporting finer-grained distinctions in the learned representations. In contrast, contrastive loss tends to compact intra-class embeddings, which may obscure subtle semantic differences. To better understand their optimization dynamics, By examining loss-decay rate, active ratio, and gradient norm, we find that contrastive loss drives many small updates early on, while triplet loss produces fewer but stronger updates that sustain learning on hard examples. Finally, across both classification and retrieval tasks on MNIST, CIFAR-10, CUB-200, and CARS196 datasets, our results consistently show that triplet loss yields superior performance, which suggests using triplet loss for detail retention and hard-sample focus, and contrastive loss for smoother, broad-based embedding refinement.