Analysis of Using Sigmoid Loss for Contrastive Learning

TL;DR

The paper analyzes how sigmoid loss shapes the geometry of embeddings in contrastive learning, motivated by the efficient SigLIP variant of CLIP. It introduces the Double-Constant Embedding Model (CCEM) to unify potential embedding structures and proves that the sigmoid-loss optimum lies within CCEM, with the structure transitioning from simplex ETF to antipodal as the temperature increases or decreases. Theoretical results specify the optimal delta-star as a monotone function of the temperature, with explicit thresholds for ETF and antipodal regimes, and synthetic experiments corroborate these phase transitions. The work provides a rigorous geometric understanding of sigmoid-loss embeddings and offers guidance for tuning temperature to achieve ETF-like alignment, aligning with SigLIP’s practical efficiency goals. Overall, CCEM offers a compact, general framework to analyze and predict embedding geometry under sigmoid-based contrastive losses across uni- and multi-modal settings.

Abstract

Contrastive learning has emerged as a prominent branch of self-supervised learning for several years. Especially, CLIP, which applies contrastive learning to large sets of captioned images, has garnered significant attention. Recently, SigLIP, a variant of CLIP, has been proposed, which uses the sigmoid loss instead of the standard InfoNCE loss. SigLIP achieves the performance comparable to CLIP in a more efficient manner by eliminating the need for a global view. However, theoretical understanding of using the sigmoid loss in contrastive learning is underexplored. In this paper, we provide a theoretical analysis of using the sigmoid loss in contrastive learning, in the perspective of the geometric structure of learned embeddings. First, we propose the double-Constant Embedding Model (CCEM), a framework for parameterizing various well-known embedding structures by a single variable. Interestingly, the proposed CCEM is proven to contain the optimal embedding with respect to the sigmoid loss. Second, we mathematically analyze the optimal embedding minimizing the sigmoid loss for contrastive learning. The optimal embedding ranges from simplex equiangular-tight-frame to antipodal structure, depending on the temperature parameter used in the sigmoid loss. Third, our experimental results on synthetic datasets coincide with the theoretical results on the optimal embedding structures.

Figures (3)

• Figure 1: The proposed double-constant embedding model in Def. \ref{['def:model']} for different $\delta$ values, when we have $N=3$ embedding vector pairs $\{({\bm{u}}_i^{\delta}, {\bm{v}}_i^{\delta})\}_{i=1}^N$ in $d=3$ dimensional space. (1) $\delta=0$ (simplex ETF), the angle between positive pair (${\bm{u}}_i^{0}$ and ${\bm{v}}_i^{0}$) is 0. (2) $\delta=1$, the angle between ${\bm{u}}_i^{1}$ and ${\bm{v}}_i^{1}$ is $\frac{\pi}{2}$. (3) $\delta = \infty$, the angle between ${\bm{u}}_i^{\infty}$ and ${\bm{v}}_i^{\infty}$ is $\pi$, forming 'antipodal' embedding.
• Figure 2: The normalized similarity $s$ of positive pairs measured for the embeddings trained by the sigmoid loss $\mathcal{L}^{\operatorname{sig}}$, for various $N$ and $t$. We test for two cases: $d=N$ (upper) and $d=\frac{N}{2}$ (below). The blue dashed line satisfies $t=\frac{N-1}{N}\log(N-3)$, and the red dashed line satisfies $t=\frac{1}{2}\log\frac{N-2}{2}$, which are the threshold values obtained in Theorem \ref{['thm:logsig']}. This plot shows that our theoretical results coincide with empirical observations.
• Figure 3: The normalized similarity $s = \frac{1}{2} (1 + \frac{1}{N}\sum_{i=1}^N{\bm{u}}_i^\top{\bm{v}}_i)$ of positive pairs measured for the embeddings trained by sigmoid loss $\mathcal{L}^{\operatorname{sig}}$, for various $N$ and $t$ when $d=N$. Unlike the setting for Fig \ref{['fig:syn_result']}, we train a encoder (two-layer fully-connected ReLU network) which outputs embeddings, rather than directly optimizing the embedding vectors.

Theorems & Definitions (31)

• Definition 1
• Corollary 1: Theorem 1 of lu2022neural
• Definition 2
• Remark 1
• Definition 3: Double-Constant Embedding Model
• Proposition 1
• Example 1
• Theorem 1
• Corollary 2
• proof