Learning with Fitzpatrick Losses

TL;DR

The paper tackles whether convex losses can be constructed for the same output-link as Fenchel-Young losses. It introduces Fitzpatrick losses, built from the Fitzpatrick function, which yield a tighter bound L_{F[∂Ω]}(y, θ) ≤ L_{Ω ⊕ Ω^*}(y, θ) while preserving the same link ∧ ŷ_Ω(θ) = ∇Ω^*(θ). The authors instantiate two concrete losses, the Fitzpatrick logistic loss and the Fitzpatrick sparsemax loss, and show they equal Fenchel-Young losses with a modified target-dependent generating function Ω_y, establishing a theoretical relationship to generalized Bregman divergences and providing a lower-bound guarantee. Empirically, they validate label-proportion estimation across 11 datasets, finding that Fitzpatrick losses are competitive with their Fenchel-Young counterparts, offering tighter convex losses at a modest computational cost, and opening the door to additional link-consistent losses in practice.

Abstract

Fenchel-Young losses are a family of convex loss functions, encompassing the squared, logistic and sparsemax losses, among others. Each Fenchel-Young loss is implicitly associated with a link function, for mapping model outputs to predictions. For instance, the logistic loss is associated with the soft argmax link function. Can we build new loss functions associated with the same link function as Fenchel-Young losses? In this paper, we introduce Fitzpatrick losses, a new family of convex loss functions based on the Fitzpatrick function. A well-known theoretical tool in maximal monotone operator theory, the Fitzpatrick function naturally leads to a refined Fenchel-Young inequality, making Fitzpatrick losses tighter than Fenchel-Young losses, while maintaining the same link function for prediction. As an example, we introduce the Fitzpatrick logistic loss and the Fitzpatrick sparsemax loss, counterparts of the logistic and the sparsemax losses. This yields two new tighter losses associated with the soft argmax and the sparse argmax, two of the most ubiquitous output layers used in machine learning. We study in details the properties of Fitzpatrick losses and in particular, we show that they can be seen as Fenchel-Young losses using a modified, target-dependent generating function. We demonstrate the effectiveness of Fitzpatrick losses for label proportion estimation.

Figures (2)

• Figure 1: We introduce Fitzpatrick losses, a new family of loss functions generated by a convex regularization function $\Omega$, that lower-bound Fenchel-Young losses generated by the same $\Omega$, while maintaining the same link function $\widehat{y}_\Omega = \nabla \Omega^*$. In particular, we use our framework to instantiate the counterparts of the logistic and sparsemax losses, two instances of Fenchel-Young losses, associated with the soft argmax and the sparse argmax. In the figures above, we plot $L(y, \theta)$, where $y = e_1$, $\theta = (s, 0)$ and $L \in \{L_{F[\partial \Omega]}, L_{\Omega \oplus \Omega^*}\}$, confirming the lower-bound property.
• Figure 2: Geometric interpretation, with $\Omega(y') = \frac{1}{2} \|y'\|^2_2$. The Fenchel-Young loss $L_{\Omega \oplus \Omega^*}(y, \theta)$ is the gap (depicted with a double-headed arrow) between $\Omega(y)$ and $\langle y, \theta \rangle - \Omega^*(\theta)$, the value at $y$ of the tangent with slope $\theta$ and intercept $-\Omega^*(\theta)$. As per Proposition \ref{['prop:fitzpatrick_func_generalized_bregman']}, the Fitzpatrick loss $L_{F[\partial \Omega]}(y, \theta)$ is equal to $L_{\Omega_y \oplus \Omega_y^*}(y, \theta)$ and is therefore equal to the gap between $\Omega_y(y) = \Omega(y)$ and $\langle y, \theta \rangle - \Omega_y^*(\theta)$, the value at $y$ of the tangent with slope $\theta$ and intercept $-\Omega_y^*(\theta)$. Since $\Omega_y(y') = \Omega(y') + D_\Omega(y, y')$, we have that $\Omega_y(y') \ge \Omega(y')$, with equality when $y = y'$. We therefore have $\Omega_y^*(\theta) \le \Omega^*(\theta)$, implying that the Fitzpatrick loss is a lower bound of the Fenchel-Young loss.

Theorems & Definitions (19)

• definition 1
• proposition 1
• proposition 2
• proposition 3
• proposition 4
• proposition 5
• proposition 6
• proposition 7
• proposition 8
• lemma 1