Geometric Insights into Focal Loss: Reducing Curvature for Enhanced Model Calibration

TL;DR

The paper addresses the calibration problem in neural networks by analyzing focal loss through a geometric lens. It shows that focal loss effectively reduces loss-surface curvature and frames this as an entropy-constrained optimization problem linked to a Maxwell–Boltzmann posterior, with support from a PAC-Bayes perspective. Empirical results reveal that curvature measures such as the Hessian's maximum eigenvalue and trace decline with higher focal gamma and correlate with improved calibration (lower ECE), and that explicit Hessian-trace regularization further enhances calibration. Overall, the work suggests curvature control as a practical, general mechanism for achieving well-calibrated predictions and motivates curvature-aware design of calibration techniques.

Abstract

The key factor in implementing machine learning algorithms in decision-making situations is not only the accuracy of the model but also its confidence level. The confidence level of a model in a classification problem is often given by the output vector of a softmax function for convenience. However, these values are known to deviate significantly from the actual expected model confidence. This problem is called model calibration and has been studied extensively. One of the simplest techniques to tackle this task is focal loss, a generalization of cross-entropy by introducing one positive parameter. Although many related studies exist because of the simplicity of the idea and its formalization, the theoretical analysis of its behavior is still insufficient. In this study, our objective is to understand the behavior of focal loss by reinterpreting this function geometrically. Our analysis suggests that focal loss reduces the curvature of the loss surface in training the model. This indicates that curvature may be one of the essential factors in achieving model calibration. We design numerical experiments to support this conjecture to reveal the behavior of focal loss and the relationship between calibration performance and curvature.

Figures (4)

• Figure 1: Changes in loss and gradient with respect to model prediction probability $p$ for different values of the hyperparameter $\gamma$ in focal loss. The vertical axes represent loss and gradient, respectively, and the horizontal axis represents model prediction probability $p$. Focal loss coincides with cross-entropy when $\gamma = 1$. For $0 <\gamma < 1$ , as shown in the right figure, the gradient does not converge (having an effect opposite to the original intention). When $\gamma \geq 1$, the gradient and loss for well-classified samples ($p$ close to 1) are smaller than those for cross-entropy.
• Figure 2: Changes in $\lambda_\text{max}(H_\text{val})$ and expected calibration error (ECE) with respect to hyperparameter $\gamma$ in focal loss training on CIFAR100 using different model architectures. For all architectures, $\lambda_\text{max}(H_\text{val})$ monotonically decreases with increasing $\gamma$, consistent with \ref{['thm:sharpness_focal_loss']}. ECE reaches its minimum for ViT architecture around $\gamma$ = 10 and for other architectures around $\gamma = 3,4$ . These results confirm the importance of curvature regularization through appropriate focal loss for achieving low ECE.
• Figure 3: Relationship between $\text{Tr}(H_\text{val})$ and expected calibration error (ECE) in focal loss training on CIFAR100 using various DNN architectures. Data points of the same color represent different values of $\gamma$. ECE reaches a minimum peak for all network architectures when $\text{Tr}(H_\text{val})$ is reduced to a certain extent rather than maximizing it. When $\text{Tr}(H_\text{val})$ is too low, training converges to a point significantly different from the convergence point in cross-entropy, resulting in ECE degradation. This indicates that applying appropriate regularization to reduce $\text{Tr}(H_\text{val})$ to a certain extent is crucial for minimizing ECE.
• Figure 4: This figure presents the results of training a 2-layer MLP with hidden-size 100 on CIFAR10 using a loss function that explicitly incorporates $\text{Tr}(H_\text{val})$ regularization. Each data point corresponds to a different learning rate and $\tau$ value. Higher $\tau$ values resulted in lower ECE and smaller $\text{Tr}(H_\text{val})$, while lower $\tau$ values led to higher ECE and $\text{Tr}(H_\text{val})$.

Theorems & Definitions (12)

• Definition 1: Expected Calibration Error naeini2015obtaining
• Definition 2: Focal Loss lin2017focal
• Lemma 1
• proof
• Theorem 1
• proof
• Corollary 1
• Corollary 2
• Theorem 2
• proof