Unconstrained Monotonic Calibration of Predictions in Deep Ranking Systems

TL;DR

The paper tackles the calibration gap in deep ranking systems by relaxing calibrator constraints and employing a monotonic neural approach. It introduces Unconstrained Monotonic Calibration (UMC), built on Unconstrained Monotonic Neural Networks (UMNN) and enhanced by a Smooth Calibration Loss (SCLoss) to guide learning toward ideal calibration. Empirical results on Avazu, AliCCP, and a large-scale online deployment in Kuaishou demonstrate superior calibration performance with minimal impact on ranking metrics and measurable business gains. The work provides practical calibration tools and reports open-source code for reproducibility and industry adoption.

Abstract

Ranking models primarily focus on modeling the relative order of predictions while often neglecting the significance of the accuracy of their absolute values. However, accurate absolute values are essential for certain downstream tasks, necessitating the calibration of the original predictions. To address this, existing calibration approaches typically employ predefined transformation functions with order-preserving properties to adjust the original predictions. Unfortunately, these functions often adhere to fixed forms, such as piece-wise linear functions, which exhibit limited expressiveness and flexibility, thereby constraining their effectiveness in complex calibration scenarios. To mitigate this issue, we propose implementing a calibrator using an Unconstrained Monotonic Neural Network (UMNN), which can learn arbitrary monotonic functions with great modeling power. This approach significantly relaxes the constraints on the calibrator, improving its flexibility and expressiveness while avoiding excessively distorting the original predictions by requiring monotonicity. Furthermore, to optimize this highly flexible network for calibration, we introduce a novel additional loss function termed Smooth Calibration Loss (SCLoss), which aims to fulfill a necessary condition for achieving the ideal calibration state. Extensive offline experiments confirm the effectiveness of our method in achieving superior calibration performance. Moreover, deployment in Kuaishou's large-scale online video ranking system demonstrates that the method's calibration improvements translate into enhanced business metrics. The source code is available at https://github.com/baiyimeng/UMC.

Figures (6)

• Figure 1: The core idea of the proposed monotonic calibrator. It solely imposes a conditional monotonicity constraint on the calibrator, enhancing the expressiveness and flexibility while avoiding excessive distortion of original predictions. Specifically, for samples $A$ and $B$, when conditioning on sample features, if the original scores satisfy $s_A>s_B$, then the calibrated scores will also satisfy $s_A^\prime>s_B^\prime$.
• Figure 2: An overview of the proposed UMC framework, which comprises the monotonic calibration architecture (with UMNN module as the core) and the calibration-aware learning strategy (with SCLoss as the core). The architecture takes the original predictions $s$ and sample features $\bm{x}$ as inputs, enforces monotonicity using the positive derivatives of UMNN, and outputs the final calibrated scores $s^\prime$ through rescaling operations. The learning strategy integrates BCELoss and SCLoss for model optimization, with SCLoss specifically designed to emphasize calibration.
• Figure 3: Results of adding SCLoss to other neural network-based baseline methods on Avazu.
• Figure 4: Results of the performance of UMC across different values of group number $N$ on Avazu.
• Figure 5: Results of the performance of UMC across different values of decay rate $\tau$ on Avazu.