Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Confidence-Aware Multi-Field Model Calibration

Yuang Zhao, Chuhan Wu, Qinglin Jia, Hong Zhu, Jia Yan, Libin Zong, Linxuan Zhang, Zhenhua Dong, Muyu Zhang

TL;DR

This work tackles miscalibration of user-action probabilities in online advertising by introducing ConfCalib, a confidence-aware, multi-field post-hoc calibration framework. It leverages binomial Wilson confidence intervals to adapt calibration intensity, and jointly calibrates across multiple feature fields via a weighted geometric fusion to mitigate data sparsity. The approach is network-free, easy to deploy online, and demonstrates superior calibration (lower Field-RCE/MF-RCE, ECE, MVCE) and maintained or improved ranking (AUC) across public and industrial datasets, with strong online gains in A/B tests. The method shows robustness to data sparsity and practical impact for advertising platforms, including Huawei’s ecosystem.

Abstract

Accurately predicting the probabilities of user feedback, such as clicks and conversions, is critical for advertisement ranking and bidding. However, there often exist unwanted mismatches between predicted probabilities and true likelihoods due to the rapid shift of data distributions and intrinsic model biases. Calibration aims to address this issue by post-processing model predictions, and field-aware calibration can adjust model output on different feature field values to satisfy fine-grained advertising demands. Unfortunately, the observed samples corresponding to certain field values can be seriously limited to make confident calibrations, which may yield bias amplification and online disturbance. In this paper, we propose a confidence-aware multi-field calibration method, which adaptively adjusts the calibration intensity based on confidence levels derived from sample statistics. It also utilizes multiple fields for joint model calibration according to their importance to mitigate the impact of data sparsity on a single field. Extensive offline and online experiments show the superiority of our method in boosting advertising performance and reducing prediction deviations.

Confidence-Aware Multi-Field Model Calibration

TL;DR

This work tackles miscalibration of user-action probabilities in online advertising by introducing ConfCalib, a confidence-aware, multi-field post-hoc calibration framework. It leverages binomial Wilson confidence intervals to adapt calibration intensity, and jointly calibrates across multiple feature fields via a weighted geometric fusion to mitigate data sparsity. The approach is network-free, easy to deploy online, and demonstrates superior calibration (lower Field-RCE/MF-RCE, ECE, MVCE) and maintained or improved ranking (AUC) across public and industrial datasets, with strong online gains in A/B tests. The method shows robustness to data sparsity and practical impact for advertising platforms, including Huawei’s ecosystem.

Abstract

Accurately predicting the probabilities of user feedback, such as clicks and conversions, is critical for advertisement ranking and bidding. However, there often exist unwanted mismatches between predicted probabilities and true likelihoods due to the rapid shift of data distributions and intrinsic model biases. Calibration aims to address this issue by post-processing model predictions, and field-aware calibration can adjust model output on different feature field values to satisfy fine-grained advertising demands. Unfortunately, the observed samples corresponding to certain field values can be seriously limited to make confident calibrations, which may yield bias amplification and online disturbance. In this paper, we propose a confidence-aware multi-field calibration method, which adaptively adjusts the calibration intensity based on confidence levels derived from sample statistics. It also utilizes multiple fields for joint model calibration according to their importance to mitigate the impact of data sparsity on a single field. Extensive offline and online experiments show the superiority of our method in boosting advertising performance and reducing prediction deviations.
Paper Structure (22 sections, 7 equations, 8 figures, 3 tables)

This paper contains 22 sections, 7 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Definition
  5. Confidence-aware Calibration
  6. Multi-field Joint Calibration
  7. Online Deployment
  8. Experiments
  9. Experimental Setup
  10. Settings
  11. Datasets
  12. Baselines
  13. Evaluation Metrics
  14. Results
  15. Main Results
  16. ...and 7 more sections

Figures (8)

  • Figure 1: An example of multi-field model calibration.
  • Figure 2: The overall framework of our ConfCalib method.
  • Figure 3: The curves of calibrated scores w.r.t. different predicted scores under different numbers of observed samples.
  • Figure 4: Online deployment diagram of ConfCalib.
  • Figure 5: Results of the ablation study on several variants of ConfCalib.
  • ...and 3 more figures