Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MLPlatt: Simple Calibration Framework for Ranking Models

Piotr Bajger, Roman Dusek, Krzysztof Galias, Paweł Młyniec, Aleksander Wawer, Paweł Zawistowski

TL;DR

This work tackles calibrating ranking model outputs into interpretable CTR probabilities without degrading ranking quality. It proposes MLPlatt, a two-stage, context-aware calibration framework that attaches a monotone MLP head to a context-embedded representation on top of a backbone ranker, enforcing monotonicity in the ranker score. Through evaluations on Allegro and AliExpress, MLPlatt achieves state-of-the-art field-aware calibration (low F-ECE) while maintaining or improving NDCG and AUC, outperforming Platt, Isotonic, ConfCalib, DESC, and RCR. The method is lightweight, plug‑and‑play, and enables unified signaling across ranking, engagement, and revenue in production pipelines by producing calibrated CTR probabilities without sacrificing ranking performance.

Abstract

Ranking models are extensively used in e-commerce for relevance estimation. These models often suffer from poor interpretability and no scale calibration, particularly when trained with typical ranking loss functions. This paper addresses the problem of post-hoc calibration of ranking models. We introduce MLPlatt: a simple yet effective ranking model calibration method that preserves the item ordering and converts ranker outputs to interpretable click-through rate (CTR) probabilities usable in downstream tasks. The method is context-aware by design and achieves good calibration metrics globally, and within strata corresponding to different values of a selected categorical field (such as user country or device), which is often important from a business perspective of an E-commerce platform. We demonstrate the superiority of MLPlatt over existing approaches on two datasets, achieving an improvement of over 10\% in F-ECE (Field Expected Calibration Error) compared to other methods. Most importantly, we show that high-quality calibration can be achieved without compromising the ranking quality.

MLPlatt: Simple Calibration Framework for Ranking Models

TL;DR

This work tackles calibrating ranking model outputs into interpretable CTR probabilities without degrading ranking quality. It proposes MLPlatt, a two-stage, context-aware calibration framework that attaches a monotone MLP head to a context-embedded representation on top of a backbone ranker, enforcing monotonicity in the ranker score. Through evaluations on Allegro and AliExpress, MLPlatt achieves state-of-the-art field-aware calibration (low F-ECE) while maintaining or improving NDCG and AUC, outperforming Platt, Isotonic, ConfCalib, DESC, and RCR. The method is lightweight, plug‑and‑play, and enables unified signaling across ranking, engagement, and revenue in production pipelines by producing calibrated CTR probabilities without sacrificing ranking performance.

Abstract

Ranking models are extensively used in e-commerce for relevance estimation. These models often suffer from poor interpretability and no scale calibration, particularly when trained with typical ranking loss functions. This paper addresses the problem of post-hoc calibration of ranking models. We introduce MLPlatt: a simple yet effective ranking model calibration method that preserves the item ordering and converts ranker outputs to interpretable click-through rate (CTR) probabilities usable in downstream tasks. The method is context-aware by design and achieves good calibration metrics globally, and within strata corresponding to different values of a selected categorical field (such as user country or device), which is often important from a business perspective of an E-commerce platform. We demonstrate the superiority of MLPlatt over existing approaches on two datasets, achieving an improvement of over 10\% in F-ECE (Field Expected Calibration Error) compared to other methods. Most importantly, we show that high-quality calibration can be achieved without compromising the ranking quality.
Paper Structure (23 sections, 4 equations, 2 figures, 6 tables)

This paper contains 23 sections, 4 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Ranking and CTR models
  3. Motivation
  4. Paper Contribution
  5. Related work
  6. Methods
  7. MLPlatt
  8. Model architecture
  9. Training procedure
  10. Integrating MLPlatt with existing pipelines
  11. Experiments
  12. Datasets
  13. Allegro
  14. AliExpress
  15. Metrics
  16. ...and 8 more sections

Figures (2)

  • Figure 1: High-level overview of the calibrated ranker model. Item ($x_{item}$) and context ($x_{ctx}$) features are passed through the ranking model to produce an uncalibrated score $r$. This score is then passed together with the context features through the MLPlatt model to output a calibrated score $c$.
  • Figure 2: MLPlatt model structure. The context vector $x_{ctx}$ is processed through the context model to produce an embedding. The embedding is concatenated with the ranker score, which is then passed to the MonoMLP model.