Rethinking Evaluation Metric for Probability Estimation Models Using Esports Data

TL;DR

This work addresses evaluating probability estimates when the true probability $p$ is unknown, with a focus on esports win-probability estimation where game-state dynamics create diverse operating conditions. It analyzes $\mathrm{Brier\ score}$ and $\mathrm{ECE}$ as calibration metrics and introduces Balance score, a binning-free, gain/loss scoring rule with key properties such as $G(p)=0$ when $\hat{p}=p$ and $|g(q;p)|=|q-p|$, linking calibration to predictive accuracy without binning. Through simulations under $\mathrm{Beta}$ distributions and real League of Legends data, Balance score consistently reflects calibration across varying distributions and data sizes, while $ECE$ suffers from binning bias and discrimination-based metrics can mislead. The Balance score offers a practical, hyperparameter-free calibration metric with potential applicability beyond esports to general probability-estimation tasks and calibration-based learning workflows.

Abstract

Probability estimation models play an important role in various fields, such as weather forecasting, recommendation systems, and sports analysis. Among several models estimating probabilities, it is difficult to evaluate which model gives reliable probabilities since the ground-truth probabilities are not available. The win probability estimation model for esports, which calculates the win probability under a certain game state, is also one of the fields being actively studied in probability estimation. However, most of the previous works evaluated their models using accuracy, a metric that only can measure the performance of discrimination. In this work, we firstly investigate the Brier score and the Expected Calibration Error (ECE) as a replacement of accuracy used as a performance evaluation metric for win probability estimation models in esports field. Based on the analysis, we propose a novel metric called Balance score which is a simple yet effective metric in terms of six good properties that probability estimation metric should have. Under the general condition, we also found that the Balance score can be an effective approximation of the true expected calibration error which has been imperfectly approximated by ECE using the binning technique. Extensive evaluations using simulation studies and real game snapshot data demonstrate the promising potential to adopt the proposed metric not only for the win probability estimation model for esports but also for evaluating general probability estimation models.

Figures (4)

• Figure 1: Problem of using accuracy as a measure of probability estimation models in esports. Compare to the image classification task which only ask for the label, even the optimal model may not be able to estimate the result of the match due to the uncertainty of the game snapshot data itself.
• Figure 2: Three plots in left side respectively refers to the plot of $p$ obtained from the generated beta distributions $Beta(2, 2)$, $Beta(1, 1)$, and $Beta(0.5, 0.5)$. Distributions of $\hat{p}$ derived by the logistic regression model trained with the real game snapshot datasets at 5, 10, and 15 minutes are also plotted on the right side.
• Figure 3: $ECE$ values with increasing binning number $M$ and the absolute Balance score of two overconfident models. Red lines with circle markers and asterisk markers respectively refers to the $ECE$ of the model with 0.1 and 0.11 tendency. Blue dashed line and dash-dotted line respectively refers to the absolute Balance score of the model with 0.1 and 0.11 tendency.
• Figure 4: $ECE$ and the absolute Balance score of overconfident model with 0.1 tendency over the increasing data size. The red line with asterisk markers and the blue line with circle markers respectively refers to $ECE$ and Balance score while the green dashed line denotes the analytic true ECE value of the evaluated model.