Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Kelly Betting as Bayesian Model Evaluation: A Framework for Time-Updating Probabilistic Forecasts

Michael Beuoy

TL;DR

The paper introduces a real-time forecast evaluation framework built on the Kelly betting criterion, treating each model as a bettor and using bankroll dynamics as a proxy for Bayesian credibility. By computing market-clearing odds and updating bets as forecasts evolve, the method yields time-sensitive credibility measures without awaiting final outcomes. Through binary and multinomial extensions and extensive simulations, the authors show that this approach can more effectively distinguish correct from incorrect models in many scenarios and that iteration further enhances discriminative power. The framework offers a practical, interpretable, and theoretically grounded alternative to traditional log-loss and Brier scoring for time-updating probabilistic forecasts across domains such as sports analytics and prediction markets.

Abstract

This paper proposes a new way of evaluating the accuracy and validity of probabilistic forecasts that change over time (such as an in-game win probability model, or an election forecast). Under this approach, each model to be evaluated is treated as a canonical Kelly bettor, and the models are pitted against each other in an iterative betting contest. The growth or decline of each model's bankroll serves as the evaluation metric. Under this approach, market consensus probabilities and implied model credibilities can be updated real time as each model updates, and do not require one to wait for the final outcome. Using a simulation model, it will be shown that this method is in general more accurate than traditional average log-loss and Brier score methods at distinguishing a correct model from an incorrect model. This Kelly approach is shown to have a direct mathematical and conceptual analogue to Bayesian inference, with bankroll serving as a proxy for Bayesian credibility.

Kelly Betting as Bayesian Model Evaluation: A Framework for Time-Updating Probabilistic Forecasts

TL;DR

The paper introduces a real-time forecast evaluation framework built on the Kelly betting criterion, treating each model as a bettor and using bankroll dynamics as a proxy for Bayesian credibility. By computing market-clearing odds and updating bets as forecasts evolve, the method yields time-sensitive credibility measures without awaiting final outcomes. Through binary and multinomial extensions and extensive simulations, the authors show that this approach can more effectively distinguish correct from incorrect models in many scenarios and that iteration further enhances discriminative power. The framework offers a practical, interpretable, and theoretically grounded alternative to traditional log-loss and Brier scoring for time-updating probabilistic forecasts across domains such as sports analytics and prediction markets.

Abstract

This paper proposes a new way of evaluating the accuracy and validity of probabilistic forecasts that change over time (such as an in-game win probability model, or an election forecast). Under this approach, each model to be evaluated is treated as a canonical Kelly bettor, and the models are pitted against each other in an iterative betting contest. The growth or decline of each model's bankroll serves as the evaluation metric. Under this approach, market consensus probabilities and implied model credibilities can be updated real time as each model updates, and do not require one to wait for the final outcome. Using a simulation model, it will be shown that this method is in general more accurate than traditional average log-loss and Brier score methods at distinguishing a correct model from an incorrect model. This Kelly approach is shown to have a direct mathematical and conceptual analogue to Bayesian inference, with bankroll serving as a proxy for Bayesian credibility.
Paper Structure (37 sections, 37 equations, 10 figures, 18 tables)

This paper contains 37 sections, 37 equations, 10 figures, 18 tables.

Table of Contents

  1. Introduction
  2. A Simple Example
  3. Binary Probabilities with Multiple Bettors
  4. Optimal Kelly betting with existing bets
  5. Determining a "market clearing price" with multiple Kelly bettors and existing bets
  6. Procedure for evaluating multiple forecast models against each other in the case of binary probabilities
  7. The General Case with Multinomial Probabilities and Multiple Bettors
  8. Kelly betting with multiple outcomes and existing bets
  9. Calculating market clearing odds with multiple outcomes and multiple bettors
  10. A self-evaluating market
  11. Bankroll as credibility: Connection to Bayes' Theorem
  12. Simulating Various Models to Demonstrate the Value of This Approach
  13. Creating a hypothetical sports contest to model and then simulating
  14. When the incorrect model has the wrong point probability
  15. When the incorrect model has faulty recency bias
  16. ...and 22 more sections

Figures (10)

  • Figure 1: Comparison of in-game win probabilities for Open Source Football and ESPN for the Seahawks--Eagles game on December 18, 2023
  • Figure 2: Implied model credibility for Open Source Football and ESPN, evaluated in-game
  • Figure 3: Weekly division win probabilities for each team, shown separately for FanGraphs and FiveThirtyEight
  • Figure 4: Market consensus division win probabilities by week for the 2022 NL East Division
  • Figure 5: Weekly model credibility for FanGraphs and FiveThirtyEight, based on the 2022 NL East division race
  • ...and 5 more figures