Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Yahtzee: Reinforcement Learning Techniques for Stochastic Combinatorial Games

Nicholas A. Pape

TL;DR

This work frames Yahtzee as a Markov Decision Process and evaluates self-play policy-gradient agents (REINFORCE, A2C, PPO) on both single-turn and full-game tasks. A2C with TD(0) emerges as the most robust approach, achieving a median score of $241.78$ over $10^5$ evaluation games, within $5\%$ of the DP-optimal solitaire score $254.59$, and attaining Yahtzee and upper-bonus rates of $34.1\%$ and $24.9\%$ respectively. The study thoroughly analyzes state/action representations, network architecture, credit assignment, entropy, and reward shaping, revealing that long-horizon credit assignment and exploration are key challenges. The results highlight Yahtzee as a meaningful benchmark for RL mid-scale problems, offering insights into learning coherent upper-bonus strategies and the trade-offs between single-turn optimization and full-game performance.

Abstract

Yahtzee is a classic dice game with a stochastic, combinatorial structure and delayed rewards, making it an interesting mid-scale RL benchmark. While an optimal policy for solitaire Yahtzee can be computed using dynamic programming methods, multiplayer is intractable, motivating approximation methods. We formulate Yahtzee as a Markov Decision Process (MDP), and train self-play agents using various policy gradient methods: REINFORCE, Advantage Actor-Critic (A2C), and Proximal Policy Optimization (PPO), all using a multi-headed network with a shared trunk. We ablate feature and action encodings, architecture, return estimators, and entropy regularization to understand their impact on learning. Under a fixed training budget, REINFORCE and PPO prove sensitive to hyperparameters and fail to reach near-optimal performance, whereas A2C trains robustly across a range of settings. Our agent attains a median score of 241.78 points over 100,000 evaluation games, within 5.0\% of the optimal DP score of 254.59, achieving the upper section bonus and Yahtzee at rates of 24.9\% and 34.1\%, respectively. All models struggle to learn the upper bonus strategy, overindexing on four-of-a-kind's, highlighting persistent long-horizon credit-assignment and exploration challenges.

Yahtzee: Reinforcement Learning Techniques for Stochastic Combinatorial Games

TL;DR

This work frames Yahtzee as a Markov Decision Process and evaluates self-play policy-gradient agents (REINFORCE, A2C, PPO) on both single-turn and full-game tasks. A2C with TD(0) emerges as the most robust approach, achieving a median score of over evaluation games, within of the DP-optimal solitaire score , and attaining Yahtzee and upper-bonus rates of and respectively. The study thoroughly analyzes state/action representations, network architecture, credit assignment, entropy, and reward shaping, revealing that long-horizon credit assignment and exploration are key challenges. The results highlight Yahtzee as a meaningful benchmark for RL mid-scale problems, offering insights into learning coherent upper-bonus strategies and the trade-offs between single-turn optimization and full-game performance.

Abstract

Yahtzee is a classic dice game with a stochastic, combinatorial structure and delayed rewards, making it an interesting mid-scale RL benchmark. While an optimal policy for solitaire Yahtzee can be computed using dynamic programming methods, multiplayer is intractable, motivating approximation methods. We formulate Yahtzee as a Markov Decision Process (MDP), and train self-play agents using various policy gradient methods: REINFORCE, Advantage Actor-Critic (A2C), and Proximal Policy Optimization (PPO), all using a multi-headed network with a shared trunk. We ablate feature and action encodings, architecture, return estimators, and entropy regularization to understand their impact on learning. Under a fixed training budget, REINFORCE and PPO prove sensitive to hyperparameters and fail to reach near-optimal performance, whereas A2C trains robustly across a range of settings. Our agent attains a median score of 241.78 points over 100,000 evaluation games, within 5.0\% of the optimal DP score of 254.59, achieving the upper section bonus and Yahtzee at rates of 24.9\% and 34.1\%, respectively. All models struggle to learn the upper bonus strategy, overindexing on four-of-a-kind's, highlighting persistent long-horizon credit-assignment and exploration challenges.
Paper Structure (64 sections, 35 equations, 19 figures, 7 tables)

This paper contains 64 sections, 35 equations, 19 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Yahtzee as a Reinforcement Learning Benchmark
  3. Objectives
  4. Related Work
  5. Policy Gradient Methods and Variance Reduction
  6. Return Estimation
  7. Policy Gradient Methods
  8. Other Variance Reduction Techniques
  9. Reward Shaping
  10. Complex Games
  11. DP Methods for Yahtzee
  12. Reinforcement Learning for Yahtzee
  13. Problem Formulation
  14. Game Description
  15. Rules of Yahtzee
  16. ...and 49 more sections

Figures (19)

  • Figure 1: Overall network architecture with shared trunk and three specialized heads
  • Figure 2: Reward Shaping: Upper bonus prediction head architecture
  • Figure 3: Single-turn and full-game performance during training.
  • Figure 4: Pareto Frontier of Single-Turn vs. Full-Game Performance
  • Figure 5: Final Performance Comparison after training on 1 million games
  • ...and 14 more figures