Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Framework for Predicting the Impact of Game Balance Changes through Meta Discovery

Akash Saravanan, Matthew Guzdial

TL;DR

This work tackles predicting how balance changes affect the competitive metagame in team-based, character-driven games by proposing a Meta Discovery Framework that combines a Battle Agent, a Team Builder, and a Simulation Environment. It instantiates the framework with self-play PPO and a heuristic agent in Pokémon Showdown to form ABC-Meta and BSD-Meta tasks, evaluating using Overlap, Edit Distance, and Spearman's Rho metrics across multiple tiers and a blank slate baseline. Results show that the approach closely approximates true metagames after balance changes, outperforming naive baselines and providing insights into the role of prior metagame knowledge, though gaps remain and there is room for improvement. The framework offers a practical, generalizable tool for game designers to anticipate metagame shifts before deploying balance changes, with openly available code and clear paths for extension to other games, horizons, and drafting dynamics. In total, the study analyzes approximately 450,000 battles (3 months × 150,000 battles per month) to simulate policy impacts and validate the predictive framework.

Abstract

A metagame is a collection of knowledge that goes beyond the rules of a game. In competitive, team-based games like Pokémon or League of Legends, it refers to the set of current dominant characters and/or strategies within the player base. Developer changes to the balance of the game can have drastic and unforeseen consequences on these sets of meta characters. A framework for predicting the impact of balance changes could aid developers in making more informed balance decisions. In this paper we present such a Meta Discovery framework, leveraging Reinforcement Learning for automated testing of balance changes. Our results demonstrate the ability to predict the outcome of balance changes in Pokémon Showdown, a collection of competitive Pokémon tiers, with high accuracy.

A Framework for Predicting the Impact of Game Balance Changes through Meta Discovery

TL;DR

This work tackles predicting how balance changes affect the competitive metagame in team-based, character-driven games by proposing a Meta Discovery Framework that combines a Battle Agent, a Team Builder, and a Simulation Environment. It instantiates the framework with self-play PPO and a heuristic agent in Pokémon Showdown to form ABC-Meta and BSD-Meta tasks, evaluating using Overlap, Edit Distance, and Spearman's Rho metrics across multiple tiers and a blank slate baseline. Results show that the approach closely approximates true metagames after balance changes, outperforming naive baselines and providing insights into the role of prior metagame knowledge, though gaps remain and there is room for improvement. The framework offers a practical, generalizable tool for game designers to anticipate metagame shifts before deploying balance changes, with openly available code and clear paths for extension to other games, horizons, and drafting dynamics. In total, the study analyzes approximately 450,000 battles (3 months × 150,000 battles per month) to simulate policy impacts and validate the predictive framework.

Abstract

A metagame is a collection of knowledge that goes beyond the rules of a game. In competitive, team-based games like Pokémon or League of Legends, it refers to the set of current dominant characters and/or strategies within the player base. Developer changes to the balance of the game can have drastic and unforeseen consequences on these sets of meta characters. A framework for predicting the impact of balance changes could aid developers in making more informed balance decisions. In this paper we present such a Meta Discovery framework, leveraging Reinforcement Learning for automated testing of balance changes. Our results demonstrate the ability to predict the outcome of balance changes in Pokémon Showdown, a collection of competitive Pokémon tiers, with high accuracy.
Paper Structure (28 sections, 6 equations, 1 figure, 7 tables)

This paper contains 28 sections, 6 equations, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Pokémon
  4. Pokémon Showdown
  5. Attributes of Competitive Pokémon Battles
  6. Pokémon Battles
  7. The Competitive Metagame of Pokémon
  8. Related Work
  9. Team-Building
  10. Metagame Discovery & Game Balance
  11. Battle Agents
  12. Action Space
  13. State Representation
  14. Reward Function
  15. Agents
  16. ...and 13 more sections

Figures (1)

  • Figure 1: The three main components of our meta discovery framework. The Battle Agent plays matches while the Team Builder generates teams (using a mix of metagame statistics and domain knowledge) for the Battle Agent to use in matches. The simulator Environment is responsible for simulating battles using the Battle Agent with the generated teams from the Team Builder and tracking the overall statistics of the system.