Equilibrium Computation in Multi-Stage Auctions and Contests

TL;DR

This paper demonstrates that deep reinforcement learning with self-play can learn equilibrium bidding strategies in continuous multi-stage games, such as sequential auctions, elimination contests, and Stackelberg-Bertrand competition, where traditional analytic solutions are intractable due to infinite-dimensional bid and signal spaces. It combines policy-gradient methods (Reinforce and PPO) with Lipschitz-based regularity and comprehensive verification to identify and certify approximate Nash equilibria across diverse model variations, including asymmetric, interdependent-prior, and risk-averse settings. A novel verification procedure is introduced and proven to yield upper bounds on equilibrium deviations that shrink as discretization granularity and sampling increase, enabling practical validation of learned equilibria. The experiments show that DRL can recover known analytical equilibria and uncover new asymmetric or interdependent-prior equilibria across multiple multi-stage scenarios, with runtimes ranging from minutes to hours, thus providing a scalable, flexible toolkit for equilibrium analysis in economic and management contexts.

Abstract

We compute equilibrium strategies in multi-stage games with continuous signal and action spaces as they are widely used in the management sciences and economics. Examples include sequential sales via auctions, multi-stage elimination contests, and Stackelberg competitions. In sequential auctions, analysts performing equilibrium analysis are required to derive not just single bids but bid functions for all possible signals or values that a bidder might have in multiple stages. Due to the continuity of the signal and action spaces, these bid functions come from an infinite dimensional space. While such models are fundamental to game theory and its applications, equilibrium strategies are rarely known. The resulting system of non-linear differential equations is considered intractable for all but elementary models. This has been limiting progress in game theory and is a barrier to its adoption in the field. We show that Deep Reinforcement Learning and self-play can learn equilibrium bidding strategies for various multi-stage games. We find equilibrium in models that have not yet been explored analytically and new asymmetric equilibrium bid functions for established models of sequential auctions. The verification of equilibrium is challenging in such games due to the continuous signal and action spaces. We introduce a verification algorithm and prove that the error of this verifier decreases when considering Lipschitz continuous strategies with increasing levels of discretization and sample sizes.

Figures (9)

• Figure 1: Approximate utility loss for different configurations of the discretization size and the number of simulations (left) and their corresponding run times (right).
• Figure 2: Equilibrium and PPO-based learned strategies in sequential sales with a first-price mechanism, two-stages, and three bidders.
• Figure 3: Asymmetric PPO-based learned strategies in sequential sales with a second-price mechanism, two stages, and three bidders.
• Figure 4: Reinforce-based learned strategies in sequential sales with a second-price mechanism, a mineral rights prior, and three risk-averse bidders with $\rho = 2.0$ in two stages.
• Figure 5: Asymmetric Reinforce-based learned strategies in the elimination contest with two weak and two strong contestants.

Theorems & Definitions (30)

• Definition 3.1: Multi-stage game myersonPerfectConditionalEEquilibria2020
• Definition 3.2: $\varepsilon$-Nash equilibrium
• Theorem 5.1: informal
• Proposition 6.1: krishna2009auction
• Proposition 6.2: zhang2008simultaneous
• Proposition 6.3: arozamenaSimultaneousVsSequential2009
• Definition A.1: Perfect recall
• Lemma C.1
• proof
• Lemma C.2