Joint-perturbation simultaneous pseudo-gradient
Carlos Martin, Tuomas Sandholm
TL;DR
This work addresses computing approximate Nash equilibria in continuous-action games with black-box utilities by introducing a joint-perturbation simultaneous pseudo-gradient (JPSPG) method. By performing a single joint perturbation across all players, it achieves a constant number of utility evaluations per iteration, $O(1)$, independent of the number of players, and integrates with standard equilibrium dynamics such as SGA, extragradient, and optimistic gradient methods. The key technical contributions include the pseudo-Jacobian framework and the joint perturbation estimator $\mathbf{g} = \frac{1}{\sigma} \mathbf{u}(\mathbf{x}+\sigma\mathbf{z}) \odot \mathbf{z}$, enabling dramatic speedups in many-player and costly-function settings. Empirical results across multi-item unit-demand auctions, knapsack auctions, sequential auctions, and continuous-action Goofspiel demonstrate substantial reductions in training time while achieving comparable or better exploitability performance.
Abstract
We study the problem of computing an approximate Nash equilibrium of a game whose strategy space is continuous without access to gradients of the utility function. Such games arise, for example, when players' strategies are represented by the parameters of a neural network. Lack of access to gradients is common in reinforcement learning settings, where the environment is treated as a black box, as well as equilibrium finding in mechanisms such as auctions, where the mechanism's payoffs are discontinuous in the players' actions. To tackle this problem, we turn to zeroth-order optimization techniques that combine pseudo-gradients with equilibrium-finding dynamics. Specifically, we introduce a new technique that requires a number of utility function evaluations per iteration that is constant rather than linear in the number of players. It achieves this by performing a single joint perturbation on all players' strategies, rather than perturbing each one individually. This yields a dramatic improvement for many-player games, especially when the utility function is expensive to compute in terms of wall time, memory, money, or other resources. We evaluate our approach on various games, including auctions, which have important real-world applications. Our approach yields a significant reduction in the run time required to reach an approximate Nash equilibrium.