Achieving Logarithmic Regret in KL-Regularized Zero-Sum Markov Games
Anupam Nayak, Tong Yang, Osman Yagan, Gauri Joshi, Yuejie Chi
TL;DR
This work addresses sample-efficient learning in two-player zero-sum games under KL regularization by exploiting a Gibbs-form best-response structure induced by the reference policy. The authors introduce Optimistic Matrix Game (OMG) for matrix games and Super-Optimistic Markov Game (SOMG) for Markov games, both leveraging best-response sampling and specialized bonuses to achieve logarithmic regret in the episode horizon $T$, with explicit dependence on the regularization strength $β$ and function approximation complexity. They establish regularization-dependent and regularization-independent regret bounds, demonstrating significant improvement over standard $\widetilde{O}(\sqrt{T})$ rates and providing finite-sample guarantees for learning an $\varepsilon$-NE under KL regularization. The methods reduce to single-agent settings in extreme cases and offer insights for alignment and self-play in LLM contexts where KL penalties encode useful priors. Overall, the results advance theoretical understanding of KL regularization in game-theoretic online learning and offer practical algorithms with provable sample efficiency benefits.
Abstract
Reverse Kullback-Leibler (KL) divergence-based regularization with respect to a fixed reference policy is widely used in modern reinforcement learning to preserve the desired traits of the reference policy and sometimes to promote exploration (using uniform reference policy, known as entropy regularization). Beyond serving as a mere anchor, the reference policy can also be interpreted as encoding prior knowledge about good actions in the environment. In the context of alignment, recent game-theoretic approaches have leveraged KL regularization with pretrained language models as reference policies, achieving notable empirical success in self-play methods. Despite these advances, the theoretical benefits of KL regularization in game-theoretic settings remain poorly understood. In this work, we develop and analyze algorithms that provably achieve improved sample efficiency under KL regularization. We study both two-player zero-sum Matrix games and Markov games: for Matrix games, we propose OMG, an algorithm based on best response sampling with optimistic bonuses, and extend this idea to Markov games through the algorithm SOMG, which also uses best response sampling and a novel concept of superoptimistic bonuses. Both algorithms achieve a logarithmic regret in $T$ that scales inversely with the KL regularization strength $β$ in addition to the standard $\widetilde{\mathcal{O}}(\sqrt{T})$ regret independent of $β$ which is attained in both regularized and unregularized settings