Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Objective Reinforcement Learning with Max-Min Criterion: A Game-Theoretic Approach

Woohyeon Byeon, Giseung Park, Jongseong Chae, Amir Leshem, Youngchul Sung

TL;DR

This work addresses MORL under a max-min fairness criterion by recasting the problem as a two-player, zero-sum, entropy-regularized game and solving it with single-loop updates. The proposed ERAM and ARAM algorithms combine mirror-descent/NPG updates for the policy with closed-form updates for the adversarial weights, enabling global last-iterate convergence and scalable runtime. Theoretical results establish convergence guarantees and complexity bounds under exact and approximate policy evaluation, while experiments show superior max-min performance, substantial memory savings, and faster training versus baselines in tabular MORL and traffic signal control tasks. The approach offers a practical path toward robust, fair, and scalable multi-objective RL in real-world resource allocation settings.

Abstract

In this paper, we propose a provably convergent and practical framework for multi-objective reinforcement learning with max-min criterion. From a game-theoretic perspective, we reformulate max-min multi-objective reinforcement learning as a two-player zero-sum regularized continuous game and introduce an efficient algorithm based on mirror descent. Our approach simplifies the policy update while ensuring global last-iterate convergence. We provide a comprehensive theoretical analysis on our algorithm, including iteration complexity under both exact and approximate policy evaluations, as well as sample complexity bounds. To further enhance performance, we modify the proposed algorithm with adaptive regularization. Our experiments demonstrate the convergence behavior of the proposed algorithm in tabular settings, and our implementation for deep reinforcement learning significantly outperforms previous baselines in many MORL environments.

Multi-Objective Reinforcement Learning with Max-Min Criterion: A Game-Theoretic Approach

TL;DR

This work addresses MORL under a max-min fairness criterion by recasting the problem as a two-player, zero-sum, entropy-regularized game and solving it with single-loop updates. The proposed ERAM and ARAM algorithms combine mirror-descent/NPG updates for the policy with closed-form updates for the adversarial weights, enabling global last-iterate convergence and scalable runtime. Theoretical results establish convergence guarantees and complexity bounds under exact and approximate policy evaluation, while experiments show superior max-min performance, substantial memory savings, and faster training versus baselines in tabular MORL and traffic signal control tasks. The approach offers a practical path toward robust, fair, and scalable multi-objective RL in real-world resource allocation settings.

Abstract

In this paper, we propose a provably convergent and practical framework for multi-objective reinforcement learning with max-min criterion. From a game-theoretic perspective, we reformulate max-min multi-objective reinforcement learning as a two-player zero-sum regularized continuous game and introduce an efficient algorithm based on mirror descent. Our approach simplifies the policy update while ensuring global last-iterate convergence. We provide a comprehensive theoretical analysis on our algorithm, including iteration complexity under both exact and approximate policy evaluations, as well as sample complexity bounds. To further enhance performance, we modify the proposed algorithm with adaptive regularization. Our experiments demonstrate the convergence behavior of the proposed algorithm in tabular settings, and our implementation for deep reinforcement learning significantly outperforms previous baselines in many MORL environments.
Paper Structure (52 sections, 8 theorems, 98 equations, 15 figures, 8 tables, 3 algorithms)

This paper contains 52 sections, 8 theorems, 98 equations, 15 figures, 8 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Multi-objective Markov decision process
  4. The Proposed Method
  5. Max-min versus min-max
  6. Two-plaer zero-sum regularized continuous game formulation
  7. The ERAM algorithm: Entropy-regularized adversary for max-min MORL
  8. The ARAM algorithm: Adaptively-regularized adversary for max-min MORL
  9. Theoretical Analysis
  10. Related Works
  11. Experiments
  12. Numerical results in tabular setting
  13. Experimental results in traffic signal control
  14. Complexity comparison
  15. Ablation study
  16. ...and 37 more sections

Key Result

Theorem 3.1

Let $(\Bar{\pi},\Bar{w})$ be a saddle point in (maxmin_equals_minmax). Then, $\Bar{\pi}$ is a solution to the entropy-regularized max-min MORL maxmin_ent_target_problem.

Figures (15)

  • Figure 1: Nash gap
  • Figure 2: Traffic signal control environment sumorl
  • Figure 3: Convergence comparison in random tabular MOMDPs: ERAM (top) vs. park (bottom). Colors indicate different random MOMDPs.
  • Figure 4: Relationship between max-min fairness and other fairness notions in MORL.
  • Figure 7: Nash gap for ARAM
  • ...and 10 more figures

Theorems & Definitions (14)

  • Theorem 3.1
  • Theorem 4.1
  • Corollary 4.2
  • Theorem 4.3
  • Corollary 4.4
  • Remark 4.5
  • Proposition F.1
  • proof
  • proof
  • Remark F.2
  • ...and 4 more