Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Welfare and Fairness in Multi-objective Reinforcement Learning

Zimeng Fan, Nianli Peng, Muhang Tian, Brandon Fain

TL;DR

This work studies fairness in multi-objective reinforcement learning by optimizing nonlinear welfare functions, with NSW as the canonical objective. It proves that maximizing expected NSW is APX-hard, even in tabular MOMDPs, and thus motivates a model-free approach that uses nonlinear welfare-aware updates and non-stationary action selection. The proposed Welfare Q-Learning algorithm is shown to converge to a unique fixed point and empirically outperforms linear scalarization, mixtures, and stationary policies on Taxi and Resource Gathering tasks. The results demonstrate that optimizing for NSW yields superior fairness-utility trade-offs and robustness to rescaling of objective magnitudes, offering a principled method for fair resource allocation in sequential decision problems.

Abstract

We study fair multi-objective reinforcement learning in which an agent must learn a policy that simultaneously achieves high reward on multiple dimensions of a vector-valued reward. Motivated by the fair resource allocation literature, we model this as an expected welfare maximization problem, for some nonlinear fair welfare function of the vector of long-term cumulative rewards. One canonical example of such a function is the Nash Social Welfare, or geometric mean, the log transform of which is also known as the Proportional Fairness objective. We show that even approximately optimal optimization of the expected Nash Social Welfare is computationally intractable even in the tabular case. Nevertheless, we provide a novel adaptation of Q-learning that combines nonlinear scalarized learning updates and non-stationary action selection to learn effective policies for optimizing nonlinear welfare functions. We show that our algorithm is provably convergent, and we demonstrate experimentally that our approach outperforms techniques based on linear scalarization, mixtures of optimal linear scalarizations, or stationary action selection for the Nash Social Welfare Objective.

Welfare and Fairness in Multi-objective Reinforcement Learning

TL;DR

This work studies fairness in multi-objective reinforcement learning by optimizing nonlinear welfare functions, with NSW as the canonical objective. It proves that maximizing expected NSW is APX-hard, even in tabular MOMDPs, and thus motivates a model-free approach that uses nonlinear welfare-aware updates and non-stationary action selection. The proposed Welfare Q-Learning algorithm is shown to converge to a unique fixed point and empirically outperforms linear scalarization, mixtures, and stationary policies on Taxi and Resource Gathering tasks. The results demonstrate that optimizing for NSW yields superior fairness-utility trade-offs and robustness to rescaling of objective magnitudes, offering a principled method for fair resource allocation in sequential decision problems.

Abstract

We study fair multi-objective reinforcement learning in which an agent must learn a policy that simultaneously achieves high reward on multiple dimensions of a vector-valued reward. Motivated by the fair resource allocation literature, we model this as an expected welfare maximization problem, for some nonlinear fair welfare function of the vector of long-term cumulative rewards. One canonical example of such a function is the Nash Social Welfare, or geometric mean, the log transform of which is also known as the Proportional Fairness objective. We show that even approximately optimal optimization of the expected Nash Social Welfare is computationally intractable even in the tabular case. Nevertheless, we provide a novel adaptation of Q-learning that combines nonlinear scalarized learning updates and non-stationary action selection to learn effective policies for optimizing nonlinear welfare functions. We show that our algorithm is provably convergent, and we demonstrate experimentally that our approach outperforms techniques based on linear scalarization, mixtures of optimal linear scalarizations, or stationary action selection for the Nash Social Welfare Objective.
Paper Structure (23 sections, 4 theorems, 10 equations, 5 figures, 1 algorithm)

This paper contains 23 sections, 4 theorems, 10 equations, 5 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions and Outline
  3. Related Work
  4. Preliminaries
  5. Problem Formulation
  6. Welfare Axioms
  7. Nash Social Welfare Function
  8. Expected Welfare
  9. Example (Expected Welfare).
  10. Optimizing Welfare
  11. Non-stationary Welfare Q-Learning
  12. Non-stationarity
  13. Convergence
  14. Experiments
  15. Metrics, Methods, and Baseline Algorithms
  16. ...and 8 more sections

Key Result

lemma 1

Hardness17 It is APX-hard to compute an indivisible allocation of goods optimizing the NSW.

Figures (5)

  • Figure 1: Example MOMDP. Dotted lines represent trajectories generated by $\pi_1$, solid line for $\pi_2$
  • Figure 2: Example of a stochastic MOMDP in which the expected Nash social welfare of stationary policies shrinks to $0$ as the dimension of rewards $n$ increases.
  • Figure 3: Simulated Environments
  • Figure 4: Experiment Results for Taxi Environment. Non-stationary Policy is Welfare Q-Learning
  • Figure 5: Experiment Results for RG Environment. Non-stationary Policy is Welfare Q-Learning

Theorems & Definitions (9)

  • definition 1
  • definition 2
  • definition 3
  • definition 4
  • lemma 1
  • theorem 1
  • theorem 2
  • definition 8
  • lemma 5: Interpreting the Fixed-Point