Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FairDICE: Fairness-Driven Offline Multi-Objective Reinforcement Learning

Woosung Kim, Jinho Lee, Jongmin Lee, Byung-Jun Lee

TL;DR

This work tackles offline multi-objective reinforcement learning (MORL) with nonlinear fairness objectives by introducing FairDICE, the first offline MORL framework that directly optimizes nonlinear welfare. It builds a regularized welfare optimization formulation that couples welfare maximization with distribution-shift control via an f-divergence between the learned stationary distribution and the offline data, enabling stable, sample-efficient learning without explicit preference sweeps. The authors establish a theoretical link showing that FairDICE implicitly optimizes the same welfare as regularized linear MORL when using optimal dual weights, and provide a practical, sample-based algorithm that extends DICE-RL to nonlinear welfare. Empirically, FairDICE demonstrates strong fairness-aware performance on discrete MO environments and continuous D4MORL benchmarks, balancing NSW, Jain’s index, and utilitarian welfare while remaining robust to dataset quality and distributional shift.

Abstract

Multi-objective reinforcement learning (MORL) aims to optimize policies in the presence of conflicting objectives, where linear scalarization is commonly used to reduce vector-valued returns into scalar signals. While effective for certain preferences, this approach cannot capture fairness-oriented goals such as Nash social welfare or max-min fairness, which require nonlinear and non-additive trade-offs. Although several online algorithms have been proposed for specific fairness objectives, a unified approach for optimizing nonlinear welfare criteria in the offline setting-where learning must proceed from a fixed dataset-remains unexplored. In this work, we present FairDICE, the first offline MORL framework that directly optimizes nonlinear welfare objective. FairDICE leverages distribution correction estimation to jointly account for welfare maximization and distributional regularization, enabling stable and sample-efficient learning without requiring explicit preference weights or exhaustive weight search. Across multiple offline benchmarks, FairDICE demonstrates strong fairness-aware performance compared to existing baselines.

FairDICE: Fairness-Driven Offline Multi-Objective Reinforcement Learning

TL;DR

This work tackles offline multi-objective reinforcement learning (MORL) with nonlinear fairness objectives by introducing FairDICE, the first offline MORL framework that directly optimizes nonlinear welfare. It builds a regularized welfare optimization formulation that couples welfare maximization with distribution-shift control via an f-divergence between the learned stationary distribution and the offline data, enabling stable, sample-efficient learning without explicit preference sweeps. The authors establish a theoretical link showing that FairDICE implicitly optimizes the same welfare as regularized linear MORL when using optimal dual weights, and provide a practical, sample-based algorithm that extends DICE-RL to nonlinear welfare. Empirically, FairDICE demonstrates strong fairness-aware performance on discrete MO environments and continuous D4MORL benchmarks, balancing NSW, Jain’s index, and utilitarian welfare while remaining robust to dataset quality and distributional shift.

Abstract

Multi-objective reinforcement learning (MORL) aims to optimize policies in the presence of conflicting objectives, where linear scalarization is commonly used to reduce vector-valued returns into scalar signals. While effective for certain preferences, this approach cannot capture fairness-oriented goals such as Nash social welfare or max-min fairness, which require nonlinear and non-additive trade-offs. Although several online algorithms have been proposed for specific fairness objectives, a unified approach for optimizing nonlinear welfare criteria in the offline setting-where learning must proceed from a fixed dataset-remains unexplored. In this work, we present FairDICE, the first offline MORL framework that directly optimizes nonlinear welfare objective. FairDICE leverages distribution correction estimation to jointly account for welfare maximization and distributional regularization, enabling stable and sample-efficient learning without requiring explicit preference weights or exhaustive weight search. Across multiple offline benchmarks, FairDICE demonstrates strong fairness-aware performance compared to existing baselines.

Paper Structure

This paper contains 43 sections, 1 theorem, 34 equations, 8 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-Objective RL and Welfare Objectives
  4. Offline RL and Offline MORL
  5. Preliminaries
  6. Markov Decision Process (MDP) and Multi-objective RL
  7. Offline Reinforcement Learning and DICE-RL framework
  8. Fair MORL as Convex Optimization
  9. Convex Formulation via stationary distribution
  10. Reformulation for sample-based optimization
  11. FairDICE: Welfare Optimization for Offline Fair MORL
  12. Regularized Welfare Optimization framework
  13. Equivalence to Regularized Linear MORL
  14. Empirical Behaviors of FairDICE
  15. MO-Four-Room Experiment
  16. ...and 28 more sections

Key Result

Proposition 1

Let $\mu_{i}^{*}$ be the optimal multipliers obtained from eq:FairDICE. Then, the optimal solutions of FairDICE and (P3-reg) with $\mu_{i}^{*}$ yield the same unique optimal policy. (Proof in Appendix Appendix:propositionproof)

Figures (8)

  • Figure 1: Visualization of FairDICE policies in MO-Four-Rooms: (a) Uniformly random policy for data collection, (b) FairDICE policy maximizing Utilitarian welfare (sum of returns), (c) FairDICE policy maximizing Nash social welfare (NSW). (d) FairDICE maximizing NSW in a domain with eight objectives. Red arrows indicate the policy, and the blue heatmap shows state visitation.
  • Figure 2: Policy performance on Random MOMDP domain across different $\alpha$ and $\beta$ values, evaluated on Nash social welfare, Utilitarian welfare, and Jain’s fairness index. Results are averaged over 1000 seeds, and reported with 95 $\%$ confidence intervals.
  • Figure 3: FairDICE-fixed with perturbed $\mu^{*}$
  • Figure 4: Nash social welfare scores for five two-objective tasks, evaluated across 30 linearly spaced preference weights. Each curve shows the average NSW over 5 seeds and 10 evaluation episodes per seed. Red line indicates the average NSW performance of FairDICE.
  • Figure 5: Raw return evaluations on five two-objective MuJoCo tasks from D4MORL. Each point represents policy performance under a specific preference weight; Pareto frontiers and dominated regions are shown.
  • ...and 3 more figures

Theorems & Definitions (1)

  • Proposition 1: Equivalence between the regularized problems