Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning

Tianchen Zhou, FNU Hairi, Haibo Yang, Jia Liu, Tian Tong, Fan Yang, Michinari Momma, Yan Gao

TL;DR

This work addresses MORL by introducing MOAC, an MGDA-inspired actor-critic algorithm capable of jointly optimizing multiple conflicting objectives in both average and discounted reward settings. It provides the first finite-time Pareto-stationary convergence and sample complexity guarantees for MORL, with convergence rates and sample complexity that are independent of the number of objectives, thanks to a momentum-based mechanism that mitigates cumulative estimation bias. The actor step uses a quadratic program to compute a common descent direction from individual objective gradients, while the critic side delivers stable, batched TD updates. Empirical results on synthetic MORL tasks and a real-world dataset validate MOAC’s effectiveness and robustness, highlighting practical initialization and bias-control benefits for multi-objective policy learning.

Abstract

Reinforcement learning with multiple, potentially conflicting objectives is pervasive in real-world applications, while this problem remains theoretically under-explored. This paper tackles the multi-objective reinforcement learning (MORL) problem and introduces an innovative actor-critic algorithm named MOAC which finds a policy by iteratively making trade-offs among conflicting reward signals. Notably, we provide the first analysis of finite-time Pareto-stationary convergence and corresponding sample complexity in both discounted and average reward settings. Our approach has two salient features: (a) MOAC mitigates the cumulative estimation bias resulting from finding an optimal common gradient descent direction out of stochastic samples. This enables provable convergence rate and sample complexity guarantees independent of the number of objectives; (b) With proper momentum coefficient, MOAC initializes the weights of individual policy gradients using samples from the environment, instead of manual initialization. This enhances the practicality and robustness of our algorithm. Finally, experiments conducted on a real-world dataset validate the effectiveness of our proposed method.

Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning

TL;DR

This work addresses MORL by introducing MOAC, an MGDA-inspired actor-critic algorithm capable of jointly optimizing multiple conflicting objectives in both average and discounted reward settings. It provides the first finite-time Pareto-stationary convergence and sample complexity guarantees for MORL, with convergence rates and sample complexity that are independent of the number of objectives, thanks to a momentum-based mechanism that mitigates cumulative estimation bias. The actor step uses a quadratic program to compute a common descent direction from individual objective gradients, while the critic side delivers stable, batched TD updates. Empirical results on synthetic MORL tasks and a real-world dataset validate MOAC’s effectiveness and robustness, highlighting practical initialization and bias-control benefits for multi-objective policy learning.

Abstract

Reinforcement learning with multiple, potentially conflicting objectives is pervasive in real-world applications, while this problem remains theoretically under-explored. This paper tackles the multi-objective reinforcement learning (MORL) problem and introduces an innovative actor-critic algorithm named MOAC which finds a policy by iteratively making trade-offs among conflicting reward signals. Notably, we provide the first analysis of finite-time Pareto-stationary convergence and corresponding sample complexity in both discounted and average reward settings. Our approach has two salient features: (a) MOAC mitigates the cumulative estimation bias resulting from finding an optimal common gradient descent direction out of stochastic samples. This enables provable convergence rate and sample complexity guarantees independent of the number of objectives; (b) With proper momentum coefficient, MOAC initializes the weights of individual policy gradients using samples from the environment, instead of manual initialization. This enhances the practicality and robustness of our algorithm. Finally, experiments conducted on a real-world dataset validate the effectiveness of our proposed method.
Paper Structure (21 sections, 11 theorems, 61 equations, 5 figures, 2 tables, 2 algorithms)

This paper contains 21 sections, 11 theorems, 61 equations, 5 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. A Multi-Objective Actor-Critic Framework
  4. Preliminaries of MORL
  5. Policy Gradient for MORL
  6. The Proposed $\mathsf{MOAC}$ Algorithmic Framework
  7. Pareto-Stationary Convergence and Sample Complexity Analysis for $\mathsf{MOAC}$
  8. Theoretical Results of the Critic of $\mathsf{MOAC}$
  9. Theoretical Results of the $\mathsf{MOAC}$ Framework
  10. Experiments
  11. Conclusion and Future Work
  12. Experimental Setup and Complementary Results
  13. Synthetic Data
  14. Real-World Data
  15. Supporting Lemmas
  16. ...and 6 more sections

Key Result

Lemma 1

For any $\bm{\theta}$, let $\pi_{\bm{\theta}}:\mathcal{S}\times\mathcal{A}\to[0,1]$ be a policy and let $J^{i}(\bm{\theta})$ be the total reward for the $i$-th objective. Then, the policy-gradient of $J^{i}(\bm{\theta})$ with respect to parameter $\bm{\theta}$ can be computed as:

Figures (5)

  • Figure 1: (a) Discounted rewards of three objectives with momentum $\eta_t=t^{-1}$; (b) Squared $\ell_2$-norm of policy gradients with different momentum coefficients.
  • Figure 2: Environment: Resource Gathering
  • Figure 3: Environment: FishWood
  • Figure 4: Resource Gathering environment. Average rewards of three objectives with momentum $\eta_t=t^{-2}$ (left), and squared $\ell_2$-norm of policy gradients (right).
  • Figure 5: FishWood environment. Discounted rewards of two conflicting objectives with momentum $\eta_t=t^{-1}$ (left), and squared $\ell_2$-norm of policy gradients (right).

Theorems & Definitions (20)

  • Definition 1: MOMDP
  • Definition 2
  • Definition 3
  • Lemma 1: Policy Gradient Theorem
  • Lemma 2
  • Theorem 3
  • Corollary 4
  • Theorem 5
  • Remark 1
  • Remark 2
  • ...and 10 more