Pareto Set Learning for Multi-Objective Reinforcement Learning

TL;DR

Multi-objective decision problems in MORL require trading off conflicting objectives, and prior methods either produce a single policy or fail to densely cover the Pareto front. This work proposes PSL-MORL, a decomposition-based MORL framework that uses a hypernetwork to generate policy parameters conditioned on a preference vector $\bm{\omega}$, yielding a continuum of personalized policies $\pi_{\phi(\bm{\omega})}$ without retraining for each weight. The paper provides theoretical analysis using Rademacher complexity and contraction mappings to establish higher model capacity and the existence of a unique fixed point for the optimal multi-objective value function. Empirically, PSL-MORL achieves superior hypervolume and sparsity on MO-MuJoCo and Fruit Tree Navigation benchmarks compared to seven MORL baselines, demonstrating its practical impact for dense and personalized Pareto front approximation.

Abstract

Multi-objective decision-making problems have emerged in numerous real-world scenarios, such as video games, navigation and robotics. Considering the clear advantages of Reinforcement Learning (RL) in optimizing decision-making processes, researchers have delved into the development of Multi-Objective RL (MORL) methods for solving multi-objective decision problems. However, previous methods either cannot obtain the entire Pareto front, or employ only a single policy network for all the preferences over multiple objectives, which may not produce personalized solutions for each preference. To address these limitations, we propose a novel decomposition-based framework for MORL, Pareto Set Learning for MORL (PSL-MORL), that harnesses the generation capability of hypernetwork to produce the parameters of the policy network for each decomposition weight, generating relatively distinct policies for various scalarized subproblems with high efficiency. PSL-MORL is a general framework, which is compatible for any RL algorithm. The theoretical result guarantees the superiority of the model capacity of PSL-MORL and the optimality of the obtained policy network. Through extensive experiments on diverse benchmarks, we demonstrate the effectiveness of PSL-MORL in achieving dense coverage of the Pareto front, significantly outperforming state-of-the-art MORL methods in the hypervolume and sparsity indicators.

Figures (1)

• Figure 1: Illustration of the proposed PSL-MORL method. The ultimate parameters of the policy network is composed of two portions, one is the parameters $\theta_2=\phi(\bm{\omega})$ generated by the hypernetwork, and the other is the initial parameters $\theta_1$ of the policy network. The left part is the input of the whole PSL-MORL, i.e., the preference randomly sampled from the uniform distribution. The middle part is the two portions, and the right part is the parameter fusion. Through mixing the parameters, we can get the final policy network and derive the loss to update our hypernetwork and the policy network. The output is the optimal parameters for our hypernetwork and the policy network parameters $\theta_1$.

Theorems & Definitions (9)

• Definition 1: Pareto Dominance
• Definition 2: Pareto Optimality
• Definition 3: Pareto Set/Front
• Theorem 1
• Definition 4
• Theorem 2
• Definition 5
• Lemma 1
• Lemma 2