Theoretical Study of Conflict-Avoidant Multi-Objective Reinforcement Learning
Yudan Wang, Peiyao Xiao, Hao Ban, Kaiyi Ji, Shaofeng Zou
TL;DR
This work tackles gradient conflicts in single-policy multi-task reinforcement learning by introducing MTAC, a dynamic-weighted actor-critic framework with two task-weight-update modes: CA (conflict-avoidant) and FC (fast-convergence). The authors establish finite-time convergence guarantees, showing MTAC-CA achieves an $\epsilon+\epsilon_{\text{app}}$-Pareto stationary policy in $\mathcal{O}(\epsilon^{-5})$ samples with an $\epsilon+\sqrt{\epsilon_{\text{app}}}$ CA distance, while MTAC-FC reduces this to $\mathcal{O}(\epsilon^{-3})$ samples at the cost of a constant CA distance. The analysis handles biased gradient estimates from function approximation by introducing a surrogate CA direction, enabling decomposition of CA-gap into critic- and approximation-related errors. Empirical results on the MT10 benchmark show MTAC-CA outperforms fixed-priority MTRL baselines, validating the practical benefits of dynamic weighting for multi-task RL.
Abstract
Multi-task reinforcement learning (MTRL) has shown great promise in many real-world applications. Existing MTRL algorithms often aim to learn a policy that optimizes individual objective functions simultaneously with a given prior preference (or weights) on different tasks. However, these methods often suffer from the issue of \textit{gradient conflict} such that the tasks with larger gradients dominate the update direction, resulting in a performance degeneration on other tasks. In this paper, we develop a novel dynamic weighting multi-task actor-critic algorithm (MTAC) under two options of sub-procedures named as CA and FC in task weight updates. MTAC-CA aims to find a conflict-avoidant (CA) update direction that maximizes the minimum value improvement among tasks, and MTAC-FC targets at a much faster convergence rate. We provide a comprehensive finite-time convergence analysis for both algorithms. We show that MTAC-CA can find a $ε+ε_{\text{app}}$-accurate Pareto stationary policy using $\mathcal{O}({ε^{-5}})$ samples, while ensuring a small $ε+\sqrt{ε_{\text{app}}}$-level CA distance (defined as the distance to the CA direction), where $ε_{\text{app}}$ is the function approximation error. The analysis also shows that MTAC-FC improves the sample complexity to $\mathcal{O}(ε^{-3})$, but with a constant-level CA distance. Our experiments on MT10 demonstrate the improved performance of our algorithms over existing MTRL methods with fixed preference.