MO-Playground: Massively Parallelized Multi-Objective Reinforcement Learning for Robotics

TL;DR

This work presents MORLAX, a new GPU-native, fast MORL algorithm, and MO-Playground, a pip-installable playground of GPU-accelerated multi-objective environments, offering 25-270x speed-ups compared to legacy CPU-based approaches whilst achieving superior Pareto front hypervolumes.

Abstract

Multi-objective reinforcement learning (MORL) is a powerful tool to learn Pareto-optimal policy families across conflicting objectives. However, unlike traditional RL algorithms, existing MORL algorithms do not effectively leverage large-scale parallelization to concurrently simulate thousands of environments, resulting in vastly increased computation time. Ultimately, this has limited MORL's application towards complex multi-objective robotics problems. To address these challenges, we present 1) MORLAX, a new GPU-native, fast MORL algorithm, and 2) MO-Playground, a pip-installable playground of GPU-accelerated multi-objective environments. Together, MORLAX and MO-Playground approximate Pareto sets within minutes, offering 25-270x speed-ups compared to legacy CPU-based approaches whilst achieving superior Pareto front hypervolumes. We demonstrate the versatility of our approach by implementing a custom BRUCE humanoid robot environment using MO-Playground and learning Pareto-optimal locomotion policies across 6 realistic objectives for BRUCE, such as smoothness, efficiency and arm swinging.

Figures (4)

• Figure 1: MO-Playground provides a suite of multi-objective environments, a framework for easily creating custom environments, and fast Pareto set approximation via GPU-accelerated massive parallelization and parameter-efficient hypernetworks.
• Figure 2: MORLAX Architecture. (a) showcases evaluating the policy hypernetwork $\mathcal{H}_\pi$ by picking a trade-off $\bm{w}$, evaluating $\mathcal{H}_\pi$, and rolling out the extracted policy in the environment. (b) showcases how MORLAX trains $\mathcal{H}_\pi$ via a parallelized sample-rollout-update procedure.
• Figure 3: MORLaX finds Pareto sets that achieve better-performing, diverse behavior compared to the baseline, HYPER-MORL hypermorl, but significantly (21-270x) faster. To visualize each environment's objectives, Pareto fronts along with composite figures of a single MORLAX policy hypernetwork are shown across extreme input trade-offs (e.g. maximum speed vs. maximum energy efficiency in MOCheetah). The displayed trajectories are generated via policies highlighted on the Pareto Fronts.
• Figure 4: We project the learned six-dimensional Pareto front representing 30,720 policies tracking a moderate, forward velocity onto two 3D plots, each with different objectives. The colors represent trade-off vectors, and are blended according to the color-coded objectives along each axis. $\pi_1$ represents a policy with arm swinging, $\pi_2$ represents a policy with rigid arms, and $\pi_3$ represents a policy with maximum smoothness. One interesting finding is that $\pi_1$ walks faster and has higher efficiency, likely an emergent benefit of arm-swing.