Interpretability by Design for Efficient Multi-Objective Reinforcement Learning

TL;DR

This work tackles multi-objective reinforcement learning by proposing LLE-MORL, a method that achieves interpretability by design through a locally linear Parameter-Performance Relationship (PPR) between policy parameters and objective outcomes. By performing directional retraining to identify local tangent directions and then extending along those directions without retraining, the approach densely and efficiently traces the Pareto front while maintaining an interpretable mapping from parameter changes to trade-offs. The authors provide theoretical guarantees (Lipschitz continuity and local manifold lift) and demonstrate empirically that LLE-MORL yields wider, higher-quality Pareto fronts with strong sample efficiency across 2D and higher-dimensional objectives, outperforming several MORL baselines. The framework is modular and training-free in the extension step, enabling easy integration as a plug-in to existing PPO-based MORL pipelines and offering practical benefits for rapid policy adaptation in changing preferences.

Abstract

Multi-objective reinforcement learning (MORL) aims at optimising several, often conflicting goals to improve the flexibility and reliability of RL in practical tasks. This is typically achieved by finding a set of diverse, non-dominated policies that form a Pareto front in the performance space. We introduce LLE-MORL, an approach that achieves interpretability by design by utilising a training scheme based on the local relationship between the parameter space and the performance space. By exploiting a locally linear map between these spaces, our method provides an interpretation of policy parameters in terms of the objectives, and this structured representation enables an efficient search within contiguous solution domains, allowing for the rapid generation of high-quality solutions without extensive retraining. Experiments across diverse continuous control domains demonstrate that LLE-MORL consistently achieves higher Pareto front quality and efficiency than state-of-the-art approaches.

Figures (5)

• Figure 1: Parameter space and performance space of the Pareto policies. (Left) 2D projection of the high‑dimensional policy parameter space. Red and blue gradient shadings and contour lines depict the scalarised reward under different preference vectors $\omega_1$ and $\omega_2$. The arrow marks a short retraining update. (Right) The policy obtained by retraining the $\theta_{w_1}$ model under $\omega_2$ (green) shifts towards the new preference as seen in performance space.
• Figure 2: Comparing independently trained policy $\theta_{w_2}$ versus retrained policy $\theta_{w'}$ based on $\theta_{w_1}$, for details see Section \ref{['valhom']}. The environment used here is the multi-objective Swimmer problem.
• Figure 3: Visualisation of the process of applying the parameter difference $\Delta\theta = \theta_{w'} - \theta_{w}$ between two related policies. The policies are obtained by first training a policy $\theta_{w}$ to a stable stage using scalarization vector $w$ and then find policy $\theta_{w'}$ by a brief additional training period with a different scalarization vector $w'$. Iterating the shift $\Delta\theta$ in the policy space induces a sequence of shifts also in the multi-objective reward space. The subfigures show results for different initial preferences: (a) A convex front is found from the two policies. (b) Although the original policy turns out to be Pareto suboptimal, the solution manifold extends into a Pareto optimal component. (c) Retraining can cause the (Pareto-suboptimal) original solution to jump to a different branch so that the corresponding solution consists of two components one of which can be ignored because of Pareto suboptimality.
• Figure 4: Pareto fronts from 2 different settings, comparing our LLE-MORL method with baselines on three 2d continuous-control benchmarks. LLE-MORL consistently achieves wider coverage and closer proximity to the true Pareto front.
• Figure 5: Manifold Map between Pareto front manifold and its local lifting. With each initialization $\theta^{(0)}$, we denote its performance $v$ and the neighbour on the Pareto front manifold to be $P_v$. Based on our assumption and conditions, the local lifting, i.e. $\Theta_v$ is also a manifold of the same dimension of $P_v$, located in the neighbourhood of $\theta^{(0)}$. The Linear extrapolation corresponds to the approximation of the tangent space of $\Theta_v$. Each time, LLE-MORL reconstruct the Pareto front branch where $v$ is located. With the appropriate choices of initialization $\theta_k^{(0)}$, it reconstructs all of the branches where the corresponding $v_k=V(\theta_k^{(0)})$ is located.

Theorems & Definitions (16)

• Definition 2.1
• Definition 2.2
• Definition 3.1
• Theorem 3.2
• Corollary 3.3
• Theorem 3.4
• Theorem 3.6
• Definition 2.1
• Theorem 2.4
• Corollary 2.5