Deep Multi-Objective Reinforcement Learning for Utility-Based Infrastructural Maintenance Optimization

TL;DR

This work advances infrastructural maintenance optimization by introducing MO-DCMAC, a utility-based MORL framework that directly optimizes policies for multiple objectives under a known non-linear utility using the ESR criterion. By integrating a distributed, multi-agent actor-critic with a distributional critic, MO-DCMAC can handle high-dimensional, partially observable maintenance problems and leverage non-trivial utilities such as FMECA. The method is validated on three Amsterdam quay-wall-inspired environments, showing superior performance over heuristic baselines in cost and reliability across two utilities, while also discussing stability and scalability challenges. The results suggest substantial practical impact for risk-aware, data-driven maintenance planning and point to future work on scaling objectives, incorporating graph-based representations, and exploring constrained or dynamic utility formulations.

Abstract

In this paper, we introduce Multi-Objective Deep Centralized Multi-Agent Actor-Critic (MO- DCMAC), a multi-objective reinforcement learning (MORL) method for infrastructural maintenance optimization, an area traditionally dominated by single-objective reinforcement learning (RL) approaches. Previous single-objective RL methods combine multiple objectives, such as probability of collapse and cost, into a singular reward signal through reward-shaping. In contrast, MO-DCMAC can optimize a policy for multiple objectives directly, even when the utility function is non-linear. We evaluated MO-DCMAC using two utility functions, which use probability of collapse and cost as input. The first utility function is the Threshold utility, in which MO-DCMAC should minimize cost so that the probability of collapse is never above the threshold. The second is based on the Failure Mode, Effects, and Criticality Analysis (FMECA) methodology used by asset managers to asses maintenance plans. We evaluated MO-DCMAC, with both utility functions, in multiple maintenance environments, including ones based on a case study of the historical quay walls of Amsterdam. The performance of MO-DCMAC was compared against multiple rule-based policies based on heuristics currently used for constructing maintenance plans. Our results demonstrate that MO-DCMAC outperforms traditional rule-based policies across various environments and utility functions.

Figures (6)

• Figure 1: This figure illustrates the training sequence for MO-DCMAC, whereby the blue nodes are the input layers, the red hidden layers, and the green output layers. Algorithm \ref{['algo:mo-dcmac']} is the pseudocode of MO-DCMAC and shows how each equation is used. Step (1) involves the actor sampling actions for each component and a collective global action from the current belief state $\mathbf{b}_t$. In step (2), these actions are processed by the environment to determine the subsequent belief state $\mathbf{b}_{t+1}$ and to generate the reward vector $\vec{r}_t$. This reward is then used with the accrued reward $\vec{R}^{-}_{t}$ for further calculations. Step (3) has the critic network predicting the joint distribution of the returns across objectives, denoted as $Z_{\psi}(\vec{z} \mid \mathbf{b}_t)$ for the current state and $Z_{\psi}(\vec{z} \mid \mathbf{b}_{t+1})$ for the next. In step (4), these predictions, along with the reward vector $\vec{r}_t$ and accrued reward $\vec{R}^{-}_{t}$, are used to compute the preference scores for the current $u_{t}$ and next state $u_{t+1}$. Step (5) employs these preference scores to calculate the advantage of the state $A_{t}$. Finally, step (6) updates the critic and actor. The overall loss combines actor and critic losses with an entropy bonus. Step (6) can be done at every step but can also be done at every $\eta$ step, whereby the actor and critic are updated from the previous $\eta$ steps.
• Figure 2: Representation of the Quay Wall Asset environment. Maintenance will only be done for the wooden components.
• Figure 3: The figure shows how we smoothed out the objective scoring for the FMECA utility. The dashed line is the normal, binned scoring from the original FMECA, and the solid line is the scoring we used to improve the training of MO-DCMAC. The increase from 5 to 10 is done at $x_{\max}$.
• Figure 4: The training plots of MO-DCMAC with the FMECA utility function. The line shows the mean over 5 runs, with the shaded area being the standard deviation.
• Figure 5: The training plots of MO-DCMAC with the Threshold utility function. The line shows the mean over 5 runs, with the shaded area being the standard deviation.