Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Human-in-the-Loop Policy Optimization for Preference-Based Multi-Objective Reinforcement Learning

Ke Li, Han Guo

TL;DR

This work addresses the challenge of preference-lean MORL by introducing a human-in-the-loop framework (CBOB) that interactively learns the DM's implicit preferences and steers policy search toward policies of interest without prior knowledge. It combines seeding with MOEA/D-like decomposition, a Gaussian-process-based preference model, and a preference-driven translation step to bias policy optimization via MOPPO, enabling efficient discovery of DM-aligned non-dominated policies. Experiments on MuJoCo and MMSD demonstrate that CBOB consistently outperforms or matches state-of-the-art conventional and preference-based MORL baselines in locating ROI-focused policies, while reducing extraneous trade-offs. The approach offers a flexible, plug-in framework with promising implications for human-AI collaboration in complex, high-dimensional MORL tasks.

Abstract

Multi-objective reinforcement learning (MORL) aims to find a set of high-performing and diverse policies that address trade-offs between multiple conflicting objectives. However, in practice, decision makers (DMs) often deploy only one or a limited number of trade-off policies. Providing too many diversified trade-off policies to the DM not only significantly increases their workload but also introduces noise in multi-criterion decision-making. With this in mind, we propose a human-in-the-loop policy optimization framework for preference-based MORL that interactively identifies policies of interest. Our method proactively learns the DM's implicit preference information without requiring any a priori knowledge, which is often unavailable in real-world black-box decision scenarios. The learned preference information is used to progressively guide policy optimization towards policies of interest. We evaluate our approach against three conventional MORL algorithms that do not consider preference information and four state-of-the-art preference-based MORL algorithms on two MORL environments for robot control and smart grid management. Experimental results fully demonstrate the effectiveness of our proposed method in comparison to the other peer algorithms.

Human-in-the-Loop Policy Optimization for Preference-Based Multi-Objective Reinforcement Learning

TL;DR

This work addresses the challenge of preference-lean MORL by introducing a human-in-the-loop framework (CBOB) that interactively learns the DM's implicit preferences and steers policy search toward policies of interest without prior knowledge. It combines seeding with MOEA/D-like decomposition, a Gaussian-process-based preference model, and a preference-driven translation step to bias policy optimization via MOPPO, enabling efficient discovery of DM-aligned non-dominated policies. Experiments on MuJoCo and MMSD demonstrate that CBOB consistently outperforms or matches state-of-the-art conventional and preference-based MORL baselines in locating ROI-focused policies, while reducing extraneous trade-offs. The approach offers a flexible, plug-in framework with promising implications for human-AI collaboration in complex, high-dimensional MORL tasks.

Abstract

Multi-objective reinforcement learning (MORL) aims to find a set of high-performing and diverse policies that address trade-offs between multiple conflicting objectives. However, in practice, decision makers (DMs) often deploy only one or a limited number of trade-off policies. Providing too many diversified trade-off policies to the DM not only significantly increases their workload but also introduces noise in multi-criterion decision-making. With this in mind, we propose a human-in-the-loop policy optimization framework for preference-based MORL that interactively identifies policies of interest. Our method proactively learns the DM's implicit preference information without requiring any a priori knowledge, which is often unavailable in real-world black-box decision scenarios. The learned preference information is used to progressively guide policy optimization towards policies of interest. We evaluate our approach against three conventional MORL algorithms that do not consider preference information and four state-of-the-art preference-based MORL algorithms on two MORL environments for robot control and smart grid management. Experimental results fully demonstrate the effectiveness of our proposed method in comparison to the other peer algorithms.
Paper Structure (33 sections, 8 equations, 11 figures, 7 tables, 2 algorithms)

This paper contains 33 sections, 8 equations, 11 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Multi-Objective Markov Decision Process
  4. Multi-Objective Optimization
  5. Related Works
  6. Single-policy MORL
  7. Multi-policy MORL
  8. IRL and Preference Learning
  9. Proposed Method
  10. Seeding Promising Policies
  11. Preference Elicitation
  12. How to collect the DM's feedback
  13. How to model the DM's preference information
  14. How to leverage the learned preference information
  15. Policy Optimization
  16. ...and 18 more sections

Figures (11)

  • Figure 1: Flowchart of CBOB. It iterates between the preference elicitation and the policy optimization modules until the stopping criterion is met, and outputs the preferred policies.
  • Figure 2: Illustration of the weight vector adjustment in Step 4. (a) The ROI is highlighted as the shaded cone region versus the evenly distributed weight vectors (denoted as $\newmoon$). (b) Adjusted weight vectors towards the ROI (denoted as $\newmoon$).
  • Figure 3: Plots of non-dominated policies obtained by CBOB versus PGMORL, RA, and MOIA with different preferences.
  • Figure 4: Selected plots of non-dominated policies obtained by CBOB vs MORL-Adaptation, META-MORL, MOMPO and MORAL.
  • Figure 5: Comparison of the weight vector specified by the DM a priori versus the ROI identified by CBOB (shaded in the gray region).
  • ...and 6 more figures

Theorems & Definitions (12)

  • Definition 1
  • Definition 2
  • Definition 3
  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Remark 5
  • Remark 6
  • Remark 7
  • ...and 2 more