Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hindsight Preference Replay Improves Preference-Conditioned Multi-Objective Reinforcement Learning

Jonaid Shianifar, Michael Schukat, Karl Mason

TL;DR

This work addresses sample inefficiency in preference-conditioned MORL by introducing Hindsight Preference Replay (HPR), a simple replay augmentation that retroactively relabels stored transitions with alternative preference vectors while preserving CAPQL’s architecture. HPR-CAPQL densifies supervision across the preference simplex and yields broader Pareto fronts, with notable gains in hypervolume and expected utility on six MO-Gymnasium tasks under a fixed budget; the most dramatic improvements occur in mo-humanoid-v5, mo-walker2d-v5, and mo-ant-2obj-v5. While most environments benefit, some tasks like mo-halfcheetah-v5 show CAPQL maintaining the HV lead, and mo-swimmer-v5 often exhibits near-draw results, highlighting a coverage–density trade-off. Overall, HPR provides a practical, low-overhead boost to MORL performance by reusing off-policy data across preferences, with implications for scalable, preference-aware control in real-world systems.

Abstract

Multi-objective reinforcement learning (MORL) enables agents to optimize vector-valued rewards while respecting user preferences. CAPQL, a preference-conditioned actor-critic method, achieves this by conditioning on weight vectors w and restricts data usage to the specific preferences under which it was collected, leaving off-policy data from other preferences unused. We introduce Hindsight Preference Replay (HPR), a simple and general replay augmentation strategy that retroactively relabels stored transitions with alternative preferences. This densifies supervision across the preference simplex without altering the CAPQL architecture or loss functions. Evaluated on six MO-Gymnasium locomotion tasks at a fixed 300000-step budget using expected utility (EUM), hypervolume (HV), and sparsity, HPR-CAPQL improves HV in five of six environments and EUM in four of six. On mo-humanoid-v5, for instance, EUM rises from $323\!\pm\!125$ to $1613\!\pm\!464$ and HV from 0.52M to 9.63M, with strong statistical support. mo-halfcheetah-v5 remains a challenging exception where CAPQL attains higher HV at comparable EUM. We report final summaries and Pareto-front visualizations across all tasks.

Hindsight Preference Replay Improves Preference-Conditioned Multi-Objective Reinforcement Learning

TL;DR

This work addresses sample inefficiency in preference-conditioned MORL by introducing Hindsight Preference Replay (HPR), a simple replay augmentation that retroactively relabels stored transitions with alternative preference vectors while preserving CAPQL’s architecture. HPR-CAPQL densifies supervision across the preference simplex and yields broader Pareto fronts, with notable gains in hypervolume and expected utility on six MO-Gymnasium tasks under a fixed budget; the most dramatic improvements occur in mo-humanoid-v5, mo-walker2d-v5, and mo-ant-2obj-v5. While most environments benefit, some tasks like mo-halfcheetah-v5 show CAPQL maintaining the HV lead, and mo-swimmer-v5 often exhibits near-draw results, highlighting a coverage–density trade-off. Overall, HPR provides a practical, low-overhead boost to MORL performance by reusing off-policy data across preferences, with implications for scalable, preference-aware control in real-world systems.

Abstract

Multi-objective reinforcement learning (MORL) enables agents to optimize vector-valued rewards while respecting user preferences. CAPQL, a preference-conditioned actor-critic method, achieves this by conditioning on weight vectors w and restricts data usage to the specific preferences under which it was collected, leaving off-policy data from other preferences unused. We introduce Hindsight Preference Replay (HPR), a simple and general replay augmentation strategy that retroactively relabels stored transitions with alternative preferences. This densifies supervision across the preference simplex without altering the CAPQL architecture or loss functions. Evaluated on six MO-Gymnasium locomotion tasks at a fixed 300000-step budget using expected utility (EUM), hypervolume (HV), and sparsity, HPR-CAPQL improves HV in five of six environments and EUM in four of six. On mo-humanoid-v5, for instance, EUM rises from to and HV from 0.52M to 9.63M, with strong statistical support. mo-halfcheetah-v5 remains a challenging exception where CAPQL attains higher HV at comparable EUM. We report final summaries and Pareto-front visualizations across all tasks.
Paper Structure (19 sections, 1 equation, 4 figures, 2 tables, 1 algorithm)

This paper contains 19 sections, 1 equation, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Literature Review
  3. Methodology: Hindsight Preference Replay (HPR)
  4. Relabeling Strategies
  5. Return-Aligned Relabeling.
  6. Neighborhood Sampling.
  7. Integration with CAPQL
  8. Practical Settings and Filters
  9. Experimental Setup
  10. Environments
  11. Training Protocol
  12. Hardware Configuration
  13. Results and Discussion
  14. Final Performance Summary
  15. Learning Behavior Over Time
  16. ...and 4 more sections

Figures (4)

  • Figure 1: EUM learning curves up to $300{,}000$ steps (mean $\pm$ std over five seeds).
  • Figure 2: Hypervolume (HV) learning curves up to $300{,}000$ steps (mean $\pm$ std over five seeds).
  • Figure 3: Sparsity (lower is better) up to $300{,}000$ steps (mean $\pm$ std over five seeds).
  • Figure 4: Final Pareto fronts (non-dominated only) after $300{,}000$ steps across all environments. Each panel shows the union of non-dominated solutions across five seeds; legend: CAPQL vs. HPR-CAPQL