GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learning

TL;DR

GraphAllocBench introduces a scalable graph-based benchmark for Preference-Conditioned Policy Learning (PCPL) in MORL, leveraging CityPlannerEnv to model graph-structured, city-scale resource allocation with diverse, potentially non-convex fronts. It couples three evaluation metrics—Hypervolume, Proportion of Non-Dominated Solutions (PNDS), and Ordering Score (OS)—to capture magnitude, diversity, and preference-consistency of solutions under varying $N$-objective preferences, where the preferences vector satisfies $\sum_i w_i = 1$. The paper demonstrates that graph-aware policies, particularly Heterogeneous Graph Neural Networks (HGNNs) with preference conditioning, outperform standard MLP baselines, especially on large-scale graphs (e.g., $|D|=|R|=100$). This work provides an extensible, realistic benchmark for advancing PCPL in high-dimensional, combinatorial settings and highlights the value of graph-based representations for complex dependency structures.

Abstract

Preference-Conditioned Policy Learning (PCPL) in Multi-Objective Reinforcement Learning (MORL) aims to approximate diverse Pareto-optimal solutions by conditioning policies on user-specified preferences over objectives. This enables a single model to flexibly adapt to arbitrary trade-offs at run-time by producing a policy on or near the Pareto front. However, existing benchmarks for PCPL are largely restricted to toy tasks and fixed environments, limiting their realism and scalability. To address this gap, we introduce GraphAllocBench, a flexible benchmark built on a novel graph-based resource allocation sandbox environment inspired by city management, which we call CityPlannerEnv. GraphAllocBench provides a rich suite of problems with diverse objective functions, varying preference conditions, and high-dimensional scalability. We also propose two new evaluation metrics -- Proportion of Non-Dominated Solutions (PNDS) and Ordering Score (OS) -- that directly capture preference consistency while complementing the widely used hypervolume metric. Through experiments with Multi-Layer Perceptrons (MLPs) and graph-aware models, we show that GraphAllocBench exposes the limitations of existing MORL approaches and paves the way for using graph-based methods such as Graph Neural Networks in complex, high-dimensional combinatorial allocation tasks. Beyond its predefined problem set, GraphAllocBench enables users to flexibly vary objectives, preferences, and allocation rules, establishing it as a versatile and extensible benchmark for advancing PCPL. Code: https://anonymous.4open.science/r/GraphAllocBench

Figures (7)

• Figure 1: Training and evaluation pipeline using the CityPlannerEnv Gymnasium environment and Proximal Policy Optimization (PPO) agent from Stable Baselines3 stable-baselines3.
• Figure 2: Performance comparison of PCPL and PD-MORL policies over 5 random seeds at 1M steps for HV Ratio, PNDS and OS. Dash marks show the mean, and dots show the outcome for each seed.
• Figure 3: Selected 2-Objective Pareto Fronts for PCPL at 1M steps over 2 random seeds: Compared to the Problem 0 baseline, the RL agent struggles with non-convex (Problem 1c) and multi-segment (Problem 2b) Pareto Fronts.
• Figure 4: PPO architecture with a shared 2-layer MLP feature extractor (128 units, SiLU) for policy and value networks, implemented with Stable Baselines3.
• Figure 5: Performance over different number of objectives (5, 10, 15, 20) for a simple increasing objective function, across 5 random seeds. The points show the mean performance, and the shaded regions show the range over 5 seeds.