AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model

TL;DR

AlignDiff tackles the problem of aligning RL agents with diverse and mutable human preferences by combining RLHF-based quantification of attribute strengths with a diffusion-model planner that can be steered at inference time for zero-shot customization. The method introduces multi-perspective human feedback, trains a transformer-based attribute-strength model, relabels data, and learns an attribute-conditioned diffusion model with a DiT backbone to plan trajectories that match target attribute vectors. Empirical results on Hopper, Walker, and Humanoid locomotion demonstrate superior performance in preference matching, rapid switching between behaviors, and coverage of diverse behavior distributions, including zero-shot tasks guided by human instructions. The work highlights a practical path toward human-AI collaboration via controllable, diffusion-driven planning, while noting inference-time efficiency and potential language-based control as avenues for future improvement.

Abstract

Aligning agent behaviors with diverse human preferences remains a challenging problem in reinforcement learning (RL), owing to the inherent abstractness and mutability of human preferences. To address these issues, we propose AlignDiff, a novel framework that leverages RL from Human Feedback (RLHF) to quantify human preferences, covering abstractness, and utilizes them to guide diffusion planning for zero-shot behavior customizing, covering mutability. AlignDiff can accurately match user-customized behaviors and efficiently switch from one to another. To build the framework, we first establish the multi-perspective human feedback datasets, which contain comparisons for the attributes of diverse behaviors, and then train an attribute strength model to predict quantified relative strengths. After relabeling behavioral datasets with relative strengths, we proceed to train an attribute-conditioned diffusion model, which serves as a planner with the attribute strength model as a director for preference aligning at the inference phase. We evaluate AlignDiff on various locomotion tasks and demonstrate its superior performance on preference matching, switching, and covering compared to other baselines. Its capability of completing unseen downstream tasks under human instructions also showcases the promising potential for human-AI collaboration. More visualization videos are released on https://aligndiff.github.io/.

Figures (13)

• Figure 1: AlignDiff achieves zero-shot human preferences aligning. Here, three robots continuously switch their behaviors based on human instructions, where $\phi$ represents no effect.
• Figure 2: Overview of AlignDiff. We begin by collecting human feedback through crowdsourcing, which is then used to train an attribute strength model $\hat{\zeta}_\theta^{\bm\alpha}$. Relabeled by it, an annotated dataset is then used to train an attribute-conditioned diffusion model $\epsilon_\phi$. With these two components, we can use AlignDiff to conduct preference alignment planning.
• Figure 3: AlignDiff, trained solely on a locomotion dataset, demonstrates the capability to accomplish unseen downstream tasks through human instructions.
• Figure 4: MAE curves. The vertical axis represents the MAE threshold, which is presented using a logarithmic scale. The horizontal axis represents the percentage of samples below the MAE threshold. Each point $(x, y)$ on the curve indicates that the algorithm has a probability of $y$ to achieve an MAE between the agent's relative attribute strength and the desired value below $x$. A larger area enclosed by the curve and the axes indicate better performance in matching human preferences.
• Figure 5: Distribution plots of attribute strength values and actual attributes are shown. The horizontal axis represents the attribute strength value $v$, while the vertical axis represents the corresponding actual attribute $u$. $p(u,v)$ denotes the probability that, given a target attribute strength value $v$, the algorithm can produce a trajectory with the actual attribute $u$. The color gradient in the distribution plot represents the probability $p(u,v)$, ranging from dark to light.