Robust Reward Alignment via Hypothesis Space Batch Cutting

TL;DR

This work introduces HSBC, a robust reward alignment framework that maintains a hypothesis space of reward models and iteratively refines it via batch-based cuts induced by disagreement-driven human preferences. By aggregating batch votes and applying a conservative cutting strategy governed by a conservativeness level γ, HSBC achieves provable robustness to unknown false preferences while bounding human query complexity with PAC-like guarantees. The method relies on a disagreement-based querying protocol and a sampling-based MPC workflow to generate trajectory ensembles and compute batch constraints f(θ, ξ^0, ξ^1, y) for hypothesis-space updates. Across six control tasks and various noise settings, HSBC matches or surpasses state-of-the-art baselines on clean data and significantly outperforms them under high rates of erroneous feedback, demonstrating strong practical impact for robotics and human-in-the-loop decision-making. The experiments also reveal insightful trade-offs with γ, η, N, and M and show HSBC’s resilience on real human data, underscoring its interpretability and applicability to real-world preference elicitation.

Abstract

Reward design in reinforcement learning and optimal control is challenging. Preference-based alignment addresses this by enabling agents to learn rewards from ranked trajectory pairs provided by humans. However, existing methods often struggle from poor robustness to unknown false human preferences. In this work, we propose a robust and efficient reward alignment method based on a novel and geometrically interpretable perspective: hypothesis space batched cutting. Our method iteratively refines the reward hypothesis space through "cuts" based on batches of human preferences. Within each batch, human preferences, queried based on disagreement, are grouped using a voting function to determine the appropriate cut, ensuring a bounded human query complexity. To handle unknown erroneous preferences, we introduce a conservative cutting method within each batch, preventing erroneous human preferences from making overly aggressive cuts to the hypothesis space. This guarantees provable robustness against false preferences, while eliminating the need to explicitly identify them. We evaluate our method in a model predictive control setting across diverse tasks. The results demonstrate that our framework achieves comparable or superior performance to state-of-the-art methods in error-free settings while significantly outperforming existing methods when handling a high percentage of erroneous human preferences.

Figures (9)

• Figure 1: Illustration of update from $\Theta_i$ to $\Theta_{i+1}$. Three constraints are induced from a preference batch of size 3, with the simplified notation $f_{i,j}(\boldsymbol{\theta}) = f(\boldsymbol{\theta}, \boldsymbol{\xi}^0_{i,j}, \boldsymbol{\xi}^1_{i,j}, y_{i,j})$. Red arrows are the directions of constraints, i.e., the region of $\{\boldsymbol{\theta}|f_{i,j}(\boldsymbol{\theta}) \geq 0\}$. New $\Theta_{i+1}$ is the regions in $\Theta_i$ satisfying all constraints.
• Figure 2: Illustration of disagreement-based preference query. Regardless of the human actual preference label (which only determines which side is to cut), the disagreement condition will always guarantee a least a portion of the hypothesis space is removed.
• Figure 3: Geometry Interpretation of the robust HSBC with false preferences. Red arrows are the direction of the constraints, i.e., the $\boldsymbol{\theta}$ region satisfying $\{\boldsymbol{\theta}|f_{i,j}(\boldsymbol{\theta}) \geq 0\}$. Left: $y_{i,3}^{\text{false}}$ is false human preference, and thus simply taking the intersection will cut out $\boldsymbol{\theta}_H$. Right: in robust HSBC, the regions with $V_i (\boldsymbol{\theta}) \geq 2$ are kept, thus $\boldsymbol{\theta}_H$ is still contained in the new hypothesis space $\Theta_{i+1}$.
• Figure 4: Task environments of experiments.
• Figure 5: Learning curves for different tasks (rows) under different rates (columns) of false human preferences. The conservativeness level in HSBC Algorithm is the same as actual human false rate. All results are reported over 5 runs. The results show our method significantly outperforms the baseline when the false rate is high, and has a comparable performance when the false rate is 0 (no false preference).

Theorems & Definitions (8)

• Lemma 4.1
• Theorem 4.2
• Lemma 5.1
• Lemma 5.2
• proof
• proof
• proof
• proof