Human-assisted Robotic Policy Refinement via Action Preference Optimization

TL;DR

This paper tackles the limited post-deployment refinement of Vision-Language-Action (VLA) models by introducing Action Preference Optimization (APO), which combines a human-robot collaboration deployment for data collection with an adaptive reweighting, binary desirability-based preference objective to learn from failure trajectories. APO relaxes the need for perfectly paired preferences by leveraging prospect-theory-inspired utilities and a KL regularization term $z_0 = KL(\pi_\theta || \pi_{ref})$, while using the reward surrogate $r_\theta(o,\hat{a}) = \log \frac{\pi_\theta(\hat{a}|o)}{\pi_{ref}(\hat{a}|o)}$ and a weighted loss $L(\pi_\theta,\pi_{ref})$ guided by $v(o,\hat{a})$. The method introduces an adaptive sample weighting $w_i = l_i/\sum l_i$ and adjusts $\lambda_D=1-e^{-eta_D w}$ and $\lambda_U=e^{-eta_U w}$ to prioritize informative samples, enabling stable updates for autoregressive VLA models. Empirical results in RoboMimic simulation and real-world tasks show APO improves generalization and robustness across diverse manipulation tasks and disruption scenarios, and the approach generalizes to other VLA architectures like $\pi_0$-FAST, with code and datasets released for public use.

Abstract

Establishing a reliable and iteratively refined robotic system is essential for deploying real-world applications. While Vision-Language-Action (VLA) models are widely recognized as the foundation model for such robotic deployment, their reliance on offline expert demonstrations critically limits their capacity for post-deployment refinement. To mitigate this limitation, we introduce Action Preference Optimization (APO), a method designed to refine VLA models by human-assisted preference alignment gathered through interaction with environments. This method begins with a human-robot collaboration framework for reliable failure correction and interaction trajectory collection through human intervention. However, directly leveraging these interaction trajectories for preference optimization is non-trivial due to the challenges of irreversible robotic actions and token distribution mismatch. To solve this, APO proposes an adaptive reweighting algorithm with binary desirability signals derived from interaction, empowering VLA models effectively suppress failure-prone actions while enhancing corrective action adaptation. Ultimately, APO equips VLA models with the crucial capability to learn from failure, paving the way for their iterative refinement and reliable deployment in dynamic environments. The experiments conducted in simulation and real-world scenarios prove superior generalization and robustness of our human-assisted framework across a variety of manipulation tasks. We believe this work could bring insights for efficient and stable optimization of VLA models through human-robot collaboration. The code and dataset are released at https://github.com/GeWu-Lab/Action-Preference-Optimization

Figures (11)

• Figure 1: Our method consists of two key components: (a) the human-robot collaboration deployment framework for reliable deployment and interaction trajectory collection with human intervention. (b) the action preference optimization process with adaptive reweighting for VLA models learning from sub-optimal interaction trajectories. The size of each circle represents its weight during training.
• Figure 2: The demonstration of our human-assisted interaction trajectory.
• Figure 2: The results on disruption scenarios.
• Figure 3: In the position disruption setting, we change the position of the stick from a fixed point to a random position from the rectangle in the Square_D0 task as illustrated in (a). In the background disruption setting, we replace the background with the gray one in the StackThree_D0 task as shown in (b). In the texture disruption setting, we replace the red blocks with the wooden ones.
• Figure 4: Lifelong learning results of APO method.