Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VLP: Vision-Language Preference Learning for Embodied Manipulation

Runze Liu, Chenjia Bai, Jiafei Lyu, Shengjie Sun, Yali Du, Xiu Li

TL;DR

This work tackles reward engineering in reinforcement learning by introducing VLP, a vision-language preference learning framework that learns a trajectory-level preference model from language-conditioned cues. It builds MTVLP, a multi-task dataset combining videos, task-language instructions, and implicit preferences, enabling robust cross-modal alignment via a cross-modal transformer. By incorporating three language-conditioned preference types (ITP, ILP, IVP) and a multi-term loss, VLP achieves accurate, generalizable preferences that can drive reward learning or direct policy optimization, often outperforming vision-language reward baselines and reducing labeling costs. The approach demonstrates strong generalization to unseen tasks and language instructions in simulated embodied manipulation, with attention maps showing language-conditioned focus on relevant scene regions. Overall, VLP advances efficient, scalable learning for embodied agents by leveraging language-conditioned vision-language alignment to produce high-quality preference signals.

Abstract

Reward engineering is one of the key challenges in Reinforcement Learning (RL). Preference-based RL effectively addresses this issue by learning from human feedback. However, it is both time-consuming and expensive to collect human preference labels. In this paper, we propose a novel \textbf{V}ision-\textbf{L}anguage \textbf{P}reference learning framework, named \textbf{VLP}, which learns a vision-language preference model to provide preference feedback for embodied manipulation tasks. To achieve this, we define three types of language-conditioned preferences and construct a vision-language preference dataset, which contains versatile implicit preference orders without human annotations. The preference model learns to extract language-related features, and then serves as a preference annotator in various downstream tasks. The policy can be learned according to the annotated preferences via reward learning or direct policy optimization. Extensive empirical results on simulated embodied manipulation tasks demonstrate that our method provides accurate preferences and generalizes to unseen tasks and unseen language instructions, outperforming the baselines by a large margin.

VLP: Vision-Language Preference Learning for Embodied Manipulation

TL;DR

This work tackles reward engineering in reinforcement learning by introducing VLP, a vision-language preference learning framework that learns a trajectory-level preference model from language-conditioned cues. It builds MTVLP, a multi-task dataset combining videos, task-language instructions, and implicit preferences, enabling robust cross-modal alignment via a cross-modal transformer. By incorporating three language-conditioned preference types (ITP, ILP, IVP) and a multi-term loss, VLP achieves accurate, generalizable preferences that can drive reward learning or direct policy optimization, often outperforming vision-language reward baselines and reducing labeling costs. The approach demonstrates strong generalization to unseen tasks and language instructions in simulated embodied manipulation, with attention maps showing language-conditioned focus on relevant scene regions. Overall, VLP advances efficient, scalable learning for embodied agents by leveraging language-conditioned vision-language alignment to produce high-quality preference signals.

Abstract

Reward engineering is one of the key challenges in Reinforcement Learning (RL). Preference-based RL effectively addresses this issue by learning from human feedback. However, it is both time-consuming and expensive to collect human preference labels. In this paper, we propose a novel \textbf{V}ision-\textbf{L}anguage \textbf{P}reference learning framework, named \textbf{VLP}, which learns a vision-language preference model to provide preference feedback for embodied manipulation tasks. To achieve this, we define three types of language-conditioned preferences and construct a vision-language preference dataset, which contains versatile implicit preference orders without human annotations. The preference model learns to extract language-related features, and then serves as a preference annotator in various downstream tasks. The policy can be learned according to the annotated preferences via reward learning or direct policy optimization. Extensive empirical results on simulated embodied manipulation tasks demonstrate that our method provides accurate preferences and generalizes to unseen tasks and unseen language instructions, outperforming the baselines by a large margin.

Paper Structure

This paper contains 46 sections, 4 equations, 4 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Problem Setting.
  4. Preference-based RL.
  5. Method
  6. Model and Dataset
  7. Model Architecture.
  8. Vision-Language Preference Dataset.
  9. Vision-Language Preference Alignment
  10. Language-conditioned Preferences.
  11. Vision-Language Preference Learning.
  12. Related Work
  13. Vision-Language Models for Reinforcement Learning.
  14. Preference-based Reinforcement Learning.
  15. Experiments
  16. ...and 31 more sections

Figures (4)

  • Figure 1: Comparison of VLP (right) with previous methods (left) of providing preference labels.
  • Figure 2: (a) Trajectory videos and language instruction are fed into the preference model to obtain a trajectory-wise preference score. (b) The cross-modal transformer obtains language-related video features and video-related language features by cross-attention mechanism.
  • Figure 3: Five simulated robotic manipulation tasks used for experimental evaluation.
  • Figure 4: Attention map visualization of Drawer Close and Door Close. The language instruction is shown at the bottom of each subfigure.