Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting the Learning Objectives of Vision-Language Reward Models

Simon Roy, Samuel Barbeau, Giovanni Beltrame, Christian Desrosiers, Nicolas Thome

TL;DR

This paper tackles the problem of comparing vision-language reward models by decoupling the learning objective from model architecture, data, and environments. It evaluates a suite of language-adapted objectives (including TCN, VIP, LIV, R3M, and a simple Triplet loss) under a unified framework using a SigLIP2 backbone and LoRA fine-tuning on Meta-World demonstrations. The key finding is that a straightforward triplet loss can surpass state-of-the-art objectives in both reward-consistency and progress-alignment metrics, suggesting that data and architectural choices rather than learning signals drive much of the reported gains. The results advocate for standardized benchmarks to fairly assess reward-model objectives and highlight the robustness of simple ranking-based losses for transfer to unseen tasks.

Abstract

Learning generalizable reward functions is a core challenge in embodied intelligence. Recent work leverages contrastive vision language models (VLMs) to obtain dense, domain-agnostic rewards without human supervision. These methods adapt VLMs into reward models through increasingly complex learning objectives, yet meaningful comparison remains difficult due to differences in training data, architectures, and evaluation settings. In this work, we isolate the impact of the learning objective by evaluating recent VLM-based reward models under a unified framework with identical backbones, finetuning data, and evaluation environments. Using Meta-World tasks, we assess modeling accuracy by measuring consistency with ground truth reward and correlation with expert progress. Remarkably, we show that a simple triplet loss outperforms state-of-the-art methods, suggesting that much of the improvements in recent approaches could be attributed to differences in data and architectures.

Revisiting the Learning Objectives of Vision-Language Reward Models

TL;DR

This paper tackles the problem of comparing vision-language reward models by decoupling the learning objective from model architecture, data, and environments. It evaluates a suite of language-adapted objectives (including TCN, VIP, LIV, R3M, and a simple Triplet loss) under a unified framework using a SigLIP2 backbone and LoRA fine-tuning on Meta-World demonstrations. The key finding is that a straightforward triplet loss can surpass state-of-the-art objectives in both reward-consistency and progress-alignment metrics, suggesting that data and architectural choices rather than learning signals drive much of the reported gains. The results advocate for standardized benchmarks to fairly assess reward-model objectives and highlight the robustness of simple ranking-based losses for transfer to unseen tasks.

Abstract

Learning generalizable reward functions is a core challenge in embodied intelligence. Recent work leverages contrastive vision language models (VLMs) to obtain dense, domain-agnostic rewards without human supervision. These methods adapt VLMs into reward models through increasingly complex learning objectives, yet meaningful comparison remains difficult due to differences in training data, architectures, and evaluation settings. In this work, we isolate the impact of the learning objective by evaluating recent VLM-based reward models under a unified framework with identical backbones, finetuning data, and evaluation environments. Using Meta-World tasks, we assess modeling accuracy by measuring consistency with ground truth reward and correlation with expert progress. Remarkably, we show that a simple triplet loss outperforms state-of-the-art methods, suggesting that much of the improvements in recent approaches could be attributed to differences in data and architectures.
Paper Structure (12 sections, 7 equations, 4 figures, 4 tables)

This paper contains 12 sections, 7 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Experiments
  4. Finetuning
  5. Evaluation
  6. Results
  7. Conclusion & Future Work
  8. Appendix
  9. Hyperparameters
  10. Example of different views in Metaworld
  11. Language goal
  12. Reward over expert trajectories

Figures (4)

  • Figure 1: The button-press environment seen from all 3 views. This example illustrates the occlusion problem often encountered in robotics, highlighting the importance of multi-view training and spatial planning.
  • Figure 2: Progression of the reward generated by each model over an expert trajectory in the button-press-v3 environment.
  • Figure 3: Progression of the reward generated by each model over an expert trajectory in the drawer-open-v3 environment.
  • Figure 4: Progression of the reward generated by each model over an expert trajectory in the door-open-v3 environment.