Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviors

TL;DR

This work investigates using video-language models as scalable behavior critics to detect undesirable behaviors in robot videos when formal verifiers are unavailable. It introduces a comprehensive benchmark of goal-reaching yet undesirable policies and evaluates VLMs on recall, precision, and error modes, revealing strong recall but notable grounding-related hallucinations. The authors propose grounding-augmented critique and a closed-loop CaP-based policy refinement pipeline, achieving high precision when grounding is perfect. Their findings provide practical guidelines for integrating VLM critiques into iterative policy improvement and highlight avenues for improving grounding and integration in embodied AI systems.

Abstract

Large-scale generative models are shown to be useful for sampling meaningful candidate solutions, yet they often overlook task constraints and user preferences. Their full power is better harnessed when the models are coupled with external verifiers and the final solutions are derived iteratively or progressively according to the verification feedback. In the context of embodied AI, verification often solely involves assessing whether goal conditions specified in the instructions have been met. Nonetheless, for these agents to be seamlessly integrated into daily life, it is crucial to account for a broader range of constraints and preferences beyond bare task success (e.g., a robot should grasp bread with care to avoid significant deformations). However, given the unbounded scope of robot tasks, it is infeasible to construct scripted verifiers akin to those used for explicit-knowledge tasks like the game of Go and theorem proving. This begs the question: when no sound verifier is available, can we use large vision and language models (VLMs), which are approximately omniscient, as scalable Behavior Critics to catch undesirable robot behaviors in videos? To answer this, we first construct a benchmark that contains diverse cases of goal-reaching yet undesirable robot policies. Then, we comprehensively evaluate VLM critics to gain a deeper understanding of their strengths and failure modes. Based on the evaluation, we provide guidelines on how to effectively utilize VLM critiques and showcase a practical way to integrate the feedback into an iterative process of policy refinement. The dataset and codebase are released at: https://guansuns.github.io/pages/vlm-critic.

Figures (10)

• Figure 1: Examples of GPT-4V critic accurately catching undesirable behaviors. Failure cases can be found in Fig. \ref{['fig:negative-examples']}.
• Figure 2: Positioning of this work. This figure also illustrates a loop of policy (re-)generation and "verification."
• Figure 3: Failure cases of GPT-4V critic. More examples can be found in Fig. \ref{['fig:appx-negative-examples']} in Appendix.
• Figure 4: Distributions of error types. GPT-4V-Augmented refers to the experiment of augmenting GPT-4V with perfect grounding feedback (Sec. \ref{['sec:expr-augmented-grounding']}).
• Figure 5: Illustrations of the initial policies and the updated policies in response to verbal critiques. A more detailed visualization can be found in Fig. \ref{['fig:full-robot-policy']} in Appendix.