Real-Time Verification of Embodied Reasoning for Generative Skill Acquisition

TL;DR

This work addresses the inefficiency and limited verification signals in learning embodied skills by introducing VERGSA, a framework that integrates real-time verification into embodied reasoning. It couples a dynamic exemplar task pool with a Process Reward Model (PRM) trained via Monte Carlo Tree Search to provide subtask-level rewards and guidance, enabling scalable and robust skill acquisition in 3D environments. Across experiments, the exemplar pool improves task and subtask success rates, while the PRM-based verifier outperforms LLM-as-a-Judge baselines, demonstrating the value of verification-driven learning for embodied tasks. The authors also contribute the first comprehensive verification dataset for embodied skill acquisition and discuss practical limitations and avenues for future work, including meta-learning and domain randomization for better sim-to-real transfer.

Abstract

Generative skill acquisition enables embodied agents to actively learn a scalable and evolving repertoire of control skills, crucial for the advancement of large decision models. While prior approaches often rely on supervision signals from generalist agents (e.g., LLMs), their effectiveness in complex 3D environments remains unclear; exhaustive evaluation incurs substantial computational costs, significantly hindering the efficiency of skill learning. Inspired by recent successes in verification models for mathematical reasoning, we propose VERGSA (Verifying Embodied Reasoning in Generative Skill Acquisition), a framework that systematically integrates real-time verification principles into embodied skill learning. VERGSA establishes 1) a seamless extension from verification of mathematical reasoning into embodied learning by dynamically incorporating contextually relevant tasks into prompts and defining success metrics for both subtasks and overall tasks, and 2) an automated, scalable reward labeling scheme that synthesizes dense reward signals by iteratively finalizing the contribution of scene configuration and subtask learning to overall skill acquisition. To the best of our knowledge, this approach constitutes the first comprehensive training dataset for verification-driven generative skill acquisition, eliminating arduous manual reward engineering. Experiments validate the efficacy of our approach: 1) the exemplar task pool improves the average task success rates by 21%, 2) our verification model boosts success rates by 24% for novel tasks and 36% for encountered tasks, and 3) outperforms LLM-as-a-Judge baselines in verification quality.

Figures (6)

• Figure 1: A motivating example.
• Figure 1: Top 2 similar tasks per novel task.
• Figure 2: The VERGSA framework.
• Figure 3: Number of subtasks in three tasks.
• Figure 4: Performance of subtask execution across 14 tasks.