Embodied Learning of Reward for Musculoskeletal Control with Vision Language Models

TL;DR

The paper introduces MoVLR, a framework that auto-discovers reward functions for high-dimensional musculoskeletal control by closing the loop between language-described goals and motion execution. It integrates policy optimization (via MPC^2) with a vision-language model that provides structured feedback and a language model that synthesizes executable reward terms, grounding abstract motion descriptions in biomechanical priors. Rewards are represented as a linear combination of terms, with weights refined through iterative multimodal feedback to align trajectories with desired coordination and stability. Across diverse locomotion and manipulation tasks, MoVLR outperforms baselines and demonstrates generalization across morphologies, illustrating a principled path toward interpretable, transferable, and scalable embodiment learning.

Abstract

Discovering effective reward functions remains a fundamental challenge in motor control of high-dimensional musculoskeletal systems. While humans can describe movement goals explicitly such as "walking forward with an upright posture," the underlying control strategies that realize these goals are largely implicit, making it difficult to directly design rewards from high-level goals and natural language descriptions. We introduce Motion from Vision-Language Representation (MoVLR), a framework that leverages vision-language models (VLMs) to bridge the gap between goal specification and movement control. Rather than relying on handcrafted rewards, MoVLR iteratively explores the reward space through iterative interaction between control optimization and VLM feedback, aligning control policies with physically coordinated behaviors. Our approach transforms language and vision-based assessments into structured guidance for embodied learning, enabling the discovery and refinement of reward functions for high-dimensional musculoskeletal locomotion and manipulation. These results suggest that VLMs can effectively ground abstract motion descriptions in the implicit principles governing physiological motor control.

Figures (18)

• Figure 1: Workflow of MoVLR. Policy optimization is performed to provide high-dimensional musculoskeletal dynamics of the reward candidate. A VLM evaluates the corresponding movement video $\boldsymbol{\zeta}^{(i)}$ to update the current best reward design $r^*$ and suggest biomechanical improvements $\mathcal{F}$ for a LLM to refine reward generation of $r^{(i+1)}$.
• Figure 2: Example inputs and outputs of the (a) VLM, and (b) LLM. The VLM analyzes a video motion sequence based on the given motion description and provides diagnostic feedback. The LLM uses this feedback to explore corresponding code modifications to the reward function.
• Figure 3: Overview of the six evaluated tasks. The top row illustrates the environment setup for each task, and the bottom row visualizes the relative weighting of learned reward terms.
• Figure 5: Progressive improvement of the musculoskeletal model’s gait across training stages based on movement video.
• Figure 8: Comparison of reward terms designed by LLM only (blue) and by human experts (green), with shared terms shown in orange, across three musculoskeletal systems.