ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations

TL;DR

ReWiND tackles the challenge of learning unseen robot manipulation tasks from language without task-specific demonstrations. It trains a language-conditioned reward model from a small demo set plus Open-X data, then offline-trains a language-conditioned policy with these rewards before fine-tuning online on new task variations. Through video rewind and instruction augmentation, the reward model achieves dense, task-generalizable guidance that aligns with policy progress and supports robust generalization. In MetaWorld and real-world bimanual robot experiments, ReWiND delivers substantial improvements in reward alignment and sample-efficient policy adaptation compared to baselines, advancing scalable, language-guided robot learning without additional demonstrations.

Abstract

We introduce ReWiND, a framework for learning robot manipulation tasks solely from language instructions without per-task demonstrations. Standard reinforcement learning (RL) and imitation learning methods require expert supervision through human-designed reward functions or demonstrations for every new task. In contrast, ReWiND starts from a small demonstration dataset to learn: (1) a data-efficient, language-conditioned reward function that labels the dataset with rewards, and (2) a language-conditioned policy pre-trained with offline RL using these rewards. Given an unseen task variation, ReWiND fine-tunes the pre-trained policy using the learned reward function, requiring minimal online interaction. We show that ReWiND's reward model generalizes effectively to unseen tasks, outperforming baselines by up to 2.4x in reward generalization and policy alignment metrics. Finally, we demonstrate that ReWiND enables sample-efficient adaptation to new tasks, beating baselines by 2x in simulation and improving real-world pretrained bimanual policies by 5x, taking a step towards scalable, real-world robot learning. See website at https://rewind-reward.github.io/.

Figures (17)

• Figure 1: Overview. We pre-train a policy and reward model from a small set of language-labeled demos. Then, we solve unseen task variations via language-guided RL without additional demos.
• Figure 2: (a): We train a reward model $R_\psi(o_{1:t}, z)$ on a small demonstration dataset $\mathcal{D}_\text{demos}$ and a curated subset of Open-X, $\mathcal{D}_\text{open-x}$, augmented with LLM-generated instructions and video rewinding. $R_\psi(o_{1:t}, z)$ predicts video progress rewards $\hat{r}_{1:T}$ from pre-trained embeddings of image observations $o_{1:T}$ and language instructions $z$, and assigns 0 progress to misaligned video-language pairs. (b): We use the trained $R_\psi(o_{1:t}, z)$ to label $\mathcal{D}_\text{demos}$ with rewards and pre-train a language-conditioned policy using offline RL. (c): For an unseen task specified by $z_\text{new}$, we fine-tune the policy with online rollouts and reward labels from $R_\psi(o_{1:t}, z_\text{new})$.
• Figure 3: Video rewind. We split a demo at intermediate timestep $i$ into forward/reverse sections. Here, the forward section shows the robot approaching the cup; the reverse section ($o_{i-1}, o_{i-2}, \ldots$) resembles dropping it.
• Figure 4: Video-Language Reward Confusion Matrix. For each unseen MetaWorld task, we compute rewards for all combinations of demonstration videos and language descriptions. ReWiND produces the most diagonal-heavy confusion matrix, indicating strong alignment between unseen demos and instructions. See \ref{['sec:appendix:additional_results:reward_analysis']} for train task results, \ref{['sec:appendix:additional_results:real_world_reward']} for real-world results.
• Figure 5: MetaWorld final performance. We plot inter-quartile means (IQMs) of success rates after 100k environment steps on 8 unseen tasks in MetaWorld. ReWiND achieves 79%.