RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedback

TL;DR

This paper tackles the challenge of sparse rewards in reinforcement fine-tuning of embodied vision-language-action agents. It introduces RFTF, which trains a temporal-value model to generate dense, time-local rewards without robot action labels and integrates this into a PPO-based RL loop with reward shaping, GAE, surrogate clipping, and adaptive KL constraints. The approach yields state-of-the-art results on CALVIN ABC-D and strong adaptation to a new environment, showcasing improved generalization and rapid transfer. By reducing reliance on labeled demonstrations and providing fine-grained feedback, RFTF enhances the practical deployment potential of embodied VLAs in diverse tasks. The work highlights a viable path toward more data-efficient, robust embodied intelligence in simulated settings, with plans to extend to real-world robots.

Abstract

Vision-Language-Action (VLA) models have demonstrated significant potential in the field of embodied intelligence, enabling agents to follow human instructions to complete complex tasks in physical environments. Existing embodied agents are often trained through behavior cloning, which requires expensive data and computational resources and is constrained by human demonstrations. To address this issue, many researchers explore the application of reinforcement fine-tuning to embodied agents. However, typical reinforcement fine-tuning methods for embodied agents usually rely on sparse, outcome-based rewards, which struggle to provide fine-grained feedback for specific actions within an episode, thus limiting the model's manipulation capabilities and generalization performance. In this paper, we propose RFTF, a novel reinforcement fine-tuning method that leverages a value model to generate dense rewards in embodied scenarios. Specifically, our value model is trained using temporal information, eliminating the need for costly robot action labels. In addition, RFTF incorporates a range of techniques, such as GAE and sample balance to enhance the effectiveness of the fine-tuning process. By addressing the sparse reward problem in reinforcement fine-tuning, our method significantly improves the performance of embodied agents, delivering superior generalization and adaptation capabilities across diverse embodied tasks. Experimental results show that embodied agents fine-tuned with RFTF achieve new state-of-the-art performance on the challenging CALVIN ABC-D with an average success length of 4.296. Moreover, RFTF enables rapid adaptation to new environments. After fine-tuning in the D environment of CALVIN for a few episodes, RFTF achieved an average success length of 4.301 in this new environment.

Figures (6)

• Figure 1: Comparison between sparse reward and dense reward. In typical reinforcement fine-tuning methods for embodied agents, only sparse, outcome-based rewards are provided, which can confuse embodied agents when encountering partially correct or incorrect episodes. In contrast, RFTF leverages a value model trained with temporal information to predict the value of each state within an episode, providing embodied agents with higher-granularity dense rewards.
• Figure 2: Training procedure of the value model. We assume that during an episode of a human-demonstrated successful embodied task, the state value increases monotonically over time.
• Figure 3: Illustration of RL fine-tuning pipeline. RFTF utilizes a value model trained with temporal information to predict the value of each state in episodes of interaction between the embodied agent and the environment, thereby providing guidance for each action in episodes to fine-tune the embodied agent.
• Figure 4: Visualization of the CALVIN benchmark. The CALVIN benchmark includes four distinct environments, differing in the positions of the LED, light bulb, slider, drawer, switch, and button, as well as the material of the table.
• Figure 5: Train curves of the value model. We show the loss curve of the value model during training and evaluation results of the value model across different epochs.