Optimizing Latent Goal by Learning from Trajectory Preference

TL;DR

This work tackles the sensitivity of instruction-following foundation policies to prompt design in open-world tasks and introduces Preference Goal-Tuning (PGT), a data-efficient post-training framework that tunes a task-specific goal latent while keeping the backbone frozen. By collecting trajectories and learning from preference signals (positive vs. negative samples), PGT yields significant performance gains across 17 Minecraft SkillForge tasks for two foundation policies, often surpassing the best human-selected prompts and even full fine-tuning in OOD environments. The approach demonstrates strong generalization, robustness to execution environments, and an efficient continual learning pathway by storing a single latent representation per task. With iterative data collection and a latent-focused objective inspired by DPO, PGT enables efficient elicitation of new skills and better long-horizon task performance with minimal computational cost.

Abstract

A glowing body of work has emerged focusing on instruction-following policies for open-world agents, aiming to better align the agent's behavior with human intentions. However, the performance of these policies is highly susceptible to the initial prompt, which leads to extra efforts in selecting the best instructions. We propose a framework named Preference Goal Tuning (PGT). PGT allows an instruction following policy to interact with the environment to collect several trajectories, which will be categorized into positive and negative samples based on preference. Then we use preference learning to fine-tune the initial goal latent representation with the categorized trajectories while keeping the policy backbone frozen. The experiment result shows that with minimal data and training, PGT achieves an average relative improvement of 72.0% and 81.6% over 17 tasks in 2 different foundation policies respectively, and outperforms the best human-selected instructions. Moreover, PGT surpasses full fine-tuning in the out-of-distribution (OOD) task-execution environments by 13.4%, indicating that our approach retains strong generalization capabilities. Since our approach stores a single latent representation for each task independently, it can be viewed as an efficient method for continual learning, without the risk of catastrophic forgetting or task interference. In short, PGT enhances the performance of agents across nearly all tasks in the Minecraft Skillforge benchmark and demonstrates robustness to the execution environment.

Figures (6)

• Figure 1: Pipeline of our Preference Goal Tuning (PGT). The process begins by selecting an initial prompt (can be video or text), encoding it into a latent representation, and allowing the policy to interact with the environment multiple times to collect trajectories. These trajectories are then classified as positive or negative based on human preferences or rewards. Then, the model is fine-tuned using the collected data, with only the latent goal embedding being trainable. Iterative training is supported.
• Figure 2: Improvements with training iterations of our methods.
• Figure 3: Comparison between full finetuning and PGT. Upper: In Distribution(ID). Lower: Out of Distribution(OOD).
• Figure 4: Different initial prompt results. Each line graph represents a different prompt, and the horizontal line represents the performance of the best human-selected prompt.
• Figure 5: Result of different parameter efficient methods. The horizontal line indicates pretraining performance. Upper: ID. Lower: OOD.