LAPP: Large Language Model Feedback for Preference-Driven Reinforcement Learning

TL;DR

LAPP presents a framework that marries large-language-model (LLM) generated trajectory preferences with reinforcement learning to bypass labor-intensive reward engineering in robotics. By prompting LLMs to label full state-action trajectories and training an online transformer-based reward predictor, LAPP continuously shapes policy optimization toward high-level, language-specified behaviors, including non-Markovian dependencies. Empirical results across quadruped locomotion and dexterous manipulation show faster convergence, higher final performance, and the ability to master complex tasks such as quadruped backflips, with successful sim-to-real transfer on a Unitree Go2. The approach demonstrates that adaptive, trajectory-level LLM feedback can robustly guide expressive robot control while reducing human annotation burden. Overall, LAPP advances scalable, preference-driven robot learning by integrating foundation-model feedback directly into the RL loop and handling long-horizon, high-dimensional control challenges.

Abstract

We introduce Large Language Model-Assisted Preference Prediction (LAPP), a novel framework for robot learning that enables efficient, customizable, and expressive behavior acquisition with minimum human effort. Unlike prior approaches that rely heavily on reward engineering, human demonstrations, motion capture, or expensive pairwise preference labels, LAPP leverages large language models (LLMs) to automatically generate preference labels from raw state-action trajectories collected during reinforcement learning (RL). These labels are used to train an online preference predictor, which in turn guides the policy optimization process toward satisfying high-level behavioral specifications provided by humans. Our key technical contribution is the integration of LLMs into the RL feedback loop through trajectory-level preference prediction, enabling robots to acquire complex skills including subtle control over gait patterns and rhythmic timing. We evaluate LAPP on a diverse set of quadruped locomotion and dexterous manipulation tasks and show that it achieves efficient learning, higher final performance, faster adaptation, and precise control of high-level behaviors. Notably, LAPP enables robots to master highly dynamic and expressive tasks such as quadruped backflips, which remain out of reach for standard LLM-generated or handcrafted rewards. Our results highlight LAPP as a promising direction for scalable preference-driven robot learning.

Figures (16)

• Figure 1: Large Language Model-Assisted Preference Prediction (LAPP) takes in language behavior instructions and generates preference feedback to guide reinforcement learning training from raw state-action robot trajectories.
• Figure 2: LAPP generates preference feedback from an LLM based on rollout trajectories pairs of raw state and actions as well as a high-level behavior instruction. A transformer-based reward predictor is trained using these preferences while simultaneously optimizing a robot policy to maximize a weighted sum of environment rewards and predicted preference rewards.
• Figure 3: Behavior Instruction Prompt Example. The LLM prompt consists of three sections: (1) defining the LLM's role and the robotic task (blue box), (2) specifying the state variables and some evaluation criteria of preference (green box), and (3) establishing rules and semantics for generating preference labels (purple box).
• Figure 4: Simulation Tasks. (a) Quadruped locomotion. The robot learns to walk forward across various terrains following given velocity commands. The terrains include the flat plane, stairs pyramids, discrete obstacles, slope pyramids, and wave-pattern hills. (b) Dexterous manipulation. Each dexterous hand has $26$ degrees of freedoms. Kettle requires the robot to pick up the kettle with one hand, and the cup with another hand, and then pour water into the kettle. Hand Over requires one hand to pass a ball to another hand. Swing Cup requires two hands to hold the cup and rotate it for $180^{\circ}$. (c) Quadruped backflip. The robot jumps in the air and rotate backwards for $360^{\circ}$, and then land on the ground.
• Figure 5: Training Efficiency. Training with LAPP converges faster in the Plane, Stairs, Obstacles, Hand Over, Swing Cup and Kettle tasks, while also exhibiting more stable performance post-convergence in Swing Cup. In the Slope and Wave tasks, LAPP performs similarly to baselines as these tasks are relatively easier for exploration, converging quickly for all algorithms.