PRIMT: Preference-based Reinforcement Learning with Multimodal Feedback and Trajectory Synthesis from Foundation Models

TL;DR

PRIMT tackles reward engineering bottlenecks in robotics PbRL by fusing multimodal foundation-model feedback through a hierarchical neuro-symbolic PSL framework and by actively synthesizing trajectories via foresight and hindsight with a causal auxiliary loss. The approach reduces human annotation needs, mitigates early-stage query ambiguity, and improves state-action credit assignment, yielding superior performance across 8 tasks and real-world deployment on a Kinova Jaco. Ablation studies show the necessity of both intra-/inter-modal fusion and trajectory synthesis components, with notable gains in early learning and final task success. While incurring additional FM costs, PRIMT demonstrates favorable cost–performance trade-offs and robust generalization to diverse robotic tasks and conditions.

Abstract

Preference-based reinforcement learning (PbRL) has emerged as a promising paradigm for teaching robots complex behaviors without reward engineering. However, its effectiveness is often limited by two critical challenges: the reliance on extensive human input and the inherent difficulties in resolving query ambiguity and credit assignment during reward learning. In this paper, we introduce PRIMT, a PbRL framework designed to overcome these challenges by leveraging foundation models (FMs) for multimodal synthetic feedback and trajectory synthesis. Unlike prior approaches that rely on single-modality FM evaluations, PRIMT employs a hierarchical neuro-symbolic fusion strategy, integrating the complementary strengths of large language models and vision-language models in evaluating robot behaviors for more reliable and comprehensive feedback. PRIMT also incorporates foresight trajectory generation, which reduces early-stage query ambiguity by warm-starting the trajectory buffer with bootstrapped samples, and hindsight trajectory augmentation, which enables counterfactual reasoning with a causal auxiliary loss to improve credit assignment. We evaluate PRIMT on 2 locomotion and 6 manipulation tasks on various benchmarks, demonstrating superior performance over FM-based and scripted baselines.

Figures (24)

• Figure 1: Overview of PRIMT, comprising two synergistic modules: 1) Hierarchical neuro-symbolic preference fusion improves the quality of synthetic feedback by leveraging the complementary strengths of VLMs and LLMs for multimodal evaluation of robot behaviors; and 2) Bidirectional trajectory synthesis mitigates early-stage query ambiguity through foresight generation and enhances credit assignment in reward learning via hindsight counterfactual augmentation with a causal auxiliary loss.
• Figure 2: Learning curves of PRIMT and baseline methods across all tasks, averaged over 5 runs with solid lines denoting the average and shaded regions representing the standard error. A moving average window of 5 steps for locomotion tasks and 10 steps for manipulation tasks is applied to improve readability.
• Figure 3: Learning curves of PRIMT and ablation models on the Door Open and PickSingleYCB tasks.
• Figure 4: Left: Distribution of preference labels, showing the proportion of correct, incorrect, and indecisive labels across different methods. Right: Reward alignment analysis, comparing the learned reward outputs of PRIMT, ablations, and baselines against ground-truth rewards. Plots on more tasks are provided in Appendix \ref{['app:extraplot']}.
• Figure 5: PRIMT successfully completes real-world block lifting and stacking tasks on a Kinova Jaco robot.