GUIDES: Guidance Using Instructor-Distilled Embeddings for Pre-trained Robot Policy Enhancement

TL;DR

GUIDES is a lightweight framework that augments pre-trained policies with semantic guidance from foundation models without requiring architectural redesign, and offers a practical and resource-efficient pathway to upgrade, rather than replace, validated robot policies.

Abstract

Pre-trained robot policies serve as the foundation of many validated robotic systems, which encapsulate extensive embodied knowledge. However, they often lack the semantic awareness characteristic of foundation models, and replacing them entirely is impractical in many situations due to high costs and the loss of accumulated knowledge. To address this gap, we introduce GUIDES, a lightweight framework that augments pre-trained policies with semantic guidance from foundation models without requiring architectural redesign. GUIDES employs a fine-tuned vision-language model (Instructor) to generate contextual instructions, which are encoded by an auxiliary module into guidance embeddings. These embeddings are injected into the policy's latent space, allowing the legacy model to adapt to this new semantic input through brief, targeted fine-tuning. For inference-time robustness, a large language model-based Reflector monitors the Instructor's confidence and, when confidence is low, initiates a reasoning loop that analyzes execution history, retrieves relevant examples, and augments the VLM's context to refine subsequent actions. Extensive validation in the RoboCasa simulation environment across diverse policy architectures shows consistent and substantial improvements in task success rates. Real-world deployment on a UR5 robot further demonstrates that GUIDES enhances motion precision for critical sub-tasks such as grasping. Overall, GUIDES offers a practical and resource-efficient pathway to upgrade, rather than replace, validated robot policies.

Figures (4)

• Figure 1: Overview of the Guides framework. The Instructor, initially fine-tuned with motion ground truth, provides step-wise instructions. These are mapped to guidance embedding via an auxiliary module, and integrated with the original model’s latent space $\phi$ to enhance $\pi_\theta$. Meanwhile, the Reflector uses Chain-of-Thought reasoning to analyze execution and infer potential risks or next steps. It queries via embedding retrieval based on the execution history.
• Figure 2: The t-SNE visualization of guidance embeddings ($\mathcal{G}_t$), colored by task category. Note the distinct clusters for manipulation, door/drawer, and appliance-related tasks, which indicate a semantically structured latent space.
• Figure 3: Experimental platform with UR5
• Figure 4: End-effector trajectory distribution (the block denotes the “strike zone”).