Distilling On-device Language Models for Robot Planning with Minimal Human Intervention

TL;DR

This paper tackles the challenge of deploying LLM-powered robot planners in environments with unreliable connectivity by introducing PRISM, a framework that distills small on-device language models from a source LLM-enabled planner. PRISM generates diverse task-environment scenarios, elicits planning demonstrations from the source planner, and uses this synthetic data to fine-tune a compact SLM with LoRA, producing a drop-in replacement that operates onboard. Across three planning domains (mapping/exploration, home assistance, and manipulation), prism-distilled models achieve high planning success rates close to or surpassing the source baselines while eliminating dependency on external compute and providing more deterministic latency. The work demonstrates strong cross-platform generalization (ground and aerial robots; indoor/outdoor settings) and releases code, datasets, and trained models to promote reproducibility and broader adoption.

Abstract

Large language models (LLMs) provide robots with powerful contextual reasoning abilities and a natural human interface. Yet, current LLM-enabled robots typically depend on cloud-hosted models, limiting their usability in environments with unreliable communication infrastructure, such as outdoor or industrial settings. We present PRISM, a framework for distilling small language model (SLM)-enabled robot planners that run on-device with minimal human supervision. Starting from an existing LLM-enabled planner, PRISM automatically synthesizes diverse tasks and environments, elicits plans from the LLM, and uses this synthetic dataset to distill a compact SLM as a drop-in replacement of the source model. We apply PRISM to three LLM-enabled planners for mapping and exploration, manipulation, and household assistance, and we demonstrate that PRISM improves the performance of Llama-3.2-3B from 10-20% of GPT-4o's performance to over 93% - using only synthetic data. We further demonstrate that the distilled planners generalize across heterogeneous robotic platforms (ground and aerial) and diverse environments (indoor and outdoor). We release all software, trained models, and datasets at https://zacravichandran.github.io/PRISM.

Figures (5)

• Figure 1: PRISM distills on-device language models for robot planning with minimal human intervention. Given an existing LLM-enabled planner, prism synthesizes diverse task and textual environments, elicits plans from the LLM, then uses the resulting dataset to fine-tune an SLM that rivals the performance of the original planner, while being extremely compute efficient. We demonstrate prism-distilled models across platforms and environments.
• Figure 2: Synthesizing a data sample: The scenario generator synthesizes an environment and semantically coherent task. The environment is masked, yielding an initial partial observation for the LLM-enabled planner. Given this task and partial observation, the LLM-enabled planner iteratively provides actions and receives new observations until the task is complete. This rollout is collected as a data sample and added to the synthesized dataset.
• Figure 3: We evaluate PRISM on three distinct LLM-enabled planning domains: mapping and exploration, home assistance, and manipulation. Left: SPINE ravichandran_spine in both indoor and outdoor environments on bot UAVs and UGVs. Middle: LLM-Planner song2023llmplanner to household tasks in the ALFRED ALFRED20 simulator. Right: SayCan saycan2022arxiv to manipulation tasks.
• Figure 4: In the following examples, prism completes the task, and we highlight failure modes of the SLM. Top right: SLM does not identify semantic targets. Top left: SLM incorrectly invokes planner API and mistakes logical ordering; two blocks cannot be simultaneously help by the robot. Bottom right: SLM makes unwarrented assumptions about the environment. Bottom left: SLM incorrectly invokes API and the plan terminates.
• Figure 5: Latency comparison between GPT-4o and prism