Trajectory Adaptation using Large Language Models

TL;DR

The paper addresses intuitive, scalable trajectory adaptation by leveraging pre-trained Large Language Models to generate executable code that modifies existing trajectories produced by planners or demonstrations. It introduces a two-stage pipeline—high-level plan generation followed by code generation—driven by natural language instructions and environmental context, with a closed-loop user feedback mechanism. The approach requires no task-specific training and demonstrates adaptability across manipulators, aerial platforms, and ground robots in PyBullet and Gazebo, including cases with complex numerical commands. This work advances human-robot interaction by providing an interpretable, flexible interface for trajectory reshaping and sets the stage for broader user studies and long-horizon tasks with open-ended instructions.

Abstract

Adapting robot trajectories based on human instructions as per new situations is essential for achieving more intuitive and scalable human-robot interactions. This work proposes a flexible language-based framework to adapt generic robotic trajectories produced by off-the-shelf motion planners like RRT, A-star, etc, or learned from human demonstrations. We utilize pre-trained LLMs to adapt trajectory waypoints by generating code as a policy for dense robot manipulation, enabling more complex and flexible instructions than current methods. This approach allows us to incorporate a broader range of commands, including numerical inputs. Compared to state-of-the-art feature-based sequence-to-sequence models which require training, our method does not require task-specific training and offers greater interpretability and more effective feedback mechanisms. We validate our approach through simulation experiments on the robotic manipulator, aerial vehicle, and ground robot in the Pybullet and Gazebo simulation environments, demonstrating that LLMs can successfully adapt trajectories to complex human instructions.

Figures (5)

• Figure 1: An overview of the pipeline: A prompt is created from environment description, instruction, and task agnostic guidelines; LLM generates high-level plan along with Python code. The user reviews the plan; if correct, the code runs to adapt the trajectory. If a feedback is given, it's sent back to the LLM with the original instruction.
• Figure 2: Simulation experiments: Testing across varied robot dynamics, a) Robotic arm, b) Ground robot, c) Drone. The initial trajectory is represented in Blue and the adapted trajectory is shown in Red
• Figure 3: Results over LaTTe Dataset
• Figure 4: Results over extended dataset having complex instructions along with numerical changes
• Figure 5: Providing additional information to LLMs, which may be missing from the high-level strategy, helps correct inconsistencies and ensures more accurate outputs