Safe LLM-Controlled Robots with Formal Guarantees via Reachability Analysis
Ahmad Hafez, Alireza Naderi Akhormeh, Amr Hegazy, Amr Alanwar
TL;DR
This work addresses the safety of deploying LLMs as controllers for robots in uncertain environments by introducing a data-driven reachability framework that operates without precise analytic models. By representing uncertainties and dynamics with constrained zonotopes and estimating a Lipschitz bound from offline data, the approach computes overapproximated forward reachable sets $\mathcal{R}_k$ and uses a differentiable, gradient-based safety layer to adjust LLM-generated plans, embedding a failsafe maneuver. The method provides formal safety guarantees through a data-driven, model-free verification process and demonstrates effectiveness on TurtleBot3 and JetRacer platforms, achieving collision avoidance under various obstacle configurations and planning horizons. The work advances practical integration of formal methods with LLM-driven robotics, enabling safe, zero-shot task execution in real-world autonomous systems with adaptability to unseen tasks.
Abstract
The deployment of Large Language Models (LLMs) in robotic systems presents unique safety challenges, particularly in unpredictable environments. Although LLMs, leveraging zero-shot learning, enhance human-robot interaction and decision-making capabilities, their inherent probabilistic nature and lack of formal guarantees raise significant concerns for safety-critical applications. Traditional model-based verification approaches often rely on precise system models, which are difficult to obtain for real-world robotic systems and may not be fully trusted due to modeling inaccuracies, unmodeled dynamics, or environmental uncertainties. To address these challenges, this paper introduces a safety assurance framework for LLM-controlled robots based on data-driven reachability analysis, a formal verification technique that ensures all possible system trajectories remain within safe operational limits. Our framework specifically investigates the problem of instructing an LLM to navigate the robot to a specified goal and assesses its ability to generate low-level control actions that successfully guide the robot safely toward that goal. By leveraging historical data to construct reachable sets of states for the robot-LLM system, our approach provides rigorous safety guarantees against unsafe behaviors without relying on explicit analytical models. We validate the framework through experimental case studies in autonomous navigation and task planning, demonstrating its effectiveness in mitigating risks associated with LLM-generated commands. This work advances the integration of formal methods into LLM-based robotics, offering a principled and practical approach to ensuring safety in next-generation autonomous systems.