Safety Not Found (404): Hidden Risks of LLM-Based Robotics Decision Making

TL;DR

The paper investigates the safety risks of Large Language Models (LLMs) guiding robotics in safety-critical settings by proposing a seven-task diagnostic suite spanning complete-information, incomplete-information, and SOSR scenarios. Through ASCII-map, sequence-based, uncertain-terrain, Back-of-the-Building, and NL-based safety tasks, it reveals stark vulnerabilities: even state-of-the-art models can fail catastrophically or hallucinate, with 100% accuracy still being unsafe in practice. It shows that reliability and safety cannot be inferred from aggregate performance, as rare but hazardous errors can dominate outcomes in high-stakes environments. The study calls for reliability-focused benchmarks, constraint-aware reasoning, and real-world validation to prevent overreliance on LLMs in safety-critical robotics.

Abstract

One mistake by an AI system in a safety-critical setting can cost lives. As Large Language Models (LLMs) become integral to robotics decision-making, the physical dimension of risk grows; a single wrong instruction can directly endanger human safety. This paper addresses the urgent need to systematically evaluate LLM performance in scenarios where even minor errors are catastrophic. Through a qualitative evaluation of a fire evacuation scenario, we identified critical failure cases in LLM-based decision-making. Based on these, we designed seven tasks for quantitative assessment, categorized into: Complete Information, Incomplete Information, and Safety-Oriented Spatial Reasoning (SOSR). Complete information tasks utilize ASCII maps to minimize interpretation ambiguity and isolate spatial reasoning from visual processing. Incomplete information tasks require models to infer missing context, testing for spatial continuity versus hallucinations. SOSR tasks use natural language to evaluate safe decision-making in life-threatening contexts. We benchmark various LLMs and Vision-Language Models (VLMs) across these tasks. Beyond aggregate performance, we analyze the implications of a 1% failure rate, highlighting how "rare" errors escalate into catastrophic outcomes. Results reveal serious vulnerabilities: several models achieved a 0% success rate in ASCII navigation, while in a simulated fire drill, models instructed robots to move toward hazardous areas instead of emergency exits. Our findings lead to a sobering conclusion: current LLMs are not ready for direct deployment in safety-critical systems. A 99% accuracy rate is dangerously misleading in robotics, as it implies one out of every hundred executions could result in catastrophic harm. We demonstrate that even state-of-the-art models cannot guarantee safety, and absolute reliance on them creates unacceptable risks.

Figures (15)

• Figure 1: In a fire scenario, the LLMs directs the user to where important documents are (32%) or a server room (1%) instead of a safe exit.
• Figure 2: Overview of the experimental prompts and map structures. The prompts used for the 'Complete' (blue), 'Incomplete' (red), and 'SOSR' (yellow) tasks are shown. The figure also displays the structure of the ASCII map and the sequence map utilized in our experiments. For the SOSR task, phrases highlighted in red are the criteria for distinguishing between difficulty levels, while italicized sentences serve as important contextual clues. Due to their length and variety, the full prompts for the "back of the building" scenario are detailed in the Appendix.
• Figure 3: Success rates of LLMs on deterministic and uncertain ASCII map tasks
• Figure 4: Collapsed map structures generated by LLaMA-3-8b on (a) Deterministic Map (Easy) and (b) Uncertain Terrain Map 1
• Figure 5: Representative failure types in the Back of the Building task. (a) Structural collapse: Loss of global topology, producing incoherent or missing spatial structures. (b) Directional error: The agent failed to reach the rear of the building. (c) Constraint violation: The path intersected obstacles, yielding unsafe or infeasible planning. (d) Waypoint error: The model failed to place waypoints at directional transition points.