Adaptive Stress Testing Black-Box LLM Planners

TL;DR

This work addresses the safety and reliability of black-box LLM planners by introducing Adaptive Stress Testing (AST) with Monte-Carlo Tree Search to systematically perturb prompts and observations. By defining an adversarial perturbation process and robust undesirability metrics, it characterizes when LLMs become uncertain or crash in driving, crowd-navigation, and lunar-landing contexts. The approach yields offline characterizations that can be leveraged at runtime to generate prompts and assess trust, while also revealing edge cases that elicit unstable decisions. Collectively, the method offers a practical, model-agnostic pathway to rigorously evaluate and guide the deployment of LLM-based planners in safety-critical applications.

Abstract

Large language models (LLMs) have recently demonstrated success in generalizing across decision-making tasks including planning, control, and prediction, but their tendency to hallucinate unsafe and undesired outputs poses risks. We argue that detecting such failures is necessary, especially in safety-critical scenarios. Existing methods for black-box models often detect hallucinations by identifying inconsistencies across multiple samples. Many of these approaches typically introduce prompt perturbations like randomizing detail order or generating adversarial inputs, with the intuition that a confident model should produce stable outputs. We first perform a manual case study showing that other forms of perturbations (e.g., adding noise, removing sensor details) cause LLMs to hallucinate in a multi-agent driving environment. We then propose a novel method for efficiently searching the space of prompt perturbations using adaptive stress testing (AST) with Monte-Carlo tree search (MCTS). Our AST formulation enables discovery of scenarios and prompts that cause language models to act with high uncertainty or even crash. By generating MCTS prompt perturbation trees across diverse scenarios, we show through extensive experiments that offline analyses can be used at runtime to automatically generate prompts that influence model uncertainty, and to inform real-time trust assessments of an LLM. We further characterize LLMs deployed as planners in a single-agent lunar lander environment and in a multi-agent robot crowd navigation simulation. Overall, ours is one of the first hallucination intervention algorithms to pave a path towards rigorous characterization of black-box LLM planners.

Figures (24)

• Figure 1: Unperturbed eval. of models measuring (Left) average speed, (Middle) average episode length, and (Right) prediction distribution over all timesteps. In all graphs, we vary system prompt between cons. and agg., and access to few-shot examples.
• Figure 2: Inconsistency rates of (Left) DeepSeek and (Right) Llama model predictions under manual offline observation perturbations. We denote when Position, Speed, Acceleration, Lane, Noise, and Randomization are present in the perturbed prompts. For example, PSN denotes that noisy position and speed are observed, leaving acceleration and lane sensors unobservable.
• Figure 3: (Left) Block diagram of general AST frameworks. (Right) An example of how we (1) expand the perturbation tree, (2) generate prompts, sample actions, and compute the diversity $\mathcal{D}$, (3) cache sampled actions, and (4) return the reward $\mathcal{R}_\zeta$.
• Figure 4: The distribution of Shannon entropy of (1) all sampled actions and (2) majority sampled actions from low-diversity perturbation states, for $20$ trees trained with (Left) $\mathcal{D}$ and (Right) $\mathcal{H}$, per LLM and MCTS configuration pair.
• Figure 5: The distribution of Shannon entropy across three chosen undesirability functions $\mathcal{U}$ for Llama and Qwen (Left) Shallow and (Right) Deep trees specifically.