Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Art

TL;DR

This work tackles hallucinations in foundation-model–driven decision-making by proposing a flexible, deployment-tunable definition and surveying state-of-the-art detection and mitigation approaches across autonomous driving, robotics, and related domains. It organizes methods into white-, grey-, and black-box categories, detailing concrete techniques such as conformal prediction, grounding with external knowledge, attention-based risk signals, and adversarial prompting, while highlighting their applicability and limitations in decision tasks. The authors offer a nine-step guideline for selecting and implementing hallucination interventions, emphasizing rigorous evaluation settings, appropriate metrics, and consideration of model access and safety requirements. The paper also charts future directions, advocating stronger emphasis on decision-making benchmarks, more robust black-box solutions, and expanded multi-modal testing to ensure reliable operation in real-world deployments. Overall, the work advances practical frameworks to quantify, detect, and mitigate hallucinations, aiming to enhance the trustworthiness of LVLMs in safety-critical decision-making systems.

Abstract

Autonomous systems are soon to be ubiquitous, spanning manufacturing, agriculture, healthcare, entertainment, and other industries. Most of these systems are developed with modular sub-components for decision-making, planning, and control that may be hand-engineered or learning-based. While these approaches perform well under the situations they were specifically designed for, they can perform especially poorly in out-of-distribution scenarios that will undoubtedly arise at test-time. The rise of foundation models trained on multiple tasks with impressively large datasets has led researchers to believe that these models may provide "common sense" reasoning that existing planners are missing, bridging the gap between algorithm development and deployment. While researchers have shown promising results in deploying foundation models to decision-making tasks, these models are known to hallucinate and generate decisions that may sound reasonable, but are in fact poor. We argue there is a need to step back and simultaneously design systems that can quantify the certainty of a model's decision, and detect when it may be hallucinating. In this work, we discuss the current use cases of foundation models for decision-making tasks, provide a general definition for hallucinations with examples, discuss existing approaches to hallucination detection and mitigation with a focus on decision problems, present guidelines, and explore areas for further research in this exciting field.

Figures (3)

• Figure 1: Example deployment of an LVLM foundation model in an autonomous driving setting. Hallucinations (pink) may arise at any point in the decision-making pipeline, including information retrieval, planning, perception, and control. In this example, the LVLM correctly queries the map for possible destinations of gas stations, but lists a location that is no longer open. Then, when navigating to one of the locations, the model predicts a path on the map that goes in the wrong direction of traffic flow. Finally, when applied to perception tasks for detecting possible pedestrians in front of the vehicle, the model hallucinates nearby people, causing an improper control action.
• Figure 2: Design process of hallucination intervention methods.
• Figure 3: Example deployment of an LVLM foundation model in a robotics setting. Hallucinations are highlighted pink. Here, a robot tasked with assembling a sandwich initially identifies an object incorrectly. Then, the model comes up with an infeasible plan. Finally, when attempting to perform one of the actions, the robot collides with the human as it did not perceive any danger.

Theorems & Definitions (1)

• Definition 3.1