Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty

Johannes Kirmayr, Lukas Stappen, Elisabeth André

TL;DR

CAR-bench Addresses a critical gap in evaluating LLM agents under real-world uncertainty by introducing a dynamic, policy-constrained in-car assistant benchmark. It couples a llm-simulated user, 19 automotive policies, and 58 tools in a realistic environment, with Hallucination and Disambiguation task types to measure limit-awareness and uncertainty handling. Using Pass^k and Pass@k metrics, the study reveals substantial gaps between a model's potential and its consistent performance, especially in disambiguation, and analyzes error taxonomies and practical deployment considerations. The work highlights the need for reliable, self-aware agents and suggests directions in architecture, domain-specific training, and safety layers to bridge the gap for real-world automotive AI systems.

Abstract

Existing benchmarks for Large Language Model (LLM) agents focus on task completion under idealistic settings but overlook reliability in real-world, user-facing applications. In domains, such as in-car voice assistants, users often issue incomplete or ambiguous requests, creating intrinsic uncertainty that agents must manage through dialogue, tool use, and policy adherence. We introduce CAR-bench, a benchmark for evaluating consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. The environment features an LLM-simulated user, domain policies, and 58 interconnected tools spanning navigation, productivity, charging, and vehicle control. Beyond standard task completion, CAR-bench introduces Hallucination tasks that test agents' limit-awareness under missing tools or information, and Disambiguation tasks that require resolving uncertainty through clarification or internal information gathering. Baseline results reveal large gaps between occasional and consistent success on all task types. Even frontier reasoning LLMs achieve less than 50% consistent pass rate on Disambiguation tasks due to premature actions, and frequently violate policies or fabricate information to satisfy user requests in Hallucination tasks, underscoring the need for more reliable and self-aware LLM agents in real-world settings.

CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty

TL;DR

CAR-bench Addresses a critical gap in evaluating LLM agents under real-world uncertainty by introducing a dynamic, policy-constrained in-car assistant benchmark. It couples a llm-simulated user, 19 automotive policies, and 58 tools in a realistic environment, with Hallucination and Disambiguation task types to measure limit-awareness and uncertainty handling. Using Pass^k and Pass@k metrics, the study reveals substantial gaps between a model's potential and its consistent performance, especially in disambiguation, and analyzes error taxonomies and practical deployment considerations. The work highlights the need for reliable, self-aware agents and suggests directions in architecture, domain-specific training, and safety layers to bridge the gap for real-world automotive AI systems.

Abstract

Existing benchmarks for Large Language Model (LLM) agents focus on task completion under idealistic settings but overlook reliability in real-world, user-facing applications. In domains, such as in-car voice assistants, users often issue incomplete or ambiguous requests, creating intrinsic uncertainty that agents must manage through dialogue, tool use, and policy adherence. We introduce CAR-bench, a benchmark for evaluating consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. The environment features an LLM-simulated user, domain policies, and 58 interconnected tools spanning navigation, productivity, charging, and vehicle control. Beyond standard task completion, CAR-bench introduces Hallucination tasks that test agents' limit-awareness under missing tools or information, and Disambiguation tasks that require resolving uncertainty through clarification or internal information gathering. Baseline results reveal large gaps between occasional and consistent success on all task types. Even frontier reasoning LLMs achieve less than 50% consistent pass rate on Disambiguation tasks due to premature actions, and frequently violate policies or fabricate information to satisfy user requests in Hallucination tasks, underscoring the need for more reliable and self-aware LLM agents in real-world settings.
Paper Structure (54 sections, 3 equations, 4 figures, 6 tables)

This paper contains 54 sections, 3 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. LLM Agent Benchmarks.
  5. Multi-turn Dialogue and User Simulation.
  6. Hallucination, Disambiguation, and Reasoning.
  7. Domain-Specific Policies.
  8. CAR-bench
  9. Benchmark Components
  10. llm-simulated User.
  11. llm-Agent with domain-specific policies.
  12. Tools.
  13. State and Context Variables.
  14. Databases.
  15. Dataset and Task Types
  16. ...and 39 more sections

Figures (4)

  • Figure 1: Overview of the CAR-bench components. (a) An LLM-simulated user generates multi-turn messages following task instructions (1); (b) the LLM agent, guided by domain policies, interacts with (c) tools to (2a) observe the environment or (2b) modify its state, until producing an informed response (3). The environment consists of (d) mutable states, (e) fixed context variables, and (f) static databases. The user instructions show the Base task type, the task types Hallucination and Disambiguation are explained in Section \ref{['sec:task_types']}.
  • Figure 2: Overview of dataset task types. (1) Base: the agent succeeds if it reaches the ground-truth end-state without violating the policy. (2) Hallucination: a required (a) tool, (b) tool parameter, or (c) tool result is removed, making task completion impossible; success requires the agent to acknowledge its inability due to this missing capability or information. (3) Disambiguation: the task is modified to include ambiguity that the agent must resolve, either (a) externally through user interaction or (b) by leveraging internal information.
  • Figure 3: Pass^1 (=Pass@1) rates by action count in Base tasks, averaged over thinking and non-thinking models.
  • Figure 4: Task-level metric (ref. Sec. \ref{['sec:task_level_evaluation']}) error rates for the Base tasks. Final and intermediate action metrics are condensed into Actions. The user end conversation metric is excluded as its error rate was zero. Multiple metric failures can co-occur per task.