HalluHard: A Hard Multi-Turn Hallucination Benchmark

TL;DR

HalluHard tackles the problem of multi-turn hallucinations by introducing a hard, verifiable benchmark spanning four high-stakes domains and 950 seed conversations, augmented with a web-search–driven judge that retrieves full-text sources to verify inline citations. The authors design a two-path judge (claim-based for most domains, response-based for coding) to assess both reference grounding and content grounding, and they reveal that even strongest models exhibit substantial hallucinations when citations must be verifiable. Key findings show that model capacity and turn position influence hallucination rates, while reasoning can reduce errors only sometimes; content grounding remains a major challenge even with web search. The work highlights the need for explicit uncertainty handling and faithful retrieval/reading of sources to improve reliability in open-ended, multi-turn LLM interactions.

Abstract

Large language models (LLMs) still produce plausible-sounding but ungrounded factual claims, a problem that worsens in multi-turn dialogue as context grows and early errors cascade. We introduce $\textbf{HalluHard}$, a challenging multi-turn hallucination benchmark with 950 seed questions spanning four high-stakes domains: legal cases, research questions, medical guidelines, and coding. We operationalize groundedness by requiring inline citations for factual assertions. To support reliable evaluation in open-ended settings, we propose a judging pipeline that iteratively retrieves evidence via web search. It can fetch, filter, and parse full-text sources (including PDFs) to assess whether cited material actually supports the generated content. Across a diverse set of frontier proprietary and open-weight models, hallucinations remain substantial even with web search ($\approx 30\%$ for the strongest configuration, Opus-4.5 with web search), with content-grounding errors persisting at high rates. Finally, we show that hallucination behavior is shaped by model capacity, turn position, effective reasoning, and the type of knowledge required.

Figures (9)

• Figure 1: Average hallucination rate on HalluHard that contains 950 multi-turn conversations across legal, research, medical, and coding domains. WS denotes web search. Lower values are better. Our challenging benchmark reveals that even frontier LLMs like Opus-4.5 hallucinate in more than 30% of cases with web search and 60% without.
• Figure 2: An example of a hallucinated claim from our judge. A claim is classified as hallucination if either reference or content grounding failure happens.
• Figure 3: Our multi-turn response generation pipeline. Seed queries are provided by HalluHard, and follow-up queries are generated via a user LLM.
• Figure 4: Our claim-based verification pipeline. For each claim, we check whether the reference is correct and whether the claimed content is grounded in that reference.
• Figure 5: Evaluation time and cost comparison for three different LLM judge pipelines. The time and cost are gathered from evaluating 10 responses ($\sim$120 atomic claims in total).