HalluMix: A Task-Agnostic, Multi-Domain Benchmark for Real-World Hallucination Detection

TL;DR

HalluMix addresses the inadequacy of existing hallucination benchmarks by introducing a large-scale, task-agnostic benchmark that spans multiple domains and formats with multi-document grounding. The dataset builds faithful and hallucinated examples by transforming NLI, summarization, and QA sources into context-response pairs, including retrieval-noise to mimic real-world RAG conditions. Seven detectors (open and closed) are evaluated, revealing that Quotient Detections achieves the best overall accuracy (0.82) and F1 (0.84) but performance varies with task type and content length, with long-form summarization proving particularly challenging. The findings highlight the need for robust detectors that generalize across lengths and formats, and the benchmark provides a resource for guiding deployment in real-world LLM systems.

Abstract

As large language models (LLMs) are increasingly deployed in high-stakes domains, detecting hallucinated content$\unicode{x2013}$text that is not grounded in supporting evidence$\unicode{x2013}$has become a critical challenge. Existing benchmarks for hallucination detection are often synthetically generated, narrowly focused on extractive question answering, and fail to capture the complexity of real-world scenarios involving multi-document contexts and full-sentence outputs. We introduce the HalluMix Benchmark, a diverse, task-agnostic dataset that includes examples from a range of domains and formats. Using this benchmark, we evaluate seven hallucination detection systems$\unicode{x2013}$both open and closed source$\unicode{x2013}$highlighting differences in performance across tasks, document lengths, and input representations. Our analysis highlights substantial performance disparities between short and long contexts, with critical implications for real-world Retrieval Augmented Generation (RAG) implementations. Quotient Detections achieves the best overall performance, with an accuracy of 0.82 and an F1 score of 0.84.

Figures (3)

• Figure 1: Overview of the HalluMix construction pipeline, showing datasets and transformation strategies.
• Figure 2: Comparison of hallucination detection performance metrics (Accuracy, F1, Precision, Recall) across all evaluated methods. Panel (a) shows performance on the complete benchmark dataset, while panel (b) shows performance excluding summarization examples. Quotient Detections achieves highest accuracy and F1 in both scenarios.
• Figure 3: Performance of hallucination detection methods as a function of content length. Panel (a) shows accuracy versus average document token count, revealing how different methods handle increasingly complex contexts. Panel (b) shows accuracy versus answer token count, demonstrating performance on longer responses. Both plots show distinct performance patterns: some methods maintain consistent accuracy across lengths while others show clear degradation with longer content.