HalluMat: Detecting Hallucinations in LLM-Generated Materials Science Content Through Multi-Stage Verification

TL;DR

The paper addresses hallucinations in LLM-generated materials science content by introducing HalluMatData, a domain-specific benchmark, and HalluMatDetector, a multi-stage verification framework combining intrinsic self-consistency checks, selective extrinsic fact verification, and graph-based contradiction analysis. It defines Paraphrased Hallucination Consistency Score as $PHCS = \sqrt{\frac{1}{N} \sum_{i=1}^{N} (H_i - \bar{H})^2}$ to quantify response variability across paraphrases and demonstrates that HalluMatDetector reduces hallucination rates by about 30% while achieving 82.2% accuracy on HalluMatData. The approach highlights subdomain variation and high-entropy queries as more prone to factual errors, providing a scalable method for domain-specific factual grounding through hierarchical retrieval and knowledge-graph analysis. The work offers a practical path toward more trustworthy AI-assisted materials discovery and can be extended to other scientific domains with improved embeddings and RLHF-based mitigation.

Abstract

Artificial Intelligence (AI), particularly Large Language Models (LLMs), is transforming scientific discovery, enabling rapid knowledge generation and hypothesis formulation. However, a critical challenge is hallucination, where LLMs generate factually incorrect or misleading information, compromising research integrity. To address this, we introduce HalluMatData, a benchmark dataset for evaluating hallucination detection methods, factual consistency, and response robustness in AI-generated materials science content. Alongside this, we propose HalluMatDetector, a multi-stage hallucination detection framework that integrates intrinsic verification, multi-source retrieval, contradiction graph analysis, and metric-based assessment to detect and mitigate LLM hallucinations. Our findings reveal that hallucination levels vary significantly across materials science subdomains, with high-entropy queries exhibiting greater factual inconsistencies. By utilizing HalluMatDetector verification pipeline, we reduce hallucination rates by 30% compared to standard LLM outputs. Furthermore, we introduce the Paraphrased Hallucination Consistency Score (PHCS) to quantify inconsistencies in LLM responses across semantically equivalent queries, offering deeper insights into model reliability.

Figures (7)

• Figure 1: Recomputed Hallucination Scores
• Figure 2: Paraphrased Hallucination Consistency Score (PHCS) Across Groups.
• Figure 3: Heatmap of fact fragments similarity. (This visualization depicts semantic similarity between fact fragments. Higher similarity suggests factual consistency.)
• Figure 4: 2D Knowledge Graph Visualization. (This graph illustrates fact clustering in AI-generated responses, where disconnected nodes indicate contradictions.)
• Figure 5: 3D Knowledge Graph Visualization. (This visualization presents relationships between fact clusters. Higher fragmentation suggests hallucination.)