Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do LLM hallucination detectors suffer from low-resource effect?

Debtanu Datta, Mohan Kishore Chilukuri, Yash Kumar, Saptarshi Ghosh, Muhammad Bilal Zafar

TL;DR

This work probes the robustness of hallucination detectors under the low-resource language effect in LLMs. It combines two multilingual QA benchmarks, mTREx and G-MMLU, across five languages and four models, evaluating three HD approaches (MAM variants, SGM, SEM) and defining a cross-language metric, TPHR, to compare task accuracy and detector performance. The study finds that detectors generally degrade far less than the underlying task accuracy in low-resource languages, with multilingual training mitigating cross-lingual transfer gaps; model-artifact detectors (MAM) consistently outperform black-box methods. Overall, the results suggest that detectors exploit clearer uncertainty signals and remain robust across languages, offering practical guidance for multilingual reliability in LLM systems. $TPHR(L) = \log_{10}\left( \frac{|\text{Accuracy}(\text{EN}) - \text{Accuracy}(L)|}{|\text{AUROC}_{\text{HD}}(\text{EN}) - \text{AUROC}_{\text{HD}}(L)|} \right)$.

Abstract

LLMs, while outperforming humans in a wide range of tasks, can still fail in unanticipated ways. We focus on two pervasive failure modes: (i) hallucinations, where models produce incorrect information about the world, and (ii) the low-resource effect, where the models show impressive performance in high-resource languages like English but the performance degrades significantly in low-resource languages like Bengali. We study the intersection of these issues and ask: do hallucination detectors suffer from the low-resource effect? We conduct experiments on five tasks across three domains (factual recall, STEM, and Humanities). Experiments with four LLMs and three hallucination detectors reveal a curious finding: As expected, the task accuracies in low-resource languages experience large drops (compared to English). However, the drop in detectors' accuracy is often several times smaller than the drop in task accuracy. Our findings suggest that even in low-resource languages, the internal mechanisms of LLMs might encode signals about their uncertainty. Further, the detectors are robust within language (even for non-English) and in multilingual setups, but not in cross-lingual settings without in-language supervision.

Do LLM hallucination detectors suffer from low-resource effect?

TL;DR

This work probes the robustness of hallucination detectors under the low-resource language effect in LLMs. It combines two multilingual QA benchmarks, mTREx and G-MMLU, across five languages and four models, evaluating three HD approaches (MAM variants, SGM, SEM) and defining a cross-language metric, TPHR, to compare task accuracy and detector performance. The study finds that detectors generally degrade far less than the underlying task accuracy in low-resource languages, with multilingual training mitigating cross-lingual transfer gaps; model-artifact detectors (MAM) consistently outperform black-box methods. Overall, the results suggest that detectors exploit clearer uncertainty signals and remain robust across languages, offering practical guidance for multilingual reliability in LLM systems. .

Abstract

LLMs, while outperforming humans in a wide range of tasks, can still fail in unanticipated ways. We focus on two pervasive failure modes: (i) hallucinations, where models produce incorrect information about the world, and (ii) the low-resource effect, where the models show impressive performance in high-resource languages like English but the performance degrades significantly in low-resource languages like Bengali. We study the intersection of these issues and ask: do hallucination detectors suffer from the low-resource effect? We conduct experiments on five tasks across three domains (factual recall, STEM, and Humanities). Experiments with four LLMs and three hallucination detectors reveal a curious finding: As expected, the task accuracies in low-resource languages experience large drops (compared to English). However, the drop in detectors' accuracy is often several times smaller than the drop in task accuracy. Our findings suggest that even in low-resource languages, the internal mechanisms of LLMs might encode signals about their uncertainty. Further, the detectors are robust within language (even for non-English) and in multilingual setups, but not in cross-lingual settings without in-language supervision.
Paper Structure (36 sections, 2 equations, 18 figures, 43 tables)

This paper contains 36 sections, 2 equations, 18 figures, 43 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multilingual QA Datasets
  4. Multilingual TREx (mTREx) dataset
  5. The Global-MMLU (G-MMLU) dataset
  6. Hallucination Detection Methods
  7. Model Artifacts Method ( MAM)
  8. SelfCheckGPT Method ( SGM)
  9. Semantic Entropy Method ( SEM)
  10. Defining Hallucinations
  11. Experimental Setup
  12. Models
  13. Prompts
  14. Evaluation Metrics
  15. Results and Observations
  16. ...and 21 more sections

Figures (18)

  • Figure 1: Comparison of task accuracy ($1$ denotes a correct answer, $0$ denotes a hallucination) of a model ( LAM-70B) with hallucination detector's (MAM) performance across languages on mTREx-Capitals dataset. When going from English to lower-resource languages, the task accuracy drops significantly. But the hallucination detector's performance remains relatively stable.
  • Figure 2: Example from mTREx-Capitals along with the response from LAM-70B across five languages. The model answers correctly in English ( EN) and German ( DE) but hallucinates in the low-resource languages.
  • Figure 3: Example prompt for mTREx (Country relationship) in English.
  • Figure 4: Prompt for G-MMLU in English.
  • Figure 5: Task accuracy vs. token compression ratio for the Capitals dataset. Low-resource languages have higher compression ratios, showing inefficient tokenization, along with lower task performance.
  • ...and 13 more figures