DSC2025 -- ViHallu Challenge: Detecting Hallucination in Vietnamese LLMs

TL;DR

This work introduces ViHallu, the first large-scale benchmark for detecting hallucinations in Vietnamese LLMs, addressing a critical gap in low-resource language evaluation. It provides a 10,000-sample dataset of (Context, Prompt, Response) triplets labeled as No Hallucination, Intrinsic, or Extrinsic, with factual, noisy, and adversarial prompts to stress robustness. Experiments show instruction-tuned LLMs with structured prompting and ensemble methods achieve Macro-F1 around $0.8480$, far surpassing the encoder baseline of $0.3283$, yet still reveal substantial room for improvement, especially on intrinsic cases. The dataset, released under CC-BY-SA 4.0, establishes a rigorous foundation for researching trustworthiness and safe deployment of Vietnamese AI systems and points to future directions in retrieval-augmented verification, span-level analysis, and multilingual expansion.

Abstract

The reliability of large language models (LLMs) in production environments remains significantly constrained by their propensity to generate hallucinations -- fluent, plausible-sounding outputs that contradict or fabricate information. While hallucination detection has recently emerged as a priority in English-centric benchmarks, low-to-medium resource languages such as Vietnamese remain inadequately covered by standardized evaluation frameworks. This paper introduces the DSC2025 ViHallu Challenge, the first large-scale shared task for detecting hallucinations in Vietnamese LLMs. We present the ViHallu dataset, comprising 10,000 annotated triplets of (context, prompt, response) samples systematically partitioned into three hallucination categories: no hallucination, intrinsic, and extrinsic hallucinations. The dataset incorporates three prompt types -- factual, noisy, and adversarial -- to stress-test model robustness. A total of 111 teams participated, with the best-performing system achieving a macro-F1 score of 84.80\%, compared to a baseline encoder-only score of 32.83\%, demonstrating that instruction-tuned LLMs with structured prompting and ensemble strategies substantially outperform generic architectures. However, the gap to perfect performance indicates that hallucination detection remains a challenging problem, particularly for intrinsic (contradiction-based) hallucinations. This work establishes a rigorous benchmark and explores a diverse range of detection methodologies, providing a foundation for future research into the trustworthiness and reliability of Vietnamese language AI systems.

Figures (2)

• Figure 1: Illustration of the ViHallu task: given a context, prompt, and model response, the system predicts whether the response is hallucinated and, if so, its type.
• Figure 2: Kernel density estimates of token lengths for Context, Prompt, and Response across dataset splits.