Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CCNU at SemEval-2025 Task 3: Leveraging Internal and External Knowledge of Large Language Models for Multilingual Hallucination Annotation

Xu Liu, Guanyi Chen

TL;DR

The paper tackles multilingual QA hallucination detection by extending Mu-SHROOM to 14 languages and requiring token-level localization. It proposes a crowdsourced-like LLM ensemble that runs in parallel, with each annotator leveraging both internal and external knowledge, avoiding fine-tuning. Experiments compare backbone LLMs (notably DeepSeek-V3 and GPT-4o variants) and show that incorporating knowledge improves $IoU$ across languages, with Hindi achieving top performance. It also discusses unsuccessful approaches and dataset quality issues, offering insights for future improvements and the interpretation of hallucinations in QA.

Abstract

We present the system developed by the Central China Normal University (CCNU) team for the Mu-SHROOM shared task, which focuses on identifying hallucinations in question-answering systems across 14 different languages. Our approach leverages multiple Large Language Models (LLMs) with distinct areas of expertise, employing them in parallel to annotate hallucinations, effectively simulating a crowdsourcing annotation process. Furthermore, each LLM-based annotator integrates both internal and external knowledge related to the input during the annotation process. Using the open-source LLM DeepSeek-V3, our system achieves the top ranking (\#1) for Hindi data and secures a Top-5 position in seven other languages. In this paper, we also discuss unsuccessful approaches explored during our development process and share key insights gained from participating in this shared task.

CCNU at SemEval-2025 Task 3: Leveraging Internal and External Knowledge of Large Language Models for Multilingual Hallucination Annotation

TL;DR

The paper tackles multilingual QA hallucination detection by extending Mu-SHROOM to 14 languages and requiring token-level localization. It proposes a crowdsourced-like LLM ensemble that runs in parallel, with each annotator leveraging both internal and external knowledge, avoiding fine-tuning. Experiments compare backbone LLMs (notably DeepSeek-V3 and GPT-4o variants) and show that incorporating knowledge improves across languages, with Hindi achieving top performance. It also discusses unsuccessful approaches and dataset quality issues, offering insights for future improvements and the interpretation of hallucinations in QA.

Abstract

We present the system developed by the Central China Normal University (CCNU) team for the Mu-SHROOM shared task, which focuses on identifying hallucinations in question-answering systems across 14 different languages. Our approach leverages multiple Large Language Models (LLMs) with distinct areas of expertise, employing them in parallel to annotate hallucinations, effectively simulating a crowdsourcing annotation process. Furthermore, each LLM-based annotator integrates both internal and external knowledge related to the input during the annotation process. Using the open-source LLM DeepSeek-V3, our system achieves the top ranking (\#1) for Hindi data and secures a Top-5 position in seven other languages. In this paper, we also discuss unsuccessful approaches explored during our development process and share key insights gained from participating in this shared task.
Paper Structure (20 sections, 3 figures, 6 tables)

This paper contains 20 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. The Mu-SHROOM Task
  3. Methodology
  4. Prompting LLMs to Mark Hallucinations
  5. Internal Knowledge
  6. External Knowledge
  7. Experiments
  8. The Choice of Backbone LLMs.
  9. Results on the Test Sets.
  10. The Effect of Marking Hallucinations in Place.
  11. Unsuccessful Approaches
  12. Ignoring Typos.
  13. Correcting before Annotating.
  14. Incorporating External Knowledge without Summarising.
  15. Discussion
  16. ...and 5 more sections

Figures (3)

  • Figure 1: An overview of our hallucination annotation system. The blue whale represents LLM (i.e., DeepSeek).
  • Figure 2: The main prompt in our system. The variables highlighted in yellow will be replaced with their corresponding desired values.
  • Figure 3: Performance in terms of IoU on 10 languages whose development sets are available.