Small Agent Can Also Rock! Empowering Small Language Models as Hallucination Detector

TL;DR

This work presents HaluAgent, an autonomous agent that equips relatively small open-source LLMs with a multi-tool framework to detect a broad spectrum of hallucinations (text, code, math) in both Chinese and English. By introducing a fine-grained three-stage detection process (sentence segmentation, tool-based verification, and reflection) and memory, and by synthesizing 2,017 detection trajectories for fine-tuning Baichuan2-Chat 7B/13B, the authors demonstrate that small models can reach or exceed GPT-4 performance on diverse in-domain and out-of-domain tasks without relying on closed-source APIs. The approach significantly improves response- and sentence-level detection, showing strong generalization and extensibility when adding new tools. The work highlights the practical potential of open-source LLMs for robust hallucination detection in real-world AI interactions, with release of data and code to support reproducibility and further research.

Abstract

Hallucination detection is a challenging task for large language models (LLMs), and existing studies heavily rely on powerful closed-source LLMs such as GPT-4. In this paper, we propose an autonomous LLM-based agent framework, called HaluAgent, which enables relatively smaller LLMs (e.g. Baichuan2-Chat 7B) to actively select suitable tools for detecting multiple hallucination types such as text, code, and mathematical expression. In HaluAgent, we integrate the LLM, multi-functional toolbox, and design a fine-grained three-stage detection framework along with memory mechanism. To facilitate the effectiveness of HaluAgent, we leverage existing Chinese and English datasets to synthesize detection trajectories for fine-tuning, which endows HaluAgent with the capability for bilingual hallucination detection. Extensive experiments demonstrate that only using 2K samples for tuning LLMs, HaluAgent can perform hallucination detection on various types of tasks and datasets, achieving performance comparable to or even higher than GPT-4 without tool enhancements on both in-domain and out-of-domain datasets. We release our dataset and code at https://github.com/RUCAIBox/HaluAgent.

Figures (7)

• Figure 1: The overview of our proposed HaluAgent. The left part shows the process of fine-tuning open-source models and detecting hallucinations. The right part illustrates the hallucination detection pipeline of HaluAgent.
• Figure 2: The usage rate of new tools and the proportion of successful detection.
• Figure 3: Case study between GPT-4 with a simple prompt and single tool, and the HaluAgent framework.
• Figure 4: Instructions of HaluAgent framework in English.
• Figure 5: Instructions of HaluAgent framework in Chinese.