Invoke Interfaces Only When Needed: Adaptive Invocation for Large Language Models in Question Answering

TL;DR

The paper tackles real-time hallucination detection in collaborative large-small LM QA systems by introducing AttenHScore, a token-level metric that captures accumulation and propagation of errors during SLM generation. It combines AttenHScore-based real-time detection with a dynamic threshold and an uncertainty-driven re-ranking mechanism to decide when to invoke LLMs, all in a plug-and-play, training-free framework. Empirical results across CoQA, SQuAD, TriviaQA, and Natural Questions show that AttenHScore improves hallucination detection and QA performance in LLM-SLM collaboration, often matching or exceeding LLM-only baselines under limited LLM-call budgets. The approach emphasizes retrieval-aware information arrangement and hotline-style safety nets, offering a practical path to cost-efficient, scalable QA systems with robust performance on complex questions.

Abstract

The collaborative paradigm of large and small language models (LMs) effectively balances performance and cost, yet its pivotal challenge lies in precisely pinpointing the moment of invocation when hallucinations arise in small LMs. Previous optimization efforts primarily focused on post-processing techniques, which were separate from the reasoning process of LMs, resulting in high computational costs and limited effectiveness. In this paper, we propose a practical invocation evaluation metric called AttenHScore, which calculates the accumulation and propagation of hallucinations during the generation process of small LMs, continuously amplifying potential reasoning errors. By dynamically adjusting the detection threshold, we achieve more accurate real-time invocation of large LMs. Additionally, considering the limited reasoning capacity of small LMs, we leverage uncertainty-aware knowledge reorganization to assist them better capture critical information from different text chunks. Extensive experiments reveal that our AttenHScore outperforms most baselines in enhancing real-time hallucination detection capabilities across multiple QA datasets, especially when addressing complex queries. Moreover, our strategies eliminate the need for additional model training and display flexibility in adapting to various transformer-based LMs.

Figures (9)

• Figure 1: Performance of large and small LMs on different QA datasets in the RAG scenario. Including 1: 2WikiMultihopQA, 2: MultiFieldQA-en, 3: Qasper, 4: MultiFieldQA-zh and 5: HotpotQA.
• Figure 2: Overview of our hallucination detection and collaborative framework.
• Figure 3: Performance comparison between the re-ranking method based on uncertainty evaluation and commonly used re-ranking models. Among them, the one starting with $G$ represents our approach, and the rest of the models are all from huggingface.
• Figure 4: Performance sensitivity to temperature on Dataset SQuAD.
• Figure 5: Performance sensitivity to top-k on Dataset SQuAD.