Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration

TL;DR

This work addresses the persistent knowledge gaps in large language models by reframing abstention as a reliability mechanism in QA. It surveys 11 baseline abstain methods across calibration, training, prompting, and consistency, and introduces two multi-LLM collaboration strategies—Cooperate and Compete—that leverage cross-model feedback to decide abstention. Across three LLMs and four knowledge-intensive tasks, the collaboration approaches achieve state-of-the-art abstain performance, with notable gains in abstain accuracy and retrieval-aware reliability. The study highlights the potential of multi-LLM collaboration to reduce hallucinations and pinpoint knowledge gaps, while also discussing computational costs and fairness considerations in abstention.

Abstract

Despite efforts to expand the knowledge of large language models (LLMs), knowledge gaps -- missing or outdated information in LLMs -- might always persist given the evolving nature of knowledge. In this work, we study approaches to identify LLM knowledge gaps and abstain from answering questions when knowledge gaps are present. We first adapt existing approaches to model calibration or adaptation through fine-tuning/prompting and analyze their ability to abstain from generating low-confidence outputs. Motivated by their failures in self-reflection and over-reliance on held-out sets, we propose two novel approaches that are based on model collaboration, i.e., LLMs probing other LLMs for knowledge gaps, either cooperatively or competitively. Extensive experiments with three LLMs on four QA tasks featuring diverse knowledge domains demonstrate that both cooperative and competitive approaches to unveiling LLM knowledge gaps achieve up to 19.3% improvements on abstain accuracy against the strongest baseline. Further analysis reveals that our proposed mechanisms could help identify failure cases in retrieval augmentation and pinpoint knowledge gaps in multi-hop reasoning.

Figures (12)

• Figure 1: Overview of collaboration-based approaches for LLM abstention: Cooperate and Compete.
• Figure 2: Four outcomes of AbstainQA.
• Figure 3: Performance of abstain mechanisms in the abstain absolute scenarios where the LLM should abstain for 100% of questions. Compete achieves the highest abstention rate on average across LLMs and datasets.
• Figure 4: Performance of Compete with Mistral-7B in the two-step abstain and retrieval setting. The proposed abstain-retrieve-abstain pipeline successfully reduces the incorrect rate by at least 21.2%.
• Figure 5: Abstain rate on Election23QA with Mistral-7B divided into where the election takes place. The lowest abstain rate for each approach among continents is highlighted in bold.