Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM Factoscope: Uncovering LLMs' Factual Discernment through Inner States Analysis

Jinwen He, Yujia Gong, Kai Chen, Zijin Lin, Chengan Wei, Yue Zhao

TL;DR

This work tackles the problem of factuality in LLM outputs by exploiting the models' inner states rather than external references. It introduces LLM Factoscope, a Siamese-network-based pipeline that collects four inner-state signals (activation maps, final output ranks, top-k indices, and top-k probabilities) from last-token processing and trains a multi-branch embedding model using triplet margin loss. Across diverse architectures, the method achieves over 96% accuracy in distinguishing factual versus non-factual outputs, with extensive ablations and generalization tests demonstrating robustness and domain transferability. The study advances reliability and transparency in LLMs by leveraging internal representations for real-time factual detection and prompts further exploration of inner workings for safer AI systems.

Abstract

Large Language Models (LLMs) have revolutionized various domains with extensive knowledge and creative capabilities. However, a critical issue with LLMs is their tendency to produce outputs that diverge from factual reality. This phenomenon is particularly concerning in sensitive applications such as medical consultation and legal advice, where accuracy is paramount. In this paper, we introduce the LLM factoscope, a novel Siamese network-based model that leverages the inner states of LLMs for factual detection. Our investigation reveals distinguishable patterns in LLMs' inner states when generating factual versus non-factual content. We demonstrate the LLM factoscope's effectiveness across various architectures, achieving over 96% accuracy in factual detection. Our work opens a new avenue for utilizing LLMs' inner states for factual detection and encourages further exploration into LLMs' inner workings for enhanced reliability and transparency.

LLM Factoscope: Uncovering LLMs' Factual Discernment through Inner States Analysis

TL;DR

This work tackles the problem of factuality in LLM outputs by exploiting the models' inner states rather than external references. It introduces LLM Factoscope, a Siamese-network-based pipeline that collects four inner-state signals (activation maps, final output ranks, top-k indices, and top-k probabilities) from last-token processing and trains a multi-branch embedding model using triplet margin loss. Across diverse architectures, the method achieves over 96% accuracy in distinguishing factual versus non-factual outputs, with extensive ablations and generalization tests demonstrating robustness and domain transferability. The study advances reliability and transparency in LLMs by leveraging internal representations for real-time factual detection and prompts further exploration of inner workings for safer AI systems.

Abstract

Large Language Models (LLMs) have revolutionized various domains with extensive knowledge and creative capabilities. However, a critical issue with LLMs is their tendency to produce outputs that diverge from factual reality. This phenomenon is particularly concerning in sensitive applications such as medical consultation and legal advice, where accuracy is paramount. In this paper, we introduce the LLM factoscope, a novel Siamese network-based model that leverages the inner states of LLMs for factual detection. Our investigation reveals distinguishable patterns in LLMs' inner states when generating factual versus non-factual content. We demonstrate the LLM factoscope's effectiveness across various architectures, achieving over 96% accuracy in factual detection. Our work opens a new avenue for utilizing LLMs' inner states for factual detection and encourages further exploration into LLMs' inner workings for enhanced reliability and transparency.
Paper Structure (19 sections, 11 figures, 5 tables)

This paper contains 19 sections, 11 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Logit Lens
  4. LLM Factual Detection
  5. Observations
  6. LLM Factoscope
  7. Overview
  8. Factual Data Collection
  9. Inner States Collection
  10. Inner States Preprocessing
  11. LLM Factoscope Model Design
  12. Evaluation
  13. Experimental Setup
  14. Effectiveness
  15. Generalization
  16. ...and 4 more sections

Figures (11)

  • Figure 1: Activation maps of layers 24-42, showing the top 1% factual-related neurons.
  • Figure 2: Case study of final output rank.
  • Figure 3: Case study of the top-4 output indices and their probabilities in the last five layers
  • Figure 4: Pipeline of the LLM Factoscope.
  • Figure 5: An instance of using LLM Factoscope.
  • ...and 6 more figures