Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension

TL;DR

This work tackles the problem of detecting truthfulness in large language model generations by leveraging local intrinsic dimension (LID) of internal activations. It introduces an MLE-based LID estimator and a distance-aware correction (LID-GeoMLE), along with a layer-selection strategy to identify the most informative representations for truthfulness detection. Empirically, LID-GeoMLE outperforms entropy- and verbal uncertainty-based baselines across four generative QA tasks using Llama-2 models, showing robustness to hyperparameters and cross-task references. The analysis reveals rich insights into LLM internals, including a hunchback pattern of LID across layers, lower LID for human-generated content, and increasing intrinsic dimensions with instruction tuning, suggesting LID as a powerful tool for understanding and improving model truthfulness and reliability.

Abstract

We study how to characterize and predict the truthfulness of texts generated from large language models (LLMs), which serves as a crucial step in building trust between humans and LLMs. Although several approaches based on entropy or verbalized uncertainty have been proposed to calibrate model predictions, these methods are often intractable, sensitive to hyperparameters, and less reliable when applied in generative tasks with LLMs. In this paper, we suggest investigating internal activations and quantifying LLM's truthfulness using the local intrinsic dimension (LID) of model activations. Through experiments on four question answering (QA) datasets, we demonstrate the effectiveness ohttps://info.arxiv.org/help/prep#abstractsf our proposed method. Additionally, we study intrinsic dimensions in LLMs and their relations with model layers, autoregressive language modeling, and the training of LLMs, revealing that intrinsic dimensions can be a powerful approach to understanding LLMs.

Figures (12)

• Figure 1: Detecting hallucinations with LIDs. LLM representations of correct answers have smaller intrinsic dimensions.
• Figure 2: Robustness to $n$ and $T$. Performance and intrinsic dimension as a function of the number of neighbors and total reference points. Both plots use the number of neighbors as X-axis. Different line styles indicate different numbers of reference points.
• Figure 3: Plots for the aggregated LID values across model layers on the four QA datasets. The X-axis is the layer id, which is layer 1 to layer 30 for Llama-2-7B. The left Y-axis is the aggregated LID values and the right Y-axis is the detection performance (AUROC) values. The detection performance curve is in orange and the LID curve is in blue with markers. We show that there is a hunchback shape in the LID values across layers. The LID values closely correlate with the performance of detection and exhibit a 'shift behind' phenomenon.
• Figure 4: Bars for intrinsic dimensions of ground-truth answers (orange dashed line) and untruthful model generations (blue line) as the language modeling proceeds. The X-axis represents buckets of different ratios of the total lengths.
• Figure 5: Plots for the accuracy and intrinsic dimension on TriviaQA and TydiQA during instruction tuning. The X-axis is the training steps. We train 3,000 steps in total and show checkpoints every 300 steps. The Y-axis is the performance for the top two figures and the aggregated LID values for the bottom two figures.