Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Non-Linear Inference Time Intervention: Improving LLM Truthfulness

Jakub Hoscilowicz, Adam Wiacek, Jan Chojnacki, Adam Cieslak, Leszek Michon, Vitalii Urbanevych, Artur Janicki

TL;DR

The paper tackles the problem of improving truthfulness in LLM outputs without fine-tuning by inspecting and biasing the model's internal representations. It introduces NL-ITI, a non-linear probing and multi-token intervention framework that identifies truth-relevant attention heads and biases their activations during inference using multi-token context. Empirical results on LLaMA-2-7B show NL-ITI significantly surpasses the original ITI and exceeds the recent Truth Forest method, achieving notable gains on TruthfulQA and strong generalization across ARC, MMLU, and OBQA. The findings demonstrate a data-efficient approach to steer LLM behavior toward truthfulness, with potential for safer and more accountable AI systems, while acknowledging the need for labeled data and avenues for unsupervised extensions.

Abstract

In this work, we explore LLM's internal representation space to identify attention heads that contain the most truthful and accurate information. We further developed the Inference Time Intervention (ITI) framework, which lets bias LLM without the need for fine-tuning. The improvement manifests in introducing a non-linear multi-token probing and multi-token intervention: Non-Linear ITI (NL-ITI), which significantly enhances performance on evaluation benchmarks. NL-ITI is tested on diverse multiple-choice datasets, including TruthfulQA, on which we report over 16% relative MC1 (accuracy of model pointing to the correct answer) improvement with respect to the baseline ITI results. Moreover, we achieved a 10% relative improvement over the recently released Truth Forest (TrFf) method that also focused on ITI improvement.

Non-Linear Inference Time Intervention: Improving LLM Truthfulness

TL;DR

The paper tackles the problem of improving truthfulness in LLM outputs without fine-tuning by inspecting and biasing the model's internal representations. It introduces NL-ITI, a non-linear probing and multi-token intervention framework that identifies truth-relevant attention heads and biases their activations during inference using multi-token context. Empirical results on LLaMA-2-7B show NL-ITI significantly surpasses the original ITI and exceeds the recent Truth Forest method, achieving notable gains on TruthfulQA and strong generalization across ARC, MMLU, and OBQA. The findings demonstrate a data-efficient approach to steer LLM behavior toward truthfulness, with potential for safer and more accountable AI systems, while acknowledging the need for labeled data and avenues for unsupervised extensions.

Abstract

In this work, we explore LLM's internal representation space to identify attention heads that contain the most truthful and accurate information. We further developed the Inference Time Intervention (ITI) framework, which lets bias LLM without the need for fine-tuning. The improvement manifests in introducing a non-linear multi-token probing and multi-token intervention: Non-Linear ITI (NL-ITI), which significantly enhances performance on evaluation benchmarks. NL-ITI is tested on diverse multiple-choice datasets, including TruthfulQA, on which we report over 16% relative MC1 (accuracy of model pointing to the correct answer) improvement with respect to the baseline ITI results. Moreover, we achieved a 10% relative improvement over the recently released Truth Forest (TrFf) method that also focused on ITI improvement.
Paper Structure (8 sections, 6 equations, 3 figures, 2 tables)

This paper contains 8 sections, 6 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Proposed improvement
  5. Experiments
  6. Evaluation
  7. Results
  8. Conclusions

Figures (3)

  • Figure 1: How MC1 correlates with KL divergence. The results were collected for ITI and NL-ITI using different hyperparameter sets ($\alpha$, heads to intervene) for TruthfulQA and OpenBookQA datasets. On each benchmark, baseline LLaMA-2-7B performance is shown.
  • Figure 2: Probing accuracy for each attention head of the LLM on TruthfulQA dataset for linear probing (ITI) -- bottom, and non-linear probing (NL-ITI) -- top. Accuracy results are 'smoothed' between neighboring attention heads (lower standard deviation).
  • Figure 3: Heat map of MC1 evaluation scores of TruthfulQA dataset for different combinations of number of tokens used during probing and intervention. The best performing model corresponds to $(\rho, \tau) = (6,4)$.