Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inference-Time Intervention in Large Language Models for Reliable Requirement Verification

Paul Darm, James Xie, Annalisa Riccardi

TL;DR

This work tackles the challenge of steering large language models (LLMs) for reliability in safety-critical MBSE tasks. It proposes inference-time intervention (ITI) to modulate model outputs by adjusting activations, avoiding gradient updates, and applies it to verify requirements over graph representations extracted from Capella models. The approach identifies a small set of attention heads whose targeted intervention, combined with self-consistency, yields near-perfect precision on holdout data and strong generalization across two space-mission designs. The results demonstrate ITI as a scalable, interpretable alternative to fine-tuning for precise requirement verification in engineering contexts, with potential for broader safety-critical applications.

Abstract

Steering the behavior of Large Language Models (LLMs) remains a challenge, particularly in engineering applications where precision and reliability are critical. While fine-tuning and prompting methods can modify model behavior, they lack the dynamic and exact control necessary for engineering applications. Inference-time intervention techniques provide a promising alternative, allowing targeted adjustments to LLM outputs. In this work, we demonstrate how interventions enable fine-grained control for automating the usually time-intensive requirement verification process in Model-Based Systems Engineering (MBSE). Using two early-stage Capella SysML models of space missions with associated requirements, we apply the intervened LLMs to reason over a graph representation of the model to determine whether a requirement is fulfilled. Our method achieves robust and reliable outputs, significantly improving over both a baseline model and a fine-tuning approach. By identifying and modifying as few as one to three specialised attention heads, we can significantly change the model's behavior. When combined with self-consistency, this allows us to achieve perfect precision on our holdout test set.

Inference-Time Intervention in Large Language Models for Reliable Requirement Verification

TL;DR

This work tackles the challenge of steering large language models (LLMs) for reliability in safety-critical MBSE tasks. It proposes inference-time intervention (ITI) to modulate model outputs by adjusting activations, avoiding gradient updates, and applies it to verify requirements over graph representations extracted from Capella models. The approach identifies a small set of attention heads whose targeted intervention, combined with self-consistency, yields near-perfect precision on holdout data and strong generalization across two space-mission designs. The results demonstrate ITI as a scalable, interpretable alternative to fine-tuning for precise requirement verification in engineering contexts, with potential for broader safety-critical applications.

Abstract

Steering the behavior of Large Language Models (LLMs) remains a challenge, particularly in engineering applications where precision and reliability are critical. While fine-tuning and prompting methods can modify model behavior, they lack the dynamic and exact control necessary for engineering applications. Inference-time intervention techniques provide a promising alternative, allowing targeted adjustments to LLM outputs. In this work, we demonstrate how interventions enable fine-grained control for automating the usually time-intensive requirement verification process in Model-Based Systems Engineering (MBSE). Using two early-stage Capella SysML models of space missions with associated requirements, we apply the intervened LLMs to reason over a graph representation of the model to determine whether a requirement is fulfilled. Our method achieves robust and reliable outputs, significantly improving over both a baseline model and a fine-tuning approach. By identifying and modifying as few as one to three specialised attention heads, we can significantly change the model's behavior. When combined with self-consistency, this allows us to achieve perfect precision on our holdout test set.

Paper Structure

This paper contains 22 sections, 9 equations, 16 figures, 6 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. NLP for Requirement Engineering
  4. Inference-Time Intervention for LLMs
  5. Methodology
  6. From System Model to Requirement Verfication
  7. LLM Prompt
  8. Intervention strategy
  9. Divide-and-Conquer Intervention Optimization
  10. Experimental Setup
  11. Experiments
  12. Validation Set
  13. Intervention Optimization
  14. Effect of Intervention Strength and Temperature on Precision and Recall Robustness
  15. KTO Fine-Tuning Baseline
  16. ...and 7 more sections

Figures (16)

  • Figure 1: Intervention on Requirement Verfication
  • Figure 2: Overview prompt construction
  • Figure 3: Sensitivity of precision to layers and groups of attention heads intervention.
  • Figure 4: Sensitivity of recall to layers and groups of attention heads intervention.
  • Figure 5: Tree plot of dividing and investigating 14th layer and groups of attention heads for sensitivity to intervention.
  • ...and 11 more figures