LAMP: Extracting Locally Linear Decision Surfaces from LLM World Models

TL;DR

LAMP addresses whether an LLM's self-reported reasons align with its decisions by constructing a local linear surrogate that maps explanation weights to predicted probabilities, using perturbations of those weights without access to gradients. The method aggregates a compact set of explanation factors and fits a ridge-regularized surrogate to capture local decision surfaces, validating the approach through multiple datasets and counterfactual perturbations. Key findings show that LAMP surrogates can predict LLM outputs with high fidelity, correlate with human judgments of explanation quality, and offer practical auditing insights, even for proprietary models. The work advances interpretable auditing by providing a gradient-free, transportable framework that can inform trust and governance in high-stakes NLP applications, with plans to extend to higher-order and interactive settings.

Abstract

We introduce LAMP (Linear Attribution Mapping Probe), a method that shines light onto a black-box language model's decision surface and studies how reliably a model maps its stated reasons to its predictions through a locally linear model approximating the decision surface. LAMP treats the model's own self-reported explanations as a coordinate system and fits a locally linear surrogate that links those weights to the model's output. By doing so, it reveals which stated factors steer the model's decisions, and by how much. We apply LAMP to three tasks: sentiment analysis, controversial-topic detection, and safety-prompt auditing. Across these tasks, LAMP reveals that many LLMs exhibit locally linear decision landscapes. In addition, these surfaces correlate with human judgments on explanation quality and, on a clinical case-file data set, aligns with expert assessments. Since LAMP operates without requiring access to model gradients, logits, or internal activations, it serves as a practical and lightweight framework for auditing proprietary language models, and enabling assessment of whether a model behaves consistently with the explanations it provides.

Figures (8)

• Figure 1: LAMP produces features that can explain the LLM decision surface as a linear combination. The decision surface is estimated by sampling perturbations around the input and regressing the resulting probabilities reported by the language model. The decision surface can then be expressed as a linear combination of LAMP-generated factors, providing an interpretable summary of the model's local decision behavior.
• Figure 2: LAMP surrogate coefficients visualized for a representative patient case. LAMP outputs a diagnosis along with a rationale. The surrogate model provides a local approximation of the decision surface, with coefficient magnitudes indicating the direction and relative influence of each factor on the LLM's reported probability of diagnosis.
• Figure 3: Example of LAMP factors-based modifications. LAMP then extracts the factors and weights from this text. Green highlights are rewritten to emphasize a more positive shift, while red highlights suggest a more negative shift.
• Figure 4: Across the board, LAMP surrogate linear models are able to predict LLM output to a small margin of error. GPT, Gemini, and Mistral are able to predict the LLM given probability just by using its locally linear surrogate (lower is better). This is better than using an intercept only surrogate (mean model) and a naive baseline (predicting 0.5 every time)
• Figure 5: Rewriting text based on LAMP factors produces localized perturbations in the explanation space. The first two principle components of the explanation space for IMDB text are plotted. Blue points represent the original text factor weightings, and orange points represent rewritten text factor weightings.