Investigating task-specific prompts and sparse autoencoders for activation monitoring

TL;DR

The paper evaluates three families of activation-monitoring methods for language models: zero-shot prompting, prompted probing of activations, and SAE-based probing (with max-pooling), plus combinations thereof. It systematically contrasts these approaches across moderation, QA, and sentiment tasks under varying data availability and compute constraints, using activations from a GPT-4o-like model. Key findings show prompted probing offers strong data efficiency and generalization, especially when inference-time compute is available, while SAE-based probing shines under limited training data or train-time compute; zero-shot prompting remains a competitive baseline. The work provides practical guidance on method selection by task and deployment constraints, and discusses limitations and safety considerations for monitoring in real-world systems.

Abstract

Language models can behave in unexpected and unsafe ways, and so it is valuable to monitor their outputs. Internal activations of language models encode additional information that could be useful for this. The baseline approach for activation monitoring is some variation of linear probing on a particular layer: starting from a labeled dataset, train a logistic regression classifier on that layer's activations. Recent work has proposed several approaches which may improve on naive linear probing, by leveraging additional computation. One class of techniques, which we call "prompted probing," leverages test time computation to improve monitoring by (1) prompting the model with a description of the monitoring task, and (2) applying a learned linear probe to resulting activations. Another class of techniques uses computation at train time: training sparse autoencoders offline to identify an interpretable basis for the activations, and e.g. max-pooling activations across tokens using that basis before applying a linear probe. However, one can also prompt the model with a description of the monitoring task and use its output directly. We develop and test novel refinements of these methods and compare them against each other. We find asking the model zero-shot is a reasonable baseline when inference-time compute is not limited; however, activation probing methods can substantially outperform this baseline given sufficient training data. Specifically, we recommend prompted probing when inference-time compute is available, due to its superior data efficiency and good generalization performance. Alternatively, if inference-time compute is limited, we find SAE-based probing methods outperform raw activation probing.

Figures (13)

• Figure 1: An illustration of our main monitoring methods on an example input. We extract residual stream activations on a model at either the last token, or pooled across all tokens after computing hidden-state activations of a sparse autoencoder.
• Figure 2: Prompted probing (orange) and SAE max-pooled probing (blue) help with classification performance across a diverse set of tasks. Prompted probing tends to help more in the low-data regime.
• Figure 3: When trained on English data and tested on English, prompted probes generalize relatively well (in the sense of strong OOD performance at a given in-distribution performance), while SAE max-pooled probes generalize poorly compared with naive last token activation probes. All methods perform more similarly when trained on single prompt data and tested on multi-turn conversation data.
• Figure 4: Last token probes (blue and green) perform best at sentiment classification. SAE pre-activation last token probing is competitive with naive last token probing.
• Figure 5: Prompted probing does not substantially stack with SAE-based probing.