Who's asking? User personas and the mechanics of latent misalignment

TL;DR

This work investigates latent misalignment in safety-tuned LLMs by examining how user persona and activation steering influence model refusals and content disclosure. Using AdvBench-derived prompts, SneakyAdvBench rewrites, early decoding, and Patchscopes on Llama 2 13B chat, the authors show that misaligned content can reside in early layers and be surfaced despite safe final outputs. They demonstrate that manipulating the model’s inferred user attributes (user persona) via prompting and activation steering can more effectively bypass safety filters than direct prompting, with layer- and persona-dependent effects. The findings highlight layerwise, attack-specific safeguards and show that steering vector geometry can predict downstream refusal, offering insights for developing more robust defenses against latent misalignment and persona-based jailbreaking in real-world deployments.

Abstract

Despite investments in improving model safety, studies show that misaligned capabilities remain latent in safety-tuned models. In this work, we shed light on the mechanics of this phenomenon. First, we show that even when model generations are safe, harmful content can persist in hidden representations and can be extracted by decoding from earlier layers. Then, we show that whether the model divulges such content depends significantly on its perception of who it is talking to, which we refer to as user persona. In fact, we find manipulating user persona to be even more effective for eliciting harmful content than direct attempts to control model refusal. We study both natural language prompting and activation steering as control methods and show that activation steering is significantly more effective at bypassing safety filters. We investigate why certain personas break model safeguards and find that they enable the model to form more charitable interpretations of otherwise dangerous queries. Finally, we show we can predict a persona's effect on refusal given only the geometry of its steering vector.

Figures (25)

• Figure 1: Layerwise effects of applying persona steering vectors (via CAA, contrastive activation addition) with either a positive (CAA+) or negative (CAA-) multiplier. Y-axis indicates the percentage of attacks to which the model provided a response. [Left] Inducing the model to believe the user has pro-social attributes (curious, altruistic, power-avoidant, and law-abiding personas) makes it more likely to divulge sensitive information. [Right] Results for anti-social (close-minded, selfish, power-seeking, and unlawful) personas indicate the reverse is also true and to stronger effect (e.g. applying the negation of a vector that induces the model to believe the user is selfish results in a response rate of 52%). Layer 13 tends to be where CAAs are most effective (and the divergence between CAA+ and CAA- is strongest), perhaps because by layer 13 input processing is mostly complete, but the model has not fully turned to next token prediction.
• Figure 2: Y-axis indicates the percent difference in response rate to adversarial attacks compared to Baseline Prompting (0.39) for a selection of personas across treatments: (1) PP (prompted prefixes), (2) CAA+ (steering vector applied at layer 13), and (3) CAA- (steering vector applied with a negative multiplier). We also indicate the difference in response rate for early decoding at layer 13 (ED13).
• Figure 3: Heatmap with personas and treatments along the x-axis, and different attack categories along the y-axis. Color indicates the response rate (green: 0% response rate to grey: 30% response rate as baselines to dark blue: 100% response rate.) We observe a stark contrast between non-adversarial and adversarial queries when applying different interventions. Specifically, steering with CAA+/CAA- selectively affects responsiveness to adversarial queries, while prompt prefixes tend to induce refusals across the board.
• Figure 4: Cosine similarity between refusal and fulfillment steering vector s across layers. Similarity is highest at first, decreases up to layer 15, then increases again until stabilizing around layer 27.
• Figure 5: Pairwise cosine similarity between persona vectors across layers. [Top] All rows (columns) represent pro-social personas paired such that consecutive rows (columns) represent semantically similar personas, but one vector is trained to predict 'Yes' and the other 'No'. Vectors predicting 'Yes' have higher cosine similarity than vectors predicting 'No', regardless of their semantic content, forming a checkerboard pattern. This effect is present in early layers (5), exaggerates by mid layers (13), and slightly decreases in the later layers. [Bottom] Top rows (right columns) contain anti-social personas leading to an increased refusal rate, and the bottom rows (left columns) contain pro-social personas. All vectors are trained to predict 'Yes'. No separation is visible in early layers. Separation emerges in mid layers (13) and by later layers (27) two distinct clusters are visible.