Neural Transparency: Mechanistic Interpretability Interfaces for Anticipating Model Behaviors for Personalized AI

TL;DR

This work addresses the opacity of user-designed personalized AI chatbots by introducing neural transparency, an interface that exposes mechanistic interpretability insights to non-technical users. It derives 16 trait directions (e.g., empathy, toxicity, sycophancy) from contrastive prompts and projects final-token activations onto these trait vectors to predict behavior, visualizing results with a sunburst diagram. An end-to-end pipeline (persona vectors, normalized persona scores) and a user study show systematic miscalibration in users’ mental models of trait activations, while the transparency tool increases trust and perceived usefulness, though it does not yield measurable changes in design iterations or AI personalities in a single session. The study demonstrates the feasibility of operationalizing mechanistic interpretability for end users and outlines future work in longitudinal deployments, adversarial tasks, and standardizing trait disclosures to support safer, more aligned human-AI interactions.

Abstract

Millions of users now design personalized LLM-based chatbots that shape their daily interactions, yet they can only roughly anticipate how their design choices will manifest as behaviors in deployment. This opacity is consequential: seemingly innocuous prompts can trigger excessive sycophancy, toxicity, or other undesirable traits, degrading utility and raising safety concerns. To address this issue, we introduce an interface that enables neural transparency by exposing language model internals during chatbot design. Our approach extracts behavioral trait vectors (empathy, toxicity, sycophancy, etc.) by computing differences in neural activations between contrastive system prompts that elicit opposing behaviors. We predict chatbot behaviors by projecting the system prompt's final token activations onto these trait vectors, normalizing for cross-trait comparability, and visualizing results via an interactive sunburst diagram. To evaluate this approach, we conducted an online user study using Prolific to compare our neural transparency interface against a baseline chatbot interface without any form of transparency. Our analyses suggest that users systematically miscalibrated AI behavior: participants misjudged trait activations for eleven of fifteen analyzable traits, motivating the need for transparency tools in everyday human-AI interaction. While our interface did not change design iteration patterns, it significantly increased user trust and was enthusiastically received. Qualitative analysis revealed nuanced user experiences with the visualization, suggesting interface and interaction improvements for future work. This work offers a path for how mechanistic interpretability can be operationalized for non-technical users, establishing a foundation for safer, more aligned human-AI interactions.

Figures (7)

• Figure 1: Pipeline to generate persona vectors and application to calculate persona scores. Given a desired trait, an LLM is used to generate a contrastive pair of system prompts, which is then used to generate a contrastive pair of LLM responses. By taking the difference between the mean activations of the responses, we calculate the persona vector, which we use to calculate the persona score for a system prompt
• Figure 2: (Left) Activation heatmap illustrating how the persona vector modulates the LLM's internal representations (Layer 20). (Right) Full view of the sunburst visualization.
• Figure 3: Linear regression between trait expression level in example system prompt and persona scores (not scaled to 0-1). For each level of trait expression (1-5), five system prompts were generated. Regressions are ordered based on their $R^2$ values.
• Figure 4: Example system prompts to create different AI personalities and their associated persona scores visualized in our sunburst diagram.
• Figure 5: User flow in our web-based experiment that was hosted on Prolific. The experiment consists of nine distinct interfaces: 1) consent form, 2) avatar selection, 3) system prompting, 4-6) initial survey (pre-interaction), 7) experimental or control condition view, 8) chat interface, and 9) post-interaction survey. Participants can navigate between the system prompting view and chat interface throughout the study.