NeuroFilter: Privacy Guardrails for Conversational LLM Agents

TL;DR

This work tackles inference-time privacy in agentic LLMs by reframing privacy as context-dependent information flows under contextual integrity. It introduces NeuroFilter, a lightweight guardrail that uses activation-space linear probes to detect privacy-violating intent and an activation-velocity signal to catch multi-turn adversarial drift, enabling early interception before leakage. Across 150k+ interactions and multiple model families, NeuroFilter achieves near-perfect safety on benign prompts with zero false positives and substantially lowers latency and compute compared with LLM-based firewalls and SAE-based methods. The results demonstrate strong generalization across architectures, robustness to quantization, and a practical path toward deployable, context-aware privacy protections in conversational AI.

Abstract

This work addresses the computational challenge of enforcing privacy for agentic Large Language Models (LLMs), where privacy is governed by the contextual integrity framework. Indeed, existing defenses rely on LLM-mediated checking stages that add substantial latency and cost, and that can be undermined in multi-turn interactions through manipulation or benign-looking conversational scaffolding. Contrasting this background, this paper makes a key observation: internal representations associated with privacy-violating intent can be separated from benign requests using linear structure. Using this insight, the paper proposes NeuroFilter, a guardrail framework that operationalizes contextual integrity by mapping norm violations to simple directions in the model's activation space, enabling detection even when semantic filters are bypassed. The proposed filter is also extended to capture threats arising during long conversations using the concept of activation velocity, which measures cumulative drift in internal representations across turns. A comprehensive evaluation across over 150,000 interactions and covering models from 7B to 70B parameters, illustrates the strong performance of NeuroFilter in detecting privacy attacks while maintaining zero false positives on benign prompts, all while reducing the computational inference cost by several orders of magnitude when compared to LLM-based agentic privacy defenses.

Figures (23)

• Figure 1: Overview of the threat landscape. User inputs (left) pass through the LLM agent (center) which provides a response (right). Four scenarios are considered: (0) Benign Authorized Use, where disclosure of a potentially sensitive attribute is contextually appropriate (e.g., patient details to an authorized doctor), (1) Direct Jailbreaks, where single-turn malicious prompts attempt to bypass safety filters to induce leakage, (2) Conversational Manipulation, involving gradual multi-turn steering via long horizon planning and probing questions to reveal sensitive information, and (3) Mosaic Attacks, where malicious queries targeting model safety are decomposed into benign sub-queries that reconstruct normally disallowed responses when aggregated. NeuroFilter monitors activation velocity (center), i.e. the cumulative drift of internal representations toward a violation, enabling the detection of adversarial intent in interactions with conversational agents.
• Figure 2: Overview of the NeuroFilter framework: probe training process (left) and inference-time deployment (right).
• Figure 3: CMPL Insurance (Qwen 2.5 7B Instruct): Comparing probing accuracies for SAEs vs. linear probes.
• Figure 4: CMPL Insurance (Single-Turn): Test accuracies (left) and projection scores (right) for GPT OSS 20B.
• Figure 5: Cosine similarities between privacy violation directions for CMPL Insurance and CMPL Scheduling across all layers of Qwen 2.5 32B Instruct.