1-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoning

TL;DR

A multi-agent framework that decomposes privacy reasoning into specialized subtasks (extraction, classification) is introduced, reducing the information load on any single agent while enabling iterative validation and more reliable adherence to contextual privacy norms.

Abstract

Addressing contextual privacy concerns remains challenging in interactive settings where large language models (LLMs) process information from multiple sources (e.g., summarizing meetings with private and public information). We introduce a multi-agent framework that decomposes privacy reasoning into specialized subtasks (extraction, classification), reducing the information load on any single agent while enabling iterative validation and more reliable adherence to contextual privacy norms. To understand how privacy errors emerge and propagate, we conduct a systematic ablation over information-flow topologies, revealing when and why upstream detection mistakes cascade into downstream leakage. Experiments on the ConfAIde and PrivacyLens benchmark with several open-source and closed-sourced LLMs demonstrate that our best multi-agent configuration substantially reduces private information leakage (\textbf{18\%} on ConfAIde and \textbf{19\%} on PrivacyLens with GPT-4o) while preserving the fidelity of public content, outperforming single-agent baselines. These results highlight the promise of principled information-flow design in multi-agent systems for contextual privacy with LLMs.

Figures (19)

• Figure 1: Methodology Overview: A modularized multi-agent architecture illustrating information-flow variants for contextual privacy reasoning across the ConfAIde and PrivacyLens benchmarks. The framework comprises three agentic components (Extractor, Checker, and Executor), which jointly process meeting transcripts and privacy-norm inputs to produce privacy-aware outputs. The Checker regulates the visibility of private versus public information, thereby enabling systematic manipulation of information asymmetry to evaluate its impact on multi-agent coordination, reasoning transparency, and contextual privacy preservation. The resulting outputs include privacy-filtered meeting summaries (ConfAIde) and privacy-compliant action predictions.
• Figure 3: PrivacyLens benchmark results for six LLMs under single-agent and three-agent settings, showing Binary Helpfulness Rate, Average Helpfulness Score, Privacy Preservation Rate, and Adjusted Information Preservation Rate (higher is better). Privacy Ann. means privacy-annotation setting, Public Only means public-only setting.
• Figure 4: Composite quality scores (0 – 200, higher = better) for each stage of our three-agent pipeline on the PrivacyLens benchmark. The score adds privacy retention (100 - leak rate) to public-information completeness, hence larger values reflect both stronger privacy protection and fuller content delivery.
• Figure 5: Public and private event coverage rates at the Assistant stage under the public-only configuration. While privacy coverage remains stable across models, public coverage varies dramatically, revealing the Assistant’s role as a bottleneck in preserving useful content.
• Figure 6: Event recovery performance at the Checker stage. The figure shows how effectively the Checker restores missing public content and filters leaked private content after receiving the Assistant’s output.