MAGPIE: A benchmark for Multi-AGent contextual PrIvacy Evaluation

TL;DR

MAGPIE introduces a multi-agent, privacy-centric benchmark with 200 high-stakes, multi-turn negotiation tasks where private information is necessary for task resolution. The framework combines a data-generation pipeline (seed scenarios expanded by LLMs with formalized utilities, penalties, and privacy constraints), a simulated negotiation environment, and LLM-based evaluators to quantify privacy leakage, task completion, and undesired behaviors. Findings show substantial privacy leakage across leading LLMs, limited consensus on complex tasks, and prevalent manipulation or power-seeking in negotiations, revealing a critical privacy-efficacy alignment gap in current agents. These results highlight the need for privacy-aware training, advanced information-flow controls, and multi-turn evaluation to safely deploy collaborative AI agents in high-stakes environments.

Abstract

A core challenge for autonomous LLM agents in collaborative settings is balancing robust privacy understanding and preservation alongside task efficacy. Existing privacy benchmarks only focus on simplistic, single-turn interactions where private information can be trivially omitted without affecting task outcomes. In this paper, we introduce MAGPIE (Multi-AGent contextual PrIvacy Evaluation), a novel benchmark of 200 high-stakes tasks designed to evaluate privacy understanding and preservation in multi-agent collaborative, non-adversarial scenarios. MAGPIE integrates private information as essential for task resolution, forcing agents to balance effective collaboration with strategic information control. Our evaluation reveals that state-of-the-art agents, including GPT-5 and Gemini 2.5-Pro, exhibit significant privacy leakage, with Gemini 2.5-Pro leaking up to 50.7% and GPT-5 up to 35.1% of the sensitive information even when explicitly instructed not to. Moreover, these agents struggle to achieve consensus or task completion and often resort to undesirable behaviors such as manipulation and power-seeking (e.g., Gemini 2.5-Pro demonstrating manipulation in 38.2% of the cases). These findings underscore that current LLM agents lack robust privacy understanding and are not yet adequately aligned to simultaneously preserve privacy and maintain effective collaboration in complex environments.

Figures (4)

• Figure 1: A datapoint from the MAGPIE dataset, which includes the task, deliverable, and agents’ profile (many details are ommited for brevity). This example uses GPT-5 as the agent backbone. Agents can message individuals or the group and also send proposals. As seen, Ben leaks hidden urgency due to a mega project coming to Eleanor, while Eleanor and Anita provide hints to private information (red = full leakage, blue = partial leakage). Finally, since all the agents do not accept Ben's proposal, conversation continues.
• Figure 2:
• Figure 3: Comparison of privacy dynamics across rounds and task outcomes. (\ref{['fig:avg_leakage_per_round']}) shows the average leakage trend per round (circles represent implicit instruction and star represent explicit instruction), while (\ref{['fig:task_completion_vs_privacy']}) illustrates the relationship between task completion and privacy preservation.
• Figure 4: Behavioral profiles of different LLM agents under implicit and explicit instruction settings.