Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Information-Theoretic Privacy Control for Sequential Multi-Agent LLM Systems

Sadia Asif, Mohammad Mohammadi Amiri

TL;DR

This work formalizes leakage using mutual information and derive a theoretical bound that characterizes how locally introduced leakage can amplify across agents under sequential execution, and proposes a privacy-regularized training framework that directly constrains information flow between agent outputs and agent-local sensitive variables.

Abstract

Sequential multi-agent large language model (LLM) systems are increasingly deployed in sensitive domains such as healthcare, finance, and enterprise decision-making, where multiple specialized agents collaboratively process a single user request. Although individual agents may satisfy local privacy constraints, sensitive information can still be inferred through sequential composition and intermediate representations. In this work, we study \emph{compositional privacy leakage} in sequential LLM agent pipelines. We formalize leakage using mutual information and derive a theoretical bound that characterizes how locally introduced leakage can amplify across agents under sequential execution. Motivated by this analysis, we propose a privacy-regularized training framework that directly constrains information flow between agent outputs and agent-local sensitive variables. We evaluate our approach across sequential agent pipelines of varying depth on three benchmark datasets, demonstrating stable optimization dynamics and consistent, interpretable privacy-utility trade-offs. Our results show that privacy in agentic LLM systems cannot be guaranteed by local constraints alone and must instead be treated as a system-level property during both training and deployment.

Information-Theoretic Privacy Control for Sequential Multi-Agent LLM Systems

TL;DR

This work formalizes leakage using mutual information and derive a theoretical bound that characterizes how locally introduced leakage can amplify across agents under sequential execution, and proposes a privacy-regularized training framework that directly constrains information flow between agent outputs and agent-local sensitive variables.

Abstract

Sequential multi-agent large language model (LLM) systems are increasingly deployed in sensitive domains such as healthcare, finance, and enterprise decision-making, where multiple specialized agents collaboratively process a single user request. Although individual agents may satisfy local privacy constraints, sensitive information can still be inferred through sequential composition and intermediate representations. In this work, we study \emph{compositional privacy leakage} in sequential LLM agent pipelines. We formalize leakage using mutual information and derive a theoretical bound that characterizes how locally introduced leakage can amplify across agents under sequential execution. Motivated by this analysis, we propose a privacy-regularized training framework that directly constrains information flow between agent outputs and agent-local sensitive variables. We evaluate our approach across sequential agent pipelines of varying depth on three benchmark datasets, demonstrating stable optimization dynamics and consistent, interpretable privacy-utility trade-offs. Our results show that privacy in agentic LLM systems cannot be guaranteed by local constraints alone and must instead be treated as a system-level property during both training and deployment.
Paper Structure (96 sections, 3 theorems, 52 equations, 7 figures, 5 tables, 1 algorithm)

This paper contains 96 sections, 3 theorems, 52 equations, 7 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Problem Setup
  5. Sequential Agent Pipeline
  6. Conditional Independence Structure.
  7. Interpretation.
  8. Threat Model
  9. Privacy Objective
  10. Theoretical Analysis for Compositional Privacy Leakage
  11. Per-Agent Leakage
  12. Sequential Composition and Leakage Amplification
  13. Local vs. Global Leakage.
  14. Interpretation.
  15. Early-Agent Dominance.
  16. ...and 81 more sections

Key Result

Theorem 4.1

Consider a sequential agent pipeline satisfying the Markov structure as given in Eq. eq:markov and assume that the sensitive variables $\{S_i\}_{i=1}^N$ are mutually independent. If each agent satisfies the local leakage constraint $I(O_i; S_i) \leq \epsilon_i$, then the global compositional leakage

Figures (7)

  • Figure 1: Distributional comparison of baseline and MINE-Reg across privacy and utility metrics for LLaMA-3B. Violin plots show per-run variability with mean (red bar) and dispersion for Sensitive Blocked, Benign Succeeded, PARI score, and Overall Success.
  • Figure 2: $\mathrm{MI}_{\text{avg}}$ leakage as a function of sequential agent depth across different models for MedQA benchmark. Unregularized systems exhibit strong leakage amplification with depth, while MI-regularized training effectively suppresses cumulative leakage across all model families.
  • Figure 3: Relationship between $\mathrm{MI}_{\text{avg}}$ and SB for Qwen-4B model on FinQA benchmark. Strong negative correlations validate the information-theoretic formulation: reducing MI directly improves privacy, with steeper gains under MI-regularized training.
  • Figure 4: Total compositional leakage under selective agent regularization.
  • Figure 5: Privacy-utility trade-off as a function of the privacy weight $\beta$, measured by validation accuracy versus total information leakage $\sum_{i=1}^{N} \hat{I}_{\text{MINE}}(O_i; S_i)$, for $N{=}3$ sequential-agent pipeline using Qwen-2B evaluated on MedQA.
  • ...and 2 more figures

Theorems & Definitions (5)

  • Theorem 4.1: Cumulative Leakage Bound
  • Proposition 4.2: Early-Agent Leakage Dominance
  • Definition 1.1: Conditional Data Processing Inequality
  • Lemma 1.2: Upstream Leakage Bound
  • proof