Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Policy Gradient Methods for Information-Theoretic Opacity in Markov Decision Processes

Chongyang Shi, Sumukha Udupa, Michael R. Dorothy, Shuo Han, Jie Fu

TL;DR

The paper defines information-theoretic opacity for MDPs by maximizing the conditional entropy of secrets given an observer's partial observations, under task-performance constraints. It proves finite-memory policies can surpass Markov policies for opacity and develops a primal–dual gradient framework to compute opacity-maximizing Markov policies using observable-operator-based gradients for $H(Z_T|Y)$. The method includes a convergence guarantee and extends to language-based opacity via a product MDP that couples the system with a DFA for the secret, with experimental validation in grid-world and graph settings showing near-optimal opacity under reward constraints and improved secrecy metrics over baselines. The work advances quantitative privacy in stochastic control by enabling optimizer-guided policy design that leverages observation noise to increase adversarial uncertainty about secrets.

Abstract

Opacity, or non-interference, is a property ensuring that an external observer cannot infer confidential information (the "secret") from system observations. We introduce an information-theoretic measure of opacity, which quantifies information leakage using the conditional entropy of the secret given the observer's partial observations in a system modeled as a Markov decision process (MDP). Our objective is to find a control policy that maximizes opacity while satisfying task performance constraints, assuming that an informed observer is aware of the control policy and system dynamics. Specifically, we consider a class of opacity called state-based opacity, where the secret is a propositional formula about the past or current state of the system, and a special case of state-based opacity called language-based opacity, where the secret is defined by a temporal logic formula (LTL) or a regular language recognized by a finite-state automaton. First, we prove that finite-memory policies can outperform Markov policies in optimizing information-theoretic opacity. Second, we develop an algorithm to compute a maximally opaque Markov policy using a primal-dual gradient-based algorithm, and prove its convergence. Since opacity cannot be expressed as a cumulative cost, we develop a novel method to compute the gradient of conditional entropy with respect to policy parameters using observable operators in hidden Markov models. The experimental results validate the effectiveness and optimality of our proposed methods.

Policy Gradient Methods for Information-Theoretic Opacity in Markov Decision Processes

TL;DR

The paper defines information-theoretic opacity for MDPs by maximizing the conditional entropy of secrets given an observer's partial observations, under task-performance constraints. It proves finite-memory policies can surpass Markov policies for opacity and develops a primal–dual gradient framework to compute opacity-maximizing Markov policies using observable-operator-based gradients for . The method includes a convergence guarantee and extends to language-based opacity via a product MDP that couples the system with a DFA for the secret, with experimental validation in grid-world and graph settings showing near-optimal opacity under reward constraints and improved secrecy metrics over baselines. The work advances quantitative privacy in stochastic control by enabling optimizer-guided policy design that leverages observation noise to increase adversarial uncertainty about secrets.

Abstract

Opacity, or non-interference, is a property ensuring that an external observer cannot infer confidential information (the "secret") from system observations. We introduce an information-theoretic measure of opacity, which quantifies information leakage using the conditional entropy of the secret given the observer's partial observations in a system modeled as a Markov decision process (MDP). Our objective is to find a control policy that maximizes opacity while satisfying task performance constraints, assuming that an informed observer is aware of the control policy and system dynamics. Specifically, we consider a class of opacity called state-based opacity, where the secret is a propositional formula about the past or current state of the system, and a special case of state-based opacity called language-based opacity, where the secret is defined by a temporal logic formula (LTL) or a regular language recognized by a finite-state automaton. First, we prove that finite-memory policies can outperform Markov policies in optimizing information-theoretic opacity. Second, we develop an algorithm to compute a maximally opaque Markov policy using a primal-dual gradient-based algorithm, and prove its convergence. Since opacity cannot be expressed as a cumulative cost, we develop a novel method to compute the gradient of conditional entropy with respect to policy parameters using observable operators in hidden Markov models. The experimental results validate the effectiveness and optimality of our proposed methods.

Paper Structure

This paper contains 27 sections, 11 theorems, 86 equations, 9 figures, 1 table.

Table of Contents

  1. INTRODUCTION
  2. Preliminary and Problem Formulation
  3. Preliminaries
  4. Problem Statement
  5. Synthesizing Constrained Opacity-Enforcement Controllers
  6. Finite-memory policy
  7. Markov Policy
  8. Finite-Memory Policy
  9. Primal-Dual Policy Gradient for Constrained Minimal Information Leakage
  10. Computing the Gradient and Hessian of Conditional Entropy
  11. The case of last-state opacity
  12. Observable Operators
  13. The case with initial-state opacity
  14. Convergence Guarantee of the Primal-Dual Policy Gradient Method
  15. Synthesizing Maximally Opacity-Enforcement Controllers For Language-Based Opacity
  16. ...and 12 more sections

Key Result

Proposition 1

The probability of an observation sequence $o_{[0:t]}$ can be written as In addition, for a fixed state $s_{T + 1} \in \mathcal{S}$ at time point $T + 1$, we have where $\mathbf{1}_{s_{T+1}}$ is a one-hot vector which assigns 1 to the $s_{T+1}$-th entry.

Figures (9)

  • Figure 1: We consider an MDP defined on a graph where the agent has two actions, $a$ and $b$. At the initial state $q_0$, if the agent takes action $a$, it remains in $q_0$. If the agent takes action $b$, it moves with equal probability $0.5$ to either state $q_1$ or state $q_2$. Both $q_1$ and $q_2$ are sink states, meaning that once the agent reaches one of these states, it remains there indefinitely.
  • Figure 2: Compare the Markov policy and finite-memory policy.
  • Figure 3: The blue robot is P1 (the agent). P1 can move in four compass directions (north, south, east, west) or remain stationary. However, the dynamics of movement are stochastic. When the robot moves in a specific direction, there is a 0.1 probability that it will also move in the nearest two directions. For instance, if the robot moves east, there is a 0.1 probability of it moving north and a 0.1 probability of it moving south, as illustrated in the image. If the robot hits the boundary or walls, it stays put. And it becomes immobilized if it enters dangerous cells.
  • Figure 4: The result of the primal-dual policy gradient method for enforcing optimal last-state opacity under the constraint on the total return.
  • Figure 5: The result of the primal-dual policy gradient method for enforcing optimal initial-state opacity under the constraint on the total return.
  • ...and 4 more figures

Theorems & Definitions (27)

  • Definition 1: Observation function of P2
  • Definition 2
  • Example 1
  • Definition 3
  • Proposition 1: jaeger2000observableoperator, udupa2025
  • Proposition 2
  • proof
  • Proposition 3
  • proof
  • Lemma 1
  • ...and 17 more