Active Inference through Incentive Design in Markov Decision Processes

TL;DR

The paper tackles inferring a follower's type in a leader–follower setting under partial observability by designing incentives that reveal distinct behavioral patterns. It formulates a bi-level optimization where the leader chooses side payments to balance information gain against incentive costs, and followers respond optimally. By employing entropy-regularized MDPs, the authors reduce the problem to a single-level gradient-based optimization, leveraging observable operators within an HMM to compute required gradients. Empirical results in stochastic grid-world environments demonstrate that the approach reduces uncertainty about follower types while controlling incentive expenditures, validating the method's practicality for active inference tasks.

Abstract

We present a method for active inference with partial observations in stochastic systems through incentive design, also known as the leader-follower game. Consider a leader agent who aims to infer a follower agent's type given a finite set of possible types. Different types of followers differ in either the dynamical model, the reward function, or both. We assume the leader can partially observe a follower's behavior in the stochastic system modeled as a Markov decision process, in which the follower takes an optimal policy to maximize a total reward. To improve inference accuracy and efficiency, the leader can offer side payments (incentives) to the followers such that different types of them, under the incentive design, can exhibit diverging behaviors that facilitate the leader's inference task. We show the problem of active inference through incentive design can be formulated as a special class of leader-follower games, where the leader's objective is to balance the information gain and cost of incentive design. The information gain is measured by the entropy of the estimated follower's type given partial observations. Furthermore, we demonstrate that this problem can be solved by reducing a single-level optimization through softmax temporal consistency between followers' policies and value functions. This reduction allows us to develop an efficient gradient-based algorithm. We utilize observable operators in the hidden Markov model (HMM) to compute the necessary gradients and demonstrate the effectiveness of our approach through experiments in stochastic grid world environments.

Figures (3)

• Figure 1: Fire rescue task in grid world environment.
• Figure 2: Behavior comparison task in grid world environment.
• Figure 3: The results of experiments.

Theorems & Definitions (10)

• Remark 1
• Definition 1
• Definition 2
• Definition 3
• Proposition 1
• proof
• Proposition 2
• Remark 2
• Example 1: Fire rescue task
• Example 2: Behavior comparison task