Variational Auto-encoder Based Solutions to Interactive Dynamic Influence Diagrams

TL;DR

The paper tackles multi-agent sequential decision making under partial observability by augmenting Interactive Dynamic Influence Diagrams (I-DIDs) with a data-driven framework based on variational autoencoders (VAEs). It introduces Zig-Zag One-Hot encoding and a tree-weighted loss to train VAEs that convert incomplete policy trees into complete, diverse representations of other agents’ behaviors, enabling better planning under unknown opponent models. A perplexity-based metric, along with diversity (MDF) and reliability (ICD) measures, guides top-K selection to balance coverage of true behaviors with decision confidence. Empirical results in tiger and UAV domains show that VAE-enabled I-DID variants often outperform classical I-DID and other baselines, demonstrating improved robustness to unknown behaviors and scalable application to complex multi-agent settings.

Abstract

Addressing multiagent decision problems in AI, especially those involving collaborative or competitive agents acting concurrently in a partially observable and stochastic environment, remains a formidable challenge. While Interactive Dynamic Influence Diagrams~(I-DIDs) have offered a promising decision framework for such problems, they encounter limitations when the subject agent encounters unknown behaviors exhibited by other agents that are not explicitly modeled within the I-DID. This can lead to sub-optimal responses from the subject agent. In this paper, we propose a novel data-driven approach that utilizes an encoder-decoder architecture, particularly a variational autoencoder, to enhance I-DID solutions. By integrating a perplexity-based tree loss function into the optimization algorithm of the variational autoencoder, coupled with the advantages of Zig-Zag One-Hot encoding and decoding, we generate potential behaviors of other agents within the I-DID that are more likely to contain their true behaviors, even from limited interactions. This new approach enables the subject agent to respond more appropriately to unknown behaviors, thus improving its decision quality. We empirically demonstrate the effectiveness of the proposed approach in two well-established problem domains, highlighting its potential for handling multi-agent decision problems with unknown behaviors. This work is the first time of using neural networks based approaches to deal with the I-DID challenge in agent planning and learning problems.

Figures (15)

• Figure 1: A dynamic influence diagram and its solutions: ($a$)the left is the dynamic influence diagram with three time steps and ($b$) the right is its solution represented as a policy tree. The blocks with the same color in ($a$) and ($b$) belong to the same time slice.
• Figure 2: By extending DID model (the blue part), the agent $i$ optimises its decision in the I-DID models with the blue part which models agent $j$'s decision making process.
• Figure 3: Reconstructing $m$ incomplete policy trees via four operators ($split$,$union$,$roulette$ and $graphying$) from behavior sequences.
• Figure 4: The principle of a VAE-based approach that leverages incomplete policy trees to generate diverse new trees, maximizing coverage of agent $j$'s true behaviors and enhancing the diversity of the overall policy tree collection.
• Figure 5: The VAE network creates new policy trees. With the Zig-Zag One-Hot encoding-decoding method designed for policy trees, VAE can handle both complete and incomplete trees, emphasizing learning from earlier nodes. By using MDF and IDF, we pick the most diverse and reliable top-$K$ trees, matching historical agent patterns closely.