Grounded Answers for Multi-agent Decision-making Problem through Generative World Model

TL;DR

This work introduces Learning before Interaction (LBI), a paradigm that grounds answers to multi-agent decision problems by training a language-guided world model and using it to simulate trial-and-error interactions. The interactive simulator comprises a VQ-VAE image tokenizer, a causal transformer dynamics model, and a bidirectional transformer reward model, all trained with language-guided supervision on VisionSMAC data. By running a converged MARL policy within the learned world model and relabeling trajectories with the reward model, LBI generates ground-truth-like image sequences as answers and demonstrates improved performance on SMAC benchmarks, including unseen tasks, along with interpretable rewards. The approach advances grounded reasoning for complex multi-agent tasks and suggests a path toward more reliable, explainable generative models in decision-making domains, albeit with noted limitations in simulator fidelity and computation time.

Abstract

Recent progress in generative models has stimulated significant innovations in many fields, such as image generation and chatbots. Despite their success, these models often produce sketchy and misleading solutions for complex multi-agent decision-making problems because they miss the trial-and-error experience and reasoning as humans. To address this limitation, we explore a paradigm that integrates a language-guided simulator into the multi-agent reinforcement learning pipeline to enhance the generated answer. The simulator is a world model that separately learns dynamics and reward, where the dynamics model comprises an image tokenizer as well as a causal transformer to generate interaction transitions autoregressively, and the reward model is a bidirectional transformer learned by maximizing the likelihood of trajectories in the expert demonstrations under language guidance. Given an image of the current state and the task description, we use the world model to train the joint policy and produce the image sequence as the answer by running the converged policy on the dynamics model. The empirical results demonstrate that this framework can improve the answers for multi-agent decision-making problems by showing superior performance on the training and unseen tasks of the StarCraft Multi-Agent Challenge benchmark. In particular, it can generate consistent interaction sequences and explainable reward functions at interaction states, opening the path for training generative models of the future.

Figures (8)

• Figure 1: Complex decision problems that require a good understanding of the environment's dynamics and the objective are still challenging for current vision-language models, e.g., the answer elicited by GPT-4 is sketchy and misleading. Instead, Learning before Interaction (LBI) enables grounded reasoning by simulating the task in the given question. LBI utilizes the simulator to train a MARL policy and generate the answer by running the converged policy on the simulator.
• Figure 2: Datasets construction and VQ-VAE training.
• Figure 3: The overview of Learning before Interaction.
• Figure 4: Visualization of the prediction from dynamics and reward model, where "np-op" and "s" denote no-operation and stopping, respectively.
• Figure 5: Images of units and terrains.