Learning Collective Dynamics of Multi-Agent Systems using Event-based Vision

TL;DR

This paper empirically demonstrate the effectiveness of event-based representation over traditional frame-based methods in predicting these collective behaviors in multi-agent systems, and presents event-based vision for Multi-Agent dynamic Prediction (evMAP), a deep learning architecture designed for real-time, accurate understanding of interaction strength and collective behavior emergence in multi-agent systems.

Abstract

This paper proposes a novel problem: vision-based perception to learn and predict the collective dynamics of multi-agent systems, specifically focusing on interaction strength and convergence time. Multi-agent systems are defined as collections of more than ten interacting agents that exhibit complex group behaviors. Unlike prior studies that assume knowledge of agent positions, we focus on deep learning models to directly predict collective dynamics from visual data, captured as frames or events. Due to the lack of relevant datasets, we create a simulated dataset using a state-of-the-art flocking simulator, coupled with a vision-to-event conversion framework. We empirically demonstrate the effectiveness of event-based representation over traditional frame-based methods in predicting these collective behaviors. Based on our analysis, we present event-based vision for Multi-Agent dynamic Prediction (evMAP), a deep learning architecture designed for real-time, accurate understanding of interaction strength and collective behavior emergence in multi-agent systems.

Figures (17)

• Figure 1: Application examples of collective dynamic prediction of multi-agent system. Multi-agent dynamic prediction is helpful for both systems that are under and beyond control.
• Figure 2: Several methods for understanding dynamics in a multi-agent system. (a) Many previous studies in multi-agent prediction require pre-processing for detecting agents. This paper focuses on scene-based perception: compared to (b-1) frame-based methods, (b-2) event-based methods demonstrate their effectiveness in understanding multi-agent dynamics.
• Figure 3: Sample flocking scenes yt_schoolyt_flock and simulators (NetLogo netlogoflocking, AgentPy foramitti2021agentpy).
• Figure 4: Simulation framework of frame- and event-based vision for multi-agent dynamic prediction. Due to the absence of existing dataset, flocking simulations netlogoflockingforamitti2021agentpy and event synthesis toolbox hu2021v2e are used to generate multi-agent dynamic sequence and convert from frame to event.
• Figure 5: (left) Strong interactions cause large changes in agents' velocities and (right) weak interactions relatively maintain individual agents' velocities.