Multi-agent Path Finding for Timed Tasks using Evolutionary Games

TL;DR

This work uses weighted automata to specify trajectory-level objectives, such that, maximal paths induced in the weighted automaton correspond to desired trajectory-level behaviors, and uses evolutionary game theory principles to train homogeneous multi-agent teams targeting homogeneous task objectives.

Abstract

Autonomous multi-agent systems such as hospital robots and package delivery drones often operate in highly uncertain environments and are expected to achieve complex temporal task objectives while ensuring safety. While learning-based methods such as reinforcement learning are popular methods to train single and multi-agent autonomous systems under user-specified and state-based reward functions, applying these methods to satisfy trajectory-level task objectives is a challenging problem. Our first contribution is the use of weighted automata to specify trajectory-level objectives, such that, maximal paths induced in the weighted automaton correspond to desired trajectory-level behaviors. We show how weighted automata-based specifications go beyond timeliness properties focused on deadlines to performance properties such as expeditiousness. Our second contribution is the use of evolutionary game theory (EGT) principles to train homogeneous multi-agent teams targeting homogeneous task objectives. We show how shared experiences of agents and EGT-based policy updates allow us to outperform state-of-the-art reinforcement learning (RL) methods in minimizing path length by nearly 30\% in large spaces. We also show that our algorithm is computationally faster than deep RL methods by at least an order of magnitude. Additionally our results indicate that it scales better with an increase in the number of agents as compared to other methods.

Figures (5)

• Figure 1: Shortest paths in the grid environment: agents $a_1$ and $a_2$ must get to designated final states (black circles) while avoiding red obstacle regions. Red dashed trajectories $\tau^1_1$ of agent $a_1$ and $\tau^2_1$ of $a_2$ satisfy the task but are not the shortest paths, green solid trajectories $\tau^1_2$ and $\tau^2_2$are the shortest paths. Arrows point towards goals.
• Figure 2: Counter-example: An agent (O) tries to go from the initial location (yellow) to the goal location (green). Under general expeditious semantics (such as in AvSTL) the trajectory in the left figure where the agent reaches the goal quickly but wanders after would receive a lower reward (area under the curve) than the one in the right figure where the agent takes a longer path to reach the goal but stays in the goal for longer.
• Figure 3: A deterministic weighted automaton defining the reach-avoid task that needs to be completed by each agent in the system. In the figure, $s^i$ refers to the current state of agent $i$ in the multi-agent system.
• Figure 4: Overview of the evolutionary based learning approach used in our algorithms.
• Figure 5: MAPF-EGT benchmarked against the algorithms: $A^*$, Monte-Carlo search, PPO, and Q learning. Timesteps to reach the goaal (Fig. a), Expected minimum distance from obstacles (greater distance indicates safer paths)(Fig. b), and clock time (seconds) required for computation (Fig. c) compared across the grid sizes: 20$\times$20, 50$\times$50, 100$\times$100, 150$\times$150 and 200$\times$200. Fig.d shows scaling in total time taken, with number of agents varied from 2 to 50 on a 100 $\times$100 grid. We also note that the $A^*$ algorithm is given a heuristic i.e. the Manhattan distance to the closest goal, information that the other algorithms are not given.

Theorems & Definitions (7)

• definition 1: Stochastic Game
• definition 2: Weighted Automata
• definition 3: Valuation function
• remark 1
• proposition 1
• lemma 1
• proof