Evaluating and Comparing Crowd Simulations: Perspectives from a Crowd Authoring Tool

TL;DR

The paper extends WebCrowds to enable end-to-end evaluation and direct comparison of multiple crowd configurations in evacuation scenarios. It introduces a new metric $\\phi$ that combines normalized time, density, speed, and traveled distance via a harmonic-mean formulation, with normalization anchored to a reference agent; a multi-configuration workflow enables parallel simulations and objective ranking of configurations. Through five scenario tests and a small expert panel, the authors show that $\\phi$ aligns with expert judgments in most cases and provides a practical alternative to the Cassol 2017 metric for selecting optimal evacuation configurations. The work demonstrates that integrating authoring, simulation, and quantitative comparison in a single tool can support safer, more effective evacuation planning and environment design in real-world settings.

Abstract

Crowd simulation is a research area widely used in diverse fields, including gaming and security, assessing virtual agent movements through metrics like time to reach their goals, speed, trajectories, and densities. This is relevant for security applications, for instance, as different crowd configurations can determine the time people spend in environments trying to evacuate them. In this work, we extend WebCrowds, an authoring tool for crowd simulation, to allow users to build scenarios and evaluate them through a set of metrics. The aim is to provide a quantitative metric that can, based on simulation data, select the best crowd configuration in a certain environment. We conduct experiments to validate our proposed metric in multiple crowd simulation scenarios and perform a comparison with another metric found in the literature. The results show that experts in the domain of crowd scenarios agree with our proposed quantitative metric.

Figures (4)

• Figure 1: The pipeline of WebCrowds. Users can use the Editor through a web browser to create simulation scenarios. Requests for a given simulation are sent to the Server, responsible for executing the simulation, generating results, and sending them back to be shown in the Browser. Elements highlighted in purple are new features in our proposed model, which include the possibility of creating multiple environments to be evaluated and compared through metrics.
• Figure 2: Example of a crowd simulation scenario with three different configurations. Blue squares represent the agents' spawn areas, the green circle at the bottom represents the agents' goal, and obstacles are represented in red. In this case, the configurations met the comparison criteria, as the total number of agents (although in different positions), the number of goals, and the environment geometries are the same.
• Figure 3: Simulations Scenarios 1 and 2. The first line presents the different crowd configurations, with the initial position of agents (blue) and goals (green). These scenarios do not contain obstacles or walls. The second line presents the occupancy maps of each crowd configuration. The third line presents the plotted trajectories of virtual agents in each crowd configuration.
• Figure 4: Simulations Scenarios 3, 4, and 5. The first line presents the different crowd configurations, with the initial position of agents (blue), the goals (green), the obstacles (red), and the walls (dark grey). The second line presents the occupancy maps of each crowd configuration. The third line presents the plotted trajectories of virtual agents in each crowd configuration.