Reconsidering the performance of DEVS modeling and simulation environments using the DEVStone benchmark
José L. Risco-Martín, Saurabh Mittal, Juan Carlos Fabero, Marina Zapater, Román Hermida
TL;DR
DEVS is a discrete-event modeling formalism, and DEVStone serves as a synthetic benchmark for evaluating simulator performance. This work revisits DEVStone by adding precise event-count equations and introducing HOmem as a simpler, analytically tractable variant of HOmod, then benchmarks five engines (aDEVS, CD++, DEVSJAVA, xDEVS, PyPDEVS) across two hardware platforms with a focus on execution time and memory footprint. Key findings show that aDEVS delivers the smallest memory footprint, while xDEVS provides the fastest execution on more complex HO configurations; HOmem matches HOmod in workload but offers simpler mathematics. The study highlights practical implications for selecting DEVS engines under varying model complexity and motivates extending DEVStone to parallel and distributed simulations.
Abstract
The Discrete Event System Specification formalism (DEVS), which supports hierarchical and modular model composition, has been widely used to understand, analyze and develop a variety of systems. DEVS has been implemented in various languages and platforms over the years. The DEVStone benchmark was conceived to generate a set of models with varied structure and behavior, and to automate the evaluation of the performance of DEVS-based simulators. However, DEVStone is still in a preliminar phase and more model analysis is required. In this paper, we revisit DEVStone introducing new equations to compute the number of events triggered. We also introduce a new benchmark, called HOmem, designed as an alternative version of HOmod, with similar CPU and memory requirements, but with an easier implementation and analytically more manageable. Finally, we compare both the performance and memory footprint of five different DEVS simulators in two different hardware platforms.