A Lean Simulation Framework for Stress Testing IoT Cloud Systems

TL;DR

The paper addresses the challenge of stress testing IoT cloud systems at scale by introducing a lean edge-to-cloud simulation framework, IoTECS, and a domain-specific language that automates simulator generation. It emphasizes three design features—symbolic IoT devices, configurable edge start-time variability, and simulation-node clustering—to achieve high scalability on commodity hardware, particularly using Docker containers. Empirical evaluations on two industrial case studies show IoTECS-based simulators outperform baseline tools (JMeter and Locust) by enabling 3.5x more IoT devices, and they quantify service capacities for UDP/TCP and MQTT clouds, along with practitioner feedback indicating significant time savings and ease of use. The work demonstrates practical applicability for stress-testing IoT cloud systems, while acknowledging limitations such as protocol scope, immutability, and offline metrics, and points to future enhancements including real-time metrics and cost-aware simulation. The release of IoTECS and artifacts supports reproducibility and adoption in industry contexts.

Abstract

The Internet of Things connects a plethora of smart devices globally across various applications like smart cities, autonomous vehicles and health monitoring. Simulation plays a key role in the testing of IoT systems, noting that field testing of a complete IoT product may be infeasible or prohibitively expensive. This paper addresses a specific yet important need in simulation-based testing for IoT: Stress testing of cloud systems. Existing stress testing solutions for IoT demand significant computational resources, making them ill-suited and costly. We propose a lean simulation framework designed for IoT cloud stress testing which enables efficient simulation of a large array of IoT and edge devices that communicate with the cloud. To facilitate simulation construction for practitioners, we develop a domain-specific language (DSL), named IoTECS, for generating simulators from model-based specifications. We provide the syntax and semantics of IoTECS and implement IoTECS using Xtext and Xtend. We assess simulators generated from IoTECS specifications for stress testing two real-world systems: a cloud-based IoT monitoring system and an IoT-connected vehicle system. Our empirical results indicate that simulators created using IoTECS: (1)achieve best performance when configured with Docker containerization; (2)effectively assess the service capacity of our case-study systems, and (3)outperform industrial stress-testing baseline tools, JMeter and Locust, by a factor of 3.5 in terms of the number of IoT and edge devices they can simulate using identical hardware resources. To gain initial insights about the usefulness of IoTECS in practice, we interviewed two engineers from our industry partner who have firsthand experience with IoTECS. Feedback from these interviews suggests that IoTECS is effective in stress testing IoT cloud systems, saving significant time and effort.

Figures (13)

• Figure 1: A schematic overview of the common tiers in IoT systems.
• Figure 2: Novel design features of our edge-to-cloud simulation framework.
• Figure 3: The design of EMU-IoT simulator Ramprasad:19
• Figure 4: The design of the Fogbed simulator Coutinho:18
• Figure 5: IoTECS's underlying conceptual model.