Lessons Learned from Building Edge Software System Testbeds

TL;DR

The paper addresses the challenge of evaluating edge software at scale without physical infrastructure by detailing five iterative edge testbeds (MockFog family and Celestial). It presents concrete architectural lessons, from network emulation and virtualization choices to cloud integration, time synchronization, and automation practices, highlighting how each iteration informed the next. The authors contribute practical guidance on when to specialize versus generalize, the importance of testing the testbed itself, and the trade-offs between containers and microVMs, as well as advanced topics like visualization and orchestration. The work has practical impact for researchers and educators by offering a cost-efficient, repeatable blueprint for building and using edge testbeds while identifying open challenges and avenues for future work.

Abstract

Edge computing requires the complex software interaction of geo-distributed, heterogeneous components. The growing research and industry interest in edge computing software systems has necessitated exploring ways of testing and evaluating edge software at scale without relying on physical infrastructure. Beyond simulation, virtual testbeds that emulate edge infrastructure can provide a cost-efficient yet realistic environment to evaluate edge software. In this experience paper, we share lessons learned from building a total of five edge software testbeds. We describe pitfalls in architecture and development as well as experiences from having students use our testbed tooling in distributed systems prototyping classes. While we remain confident that building custom testbed tooling is the right approach for edge computing researchers and practitioners alike, we hope this paper allows others to avoid common mistakes and benefit from our experience.

Figures (7)

• Figure 1: Edge testbeds commonly deploy each infrastructure node in its own isolation or virtualization environment and network them with emulated network characteristics. A coordinating component modifies node and network capabilities. The system-under-test (SUT), i.e., the software systems, can then run on each virtual edge node.
• Figure 2: An example MockFog 2.0 topology where an application developer deploys a MockFog infrastructure and topology plan to the node managerpaper_hasenburg2021_mockfog2. Each edge and cloud node is a dedicated virtual machine running the node agent, which translates control message from the node manager, and the application itself.
• Figure 3: A Celestial emulation is based on a real-time satellite simulation at the coordinator, which instructs hosts on which Firecracker microVMs emulate groundstation and satellite servers. All hosts are connected with a network plane on which delays and bandwidth limits between nodes are emulated paper_pfandzelter2022_celestialtechreport_pfandzelter2022_celestial_extendedpfandzelter2023celestialdemo
• Figure 4: Setup time for $500 \times (500-1)$ links in Celestial with NetEm and our eBPF-based approach.
• Figure 5: In MockFog and Celestial (among other testbeds), a logical topology (gray), e.g., with node A connecting to node C via node B, is reduced to a fully meshed topology. The newly introduced emulated link between A and C (shown as a dashed red line) inherits the summed latency and minimum bandwidth of the logical path between A and C.