How to Evaluate Distributed Coordination Systems? -- A Survey and Analysis
Bekir Turkkan, Elvis Rodrigues, Tevfik Kosar, Aleksey Charapko, Ailidani Ailijiang, Murat Demirbas
TL;DR
This paper tackles the absence of standard benchmarks for distributed coordination systems by surveying evaluation practices across consensus algorithms, coordination services, and applications. It catalogs metrics, topologies, and workload factors used in existing studies, and critically assesses their deficiencies, particularly in data access patterns, ownership, and WAN-scale testing. The authors propose core benchmarking requirements—flexibility, WAN awareness, scalability, and ease of adoption—and argue for a unified, extensible suite that can fairly compare diverse coordination technologies. The work highlights practical implications for researchers and practitioners aiming to rigorously assess performance, availability, scalability, and consistency of distributed coordination systems.
Abstract
Coordination services and protocols are critical components of distributed systems and are essential for providing consistency, fault tolerance, and scalability. However, due to the lack of standard benchmarking and evaluation tools for distributed coordination services, coordination service developers/researchers either use a NoSQL standard benchmark and omit evaluating consistency, distribution, and fault tolerance; or create their own ad-hoc microbenchmarks and skip comparability with other services. In this study, we analyze and compare the evaluation mechanisms for known and widely used consensus algorithms, distributed coordination services, and distributed applications built on top of these services. We identify the most important requirements of distributed coordination service benchmarking, such as the metrics and parameters for the evaluation of the performance, scalability, availability, and consistency of these systems. Finally, we discuss why the existing benchmarks fail to address the complex requirements of distributed coordination system evaluation.