ScalerEval: Automated and Consistent Evaluation Testbed for Auto-scalers in Microservices
Shuaiyu Xie, Jian Wang, Yang Luo, Yunqing Yong, Yuzhen Tan, Bing Li
TL;DR
This paper tackles the challenge of objectively evaluating auto-scalers for microservices, which is hampered by complex infrastructure, manual operations, and inconsistent benchmarks. It introduces ScalerEval, an automated, consistent testbed that automates benchmark initialization, scaler registration, workload injection, metric collection, and performance assessment, supported by a scalable scaler template and essential interfaces; the workflow can be executed with a single command. The authors implement and evaluate state-of-the-art auto-scalers on two popular microservice benchmarks, demonstrating improved reproducibility and fairer comparisons, with metrics such as $SVR$ and $SR$ guiding performance and cost trade-offs. The work advances reproducibility and efficiency in auto-scaler research and provides an open-source platform for researchers to rapidly develop and compare scaling strategies, while outlining plans to extend to vertical scaling and additional orchestration tasks.
Abstract
Auto-scaling is an automated approach that dynamically provisions resources for microservices to accommodate fluctuating workloads. Despite the introduction of many sophisticated auto-scaling algorithms, evaluating auto-scalers remains time-consuming and labor-intensive, as it requires the implementation of numerous fundamental interfaces, complex manual operations, and in-depth domain knowledge. Besides, frequent human intervention can inevitably introduce operational errors, leading to inconsistencies in the evaluation of different auto-scalers. To address these issues, we present ScalerEval, an end-to-end automated and consistent testbed for auto-scalers in microservices. ScalerEval integrates essential fundamental interfaces for implementation of auto-scalers and further orchestrates a one-click evaluation workflow for researchers. The source code is publicly available at \href{https://github.com/WHU-AISE/ScalerEval}{https://github.com/WHU-AISE/ScalerEval}.