DRPC: Distributed Reinforcement Learning Approach for Scalable Resource Provisioning in Container-based Clusters

TL;DR

DRPC tackles autoscaling scalability in container-based microservice clusters by decentralizing decision-making through a TD3-based distributed reinforcement learning framework, complemented by imitation learning to align local deployments with global policy. The architecture splits learning into a Central Teacher Network and Distributed Student Deployment Networks, aided by a GRU-based workload predictor and MTFS-based forecasting within a MAPe-K style loop. Empirical results on a Kubernetes testbed with TrainTicket-like workloads and Alibaba traces show DRPC achieves about a 15% reduction in average response time and a 24% reduction in failed requests under high load, outperforming KuScal, CoScal, and FIRM in throughput and QoS. The work demonstrates that asynchronous, distributed autoscaling can improve scalability and QoS for large microservice ecosystems, with open-source code to facilitate adoption and further research.

Abstract

Microservices have transformed monolithic applications into lightweight, self-contained, and isolated application components, establishing themselves as a dominant paradigm for application development and deployment in public clouds such as Google and Alibaba. Autoscaling emerges as an efficient strategy for managing resources allocated to microservices' replicas. However, the dynamic and intricate dependencies within microservice chains present challenges to the effective management of scaled microservices. Additionally, the centralized autoscaling approach can encounter scalability issues, especially in the management of large-scale microservice-based clusters. To address these challenges and enhance scalability, we propose an innovative distributed resource provisioning approach for microservices based on the Twin Delayed Deep Deterministic Policy Gradient algorithm. This approach enables effective autoscaling decisions and decentralizes responsibilities from a central node to distributed nodes. Comparative results with state-of-the-art approaches, obtained from a realistic testbed and traces, indicate that our approach reduces the average response time by 15% and the number of failed requests by 24%, validating improved scalability as the number of requests increases.

Figures (4)

• Figure 1: Response time of centralized approaches when the number of requests increases significantly
• Figure 2: System Model of DRPC
• Figure 3: Performance comparison of KuScal, FIRM, CoScal, and DRPC
• Figure 4: CDF comparison and reward convergence