StraightLine: An End-to-End Resource-Aware Scheduler for Machine Learning Application Requests

TL;DR

StraightLine addresses the gap in ML deployment by providing an end-to-end resource-aware scheduler for heterogeneous infrastructure. It introduces an empirical dynamic placing algorithm that selects between serverless, Docker containers, and local Flask-backed services based on workload features, including the frequency $f_t$ and data size $r_d$, guided by thresholds $F$ and $D$. The design comprises three layers: model containerization with NVIDIA-Docker, container customization for RESTful/serverless deployment, and real-time scheduling to minimize latency and failures. Evaluation on a real hybrid testbed with RESTful APIs, serverless platforms, Docker, and VMs demonstrates improved deployment performance and adaptability to resource heterogeneity.

Abstract

The life cycle of machine learning (ML) applications consists of two stages: model development and model deployment. However, traditional ML systems (e.g., training-specific or inference-specific systems) focus on one particular stage or phase of the life cycle of ML applications. These systems often aim at optimizing model training or accelerating model inference, and they frequently assume homogeneous infrastructure, which may not always reflect real-world scenarios that include cloud data centers, local servers, containers, and serverless platforms. We present StraightLine, an end-to-end resource-aware scheduler that schedules the optimal resources (e.g., container, virtual machine, or serverless) for different ML application requests in a hybrid infrastructure. The key innovation is an empirical dynamic placing algorithm that intelligently places requests based on their unique characteristics (e.g., request frequency, input data size, and data distribution). In contrast to existing ML systems, StraightLine offers end-to-end resource-aware placement, thereby it can significantly reduce response time and failure rate for model deployment when facing different computing resources in the hybrid infrastructure.

Figures (8)

• Figure 1: The workflow of StraightLine.
• Figure 2: The core information in the dockerfile.
• Figure 3: Training and validation accuracy and loss for the same task running by the CPU and GPU cluster respectively.
• Figure 4: The performance of a Flask API server implemented on a local web server and in-house data center. (a) shows the failure rate of the local web server and the in-house data center. (b) and (c) show the session length of the local web server and the in-house data center, respectively. Total sessions start from 10 requests per 180 seconds to 2,000 requests per 180 seconds against the Xception model.
• Figure 5: The results show the comparison for the same Xception application Docker implemented on AWS Lambda with 2GB and 3GB provisioned respectively. (a) shows the failed rate directly impacted by the provisioned memory. (b) and (c) show that the application latency is not affected much by the load changes.