Learning to Schedule: A Supervised Learning Framework for Network-Aware Scheduling of Data-Intensive Workloads

TL;DR

The paper tackles the problem of network-aware scheduling for data-intensive workloads by introducing a supervised learning-based scheduler that predicts per-node job completion times using real-time telemetry and job configurations. It runs as an external, user-space component that ranks candidate nodes with a trained regression model and enforces placement via Kubernetes nodeAffinity, without altering the control plane. Using a multi-site FABRIC deployment and Spark workloads, the approach demonstrates substantial gains in placement accuracy (34–54% Top-1, 34–62% Top-2) over the default Kubernetes scheduler, validating the viability of offline-trained, network-aware scheduling for production-like environments. The work highlights the practicality of combining telemetry, feature construction, and offline learning to improve data-intensive job performance, with clear paths for scaling, cost analysis, and richer network instrumentation in future work.

Abstract

Distributed cloud environments hosting data-intensive applications often experience slowdowns due to network congestion, asymmetric bandwidth, and inter-node data shuffling. These factors are typically not captured by traditional host-level metrics like CPU or memory. Scheduling without accounting for these conditions can lead to poor placement decisions, longer data transfers, and suboptimal job performance. We present a network-aware job scheduler that uses supervised learning to predict the completion time of candidate jobs. Our system introduces a prediction-and-ranking mechanism that collects real-time telemetry from all nodes, uses a trained supervised model to estimate job duration per node, and ranks them to select the best placement. We evaluate the scheduler on a geo-distributed Kubernetes cluster deployed on the FABRIC testbed by running network-intensive Spark workloads. Compared to the default Kubernetes scheduler, which makes placement decisions based on current resource availability alone, our proposed supervised scheduler achieved 34-54% higher accuracy in selecting optimal nodes for job placement. The novelty of our work lies in the demonstration of supervised learning for real-time, network-aware job scheduling on a multi-site cluster.

Figures (4)

• Figure 1: Overview of the system architecture.
• Figure 2: Average latency per node across five runs of Sort.
• Figure 3: Average transmit bandwidth per node across five runs of Sort.
• Figure 4: Geographical layout of the cluster across three FABRIC sites with RTT measurements shown along the connecting lines.