Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Morpheus: Lightweight RTT Prediction for Performance-Aware Load Balancing

Panagiotis Giannakopoulos, Bart van Knippenberg, Kishor Chandra Joshi, Nicola Calabretta, George Exarchakos

TL;DR

Morpheus tackles the challenge of proactive, RTT-aware load balancing in co-located edge-cloud environments. It presents lightweight, runtime RTT predictors that learn from live Kubernetes monitoring data, selecting correlated metrics and efficient models to predict per-request RTT with high accuracy and low overhead. The framework integrates data collection, model training, and prediction in parallel, and its impact on load balancing is demonstrated via simulation, showing substantial reductions in completion time and resource waste when RTT predictions guide replica selection. The work highlights practical deployment considerations, including dataset balancing, monitoring bottlenecks, and heterogeneity-aware strategies, and points toward production-ready integration and monitoring-system optimizations as future directions.

Abstract

Distributed applications increasingly demand low end-to-end latency, especially in edge and cloud environments where co-located workloads contend for limited resources. Traditional load-balancing strategies are typically reactive and rely on outdated or coarse-grained metrics, often leading to suboptimal routing decisions and increased tail latencies. This paper investigates the use of round-trip time (RTT) predictors to enhance request routing by anticipating application latency. We develop lightweight and accurate RTT predictors that are trained on time-series monitoring data collected from a Kubernetes-managed GPU cluster. By leveraging a reduced set of highly correlated monitoring metrics, our approach maintains low overhead while remaining adaptable to diverse co-location scenarios and heterogeneous hardware. The predictors achieve up to 95% accuracy while keeping the prediction delay within 10% of the application RTT. In addition, we identify the minimum prediction accuracy threshold and key system-level factors required to ensure effective predictor deployment in resource-constrained clusters. Simulation-based evaluation demonstrates that performance-aware load balancing can significantly reduce application RTT and minimize resource waste. These results highlight the feasibility of integrating predictive load balancing into future production systems.

Morpheus: Lightweight RTT Prediction for Performance-Aware Load Balancing

TL;DR

Morpheus tackles the challenge of proactive, RTT-aware load balancing in co-located edge-cloud environments. It presents lightweight, runtime RTT predictors that learn from live Kubernetes monitoring data, selecting correlated metrics and efficient models to predict per-request RTT with high accuracy and low overhead. The framework integrates data collection, model training, and prediction in parallel, and its impact on load balancing is demonstrated via simulation, showing substantial reductions in completion time and resource waste when RTT predictions guide replica selection. The work highlights practical deployment considerations, including dataset balancing, monitoring bottlenecks, and heterogeneity-aware strategies, and points toward production-ready integration and monitoring-system optimizations as future directions.

Abstract

Distributed applications increasingly demand low end-to-end latency, especially in edge and cloud environments where co-located workloads contend for limited resources. Traditional load-balancing strategies are typically reactive and rely on outdated or coarse-grained metrics, often leading to suboptimal routing decisions and increased tail latencies. This paper investigates the use of round-trip time (RTT) predictors to enhance request routing by anticipating application latency. We develop lightweight and accurate RTT predictors that are trained on time-series monitoring data collected from a Kubernetes-managed GPU cluster. By leveraging a reduced set of highly correlated monitoring metrics, our approach maintains low overhead while remaining adaptable to diverse co-location scenarios and heterogeneous hardware. The predictors achieve up to 95% accuracy while keeping the prediction delay within 10% of the application RTT. In addition, we identify the minimum prediction accuracy threshold and key system-level factors required to ensure effective predictor deployment in resource-constrained clusters. Simulation-based evaluation demonstrates that performance-aware load balancing can significantly reduce application RTT and minimize resource waste. These results highlight the feasibility of integrating predictive load balancing into future production systems.
Paper Structure (24 sections, 11 equations, 11 figures, 5 tables)

This paper contains 24 sections, 11 equations, 11 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Predictor architecture
  4. Data collection process
  5. Model Training Process
  6. Prediction
  7. Experimental Setup
  8. Applications
  9. Frameworks
  10. Compute Infrastructure
  11. Experimental Setup
  12. Evaluation of predictors
  13. Selected correlation algorithms
  14. Selected predictor configuration
  15. Prediction adaptation on dynamic co-location
  16. ...and 9 more sections

Figures (11)

  • Figure 1: High-level architecture for integrating RTT predictors into the load balancer of a resource cluster.
  • Figure 2: Overview of the RTT prediction methodology.
  • Figure 3: Cluster architecture. All nodes are interconnected via 1 Gbps Ethernet.
  • Figure 4: Proportion of monitoring metrics for each application for which each correlation method yields the highest correlation with RTT.
  • Figure 5: Proportion of selected RTT predictor configurations by model type, number of metrics, and observation window (in seconds) across all applications and nodes.
  • ...and 6 more figures