LA-IMR: Latency-Aware, Predictive In-Memory Routing and Proactive Autoscaling for Tail-Latency-Sensitive Cloud Robotics

TL;DR

LA-IMR tackles tail latency in hybrid edge–cloud AI inference by coupling a closed-form, utilisation-driven latency model with an in-memory, SLO-aware control loop that routes requests, offloads traffic, and scales replicas proactively. The end-to-end latency is decomposed into processing, network, and queueing components, enabling two complementary latency functions $g_{m,i}(\boldsymbol{\lambda})$ and $g_{m,i}(N_{m,i})$ to guide millisecond-scale routing and capacity planning. A quality-differentiated multi-queue scheduler and a custom-metric Kubernetes autoscaler enable just-in-time scaling, reducing P99 latency by up to $20.7\%$ and substantially lowering tail variance on bursty vision workloads like YOLOv5m and EfficientDet. Experiments across a cloud–edge continuum validate that predictive routing and proactive autoscaling suppress long-tail spikes, delivering more stable, tail-tolerant cloud robotics inference services.

Abstract

Hybrid cloud-edge infrastructures now support latency-critical workloads ranging from autonomous vehicles and surgical robotics to immersive AR/VR. However, they continue to experience crippling long-tail latency spikes whenever bursty request streams exceed the capacity of heterogeneous edge and cloud tiers. To address these long-tail latency issues, we present Latency-Aware, Predictive In-Memory Routing and Proactive Autoscaling (LA-IMR). This control layer integrates a closed-form, utilization-driven latency model with event-driven scheduling, replica autoscaling, and edge-to-cloud offloading to mitigate 99th-percentile (P99) delays. Our analytic model decomposes end-to-end latency into processing, network, and queuing components, expressing inference latency as an affine power-law function of instance utilization. Once calibrated, it produces two complementary functions that drive: (i) millisecond-scale routing decisions for traffic offloading, and (ii) capacity planning that jointly determines replica pool sizes. LA-IMR enacts these decisions through a quality-differentiated, multi-queue scheduler and a custom-metric Kubernetes autoscaler that scales replicas proactively -- before queues build up -- rather than reactively based on lagging CPU metrics. Across representative vision workloads (YOLOv5m and EfficientDet) and bursty arrival traces, LA-IMR reduces P99 latency by up to 20.7 percent compared to traditional latency-only autoscaling, laying a principled foundation for next-generation, tail-tolerant cloud-edge inference services.

Figures (8)

• Figure 1: LA‑IMR: an in‑memory SLO‑aware controller that routes requests across edge–cloud tiers.
• Figure 2: The inference latency measured in the real operations and predicted by Eq. \ref{['eq:inference-delay-final']} with $\alpha_i$=0.73, $\beta_{m,i}$ = 1.29, and $\gamma$=1.49 (3 CPUs per replica).
• Figure 3: Latency metrics for user robot19 under varying arrival rates, showing super-linear growth in average, P95, and P99 latencies.
• Figure 4: Inference latency comparison between the microservice and monolithic service architecture as the number of replica $N_{m,i}$ increases when the arrival rate $\lambda$=4 is given. Overall, the microservice architecture shows the superior latency.
• Figure 5: Real-time latency prediction uses the arrival rate $\lambda$ to meet the target latency $\tau$. If latency exceeds $\tau$, the system increases replicas $N_{m,i}$. This prediction also enables proactive offloading based on $\lambda$ and $N_{m,i}$.