MIDAS: Adaptive Proxy Middleware for Mitigating Metadata Hotspots in HPC I/O at Scale

TL;DR

MIDAS tackles metadata hotspots in HPC I/O by inserting an adaptive middleware layer that sits between clients and metadata servers. It combines namespace-aware load balancing, cooperative caching with leases, and a self-stabilizing control loop to dynamically adjust routing and cache lifetimes, all without kernel or backend modifications. Theoretical analysis and controlled experiments show reductions in average queue lengths by about $23\%$ and hotspots by up to $80\%$ relative to round-robin baselines, demonstrating improved scalability and tail latency under bursty workloads. This middleware-anchored approach offers backend-agnostic performance gains and stability, with practical implications for bursty HPC and cloud-scale storage systems.

Abstract

Metadata hotspots remain one of the key obstacles to scalable Input/Output (I/O) in both High-Performance Computing (HPC) and cloud-scale storage environments. Situations such as job start-ups, checkpoint storms, or heavily skewed namespace access can trigger thousands of concurrent metadata requests against a small subset of servers. The result is long queues, inflated tail latencies, and reduced system throughput. Prior efforts including static namespace partitioning, backend-specific extensions, and kernel-level modifications address parts of the problem, but they often prove too rigid, intrusive to deploy, or unstable under shifting workloads. We present MIDAS, an adaptive middleware layer that operates transparently between clients and metadata servers, requiring no changes to kernels or storage backends. The design brings together three mechanisms: (i) a namespace-aware load balancer that enhances consistent hashing with power-of-d sampling informed by live telemetry, (ii) a cooperative caching layer that preserves backend semantics through leases, invalidations, or adaptive timeouts, and (iii) a self-stabilizing control loop that dynamically adjusts routing aggressiveness and cache lifetimes while avoiding oscillations under bursty workloads. Analysis of the model and controlled experiments show that MIDAS reduces average queue lengths by roughly 23% and mitigates worst-case hotspots by up to 80% when compared to round-robin scheduling. These findings highlight that a stability-aware, middleware-based strategy can provide backend-agnostic improvements to metadata management, enabling better scalability in bursty scenarios, more predictable tail latencies, and stronger overall system performance.

Figures (4)

• Figure 1: Architecture of
• Figure 2: Traffic patterns across different workloads. Each dot corresponds to an request.
• Figure 3: Queue length over time using round-robin distribution (Lustre baseline).
• Figure 4: Queue length over time using distribution (power-of-$d$).