Rank-Aware Resource Scheduling for Tightly-Coupled MPI Workloads on Kubernetes

Abstract

Fully provisioned Message Passing Interface (MPI) parallelism achieves near-optimal wall-clock time for Computational Fluid Dynamics (CFD) solvers. This work addresses a complementary question for shared, cloud-managed clusters: can fine-grained CPU provisioning reduce resource reservation of low-load subdomains, improving cluster packing efficiency without unacceptably degrading performance? We propose rank-aware resource scheduling on Kubernetes, mapping each MPI rank to a pod whose CPU request is proportional to its subdomain cell count. We also demonstrate In-Place Pod Vertical Scaling (Kubernetes v1.35 GA) for mid-simulation CPU adjustment without pod restart. Three findings emerge. First, hard CPU limits via the Linux CFS bandwidth controller cause 78x slowdown through cascading stalls at MPI_Allreduce barriers; requests-only allocation eliminates throttling entirely. Second, on non-burstable c5.xlarge instances, concentric decomposition with equal CPU is 19% faster than the Scotch baseline, while adding proportional CPU yields a further 3% improvement. Third, at 16 MPI ranks on 101K-cell meshes, proportional allocation is 20% faster than equal allocation while reducing sparse-subdomain provisioned CPU by 82%, freeing 6.5 vCPU of scheduling headroom. Experiments are conducted on AWS EC2 c5.xlarge clusters (4-16 ranks) running k3s v1.35. All scripts and data are released as open source.

Figures (9)

• Figure 1: Cluster packing comparison. (a) Traditional MPI: each rank reserves 1.0 vCPU, leaving 4.0 vCPU idle across two nodes. (b) Rank-aware K8s scheduling: proportional CPU frees headroom for co-resident jobs (orange, green), eliminating idle waste.
• Figure 2: pitzDaily mesh coloured by Scotch subdomain. Top: equal decomposition (C2 baseline), all four subdomains $\approx$3 056 cells each. Bottom: proportional decomposition, processorWeights $(1,1,5,15)$ gives 556 / 556 / 2 778 / 8 335 cells. The downstream channel (Proc 3, red) receives the most cells and the highest CPU budget; the near-step region (Proc 0--1, blue/green) receives the fewest.
• Figure 3: NACA 0012 mesh coloured by subdomain assignment. Top row (a, b): equal Scotch decomposition (C2 baseline), $\approx$4 050 cells per subdomain in four angular sectors. Bottom row (c, d): distance-based proportional decomposition (4:3:2:1), forming concentric rings around the aerofoil. Left column: full domain ($\pm$20$c$); right column: near-field zoom showing the subdomain structure around the aerofoil surface. The concentric decomposition aligns subdomain boundaries with the radial cost gradient: Proc 0 (innermost, 6 480 cells) covers the boundary layer; Proc 3 (outermost, 1 620 cells) covers the far-field.
• Figure 4: Architectural comparison using processorWeights $(1,1,5,15)$. (a) Traditional MPI assigns equal CPU (0.25 of total) per rank regardless of subdomain size; red arrows mark the mismatch between workload and allocation for Proc 3. (b) K8s proportional allocation aligns CPU fraction with cell fraction, eliminating idle capacity in the sparse subdomains.
• Figure 5: Wall-clock time for all seven configurations on pitzDaily (291--293 iterations to convergence). The dashed blue line marks the C2 cross-node K8s baseline (35 s). Note the logarithmic scale: C3 (proportional with hard limits) requires 2738 s, a 78$\times$ increase over C2 caused by CFS bandwidth throttling.