Extending the Control Plane of Container Orchestrators for I/O Virtualization
Garegin Grigoryan, Minseok Kwon, M. Mustafa Rafique
TL;DR
This work addresses the limited controllability of SR-IOV in container orchestrators by proposing ConRDMA, a control-plane extension for Kubernetes that enables per-pod RDMA bandwidth management. It introduces a hardware daemon set on each node, a scheduler extender on the master, and a modified CNI to allocate and move RDMA VFs into pod namespaces, with per-VF bandwidth limits. Evaluation shows that ConRDMA achieves more accurate, demand-driven bandwidth allocation with minimal impact on latency and enables multi-interface per-pod RDMA configurations. The approach preserves SR-IOV advantages while improving manageability and efficiency in heterogeneous clusters, offering practical guidance for real-world deployment.
Abstract
Single Root Input/Output Virtualization (SR-IOV) is a standard technology for forking a single PCI express device and providing it to applications while ensuring performance isolation. It enables container orchestrators to share a limited number of physical network interfaces without incurring significant virtualization overhead. The allocation of virtualized network devices to containers, however, needs to be more configurable based on the bandwidth needs of running applications. Moreover, container orchestrators' network control over the virtualized interfaces is limited by the abilities of SR-IOV. We explore the design considerations for a system with controlled SR-IOV virtualization and present ConRDMA, a novel architecture that enables fine control of RDMA virtualization for containers. Our evaluation shows that ConRDMA enables containers to use RDMA allocated bandwidth more efficiently and to select best-suited nodes to meet their varying communication requirements.