LIDC: A Location Independent Multi-Cluster Computing Framework for Data Intensive Science
Sankalpa Timilsina, Susmit Shannigrahi
TL;DR
This work tackles the challenge of locating and orchestrating compute across geographically distributed, multi-organizational resources. It introduces LIDC, a decentralized, name-based framework that uses semantic names and Named Data Networking to route compute requests to named Kubernetes endpoints and data lakes, removing the need for prior cluster knowledge. The approach couples network-layer matchmaking with in-cluster services, supports data-aware computation, and demonstrates a Genomics workflow deployment on a MicroK8s/GCP testbed, including modular per-application validations. The results indicate potential for dynamic, resilient, and scalable data-intensive workflows with reduced manual configuration, offering practical benefits for cross-institutional scientific computing.
Abstract
Scientific communities are increasingly using geographically distributed computing platforms. The current methods of compute placement predominantly use logically centralized controllers such as Kubernetes (K8s) to match tasks to available resources. However, this centralized approach is unsuitable in multi-organizational collaborations. Furthermore, workflows often need to use manual configurations tailored for a single platform and cannot adapt to dynamic changes across infrastructure. Our work introduces a decentralized control plane for placing computations on geographically dispersed compute clusters using semantic names. We assign semantic names to computations to match requests with named Kubernetes (K8s) service endpoints. We show that this approach provides multiple benefits. First, it allows placement of computational jobs to be independent of location, enabling any cluster with sufficient resources to execute the computation. Second, it facilitates dynamic compute placement without requiring prior knowledge of cluster locations or predefined configurations.