Hybrid Cloud Architectures for Research Computing: Applications and Use Cases

TL;DR

This paper addresses the fragmentation of research computing by advocating hybrid cloud architectures that integrate grid and cloud environments to optimize performance, cost, and accessibility. It surveys deployment models, containerization, workflow management, and common execution platforms, and presents five practical, layer‑based architectural approaches demonstrated within the ELIXIR Compute Platform. The work contributes a governance‑oriented roadmap for adopting hybrid/multi‑cloud in research, highlighting federated computing, data provenance, interoperability, and security as central challenges. The findings have practical impact by guiding the European research ecosystem toward interoperable, scalable, and sustainable infrastructures aligned with EOSC and ELIXIR objectives.

Abstract

Scientific research increasingly depends on robust and scalable IT infrastructures to support complex computational workflows. With the proliferation of services provided by research infrastructures, NRENs, and commercial cloud providers, researchers must navigate a fragmented ecosystem of computing environments, balancing performance, cost, scalability, and accessibility. Hybrid cloud architectures offer a compelling solution by integrating multiple computing environments to enhance flexibility, resource efficiency, and access to specialised hardware. This paper provides a comprehensive overview of hybrid cloud deployment models, focusing on grid and cloud platforms (OpenPBS, SLURM, OpenStack, Kubernetes) and workflow management tools (Nextflow, Snakemake, CWL). We explore strategies for federated computing, multi-cloud orchestration, and workload scheduling, addressing key challenges such as interoperability, data security, reproducibility, and network performance. Drawing on implementations from life sciences, as coordinated by the ELIXIR Compute Platform and their integration into a wider EOSC context, we propose a roadmap for accelerating hybrid cloud adoption in research computing, emphasising governance frameworks and technical solutions that can drive sustainable and scalable infrastructure development.

Figures (5)

• Figure 1: Service type abstractions from bottom to top and associated level of responsibility between customer and vendor concerning implementation and operation. IaaS: Infrastructure-as-a-Service; PaaS: Platform-as-a-Service; CaaS: Container-as-as-Service; SaaS: Software-as-a-Service; FaaS: Function-as-a-Service. From bottom to top, more responsibility concerning development, provisioning and operations are shifted from the customer towards the vendor.
• Figure 2: Deployment and execution flow for Nextflow and SLURM environments deployed independently to three different cloud or on-premise environments. Data flows from the source (left), is then split into batches that are then executed independently in one of the three environments, before result data is recombined in a common storage (right), FTP in this case.
• Figure 3: An implementation of the advanced solution in the second approach. The federated computing across multiple HPC clusters and multiple clouds (e.g. GCP & AWS) was used for the systematic analysis of SARS-CoV-2 genomes in the COVID-19 pandemic.
• Figure 4: Schematic of a BiBiGrid Multi-Cloud Cluster. The main deployment, including the master server has control over local and remote job scheduling (the latter via VPN tunnel). Usage of Dnsmasq and Wireguard to set up the VPN allows the creation of a virtual execution environment spanning multiple clouds. Data access for reading and writing is provided by a central NFS server, also accessible to the remote clouds via VPN.
• Figure 5: Workflow execution via a TES Gateway allows fine-granular scheduling of individual sequential or parallel workflow steps to suitable TES nodes, e.g. the geographically closest ones, for execution. Integration between the TES Gateway, nodes and clients allows for transparent execution of map / reduce or scatter / gather-type workflow tasks. Due to its support for multiple workflow engines like Nextflow, Snakemake and CWL, TES enables workflow agnostic execution.