Bringing Computation to the data: Interoperable serverless function execution for astrophysical data analysis in the SRCNet

TL;DR

This work demonstrates the feasibility of integrating Function-as-a-Service into the federated SRCNet infrastructure to enable data-proximate computation for the SKA era. By decomposing radio-astronomy workflows into containerised functions and interconnecting them via global and local SRCNet services, GateKeeper, and IVOA DataLink, the study shows how to bring computation to distributed data while maintaining security and provenance. The gaussconv implementation serves as a concrete validation of deployment, registration, and end-user access within a Kubernetes-based, CI/CD-driven pipeline. The approach promises scalable, reproducible, and policy-compliant analyses across SKA-scale data, with broader applicability to other large-scale, data-intensive scientific domains. Future work aims to enable function chaining, improved scheduling across sites, and enhanced provenance to further generalise federated, function-oriented computation.

Abstract

Serverless computing is a paradigm in which the underlying infrastructure is fully managed by the provider, enabling applications and services to be executed with elastic resource provisioning and minimal operational overhead. A core model within this paradigm is Function-as-a-Service (FaaS), where lightweight functions are deployed and triggered on demand, scaling seamlessly with workload. FaaS offers flexibility, cost-effectiveness, and fine-grained scalability, qualities particularly relevant for large-scale scientific infrastructures where data volumes are too large to centralise and computation must increasingly occur close to the data. The Square Kilometre Array Observatory (SKAO) exemplifies this challenge. Once operational, it will generate about 700~PB of data products annually, distributed across the SKA Regional Centre Network (SRCNet), a federation of international centres providing storage, computing, and analysis services. In such a context, FaaS offers a mechanism to bring computation to the data. We studied the principles of serverless and FaaS computing and explored their application to radio astronomy workflows. Representative functions for astrophysical data analysis were developed and deployed, including micro-functions derived from existing libraries and wrappers around domain-specific applications. In particular, a Gaussian convolution function was implemented and integrated within the SRCNet ecosystem. The use case demonstrates that FaaS can be embedded into the existing SRCNet ecosystem of services, allowing functions to run directly at sites where data replicas are stored. This reduces latency, minimises transfers, and improves efficiency, aligning with federated, data-proximate computation. The results show that serverless models provide a scalable and efficient pathway to address the data volumes of the SKA era.

Figures (7)

• Figure 1: Model of a pipeline to enable a function on a FaaS environment, that encompasses everything from the creation of the Python code for the function to its direct use in a Notebook, including building, deployment, and registration in a catalogue.
• Figure 2: Decomposition of a radio-astronomy pipeline: a monolithic script with wsclean and gaussconv is decomposed into containerised functions and exposed as FaaS HTTP endpoints. Functions are then deployed at SRCNet nodes hosting the dataset replicas to enable data-proximate execution. The same function may have variants, for example with improvements in implementation such as GPU support gaussconv-gpu, which can even run on nodes enabled for this purpose.
• Figure 3: SRCNet architecture highlighting global services (SKAO IAM, SRCNet Permissions API, SRCNet Site-Capabilities, SKAO Rucio Datalake, SRCNet IVOA DataLink) and site-local services (compute/storage infrastructure, Jupyter Notebooks/CARTA, deployed functions, SRCNet GateKeeper). This layered architecture provides the substrate for integrating function-oriented workflows, such as the gaussconv example and other.
• Figure 4: This figure includes the components necessary for the operation of any FaaS environment. Specifically, it focuses on the orchestration layer (Kubernetes), the storage layer for the cluster and services on the orchestrator, as well as the functions (both the FaaS platform and native mode), and the container registry, necessary for archiving container images. The automated deployment files reside in a repository in GitLab that allows CI/CD to be used for this purpose, together with a Flux CI/CD service.
• Figure 5: Workflow of the steps required to integrate a new function into the SRCNet ecosystem: development and containerisation, CI/CD deployment, SRCNet GateKeeper configuration, registration in SRCNet Site-Capabilities, integration with SRCNet Permissions API and IVOA DataLink, and final exposure through the astroquery.srcnet wrapper.