Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Advanced computing for reproducibility of astronomy Big Data Science, with a showcase of AMIGA and the SKA Science prototype

Julián Garrido, Susana Sánchez, Edgar Ribeiro João, Roger Ianjamasimanana, Manuel Parra, Lourdes Verdes-Montenegro

TL;DR

The paper addresses reproducibility challenges in SKA Big Data by proposing interoperable data models, semantic descriptions of SRCNet resources, and distributed analysis services to enable provable, federated science. It introduces the SKA-ProvSDP provenance model and a JSON-LD SRCNet data model to connect high-level workflows with low-level processing across heterogeneous infrastructures, and discusses SaaS and FaaS paradigms to improve accessibility and portability. Through AMIGA case studies on HCG16 and MeerKAT HCGs, it demonstrates end-to-end reproducibility using containerised pipelines, workflow engines (e.g., Snakemake, CARACAL), and provenance capture, complemented by notebooks, Binder, and VO data publication. The results argue that reproducibility must be a first-class architectural requirement for the SRCNet and SKAO, enabling Open Science, FAIR data practices, and scalable, verifiable science in the Big Data era.

Abstract

The Square Kilometre Array Observatory (SKAO) faces unprecedented technological challenges due to the vast scale and complexity of its data. This paper provides an overview of research by the AMIGA group to address these computing and reproducibility challenges. We present advancements in semantic data models, analysis services integrated into federated infrastructures, and the application to astronomy studies of techniques that enhance research transparency. By showcasing these astronomy work, we demonstrate that achieving reproducible science in the Big Data era is feasible. However, we conclude that for the SKAO to succeed, the development of the SKA Regional Centre Network (SRCNet) must explicitly incorporate these reproducibility requirements into its fundamental architectural design. Embedding these standards is crucial to enable the global community to conduct verifiable and sustainable research within a federated environment.

Advanced computing for reproducibility of astronomy Big Data Science, with a showcase of AMIGA and the SKA Science prototype

TL;DR

The paper addresses reproducibility challenges in SKA Big Data by proposing interoperable data models, semantic descriptions of SRCNet resources, and distributed analysis services to enable provable, federated science. It introduces the SKA-ProvSDP provenance model and a JSON-LD SRCNet data model to connect high-level workflows with low-level processing across heterogeneous infrastructures, and discusses SaaS and FaaS paradigms to improve accessibility and portability. Through AMIGA case studies on HCG16 and MeerKAT HCGs, it demonstrates end-to-end reproducibility using containerised pipelines, workflow engines (e.g., Snakemake, CARACAL), and provenance capture, complemented by notebooks, Binder, and VO data publication. The results argue that reproducibility must be a first-class architectural requirement for the SRCNet and SKAO, enabling Open Science, FAIR data practices, and scalable, verifiable science in the Big Data era.

Abstract

The Square Kilometre Array Observatory (SKAO) faces unprecedented technological challenges due to the vast scale and complexity of its data. This paper provides an overview of research by the AMIGA group to address these computing and reproducibility challenges. We present advancements in semantic data models, analysis services integrated into federated infrastructures, and the application to astronomy studies of techniques that enhance research transparency. By showcasing these astronomy work, we demonstrate that achieving reproducible science in the Big Data era is feasible. However, we conclude that for the SKAO to succeed, the development of the SKA Regional Centre Network (SRCNet) must explicitly incorporate these reproducibility requirements into its fundamental architectural design. Embedding these standards is crucial to enable the global community to conduct verifiable and sustainable research within a federated environment.
Paper Structure (11 sections, 3 figures)

This paper contains 11 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Data models to improve interoperability in the SKA Science platforms
  3. Science Data Processor data model
  4. The SRCNet data model
  5. Advance data analysis services in Science platforms
  6. Software as a service in distributed computing infrastructures
  7. Function as a service in the science platform
  8. Reproducibility in AMIGA science studies
  9. Reproducibility in the HCG16 study
  10. Reproducibility in a MeerKAT study on HCGs
  11. Conclusions

Figures (3)

  • Figure 1: Conceptual view of the SRCNet data model showing the relationships between SRCNet, SRCs, Nodes, and Services. Services include Jupyter Notebook, CARTA, CANFAR, Rucio SE, Monitoring, and SODA.
  • Figure 2: Architecture illustrating how alternative solutions can be combined by advanced analysis tools to enable seamless access to a variety of computing infrastructures.
  • Figure 3: Architecture for reproducibility in the HCG16 study. The top layer represents data sources and scientific archives for original and derived datasets. The bottom layer includes repositories for methods and code (e.g., GitHub, container registries). The middle layer is divided into two components: reproducibility of the computational pipeline and reproducibility of figures. The bottom layer shows the underlying infrastructures (cloud, grid, HPC, Kubernetes) supporting deployment and execution.