Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Missing Adapter Layer for Research Computing

Bowen Li, Jiazhu Xie, Chelsea Wang, Alessandro Umberto D'Aloia, Ziqi Xu, Fengling Han

Abstract

Higher Degree by Research (HDR) candidates increasingly depend on cloud-provisioned virtual machines and local GPU hardware for their computational experiments, yet a persistent and under-addressed gap exists between having compute resources and using them productively. Cloud and infrastructure teams can provision virtual machines, but the path from a raw VM to a reproducible, GPU-ready research environment remains a significant barrier for researchers who are domain experts, not systems engineers. We identify this gap as a missing adapter layer between cloud provisioning and interactive research work. We present a lightweight, open-source solution built on k3s and Coder that implements this adapter layer and is already in active use in our research workspace environment. Our CI/CD pipeline connects GitHub directly to the local cluster, deploying research projects in under five minutes. We define a concrete metrics framework for evaluating this layer -- covering deployment latency, environment reproducibility, onboarding friction, and resource utilisation -- and establish baselines against which improvements can be measured.

The Missing Adapter Layer for Research Computing

Abstract

Higher Degree by Research (HDR) candidates increasingly depend on cloud-provisioned virtual machines and local GPU hardware for their computational experiments, yet a persistent and under-addressed gap exists between having compute resources and using them productively. Cloud and infrastructure teams can provision virtual machines, but the path from a raw VM to a reproducible, GPU-ready research environment remains a significant barrier for researchers who are domain experts, not systems engineers. We identify this gap as a missing adapter layer between cloud provisioning and interactive research work. We present a lightweight, open-source solution built on k3s and Coder that implements this adapter layer and is already in active use in our research workspace environment. Our CI/CD pipeline connects GitHub directly to the local cluster, deploying research projects in under five minutes. We define a concrete metrics framework for evaluating this layer -- covering deployment latency, environment reproducibility, onboarding friction, and resource utilisation -- and establish baselines against which improvements can be measured.
Paper Structure (18 sections, 2 figures, 4 tables)

This paper contains 18 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. The Adapter Layer Problem
  3. Architecture and Implementation
  4. Cluster Layer: k3s for Lightweight Orchestration
  5. Workspace Layer: Coder for Self-Service Lifecycle Management
  6. Environment Layer: Versioned Container Images
  7. CI/CD Pipeline: Project Deployment in Under Five Minutes
  8. Metrics Framework and Baselines
  9. Metric 1: Workspace Deployment Latency
  10. Metric 2: Environment Reproducibility Rate
  11. Metric 3: Onboarding Time to First Experiment
  12. Metric 4: GPU Utilisation and Idle Capacity
  13. Metrics Summary
  14. Related Work
  15. Discussion and Conclusion
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Gap-to-solution mapping. Each identified adapter layer gap maps to one or more architectural components introduced in Section \ref{['sec:architecture']}: versioned container images, the k3s scheduling layer, and the Coder workspace layer.
  • Figure 2: System architecture overview. The adapter layer (centre) sits between raw provisioned compute resources (EC2 instances, local GPU nodes) and the HDR candidate's interactive workspace. It comprises three components: versioned container images for environment reproducibility, k3s for cluster scheduling, and Coder for self-service workspace lifecycle management.