Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Applying Large-Scale Distributed Computing to Structural Bioinformatics -- Bridging Legacy HPC Clusters With Big Data Technologies Using kafka-slurm-agent

Pawel Rubach

TL;DR

The paper addresses scaling knot-detection computations for AlphaFold-scale data by bridging legacy Slurm HPC clusters with modern distributed computing. It introduces the Kafka Slurm Agent (KSA), a Python-based, Kafka-backed orchestration framework comprising four agents and a REST API to submit, monitor, and aggregate results across clusters and workstations. Its contributions include a detailed architecture, implementation guidance, and demonstration on AlphaKnot 2.0 with large-scale AlphaFold-derived structures, supported by a PyPI package and demo templates. The work enables Big Data-scale structural bioinformatics by reducing admin overhead and extending compute reach across heterogeneous resources, signaling practical impact for large-scale protein topology analyses.

Abstract

This paper presents the Kafka Slurm Agent (KSA), an open source (Apache 2.0 license) distributed computing and stream processing engine designed to help researchers distribute Python-based computational tasks across multiple Slurm-managed HPC clusters and workstations. Written entirely in Python, this extensible framework utilizes an Apache Kafka broker for asynchronous communication between its components. It is intended for non-expert users and does not require administrative privileges or additional libraries to run on Slurm. The framework's development was driven by the introduction of the AlphaFold protein structure prediction model, specifically, it was first created to facilitate the detection of knots in protein chains within structures predicted by AlphaFold. KSA has since been applied to several structural bioinformatics research projects, among others, leading to the discovery of new knotted proteins with previously unknown knot types. These knotted structures are now part of the AlphaKnot 2.0 web server and database, where KSA is applied to manage the knot detection process for user-uploaded structures.

Applying Large-Scale Distributed Computing to Structural Bioinformatics -- Bridging Legacy HPC Clusters With Big Data Technologies Using kafka-slurm-agent

TL;DR

The paper addresses scaling knot-detection computations for AlphaFold-scale data by bridging legacy Slurm HPC clusters with modern distributed computing. It introduces the Kafka Slurm Agent (KSA), a Python-based, Kafka-backed orchestration framework comprising four agents and a REST API to submit, monitor, and aggregate results across clusters and workstations. Its contributions include a detailed architecture, implementation guidance, and demonstration on AlphaKnot 2.0 with large-scale AlphaFold-derived structures, supported by a PyPI package and demo templates. The work enables Big Data-scale structural bioinformatics by reducing admin overhead and extending compute reach across heterogeneous resources, signaling practical impact for large-scale protein topology analyses.

Abstract

This paper presents the Kafka Slurm Agent (KSA), an open source (Apache 2.0 license) distributed computing and stream processing engine designed to help researchers distribute Python-based computational tasks across multiple Slurm-managed HPC clusters and workstations. Written entirely in Python, this extensible framework utilizes an Apache Kafka broker for asynchronous communication between its components. It is intended for non-expert users and does not require administrative privileges or additional libraries to run on Slurm. The framework's development was driven by the introduction of the AlphaFold protein structure prediction model, specifically, it was first created to facilitate the detection of knots in protein chains within structures predicted by AlphaFold. KSA has since been applied to several structural bioinformatics research projects, among others, leading to the discovery of new knotted proteins with previously unknown knot types. These knotted structures are now part of the AlphaKnot 2.0 web server and database, where KSA is applied to manage the knot detection process for user-uploaded structures.

Paper Structure

This paper contains 8 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Motivation
  3. Architecture
  4. Applications in Structural Bioinformatics
  5. Implementation
  6. Performance
  7. Discussion and other solutions
  8. Conclusions

Figures (3)

  • Figure 1: Kafka-slurm-agent's architecture showing components, and message flow between them.
  • Figure 2: AlphaKnot 2.0 showing the "von Willebrand factor A domain-containing protein 5A" from the human proteome, UniProt ID: O00534, with the main knot of type $6_{3}$, https://alphaknot.cent.uw.edu.pl/view/O00534/
  • Figure 3: UML class diagram showing the basic ClusterComputing class and its methods along with an example of a user-defined MatrixComputing class that implements the script that will be run by KSA.