Compendium Manager: a tool for coordination of workflow management instances for bulk data processing in Python

TL;DR

The paper addresses the challenge of coordinating hundreds to thousands of independent bioinformatics pipelines across multiple projects, a task not fully supported by existing workflow managers. It proposes Compendium Manager, a lightweight Python CLI that uses a SQLite database to store metadata, monitor progress, and consolidate results, while orchestrating Snakemake pipelines at scale. The authors demonstrate the approach in the context of the Human Microbiome Compendium, detailing commands for loading metadata, provisioning environments, and automating project queuing and evaluation. The work offers a practical, open-source framework for scalable meta-analyses of public sequencing data and reduces manual intervention, paving the way for reproducible, comprehensive compendia.

Abstract

Compendium Manager is a command-line tool written in Python to automate the provisioning, launch, and evaluation of bioinformatics pipelines. Although workflow management tools such as Snakemake and Nextflow enable users to automate the processing of samples within a single sequencing project, integrating many datasets in bulk requires launching and monitoring hundreds or thousands of pipelines. We present the Compendium Manager, a lightweight command-line tool to enable launching and monitoring analysis pipelines at scale. The tool can gauge progress through a list of projects, load results into a shared database, and record detailed processing metrics for later evaluation and reproducibility.

Figures (1)

• Figure 1: Workflow illustration. A flow chart indicating the processes followed at the prompting of a user for an example amplicon sequencing workflow. The green square indicates user interaction. Yellow squares are data sources such as databases or APIs. The orange boxes indicate steps launched and monitored by the Compendium Manager tool. Blue boxes indicate steps that are automated by Snakemake.