Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BOOM and Babamul: a real-time, multi-survey, optical alert broker system operating at scale

Theophile Jegou du Laz, Michael W. Coughlin, Peter Bachant, Jacob E. Simones, Thomas Culino, Antoine Le Calloch, Sushant Sharma Chaudhary, Xander J. Hall, Tyler Barna, Daniel Warshofsky, Matthew Graham, Mansi M. Kasliwal, Ashish Mahabal, Joshua S. Bloom, Antonella Palmese, Frank J. Masci, Steven L. Groom, Richard Dekany, Reed L. Riddle, George Helou

TL;DR

The paper presents BOOM, a Rust-based, real-time alert broker capable of multi-survey, cross-match processing to meet LSST-scale data streams. It describes a multi-tier architecture using Kafka for input, Valkey for in-memory queuing, and MongoDB for a queryable, cross-matchable archive, with dedicated ingestion, enrichment, and filtering workers and an ONNX-based inference path. Key contributions include a scalable, batched processing framework, a no-code filter-builder, and integration pathways with SkyPortal and the upcoming Babamul public broker for LSST-era data. The results demonstrate substantial throughput gains, realistic hardware guidance, and successful cross-survey demonstrations (e.g., ZTF+DECam), underscoring BOOM’s readiness for Rubin-era operations.

Abstract

With the arrival of ever higher throughput wide-field surveys and a multitude of multi-messenger and multi-wavelength instruments to complement them, software capable of harnessing these associated data streams is urgently required. To meet these needs, a number of community supported alert brokers have been built, currently focused on processing of Zwicky Transient Facility (ZTF; $\sim 10^5$-$10^6$ alerts per night) with an eye towards Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST; $\sim 2 \times 10^7$ alerts per night). Building upon the system that successfully ran in production for ZTF's first seven years of operation, we introduce BOOM (Burst & Outburst Observations Monitor), an analysis framework focused on real-time, joint brokering of these alert streams. BOOM harnesses the performance of a Rust-based software stack relying on a non-relational MongoDB database combined with a Valkey in-memory processing queue and a Kafka cluster for message sharing. With this system, we demonstrate feature parity with the existing ZTF system with a throughput $\sim 8 \times$ higher. We describe the workflow that enables the real-time processing as well as the results with custom filters we have built to demonstrate the system's capabilities. In conclusion, we present the development roadmap for both BOOM and Babamul - the public-facing LSST alert broker built atop BOOM - as we begin the Rubin era.

BOOM and Babamul: a real-time, multi-survey, optical alert broker system operating at scale

TL;DR

The paper presents BOOM, a Rust-based, real-time alert broker capable of multi-survey, cross-match processing to meet LSST-scale data streams. It describes a multi-tier architecture using Kafka for input, Valkey for in-memory queuing, and MongoDB for a queryable, cross-matchable archive, with dedicated ingestion, enrichment, and filtering workers and an ONNX-based inference path. Key contributions include a scalable, batched processing framework, a no-code filter-builder, and integration pathways with SkyPortal and the upcoming Babamul public broker for LSST-era data. The results demonstrate substantial throughput gains, realistic hardware guidance, and successful cross-survey demonstrations (e.g., ZTF+DECam), underscoring BOOM’s readiness for Rubin-era operations.

Abstract

With the arrival of ever higher throughput wide-field surveys and a multitude of multi-messenger and multi-wavelength instruments to complement them, software capable of harnessing these associated data streams is urgently required. To meet these needs, a number of community supported alert brokers have been built, currently focused on processing of Zwicky Transient Facility (ZTF; - alerts per night) with an eye towards Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST; alerts per night). Building upon the system that successfully ran in production for ZTF's first seven years of operation, we introduce BOOM (Burst & Outburst Observations Monitor), an analysis framework focused on real-time, joint brokering of these alert streams. BOOM harnesses the performance of a Rust-based software stack relying on a non-relational MongoDB database combined with a Valkey in-memory processing queue and a Kafka cluster for message sharing. With this system, we demonstrate feature parity with the existing ZTF system with a throughput higher. We describe the workflow that enables the real-time processing as well as the results with custom filters we have built to demonstrate the system's capabilities. In conclusion, we present the development roadmap for both BOOM and Babamul - the public-facing LSST alert broker built atop BOOM - as we begin the Rubin era.

Paper Structure

This paper contains 18 sections, 5 figures.

Table of Contents

  1. Introduction
  2. BOOM Framework
  3. Input/Output through Apache Kafka
  4. Job scheduling with Valkey
  5. Spatial query-ready database with MongoDB
  6. Parallelized and Distributed Alert Processing
  7. Alert Ingestion Worker
  8. Enrichment Worker
  9. Filter Worker
  10. A RESTful API for archival searches
  11. Deployments, Science Validation and First Results
  12. Throughput Testing
  13. Hardware Requirements
  14. Deployment Approaches
  15. Integration with SkyPortal
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Flowchart for BOOM.
  • Figure 2: Decision tree workflow of each boom worker
  • Figure 3: Scalability testing results for BOOM and Kowalski. The dashed horizontal line represents the average alert production rate of the Rubin observatory: 10,000 alerts for every 30-second exposure, or $\sim 333$ alert/s.
  • Figure 4: Filter Builder
  • Figure 5: Alert photometry of SN 2025kwy, a young supernova candidate first detected by DECam (C202505201402422m202612) and later observed by ZTF (ZTF25aaqsuda).