Validating Alerts in Cloud-Native Observability
Maria C. Borges, Julian Legler, Lucca Di Benedetto
TL;DR
This paper tackles the problem of designing and validating alerts in cloud-native systems, where threshold tuning and false positives can undermine reliability. It introduces an alerting extension to OXN, enabling online fault detection experiments by deploying a SUE, injecting faults, and collecting Prometheus alerts to measure detections. The demonstration applies two alert design strategies to the OpenTelemetry Astronomy Shop Demo, showing how window size and duration affect firing behavior. The approach offers practitioners a way to compare alert rules systematically at design time, reducing runtime misfires and guiding data-driven alert optimization; future work includes batch exploration and AI-assisted observability design.
Abstract
Observability and alerting form the backbone of modern reliability engineering. Alerts help teams catch faults early before they turn into production outages and serve as first clues for troubleshooting. However, designing effective alerts is challenging. They need to strike a fine balance between catching issues early and minimizing false alarms. On top of this, alerts often cover uncommon faults, so the code is rarely executed and therefore rarely checked. To address these challenges, several industry practitioners advocate for testing alerting code with the same rigor as application code. Still, there's a lack of tools that support such systematic design and validation of alerts. This paper introduces a new alerting extension for the observability experimentation tool OXN. It lets engineers experiment with alerts early during development. With OXN, engineers can now tune rules at design time and routinely validate the firing behavior of their alerts, avoiding future problems at runtime.