DrP: Meta's Efficient Investigations Platform at Scale

TL;DR

DrP addresses the rising complexity of investigations in large-scale, multi-domain systems by codifying manual playbooks into analyzers using a flexible SDK, and by deploying a scalable backend that executes these analyzers with robust backtesting, integration, and post-processing capabilities. It demonstrates substantial production impact at Meta, including reductions in MTTR and on-call toil across hundreds of teams and thousands of analyses per day. The work highlights key design choices—analyzer chaining, per-group binaries, and a community-driven library model—that enable scalable, reliable investigations while integrating with existing incident workflows. Looking forward, DrP explores Generative AI-based analysis to further automate and democratize incident response.

Abstract

Investigations are a significant step in the operational workflows for large scale systems across multiple domains such as services, data, AI/ML, mobile. Investigation processes followed by on-call engineers are often manual or rely on ad-hoc scripts. This leads to inefficient investigations resulting in increased time to mitigate and isolate failures/SLO violations. It also contributes to on-call toil and poor productivity leading to multiple hours/days spent in triaging/debugging incidents. In this paper, we present DrP, an end-to-end framework and system to automate investigations that reduces the mean time to resolve incidents (MTTR) and reduces on-call toil. DrP consists of an expressive and flexible SDK to author investigation playbooks in code (called analyzers), a scalable backend system to execute these automated playbooks, plug-ins to integrate playbooks into mainstream workflows such as alerts and incident management tools, and a post-processing system to take actions on investigations including mitigation steps. We have implemented and deployed DrP at large scale at Meta covering 300+ teams, 2000+ analyzers, across a large set of use cases across domains such as services, core infrastructure, AI/ML, hardware, mobile. DrP has been running in production for the past 5 years and executes 50K automated analyses per day. Overall, our results and experience show that DrP has been able to reduce average MTTR by 20 percent at large scale (with over 80 percent for some teams) and has significantly improved on-call productivity.

Figures (6)

• Figure 1: Rapid increase in number of investigation steps with increasing complexity. Complex systems typically require a lot more queries over multiple data sources to triage the incident.
• Figure 2: Example of a complex investigation - ML prediction debugging
• Figure 3: High-level Authoring Overview
• Figure 4: DrP declarative query API vs Raw SQL based querying
• Figure 5: Example of Analyzer Chaining