Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GraphWeaver: Billion-Scale Cybersecurity Incident Correlation

Scott Freitas, Amir Gharib

TL;DR

GraphWeaver tackles the problem of scaling incident correlation to billions of alerts in large enterprises by shifting to a geo-distributed, graph-based framework integrated with Microsoft Defender XDR. It introduces a PySpark-driven pipeline, threat-intelligence integration, and a minimum spanning tree-based compression to reduce storage and computation, complemented by a human-in-the-loop feedback loop for continual refinement. The approach achieves a 99% true-correlation rate and a 7% reduction in singleton incidents, plus a 7.4x compression in correlations, validating its practical impact on SOC efficiency and scalability. Deployed worldwide, GraphWeaver demonstrates the practicality of industry-scale graph correlation, while outlining open research directions in verification at scale, knowledge-graph–assisted decision making, and advanced gap detection to guide future work.

Abstract

In the dynamic landscape of large enterprise cybersecurity, accurately and efficiently correlating billions of security alerts into comprehensive incidents is a substantial challenge. Traditional correlation techniques often struggle with maintenance, scaling, and adapting to emerging threats and novel sources of telemetry. We introduce GraphWeaver, an industry-scale framework that shifts the traditional incident correlation process to a data-optimized, geo-distributed graph based approach. GraphWeaver introduces a suite of innovations tailored to handle the complexities of correlating billions of shared evidence alerts across hundreds of thousands of enterprises. Key among these innovations are a geo-distributed database and PySpark analytics engine for large-scale data processing, a minimum spanning tree algorithm to optimize correlation storage, integration of security domain knowledge and threat intelligence, and a human-in-the-loop feedback system to continuously refine key correlation processes and parameters. GraphWeaver is integrated into the Microsoft Defender XDR product and deployed worldwide, handling billions of correlations with a 99% accuracy rate, as confirmed by customer feedback and extensive investigations by security experts. This integration has not only maintained high correlation accuracy but reduces traditional correlation storage requirements by 7.4x. We provide an in-depth overview of the key design and operational features of GraphWeaver, setting a precedent as the first cybersecurity company to openly discuss these critical capabilities at this level of depth.

GraphWeaver: Billion-Scale Cybersecurity Incident Correlation

TL;DR

GraphWeaver tackles the problem of scaling incident correlation to billions of alerts in large enterprises by shifting to a geo-distributed, graph-based framework integrated with Microsoft Defender XDR. It introduces a PySpark-driven pipeline, threat-intelligence integration, and a minimum spanning tree-based compression to reduce storage and computation, complemented by a human-in-the-loop feedback loop for continual refinement. The approach achieves a 99% true-correlation rate and a 7% reduction in singleton incidents, plus a 7.4x compression in correlations, validating its practical impact on SOC efficiency and scalability. Deployed worldwide, GraphWeaver demonstrates the practicality of industry-scale graph correlation, while outlining open research directions in verification at scale, knowledge-graph–assisted decision making, and advanced gap detection to guide future work.

Abstract

In the dynamic landscape of large enterprise cybersecurity, accurately and efficiently correlating billions of security alerts into comprehensive incidents is a substantial challenge. Traditional correlation techniques often struggle with maintenance, scaling, and adapting to emerging threats and novel sources of telemetry. We introduce GraphWeaver, an industry-scale framework that shifts the traditional incident correlation process to a data-optimized, geo-distributed graph based approach. GraphWeaver introduces a suite of innovations tailored to handle the complexities of correlating billions of shared evidence alerts across hundreds of thousands of enterprises. Key among these innovations are a geo-distributed database and PySpark analytics engine for large-scale data processing, a minimum spanning tree algorithm to optimize correlation storage, integration of security domain knowledge and threat intelligence, and a human-in-the-loop feedback system to continuously refine key correlation processes and parameters. GraphWeaver is integrated into the Microsoft Defender XDR product and deployed worldwide, handling billions of correlations with a 99% accuracy rate, as confirmed by customer feedback and extensive investigations by security experts. This integration has not only maintained high correlation accuracy but reduces traditional correlation storage requirements by 7.4x. We provide an in-depth overview of the key design and operational features of GraphWeaver, setting a precedent as the first cybersecurity company to openly discuss these critical capabilities at this level of depth.
Paper Structure (15 sections, 1 theorem, 1 figure, 2 tables, 1 algorithm)

This paper contains 15 sections, 1 theorem, 1 figure, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Network Intrusion Detection Systems
  5. Security Information & Event Management
  6. Extended Detection and Response
  7. Problem Formulation
  8. GraphWeaver Architecture
  9. Scalable Alert Correlation
  10. Incident Graph Mining
  11. Parameter Optimization & Gap Discovery
  12. Computational Complexity
  13. Deployment and Impact
  14. Open Research Directions
  15. Conclusion

Key Result

Lemma 1

The time and space complexity of GraphWeaver is $O(s + t + c \log c)$ and $O(sm + tm + cm)$, respectively.

Figures (1)

  • Figure 1: Overview of the GraphWeaver architecture from the perspective of a single geographic region. Alert telemetry is retrieved from an ADLS system and processed into two PySpark dataframes: one containing the latest 35 minutes of data, and the other the last 2 days. These dataframes are then correlated based on shared entities between pairs of alerts. Subsequently, threat intelligence and security domain knowledge are applied to filter out invalid correlations. The refined correlations are then centralized to the main node and converted into a graph, enabling a minimum spanning tree algorithm to remove redundant incident correlations. Finally, a human-in-the-loop feedback system is employed, where threat researchers review correlation reports to optimize parameters, review assumptions, and pinpoint potential correlation gaps.

Theorems & Definitions (1)

  • Lemma 1