Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative AI in Live Operations: Evidence of Productivity Gains in Cybersecurity and Endpoint Management

James Bono, Justin Grana, Kleanthis Karakolios, Pruthvi Hanumanthapura Ramakrishna, Ankit Srivastava

TL;DR

The paper analyzes the productivity effects of generative AI tools in live security and IT operations by treating Microsoft Security Copilot adoption as the intervention and using causal inference methods on telemetry data. Across four metrics—alerts per incident, probability of incident reopening, DLP MTTC, and device policy conflict MTTR—it finds robust, statistically significant improvements (e.g., ~22.9% fewer alerts per incident, ~68% lower reopening probability, ~18.4% MTTC reduction for web-portal true positives, and ~54% MTTR reduction). The study leverages difference-in-differences, propensity-score matching, and two-way fixed effects to compare adopters and matched non-adopters, while repeatedly testing robustness to data filtering, classification source, and window definitions. Despite weaker causal claims due to observational data, the results suggest substantial real-world productivity gains from GAI in security operations and endpoint management, highlighting practical implications for the deployment of Copilot-like tools. Limitations include potential selection bias and unobserved confounders, motivating future randomized control or natural experiments to establish stronger causal evidence.

Abstract

We measure the association between generative AI (GAI) tool adoption and four metrics spanning security operations, information protection, and endpoint management: 1) number of security alerts per incident, 2) probability of security incident reopenings, 3) time to classify a data loss prevention alert, and 4) time to resolve device policy conflicts. We find that GAI is associated with robust and statistically and practically significant improvements in the four metrics. Although unobserved confounders inhibit causal identification, these results are among the first to use observational data from live operations to investigate the relationship between GAI adoption and security operations, data loss prevention, and device policy management.

Generative AI in Live Operations: Evidence of Productivity Gains in Cybersecurity and Endpoint Management

TL;DR

The paper analyzes the productivity effects of generative AI tools in live security and IT operations by treating Microsoft Security Copilot adoption as the intervention and using causal inference methods on telemetry data. Across four metrics—alerts per incident, probability of incident reopening, DLP MTTC, and device policy conflict MTTR—it finds robust, statistically significant improvements (e.g., ~22.9% fewer alerts per incident, ~68% lower reopening probability, ~18.4% MTTC reduction for web-portal true positives, and ~54% MTTR reduction). The study leverages difference-in-differences, propensity-score matching, and two-way fixed effects to compare adopters and matched non-adopters, while repeatedly testing robustness to data filtering, classification source, and window definitions. Despite weaker causal claims due to observational data, the results suggest substantial real-world productivity gains from GAI in security operations and endpoint management, highlighting practical implications for the deployment of Copilot-like tools. Limitations include potential selection bias and unobserved confounders, motivating future randomized control or natural experiments to establish stronger causal evidence.

Abstract

We measure the association between generative AI (GAI) tool adoption and four metrics spanning security operations, information protection, and endpoint management: 1) number of security alerts per incident, 2) probability of security incident reopenings, 3) time to classify a data loss prevention alert, and 4) time to resolve device policy conflicts. We find that GAI is associated with robust and statistically and practically significant improvements in the four metrics. Although unobserved confounders inhibit causal identification, these results are among the first to use observational data from live operations to investigate the relationship between GAI adoption and security operations, data loss prevention, and device policy management.

Paper Structure

This paper contains 23 sections, 4 equations, 2 figures, 9 tables.

Table of Contents

  1. Introduction and Background
  2. Related Work
  3. Domain Background
  4. What is Security Event Management?
  5. What is Device Policy Management?
  6. What is Copilot?
  7. Security, Data Protection, and Device Management Productivity Metrics
  8. Alerts per Incident and Security Incident Reopenings
  9. Data and Method
  10. Results
  11. Robustness
  12. Data Loss Prevention Alerts
  13. Data and Method
  14. Data
  15. Method
  16. ...and 8 more sections

Figures (2)

  • Figure 1: An example of incident summarization for a Business Email Compromise Incident
  • Figure 2: Empirical Distribution of MTTC