What's DAT? Three Case Studies of Measuring Software Development Productivity at Meta With Diff Authoring Time

TL;DR

This paper introduces Diff Authoring Time (DAT), a time-based productivity metric measured via privacy-preserving telemetry integrated with version control, the IDE, and the OS. It validates DAT through multiple studies and demonstrates three industrial case studies—Typed Mocking in Hack (14% improvement), React auto-memoization (33% improvement), and code sharing (over 50% improvement, thousands of hours saved). By providing precise, diff-level attribution, DAT supports rigorous, experiment-driven evaluation of internal software development productivity at industry scale. The work highlights practical implications for prioritization, culture, and future research on long-standing software engineering questions, such as whether types improve developer efficiency, and outlines planned extensions to broaden coverage and tooling.

Abstract

This paper introduces Diff Authoring Time (DAT), a powerful, yet conceptually simple approach to measuring software development productivity that enables rigorous experimentation. DAT is a time based metric, which assess how long engineers take to develop changes, using a privacy-aware telemetry system integrated with version control, the IDE, and the OS. We validate DAT through observational studies, surveys, visualizations, and descriptive statistics. At Meta, DAT has powered experiments and case studies on more than 20 projects. Here, we highlight (1) an experiment on introducing mock types (a 14% DAT improvement), (2) the development of automatic memoization in the React compiler (33% improvement), and (3) an estimate of thousands of DAT hours saved annually through code sharing (> 50% improvement). DAT offers a precise, yet high-coverage measure for development productivity, aiding business decisions. It enhances development efficiency by aligning the internal development workflow with the experiment-driven culture of external product development. On the research front, DAT has enabled us to perform rigorous experimentation on long-standing software engineering questions such as "do types make development more efficient?"

Figures (7)

• Figure 1: Example illustration of the most relevant DAT-covered and uncovered activities for hypothetical diffs D123 and D987, and a comparison with other related metrics Code Gestation Time (CGT) and Time Spent by Diff (TSD).
• Figure 2: Explanation of the core DAT commit matching algorithm.
• Figure 3: 99th percentile winsorized average DAT for different algorithms, across a range of several months. The figure clearly shows the stability of DAT as a top-line metric.
• Figure 4: Human DAT estimates (solid) and actual DAT values for an earlier DAT version (dashed).
• Figure 5: A sample timeline of activities (raw telemetry, bottom row), and different DAT algorithms.