Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Just-in-Time Catching Test Generation at Meta

Matthew Becker, Yifei Chen, Nicholas Cochran, Pouyan Ghasemi, Abhishek Gulati, Mark Harman, Zachary Haluza, Mehrdad Honarkhah, Herve Robert, Jiacheng Liu, Weini Liu, Sreeja Thummala, Xiaoning Yang, Rui Xin, Sophie Zeng

TL;DR

This paper presents Just-in-Time catching test generation (Catching JiTTest) at Meta, designed to surface bugs in extremely large codebases before changes land. It introduces diff-aware workflows (dodgy diff and intent-aware) that leverage diff context and inferred intent to generate weak catching tests far more effectively than traditional hardening baselines. To manage noise, the authors deploy three assessors (LLM-based ensemble and RubFake rule-based) that significantly reduce human review workload and improve precision, achieving an engineer-experienced true positive rate of about $19.5\%$ with negligible false-positive drag. Real-world deployment shows that eight strong catches would have prevented serious production failures, underscoring the practical value and scalability of Just-in-Time catching in large-scale software engineering. The work paves the way for more targeted, low-friction testing in high-velocity environments and suggests avenues for improving diff understanding and assessor accuracy.

Abstract

We report on Just-in-Time catching test generation at Meta, designed to prevent bugs in large scale backend systems of hundreds of millions of line of code. Unlike traditional hardening tests, which pass at generation time, catching tests are meant to fail, surfacing bugs before code lands. The primary challenge is to reduce development drag from false positive test failures. Analyzing 22,126 generated tests, we show code-change-aware methods improve candidate catch generation 4x over hardening tests and 20x over coincidentally failing tests. To address false positives, we use rule-based and LLM-based assessors. These assessors reduce human review load by 70%. Inferential statistical analysis showed that human-accepted code changes are assessed to have significantly more false positives, while human-rejected changes have significantly more true positives. We reported 41 candidate catches to engineers; 8 were confirmed to be true positives, 4 of which would have led to serious failures had they remained uncaught. Overall, our results show that Just-in-Time catching is scalable, industrially applicable, and that it prevents serious failures from reaching production.

Just-in-Time Catching Test Generation at Meta

TL;DR

This paper presents Just-in-Time catching test generation (Catching JiTTest) at Meta, designed to surface bugs in extremely large codebases before changes land. It introduces diff-aware workflows (dodgy diff and intent-aware) that leverage diff context and inferred intent to generate weak catching tests far more effectively than traditional hardening baselines. To manage noise, the authors deploy three assessors (LLM-based ensemble and RubFake rule-based) that significantly reduce human review workload and improve precision, achieving an engineer-experienced true positive rate of about with negligible false-positive drag. Real-world deployment shows that eight strong catches would have prevented serious production failures, underscoring the practical value and scalability of Just-in-Time catching in large-scale software engineering. The work paves the way for more targeted, low-friction testing in high-velocity environments and suggests avenues for improving diff understanding and assessor accuracy.

Abstract

We report on Just-in-Time catching test generation at Meta, designed to prevent bugs in large scale backend systems of hundreds of millions of line of code. Unlike traditional hardening tests, which pass at generation time, catching tests are meant to fail, surfacing bugs before code lands. The primary challenge is to reduce development drag from false positive test failures. Analyzing 22,126 generated tests, we show code-change-aware methods improve candidate catch generation 4x over hardening tests and 20x over coincidentally failing tests. To address false positives, we use rule-based and LLM-based assessors. These assessors reduce human review load by 70%. Inferential statistical analysis showed that human-accepted code changes are assessed to have significantly more false positives, while human-rejected changes have significantly more true positives. We reported 41 candidate catches to engineers; 8 were confirmed to be true positives, 4 of which would have led to serious failures had they remained uncaught. Overall, our results show that Just-in-Time catching is scalable, industrially applicable, and that it prevents serious failures from reaching production.
Paper Structure (33 sections, 1 figure, 12 tables)

This paper contains 33 sections, 1 figure, 12 tables.

Table of Contents

  1. Introduction
  2. Baselines for Just in Time Catching
  3. Coincidental regression catches found during ACH hardening deployment
  4. Hardening test generation workflows as catching workflows
  5. Diff-Aware Workflows
  6. Intent aware workflow
  7. Intent-aware baseline for intent computation
  8. Dodgy diff: intent-unaware workflow
  9. Results of applying the workflows
  10. Results for co-incidental catches
  11. Results for hardening baseline
  12. Results for diff-aware weak catch generation
  13. Initial Evidence that significant real world bugs will be prevented by targeted catch
  14. Determining whether to reach out
  15. Reducing false positives and identifying likely true positives
  16. ...and 18 more sections

Figures (1)

  • Figure 1: Architecture of 'Dodgy diff' and Intent-Aware Workflows for generating Just-in-Time Catches.