Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges
Mark Harman, Peter O'Hearn, Shubho Sengupta
TL;DR
This work reframes automated LLM-based software testing around hardening and catching tests, introducing the Just-in-Time JiTTest paradigm to catch regressions and new functionality before production. It provides formal definitions for hardening/catching, distinguishes regression-catching from functionality-catching, and analyzes deployment strategies, including the benefits and risks of engineer involvement. The paper reports initial Meta results with ACH and mutation-guided testing, and presents a comprehensive research agenda, including the Catching JiTTest Challenge and oracle scavenging for non-executable information. Overall, the approach seeks to move beyond the regression-coverage paradigm toward verifiable, timely, and high-signal testing that can reveal latent bugs while reducing false positives. The practical impact lies in enabling more reliable, just-in-time test generation for large-scale LLM-assisted software development and maintenance.
Abstract
Despite decades of research and practice in automated software testing, several fundamental concepts remain ill-defined and under-explored, yet offer enormous potential real-world impact. We show that these concepts raise exciting new challenges in the context of Large Language Models for software test generation. More specifically, we formally define and investigate the properties of hardening and catching tests. A hardening test is one that seeks to protect against future regressions, while a catching test is one that catches such a regression or a fault in new functionality introduced by a code change. Hardening tests can be generated at any time and may become catching tests when a future regression is caught. We also define and motivate the Catching 'Just-in-Time' (JiTTest) Challenge, in which tests are generated 'just-in-time' to catch new faults before they land into production. We show that any solution to Catching JiTTest generation can also be repurposed to catch latent faults in legacy code. We enumerate possible outcomes for hardening and catching tests and JiTTests, and discuss open research problems, deployment options, and initial results from our work on automated LLM-based hardening at Meta. This paper was written to accompany the keynote by the authors at the ACM International Conference on the Foundations of Software Engineering (FSE) 2025. Author order is alphabetical. The corresponding author is Mark Harman.