Enhancing Testing at Meta with Rich-State Simulated Populations

TL;DR

This work demonstrates the deployment and impact of Rich-State Simulated Populations at Meta for both automated and manual testing of Facebook, Messenger, and Instagram on iOS and Android. By treating test users as first-class agents that accrue state through simulated interactions, the approach yields substantial gains in code and endpoint coverage and a marked increase in fault revelation, compared to traditional Empty State testing. The study introduces the Populations Manager and two workflows (Sapienz automated testing and Test Universe manual testing) and provides empirical evidence from shadow deployments and Alpha builds. It also documents the uptake of the Test Universe by tens of thousands of engineers, emphasizing privacy-safe, isolated simulation via WW and the evolving nature of test populations. The results suggest rich-state testing as a practical, scalable enhancement for system-level testing of highly interactive platforms, with clear avenues for future work including state-aware algorithms and reproducibility strategies.

Abstract

This paper reports the results of the deployment of Rich-State Simulated Populations at Meta for both automated and manual testing. We use simulated users (aka test users) to mimic user interactions and acquire state in much the same way that real user accounts acquire state. For automated testing, we present empirical results from deployment on the Facebook, Messenger, and Instagram apps for iOS and Android Platforms. These apps consist of tens of millions of lines of code, communicating with hundreds of millions of lines of backend code, and are used by over 2 billion people every day. Our results reveal that rich state increases average code coverage by 38\%, and endpoint coverage by 61\%. More importantly, it also yields an average increase of 115\% in the faults found by automated testing. The rich-state test user populations are also deployed in a (continually evolving) Test Universe; a web-enabled simulation platform for privacy-safe manual testing, which has been used by over 21,000 Meta engineers since its deployment in November 2022.

Figures (10)

• Figure 1: The Populations Manager workflow. This is how Rich State populations are created and evolved. For Sapienz, we only evolve once on initial creation, and use the 1st generation only. For the Test Universe, we evolve once per day, so we are currently on the Nth Population where N is the number of days since the population has been created.
• Figure 2: Populations Manager architecture. The server queries the configuration database (1) and receives a list of current populations and their settings (2). For each population, the server maintains a runner (3), that is responsible for monitoring and scheduling new jobs to create additional test users (4).
• Figure 3: Sizes of test user populations deployed at Meta in the Populations Manager framework.
• Figure 4: Total number of test users in the Test Universe between November 2022 and August 2023.
• Figure 5: Cumulative number of Meta employees who claimed a test user from the Test Universe between November 2022 and August 2023. As can be seen, there has been a steady rise in adoption over the period of deployment, to the point where over 21,000 employees have claimed a test user.