Fuzzing Microservices: A Series of User Studies in Industry on Industrial Systems with EvoMaster

TL;DR

This study evaluates EvoMaster, a search-based white-box fuzzer for Web APIs, within large-scale, industrial microservice contexts. Through three user studies (2021 and 2023 at Meituan, plus 2024 with practitioners from five other companies), the authors assess usability, effectiveness in line coverage and fault detection, and integration into industrial pipelines, highlighting RPC-native fuzzing improvements while identifying persistent challenges in state reset, data scale, and test readability. The findings show EvoMaster can uncover faults and boost coverage in real industrial SUTs, and it demonstrates potential for CI-based automation, but the work also reveals significant usability and generalizability hurdles that the research community must address for broader adoption. Overall, the paper contributes a valuable industrial perspective on automated test generation for microservices and documents a practical roadmap for advancing white-box fuzzing in enterprise settings.

Abstract

With several microservice architectures comprising of thousands of web services, used to serve 630 million customers, companies like Meituan face several challenges in the verification and validation of their software. This paper reports on our experience of integrating EvoMaster (a search-based white-box fuzzer) in the testing processes at Meituan over almost 2 years. Two user studies were carried out in 2021 and in 2023 to evaluate two versions of EvoMaster, respectively, in tackling the test generation for industrial web services which are parts of a large e-commerce microservice system. The two user studies involve in total 321,131 lines of code from five APIs and 27 industrial participants at Meituan. Questionnaires and interviews were carried out in both user studies with employees at Meituan. The two user studies demonstrate clear advantages of EvoMaster (i.e., code coverage and fault detection) and the urgent need to have such a fuzzer in industrial microservices testing. To study how these results could generalize, a follow up user study was done in 2024 with five engineers in the five different companies. Our results show that, besides their clear usefulness, there are still many critical challenges that the research community needs to investigate to improve performance further.

Figures (9)

• Figure 1: An example of OpenAPI specification
• Figure 2: An example of the SUT with its connected services
• Figure 3: An overview of EvoMaster
• Figure 4: Answers provided by industrial partners about the difficulties on applying EvoMaster (QAs)
• Figure 5: Average covered targets (y-axis) with 30m (green line), 1h (red line) and 10h (blue line) throughout the search for industrial applications, reported at 5% intervals of the used budget allocated for the search (x-axis)