Grey-Box Fuzzing in Constrained Ultra-Large Systems: Lessons for SE Community

TL;DR

The paper addresses the challenge of fuzz testing ultra-large FinTech microservices under restricted production access and rigid test environments. It introduces SandBoxFuzz, a scalable grey-box fuzzing framework that uses aspect-oriented programming and runtime reflection to mine input specifications and generate valid inputs at runtime, avoiding reliance on invasive coverage agents. In real-world deployments at Ant Group, SandBoxFuzz achieves a 7.5% improvement in branch coverage and uncovers 1,850 additional exceptions while reducing setup time from hours to minutes, demonstrating strong practical utility. The work also contrasts with FatFuzz and highlights open-source deployment as a practical aid for researchers and practitioners working with large-scale, constrained industrial systems.

Abstract

Testing ultra-large microservices-based FinTech systems presents significant challenges, including restricted access to production environments, complex dependencies, and stringent security constraints. We propose SandBoxFuzz, a scalable grey-box fuzzing technique that addresses these limitations by leveraging aspect-oriented programming and runtime reflection to enable dynamic specification mining, generating targeted inputs for constrained environments. SandBoxFuzz also introduces a log-based coverage mechanism, seamlessly integrated into the build pipeline, eliminating the need for runtime coverage agents that are often infeasible in industrial settings. SandBoxFuzz has been successfully deployed to Ant Group's production line and, compared to an initial solution built on a state-of-the-art fuzzing framework, it demonstrates superior performance in their microservices software. SandBoxFuzz achieves a 7.5% increase in branch coverage, identifies 1,850 additional exceptions, and reduces setup time from hours to minutes, highlighting its effectiveness and practical utility in a real-world industrial environment. By open-sourcing SandBoxFuzz, we provide a practical and effective tool for researchers and practitioners to test large-scale microservices systems.

Figures (9)

• Figure 1: An example of microservices service calls.
• Figure 2: Main features of iTest test framework.
• Figure 3: An example of instrumenting print statements.
• Figure 4: Overview of SandBoxFuzz's architecture and workflow.
• Figure 5: A code example of sandbox interception.