Policy Testing with MDPFuzz (Replicability Study)

TL;DR

The paper critically re-examines MDPFuzz through a dual lens of reproduction and replication. It identifies significant implementation bugs and algorithmic discrepancies in the original release, then re-implements the method faithfully and extends evaluation with three new environments and a random-testing baseline. Across both studies, the authors find that the coverage-guided component of MDPFuzz offers little consistent fault-detection advantage over a simpler Fuzzer or Random Testing, challenging prior conclusions. The work highlights replication as essential in RL-based policy testing and provides reusable artifacts to catalyze further research and standardization in evaluation practices.

Abstract

In recent years, following tremendous achievements in Reinforcement Learning, a great deal of interest has been devoted to ML models for sequential decision-making. Together with these scientific breakthroughs/advances, research has been conducted to develop automated functional testing methods for finding faults in black-box Markov decision processes. Pang et al. (ISSTA 2022) presented a black-box fuzz testing framework called MDPFuzz. The method consists of a fuzzer whose main feature is to use Gaussian Mixture Models (GMMs) to compute coverage of the test inputs as the likelihood to have already observed their results. This guidance through coverage evaluation aims at favoring novelty during testing and fault discovery in the decision model. Pang et al. evaluated their work with four use cases, by comparing the number of failures found after twelve-hour testing campaigns with or without the guidance of the GMMs (ablation study). In this paper, we verify some of the key findings of the original paper and explore the limits of MDPFuzz through reproduction and replication. We re-implemented the proposed methodology and evaluated our replication in a large-scale study that extends the original four use cases with three new ones. Furthermore, we compare MDPFuzz and its ablated counterpart with a random testing baseline. We also assess the effectiveness of coverage guidance for different parameters, something that has not been done in the original evaluation. Despite this parameter analysis and unlike Pang et al.'s original conclusions, we find that in most cases, the aforementioned ablated Fuzzer outperforms MDPFuzz, and conclude that the coverage model proposed does not lead to finding more faults.

Figures (7)

• Figure 1: Summary of the reproduction results as the number of faults found over time. The lines show the median results over 3 executions, and the shaded areas draw the interquartile ranges (IQRs). The results of the threats to validity studied are plotted with dashed and dotted lines. The timestamps indicate the time by which Fuzzer-O tests as many states as MDPFuzz-O.
• Figure 2: Evolution of the number of faults found with the replicate (Fuzzer-R and MDPFuzz-R) and Random Testing (RT). The lines show the median results over 5 executions (3 for CARLA), with their relative IQRs as shaded areas.
• Figure 3: Running time of the replication study. We report the total, testing ($\pi$-environment interactions) and coverage times of the three methods for each use case. The bars show the median values of 5 executions (3 for CARLA), in minutes.
• Figure 4: Fault distribution of $\pi$ for the Lunar Lander use case, where the inputs describe the initial force applied to the agent. The horizontal and vertical axes show the components of the forces in the x and y directions, respectively. The input space ($[-1000, 1000]^2$) has been evenly sampled with $4.10^6$ points.
• Figure 5: TSNE projections of the faults found by the three methods in the replication study (i.e., Fuzzer-R, MDPFuzz-R and RT).