Fuzzy Logic Guided Reward Function Variation: An Oracle for Testing Reinforcement Learning Programs
Shiyu Zhang, Haoyang Song, Qixin Wang, Yu Pei
TL;DR
The paper tackles the oracle problem in reinforcement learning (RL) testing by introducing a fuzzy-logic-based automated oracle that leverages RL-specific heuristics to assess learning progress. It defines an intended policy $\pi^*$ and a fuzzy reward function to compute a policy-compliance-value $\mu_e$, tracking its trend over $E$ training epochs and using TrendAnalysis to detect abnormal behavior after convergence. The approach is evaluated against a human oracle using Stable Baselines3 APIs across two environments (FrozenLake and MountainCarContinuous) and three algorithms (DQN, A2C, PPO), demonstrating superior performance in more complex settings and competitive results in simpler ones. Key contributions include the formalization of CalcPolicyComplianceValueTimeSeries and TrendAnalysis, a practical parameter study (e.g., $I$, $E$, $n$, $\Delta$, $\theta_{orcl}$), and an empirical comparison that highlights the method’s potential for scalable automated RL testing. The findings suggest that fuzzy-logic-based oracles can enhance reliability and efficiency in RL program testing, particularly for complex environments, while indicating avenues for further improvement and integration with other testing paradigms.
Abstract
Reinforcement Learning (RL) has gained significant attention across various domains. However, the increasing complexity of RL programs presents testing challenges, particularly the oracle problem: defining the correctness of the RL program. Conventional human oracles struggle to cope with the complexity, leading to inefficiencies and potential unreliability in RL testing. To alleviate this problem, we propose an automated oracle approach that leverages RL properties using fuzzy logic. Our oracle quantifies an agent's behavioral compliance with reward policies and analyzes its trend over training episodes. It labels an RL program as "Buggy" if the compliance trend violates expectations derived from RL characteristics. We evaluate our oracle on RL programs with varying complexities and compare it with human oracles. Results show that while human oracles perform well in simpler testing scenarios, our fuzzy oracle demonstrates superior performance in complex environments. The proposed approach shows promise in addressing the oracle problem for RL testing, particularly in complex cases where manual testing falls short. It offers a potential solution to improve the efficiency, reliability, and scalability of RL program testing. This research takes a step towards automated testing of RL programs and highlights the potential of fuzzy logic-based oracles in tackling the oracle problem.