Parameter Stress Analysis in Reinforcement Learning: Applying Synaptic Filtering to Policy Networks

TL;DR

The results highlight the presence of antifragile parameters that enhance policy performance under stress, demonstrating the potential of targeted filtering techniques to improve RL policy adaptability.

Abstract

This paper explores reinforcement learning (RL) policy robustness by systematically analyzing network parameters under internal and external stresses. \textcolor{black}{We apply synaptic filtering methods using high-pass, low-pass, and pulse-wave filters from} \citep{pravin2024fragility}, as an internal stress by selectively perturbing parameters, while adversarial attacks apply external stress through modified agent observations. This dual approach enables the classification of parameters as \textit{fragile}, \textit{robust}, or \textit{antifragile}, based on their influence on policy performance in clean and adversarial settings. Parameter scores are defined to quantify these characteristics, and the framework is validated on proximal policy optimization (PPO)-trained agents in Mujoco continuous control environments. The results highlight the presence of antifragile parameters that enhance policy performance under stress, demonstrating the potential of targeted filtering techniques to improve RL policy adaptability. These insights provide a foundation for future advancements in the design of robust and antifragile RL systems.

Figures (5)

• Figure 1: Framework for training and evaluating robust and antifragile policies in RL.
• Figure 2: Global parameter filtering statics for mujoco environment
• Figure 3: Impact of FGSM, BIM, and PGD Attacks on Policy Performance on Mujoco Environments.
• Figure 4: The left column shows parameter scores ($S_{\alpha_i}$) for clean environments. The middle column shows parameter scores ($S^{\epsilon_k}$) under FGSM adversarial perturbations with $\epsilon = 2.0$, while the right column depicts the difference in parameter scores ($\Delta S^{\epsilon_k}_{\alpha_i}$).
• Figure 5: Heatmaps of parameter scores under FGSM adversarial attack across environments, showing synaptic filtering methods (High-Pass, Low-Pass, Pulse Wave). The x-axis represents filtering thresholds $\alpha$, and the y-axis denotes stress magnitudes $\epsilon$. Red indicates antifragility, blue indicates fragility.