Why the Maximum Second Derivative of Activations Matters for Adversarial Robustness

Abstract

This work investigates the critical role of activation function curvature -- quantified by the maximum second derivative $\max|σ''|$ -- in adversarial robustness. Using the Recursive Curvature-Tunable Activation Family (RCT-AF), which enables precise control over curvature through parameters $α$ and $β$, we systematically analyze this relationship. Our study reveals a fundamental trade-off: insufficient curvature limits model expressivity, while excessive curvature amplifies the normalized Hessian diagonal norm of the loss, leading to sharper minima that hinder robust generalization. This results in a non-monotonic relationship where optimal adversarial robustness consistently occurs when $\max|σ''|$ falls within 4 to 10, a finding that holds across diverse network architectures, datasets, and adversarial training methods. We provide theoretical insights into how activation curvature affects the diagonal elements of the hessian matrix of the loss, and experimentally demonstrate that the normalized Hessian diagonal norm exhibits a U-shaped dependence on $\max|σ''|$, with its minimum within the optimal robustness range, thereby validating the proposed mechanism.

Figures (7)

• Figure 1: Comparison of RCT-AF activation functions with tunable rectification strength via $\alpha$ parameter ($\alpha = 1, 5, 10, 15, 20$). (a) $\beta = 0$: Baseline form. (b) $\beta = 1$: First-order variant. (c) $\beta = 2$: Second-order variant. Across all configurations, increasing $\alpha$ systematically strengthens the asymmetric treatment of negative versus positive inputs, enabling precise control over the activation function's rectification characteristics.
• Figure 2: Second derivative analysis of symmetric RCT-AF activation functions for $\alpha \in \{1, 5, 10, 15, 20\}$. (a) $\beta = 0$: $\sigma"_{\beta=0}(x)$ yields a single extremum at $x=0$ with value $\alpha/4$. (b) $\beta = 1$: $\sigma"_{\beta=1}(x)$ produces three critical points: a maximum at $x=0$ ($\alpha/2$) and two symmetric minima. (c) $\beta = 2$: $\sigma"_{\beta=2}(x)$ exhibits five critical points: a central maximum at $x=0$ ($\alpha$) and four symmetric extremal points. All functions are even and symmetric about the $y$-axis.
• Figure 3: Robustness analysis of RCT-AF activations under DAJAT adversarial training on CIFAR-10 with ResNet-18. (a) Robust accuracy vs. $\alpha$ for $\beta=0,1,2$, with $\alpha$ varying from 1 to 50 in steps of 1. The curves differ significantly across $\beta$ values, indicating that $\alpha$ alone does not determine robustness. (b) Robust accuracy vs. $\max_x\sigma"(x)$ for the same models, where $\max_x\sigma"(x) = \alpha/4$ for $\beta=0$, $\alpha/2$ for $\beta=1$, and $\alpha$ for $\beta=2$. When $\max_x\sigma"(x) < 15$, the curves nearly overlap, showing that maximum curvature, not $\beta$, primarily governs robustness.
• Figure 4: Adversarial robustness vs. maximum second derivative $\max|\sigma"|$ for ResNet-18 and WideResNet-28-10 trained with TRADES on CIFAR-10 using RCT-AF ($\beta=1$). Both architectures exhibit the same inverted-U relationship, with robustness peaking when $\max|\sigma"|$ lies between 4 and 10. The wider network (WideResNet-28-10) achieves higher absolute robust accuracy but follows an identical dependence on activation curvature, demonstrating that the identified optimal curvature range is architecture-agnostic.
• Figure 5: Comparison of adversarial robustness vs. $\max|\sigma"|$ across three adversarial training methods—DAJAT, DKL, and TRADES—on CIFAR-10 with ResNet-18 and RCT-AF ($\beta=1$). All methods exhibit the same qualitative behavior: robustness improves with curvature up to an optimum ($\max|\sigma"| \approx 4\text{--}10$) and declines thereafter.