Automated Design of Linear Bounding Functions for Sigmoidal Nonlinearities in Neural Networks

TL;DR

This work tackles robustness verification of neural networks with sigmoidal activations, where complete verification is intractable and convex relaxations introduce imprecision. It introduces an automated tangent-point search guided by SMAC to tighten the linear bounds used in convex relaxation, thereby improving the global lower bound $g^*$ and certification rate. Across Sigmoid and Tanh networks on CIFAR-10 and MNIST, the method yields substantial improvements in average lower bounds and, in many cases, enables robustness certification that the baseline could not achieve. The approach demonstrates that automated configuration can systematically enhance bounds for nonpiecewise linear activations, with potential applicability to a broader class of activation functions and verification tasks.

Abstract

The ubiquity of deep learning algorithms in various applications has amplified the need for assuring their robustness against small input perturbations such as those occurring in adversarial attacks. Existing complete verification techniques offer provable guarantees for all robustness queries but struggle to scale beyond small neural networks. To overcome this computational intractability, incomplete verification methods often rely on convex relaxation to over-approximate the nonlinearities in neural networks. Progress in tighter approximations has been achieved for piecewise linear functions. However, robustness verification of neural networks for general activation functions (e.g., Sigmoid, Tanh) remains under-explored and poses new challenges. Typically, these networks are verified using convex relaxation techniques, which involve computing linear upper and lower bounds of the nonlinear activation functions. In this work, we propose a novel parameter search method to improve the quality of these linear approximations. Specifically, we show that using a simple search method, carefully adapted to the given verification problem through state-of-the-art algorithm configuration techniques, improves the average global lower bound by 25% on average over the current state of the art on several commonly used local robustness verification benchmarks.

Figures (3)

• Figure 1: Linear bounding rules for different cases of the Sigmoid activation function. The x-axis shows the pre-activation bounds, while the y-axis indicates the output of the activation function.
• Figure 2: Experimental results obtained for Sigmoid-based networks. Each dot represents a problem instance and the global lower bound, i.e., the value of $g^{\ast}$, for that instance achieved by the baseline approach (x-axis) vs our method (y-axis).
• Figure 3: Probability density functions of the values of $\hat{z}^{\ast}_j$ obtained by our method as well as vanilla $\mathrm{CROWN}$ for lower and upper bounding functions per activation layer. Remember that $\hat{z}^{\ast}_j$ determines the tangent point of the bounding function of a given node $j$.

Theorems & Definitions (2)

• definition thmcounterdefinition: Convex relaxation of activation functions
• definition thmcounterdefinition: Convex relaxation of neural networks