Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Optimal Branching of Linear and Semidefinite Relaxations for Neural Network Robustness Certification

Brendon G. Anderson, Ziye Ma, Jingqi Li, Somayeh Sojoudi

TL;DR

This work enhances robustness certification of ReLU networks by introducing partitioned LP and SDP relaxations within a branch-and-bound framework. By partitioning the input uncertainty set and solving convex relaxations on each part, the authors prove both validity and tightening of the relaxations, and derive principled, worst-case-guided branching schemes for single-hidden-layer networks. They establish that optimizing the true LP relaxation error is NP-hard, motivating tractable worst-case partitioning rules, and derive a rank-1 gap based SDP partition strategy that favors uniform coordinate-wise partitions. Empirical results on MNIST, CIFAR-10, and Wisconsin Breast Cancer demonstrate substantial gains in certified robustness for both LP and SDP approaches, with the SDP method particularly advantageous in deeper networks and large-scale benchmarks, achieving performance competitive with state-of-the-art verifiers. The paper also extends LP branching to multi-layer settings, offering guidance on regime-specific applicability of branched LP vs branched SDP in practical certification tasks.

Abstract

In this paper, we study certifying the robustness of ReLU neural networks against adversarial input perturbations. To diminish the relaxation error suffered by the popular linear programming (LP) and semidefinite programming (SDP) certification methods, we take a branch-and-bound approach to propose partitioning the input uncertainty set and solving the relaxations on each part separately. We show that this approach reduces relaxation error, and that the error is eliminated entirely upon performing an LP relaxation with a partition intelligently designed to exploit the nature of the ReLU activations. To scale this approach to large networks, we consider using a coarser partition whereby the number of parts in the partition is reduced. We prove that computing such a coarse partition that directly minimizes the LP relaxation error is NP-hard. By instead minimizing the worst-case LP relaxation error, we develop a closed-form branching scheme in the single-hidden layer case. We extend the analysis to the SDP, where the feasible set geometry is exploited to design a branching scheme that minimizes the worst-case SDP relaxation error. Experiments on MNIST, CIFAR-10, and Wisconsin breast cancer diagnosis classifiers demonstrate significant increases in the percentages of test samples certified. By independently increasing the input size and the number of layers, we empirically illustrate under which regimes the branched LP and branched SDP are best applied. Finally, we extend our LP branching method into a multi-layer branching heuristic, which attains comparable performance to prior state-of-the-art heuristics on large-scale, deep neural network certification benchmarks.

Towards Optimal Branching of Linear and Semidefinite Relaxations for Neural Network Robustness Certification

TL;DR

This work enhances robustness certification of ReLU networks by introducing partitioned LP and SDP relaxations within a branch-and-bound framework. By partitioning the input uncertainty set and solving convex relaxations on each part, the authors prove both validity and tightening of the relaxations, and derive principled, worst-case-guided branching schemes for single-hidden-layer networks. They establish that optimizing the true LP relaxation error is NP-hard, motivating tractable worst-case partitioning rules, and derive a rank-1 gap based SDP partition strategy that favors uniform coordinate-wise partitions. Empirical results on MNIST, CIFAR-10, and Wisconsin Breast Cancer demonstrate substantial gains in certified robustness for both LP and SDP approaches, with the SDP method particularly advantageous in deeper networks and large-scale benchmarks, achieving performance competitive with state-of-the-art verifiers. The paper also extends LP branching to multi-layer settings, offering guidance on regime-specific applicability of branched LP vs branched SDP in practical certification tasks.

Abstract

In this paper, we study certifying the robustness of ReLU neural networks against adversarial input perturbations. To diminish the relaxation error suffered by the popular linear programming (LP) and semidefinite programming (SDP) certification methods, we take a branch-and-bound approach to propose partitioning the input uncertainty set and solving the relaxations on each part separately. We show that this approach reduces relaxation error, and that the error is eliminated entirely upon performing an LP relaxation with a partition intelligently designed to exploit the nature of the ReLU activations. To scale this approach to large networks, we consider using a coarser partition whereby the number of parts in the partition is reduced. We prove that computing such a coarse partition that directly minimizes the LP relaxation error is NP-hard. By instead minimizing the worst-case LP relaxation error, we develop a closed-form branching scheme in the single-hidden layer case. We extend the analysis to the SDP, where the feasible set geometry is exploited to design a branching scheme that minimizes the worst-case SDP relaxation error. Experiments on MNIST, CIFAR-10, and Wisconsin breast cancer diagnosis classifiers demonstrate significant increases in the percentages of test samples certified. By independently increasing the input size and the number of layers, we empirically illustrate under which regimes the branched LP and branched SDP are best applied. Finally, we extend our LP branching method into a multi-layer branching heuristic, which attains comparable performance to prior state-of-the-art heuristics on large-scale, deep neural network certification benchmarks.

Paper Structure

This paper contains 37 sections, 16 theorems, 163 equations, 8 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Branch-and-Bound Certification
  3. Contributions
  4. Outline
  5. Notations
  6. Problem Statement
  7. Description of the Network and Uncertainty
  8. The Robustness Certification Problem
  9. LP Relaxation of the Network Constraints
  10. SDP Relaxation of the Network Constraints
  11. Partitioned LP Relaxation
  12. Properties of Partitioned Relaxation
  13. Partitioning Gives Valid Relaxation
  14. Tightening of the Relaxation
  15. Motivating Partition
  16. ...and 22 more sections

Key Result

Proposition 3

Let $\{\mathcal{X}^{(j)} \subseteq \mathcal{X} : j\in\{1,2,\dots,p\}\}$ be a partition of $\mathcal{X}$. Then, it holds that

Figures (8)

  • Figure 1: The set $\hat{f}(\mathcal{X})$ is a convex outer approximation of the nonconvex set $f(\mathcal{X})$. If the outer approximation is safe, i.e., $\hat{f}(\mathcal{X})\subseteq \mathcal{S}$, then so is $f(\mathcal{X})$.
  • Figure 2: This scenario shows that if the convex outer approximation $\hat{f}(\mathcal{X})$ is too large, meaning the relaxation is too loose, then the convex approach fails to issue a certificate of robustness.
  • Figure 3: Relaxed ReLU constraint set $\mathcal{N}_\textup{LP}^{[k]}$ at a single neuron $i$ in layer $k$ of the network.
  • Figure 4: Partitioning based on row $w_i^\top$ of the weight matrix. This partition results in an exact ReLU constraint in coordinate $i$ over the two resulting input parts $\mathcal{X}_i^{(1)} = \{x\in\mathcal{X} : w_i^\top x \ge 0\}$ and $\mathcal{X}_i^{(2)}=\mathcal{X}\setminus\mathcal{X}_i^{(1)}$.
  • Figure 5: Geometry of the SDP relaxation in coordinate $i$ over part $j$ of the partition. The shaded region shows the feasible $X_i$ satisfying the input constraint raghunathan2018semidefinite.
  • ...and 3 more figures

Theorems & Definitions (21)

  • Definition 1: Partition
  • Remark 2
  • Proposition 3: Partitioned relaxation bound
  • Proposition 4: Improving the LP relaxation bound
  • Definition 5: Diameter
  • Proposition 6: Diameter bound
  • Proposition 7: Motivating partition
  • Theorem 8: Worst-case relaxation bound
  • Lemma 9: Two-part bound
  • Theorem 10: Worst-case optimal LP branching
  • ...and 11 more