Certified Robustness for Deep Equilibrium Models via Serialized Random Smoothing

TL;DR

This study reveals that simply applying randomized smoothing to certify DEQs provides certified robustness generalized to large-scale datasets but incurs extremely expensive computation costs, and proposes a novel Serialized Randomized Smoothing (SRS) approach that leverages historical information to reduce computational redundancy.

Abstract

Implicit models such as Deep Equilibrium Models (DEQs) have emerged as promising alternative approaches for building deep neural networks. Their certified robustness has gained increasing research attention due to security concerns. Existing certified defenses for DEQs employing deterministic certification methods such as interval bound propagation and Lipschitz-bounds can not certify on large-scale datasets. Besides, they are also restricted to specific forms of DEQs. In this paper, we provide the first randomized smoothing certified defense for DEQs to solve these limitations. Our study reveals that simply applying randomized smoothing to certify DEQs provides certified robustness generalized to large-scale datasets but incurs extremely expensive computation costs. To reduce computational redundancy, we propose a novel Serialized Randomized Smoothing (SRS) approach that leverages historical information. Additionally, we derive a new certified radius estimation for SRS to theoretically ensure the correctness of our algorithm. Extensive experiments and ablation studies on image recognition demonstrate that our algorithm can significantly accelerate the certification of DEQs by up to 7x almost without sacrificing the certified accuracy. Our code is available at https://github.com/WeizhiGao/Serialized-Randomized-Smoothing.

Figures (12)

• Figure 1: Illustrations of the standard DEQ and our SRS-DEQ. The representation for each sample goes through $D$ layers in standard DEQ. Our SRS-DEQ uses the previous representation as the initialization and converges to the fixed point with a few layers ($S\ll D$). After get all the predictions, SRS-DEQ makes use of correlation-eliminated certification to estimate the certified radius.
• Figure 2: RRD histogram with MDEQ-LARGE with 20 bins.
• Figure 3: Gap histogram of MDEQ-LARGE and $\overline{p_m}$ histogram of MDEQ-LARGE with 10 bins.
• Figure 4: RRD histogram with MDEQ-SMALL models. There are 10 bins in each histogram.
• Figure 5: The illustration of our correlation-eliminated certification. If we input the noisy panda images into the standard DEQ and our SRS-DEQ, there will be some misalignment due to the correlation introduced by SRS. Our method conservatively converts these predictions back to the correct ones. For instance, the predictions of ${\mathbf{x}}+\epsilon_2$ are different with RS and SRS. Therefore, the prediction of ${\mathbf{x}}+\epsilon_2$ will not be counted as the most probable class $\hat{c}_A$. Finally, we use these converted predictions to calculate the certified radius to recover the standard DEQ's predictions. In the implementation, we try to estimate the number of these converted predictions instead of using the standard DEQ to get the inferences.

Theorems & Definitions (2)

• Theorem 3.1: Correlation-Eliminated Certification
• Theorem