Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On damage of interpolation to adversarial robustness in regression

Jingfu Peng, Yuhong Yang

TL;DR

This work investigates how interpolation influences adversarial robustness in nonparametric regression with true functions in a Hölder class. By framing a minimax analysis over δ-interpolating estimators, it reveals that higher degrees of interpolation can degrade robustness under future X-attacks and uncovers phase transitions and a curse of sample size, with dimensionality sometimes mitigating adverse effects. The results connect to deep networks by showing that interpolating models, including many DNNs, cannot attain optimal robustness under adversarial perturbations, motivating cautious training practices. Complemented by synthetic and real-data experiments, the paper provides theoretical and empirical insight into the robustness penalties of interpolation and informs strategies to improve robust generalization in practice.

Abstract

Deep neural networks (DNNs) typically involve a large number of parameters and are trained to achieve zero or near-zero training error. Despite such interpolation, they often exhibit strong generalization performance on unseen data, a phenomenon that has motivated extensive theoretical investigations. Comforting results show that interpolation indeed may not affect the minimax rate of convergence under the squared error loss. In the mean time, DNNs are well known to be highly vulnerable to adversarial perturbations in future inputs. A natural question then arises: Can interpolation also escape from suboptimal performance under a future $X$-attack? In this paper, we investigate the adversarial robustness of interpolating estimators in a framework of nonparametric regression. A finding is that interpolating estimators must be suboptimal even under a subtle future $X$-attack, and achieving perfect fitting can substantially damage their robustness. An interesting phenomenon in the high interpolation regime, which we term the curse of simple size, is also revealed and discussed. Numerical experiments support our theoretical findings.

On damage of interpolation to adversarial robustness in regression

TL;DR

This work investigates how interpolation influences adversarial robustness in nonparametric regression with true functions in a Hölder class. By framing a minimax analysis over δ-interpolating estimators, it reveals that higher degrees of interpolation can degrade robustness under future X-attacks and uncovers phase transitions and a curse of sample size, with dimensionality sometimes mitigating adverse effects. The results connect to deep networks by showing that interpolating models, including many DNNs, cannot attain optimal robustness under adversarial perturbations, motivating cautious training practices. Complemented by synthetic and real-data experiments, the paper provides theoretical and empirical insight into the robustness penalties of interpolation and informs strategies to improve robust generalization in practice.

Abstract

Deep neural networks (DNNs) typically involve a large number of parameters and are trained to achieve zero or near-zero training error. Despite such interpolation, they often exhibit strong generalization performance on unseen data, a phenomenon that has motivated extensive theoretical investigations. Comforting results show that interpolation indeed may not affect the minimax rate of convergence under the squared error loss. In the mean time, DNNs are well known to be highly vulnerable to adversarial perturbations in future inputs. A natural question then arises: Can interpolation also escape from suboptimal performance under a future -attack? In this paper, we investigate the adversarial robustness of interpolating estimators in a framework of nonparametric regression. A finding is that interpolating estimators must be suboptimal even under a subtle future -attack, and achieving perfect fitting can substantially damage their robustness. An interesting phenomenon in the high interpolation regime, which we term the curse of simple size, is also revealed and discussed. Numerical experiments support our theoretical findings.
Paper Structure (34 sections, 7 theorems, 94 equations, 5 figures)

This paper contains 34 sections, 7 theorems, 94 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Contributions
  3. Other related work
  4. Outline
  5. Problem setup
  6. Basic settings
  7. Adversarial risks
  8. Interpolating estimator
  9. Main results
  10. Minimax rates
  11. Phase transitions
  12. An interesting phenomenon in adversarial learning under future $X$-attacks: curse of sample size and blessing of dimensionality
  13. Implications for adversarial robustness in DNNs
  14. Numerical experiments
  15. Synthetic simulation
  16. ...and 19 more sections

Key Result

Theorem 1

Suppose Assumption ass:bounded_density holds and $\xi_i$ are i.i.d. $N(0,\sigma^2)$. We have the following minimax lower bound: where $\mathcal{S}_p(x,r) \triangleq \{ i: X_i \in \mathcal{X} \cap B_p(x,r) \}$ denotes the index set of points within the adversarial ball $B_p(x,r)$.

Figures (5)

  • Figure 1: Minimax rate in the low interpolation regime: Panel (a) illustrates the case where $1 \leq d \leq 4$, while panel (b) corresponds to $d \geq 5$. The quantity in the grey rectangle denotes the dominant rates in the respective parameter spaces.
  • Figure 2: Minimax rate in the high interpolation regime: Panel (a) illustrates the case where $1 \leq d \leq 4$, while panel (b) corresponds to $d \geq 5$. The quantity in the grey rectangle denotes the dominant rates in the respective parameter spaces.
  • Figure 3: Adversarial risk of the five competing methods: results for Case 1 are shown in row (a), Case 2 in row (b), and Case 3 in row (c).
  • Figure 4: Standard and adversarial risks of the neural network across epochs. The $\diamond$ symbols indicate the minimum value along each risk curve.
  • Figure 5: Density of the optimal number of epochs at which the minimum error is attained over 100 replications.

Theorems & Definitions (13)

  • Definition 1
  • Theorem 1: Lower bound
  • Theorem 2: Upper bound
  • Example 1: Simple $\delta$-interpolator
  • Remark 1
  • Example 2: Shrinking-neighborhood $\delta$-interpolator
  • Remark 2
  • Theorem 3: Low interpolation regime
  • Theorem 4: Moderate interpolation regime
  • Theorem 5: High interpolation regime
  • ...and 3 more