Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Method

Mihyeon Kim, Juhyoung Park, Youngbin Kim

TL;DR

This work tackles adversarial vulnerability in pre-trained language models during fine-tuning by treating BERT as a continuous-time dynamical system and analyzing forward and backward Euler discretizations. It introduces IM-connection, an implicit Euler-based module implemented via gradient descent to yield an inherently robust architecture (IM-BERT) without added parameters or adversarial training. Empirical results on AdvGLUE show IM-BERT improves average test accuracy by about $8.3$ percentage points and delivers up to $5.9$ percentage points gains in low-resource settings, outperforming several baselines. While the method incurs time-cost due to the implicit solve, selective layer deployment mitigates this, and the work paves the way for extending this architecture-level robustness to other PLMs and numerical methods.

Abstract

Pre-trained Language Models (PLMs) have achieved remarkable performance on diverse NLP tasks through pre-training and fine-tuning. However, fine-tuning the model with a large number of parameters on limited downstream datasets often leads to vulnerability to adversarial attacks, causing overfitting of the model on standard datasets. To address these issues, we propose IM-BERT from the perspective of a dynamic system by conceptualizing a layer of BERT as a solution of Ordinary Differential Equations (ODEs). Under the situation of initial value perturbation, we analyze the numerical stability of two main numerical ODE solvers: the explicit and implicit Euler approaches. Based on these analyses, we introduce a numerically robust IM-connection incorporating BERT's layers. This strategy enhances the robustness of PLMs against adversarial attacks, even in low-resource scenarios, without introducing additional parameters or adversarial training strategies. Experimental results on the adversarial GLUE (AdvGLUE) dataset validate the robustness of IM-BERT under various conditions. Compared to the original BERT, IM-BERT exhibits a performance improvement of approximately 8.3\%p on the AdvGLUE dataset. Furthermore, in low-resource scenarios, IM-BERT outperforms BERT by achieving 5.9\%p higher accuracy.

IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Method

TL;DR

This work tackles adversarial vulnerability in pre-trained language models during fine-tuning by treating BERT as a continuous-time dynamical system and analyzing forward and backward Euler discretizations. It introduces IM-connection, an implicit Euler-based module implemented via gradient descent to yield an inherently robust architecture (IM-BERT) without added parameters or adversarial training. Empirical results on AdvGLUE show IM-BERT improves average test accuracy by about percentage points and delivers up to percentage points gains in low-resource settings, outperforming several baselines. While the method incurs time-cost due to the implicit solve, selective layer deployment mitigates this, and the work paves the way for extending this architecture-level robustness to other PLMs and numerical methods.

Abstract

Pre-trained Language Models (PLMs) have achieved remarkable performance on diverse NLP tasks through pre-training and fine-tuning. However, fine-tuning the model with a large number of parameters on limited downstream datasets often leads to vulnerability to adversarial attacks, causing overfitting of the model on standard datasets. To address these issues, we propose IM-BERT from the perspective of a dynamic system by conceptualizing a layer of BERT as a solution of Ordinary Differential Equations (ODEs). Under the situation of initial value perturbation, we analyze the numerical stability of two main numerical ODE solvers: the explicit and implicit Euler approaches. Based on these analyses, we introduce a numerically robust IM-connection incorporating BERT's layers. This strategy enhances the robustness of PLMs against adversarial attacks, even in low-resource scenarios, without introducing additional parameters or adversarial training strategies. Experimental results on the adversarial GLUE (AdvGLUE) dataset validate the robustness of IM-BERT under various conditions. Compared to the original BERT, IM-BERT exhibits a performance improvement of approximately 8.3\%p on the AdvGLUE dataset. Furthermore, in low-resource scenarios, IM-BERT outperforms BERT by achieving 5.9\%p higher accuracy.
Paper Structure (19 sections, 9 equations, 1 figure, 6 tables, 1 algorithm)

This paper contains 19 sections, 9 equations, 1 figure, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Robust Learning against Adversarial Attack
  4. ODE Neural Network
  5. Method
  6. Neural Network as an ODE Solver
  7. Residual Connection as Euler Method
  8. Stability of Implicit Connection in Perturbation
  9. IM-BERT
  10. Experiment
  11. Robustness against Various Adversarial Attacks
  12. Robustness in Low Resource Scenarios
  13. Ablation Study
  14. Evaluating the Inherent Robustness
  15. Conclusion
  16. ...and 4 more sections

Figures (1)

  • Figure 1: The $l$-th layer in each architecture The perturbed inputs pass through the layer to estimate the hidden states. Blue represents the corrected hidden state, while red indicates the opposite. Blue connections represent explicit residual connections, while red connections represent IM-connections. (a) BERT (b) EX-BERT In the layer, the layers are connected with the explicit method. (c) IM-BERT The hidden states taken from the $l$-th layer are updated in IM-connection.