ROIC-DM: Robust Text Inference and Classification via Diffusion Model

TL;DR

This paper introduces an innovative model for robust text inference and classification, built upon diffusion models (ROIC-DM), which outperforms traditional language models in robustness, even when the latter are fortified with advanced defense mechanisms.

Abstract

While language models have made many milestones in text inference and classification tasks, they remain susceptible to adversarial attacks that can lead to unforeseen outcomes. Existing works alleviate this problem by equipping language models with defense patches. However, these defense strategies often rely on impractical assumptions or entail substantial sacrifices in model performance. Consequently, enhancing the resilience of the target model using such defense mechanisms is a formidable challenge. This paper introduces an innovative model for robust text inference and classification, built upon diffusion models (ROIC-DM). Benefiting from its training involving denoising stages, ROIC-DM inherently exhibits greater robustness compared to conventional language models. Moreover, ROIC-DM can attain comparable, and in some cases, superior performance to language models, by effectively incorporating them as advisory components. Extensive experiments conducted with several strong textual adversarial attacks on three datasets demonstrate that (1) ROIC-DM outperforms traditional language models in robustness, even when the latter are fortified with advanced defense mechanisms; (2) ROIC-DM can achieve comparable and even better performance than traditional language models by using them as advisors.

Figures (2)

• Figure 1: Illustration of our proposed Robust text Inference and Classification Diffusion Model (ROIC-DM). The figure on the left illustrates the diffusion phase and the reverse phase. The figure on the right illustrates the architecture of noise estimator $f_{\theta}$. $\mathbf{x}$ is the input text; $\mathbf{y}$ is the categorical label; $t$ is the time step number; $\epsilon_{\theta}$ is the predicted noise. The $\odot$ is the element-wise product. The $\oplus$ indicates element-wise summation.
• Figure 2: The training loss trend for ROIC-DM and ROIC-DM(-advisor) on the AG NEWS dataset.