CR-UTP: Certified Robustness against Universal Text Perturbations on Large Language Models

TL;DR

This work tackles the problem of certifying robustness of pretrained language models against universal text perturbations (UTPs) and input-specific perturbations (ISTPs). It introduces CR-UTP, a three-component framework that combines adapting random smoothing to PLMs, a reinforcement-learning based Superior Prompt Search, and a Superior Prompt Ensemble to maintain high certified accuracy under heavy masking. The method achieves state-of-the-art certified robustness and substantial reductions in attack success rate across multiple datasets and large-language models, outperforming prior random masking and adversarial-training baselines. Practically, CR-UTP provides a reusable offline prompt search and an efficient ensemble-based inference path that enhances robustness in high-stakes NLP applications.

Abstract

It is imperative to ensure the stability of every prediction made by a language model; that is, a language's prediction should remain consistent despite minor input variations, like word substitutions. In this paper, we investigate the problem of certifying a language model's robustness against Universal Text Perturbations (UTPs), which have been widely used in universal adversarial attacks and backdoor attacks. Existing certified robustness based on random smoothing has shown considerable promise in certifying the input-specific text perturbations (ISTPs), operating under the assumption that any random alteration of a sample's clean or adversarial words would negate the impact of sample-wise perturbations. However, with UTPs, masking only the adversarial words can eliminate the attack. A naive method is to simply increase the masking ratio and the likelihood of masking attack tokens, but it leads to a significant reduction in both certified accuracy and the certified radius due to input corruption by extensive masking. To solve this challenge, we introduce a novel approach, the superior prompt search method, designed to identify a superior prompt that maintains higher certified accuracy under extensive masking. Additionally, we theoretically motivate why ensembles are a particularly suitable choice as base prompts for random smoothing. The method is denoted by superior prompt ensembling technique. We also empirically confirm this technique, obtaining state-of-the-art results in multiple settings. These methodologies, for the first time, enable high certified accuracy against both UTPs and ISTPs. The source code of CR-UTP is available at \url {https://github.com/UCFML-Research/CR-UTP}.

Figures (5)

• Figure 1: Illustration of the prediction distributions. A superior prompt exhibits greater robustness compared to a vanilla prompt, with ensembled prompts showing even higher robustness. Different colors represent various classes, and different radii indicate varying levels of perturbation. The bars demonstrate the output class probabilities for the smoothed PLMs given corresponding prompts. $\underline{p_A}$ represents the minimum probability of the majority class, and $\overline{p_B}$ indicates the maximum probability of the second-most likely class.
• Figure 2: (a) Our CR-UTP shows higher certified robustness accuracy and (b) Our CR-UTP significantly reduces ASR.
• Figure 3: Overview of CR-UTP. CR-UTP leverages superior prompt search and prompt ensembling techniques to enhance the certified robustness of PLMs.
• Figure 4: (a) Clean accuracy and (b) variance of proposed methods under different mask ratio.
• Figure 5: Clean accuracy and variance of CR-UTP under different ensemble numbers.