Noise-BERT: A Unified Perturbation-Robust Framework with Noise Alignment Pre-training for Noisy Slot Filling Task

TL;DR

Noise-BERT addresses the slot filling robustness problem under input perturbations in task-oriented dialogue systems. It introduces two Noise Alignment Pre-training tasks (Slot Masked Prediction and Sentence Noisiness Discrimination) and combines them with contrastive fine-tuning and adversarial training. The approach leverages multi-level data augmentation via NLPAug to strengthen slot-boundary and label representations under noisy conditions. Empirical results on RADDLE and SNIPS-based PSSAT settings show substantial robustness gains across single and mixed perturbations, with ablations confirming the value of each component. This work advances practical robustness for slot filling in noisy real-world dialogue systems.

Abstract

In a realistic dialogue system, the input information from users is often subject to various types of input perturbations, which affects the slot-filling task. Although rule-based data augmentation methods have achieved satisfactory results, they fail to exhibit the desired generalization when faced with unknown noise disturbances. In this study, we address the challenges posed by input perturbations in slot filling by proposing Noise-BERT, a unified Perturbation-Robust Framework with Noise Alignment Pre-training. Our framework incorporates two Noise Alignment Pre-training tasks: Slot Masked Prediction and Sentence Noisiness Discrimination, aiming to guide the pre-trained language model in capturing accurate slot information and noise distribution. During fine-tuning, we employ a contrastive learning loss to enhance the semantic representation of entities and labels. Additionally, we introduce an adversarial attack training strategy to improve the model's robustness. Experimental results demonstrate the superiority of our proposed approach over state-of-the-art models, and further analysis confirms its effectiveness and generalization ability.

Figures (3)

• Figure 1: The impact of various types of input perturbations on the slot filling task in real dialogue systems
• Figure 2: The overall architecture of our proposed Noise-BERT framework
• Figure 3: The t-SNE visualization of the entity representations with different types on mixed perturbation.