Measure and Improve Robustness in NLP Models: A Survey

TL;DR

NLP models remain fragile under unseen conditions, motivating a unified survey of robustness that treats adversarial perturbations and distribution shifts within a common framework. The paper defines robustness via performance on shifted data, surveys identification methods (human-prior analyses and model-driven approaches), and organizes mitigation into data-driven, model/training-based, inductive-prior-based, and causal-intervention strategies, while highlighting connections to dataset biases and calibration. It contributes a conceptual schema, a catalog of tasks and benchmarks, and open questions that guide future work toward unified evaluation, end-to-end mitigation, and human-centered robustness. This work aims to inform safer real-world deployment and to catalyze cross-disciplinary progress in robustness for NLP, with formalism such as robust accuracy $ ext{RobustAccuracy} = \

Abstract

As NLP models achieved state-of-the-art performances over benchmarks and gained wide applications, it has been increasingly important to ensure the safe deployment of these models in the real world, e.g., making sure the models are robust against unseen or challenging scenarios. Despite robustness being an increasingly studied topic, it has been separately explored in applications like vision and NLP, with various definitions, evaluation and mitigation strategies in multiple lines of research. In this paper, we aim to provide a unifying survey of how to define, measure and improve robustness in NLP. We first connect multiple definitions of robustness, then unify various lines of work on identifying robustness failures and evaluating models' robustness. Correspondingly, we present mitigation strategies that are data-driven, model-driven, and inductive-prior-based, with a more systematic view of how to effectively improve robustness in NLP models. Finally, we conclude by outlining open challenges and future directions to motivate further research in this area.