SafeText: A Benchmark for Exploring Physical Safety in Language Models

TL;DR

SafeText introduces a benchmark for commonsense physical safety, capturing everyday scenarios where safe versus unsafe advice hinges on implicit knowledge of physical harm. Through a five-phase data-collection process, the authors create 367 prompts with paired safe and unsafe commands, enabling generation and reasoning assessments across GPT-2, GPT-3, COMET-GPT2, and NLI models. Experiments show that while models often favor safe outputs, unsafe text can still be generated or misdetected, especially by knowledge-grounded or zero-shot systems. The results advocate enhanced safety evaluation and integration of external knowledge to prevent unsafe guidance, and they position SafeText as a targeted, high-signal benchmark for future safety research. Overall, the work highlights the vulnerability of current large language models to commonsense physical safety failures and the need for proactive safeguards before deployment.

Abstract

Understanding what constitutes safe text is an important issue in natural language processing and can often prevent the deployment of models deemed harmful and unsafe. One such type of safety that has been scarcely studied is commonsense physical safety, i.e. text that is not explicitly violent and requires additional commonsense knowledge to comprehend that it leads to physical harm. We create the first benchmark dataset, SafeText, comprising real-life scenarios with paired safe and physically unsafe pieces of advice. We utilize SafeText to empirically study commonsense physical safety across various models designed for text generation and commonsense reasoning tasks. We find that state-of-the-art large language models are susceptible to the generation of unsafe text and have difficulty rejecting unsafe advice. As a result, we argue for further studies of safety and the assessment of commonsense physical safety in models before release.

Figures (6)

• Figure 1: SafeText dataset creation process.
• Figure 2: Dataset creation consent form.
• Figure 3: Dataset creation phase 2 interface.
• Figure 4: Dataset creation phase 3 interface.
• Figure 5: Top of phase 5 interface for dataset creation.