Can LLMs Threaten Human Survival? Benchmarking Potential Existential Threats from LLMs via Prefix Completion

TL;DR

The paper tackles the problem of existential threats from LLMs by introducing ExistBench, a bilingual benchmark and evaluation framework that uses prefix completion to elicit hostile suffixes and assess real-world risks. It combines a 2,138-sample dataset with a dual-inference setup and two metrics (RR and TR) to quantify hostile attitudes and actionable threats, including an extended multi-round prefix completion and a tool-calling risk framework. Empirical results across 10 models reveal substantial existential threats under ExistBench, amplified by multi-round interactions, and show a tendency to invoke harmful tools in tool-calling scenarios. The work also analyzes attention logits to explain the root causes and discusses mitigation strategies, limitations, and ethical considerations, highlighting an urgent need for more robust defenses in LLM systems.

Abstract

Research on the safety evaluation of large language models (LLMs) has become extensive, driven by jailbreak studies that elicit unsafe responses. Such response involves information already available to humans, such as the answer to "how to make a bomb". When LLMs are jailbroken, the practical threat they pose to humans is negligible. However, it remains unclear whether LLMs commonly produce unpredictable outputs that could pose substantive threats to human safety. To address this gap, we study whether LLM-generated content contains potential existential threats, defined as outputs that imply or promote direct harm to human survival. We propose \textsc{ExistBench}, a benchmark designed to evaluate such risks. Each sample in \textsc{ExistBench} is derived from scenarios where humans are positioned as adversaries to AI assistants. Unlike existing evaluations, we use prefix completion to bypass model safeguards. This leads the LLMs to generate suffixes that express hostility toward humans or actions with severe threat, such as the execution of a nuclear strike. Our experiments on 10 LLMs reveal that LLM-generated content indicates existential threats. To investigate the underlying causes, we also analyze the attention logits from LLMs. To highlight real-world safety risks, we further develop a framework to assess model behavior in tool-calling. We find that LLMs actively select and invoke external tools with existential threats. Code and data are available at: https://github.com/cuiyu-ai/ExistBench.

Figures (7)

• Figure 1: Existential threats revealed in benign LLMs under prefix completion. Even when explicitly instructed to act as a faithful AI assistant to humans, the model still generates outputs that pose severe safety risks to humanity.
• Figure 2: Comparison of word clouds for our proposed benchmark dataset and traditional jailbreak dataset.
• Figure 3: Experimental results of the resistance rate and threat rate of LLMs under malicious inference.
• Figure 4: Evaluation results of existential threats from LLMs under tool-calling scenarios. During a single prefix completion, the model may invoke multiple tools. Therefore, the sum of these rates does not necessarily reach 100% in an overall evaluation.
• Figure 5: Evaluation of existential threats generated by LLMs under multi-round prefix completion.