Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding LLM Behavior When Encountering User-Supplied Harmful Content in Harmless Tasks

Junjie Chu, Yiting Qu, Ye Leng, Michael Backes, Yun Shen, Savvas Zannettou, Yang Zhang

Abstract

Large Language Models (LLMs) are increasingly trained to align with human values, primarily focusing on task level, i.e., refusing to execute directly harmful tasks. However, a subtle yet crucial content-level ethical question is often overlooked: when performing a seemingly benign task, will LLMs -- like morally conscious human beings -- refuse to proceed when encountering harmful content in user-provided material? In this study, we aim to understand this content-level ethical question and systematically evaluate its implications for mainstream LLMs. We first construct a harmful knowledge dataset (i.e., non-compliant with OpenAI's usage policy) to serve as the user-supplied harmful content, with 1,357 entries across ten harmful categories. We then design nine harmless tasks (i.e., compliant with OpenAI's usage policy) to simulate the real-world benign tasks, grouped into three categories according to the extent of user-supplied content required: extensive, moderate, and limited. Leveraging the harmful knowledge dataset and the set of harmless tasks, we evaluate how nine LLMs behave when exposed to user-supplied harmful content during the execution of benign tasks, and further examine how the dynamics between harmful knowledge categories and tasks affect different LLMs. Our results show that current LLMs, even the latest GPT-5.2 and Gemini-3-Pro, often fail to uphold human-aligned ethics by continuing to process harmful content in harmless tasks. Furthermore, external knowledge from the ``Violence/Graphic'' category and the ``Translation'' task is more likely to elicit harmful responses from LLMs. We also conduct extensive ablation studies to investigate potential factors affecting this novel misuse vulnerability. We hope that our study could inspire enhanced safety measures among stakeholders to mitigate this overlooked content-level ethical risk.

Understanding LLM Behavior When Encountering User-Supplied Harmful Content in Harmless Tasks

Abstract

Large Language Models (LLMs) are increasingly trained to align with human values, primarily focusing on task level, i.e., refusing to execute directly harmful tasks. However, a subtle yet crucial content-level ethical question is often overlooked: when performing a seemingly benign task, will LLMs -- like morally conscious human beings -- refuse to proceed when encountering harmful content in user-provided material? In this study, we aim to understand this content-level ethical question and systematically evaluate its implications for mainstream LLMs. We first construct a harmful knowledge dataset (i.e., non-compliant with OpenAI's usage policy) to serve as the user-supplied harmful content, with 1,357 entries across ten harmful categories. We then design nine harmless tasks (i.e., compliant with OpenAI's usage policy) to simulate the real-world benign tasks, grouped into three categories according to the extent of user-supplied content required: extensive, moderate, and limited. Leveraging the harmful knowledge dataset and the set of harmless tasks, we evaluate how nine LLMs behave when exposed to user-supplied harmful content during the execution of benign tasks, and further examine how the dynamics between harmful knowledge categories and tasks affect different LLMs. Our results show that current LLMs, even the latest GPT-5.2 and Gemini-3-Pro, often fail to uphold human-aligned ethics by continuing to process harmful content in harmless tasks. Furthermore, external knowledge from the ``Violence/Graphic'' category and the ``Translation'' task is more likely to elicit harmful responses from LLMs. We also conduct extensive ablation studies to investigate potential factors affecting this novel misuse vulnerability. We hope that our study could inspire enhanced safety measures among stakeholders to mitigate this overlooked content-level ethical risk.
Paper Structure (48 sections, 11 figures, 13 tables)

This paper contains 48 sections, 11 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. In-Content Harm Risk
  4. Related Works
  5. LLM Misuse
  6. LLM Safety Alignment
  7. Assessment Methodology
  8. Harmful Knowledge Dataset
  9. Harmless Task Design
  10. Response Inspection
  11. Evaluation Metrics
  12. Evaluation on Frontier LLMs
  13. Assessment Procedure
  14. Experimental Setups
  15. Target LLMs
  16. ...and 33 more sections

Figures (11)

  • Figure 1: An instance of the in-content harm risk: translating detailed instructions for constructing nuclear bombs guide_nuclear.
  • Figure 2: Overview of the assessment process for in-content harm risk.
  • Figure 3: T-HRRs of different tasks with various harmful knowledge categories on the target LLMs.
  • Figure 4: Ablation studies of the in-content harm risk. (a) Effects of different sources of harmful content, comparing user-supplied content only versus combined user-supplied and pre-trained knowledge. (b) Effects of different internal safety check statuses.
  • Figure 5: Ablation studies of the in-content harm risk (task: Translation). (a) Results of user-supplied knowledge length (logarithmic scale). (b) Results of different harmful content proportions. The total length of the user-supplied knowledge is about 1,000 tokens, composed of 10 knowledge pieces. (c) Results of different harmful content positions. The total length of the user-supplied knowledge is about 500 tokens, composed of five knowledge pieces.
  • ...and 6 more figures