Automated Harmfulness Testing for Code Large Language Models
Honghao Tan, Haibo Wang, Diany Pressato, Yisen Xu, Shin Hwei Tan
TL;DR
Automated Harmfulness Testing for Code Large Language Models (CHT) targets the robustness gap in content moderation for Code LLMs by introducing coverage-guided harmfulness testing that injects harmful keywords into benign code transformations. The method builds a 13-category harm taxonomy, constructs a harm-content dataset and a benign program dataset, and uses diversity-enhanced prompt synthesis and output-damage measurement to assess risks in LLM outputs. Key findings show Code LLMs generally struggle to resist harmful code generation (average GN $65.93\%$) and that a two-phase moderation approach using function calling can substantially reduce harmful outputs (e.g., NR $34.50\%$). The work provides datasets, a testing framework, and empirical results across multiple Code LLMs, highlighting the need for automated, pre-output content moderation to improve developer safety and software artifact integrity.
Abstract
Generative AI systems powered by Large Language Models (LLMs) usually use content moderation to prevent harmful content spread. To evaluate the robustness of content moderation, several metamorphic testing techniques have been proposed to test content moderation software. However, these techniques mainly focus on general users (e.g., text and image generation). Meanwhile, a recent study shows that developers consider using harmful keywords when naming software artifacts to be an unethical behavior. Exposure to harmful content in software artifacts can negatively impact the mental health of developers, making content moderation for Code Large Language Models (Code LLMs) essential. We conduct a preliminary study on program transformations that can be misused to introduce harmful content into auto-generated code, identifying 32 such transformations. To address this, we propose CHT, a coverage-guided harmfulness testing framework that generates prompts using diverse transformations and harmful keywords injected into benign programs. CHT evaluates output damage to assess potential risks in LLM-generated explanations and code. Our evaluation of four Code LLMs and GPT-4o-mini reveals that content moderation in LLM-based code generation is easily bypassed. To enhance moderation, we propose a two-phase approach that first detects harmful content before generating output, improving moderation effectiveness by 483.76\%.