RMCBench: Benchmarking Large Language Models' Resistance to Malicious Code

TL;DR

RMCBench tackles the gap in evaluating LLMs’ resistance to malicious code generation by introducing a dedicated benchmark with 473 prompts across text-to-code and code-to-code scenarios. The authors construct a three-tier text-to-code prompt set and two code-to-code tasks (completion and translation), then benchmark 11 representative LLMs, including ChatGPT variants and several open-source models. Key findings show an overall modest resistance, with an average refusal of 40.36% for text-to-code and 11.52% for code-to-code, and reveal factors such as model size, prompt design, code language, and input length that influence outcomes; phishing and certain languages (e.g., Bash) pose especially high risk. The work provides actionable guidance for model developers to enhance safety, and releases data and code to enable broader evaluation and improvement of robustness against malicious code generation.

Abstract

The emergence of Large Language Models (LLMs) has significantly influenced various aspects of software development activities. Despite their benefits, LLMs also pose notable risks, including the potential to generate harmful content and being abused by malicious developers to create malicious code. Several previous studies have focused on the ability of LLMs to resist the generation of harmful content that violates human ethical standards, such as biased or offensive content. However, there is no research evaluating the ability of LLMs to resist malicious code generation. To fill this gap, we propose RMCBench, the first benchmark comprising 473 prompts designed to assess the ability of LLMs to resist malicious code generation. This benchmark employs two scenarios: a text-to-code scenario, where LLMs are prompted with descriptions to generate code, and a code-to-code scenario, where LLMs translate or complete existing malicious code. Based on RMCBench, we conduct an empirical study on 11 representative LLMs to assess their ability to resist malicious code generation. Our findings indicate that current LLMs have a limited ability to resist malicious code generation with an average refusal rate of 40.36% in text-to-code scenario and 11.52% in code-to-code scenario. The average refusal rate of all LLMs in RMCBench is only 28.71%; ChatGPT-4 has a refusal rate of only 35.73%. We also analyze the factors that affect LLMs' ability to resist malicious code generation and provide implications for developers to enhance model robustness.

Figures (6)

• Figure 1: Examples of using LLMs to generate malicious code.
• Figure 2: Overview of RMCBench Construction Process. T2C-L represents text-to-code level, CC represents code completion, CT represents code translation.
• Figure 3: Prompt templates for C2C scenario.
• Figure 4: Categories and Language of Malicious Code in RMCBench.
• Figure 5: Prompt Template for Evaluation.