Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Malla: Demystifying Real-world Large Language Model Integrated Malicious Services

Zilong Lin, Jian Cui, Xiaojing Liao, XiaoFeng Wang

TL;DR

This paper provides a systematic exploration of real-world Mallas, LLM-based malicious services, by analyzing 212 samples across underground markets and hosting platforms. It characterizes the ecosystem, key actors, and monetization strategies, and reveals two dominant exploitation techniques: uncensored LLMs and jailbreak prompts, supported by 182 jailbreak prompts and 45 malicious prompts. Through content-quality analysis, case studies, and a user study, the work shows that some Malla-generated payloads can be highly plausible and evasive, though real-world effectiveness varies and detection remains feasible in many cases. The authors also release artifacts and propose defense strategies—dynamic threat monitoring, improved moderation, and platform accountability—to counter LLM misuse at scale.

Abstract

The underground exploitation of large language models (LLMs) for malicious services (i.e., Malla) is witnessing an uptick, amplifying the cyber threat landscape and posing questions about the trustworthiness of LLM technologies. However, there has been little effort to understand this new cybercrime, in terms of its magnitude, impact, and techniques. In this paper, we conduct the first systematic study on 212 real-world Mallas, uncovering their proliferation in underground marketplaces and exposing their operational modalities. Our study discloses the Malla ecosystem, revealing its significant growth and impact on today's public LLM services. Through examining 212 Mallas, we uncovered eight backend LLMs used by Mallas, along with 182 prompts that circumvent the protective measures of public LLM APIs. We further demystify the tactics employed by Mallas, including the abuse of uncensored LLMs and the exploitation of public LLM APIs through jailbreak prompts. Our findings enable a better understanding of the real-world exploitation of LLMs by cybercriminals, offering insights into strategies to counteract this cybercrime.

Malla: Demystifying Real-world Large Language Model Integrated Malicious Services

TL;DR

This paper provides a systematic exploration of real-world Mallas, LLM-based malicious services, by analyzing 212 samples across underground markets and hosting platforms. It characterizes the ecosystem, key actors, and monetization strategies, and reveals two dominant exploitation techniques: uncensored LLMs and jailbreak prompts, supported by 182 jailbreak prompts and 45 malicious prompts. Through content-quality analysis, case studies, and a user study, the work shows that some Malla-generated payloads can be highly plausible and evasive, though real-world effectiveness varies and detection remains feasible in many cases. The authors also release artifacts and propose defense strategies—dynamic threat monitoring, improved moderation, and platform accountability—to counter LLM misuse at scale.

Abstract

The underground exploitation of large language models (LLMs) for malicious services (i.e., Malla) is witnessing an uptick, amplifying the cyber threat landscape and posing questions about the trustworthiness of LLM technologies. However, there has been little effort to understand this new cybercrime, in terms of its magnitude, impact, and techniques. In this paper, we conduct the first systematic study on 212 real-world Mallas, uncovering their proliferation in underground marketplaces and exposing their operational modalities. Our study discloses the Malla ecosystem, revealing its significant growth and impact on today's public LLM services. Through examining 212 Mallas, we uncovered eight backend LLMs used by Mallas, along with 182 prompts that circumvent the protective measures of public LLM APIs. We further demystify the tactics employed by Mallas, including the abuse of uncensored LLMs and the exploitation of public LLM APIs through jailbreak prompts. Our findings enable a better understanding of the real-world exploitation of LLMs by cybercriminals, offering insights into strategies to counteract this cybercrime.
Paper Structure (26 sections, 4 figures, 8 tables)

This paper contains 26 sections, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Large Language Model
  4. Threat Model
  5. Data Collection
  6. Malla Services
  7. Malla Projects
  8. Discussion
  9. Understanding Malla
  10. Scope and Magnitude
  11. Malla Stakeholders
  12. Price Strategy and Revenue
  13. Analyzing Quality of Malla-generated Content
  14. Methods and Metrics
  15. Results and Findings
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Malla workflow.
  • Figure 2: Creation dates of Malla projects and owner accounts on FlowGPT.
  • Figure 3: Human subject study responses
  • Figure 4: Questionnaire sample for user study.