Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficacy of Utilizing Large Language Models to Detect Public Threat Posted Online

Taeksoo Kwon, Connor Kim

TL;DR

The study evaluates large language models for detecting online public threats using Korean post titles, employing a 30-trial setup with 40 safe and 10 threat items per trial and assessing accuracy via a chi-square goodness-of-fit test at $\alpha = 0.05$. GPT-4 achieves the strongest performance, with non-threat accuracy around $97.9\%$ and threat accuracy at $100\%$, while GPT-3.5-turbo-1106 trails on some measures and PaLM offers cost advantages at the expense of occasional formatting challenges. The results support the potential of LLM-assisted content moderation at scale but flag critical considerations around biases, transparency, and ethical oversight before deployment. Language and domain specificity (Korean data) further influence outcomes, underscoring the need for careful validation across contexts and ongoing governance. Overall, the work highlights practical avenues for integrating LLMs into threat-detection workflows while calling for robust safeguards and iterative refinements.

Abstract

This paper examines the efficacy of utilizing large language models (LLMs) to detect public threats posted online. Amid rising concerns over the spread of threatening rhetoric and advance notices of violence, automated content analysis techniques may aid in early identification and moderation. Custom data collection tools were developed to amass post titles from a popular Korean online community, comprising 500 non-threat examples and 20 threats. Various LLMs (GPT-3.5, GPT-4, PaLM) were prompted to classify individual posts as either "threat" or "safe." Statistical analysis found all models demonstrated strong accuracy, passing chi-square goodness of fit tests for both threat and non-threat identification. GPT-4 performed best overall with 97.9% non-threat and 100% threat accuracy. Affordability analysis also showed PaLM API pricing as highly cost-efficient. The findings indicate LLMs can effectively augment human content moderation at scale to help mitigate emerging online risks. However, biases, transparency, and ethical oversight remain vital considerations before real-world implementation.

Efficacy of Utilizing Large Language Models to Detect Public Threat Posted Online

TL;DR

The study evaluates large language models for detecting online public threats using Korean post titles, employing a 30-trial setup with 40 safe and 10 threat items per trial and assessing accuracy via a chi-square goodness-of-fit test at . GPT-4 achieves the strongest performance, with non-threat accuracy around and threat accuracy at , while GPT-3.5-turbo-1106 trails on some measures and PaLM offers cost advantages at the expense of occasional formatting challenges. The results support the potential of LLM-assisted content moderation at scale but flag critical considerations around biases, transparency, and ethical oversight before deployment. Language and domain specificity (Korean data) further influence outcomes, underscoring the need for careful validation across contexts and ongoing governance. Overall, the work highlights practical avenues for integrating LLMs into threat-detection workflows while calling for robust safeguards and iterative refinements.

Abstract

This paper examines the efficacy of utilizing large language models (LLMs) to detect public threats posted online. Amid rising concerns over the spread of threatening rhetoric and advance notices of violence, automated content analysis techniques may aid in early identification and moderation. Custom data collection tools were developed to amass post titles from a popular Korean online community, comprising 500 non-threat examples and 20 threats. Various LLMs (GPT-3.5, GPT-4, PaLM) were prompted to classify individual posts as either "threat" or "safe." Statistical analysis found all models demonstrated strong accuracy, passing chi-square goodness of fit tests for both threat and non-threat identification. GPT-4 performed best overall with 97.9% non-threat and 100% threat accuracy. Affordability analysis also showed PaLM API pricing as highly cost-efficient. The findings indicate LLMs can effectively augment human content moderation at scale to help mitigate emerging online risks. However, biases, transparency, and ethical oversight remain vital considerations before real-world implementation.
Paper Structure (26 sections, 4 equations, 4 figures, 4 tables)

This paper contains 26 sections, 4 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Recent Trends in Online Threats in America
  4. Trends in South Korea: Tragedy at Sillim Station
  5. The Rise of Large Language Models: What are They?
  6. Popularity and Affordability of Large Language Models
  7. Cases of Using LLM to Moderate Content
  8. Limitations of Human Content Moderation
  9. Ethical Considerations
  10. Future Directions
  11. Methodology
  12. Research Design
  13. Data Collection and Analysis
  14. Prompt Engineering
  15. Evaluation Criteria
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Example Public Threat Post Titles Collected namu_silim_incidents
  • Figure 2: Non-threat Accuracy Comparison
  • Figure 3: Threat Accuracy Comparison
  • Figure 4: Screenshot of Manually Interpreting PaLM API's Result