Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowledge Acquisition on Mass-shooting Events via LLMs for AI-Driven Justice

Benign John Ihugba, Afsana Nasrin, Ling Wu, Lin Li, Lijun Qian, Xishuang Dong

TL;DR

This paper tackles the problem of extracting structured knowledge from mass-shooting event narratives to support AI-driven justice. It constructs the first mass-shooting NER dataset with 41 subcategories across offense, offender, victim, and environment, sourced from Mother Jones and annotated via a BIO scheme. Using few-shot prompts, the authors benchmark GPT-3.5, GPT-4o, and o1-mini, finding that GPT-4o delivers the strongest Micro Precision, Recall, and F1, while o1-mini provides a more efficient alternative; performance generally improves with more shots. The work demonstrates the viability of LLM-based NER for domain-specific information extraction and points toward building knowledge graphs for investigations and policy, with future work focusing on minority-category extraction and relation mining for richer knowledge graphs.

Abstract

Mass-shooting events pose a significant challenge to public safety, generating large volumes of unstructured textual data that hinder effective investigations and the formulation of public policy. Despite the urgency, few prior studies have effectively automated the extraction of key information from these events to support legal and investigative efforts. This paper presented the first dataset designed for knowledge acquisition on mass-shooting events through the application of named entity recognition (NER) techniques. It focuses on identifying key entities such as offenders, victims, locations, and criminal instruments, that are vital for legal and investigative purposes. The NER process is powered by Large Language Models (LLMs) using few-shot prompting, facilitating the efficient extraction and organization of critical information from diverse sources, including news articles, police reports, and social media. Experimental results on real-world mass-shooting corpora demonstrate that GPT-4o is the most effective model for mass-shooting NER, achieving the highest Micro Precision, Micro Recall, and Micro F1-scores. Meanwhile, o1-mini delivers competitive performance, making it a resource-efficient alternative for less complex NER tasks. It is also observed that increasing the shot count enhances the performance of all models, but the gains are more substantial for GPT-4o and o1-mini, highlighting their superior adaptability to few-shot learning scenarios.

Knowledge Acquisition on Mass-shooting Events via LLMs for AI-Driven Justice

TL;DR

This paper tackles the problem of extracting structured knowledge from mass-shooting event narratives to support AI-driven justice. It constructs the first mass-shooting NER dataset with 41 subcategories across offense, offender, victim, and environment, sourced from Mother Jones and annotated via a BIO scheme. Using few-shot prompts, the authors benchmark GPT-3.5, GPT-4o, and o1-mini, finding that GPT-4o delivers the strongest Micro Precision, Recall, and F1, while o1-mini provides a more efficient alternative; performance generally improves with more shots. The work demonstrates the viability of LLM-based NER for domain-specific information extraction and points toward building knowledge graphs for investigations and policy, with future work focusing on minority-category extraction and relation mining for richer knowledge graphs.

Abstract

Mass-shooting events pose a significant challenge to public safety, generating large volumes of unstructured textual data that hinder effective investigations and the formulation of public policy. Despite the urgency, few prior studies have effectively automated the extraction of key information from these events to support legal and investigative efforts. This paper presented the first dataset designed for knowledge acquisition on mass-shooting events through the application of named entity recognition (NER) techniques. It focuses on identifying key entities such as offenders, victims, locations, and criminal instruments, that are vital for legal and investigative purposes. The NER process is powered by Large Language Models (LLMs) using few-shot prompting, facilitating the efficient extraction and organization of critical information from diverse sources, including news articles, police reports, and social media. Experimental results on real-world mass-shooting corpora demonstrate that GPT-4o is the most effective model for mass-shooting NER, achieving the highest Micro Precision, Micro Recall, and Micro F1-scores. Meanwhile, o1-mini delivers competitive performance, making it a resource-efficient alternative for less complex NER tasks. It is also observed that increasing the shot count enhances the performance of all models, but the gains are more substantial for GPT-4o and o1-mini, highlighting their superior adaptability to few-shot learning scenarios.

Paper Structure

This paper contains 15 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Task Definition
  3. Data Collection and Annotation
  4. Data Collection and Preparation
  5. Data Annotation
  6. LLMs and Prompt Engineering
  7. LLMs
  8. Prompt Engineering
  9. Experiments
  10. Evaluation Metrics
  11. Experimental Results and Discussions
  12. Related Work
  13. AI-Driven Justice
  14. LLM-Based NER
  15. Conclusion and Future Work

Figures (4)

  • Figure 1: An example of NER on a mass-shooting event for knowledge acquisition. It includes various types of named entities including Offender Name (ON), Offender Age (OA), Criminal Instrument (CI), Actions, Type of Premises Entered (TPE), Incident Date (I-Date), Incident Hour (I-Hour), Number of Deaths (ND), Agency Information (AI), Motivation, Shooter Suicidality (SS).
  • Figure 2: Distributions of named entities involved in mass-shooting events.
  • Figure 3: Prompt schema of NER on mass-shooting events.
  • Figure 4: Performance comparison regarding F1-score across different types of named entities with 14 shots.