Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Reasoning LLMs for Suicide Screening with the Columbia-Suicide Severity Rating Scale

Avinash Patil, Siru Tao, Amardeep Gedhu

TL;DR

This paper evaluates zero-shot large language models on automated suicide risk assessment using the Columbia-Suicide Severity Rating Scale (C-SSRS) for Reddit r/SuicideWatch posts. It compares Claude, GPT, Mistral, LLaMA, and Gemini across an ordinal 0–6 severity scale, with human annotations serving as ground truth; Claude and GPT align most closely with humans, while Mistral achieves the best ordinal calibration (lower error metrics). The study analyzes confusion patterns, error sources (notably adjacent-level misclassifications and intent vs. action distinctions), and ethical considerations, advocating for human-in-the-loop deployment and transparent reporting. Overall, the results demonstrate promising capabilities of zero-shot LLMs for scalable digital triage in online mental health contexts, while underscoring the need for interpretability, bias mitigation, and cross-platform validation before clinical or moderator deployment.

Abstract

Suicide prevention remains a critical public health challenge. While online platforms such as Reddit's r/SuicideWatch have historically provided spaces for individuals to express suicidal thoughts and seek community support, the advent of large language models (LLMs) introduces a new paradigm-where individuals may begin disclosing ideation to AI systems instead of humans. This study evaluates the capability of LLMs to perform automated suicide risk assessment using the Columbia-Suicide Severity Rating Scale (C-SSRS). We assess the zero-shot performance of six models-including Claude, GPT, Mistral, and LLaMA-in classifying posts across a 7-point severity scale (Levels 0-6). Results indicate that Claude and GPT closely align with human annotations, while Mistral achieves the lowest ordinal prediction error. Most models exhibit ordinal sensitivity, with misclassifications typically occurring between adjacent severity levels. We further analyze confusion patterns, misclassification sources, and ethical considerations, underscoring the importance of human oversight, transparency, and cautious deployment. Full code and supplementary materials are available at https://github.com/av9ash/llm_cssrs_code.

Evaluating Reasoning LLMs for Suicide Screening with the Columbia-Suicide Severity Rating Scale

TL;DR

This paper evaluates zero-shot large language models on automated suicide risk assessment using the Columbia-Suicide Severity Rating Scale (C-SSRS) for Reddit r/SuicideWatch posts. It compares Claude, GPT, Mistral, LLaMA, and Gemini across an ordinal 0–6 severity scale, with human annotations serving as ground truth; Claude and GPT align most closely with humans, while Mistral achieves the best ordinal calibration (lower error metrics). The study analyzes confusion patterns, error sources (notably adjacent-level misclassifications and intent vs. action distinctions), and ethical considerations, advocating for human-in-the-loop deployment and transparent reporting. Overall, the results demonstrate promising capabilities of zero-shot LLMs for scalable digital triage in online mental health contexts, while underscoring the need for interpretability, bias mitigation, and cross-platform validation before clinical or moderator deployment.

Abstract

Suicide prevention remains a critical public health challenge. While online platforms such as Reddit's r/SuicideWatch have historically provided spaces for individuals to express suicidal thoughts and seek community support, the advent of large language models (LLMs) introduces a new paradigm-where individuals may begin disclosing ideation to AI systems instead of humans. This study evaluates the capability of LLMs to perform automated suicide risk assessment using the Columbia-Suicide Severity Rating Scale (C-SSRS). We assess the zero-shot performance of six models-including Claude, GPT, Mistral, and LLaMA-in classifying posts across a 7-point severity scale (Levels 0-6). Results indicate that Claude and GPT closely align with human annotations, while Mistral achieves the lowest ordinal prediction error. Most models exhibit ordinal sensitivity, with misclassifications typically occurring between adjacent severity levels. We further analyze confusion patterns, misclassification sources, and ethical considerations, underscoring the importance of human oversight, transparency, and cautious deployment. Full code and supplementary materials are available at https://github.com/av9ash/llm_cssrs_code.
Paper Structure (30 sections, 3 figures, 1 table)

This paper contains 30 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Suicide Risk Assessment in NLP
  4. Large Language Models for Mental Health
  5. The Columbia-Suicide Severity Rating Scale (C-SSRS) in Computational Psychiatry and Risk Assessment
  6. Dataset
  7. Exploratory Data Analysis:
  8. Annotation and Labeling
  9. Methodology
  10. Experimental Setup
  11. Large Language Models
  12. Evaluation Metrics
  13. Results
  14. Overall Performance
  15. Confusion Matrix Analysis
  16. ...and 15 more sections

Figures (3)

  • Figure 1: C-SSRS: Suicide Risk Screening Tool
  • Figure 5: Word cloud visualizations (Severity 0-6).
  • Figure 6: Confusion matrices for language identification across seven machine translation models. Darker diagonal entries indicate correct classifications.