Uncovering Misattributed Suicide Causes through Annotation Inconsistency Detection in Death Investigation Notes

TL;DR

This work tackles misattributed suicide causes arising from annotation inconsistencies in NVDRS death investigation notes. It presents a transformer-based NLP pipeline coupled with a cross-validation–like framework to detect inconsistencies, identify and manually rectify problematic annotations, and re-evaluate model performance across states. The findings reveal substantial cross-state inconsistencies, demonstrate that removing or correcting identified mistakes improves generalization (including cross-state F1 gains), and show demographic associations (odds ratios) can shift after correction, underscoring the importance of data quality. The approach enhances NVDRS data reliability, enabling more accurate inference of suicide causes and better-informed prevention policies, while acknowledging limitations such as computational cost and the need for scalable automatic correction methods.

Abstract

Data accuracy is essential for scientific research and policy development. The National Violent Death Reporting System (NVDRS) data is widely used for discovering the patterns and causes of death. Recent studies suggested the annotation inconsistencies within the NVDRS and the potential impact on erroneous suicide-cause attributions. We present an empirical Natural Language Processing (NLP) approach to detect annotation inconsistencies and adopt a cross-validation-like paradigm to identify problematic instances. We analyzed 267,804 suicide death incidents between 2003 and 2020 from the NVDRS. Our results showed that incorporating the target state's data into training the suicide-crisis classifier brought an increase of 5.4% to the F-1 score on the target state's test set and a decrease of 1.1% on other states' test set. To conclude, we demonstrated the annotation inconsistencies in NVDRS's death investigation notes, identified problematic instances, evaluated the effectiveness of correcting problematic instances, and eventually proposed an NLP improvement solution.

Figures (6)

• Figure 1: $\Delta \text{F1}\xspace$'s on the test sets of the target state ($\uparrow$) and other states ($\downarrow$). (A) Physical Health, (B) Family Relationship, (C) Mental Health Crisis.
• Figure 2: Prediction error count distributions (log scale) of Ohio and Colorado. (A) Physical Health, (B) Family Relationship, (C) Mental Health Crisis. Data instances with a prediction error count equal to 5 will be identified as potential mistakes.
• Figure 3: Comparison of F1 scores between models trained using 'Original' (before removing the identified potential mistakes), and 'PMs Removed' (after removing the identified potential mistakes). (A) Physical Health, (B) Family Relationship, (C) Mental Health Crisis. PMs - Potential Mistakes. The asterisk indicates statistical significance.
• Figure 4: Comparisons of average micro F1 scores for Family Relationship Crisis when we gradually feed more training data in an incremental manner to the model, (a) Other states’ test set, (b) Ohio's test set. In either subplot, the black vertical dashed line on the left denotes when Ohio's data have all been fed to the model for training data Ohio+Others and CorrectedOhio+Others, while the red vertical dashed line on the right denotes when we start to feed Ohio’s data to the model for Others+Ohio and Others+CorrectedOhio.
• Figure 5: Annotation inconsistency example and our proposed framework. In Step 1, the size of other states' $Train_2$ set equals the size of the target state's $Train$ set, ensuring the three new training sets are of the same size. In Step 2, the $k$-fold cross-validation procedure is repeated n times using different random seeds. For each data instance, we recorded its prediction error counts, and eventually identified the problematic instances by thresholding the prediction error counts. PMs - Potential Mistakes.