Tri-Modal Severity Fused Diagnosis across Depression and Post-traumatic Stress Disorders

TL;DR

The paper introduces the first tri-modal framework for joint, severity-aware diagnosis of depression and PTSD, integrating text, audio, and facial cues to produce graded severities for PHQ-8 and PTSD with calibrated probabilities. It employs compact feature extractors (text via MPNet sentence embeddings, audio via log-Mel statistics, and OpenFace-based facial descriptors) fused through a late, calibrated classifier that provides per-disorder predictions and feature attributions. Evaluated on DAIC-WOZ and E-DAIC with stratified cross-validation, the approach achieves competitive accuracy while delivering improved decision-curve utility and robustness to missing modalities, and demonstrates distinct modality contributions for each disorder. SHAP-based interpretability links predictive cues to linguistics and behavior, supporting clinician trust and potential human-in-the-loop deployment, with reproducible pipelines and public code release. The work advances clinical utility by moving beyond binary detection to severity-aware, multi-disorder assessment suitable for triage and stepped-care decision support in mental health care.

Abstract

Depression and post traumatic stress disorder (PTSD) often co-occur with connected symptoms, complicating automated assessment, which is often binary and disorder specific. Clinically useful diagnosis needs severity aware cross disorder estimates and decision support explanations. Our unified tri modal affective severity framework synchronizes and fuses interview text with sentence level transformer embeddings, audio with log Mel statistics with deltas, and facial signals with action units, gaze, head and pose descriptors to output graded severities for diagnosing both depression (PHQ-8; 5 classes) and PTSD (3 classes). Standardized features are fused via a calibrated late fusion classifier, yielding per disorder probabilities and feature-level attributions. This severity aware tri-modal affective fusion approach is demoed on multi disorder concurrent depression and PTSD assessment. Stratified cross validation on DAIC derived corpora outperforms unimodal/ablation baselines. The fused model matches the strongest unimodal baseline on accuracy and weighted F1, while improving decision curve utility and robustness under noisy or missing modalities. For PTSD specifically, fusion reduces regression error and improves class concordance. Errors cluster between adjacent severities; extreme classes are identified reliably. Ablations show text contributes most to depression severity, audio and facial cues are critical for PTSD, whereas attributions align with linguistic and behavioral markers. Our approach offers reproducible evaluation and clinician in the loop support for affective clinical decision making.

Figures (12)

• Figure 1: Architecture overview. Text (768-D), audio (256-D), and face (512-D) embeddings are standardized and fused (1,536-D) for boosted late-fusion classification (Depression, PTSD). Calibrated probabilities enable severity-aware diagnosis.
• Figure 2: Representative log-Mel spectrogram. Example extracted at 16 kHz with $n_{\text{mels}}=64$. Bright vertical stripes correspond to voiced segments with strong harmonic energy, while dark gaps reflect silences and low-energy speech.
• Figure 3: Per-class F1 for fusion models. Extremes (none/severe) are easier; middle tiers remain challenging.
• Figure 4: Confusion matrix Depression (fusion). Most errors occur between adjacent severities.
• Figure 5: Confusion matrix PTSD (fusion). Clear separation for severe PTSD; residual overlap in mild vs. moderate.