Interpreting Multimodal Communication at Scale in Short-Form Video: Visual, Audio, and Textual Mental Health Discourse on TikTok

TL;DR

The study tackles interpreting how text, visuals, and audio jointly influence engagement in short-form videos by proposing a scalable, interpretable multimodal pipeline that uses zero-shot feature extraction and SHAP-based attribution. The SHAP framework decomposes predictions as $f(x) = E[f(X)] + \,\sum_i \phi_i(x)$ and includes interaction terms $f(x) = E[f(X)] + \,\sum_i \phi_{ii}(x) + \,\sum_{i<j} \phi_{ij}(x)$, enabling measurement of both individual modality effects and cross-modal synergies. Applied to 162,965 TikTok videos about social anxiety disorder, the approach finds facial expressions predict engagement more reliably than textual sentiment, informational content attracts more attention than emotional support, and cross-modal interactions exhibit threshold-dependent effects (e.g., meme formats amplifying humor only when strongly present). The authors contribute a reproducible, theory-driven framework for interpretable multimodal analysis with implications for mental health communication in algorithmically mediated environments and for broader social science research across domains.

Abstract

Short-form video platforms integrate text, visuals, and audio into complex communicative acts, yet existing research analyzes these modalities in isolation, lacking scalable frameworks to interpret their joint contributions. This study introduces a pipeline combining automated multimodal feature extraction with Shapley value-based interpretability to analyze how text, visuals, and audio jointly influence engagement. Applying this framework to 162,965 TikTok videos and 814,825 images about social anxiety disorder (SAD), we find that facial expressions outperform textual sentiment in predicting viewership, informational content drives more attention than emotional support, and cross-modal synergies exhibit threshold-dependent effects. These findings demonstrate how multimodal analysis reveals interaction patterns invisible to single-modality approaches. Methodologically, we contribute a reproducible framework for interpretable multimodal research applicable across domains; substantively, we advance understanding of mental health communication in algorithmically mediated environments.

Figures (9)

• Figure 1: Overview of the multimodal analysis pipeline
• Figure 2: a) SHAP beeswarm plot of features from the visual, text, and audio modalities and b) point-weighted aggregate SHAP values as a forest plot.
• Figure 3: Mean directional SHAP contributions for textual (blue), audio (red), and visual (green) features.
• Figure 4: SHAP interaction plots illustrating cross-modal feature effects on viewership. Y-axis shows SHAP Values. X-axis shows one feature's magnitude. Blue represents low feature magnitude and red represents high magnitude.
• Figure 5: Piecewise regression coefficients from SHAP interaction values across quadrants. Each interaction pair is represented by a distinct marker. Four quadrants represent different combinations of feature values: (-, -) indicates both features below median threshold; (+, +) indicates both above threshold; (-, +) and (+, -) represent mixed conditions where one feature is above and one below.