Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bigger But Not Better: Small Neural Language Models Outperform Large Language Models in Detection of Thought Disorder

Changye Li, Weizhe Xu, Serguei Pakhomov, Ellen Bradley, Dror Ben-Zeev, Trevor Cohen

TL;DR

The paper demonstrates that smaller language models, when evaluated with sliding-window perplexity, can outperform larger language models in detecting positive Formal Thought Disorder from speech, challenging the notion that bigger is always better for clinical NLP. Using AVH diary transcripts and clinical interviews, the study shows that 64-token windows with mid-sized LMs yield the strongest correlations with clinician ratings (up to $\rho$ ≈ $0.486$ for TALD and substantial PANSS associations), while global perplexity offers limited diagnostic utility. The findings support a privacy-preserving, on-device approach to FTD screening that reduces computational and data-transfer burdens, enabling scalable deployment in diverse clinical and real-world settings. The work also highlights a non-linear relationship between model size, context window, and diagnostic sensitivity, suggesting calibrated combinations to optimize performance and resource use.

Abstract

Disorganized thinking is a key diagnostic indicator of schizophrenia-spectrum disorders. Recently, clinical estimates of the severity of disorganized thinking have been shown to correlate with measures of how difficult speech transcripts would be for large language models (LLMs) to predict. However, LLMs' deployment challenges -- including privacy concerns, computational and financial costs, and lack of transparency of training data -- limit their clinical utility. We investigate whether smaller neural language models can serve as effective alternatives for detecting positive formal thought disorder, using the same sliding window based perplexity measurements that proved effective with larger models. Surprisingly, our results show that smaller models are more sensitive to linguistic differences associated with formal thought disorder than their larger counterparts. Detection capability declines beyond a certain model size and context length, challenging the common assumption of ``bigger is better'' for LLM-based applications. Our findings generalize across audio diaries and clinical interview speech samples from individuals with psychotic symptoms, suggesting a promising direction for developing efficient, cost-effective, and privacy-preserving screening tools that can be deployed in both clinical and naturalistic settings.

Bigger But Not Better: Small Neural Language Models Outperform Large Language Models in Detection of Thought Disorder

TL;DR

The paper demonstrates that smaller language models, when evaluated with sliding-window perplexity, can outperform larger language models in detecting positive Formal Thought Disorder from speech, challenging the notion that bigger is always better for clinical NLP. Using AVH diary transcripts and clinical interviews, the study shows that 64-token windows with mid-sized LMs yield the strongest correlations with clinician ratings (up to ≈ for TALD and substantial PANSS associations), while global perplexity offers limited diagnostic utility. The findings support a privacy-preserving, on-device approach to FTD screening that reduces computational and data-transfer burdens, enabling scalable deployment in diverse clinical and real-world settings. The work also highlights a non-linear relationship between model size, context window, and diagnostic sensitivity, suggesting calibrated combinations to optimize performance and resource use.

Abstract

Disorganized thinking is a key diagnostic indicator of schizophrenia-spectrum disorders. Recently, clinical estimates of the severity of disorganized thinking have been shown to correlate with measures of how difficult speech transcripts would be for large language models (LLMs) to predict. However, LLMs' deployment challenges -- including privacy concerns, computational and financial costs, and lack of transparency of training data -- limit their clinical utility. We investigate whether smaller neural language models can serve as effective alternatives for detecting positive formal thought disorder, using the same sliding window based perplexity measurements that proved effective with larger models. Surprisingly, our results show that smaller models are more sensitive to linguistic differences associated with formal thought disorder than their larger counterparts. Detection capability declines beyond a certain model size and context length, challenging the common assumption of ``bigger is better'' for LLM-based applications. Our findings generalize across audio diaries and clinical interview speech samples from individuals with psychotic symptoms, suggesting a promising direction for developing efficient, cost-effective, and privacy-preserving screening tools that can be deployed in both clinical and naturalistic settings.

Paper Structure

This paper contains 19 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. FTD in schizophrenia
  4. Assessing FTD and SSDs with NLP methods
  5. Methods
  6. Data
  7. AVH Dataset
  8. Clinical Interview Dataset
  9. Language models
  10. Global and sliding window PPLs
  11. Results
  12. Global PPL as a proxy for FTD-related linguistic patterns
  13. Sliding window PPL performance
  14. The AVH dataset
  15. The clinical interview dataset
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Global PPLs estimated by the Pythia suite. The shaded area represents 95% confidence intervals of the global PPLs estimated from a pre-trained Pythia model.
  • Figure A.1: The distribution of sliding window PPLs using a sliding window of 64 tokens on the AVH dataset, where the x-axis represents the index of sliding window PPLs in transcripts. The shaded area represents the 95% confidence interval of the estimated sliding window PPL on a given index. The label of 0 is defined where TALD $<$ 3, serving as a proxy label for cognitively healthy individuals, whereas the label of 1 serves as the proxy label of FTD individuals.
  • Figure A.2: The distribution of sliding window PPLs using the sliding window of 16 tokens on the clinical interview dataset, where the x-axis represents the index of sliding window PPLs in transcripts. The shaded area represents the 95% confidence interval of the estimated sliding window PPL on a given index.