Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tuning In: Analysis of Audio Classifier Performance in Clinical Settings with Limited Data

Hamza Mahdi, Eptehal Nashnoush, Rami Saab, Arjun Balachandar, Rishit Dagli, Lucas X. Perri, Houman Khosravani

TL;DR

This work tackles the challenge of audio-based clinical diagnostics under limited real-world data. It compares CNNs, RNNs, and transformer architectures with diverse pre-training (Imagenet, AudioSet, US8K, ESC50) and preprocessing (Mel RGB, Mel mono, Superlets) on two stroke-related audio datasets (NIHSS-based speech and vowel sounds). CNNs, particularly DenseNet variants and ConvNeXt, often rival or surpass transformers in small-data regimes, with DenseNet-Contrastive US8K achieving exceptional specificity and F1, while AST demonstrates training efficiency and strong sensitivity. The study provides actionable guidance on model selection and preprocessing for clinical audio biomarkers, and introduces two novel datasets that enable future research in dysphagia and stroke assessment through audio signals.

Abstract

This study assesses deep learning models for audio classification in a clinical setting with the constraint of small datasets reflecting real-world prospective data collection. We analyze CNNs, including DenseNet and ConvNeXt, alongside transformer models like ViT, SWIN, and AST, and compare them against pre-trained audio models such as YAMNet and VGGish. Our method highlights the benefits of pre-training on large datasets before fine-tuning on specific clinical data. We prospectively collected two first-of-their-kind patient audio datasets from stroke patients. We investigated various preprocessing techniques, finding that RGB and grayscale spectrogram transformations affect model performance differently based on the priors they learn from pre-training. Our findings indicate CNNs can match or exceed transformer models in small dataset contexts, with DenseNet-Contrastive and AST models showing notable performance. This study highlights the significance of incremental marginal gains through model selection, pre-training, and preprocessing in sound classification; this offers valuable insights for clinical diagnostics that rely on audio classification.

Tuning In: Analysis of Audio Classifier Performance in Clinical Settings with Limited Data

TL;DR

This work tackles the challenge of audio-based clinical diagnostics under limited real-world data. It compares CNNs, RNNs, and transformer architectures with diverse pre-training (Imagenet, AudioSet, US8K, ESC50) and preprocessing (Mel RGB, Mel mono, Superlets) on two stroke-related audio datasets (NIHSS-based speech and vowel sounds). CNNs, particularly DenseNet variants and ConvNeXt, often rival or surpass transformers in small-data regimes, with DenseNet-Contrastive US8K achieving exceptional specificity and F1, while AST demonstrates training efficiency and strong sensitivity. The study provides actionable guidance on model selection and preprocessing for clinical audio biomarkers, and introduces two novel datasets that enable future research in dysphagia and stroke assessment through audio signals.

Abstract

This study assesses deep learning models for audio classification in a clinical setting with the constraint of small datasets reflecting real-world prospective data collection. We analyze CNNs, including DenseNet and ConvNeXt, alongside transformer models like ViT, SWIN, and AST, and compare them against pre-trained audio models such as YAMNet and VGGish. Our method highlights the benefits of pre-training on large datasets before fine-tuning on specific clinical data. We prospectively collected two first-of-their-kind patient audio datasets from stroke patients. We investigated various preprocessing techniques, finding that RGB and grayscale spectrogram transformations affect model performance differently based on the priors they learn from pre-training. Our findings indicate CNNs can match or exceed transformer models in small dataset contexts, with DenseNet-Contrastive and AST models showing notable performance. This study highlights the significance of incremental marginal gains through model selection, pre-training, and preprocessing in sound classification; this offers valuable insights for clinical diagnostics that rely on audio classification.
Paper Structure (33 sections, 1 figure, 3 tables)

This paper contains 33 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Key Contributions.
  3. Related Work
  4. Audio Classification
  5. Downstream Effects of Transforms
  6. Method
  7. Participants
  8. Data Collection
  9. Data Analysis
  10. Pre-training on Public Datasets
  11. Imagenet
  12. AudioSet
  13. US8K
  14. ESC50
  15. Exploring Network Architectures for Audio Analysis
  16. ...and 18 more sections

Figures (1)

  • Figure 1: Schematic Overview of the Audio Classification Workflow for Stroke Assessment. The process begins with the collection of voice recordings, including both sustained vowel sounds (/a/, /e/, /i/, /o/, /u/) and speech following the NIHSS, represented by icons for hand, feather, key, chair, etc. These recordings are then segmented using an overlapping patch method to prepare for pre-processing. Subsequently, audio segments are transformed into two types of visual representations: mel-spectrograms and superlet transforms. The final stage involves inputting the processed spectrograms into an array of machine learning models—CNNs, transformers, and RNNs—to predict the outcome of the TOR-BSST$\copyright$ as either "pass" or "fail".