Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Semi-Supervised Diseased Detection from Speech Dialogues with Multi-Level Data Modeling

Xingyuan Li, Mengyue Wu

TL;DR

This work tackles disease detection from long-form clinical dialogues by formulating a three-level, audio-only semi-supervised framework that jointly learns session-, clip-, and frame-level representations and updates high-quality pseudo-labels online. The method processes entire dialogues in one pass, uses a Transformer-based session encoder, an RNN for clip-level refinement, and a Siamese EMA setup for frame-level consistency, all optimized in a single-stage loss. It demonstrates strong data efficiency, achieving near fully-supervised performance with a fraction of labeled data, and shows robustness across languages and encoders while maintaining competitive fully-supervised results. The approach avoids multi-modal reliance, enabling more scalable, language-agnostic deployment in real-world clinical settings.

Abstract

Detecting medical conditions from speech acoustics is fundamentally a weakly-supervised learning problem: a single, often noisy, session-level label must be linked to nuanced patterns within a long, complex audio recording. This task is further hampered by severe data scarcity and the subjective nature of clinical annotations. While semi-supervised learning (SSL) offers a viable path to leverage unlabeled data, existing audio methods often fail to address the core challenge that pathological traits are not uniformly expressed in a patient's speech. We propose a novel, audio-only SSL framework that explicitly models this hierarchy by jointly learning from frame-level, segment-level, and session-level representations within unsegmented clinical dialogues. Our end-to-end approach dynamically aggregates these multi-granularity features and generates high-quality pseudo-labels to efficiently utilize unlabeled data. Extensive experiments show the framework is model-agnostic, robust across languages and conditions, and highly data-efficient-achieving, for instance, 90\% of fully-supervised performance using only 11 labeled samples. This work provides a principled approach to learning from weak, far-end supervision in medical speech analysis.

Semi-Supervised Diseased Detection from Speech Dialogues with Multi-Level Data Modeling

TL;DR

This work tackles disease detection from long-form clinical dialogues by formulating a three-level, audio-only semi-supervised framework that jointly learns session-, clip-, and frame-level representations and updates high-quality pseudo-labels online. The method processes entire dialogues in one pass, uses a Transformer-based session encoder, an RNN for clip-level refinement, and a Siamese EMA setup for frame-level consistency, all optimized in a single-stage loss. It demonstrates strong data efficiency, achieving near fully-supervised performance with a fraction of labeled data, and shows robustness across languages and encoders while maintaining competitive fully-supervised results. The approach avoids multi-modal reliance, enabling more scalable, language-agnostic deployment in real-world clinical settings.

Abstract

Detecting medical conditions from speech acoustics is fundamentally a weakly-supervised learning problem: a single, often noisy, session-level label must be linked to nuanced patterns within a long, complex audio recording. This task is further hampered by severe data scarcity and the subjective nature of clinical annotations. While semi-supervised learning (SSL) offers a viable path to leverage unlabeled data, existing audio methods often fail to address the core challenge that pathological traits are not uniformly expressed in a patient's speech. We propose a novel, audio-only SSL framework that explicitly models this hierarchy by jointly learning from frame-level, segment-level, and session-level representations within unsegmented clinical dialogues. Our end-to-end approach dynamically aggregates these multi-granularity features and generates high-quality pseudo-labels to efficiently utilize unlabeled data. Extensive experiments show the framework is model-agnostic, robust across languages and conditions, and highly data-efficient-achieving, for instance, 90\% of fully-supervised performance using only 11 labeled samples. This work provides a principled approach to learning from weak, far-end supervision in medical speech analysis.
Paper Structure (23 sections, 7 equations, 3 figures, 6 tables)

This paper contains 23 sections, 7 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Methods
  4. Session-level
  5. Clip-level
  6. Frame-level
  7. online single-stage training recipe
  8. Experimental Setup
  9. Results and Analysis
  10. Semi-supervised and fully-supervised setting results
  11. Data Efficiency.
  12. Performance Gains.
  13. Full Supervision Enhancement.
  14. Comparisons with existing works.
  15. Pseudo-label Analysis.
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Pathological speech detection in clinical diagnostic dialogues.
  • Figure 2: Model architecture overview. Our framework operates at three hierarchical levels: session-level (global dialogue representation via Transformer), clip-level (utterance-level modeling via RNN with pseudo-labels), and frame-level (acoustic feature consistency via MSE loss). The teacher-student framework with Exponential Moving Average (EMA) updates enables dynamic pseudo-label refinement during training.
  • Figure 3: Evolution of pseudo-label accuracy during training on depression detection.