Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pre-Trained Foundation Model representations to uncover Breathing patterns in Speech

Vikramjit Mitra, Anirban Chatterjee, Ke Zhai, Helen Weng, Ayuko Hill, Nicole Hay, Christopher Webb, Jamie Cheng, Erdrin Azemi

TL;DR

Speech-based estimation of respiration time-series data from speech segments obtained from subjects speaking to a close-talking microphone device is investigated and the use of pre-trained representations obtained from a foundation model can be used to estimate respiration-time-series with low root-mean-squared error and high correlation coefficient.

Abstract

The process of human speech production involves coordinated respiratory action to elicit acoustic speech signals. Typically, speech is produced when air is forced from the lungs and is modulated by the vocal tract, where such actions are interspersed by moments of breathing in air (inhalation) to refill the lungs again. Respiratory rate (RR) is a vital metric that is used to assess the overall health, fitness, and general well-being of an individual. Existing approaches to measure RR (number of breaths one takes in a minute) are performed using specialized equipment or training. Studies have demonstrated that machine learning algorithms can be used to estimate RR using bio-sensor signals as input. Speech-based estimation of RR can offer an effective approach to measure the vital metric without requiring any specialized equipment or sensors. This work investigates a machine learning based approach to estimate RR from speech segments obtained from subjects speaking to a close-talking microphone device. Data were collected from N=26 individuals, where the groundtruth RR was obtained through commercial grade chest-belts and then manually corrected for any errors. A convolutional long-short term memory network (Conv-LSTM) is proposed to estimate respiration time-series data from the speech signal. We demonstrate that the use of pre-trained representations obtained from a foundation model, such as Wav2Vec2, can be used to estimate respiration-time-series with low root-mean-squared error and high correlation coefficient, when compared with the baseline. The model-driven time series can be used to estimate $RR$ with a low mean absolute error (MAE) ~ 1.6 breaths/min.

Pre-Trained Foundation Model representations to uncover Breathing patterns in Speech

TL;DR

Speech-based estimation of respiration time-series data from speech segments obtained from subjects speaking to a close-talking microphone device is investigated and the use of pre-trained representations obtained from a foundation model can be used to estimate respiration-time-series with low root-mean-squared error and high correlation coefficient.

Abstract

The process of human speech production involves coordinated respiratory action to elicit acoustic speech signals. Typically, speech is produced when air is forced from the lungs and is modulated by the vocal tract, where such actions are interspersed by moments of breathing in air (inhalation) to refill the lungs again. Respiratory rate (RR) is a vital metric that is used to assess the overall health, fitness, and general well-being of an individual. Existing approaches to measure RR (number of breaths one takes in a minute) are performed using specialized equipment or training. Studies have demonstrated that machine learning algorithms can be used to estimate RR using bio-sensor signals as input. Speech-based estimation of RR can offer an effective approach to measure the vital metric without requiring any specialized equipment or sensors. This work investigates a machine learning based approach to estimate RR from speech segments obtained from subjects speaking to a close-talking microphone device. Data were collected from N=26 individuals, where the groundtruth RR was obtained through commercial grade chest-belts and then manually corrected for any errors. A convolutional long-short term memory network (Conv-LSTM) is proposed to estimate respiration time-series data from the speech signal. We demonstrate that the use of pre-trained representations obtained from a foundation model, such as Wav2Vec2, can be used to estimate respiration-time-series with low root-mean-squared error and high correlation coefficient, when compared with the baseline. The model-driven time series can be used to estimate with a low mean absolute error (MAE) ~ 1.6 breaths/min.
Paper Structure (10 sections, 4 equations, 6 figures, 3 tables)

This paper contains 10 sections, 4 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Data
  3. Analysis
  4. Methods
  5. Acoustic Features
  6. Features from Pre-Trained Models
  7. Model
  8. Salient representations
  9. Results
  10. Conclusion

Figures (6)

  • Figure 1: Spectrogram speech [top] and the corresponding chest-belt pressure measurement (in Newton) [bottom].
  • Figure 2: Histogram of $RR$ (in br/min) estimated from chest-belt data in the dataset.
  • Figure 3: Mean and std-dev $RR$ (in br/min) by speakers.
  • Figure 4: (A) Architecture of the single-feature (Conv-LSTM) network, and (B) Feature-fused network
  • Figure 5: Segment-level performance by number of LSTM layers for models trained with MFB and Wav2Vec2 features
  • ...and 1 more figures