Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction

Yuwei Zhang, Tong Xia, Aaqib Saeed, Cecilia Mascolo

TL;DR

RespLLM is proposed, a novel multimodal large language model (LLM) framework that unifies text and audio representations for respiratory health prediction and leverages the extensive prior knowledge of pretrained LLMs and enables effective audio-text fusion through cross-modal attentions.

Abstract

The high incidence and mortality rates associated with respiratory diseases underscores the importance of early screening. Machine learning models can automate clinical consultations and auscultation, offering vital support in this area. However, the data involved, spanning demographics, medical history, symptoms, and respiratory audio, are heterogeneous and complex. Existing approaches are insufficient and lack generalizability, as they typically rely on limited training data, basic fusion techniques, and task-specific models. In this paper, we propose RespLLM, a novel multimodal large language model (LLM) framework that unifies text and audio representations for respiratory health prediction. RespLLM leverages the extensive prior knowledge of pretrained LLMs and enables effective audio-text fusion through cross-modal attentions. Instruction tuning is employed to integrate diverse data from multiple sources, ensuring generalizability and versatility of the model. Experiments on five real-world datasets demonstrate that RespLLM outperforms leading baselines by an average of 4.6% on trained tasks, 7.9% on unseen datasets, and facilitates zero-shot predictions for new tasks. Our work lays the foundation for multimodal models that can perceive, listen to, and understand heterogeneous data, paving the way for scalable respiratory health diagnosis.

RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction

TL;DR

RespLLM is proposed, a novel multimodal large language model (LLM) framework that unifies text and audio representations for respiratory health prediction and leverages the extensive prior knowledge of pretrained LLMs and enables effective audio-text fusion through cross-modal attentions.

Abstract

The high incidence and mortality rates associated with respiratory diseases underscores the importance of early screening. Machine learning models can automate clinical consultations and auscultation, offering vital support in this area. However, the data involved, spanning demographics, medical history, symptoms, and respiratory audio, are heterogeneous and complex. Existing approaches are insufficient and lack generalizability, as they typically rely on limited training data, basic fusion techniques, and task-specific models. In this paper, we propose RespLLM, a novel multimodal large language model (LLM) framework that unifies text and audio representations for respiratory health prediction. RespLLM leverages the extensive prior knowledge of pretrained LLMs and enables effective audio-text fusion through cross-modal attentions. Instruction tuning is employed to integrate diverse data from multiple sources, ensuring generalizability and versatility of the model. Experiments on five real-world datasets demonstrate that RespLLM outperforms leading baselines by an average of 4.6% on trained tasks, 7.9% on unseen datasets, and facilitates zero-shot predictions for new tasks. Our work lays the foundation for multimodal models that can perceive, listen to, and understand heterogeneous data, paving the way for scalable respiratory health diagnosis.
Paper Structure (29 sections, 5 figures, 7 tables)

This paper contains 29 sections, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. ML for Respiratory Health
  4. LLMs for Health
  5. Methodology
  6. Model Architecture
  7. Text embedding.
  8. Audio Encoder with Projector.
  9. LLM and LoRA.
  10. Model Training
  11. Data Curation.
  12. Instruction Tuning.
  13. Zero-shot Prediction.
  14. Experiments
  15. Datasets and Tasks
  16. ...and 14 more sections

Figures (5)

  • Figure 1: Automated consultation and auscultation for respiratory health screening.
  • Figure 2: Multimodal models for respiratory health prediction. (a) Existing concatenation-based fusion method. (b) Our LLM-based fusion method.
  • Figure 3: Existing DMS encoding methods. (a) Pre-defined mapping. (b) Text embedding.
  • Figure 4: The model architecture of RespLLM. Text embeddings from task prompts and personal DMS, along with audio embeddings from respiratory sounds, are sequentialized as input for the LLM consisting of multiple transformer blocks.
  • Figure 5: Examples of instructions used in our work. The variables that differ across samples and datasets are highlighted. For any missing data in a field, the corresponding description is omitted.