Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing

Ziyang Ma, Guanrou Yang, Wenxi Chen, Zhifu Gao, Yexing Du, Xiquan Li, Zhisheng Zheng, Haina Zhu, Jianheng Zhuo, Zheshu Song, Ruiyang Xu, Tiranrui Wang, Yifan Yang, Yanqiao Zhu, Zhikang Niu, Liumeng Xue, Yinghao Ma, Ruibin Yuan, Shiliang Zhang, Kai Yu, Eng Siong Chng, Xie Chen

TL;DR

SLAM-LLM is an open-source deep learning framework designed to train customized MLLMs, focused on speech, language, audio, and music processing, and provides a modular configuration of different encoders, projectors, LLMs, and parameter-efficient fine-tuning plugins.

Abstract

The recent surge in open-source Multimodal Large Language Models (MLLM) frameworks, such as LLaVA, provides a convenient kickoff for artificial intelligence developers and researchers. However, most of the MLLM frameworks take vision as the main input modality, and provide limited in-depth support for the modality of speech, audio, and music. This situation hinders the development of audio-language models, and forces researchers to spend a lot of effort on code writing and hyperparameter tuning. We present SLAM-LLM, an open-source deep learning framework designed to train customized MLLMs, focused on speech, language, audio, and music processing. SLAM-LLM provides a modular configuration of different encoders, projectors, LLMs, and parameter-efficient fine-tuning plugins. SLAM-LLM also includes detailed training and inference recipes for mainstream tasks, along with high-performance checkpoints like LLM-based Automatic Speech Recognition (ASR), Automated Audio Captioning (AAC), and Music Captioning (MC). Some of these recipes have already reached or are nearing state-of-the-art performance, and some relevant techniques have also been accepted by academic papers. We hope SLAM-LLM will accelerate iteration, development, data engineering, and model training for researchers. We are committed to continually pushing forward audio-based MLLMs through this open-source framework, and call on the community to contribute to the LLM-based speech, audio and music processing.

SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing

TL;DR

SLAM-LLM is an open-source deep learning framework designed to train customized MLLMs, focused on speech, language, audio, and music processing, and provides a modular configuration of different encoders, projectors, LLMs, and parameter-efficient fine-tuning plugins.

Abstract

The recent surge in open-source Multimodal Large Language Models (MLLM) frameworks, such as LLaVA, provides a convenient kickoff for artificial intelligence developers and researchers. However, most of the MLLM frameworks take vision as the main input modality, and provide limited in-depth support for the modality of speech, audio, and music. This situation hinders the development of audio-language models, and forces researchers to spend a lot of effort on code writing and hyperparameter tuning. We present SLAM-LLM, an open-source deep learning framework designed to train customized MLLMs, focused on speech, language, audio, and music processing. SLAM-LLM provides a modular configuration of different encoders, projectors, LLMs, and parameter-efficient fine-tuning plugins. SLAM-LLM also includes detailed training and inference recipes for mainstream tasks, along with high-performance checkpoints like LLM-based Automatic Speech Recognition (ASR), Automated Audio Captioning (AAC), and Music Captioning (MC). Some of these recipes have already reached or are nearing state-of-the-art performance, and some relevant techniques have also been accepted by academic papers. We hope SLAM-LLM will accelerate iteration, development, data engineering, and model training for researchers. We are committed to continually pushing forward audio-based MLLMs through this open-source framework, and call on the community to contribute to the LLM-based speech, audio and music processing.
Paper Structure (53 sections, 5 figures, 17 tables)

This paper contains 53 sections, 5 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Multimodal Large Language Model Framework
  3. Design of SLAM-LLM
  4. Overview of SLAM-LLM
  5. Modular Design
  6. Task Support
  7. LLM-based Speech Processing
  8. Automatic Speech Recognition (ASR)
  9. Model setup
  10. Datasets
  11. Experiments on different supervised speech encoders
  12. Experiments on different self-supervised speech encoders
  13. Comparison with Non-LLM-based ASR models
  14. Experiments on minority language datasets
  15. Experiments on the code switching dataset
  16. ...and 38 more sections

Figures (5)

  • Figure 1: A brief workflow in the SLAM-LLM framework. By specifying model, training, and data configurations in a YAML file, customized requirements are met efficiently. The depicted example demonstrates an LLM-based Southeast Asian ASR model using a Southeast Asian language LLM Sailor-7B and corresponding speech dataset GigaSpeech 2.
  • Figure 2: SLAM-LLM modularizes the Encoder-Projector-LLM three-part components, which are assembled according to the configuration at training and inference time.
  • Figure 3: (a) LLM-based Visual Contextual ASR in SLAM-LLM framework. (b) LLM-based Contextual Biasing ASR in SLAM-LLM framework.
  • Figure 4: The model architecture of LLM-based visual speech recognition (VSR) in SLAM-LLM framework.
  • Figure 5: (a). LLM-based vanilla automated audio captioning in SLAM-LLM framework (b). LLM-based zero-shot automated audio captioning in SLAM-LLM framework