Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OSUM-Pangu: An Open-Source Multidimension Speech Understanding Foundation Model Built upon OpenPangu on Ascend NPUs

Yujie Liao, Xuelong Geng, Hongfei Xue, Shuiyuan Wang, Lei Xie

TL;DR

OSUM-Pangu is presented, a fully open-source speech understanding foundation model developed on a completely non-CUDA software and hardware stack, providing a reproducible, non-CUDA baseline for the open-source speech community, promoting the independent evolution of multimodal intelligence.

Abstract

Recent advancements in Speech Large Language Models have significantly enhanced multi-dimensional speech understanding. However, the majority of high-performance frameworks are predominantly optimized for GPU centric ecosystems and proprietary backbones, creating a significant gap for deployment on non-CUDA computing infrastructures. In this paper, we present OSUM-Pangu, a fully open-source speech understanding foundation model developed on a completely non-CUDA software and hardware stack. By integrating an audio encoder with the openPangu-7B LLM backbone, we successfully implement the entire training and inference pipeline on the Ascend NPU platform. To facilitate efficient task alignment under non-CUDA resource constraints, we adopt a practical training process that sequentially bridges speech perception and user intent recognition. Experimental results demonstrate that OSUM-Pangu achieves task accuracy comparable to mainstream GPU-based models while maintaining robust natural language interaction capabilities. Our work provides a reproducible, non-CUDA baseline for the open-source speech community, promoting the independent evolution of multimodal intelligence.

OSUM-Pangu: An Open-Source Multidimension Speech Understanding Foundation Model Built upon OpenPangu on Ascend NPUs

TL;DR

OSUM-Pangu is presented, a fully open-source speech understanding foundation model developed on a completely non-CUDA software and hardware stack, providing a reproducible, non-CUDA baseline for the open-source speech community, promoting the independent evolution of multimodal intelligence.

Abstract

Recent advancements in Speech Large Language Models have significantly enhanced multi-dimensional speech understanding. However, the majority of high-performance frameworks are predominantly optimized for GPU centric ecosystems and proprietary backbones, creating a significant gap for deployment on non-CUDA computing infrastructures. In this paper, we present OSUM-Pangu, a fully open-source speech understanding foundation model developed on a completely non-CUDA software and hardware stack. By integrating an audio encoder with the openPangu-7B LLM backbone, we successfully implement the entire training and inference pipeline on the Ascend NPU platform. To facilitate efficient task alignment under non-CUDA resource constraints, we adopt a practical training process that sequentially bridges speech perception and user intent recognition. Experimental results demonstrate that OSUM-Pangu achieves task accuracy comparable to mainstream GPU-based models while maintaining robust natural language interaction capabilities. Our work provides a reproducible, non-CUDA baseline for the open-source speech community, promoting the independent evolution of multimodal intelligence.
Paper Structure (14 sections, 6 equations, 2 figures, 4 tables)

This paper contains 14 sections, 6 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Model Architecture
  5. Training
  6. Implementation on NPU Infrastructure
  7. Instruction Following
  8. Experiments
  9. Datasets
  10. Setup
  11. Main Results
  12. Instruction Following Results
  13. Speech-to-Text Chat Results
  14. Conclusions

Figures (2)

  • Figure 1: Illustration of the OSUM-Pangu model inference workflow. The architecture processes text instructions (left/right) and speech signals (middle) through a Transformer-based adapter and openPangu backbone, producing structured outputs with task-specific tags.
  • Figure 2: The three-stage training pipeline. Stage I: Tag-based speech alignment. Stage II: Pure text-based intent parsing. Stage III: Joint multimodal integration for intent-driven speech understanding.