Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-modal Knowledge Transfer Learning as Graph Matching Based on Optimal Transport for ASR

Xugang Lu, Peng Shen, Yu Tsao, Hisashi Kawai

TL;DR

The paper tackles cross-modal linguistic knowledge transfer from pretrained language models to end-to-end ASR by modeling utterances as ordered graphs and aligning acoustic and linguistic graphs via Graph Matching Optimal Transport (GM-OT). GM-OT combines node-level Wasserstein distance and edge-level Gromov-Wasserstein distance into a fused objective L_{FGWD} = (1-\alpha)L_{WD} + \alpha L_{GWD}, with temporal consistency constraints to preserve sequence structure, solved efficiently via Sinkhorn iterations. The approach unifies and subsumes prior OT-based alignment methods as special cases and demonstrates significant CER improvements on AISHELL-1 for a CTC-based ASR system with PLM knowledge transfer. This structured, graph-aware cross-modal transfer reduces reliance on external LMs and enhances knowledge transfer efficiency, offering a principled framework for future cross-modal sequence learning.

Abstract

Transferring linguistic knowledge from a pretrained language model (PLM) to acoustic feature learning has proven effective in enhancing end-to-end automatic speech recognition (E2E-ASR). However, aligning representations between linguistic and acoustic modalities remains a challenge due to inherent modality gaps. Optimal transport (OT) has shown promise in mitigating these gaps by minimizing the Wasserstein distance (WD) between linguistic and acoustic feature distributions. However, previous OT-based methods overlook structural relationships, treating feature vectors as unordered sets. To address this, we propose Graph Matching Optimal Transport (GM-OT), which models linguistic and acoustic sequences as structured graphs. Nodes represent feature embeddings, while edges capture temporal and sequential relationships. GM-OT minimizes both WD (between nodes) and Gromov-Wasserstein distance (GWD) (between edges), leading to a fused Gromov-Wasserstein distance (FGWD) formulation. This enables structured alignment and more efficient knowledge transfer compared to existing OT-based approaches. Theoretical analysis further shows that prior OT-based methods in linguistic knowledge transfer can be viewed as a special case within our GM-OT framework. We evaluate GM-OT on Mandarin ASR using a CTC-based E2E-ASR system with a PLM for knowledge transfer. Experimental results demonstrate significant performance gains over state-of-the-art models, validating the effectiveness of our approach.

Cross-modal Knowledge Transfer Learning as Graph Matching Based on Optimal Transport for ASR

TL;DR

The paper tackles cross-modal linguistic knowledge transfer from pretrained language models to end-to-end ASR by modeling utterances as ordered graphs and aligning acoustic and linguistic graphs via Graph Matching Optimal Transport (GM-OT). GM-OT combines node-level Wasserstein distance and edge-level Gromov-Wasserstein distance into a fused objective L_{FGWD} = (1-\alpha)L_{WD} + \alpha L_{GWD}, with temporal consistency constraints to preserve sequence structure, solved efficiently via Sinkhorn iterations. The approach unifies and subsumes prior OT-based alignment methods as special cases and demonstrates significant CER improvements on AISHELL-1 for a CTC-based ASR system with PLM knowledge transfer. This structured, graph-aware cross-modal transfer reduces reliance on external LMs and enhances knowledge transfer efficiency, offering a principled framework for future cross-modal sequence learning.

Abstract

Transferring linguistic knowledge from a pretrained language model (PLM) to acoustic feature learning has proven effective in enhancing end-to-end automatic speech recognition (E2E-ASR). However, aligning representations between linguistic and acoustic modalities remains a challenge due to inherent modality gaps. Optimal transport (OT) has shown promise in mitigating these gaps by minimizing the Wasserstein distance (WD) between linguistic and acoustic feature distributions. However, previous OT-based methods overlook structural relationships, treating feature vectors as unordered sets. To address this, we propose Graph Matching Optimal Transport (GM-OT), which models linguistic and acoustic sequences as structured graphs. Nodes represent feature embeddings, while edges capture temporal and sequential relationships. GM-OT minimizes both WD (between nodes) and Gromov-Wasserstein distance (GWD) (between edges), leading to a fused Gromov-Wasserstein distance (FGWD) formulation. This enables structured alignment and more efficient knowledge transfer compared to existing OT-based approaches. Theoretical analysis further shows that prior OT-based methods in linguistic knowledge transfer can be viewed as a special case within our GM-OT framework. We evaluate GM-OT on Mandarin ASR using a CTC-based E2E-ASR system with a PLM for knowledge transfer. Experimental results demonstrate significant performance gains over state-of-the-art models, validating the effectiveness of our approach.
Paper Structure (15 sections, 18 equations, 3 figures, 2 tables)

This paper contains 15 sections, 18 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed method
  3. Feature exploration in acoustic and linguistic encoders
  4. Feature alignment based on Optimal Transport
  5. Matching on nodes: Minimizing Wasserstein distance
  6. Matching on edges: Minimizing Gromov-Wasserstein distance
  7. Matching on ordered graph: Minimizing Fused WD and GWD
  8. Temporal consistency in graph topology matching
  9. Loss functions in knowledge transfer learning
  10. Experiments and results
  11. Model architecture
  12. Hyper-parameters in model learning
  13. Behaviors of the GM-OT based feature alignments
  14. ASR results
  15. Conclusion and future work

Figures (3)

  • Figure 1: The proposed cross-modal alignment and knowledge transfer model for ASR based on GM-OT.
  • Figure 2: Feature alignment between acoustic and linguistic modalities, (a) OT on unordered set, (b) OT on set with pair-wised relevance, (c) OT on set with temporal consistent topology structure.
  • Figure 3: Coupling between acoustic feature and linguistic tokens with changing different parameters, (a) weighting between WD and GWD (varying $\alpha$), (b) controlling temporal consistency between cross-modalities (varying $\rho$) (fixed $\alpha=0.1$), (c) with entropy regularization (varying $\beta$) (fixed $\alpha=0.1$, $\rho=0.5$).