Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Textless Dependency Parsing by Labeled Sequence Prediction

Shunsuke Kando, Yusuke Miyao, Jason Naradowsky, Shinnosuke Takamichi

TL;DR

This paper investigates textless dependency parsing from speech and compares a cascading baseline (Wav2tree) with a proposed textless approach that directly predicts a labeled sequence representing the dependency tree from speech. The textless method uses a CTC loss to map speech representations to a sequence encoding words and dependency annotations, while Wav2tree first transcribes the speech and then parses word-level features. Experiments on French Orféo Treebank and English Switchboard show that Wav2tree generally excels, especially for long-distance dependencies, but the textless model gains when sentence-level prosody provides disambiguating cues, such as stress. The findings highlight the value of integrating both word-level representations and sentence-level prosody for robust spoken-language parsing and release public code and models to facilitate future work.

Abstract

Traditional spoken language processing involves cascading an automatic speech recognition (ASR) system into text processing models. In contrast, "textless" methods process speech representations without ASR systems, enabling the direct use of acoustic speech features. Although their effectiveness is shown in capturing acoustic features, it is unclear in capturing lexical knowledge. This paper proposes a textless method for dependency parsing, examining its effectiveness and limitations. Our proposed method predicts a dependency tree from a speech signal without transcribing, representing the tree as a labeled sequence. scading method outperforms the textless method in overall parsing accuracy, the latter excels in instances with important acoustic features. Our findings highlight the importance of fusing word-level representations and sentence-level prosody for enhanced parsing performance. The code and models are made publicly available: https://github.com/mynlp/SpeechParser.

Textless Dependency Parsing by Labeled Sequence Prediction

TL;DR

This paper investigates textless dependency parsing from speech and compares a cascading baseline (Wav2tree) with a proposed textless approach that directly predicts a labeled sequence representing the dependency tree from speech. The textless method uses a CTC loss to map speech representations to a sequence encoding words and dependency annotations, while Wav2tree first transcribes the speech and then parses word-level features. Experiments on French Orféo Treebank and English Switchboard show that Wav2tree generally excels, especially for long-distance dependencies, but the textless model gains when sentence-level prosody provides disambiguating cues, such as stress. The findings highlight the value of integrating both word-level representations and sentence-level prosody for robust spoken-language parsing and release public code and models to facilitate future work.

Abstract

Traditional spoken language processing involves cascading an automatic speech recognition (ASR) system into text processing models. In contrast, "textless" methods process speech representations without ASR systems, enabling the direct use of acoustic speech features. Although their effectiveness is shown in capturing acoustic features, it is unclear in capturing lexical knowledge. This paper proposes a textless method for dependency parsing, examining its effectiveness and limitations. Our proposed method predicts a dependency tree from a speech signal without transcribing, representing the tree as a labeled sequence. scading method outperforms the textless method in overall parsing accuracy, the latter excels in instances with important acoustic features. Our findings highlight the importance of fusing word-level representations and sentence-level prosody for enhanced parsing performance. The code and models are made publicly available: https://github.com/mynlp/SpeechParser.
Paper Structure (18 sections, 2 equations, 5 figures, 3 tables)

This paper contains 18 sections, 2 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Wav2tree: A Cascading Method
  3. ASR Module
  4. Dependency Parsing Module
  5. Handling ASR Errors in Training
  6. Textless Dependency Parsing
  7. Recovering Dependency Tree from Labeled Sequence
  8. Experimental Settings
  9. Dataset
  10. Preprocessing
  11. Model
  12. Optimization
  13. Evaluation
  14. Result and Discussion
  15. Analysis 1: Prediction Accuracy of Head Position
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Comparison of Wav2tree pupierEndtoEndDependencyParsing2022 (cascading) and the proposed method (textless). While Wav2tree includes an ASR module, our proposed method directly predicts a dependency tree (represented as a labeled sequence of tokens).
  • Figure 2: An example of a dependency tree. Each word is annotated with its part-of-speech, head, and dependency relation.
  • Figure 3: The rules for obtaining the oracle tree. The upper displays the gold dependency tree; the lower displays ASR results and oracles. Annotations added after rewriting are highlighted in red; deleted are in gray.
  • Figure 4: A labeled sequence representing a dependency tree in Figure \ref{['fig:dep-tree']} and its BPE tokenization. Spaces are indicated by "_".
  • Figure 5: Prediction accuracy of the relative position of the head (left: Orféo Treebank, right: Switchboard). Bars show log frequencies; lines show accuracies.