Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Multilingual Voice Toxicity Detection with Speech-Text Alignment

Joseph Liu, Mahesh Kumar Nandwana, Janne Pylkkönen, Hannes Heikinheimo, Morgan McGuire

TL;DR

This work proposes a novel framework that utilizes cross-modal learning to integrate the semantic embedding of text into a multilabel speech toxicity classifier during training that enables to incorporate textual information during training while still requiring only audio during inference.

Abstract

Toxicity classification for voice heavily relies on the semantic content of speech. We propose a novel framework that utilizes cross-modal learning to integrate the semantic embedding of text into a multilabel speech toxicity classifier during training. This enables us to incorporate textual information during training while still requiring only audio during inference. We evaluate this classifier on large-scale datasets with real-world characteristics to validate the effectiveness of this framework. Through ablation studies, we demonstrate that general-purpose semantic text embeddings are rich and aligned with speech for toxicity classification purposes. Conducting experiments across multiple languages at scale, we show improvements in voice toxicity classification across five languages and different toxicity categories.

Enhancing Multilingual Voice Toxicity Detection with Speech-Text Alignment

TL;DR

This work proposes a novel framework that utilizes cross-modal learning to integrate the semantic embedding of text into a multilabel speech toxicity classifier during training that enables to incorporate textual information during training while still requiring only audio during inference.

Abstract

Toxicity classification for voice heavily relies on the semantic content of speech. We propose a novel framework that utilizes cross-modal learning to integrate the semantic embedding of text into a multilabel speech toxicity classifier during training. This enables us to incorporate textual information during training while still requiring only audio during inference. We evaluate this classifier on large-scale datasets with real-world characteristics to validate the effectiveness of this framework. Through ablation studies, we demonstrate that general-purpose semantic text embeddings are rich and aligned with speech for toxicity classification purposes. Conducting experiments across multiple languages at scale, we show improvements in voice toxicity classification across five languages and different toxicity categories.
Paper Structure (17 sections, 4 equations, 2 figures, 3 tables)

This paper contains 17 sections, 4 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Data Pipeline
  3. Proposed System
  4. Model Architecture
  5. Speech Toxicity Classifier
  6. Text Encoder
  7. Text Injection Framework
  8. Experiments
  9. Training protocol
  10. Training & Evaluation Datasets
  11. Results
  12. Monolingual
  13. Text Injection Loss
  14. Optimal Loss Hyperparameters
  15. Optimal Layer Selection
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Schematic diagram of proposed text injection training pipeline.
  • Figure 2: Results comparing multilingual text injection against baseline across 5 languages along with their 95% confidence intervals.