Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient data selection employing Semantic Similarity-based Graph Structures for model training

Roxana Petcu, Subhadeep Maji

TL;DR

Semantics for data SAliency in Model performance Estimation (SeSaME) is an efficient data sampling mechanism solely based on textual information without passing the data through a compute-heavy model or other intensive pre-processing transformations.

Abstract

Recent developments in natural language processing (NLP) have highlighted the need for substantial amounts of data for models to capture textual information accurately. This raises concerns regarding the computational resources and time required for training such models. This paper introduces Semantics for data SAliency in Model performance Estimation (SeSaME). It is an efficient data sampling mechanism solely based on textual information without passing the data through a compute-heavy model or other intensive pre-processing transformations. The application of this approach is demonstrated in the use case of low-resource automated speech recognition (ASR) models, which excessively rely on text-to-speech (TTS) calls when using augmented data. SeSaME learns to categorize new incoming data points into speech recognition difficulty buckets by employing semantic similarity-based graph structures and discrete ASR information from homophilous neighbourhoods through message passing. The results indicate reliable projections of ASR performance, with a 93% accuracy increase when using the proposed method compared to random predictions, bringing non-trivial information on the impact of textual representations in speech models. Furthermore, a series of experiments show both the benefits and challenges of using the ASR information on incoming data to fine-tune the model. We report a 7% drop in validation loss compared to random sampling, 7% WER drop with non-local aggregation when evaluating against a highly difficult dataset, and 1.8% WER drop with local aggregation and high semantic similarity between datasets.

Efficient data selection employing Semantic Similarity-based Graph Structures for model training

TL;DR

Semantics for data SAliency in Model performance Estimation (SeSaME) is an efficient data sampling mechanism solely based on textual information without passing the data through a compute-heavy model or other intensive pre-processing transformations.

Abstract

Recent developments in natural language processing (NLP) have highlighted the need for substantial amounts of data for models to capture textual information accurately. This raises concerns regarding the computational resources and time required for training such models. This paper introduces Semantics for data SAliency in Model performance Estimation (SeSaME). It is an efficient data sampling mechanism solely based on textual information without passing the data through a compute-heavy model or other intensive pre-processing transformations. The application of this approach is demonstrated in the use case of low-resource automated speech recognition (ASR) models, which excessively rely on text-to-speech (TTS) calls when using augmented data. SeSaME learns to categorize new incoming data points into speech recognition difficulty buckets by employing semantic similarity-based graph structures and discrete ASR information from homophilous neighbourhoods through message passing. The results indicate reliable projections of ASR performance, with a 93% accuracy increase when using the proposed method compared to random predictions, bringing non-trivial information on the impact of textual representations in speech models. Furthermore, a series of experiments show both the benefits and challenges of using the ASR information on incoming data to fine-tune the model. We report a 7% drop in validation loss compared to random sampling, 7% WER drop with non-local aggregation when evaluating against a highly difficult dataset, and 1.8% WER drop with local aggregation and high semantic similarity between datasets.
Paper Structure (9 sections, 2 equations, 2 figures, 5 tables)

This paper contains 9 sections, 2 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Train Pass
  5. Fine Tune Pass
  6. Experimental Setup
  7. Results and Discussion
  8. Conclusion
  9. Notations

Figures (2)

  • Figure 1: Train pass from left to right: Given dataset $\mathcal{D}_\text{train}$; Pass it to the TTS engine $\tau$ for transforming textual sentences into synthetic audio; Train ASR model $\alpha$; Compute WER on the predictions of $\alpha$; Bucket the observed WER into seven ordinal classes; Create a graph structure $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ with BERT embeddings as node features, bucketed WER as labels, and edges between semantically similar nodes; Train GNN $\gamma$ on the label recovery task.
  • Figure 2: Fine tune pass from left to right: Given new dataset $\mathcal{D}_\text{holdout}$; Add it to $\mathcal{G}$ to create a new graph structure $\bar{\mathcal{G}} = (\bar{\mathcal{V}}, \bar{\mathcal{E}})$; Make forward pass to the pre-trained GNN to retrieve labels for $\mathcal{D}_\text{holdout}$; Sample the nodes predicted as highly difficult points; Fine-tune $\alpha$ with the sampled data; Get WER performance and compare with baselines.