Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Performance and Complexity Trade-off Optimization of Speech Models During Training

Esteban Gómez, Tom Bäckström

TL;DR

This work tackles the problem of balancing model performance with computational cost during training by introducing differentiable feature noise injection and dynamic complexity layers that adjust network capacity in real time. By defining Dynamic, Adaptive, and Fixed Complexity Layers, the method enables SGD-based optimization to jointly tune accuracy and resource usage, consolidating to compact architectures for inference. Three case studies—synthetic filter banks, voice activity detection, and audio anti-spoofing—demonstrate substantial complexity reductions (up to ~84%) with minimal to substantial performance changes, including notable EER improvements in anti-spoofing. The approach yields practical benefits for on-device speech processing and can be combined with other efficiency techniques, with code openly available for further research.

Abstract

In speech machine learning, neural network models are typically designed by choosing an architecture with fixed layer sizes and structure. These models are then trained to maximize performance on metrics aligned with the task's objective. While the overall architecture is usually guided by prior knowledge of the task, the sizes of individual layers are often chosen heuristically. However, this approach does not guarantee an optimal trade-off between performance and computational complexity; consequently, post hoc methods such as weight quantization or model pruning are typically employed to reduce computational cost. This occurs because stochastic gradient descent (SGD) methods can only optimize differentiable functions, while factors influencing computational complexity, such as layer sizes and floating-point operations per second (FLOP/s), are non-differentiable and require modifying the model structure during training. We propose a reparameterization technique based on feature noise injection that enables joint optimization of performance and computational complexity during training using SGD-based methods. Unlike traditional pruning methods, our approach allows the model size to be dynamically optimized for a target performance-complexity trade-off, without relying on heuristic criteria to select which weights or structures to remove. We demonstrate the effectiveness of our method through three case studies, including a synthetic example and two practical real-world applications: voice activity detection and audio anti-spoofing. The code related to our work is publicly available to encourage further research.

Performance and Complexity Trade-off Optimization of Speech Models During Training

TL;DR

This work tackles the problem of balancing model performance with computational cost during training by introducing differentiable feature noise injection and dynamic complexity layers that adjust network capacity in real time. By defining Dynamic, Adaptive, and Fixed Complexity Layers, the method enables SGD-based optimization to jointly tune accuracy and resource usage, consolidating to compact architectures for inference. Three case studies—synthetic filter banks, voice activity detection, and audio anti-spoofing—demonstrate substantial complexity reductions (up to ~84%) with minimal to substantial performance changes, including notable EER improvements in anti-spoofing. The approach yields practical benefits for on-device speech processing and can be combined with other efficiency techniques, with code openly available for further research.

Abstract

In speech machine learning, neural network models are typically designed by choosing an architecture with fixed layer sizes and structure. These models are then trained to maximize performance on metrics aligned with the task's objective. While the overall architecture is usually guided by prior knowledge of the task, the sizes of individual layers are often chosen heuristically. However, this approach does not guarantee an optimal trade-off between performance and computational complexity; consequently, post hoc methods such as weight quantization or model pruning are typically employed to reduce computational cost. This occurs because stochastic gradient descent (SGD) methods can only optimize differentiable functions, while factors influencing computational complexity, such as layer sizes and floating-point operations per second (FLOP/s), are non-differentiable and require modifying the model structure during training. We propose a reparameterization technique based on feature noise injection that enables joint optimization of performance and computational complexity during training using SGD-based methods. Unlike traditional pruning methods, our approach allows the model size to be dynamically optimized for a target performance-complexity trade-off, without relying on heuristic criteria to select which weights or structures to remove. We demonstrate the effectiveness of our method through three case studies, including a synthetic example and two practical real-world applications: voice activity detection and audio anti-spoofing. The code related to our work is publicly available to encourage further research.
Paper Structure (27 sections, 9 equations, 8 figures)

This paper contains 27 sections, 9 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Model pruning
  4. Knowledge distillation
  5. Neural architecture search (NAS)
  6. Dynamic neural networks
  7. Proposed method
  8. Method
  9. Derivation
  10. Implementation details
  11. Case study 1: Dynamic complexity filter bank
  12. Introduction
  13. Experimental setup
  14. Results and discussion
  15. Case study 2: Voice activity detection
  16. ...and 12 more sections

Figures (8)

  • Figure 1: Dynamic (DCL), adaptive (ACL) and fixed (FCL) complexity layers. During training, DCL can modify their output shape, whereas ACL will adapt their input shape to remain compatible, and FCL have fixed shape during training. At inference time, all layers are consolidated to their final fixed shape.
  • Figure 2: Training progression of a single linear DCL to approximate a 32-band Bark filter bank. The left image shows early results after 100 steps. The middle image displays results after 3000 steps, where the estimation closely matches the target and the network's capacity is reduced by 30% ($\lambda=0.7$). The right image presents the target filter bank.
  • Figure 3: Overparameterization factor reduction for different increasing values of $\beta$, expressed as the relationship between input overparameterization factor and output overparameterization factor. Values along the gray dashed diagonal line indicate no overparameterization reduction.
  • Figure 4: Speech presence probability network. (a) represents the fixed complexity network (CRNNVAD), and (b) is the dynamic complexity variant (DynCRNNVAD). Blocks with dotted line borders can be resized during training as part of the complexity optimization process.
  • Figure 5: Comparison between CRNNVAD baseline and the DynCRNNVAD (DCRNNVAD) models in terms of complexity, performance, and filter bank size. The x-axis shows [bracket-unit-denominator = false], and the y-axis indicates EER on the VCTK+ESC-50 set. Numbers next to model figures denote filter bank bands. Numerical suffixes in model names denote minimum allowed complexity percentage. Silero VAD is also used as an additional benchmark, though its complexity is not reported.
  • ...and 3 more figures