SpokeN-100: A Cross-Lingual Benchmarking Dataset for The Classification of Spoken Numbers in Different Languages

TL;DR

SpokeN-100 addresses the need for a cross-lingual TinyDL benchmark by delivering a synthetic dataset of spoken numbers from $0$ to $99$ across English, German, French, and Mandarin, built from a language model and text-to-speech with $12{,}800$ samples from $32$ speakers. It establishes two classification tasks (language and number) and evaluates a range of baselines alongside an evolutionary neural architecture search (EvoNAS) workflow to produce MCU-deployable models on an ARM Cortex-M4 device, with careful data quality checks using $F_0$ analysis and UMAP-based diversity visualization. Key contributions include a fully synthetic, linguistically diverse dataset with standardized segmentation and quality controls, baseline and NAS-based benchmarks for TinyDL, and practical on-device models with measured latency (e.g., $\approx 368$ ms) and limited numbers accuracy due to hardware constraints. SpokeN-100 enables cross-lingual, edge-focused speech recognition research and provides a scalable benchmark for NAS and TinyDL on microcontrollers, with potential for future expansions to more languages and speakers.

Abstract

Benchmarking plays a pivotal role in assessing and enhancing the performance of compact deep learning models designed for execution on resource-constrained devices, such as microcontrollers. Our study introduces a novel, entirely artificially generated benchmarking dataset tailored for speech recognition, representing a core challenge in the field of tiny deep learning. SpokeN-100 consists of spoken numbers from 0 to 99 spoken by 32 different speakers in four different languages, namely English, Mandarin, German and French, resulting in 12,800 audio samples. We determine auditory features and use UMAP (Uniform Manifold Approximation and Projection for Dimension Reduction) as a dimensionality reduction method to show the diversity and richness of the dataset. To highlight the use case of the dataset, we introduce two benchmark tasks: given an audio sample, classify (i) the used language and/or (ii) the spoken number. We optimized state-of-the-art deep neural networks and performed an evolutionary neural architecture search to find tiny architectures optimized for the 32-bit ARM Cortex-M4 nRF52840 microcontroller. Our results represent the first benchmark data achieved for SpokeN-100.

Figures (3)

• Figure 1: Overview of this study as a flow diagram.
• Figure 2: Descriptive statistics of the dataset. a) Mean audio sample length of each spoken number across all speakers for four different languages. The shaded marks the standard deviation. b) Determined fundamental frequencies $F_0$ for each speaker. $F_0$ was calculated for each audio sample and then averaged across all speakers. The shaded area indicates the standard deviation.
• Figure 3: Low dimensional visualization of all audio samples with UMAP. a) UMAP embedding color-coded by language. b) UMAP embedding color-coded by speaker. Each speaker is assigned a value between 0 and 1 to create a gradient visualization. c) UMAP embedding color-coded by spoken number.