Edge-Based Speech Transcription and Synthesis for Kinyarwanda and Swahili Languages

TL;DR

The paper addresses the scarcity of robust speech processing tools for Kinyarwanda and Swahili in East Africa by proposing a cascaded edge-cloud framework that splits STT and TTS workloads between edge devices and the cloud. It uses Whisper for STT and SpeechT5 for TTS, with a cascading mechanism that dispatches to the cloud when edge inference fails to meet quality or latency thresholds. Data collected from Mozilla Common Voice and experiments on a 1.7 GHz edge device with limited bandwidth demonstrate modest edge memory footprints (9.5% for SpeechT5, 14% for Whisper) and practical latency improvements, supporting operation at bandwidths around 1 MB/s. The results indicate the approach can provide accurate and timely transcription and synthesis for low-resource languages in infrastructure-constrained settings, with potential applicability to other languages and deployment scenarios. Future work includes further model compression, broader data collection, and real-world deployment studies.

Abstract

This paper presents a novel framework for speech transcription and synthesis, leveraging edge-cloud parallelism to enhance processing speed and accessibility for Kinyarwanda and Swahili speakers. It addresses the scarcity of powerful language processing tools for these widely spoken languages in East African countries with limited technological infrastructure. The framework utilizes the Whisper and SpeechT5 pre-trained models to enable speech-to-text (STT) and text-to-speech (TTS) translation. The architecture uses a cascading mechanism that distributes the model inference workload between the edge device and the cloud, thereby reducing latency and resource usage, benefiting both ends. On the edge device, our approach achieves a memory usage compression of 9.5% for the SpeechT5 model and 14% for the Whisper model, with a maximum memory usage of 149 MB. Experimental results indicate that on a 1.7 GHz CPU edge device with a 1 MB/s network bandwidth, the system can process a 270-character text in less than a minute for both speech-to-text and text-to-speech transcription. Using real-world survey data from Kenya, it is shown that the cascaded edge-cloud architecture proposed could easily serve as an excellent platform for STT and TTS transcription with good accuracy and response time.

Figures (11)

• Figure 1: A generalized transformer architecture. The inputs are speech and text data for STT and TTS respectively. The outputs are text and speech for STT and TTS respectively. For both tasks, the decoder's inference is autoregressive
• Figure 2: The cascading architecture is presented. On the left, the edge-level model is used to produce output. When the output's quality does not meet a predefined threshold, the internal data representations will be sent to the cloud for enhanced processing.
• Figure 3: The Whisper architecture a simple end-to-end approach, implemented as an encoder-decoder Transformer Alec.
• Figure 4: The model architecture of SpeechT5, which contains an encoder-decoder module and six modal specific pre/post-netsJunyi.
• Figure 5: Simulation results of a 12-characters and a 270-characters Swahili text’s speech synthesis time against different bandwidth values. The CPU time is the time taken by the edge device in computation while the wall time is the total elapsed time.