Quantized Approximate Signal Processing (QASP): Towards Homomorphic Encryption for audio

TL;DR

This work tackles privacy risks in audio ML by enabling inference directly on encrypted raw audio using fully homomorphic encryption (FHE), preserving confidentiality during deployment. It introduces Quantized Approximate Signal Processing (QASP), a framework that reformulates time-frequency transforms (STFT, Mel filterbanks, MFCCs, gammatone) as quantized neural-network layers and pairs them with approximate STFT methods that provide theoretical error bounds under a 16-bit FHE budget. The authors validate their approach on VocalSet and OxVoc, showing that carefully chosen approximations (e.g., dilation, frequency-dependent windows, Poorman) reduce quantization error and enable private audio descriptors and CNN classifiers with performance close to clear computations. This work offers a practical path toward production-ready privacy-preserving audio analytics, albeit with current FHE-speed limitations that motivate further cryptographic advances.

Abstract

Audio and speech data are increasingly used in machine learning applications such as speech recognition, speaker identification, and mental health monitoring. However, the passive collection of this data by audio listening devices raises significant privacy concerns. Fully homomorphic encryption (FHE) offers a promising solution by enabling computations on encrypted data and preserving user privacy. Despite its potential, prior attempts to apply FHE to audio processing have faced challenges, particularly in securely computing time frequency representations, a critical step in many audio tasks. Here, we addressed this gap by introducing a fully secure pipeline that computes, with FHE and quantized neural network operations, four fundamental time-frequency representations: Short-Time Fourier Transform (STFT), Mel filterbanks, Mel-frequency cepstral coefficients (MFCCs), and gammatone filters. Our methods also support the private computation of audio descriptors and convolutional neural network (CNN) classifiers. Besides, we proposed approximate STFT algorithms that lighten computation and bit use for statistical and machine learning analyses. We ran experiments on the VocalSet and OxVoc datasets demonstrating the fully private computation of our approach. We showed significant performance improvements with STFT approximation in private statistical analysis of audio markers, and for vocal exercise classification with CNNs. Our results reveal that our approximations substantially reduce error rates compared to conventional STFT implementations in FHE. We also demonstrated a fully private classification based on the raw audio for gender and vocal exercise classification. Finally, we provided a practical heuristic for parameter selection, making quantized approximate signal processing accessible to researchers and practitioners aiming to protect sensitive audio data.

Figures (5)

• Figure 1: Schematic representation of secure audio processing with fully homomorphic encryption (FHE): client audio is not accessed in plaintext, and can be processed in encrypted form. $\mathcal{T}$ is the MFCC transformation, $g$ is a combination of subsequent operations (e.g. a classifier layer) over MFCCs.
• Figure 2: Top: conventional computation of the spectrogram, or squared magnitude of the STFT. Bottom: our quantized approximate formulations of the STFT energy.
• Figure 3: Comparison of log-scale spectrograms with FHE-friendly transformation corresponding to lowest Euclidean distance, of a given audio from the Vocalset dataset.
• Figure 4: Example of a scatter plot of p-values computed in clear or in FHE with the conventional STFT formulation. Only the mean of gammatone standard deviations (M-GStds) audio descriptor is plotted. The data is from VocalSet. The arrow points to statistical tests in clear and FHE for the pair of audio classes ("scales slow forte", "scales vocal fry").
• Figure 5: Distributions of bit width controlling the input quantization ($B_i$), output quantization ($B_o$), weights of convolutions ($B_w$) and intermediary activations ($B_m$), in the best-performing models for all approaches, datasets and tasks.