Transformer Redesign for Late Fusion of Audio-Text Features on Ultra-Low-Power Edge Hardware

TL;DR

This work addresses the challenge of real-time multimodal emotion recognition on ultra-low-power edge devices by introducing a hardware-aware late-fusion model that combines keyword-based textual features with acoustic spectrogram embeddings. The architecture nests a lightweight ViT-based audio encoder with a frozen DSResNet-SE keyword-spotting module, feeding a concatenated embedding into a final classifier, and is designed for quantization and deployment on the Coral Edge TPU. The system demonstrates practical edge capabilities, achieving privacy-preserving inference within a tight 1.8 MB footprint, and reports notable improvements over unimodal baselines on re-recorded samples, with macro F1 gains and low latency (21–23 ms) on MCU hardware. The approach shows that accurate, real-time multimodal emotion inference is feasible on microcontroller-class devices through task-specific fusion and hardware-guided model design, with future directions including on-device personalization, trigger-based activation to extend battery life, and wearable validation.

Abstract

Deploying emotion recognition systems in real-world environments where devices must be small, low-power, and private remains a significant challenge. This is especially relevant for applications such as tension monitoring, conflict de-escalation, and responsive wearables, where cloud-based solutions are impractical. Multimodal emotion recognition has advanced through deep learning, but most systems remain unsuitable for deployment on ultra-constrained edge devices. Prior work typically relies on powerful hardware, lacks real-time performance, or uses unimodal input. This paper addresses that gap by presenting a hardware-aware emotion recognition system that combines acoustic and linguistic features using a late-fusion architecture optimised for Edge TPU. The design integrates a quantised transformer-based acoustic model with frozen keyword embeddings from a DSResNet-SE network, enabling real-time inference within a 1.8MB memory budget and 21-23ms latency. The pipeline ensures spectrogram alignment between training and deployment using MicroFrontend and MLTK. Evaluation on re-recorded, segmented IEMOCAP samples captured through the Coral Dev Board Micro microphone shows a 6.3% macro F1 improvement over unimodal baselines. This work demonstrates that accurate, real-time multimodal emotion inference is achievable on microcontroller-class edge platforms through task-specific fusion and hardware-guided model design.

Figures (8)

• Figure 1: Comparison of mel-spectrograms: (a) generated on Coral Micro, and (b) re-recorded as .wav and processed using MLTK. The horizontal axis denotes time (s), and the vertical axis denotes frequency (Hz); color indicates intensity in dB.
• Figure 2: System Architecture
• Figure 3: Squeeze and Excitation (SE) block architecture
• Figure 4: DSResNet-SE model architecture, with a magnified residual block
• Figure 5: Training loss and accuracy of the DSResNet-SE model's best trial run. The red horizontal dashed line represents the epoch where the learning rate decayed.