The Latency Wall: Benchmarking Off-the-Shelf Emotion Recognition for Real-Time Virtual Avatars

TL;DR

This work benchmarks off-the-shelf emotion recognition models for real-time virtual avatars under a strict latency budget of $140~\mathrm{ms}$, revealing a clear split between robust face detection and a latency-laden emotion classification stage. Using the UIBVFED dataset, the study compares YOLO-based detectors (v8/v11/v12, Medium and Nano) with zero-shot classifiers (CLIP, SigLIP) and a domain-tuned ViT-FER, all evaluated on CPU hardware. Results show 100% detection accuracy across stylized avatars but a pronounced Latency Wall for emotion classification: the best avatar-facing setup (YOLOv11n + ViT-FER) runs around $193.58~\mathrm{ms}$ for classification, still exceeding the 130 ms AI budget after accounting for rendering, while zero-shot methods either underperform in accuracy or exceed latency budgets. On FER-2013 real faces, some models fare better, yet the overall conclusion remains: off-the-shelf transformers are not yet suitable for real-time, accessible VR therapy without significant model distillation and domain-specific optimization. The findings call for lightweight, domain-tailored architectures (e.g., distilled CNNs) to bridge the gap between perceptual quality and strict real-time constraints in therapeutic VR applications.

Abstract

In the realm of Virtual Reality (VR) and Human-Computer Interaction (HCI), real-time emotion recognition shows promise for supporting individuals with Autism Spectrum Disorder (ASD) in improving social skills. This task requires a strict latency-accuracy trade-off, with motion-to-photon (MTP) latency kept below 140 ms to maintain contingency. However, most off-the-shelf Deep Learning models prioritize accuracy over the strict timing constraints of commodity hardware. As a first step toward accessible VR therapy, we benchmark State-of-the-Art (SOTA) models for Zero-Shot Facial Expression Recognition (FER) on virtual characters using the UIBVFED dataset. We evaluate Medium and Nano variants of YOLO (v8, v11, and v12) for face detection, alongside general-purpose Vision Transformers including CLIP, SigLIP, and ViT-FER.Our results on CPU-only inference demonstrate that while face detection on stylized avatars is robust (100% accuracy), a "Latency Wall" exists in the classification stage. The YOLOv11n architecture offers the optimal balance for detection (~54 ms). However, general-purpose Transformers like CLIP and SigLIP fail to achieve viable accuracy (<23%) or speed (>150 ms) for real-time loops. This study highlights the necessity for lightweight, domain-specific architectures to enable accessible, real-time AI in therapeutic settings.

Figures (1)

• Figure 1: Images from the UIBVFED dataset