Quality assessment of 3D human animation: Subjective and objective evaluation

TL;DR

This work tackles the lack of perceptually validated quality metrics for non-parametric, geometrically dense 3D human animations. It introduces 4DHumanPercept, a dataset of 4D human animations distorted along controlled geometry and motion factors, with corresponding subjective MOS labels gathered in a user study; and 4DHumanQA, a linear regression-based perceptual metric trained on geometry- and motion-focused features to predict MOS. On a test set, 4DHumanQA achieves a PLCC of about $0.917$ and an SROCC of about $0.961$, outperforming a state-of-the-art deep-learning baseline (LPIPS) for this task. The results demonstrate that a carefully crafted, feature-based predictor can closely match human perception for non-parametric 3D human animations, enabling accurate perceptual evaluation without relying on parametric body models. The work lays the groundwork for scalable perceptual quality assessment in VH research and applications, with public data and code to facilitate further development.

Abstract

Virtual human animations have a wide range of applications in virtual and augmented reality. While automatic generation methods of animated virtual humans have been developed, assessing their quality remains challenging. Recently, approaches introducing task-oriented evaluation metrics have been proposed, leveraging neural network training. However, quality assessment measures for animated virtual humans that are not generated with parametric body models have yet to be developed. In this context, we introduce a first such quality assessment measure leveraging a novel data-driven framework. First, we generate a dataset of virtual human animations together with their corresponding subjective realism evaluation scores collected with a user study. Second, we use the resulting dataset to learn predicting perceptual evaluation scores. Results indicate that training a linear regressor on our dataset results in a correlation of 90%, which outperforms a state of the art deep learning baseline.

Figures (6)

• Figure 1: We conduct a perceptual evaluation to collect subjective scores for visual distortions of generated 3D human animations with respect to corresponding references, which are the acquired 3D reconstructions of real actors. We use the resulting "4DHumanPercept" dataset to first analyse the factors influencing human motion realism, and second, to learn a data-driven model called "4DHumanQA" that predicts a perceptual score for 3D human animation realism.
• Figure 2: Illustration of the 8 source models selected from 4DHumanOutfit armando20234dhumanoutfit.
• Figure 3: Generated 4D humans with the 6 different simulated distortions at the highest strength. The distortions were applied on two complementary source models, each representing a different subject performing a different motion, and dressed in distinct clothing.
• Figure 4: Screenshot of the user study developed with PsychoPy.
• Figure 5: Graphs representing the distribution of participants' "Opinion" responses with the calculated MOS and 95% confidence intervals (whiskers) for the factor Distortion strength, for all Distortion types. Lines marked with * denote significant differences at $p < 0.05$, ** $p < 0.01$, and *** $p < 0.001$ (post hoc test: Tukey HSD).