Statistical Blendshape Calculation and Analysis for Graphics Applications

TL;DR

This paper addresses real-time facial avatar animation on low-power devices by predicting blendshape coefficients from monocular webcam input. It introduces a pipeline of affine transformation, segmentation, data transformation, regression, and smoothing, formalized as $f_b=F(R(T_d(S(T_a(X)))))$, to convert landmarks into blendshape coefficients. The authors train independent statistical models for most blendshapes, apply bias correction and autocorrelation-aware smoothing, and validate against ARKit 6 with competitive accuracy while requiring modest hardware. Experiments use real and synthetic data, with 18,209 frames over 21 videos, and show strong per-blendshape performance and CPU-only real-time operation. This work enables practical, high-quality facial animation on standard PCs and low-power devices without relying on depth sensors.

Abstract

With the development of virtualization and AI, real-time facial avatar animation is widely used in entertainment, office, business and other fields. Against this background, blendshapes have become a common industry animation solution because of their relative simplicity and ease of interpretation. Aiming for real-time performance and low computing resource dependence, we independently developed an accurate blendshape prediction system for low-power VR applications using a standard webcam. First, blendshape feature vectors are extracted through affine transformation and segmentation. Through further transformation and regression analysis, we were able to identify models for most blendshapes with significant predictive power. Post-processing was used to further improve response stability, including smoothing filtering and nonlinear transformations to minimize error. Experiments showed the system achieved accuracy similar to ARKit 6. Our model has low sensor/hardware requirements and realtime response with a consistent, accurate and smooth visual experience.

Figures (11)

• Figure 1: Effect of the affine transformation matrices on (a) Raw data where (b) $R_1$, (c) $R_2$ (d) $R_3$ are applied in succession
• Figure 2: Correlation for JawOpen. (a) Linear correlation magnitude for all points on the face for JawOpen. Yellow points are highly correlated, while black is uncorrelated. (b) Behavior of highly correlated (blue) and uncorrelated (red) sample points. This comparison demonstrates distinct behavior that justifies segmentation before regression modeling.
• Figure 3: Statistical support for keypoint selection and segmentation. (a)-(d) Shows further statistical analysis. (a) Landmark rank order, (b) 3D correlation vs. 1D linear regression, (c) SVR + RBF kernel vs 3D correlation, (d) F-regression vs. SVR + RBF.
• Figure 4: Data transformation and sample regression for CheekPuff, and resulting coefficients of determination ($R^2$). (a) Shows a multiple exponential model, (b) multiple log-exponential model, (c) polynomial (deg=2) , (d) Standard OLS Linear Regression (e) Partial Least Squares projection (PLS, deg=1), (e). Beneath each is a truth comparison for (f) exponential (g) log-exponential, (h) polynomial, (i) Linear (j) PLS. Statistical measures of performance include Mean squared errorjohnson_applied_2002, Durbin-Watson statisticdurbin_testing_1950, Breush-Pagan testbreusch_simple_1979, Fisher-Statisticweir_estimating_1984, Pearson correlationpearson_x_1900 and $\xi$ correlationchatterjee_new_2021.
• Figure 5: Regression models for simple blendshape classes. (a) Single element of a multiple linear regression model. (b) A 1 component PCA model using 8 different data vectors. (c) A PLS model created using 8 keypoint displacement vectors. (g) A Gaussian process regression model showing the 95% confidence interval for the probability distribution.