exploreCOSMOS: Interactive Exploration of Conditional Statistical Shape Models in the Web-Browser

TL;DR

The paper addresses accessibility barriers in statistical shape modeling by introducing exploreCOSMOS, a browser-based tool for interactive visualization and posterior reconstruction of 3D faces from partial observations. It adopts the standard SSM framework, using $s=\mu+Q\alpha$ with $\alpha \sim \mathcal{N}(0,I)$, and updates to a posterior mean via partial observations $s_p=\mu_p+Q_p\alpha$, computed efficiently on-device. The implementation relies on three.js, TensorFlow.js, and Statismo-format models to run entirely in the browser, enabling private data handling and easy sharing. The results show intuitive face manipulation, a nose-reconstruction example, and a small user study, with open-source availability for teaching, design, and clinical planning.

Abstract

Statistical Shape Models of faces and various body parts are heavily used in medical image analysis, computer vision and visualization. Whilst the field is well explored with many existing tools, all of them aim at experts, which limits their applicability. We demonstrate the first tool that enables the convenient exploration of statistical shape models in the browser, with the capability to manipulate the faces in a targeted manner. This manipulation is performed via a posterior model given partial observations. We release our code and application on GitHub https://github.com/maximilian-hahn/exploreCOSMOS

Figures (5)

• Figure 1.1: Screenshot of the proposed user interface after generating a random face from the Basel Face Model 2019 bfm2017paper and its corresponding principal components on the right. We can now modify the face by adding observations, e.g., here we move the tip of the nose forward and then compute and display the mean face shape of the posterior; size of controls adjusted for readability.
• Figure 1.2: Examples of face manipulations by moving specific points (original position in red, new position in green). The statistical model enables these manipulations and the resulting faces appear natural (from left to right): receding and protruding chin, long face, short face and asymmetric face.
• Figure 1.3: Our interactive design process is structured as follows: we select vertex points on the face surface and decide if they should stay at the current position or if their position should be modified. The posterior is then calculated based on those observations and displayed to the user. The user can then refine the designed face by readjusting existing observations or adding new observations.
• Figure 1.4: Potential application: We guide different versions for a nose reconstruction for a given face (many observations) and few guiding landmarks on the nose (additional observations) (from left to right): initial shape with missing nose and landmarks, slim, wide, big, small and hooked.
• Figure 1.5: Possible solutions for the tasks in our user study (from left to right): initial shape, shape with long nose, shape with adjusted principal components and two alien-like shapes.