OXSeg: Multidimensional attention UNet-based lip segmentation using semi-supervised lip contours

TL;DR

This work tackles the challenge of robust upper-lip segmentation under variable image quality and limited ground-truth masks, with an application to fetal alcohol syndrome (FAS) assessment. It introduces a multidimensional input strategy based on local binary pattern features and an attention-augmented UNet that is applied in a sequential manner to refine lip boundaries, aided by a semi-supervised mask generation from anatomical landmarks. The segmented lips are compressed into a latent representation via an autoencoder, which is then used by a 3D-CNN and a GAN classifier to distinguish FAS from controls, achieving high accuracy, notably 98.55% in the African cohort with GAN. The results demonstrate improved boundary reconstruction around Cupid's bow and show the latent space as a clinically informative descriptor, suggesting practical utility in objective lip-thickness evaluation and FAS screening, while pointing to future unsupervised enhancements.

Abstract

Lip segmentation plays a crucial role in various domains, such as lip synchronization, lipreading, and diagnostics. However, the effectiveness of supervised lip segmentation is constrained by the availability of lip contour in the training phase. A further challenge with lip segmentation is its reliance on image quality , lighting, and skin tone, leading to inaccuracies in the detected boundaries. To address these challenges, we propose a sequential lip segmentation method that integrates attention UNet and multidimensional input. We unravel the micro-patterns in facial images using local binary patterns to build multidimensional inputs. Subsequently, the multidimensional inputs are fed into sequential attention UNets, where the lip contour is reconstructed. We introduce a mask generation method that uses a few anatomical landmarks and estimates the complete lip contour to improve segmentation accuracy. This mask has been utilized in the training phase for lip segmentation. To evaluate the proposed method, we use facial images to segment the upper lips and subsequently assess lip-related facial anomalies in subjects with fetal alcohol syndrome (FAS). Using the proposed lip segmentation method, we achieved a mean dice score of 84.75%, and a mean pixel accuracy of 99.77% in upper lip segmentation. To further evaluate the method, we implemented classifiers to identify those with FAS. Using a generative adversarial network (GAN), we reached an accuracy of 98.55% in identifying FAS in one of the study populations. This method could be used to improve lip segmentation accuracy, especially around Cupid's bow, and shed light on distinct lip-related characteristics of FAS.

Figures (10)

• Figure 1: 5-point Likert scale for lip thickness score ( adopted from astley2015palpebral).
• Figure 2: The model architecture. The model takes the RGB images as input, and outputs the FAS status.
• Figure 3: Multidimensional input construction. A) RGB image, B) LBP image, and C) GLBP image. Note that the lip area is zoomed in for visualization purposes.
• Figure 4: Overlay of the anatomical landmarks on the facial regions. Blue depicts the upper lip landmarks, and green denotes other landmarks used for image alignment.
• Figure 5: Venn diagram for the data distribution. The pink spectrum denotes the VBLS group, and the green spectrum shows subjects with a known FAS status. The VBLS and FAS status intersection is then subdivided into two groups: control and FAS. The VBLS intersection control is denoted with an orange spectrum, and the VBLS intersection FAS is depicted with a blue spectrum.