DermSynth3D: Synthesis of in-the-wild Annotated Dermatology Images

TL;DR

DermSynth3D tackles the data bottleneck in dermatology by generating photo-realistic 2D skin images with dense ground-truth annotations through blending 2D lesion patterns onto 3D textured human meshes using a differentiable renderer. The framework automatically places lesions on anatomically suitable regions, blends them into texture maps with a deep optimization informed by content, style, and gradient cues, and renders multi-view images with varied lighting and backgrounds to produce richly labeled datasets. Across wound detection and lesion/skin segmentation tasks, models trained on DermSynth3D data demonstrate improved generalization to real in-the-wild images and cross-domain datasets, even with limited real data, while the authors acknowledge limitations and outline future directions including domain adaptation and diffusion-based approaches. By releasing an open-source, modular toolchain, DermSynth3D provides a scalable path to expand dermatology datasets and support diverse downstream tasks such as longitudinal lesion tracking and body-part-aware analytics. The work highlights the practical significance of synthetic, annotated dermatology data for advancing DL-based skin analysis in resource-constrained or ethically sensitive settings.

Abstract

In recent years, deep learning (DL) has shown great potential in the field of dermatological image analysis. However, existing datasets in this domain have significant limitations, including a small number of image samples, limited disease conditions, insufficient annotations, and non-standardized image acquisitions. To address these shortcomings, we propose a novel framework called DermSynth3D. DermSynth3D blends skin disease patterns onto 3D textured meshes of human subjects using a differentiable renderer and generates 2D images from various camera viewpoints under chosen lighting conditions in diverse background scenes. Our method adheres to top-down rules that constrain the blending and rendering process to create 2D images with skin conditions that mimic in-the-wild acquisitions, ensuring more meaningful results. The framework generates photo-realistic 2D dermoscopy images and the corresponding dense annotations for semantic segmentation of the skin, skin conditions, body parts, bounding boxes around lesions, depth maps, and other 3D scene parameters, such as camera position and lighting conditions. DermSynth3D allows for the creation of custom datasets for various dermatology tasks. We demonstrate the effectiveness of data generated using DermSynth3D by training DL models on synthetic data and evaluating them on various dermatology tasks using real 2D dermatological images. We make our code publicly available at https://github.com/sfu-mial/DermSynth3D.

Figures (19)

• Figure 1: Standardized vs in-the-wild skin lesion images ($*$: dermoscopy, all others: clinical).
• Figure 2: A figure depicting the essential notations illustrating the conditions for camera positioning, which require that the camera is positioned outside the mesh and that the mesh does not obstruct the light rays.
• Figure 3: Overview of our proposed framework DermSynth3D. The pipeline takes 2D segmented skin conditions and texture image of a 3D mesh as input, and blends the skin condition onto it to produce a lesion blended texture map. After blending, 2D views of the mesh are rendered from various camera viewpoints, under different lighting conditions, and combined with background images to create a synthetic dermatology dataset.
• Figure 4: The different variants of texture images produced by DermSynth3D with a corresponding close view of skin -conditions. Left to right: Original texture image , Binary mask indicating the non-skin regions , texture image with the "pasted" skin conditions $T_p$, a mask of the texture image showing the localizations of the pasted skin -conditions , a texture image $T_d$ created by dilating the masked lesion $s$, a texture image $T_b$ with the "blended" skin conditions.
• Figure 5: Rendered images from the same camera viewpoint (4 examples; one per row), showcasing blended lesions (b-g) on the original texture map (a). Column (a) shows the rendered views of the original texture map with a combination of Ambient, Diffuse and Specular color values for Point Lights. The images in columns b-d are rendered under different light source positions (b and c) and intensities (c and d), while keeping the material properties constant. The images in columns e-g are rendered by changing the material's reflectivity (e), shininess (f), and a combination of both (g), while keeping the lighting parameters same as d.