Rendering-Refined Stable Diffusion for Privacy Compliant Synthetic Data

TL;DR

R Rendering-Refined Stable Diffusion is introduced, a pipeline that combines 3D-rendering with Stable Diffusion, enabling prompt-based control over human attributes while preserving posture, unlike standard diffusion models that fail to retain posture or GANs that lack realism and flexible attribute control.

Abstract

Growing privacy concerns and regulations like GDPR and CCPA necessitate pseudonymization techniques that protect identity in image datasets. However, retaining utility is also essential. Traditional methods like masking and blurring degrade quality and obscure critical context, especially in human-centric images. We introduce Rendering-Refined Stable Diffusion (RefSD), a pipeline that combines 3D-rendering with Stable Diffusion, enabling prompt-based control over human attributes while preserving posture. Unlike standard diffusion models that fail to retain posture or GANs that lack realism and flexible attribute control, RefSD balances posture preservation, realism, and customization. We also propose HumanGenAI, a framework for human perception and utility evaluation. Human perception assessments reveal attribute-specific strengths and weaknesses of RefSD. Our utility experiments show that models trained on RefSD pseudonymized data outperform those trained on real data in detection tasks, with further performance gains when combining RefSD with real data. For classification tasks, we consistently observe performance improvements when using RefSD data with real data, confirming the utility of our pseudonymized data.

Figures (27)

• Figure 1: Rendering-Refined Stable Diffusion (RefSD) pseudonymizes while preserving posture by combining 3D-rendered poses with attribute-driven human generation, as shown in (a). Two examples of pseudonymized images processed by RefSD are shown in (b).
• Figure 2: Rendering-Refined Stable Diffusion (RefSD) Pipeline: following body mesh estimation , we render a synthetic human pyrender. A privacy mask for the original subject is then applied , merging the synthetic human to replace sensitive data . Finally, SD generates human-like images with attribute-controlled prompts .
• Figure 3: Comparison of SD rombach2022high, DP2 hukkelaas2023deepprivacy2, and our RefSD for posture-preserving pseudonymization. RefSD achieves superior alignment and realism. More in Supplementary Material.
• Figure 4: Overview of the HumanGenAI framework. $\phi$: human (annotator) perception evaluations, $\psi$: vision training evaluations.
• Figure 5: Mean annotator scores for Prompt Complexity ($\phi_{\text{A}}$) for Ethnicity. Remaining (Age, Gender, Emotion, Face Attributes) provided in Supplementary Material.