DepGAN: Leveraging Depth Maps for Handling Occlusions and Transparency in Image Composition

TL;DR

DepGAN addresses occlusion and transparency in image composition by integrating depth maps and alpha channels into a conditional GAN framework. It introduces a Depth Aware Loss to enforce depth-consistent occlusion boundaries and leverages a Spatial Transformer Network to precisely align foregrounds with backgrounds, aided by a PatchGAN discriminator. The approach is validated across real and synthetic datasets, achieving superior placement semantics and more accurate occlusion/transparency handling than state-of-the-art methods, and is supported by a new aerial dataset for context-aware placement. The work advances practical image compositing by incorporating 3D scene information, enabling more realistic and semantically coherent composites with potential impact on graphics pipelines and automated editing tools.

Abstract

Image composition is a complex task which requires a lot of information about the scene for an accurate and realistic composition, such as perspective, lighting, shadows, occlusions, and object interactions. Previous methods have predominantly used 2D information for image composition, neglecting the potentials of 3D spatial information. In this work, we propose DepGAN, a Generative Adversarial Network that utilizes depth maps and alpha channels to rectify inaccurate occlusions and enhance transparency effects in image composition. Central to our network is a novel loss function called Depth Aware Loss which quantifies the pixel wise depth difference to accurately delineate occlusion boundaries while compositing objects at different depth levels. Furthermore, we enhance our network's learning process by utilizing opacity data, enabling it to effectively manage compositions involving transparent and semi-transparent objects. We tested our model against state-of-the-art image composition GANs on benchmark (both real and synthetic) datasets. The results reveal that DepGAN significantly outperforms existing methods in terms of accuracy of object placement semantics, transparency and occlusion handling, both visually and quantitatively. Our code is available at https://amrtsg.github.io/DepGAN/.

Figures (24)

• Figure 1: Comparison between CompositionalGAN compgan, Photoshop photoshop, and DepGAN (our work) on (a) handling occlusion, (b) handling transparency, and (c) contextual placement semantic when compositing a foreground with a background.
• Figure 2: Overall architecture of DepGAN.
• Figure 3: Left (G): The generator part of DepGAN. The Depth Aware Loss which penalizes the generator when generating foreground regions in composite image where the varying depth values are lighter is our addition to the network. The detailed architecture of our generator is provided in the supplementary materials. Right (D): A diagram of our discriminator, based off the PatchGAN discriminator patchgan, which assesses whether each N × N patch within an image is genuine or generated. This approach allows for a finer-grained analysis of image structure.
• Figure 4: Left. An image of the binary mask where each foreground pixel imposes a penalty or reward on the generator. Lighter pixels indicate higher penalties. The depth mask applied to the predicted image and ground truth highlights only important areas, ensuring the foreground does not overlap the background.
• Figure 5: Evaluation on STRAT’s dataset STRAT: Both DepGAN and Compositional GAN show impressive performance, but in cases marked (A) and (B), only DepGAN is able to handle the transparency of the glasses.