RPNR: Robust-Perception Neural Reshading

TL;DR

RPNR addresses the challenge of coherently reshading arbitrarily shaped source fragments for AR insertion using only two images and unpaired training. It introduces a DIP-based renderer coupled with three auxiliary networks—Robust-Perception, Albedo-Shading Net, and Normal-Shading Discriminator—to enforce shading consistency while preserving object albedo. The method forms the final image via a Cut-and-Paste composition and optimizes three losses (shading consistency, shading–normal consistency, and perceptual similarity) to guide the DIP. While results yield believable shading, the authors note limitations in depth and realism due to dataset constraints and simplified lighting models, highlighting opportunities for improved losses and more sophisticated lighting modeling in future work.

Abstract

Augmented Reality (AR) applications necessitates methods of inserting needed objects into scenes captured by cameras in a way that is coherent with the surroundings. Common AR applications require the insertion of predefined 3D objects with known properties and shape. This simplifies the problem since it is reduced to extracting an illumination model for the object in that scene by understanding the surrounding light sources. However, it is often not the case that we have information about the properties of an object, especially when we depart from a single source image. Our method renders such source fragments in a coherent way with the target surroundings using only these two images. Our pipeline uses a Deep Image Prior (DIP) network based on a U-Net architecture as the main renderer, alongside robust-feature extracting networks that are used to apply needed losses. Our method does not require any pair-labeled data, and no extensive training on a dataset. We compare our method using qualitative metrics to the baseline methods such as Cut and Paste, Cut And Paste Neural Rendering, and Image Harmonization

Figures (1)

• Figure 1: The U-Net architecture