Neural-Polyptych: Content Controllable Painting Recreation for Diverse Genres

TL;DR

Neural-Polyptych introduces a content-controllable framework for large-scale, high-fidelity painting recreation by integrating interactive sketches with fragments of original artworks. The core architecture combines a two-branch GAN (encoder-decoder and super-resolution enhancer) with a Reference Bank and a Correspondence Attention module to enforce semantic detail and global consistency. A multi-scale training regime and patch-based inference enable megapixel-scale outputs, with per-painting models trained on diverse Eastern and Western genres. The approach demonstrates superior qualitative and quantitative performance, supports non-expert user input, and enables applications such as large-scale painting generation, texture synthesis, genre switching, and mural restoration. Despite promising results, it notes limitations around semantic masking for nuanced content and performance with sparse inputs.

Abstract

To bridge the gap between artists and non-specialists, we present a unified framework, Neural-Polyptych, to facilitate the creation of expansive, high-resolution paintings by seamlessly incorporating interactive hand-drawn sketches with fragments from original paintings. We have designed a multi-scale GAN-based architecture to decompose the generation process into two parts, each responsible for identifying global and local features. To enhance the fidelity of semantic details generated from users' sketched outlines, we introduce a Correspondence Attention module utilizing our Reference Bank strategy. This ensures the creation of high-quality, intricately detailed elements within the artwork. The final result is achieved by carefully blending these local elements while preserving coherent global consistency. Consequently, this methodology enables the production of digital paintings at megapixel scale, accommodating diverse artistic expressions and enabling users to recreate content in a controlled manner. We validate our approach to diverse genres of both Eastern and Western paintings. Applications such as large painting extension, texture shuffling, genre switching, mural art restoration, and recomposition can be successfully based on our framework.

Figures (19)

• Figure 1: Working flow of our approach. Firstly, we choose specific regions of interest from the original painting that are earmarked for subsequent recomposition. Subsequently, the user is provided with interactive sketches, allowing for unconstrained design inputs. Using a neural network, our method transforms these inputs into visually captivating digital paintings.
• Figure 2: Diverse components with detail from an artwork.
• Figure 3: Overview of our network architecture.
• Figure 4: Visualization of the clustered results of reference bank. Three representative examples of the category are showcased on the right side of each dot chart. The frame color of each representative case matches the dots' color in the charts. Categories with a small number of samples $\left(< N\right)$ will be processed as outliers.
• Figure 5: Correspondence Attention.