Multispectral Texture Synthesis using RGB Convolutional Neural Networks

TL;DR

This work proposes two solutions to extend red-green–blue texture synthesis algorithms to multispectral imaging (MSI), and demonstrates that they can be used to perform exemplar-based texture synthesis, achieve good visual quality, and come close to state-of-the-art methods on RGB bands.

Abstract

State-of-the-art RGB texture synthesis algorithms rely on style distances that are computed through statistics of deep features. These deep features are extracted by classification neural networks that have been trained on large datasets of RGB images. Extending such synthesis methods to multispectral images is not straightforward, since the pre-trained networks are designed for and have been trained on RGB images. In this work, we propose two solutions to extend these methods to multispectral imaging. Neither of them require additional training of the neural network from which the second order neural statistics are extracted. The first one consists in optimizing over batches of random triplets of spectral bands throughout training. The second one projects multispectral pixels onto a 3 dimensional space. We further explore the benefit of a color transfer operation upstream of the projection to avoid the potentially abnormal color distributions induced by the projection. Our experiments compare the performances of the various methods through different metrics. We demonstrate that they can be used to perform exemplar-based texture synthesis, achieve good visual quality and comes close to state-of-the art methods on RGB bands.

Figures (9)

• Figure 1: Examples of unusual color distributions induced by the PCA projection of Sentinel-2 cloud field images.
• Figure 2: Scheme of the proposed use of color transfer for multispectral texture synthesis. A projection $P$ is used to obtain 3-channels images from the exemplar and synthetic multispectral images. Then, a color transfer operator $T_C$ is applied to both images to match their color statistics to the one of a reference palette image. Finally, statistics of deep features are extracted from the so-obtained images to perform texture synthesis.
• Figure 3: (Left) Eigenvalues of the used PCA over 11-band Sentinel-2 images. The first component encompasses $95\%$ of the variance, the 3 first $99\%$. (Right) 3 first eigenvectors of the used PCA over 11-band Sentinel-2 images (denoted as $u_i$).
• Figure 4: Exemples of palette images used for the proposed color transfer.
• Figure 5: Examples of multispectral texture synthesis using different style distances as optimization objective. The images displayed are obtained by pooling the 11 spectral bands to obtain 3-channels images allowing visual representation of multispectral images (channel 1: bands 1,2,3,4; channel 2: bands 5,6,7,8; channel 3: bands 9,11,12). Exemplar textures are shown on the left, and each following column correspond to a previously presented method of synthesis, label are shown on the bottom. Images that optimize the stochastic loss allow a visual appearance more faithful to the original textures than the ones minimizing a projected style distance, with no grid-like artifacts observed in the first ones. The introduction of color transfer upstream of a projection enables far less of these defects in the generated images.