Multi-Texture Synthesis through Signal Responsive Neural Cellular Automata

TL;DR

This work addresses the need for multiple textures from a single neural cellular automaton (NCA) by encoding texture identity in genome channels within the seed, enabling up to $2^{n_g}$ textures and supporting interpolation and grafting. The authors implement a single NCA with fixed perception kernels and a 1x1 update network, trained via backpropagation through time using a differentiable VGG16 observer and Sliced Wasserstein Loss, alongside an overflow term, and stabilized by a pool of $1024$ seeds. They extend to multi-texture generation by allocating $n_g$ genomic channels, cycle through genomes during training, and demonstrate interpolation between textures as well as regeneration and grafting within one automaton. Results on texture datasets show that genome channels preserve texture identity, SWL effectively captures texture style for irregular patterns, and a hybrid loss (e.g., OTT) can help regular patterns, with compelling demonstrations of interpolation and texture cohabitation. The findings suggest a scalable, editable, and compact approach to texture synthesis with potential extensions to 3D textures and hardware implementations.

Abstract

Neural Cellular Automata (NCA) have proven to be effective in a variety of fields, with numerous biologically inspired applications. One of the fields, in which NCAs perform well is the generation of textures, modelling global patterns from local interactions governed by uniform and coherent rules. This paper aims to enhance the usability of NCAs in texture synthesis by addressing a shortcoming of current NCA architectures for texture generation, which requires separately trained NCA for each individual texture. In this work, we train a single NCA for the evolution of multiple textures, based on individual examples. Our solution provides texture information in the state of each cell, in the form of an internally coded genomic signal, which enables the NCA to generate the expected texture. Such a neural cellular automaton not only maintains its regenerative capability but also allows for interpolation between learned textures and supports grafting techniques. This demonstrates the ability to edit generated textures and the potential for them to merge and coexist within the same automaton. We also address questions related to the influence of the genomic information and the cost function on the evolution of the NCA.

Figures (22)

• Figure 1: One NCA pass.
• Figure 2: The NN that models the update rule based on the perception output.
• Figure 3: A training step. The NCA runs iteratively through the NN for $t$ steps, as illustrated in Figure \ref{['fig:nca-pass']}. The loss is then calculated and the the NN's parameters are updated by backpropagation through time.
• Figure 4: VGG16 architecture
• Figure 5: The loss calculation process. The feature distributions for the specified $L$ layers are extracted by passing the state's RGB channels through VGG16. They are then matched to those of the example image using SWL.