Neural Cellular Automata: From Cells to Pixels

TL;DR

The paper addresses the resolution bottleneck of Neural Cellular Automata by decoupling dynamics from appearance: NCAs run on a coarse grid while a lightweight Local Pattern Producing Network (LPPN) renders high-resolution outputs as a neural field conditioned on local cell states and coordinates. End-to-end training uses task-specific losses for texture synthesis and morphology growth, including patch-based multi-scale OT texture loss, PBR cross-map alignment via pseudo targets, auto-correlation regularization, and LPIPS perceptual terms. The hybrid NCA+LPPN achieves real-time, high-resolution outputs across 2D/3D grids and meshes (e.g., $96$–$128^2$ NCA grids render to $768$–$1024^2$ or higher resolutions) while preserving the self-organizing properties of NCAs such as robustness and regeneration. An interactive web demo in the browser demonstrates practical deployment, highlighting the approach’s potential for scalable, generative self-organization in graphics and texture synthesis. The work lays groundwork for extending to full 3D assets, higher resolutions, and tighter integration between rendering and learning objectives, offering a pathway to efficient, controllable, high-fidelity self-organizing systems.

Abstract

Neural Cellular Automata (NCAs) are bio-inspired dynamical systems in which identical cells iteratively apply a learned local update rule to self-organize into complex patterns, exhibiting regeneration, robustness, and spontaneous dynamics. Despite their success in texture synthesis and morphogenesis, NCAs remain largely confined to low-resolution outputs. This limitation stems from (1) training time and memory requirements that grow quadratically with grid size, (2) the strictly local propagation of information that impedes long-range cell communication, and (3) the heavy compute demands of real-time inference at high resolution. In this work, we overcome this limitation by pairing an NCA that evolves on a coarse grid with a lightweight implicit decoder that maps cell states and local coordinates to appearance attributes, enabling the same model to render outputs at arbitrary resolution. Moreover, because both the decoder and NCA updates are local, inference remains highly parallelizable. To supervise high-resolution outputs efficiently, we introduce task-specific losses for morphogenesis (growth from a seed) and texture synthesis with minimal additional memory and computation overhead. Our experiments across 2D/3D grids and mesh domains demonstrate that our hybrid models produce high-resolution outputs in real-time, and preserve the characteristic self-organizing behavior of NCAs.

Figures (19)

• Figure 1: Summary of results. Our proposed method enables NCAs to generate high-quality outputs with minimal extra cost. Our method is applicable to different NCA architectures and training targets. Left: Growing 2D shapes and images from a single seed; Middle: Texture synthesis in 2D; Right: Texture synthesis on 3D Meshes. Online interactive demos are available at https://cells2pixels.github.io.
• Figure 2: Sample Output of Our Hybrid Model. The NCA evolves on a coarse $128 \times 128$ lattice (left); Our Local Pattern Producing Network (LPPN) acts as a neural field decoder that decouples the render resolution from the NCA grid size. We sample this field at $1024 \times 1024$ and, without retraining, at $8192 \times 8192$ in the magnified inset (right).
• Figure 3: Hybrid NCA + LPPN Overview.Left: The NCA operates on a coarse lattice of cells (in this example vertices of a mesh) Center: A sampling point $\mathrm{\mathbf{p}}$ (red dot) inside a triangle primitive, whose vertices correspond to NCA cells $\bm{s}_i,\,\bm{s}_j,\,\bm{s}_k$. The local coordinate$u(\mathrm{\mathbf{p}})$ expresses the point's position inside the primitive, while the locally averaged cell state$\bar{\bm{s}}(\mathrm{\mathbf{p}})$ is obtained by interpolating the surrounding cell states. Right: A shared lightweight MLP, LPPN, receives $(\bar{\bm{s}}(\mathrm{\mathbf{p}}),u(\mathrm{\mathbf{p}}))$ as input and outputs the appearance features, such as color and surface normal, at point $\mathrm{\mathbf{p}}$.
• Figure 4: Representative examples of primitives. Vertices correspond to neighboring cells of an arbitrary sampling point $\mathrm{\mathbf{p}}$.
• Figure 5: Local coordinate transformations. Raw $u$-coordinates visualized as RGB suffer discontinuities at primitive boundaries. (Top Right) Applying trigonometric functions to Cartesian coordinates enforces $C^{0}$ continuity at boundaries for rectangle primitives. Bottom Right (a): Sorting the barycentric coordinates enforces $C^{0}$ continuity but yields an imbalanced dynamic range (red color dominates). (b) Applying a remapping equalizes the range, giving a uniform, continuous positional field that is easier for the LPPN to digest.