Neural Cellular Automata and Deep Equilibrium Models

TL;DR

This work analyzes the connections between Neural Cellular Automata (NCA) and Deep Equilibrium Models (DEQ), arguing that both frameworks share local, weight-tied dynamics and implicit equilibria. It demonstrates, via a compact MNIST experiment with a single explicit convolutional layer and a DEQ layer, that DEQ training can yield NCA-like self-organization and robust performance with few parameters. The paper further discusses how implicit differentiation and PDE-inspired viewpoints can unify the two approaches, suggesting practical and theoretical benefits from their integration. Overall, the study highlights a pathway to memory-efficient, dynamically rich models that leverage self-organization and stable equilibrium computation for future research and potential hardware implementations.

Abstract

This essay discusses the connections and differences between two emerging paradigms in deep learning, namely Neural Cellular Automata and Deep Equilibrium Models, and train a simple Deep Equilibrium Convolutional model to demonstrate the inherent similarity of NCA and DEQ based methods. Finally, this essay speculates about ways to combine theoretical and practical aspects of both approaches for future research.

Figures (2)

• Figure 1: Visualizing the hidden channels of the inner iteration in a Deep Equilibrium MNIST classification model
• Figure 2: A truncated image can still be used to generate the equilibrium state for classification