Dynamic Dropout: Leveraging Conway's Game of Life for Neural Networks Regularization

TL;DR

The paper tackles overfitting in neural networks and the limitations of static dropout by introducing Dynamic Dropout, a GoL-inspired regularization that evolves a lattice to yield adaptive, structured neuron deactivation during training. The method defines a binary lattice $L$ that follows Conway's Game of Life rules and modulates layer activations via $\tilde{z}^{(l)} = z^{(l)} \odot (1 - L^{(t)}_{l})$, with occasional reactivation to prevent stagnation, while using standard cross-entropy loss. Empirical evaluation on CIFAR-10 shows that Dynamic Dropout achieves higher training accuracy than classical dropout variants but can exhibit some overfitting on validation in shallower networks, while deeper architectures benefit from reduced generalization gaps and more robust regularization due to GoL pattern dynamics. The work highlights the potential of GoL-based, self-organizing regularization for deeper architectures and outlines avenues for extending the approach to CNNs, transformers, and combinatorial integration with other regularization techniques.

Abstract

Regularization techniques play a crucial role in preventing overfitting and improving the generalization performance of neural networks. Dropout, a widely used regularization technique, randomly deactivates units during training to introduce redundancy and prevent co-adaptation among neurons. Despite its effectiveness, dropout has limitations, such as its static nature and lack of interpretability. In this paper, we propose a novel approach to regularization by substituting dropout with Conway's Game of Life (GoL), a cellular automata with simple rules that govern the evolution of a grid of cells. We introduce dynamic unit deactivation during training by representing neural network units as cells in a GoL grid and applying the game's rules to deactivate units. This approach allows for the emergence of spatial patterns that adapt to the training data, potentially enhancing the network's ability to generalize. We demonstrate the effectiveness of our approach on the CIFAR-10 dataset, showing that dynamic unit deactivation using GoL achieves comparable performance to traditional dropout techniques while offering insights into the network's behavior through the visualization of evolving patterns. Furthermore, our discussion highlights the applicability of our proposal in deeper architectures, demonstrating how it enhances the performance of different dropout techniques.

Figures (5)

• Figure 1: Dynamic Dropout at consecutive training epochs.
• Figure 2: The evolution of GoL can sometimes result in fully activated configurations, which may contribute to overfitting, as observed in epoch $t+2$. To mitigate this, the algorithm randomly activates a subset of units, as depicted in epoch $t+4$, prompting a new iteration of GoL from that point onward.
• Figure 3: Accuracy achieved by each approach.
• Figure 4: Loss achieved by each approach. AD validation losses in (b) and (c) exceeded the chart's display limits.
• Figure 5: Layouts of the GoL lattice at epochs 1, 10, and 20 for the studied configurations.