Guiding Evolutionary AutoEncoder Training with Activation-Based Pruning Operators

TL;DR

The paper addresses the inefficiency of over-parameterized deep models by pruning autoencoders (AEs) using evolutionary computation. It introduces activation-guided pruning operators (VARIANCE and CONJUNCTIVE) and a RANDOM baseline, evaluated within canonical AE training and the Lipi-AE-S cooperative coevolution framework that employs a ring topology of $Z$ cells with neighborhood size $s=2r+1$. Six pruning schedules and three operator variants are explored, optimized with reconstruction loss and preserved parameter counts as metrics. Key findings show that CONJUNCTIVE can match unpruned canonical performance while reducing parameters, and that RANDOM with exponential pruning often yields the best results in Lipi-AE-S, highlighting the distinct advantages of activation-guided pruning in low-dimensional settings and of population-based coevolution for robustness and diversity.

Abstract

This study explores a novel approach to neural network pruning using evolutionary computation, focusing on simultaneously pruning the encoder and decoder of an autoencoder. We introduce two new mutation operators that use layer activations to guide weight pruning. Our findings reveal that one of these activation-informed operators outperforms random pruning, resulting in more efficient autoencoders with comparable performance to canonically trained models. Prior work has established that autoencoder training is effective and scalable with a spatial coevolutionary algorithm that cooperatively coevolves a population of encoders with a population of decoders, rather than one autoencoder. We evaluate how the same activity-guided mutation operators transfer to this context. We find that random pruning is better than guided pruning, in the coevolutionary setting. This suggests activation-based guidance proves more effective in low-dimensional pruning environments, where constrained sample spaces can lead to deviations from true uniformity in randomization. Conversely, population-driven strategies enhance robustness by expanding the total pruning dimensionality, achieving statistically uniform randomness that better preserves system dynamics. We experiment with pruning according to different schedules and present best combinations of operator and schedule for the canonical and coevolving populations cases.

Figures (18)

• Figure 1: Overview Lipi-AE. A spatial ring topology of size 6 with radius one is shown. The neighbors are copied to form a subpopulation. The Euclidean product of AEs are evaluated. High quality encoders and decoders are selected. Then encoders and decoders are varied through training and replaced. Finally, the Euclidean product of AEs are evaluated and the encoder and decoder at the node are updated by the new best encoder and decoder.
• Figure 2: Overview of mechanistic interpretation of an AE compute graph that is fully connected. Gray illustrates node weight parameters. Blue illustrates low activations for input data which leads to different paths, based on activation at node. Red illustrates changes in parameter and activation between original ANN \ref{['fig:mi_overview_org']} and pruned ANN \ref{['fig:mi_overview_pruned']}
• Figure 3: Illustration of the Binary Clustering Problem Hemberg2024ae. Sample vectors and centroids are shown.
• Figure 4: Best-performing combinations of pruning methods and schedules for Lipi-AE-S training (default and reduced network sizes) compared to canonical AE training
• Figure 5: Comparative performance of best pruning configs across training paradigms: Lipi-AE-S (standard and reduced networks) versus canonical AE training