Multiobjective Evolutionary Pruning of Deep Neural Networks with Transfer Learning for improving their Performance and Robustness
Javier Poyatos, Daniel Molina, Aitor Martínez, Javier Del Ser, Francisco Herrera
TL;DR
MO-EvoPruneDeepTL tackles pruning in deep networks under a transfer-learning setting by framing pruning as a three-objective optimization over sparse last-layer patterns: $ACC_{val}$, $MEM$, and $AUROC$. It uses ODIN-based OoD detection as a robustness objective and evolving sparse last-layer connections via NSGA-II to balance accuracy, complexity, and robustness. The approach yields diverse Pareto fronts, identifies consistent pruning patterns with Grad-CAM, and demonstrates that ensembles of Pareto-front models can improve both accuracy and OoD robustness across multiple datasets. This work advances MONAS by integrating pruning, transfer learning, and OoD detection into a coherent multi-objective optimization framework with practical ensemble benefits.
Abstract
Evolutionary Computation algorithms have been used to solve optimization problems in relation with architectural, hyper-parameter or training configuration, forging the field known today as Neural Architecture Search. These algorithms have been combined with other techniques such as the pruning of Neural Networks, which reduces the complexity of the network, and the Transfer Learning, which lets the import of knowledge from another problem related to the one at hand. The usage of several criteria to evaluate the quality of the evolutionary proposals is also a common case, in which the performance and complexity of the network are the most used criteria. This work proposes MO-EvoPruneDeepTL, a multi-objective evolutionary pruning algorithm. MO-EvoPruneDeepTL uses Transfer Learning to adapt the last layers of Deep Neural Networks, by replacing them with sparse layers evolved by a genetic algorithm, which guides the evolution based in the performance, complexity and robustness of the network, being the robustness a great quality indicator for the evolved models. We carry out different experiments with several datasets to assess the benefits of our proposal. Results show that our proposal achieves promising results in all the objectives, and direct relation are presented among them. The experiments also show that the most influential neurons help us explain which parts of the input images are the most relevant for the prediction of the pruned neural network. Lastly, by virtue of the diversity within the Pareto front of pruning patterns produced by the proposal, it is shown that an ensemble of differently pruned models improves the overall performance and robustness of the trained networks.