Optimizing Dense Feed-Forward Neural Networks

TL;DR

ODF2NNA addresses the challenge of designing efficient dense feed-forward networks by coupling pruning with transfer learning. It constructs a large, uniform topology, prunes to extract useful units via a data-driven variability threshold controlled by $\epsilon$, and rebuilds a refined, lighter model that is lightly retrained to retain or improve performance. Across classification and regression benchmarks, the method achieves substantial parameter reductions (often >70%) with equal or better accuracy and demonstrates strong performance on both small-scale and large-scale datasets, including MNIST, Hepmass, and Higgs. The results support the practical value of topology-aware pruning with knowledge transfer for energy- and memory-constrained deployments, outperforming many existing pruning approaches and suggesting that topology optimization may be more effective than purely backpropagation-driven training for dense networks.

Abstract

Deep learning models have been widely used during the last decade due to their outstanding learning and abstraction capacities. However, one of the main challenges any scientist has to face using deep learning models is to establish the network's architecture. Due to this difficulty, data scientists usually build over complex models and, as a result, most of them result computationally intensive and impose a large memory footprint, generating huge costs, contributing to climate change and hindering their use in computational-limited devices. In this paper, we propose a novel feed-forward neural network constructing method based on pruning and transfer learning. Its performance has been thoroughly assessed in classification and regression problems. Without any accuracy loss, our approach can compress the number of parameters by more than 70%. Even further, choosing the pruning parameter carefully, most of the refined models outperform original ones. We also evaluate the transfer learning level comparing the refined model and the original one training from scratch a neural network with the same hyper parameters as the optimized model. The results obtained show that our constructing method not only helps in the design of more efficient models but also more effective ones.

Figures (9)

• Figure 1: Algorithm 1. Flowchart
• Figure 2: Algorithm 2. Flowchart
• Figure 3: Implicit subnet for the candidate unit (red)
• Figure 4: Algorithm 3. Flowchart
• Figure 5: Algorithm 4. Flowchart