BEND: Bagging Deep Learning Training Based on Efficient Neural Network Diffusion

TL;DR

The proposed BEND algorithm can consistently outperform the mean and median accuracies of both the original trained model and the diffused model and provides a new paradigm for future deep learning training and inference.

Abstract

Bagging has achieved great success in the field of machine learning by integrating multiple base classifiers to build a single strong classifier to reduce model variance. The performance improvement of bagging mainly relies on the number and diversity of base classifiers. However, traditional deep learning model training methods are expensive to train individually and difficult to train multiple models with low similarity in a restricted dataset. Recently, diffusion models, which have been tremendously successful in the fields of imaging and vision, have been found to be effective in generating neural network model weights and biases with diversity. We creatively propose a Bagging deep learning training algorithm based on Efficient Neural network Diffusion (BEND). The originality of BEND comes from the first use of a neural network diffusion model to efficiently build base classifiers for bagging. Our approach is simple but effective, first using multiple trained model weights and biases as inputs to train autoencoder and latent diffusion model to realize a diffusion model from noise to valid neural network parameters. Subsequently, we generate several base classifiers using the trained diffusion model. Finally, we integrate these ba se classifiers for various inference tasks using the Bagging method. Resulting experiments on multiple models and datasets show that our proposed BEND algorithm can consistently outperform the mean and median accuracies of both the original trained model and the diffused model. At the same time, new models diffused using the diffusion model have higher diversity and lower cost than multiple models trained using traditional methods. The BEND approach successfully introduces diffusion models into the new deep learning training domain and provides a new paradigm for future deep learning training and inference.

Figures (5)

• Figure 1: Deep Learning Training Process
• Figure 2: BEND Framework. Our approach consists of three processes, named Construct Model Parameter Subsets, Diffuse New Parameters, and Bagging Weak Classifier Results. Construct Model Parameter Subsets only appears during training. It delivers a subset of trained model parameters. Diffuse New Parameter emerges in both the training and inference phases. Diffuse New Parameter emerges in both the training and inference phases. In the inference phase, it uses noise to diffuse the new model parameters. Bagging Weak Classifier Results occurs only in the inference phase. It ensembles multiple predictions using the sBEND or aBEND methods.
• Figure 3: An example of Bagging.There are three samples (sample1, sample2 and sample3) and three classifiers (classifier1, classifier2 and classifier3). ✓ means correctly classified and ✗ indicates incorrectly classified.
• Figure 4: Diversity of Different Model Parameters
• Figure 5: Test Set Accuracy vs. the Number of Subsets of Model Parameters ($k$) and the Number of Diffused base classifiers ($m$)