An Effective Weight Initialization Method for Deep Learning: Application to Satellite Image Classification

TL;DR

This work addresses the challenge of weight initialization for CNN-based satellite image classification by introducing a variance-preserving initialization. The method derives forward and backward pass variance constraints, yielding a target weight variance of $Var[W]=\frac{2}{fan_{in}+fan_{out}}$ and providing normal or uniform sampling options to maintain stable gradient flow. The approach is validated across six real-world RS datasets and a non-RS benchmark (CIFAR-100), demonstrating consistent improvements in precision, recall, F1-score, and validation accuracy with negligible computational overhead. The authors provide open-source code and show the technique's practical impact for enhancing DL-based satellite image classification pipelines.

Abstract

The growing interest in satellite imagery has triggered the need for efficient mechanisms to extract valuable information from these vast data sources, providing deeper insights. Even though deep learning has shown significant progress in satellite image classification. Nevertheless, in the literature, only a few results can be found on weight initialization techniques. These techniques traditionally involve initializing the networks' weights before training on extensive datasets, distinct from fine-tuning the weights of pre-trained networks. In this study, a novel weight initialization method is proposed in the context of satellite image classification. The proposed weight initialization method is mathematically detailed during the forward and backward passes of the convolutional neural network (CNN) model. Extensive experiments are carried out using six real-world datasets. Comparative analyses with existing weight initialization techniques made on various well-known CNN models reveal that the proposed weight initialization technique outperforms the previous competitive techniques in classification accuracy. The complete code of the proposed technique, along with the obtained results, is available at https://github.com/WadiiBoulila/Weight-Initialization

Figures (10)

• Figure 1: Main steps of the proposed approach.
• Figure 2: Illustration of the weight initialization process for deep learning networks, where $W$ represents the weights being initialized.
• Figure 3: Illustration of the forward pass process, focusing on the activation of unit $y_{1}$ within the network.
• Figure 4: Illustration of the backward pass process, with a focus on the unit $x_{1}$ to elucidate the proposed weight initialization impact.
• Figure 5: A sample from the satellite image dataset.