Mitigating Gradient Overlap in Deep Residual Networks with Gradient Normalization for Improved Non-Convex Optimization

TL;DR

ZNorm adjusts the gradient scale, standardizing gradients across layers and reducing the negative impact of overlapping gradients, suggesting that ZNorm can affect the gradient flow, enhancing performance in large-scale data processing where accuracy is critical.

Abstract

In deep learning, Residual Networks (ResNets) have proven effective in addressing the vanishing gradient problem, allowing for the successful training of very deep networks. However, skip connections in ResNets can lead to gradient overlap, where gradients from both the learned transformation and the skip connection combine, potentially resulting in overestimated gradients. This overestimation can cause inefficiencies in optimization, as some updates may overshoot optimal regions, affecting weight updates. To address this, we examine Z-score Normalization (ZNorm) as a technique to manage gradient overlap. ZNorm adjusts the gradient scale, standardizing gradients across layers and reducing the negative impact of overlapping gradients. Our experiments demonstrate that ZNorm improves training process, especially in non-convex optimization scenarios common in deep learning, where finding optimal solutions is challenging. These findings suggest that ZNorm can affect the gradient flow, enhancing performance in large-scale data processing where accuracy is critical.

Figures (3)

• Figure 1: Comparison of Gradient Magnitudes in Different Scenarios. This figure visualizes the effects of gradient adjustments in a simulated 3D gradient field, focusing on the impact of residual connections and normalization techniques. Subfigure A displays the original gradients' magnitudes without modifications, representing the initial gradient structure. Subfigure B shows the gradients' magnitudes with residual connections, simulating the effect of gradient overlap often observed in residual networks. Subfigure (C) illustrates the gradients' magnitudes after applying ZNorm yun2024znorm. Subfigure (D) depicts the difference in gradient magnitude between the ZNorm and residual gradients he2016deep. The visualizations emphasize how different normalization techniques and residual structures impact the gradient landscape.
• Figure 2: Comparison of Gradient Directions Across Various Scenarios. This figure demonstrates how gradient adjustments affect a simulated 3D gradient field, highlighting the role of residual connections and normalization methods. Subfigure A presents the initial structure by displaying the unaltered gradient magnitudes and directions. Subfigure B illustrates the effect of residual connections on gradient magnitudes and directions, replicating the typical gradient overlap seen in residual networks. Subfigure C shows the gradient magnitudes and directions after ZNorm yun2024znorm is applied, and Subfigure D reveals the difference in magnitude and direction between gradients processed with ZNorm yun2024znorm and those from residual connections he2016deep. These visualizations underscore the impact of distinct normalization methods and residual connections on the gradient field.
• Figure 3: Test accuracy comparison on CIFAR-10 Dataset krizhevsky2009learning for Deep Residual Networks he2016deep and gradient normalization techniques centeryun2024znormclip.