Advancing the Understanding of Fixed Point Iterations in Deep Neural Networks: A Detailed Analytical Study

TL;DR

This study conducts a detailed analysis of fixed point iterations in a vector-valued function modeled by neural networks that establishes a sufficient condition for the existence of multiple fixed points of looped neural networks based on varying input regions and expands the examination to include a robust version of fixed point iterations.

Abstract

Recent empirical studies have identified fixed point iteration phenomena in deep neural networks, where the hidden state tends to stabilize after several layers, showing minimal change in subsequent layers. This observation has spurred the development of practical methodologies, such as accelerating inference by bypassing certain layers once the hidden state stabilizes, selectively fine-tuning layers to modify the iteration process, and implementing loops of specific layers to maintain fixed point iterations. Despite these advancements, the understanding of fixed point iterations remains superficial, particularly in high-dimensional spaces, due to the inadequacy of current analytical tools. In this study, we conduct a detailed analysis of fixed point iterations in a vector-valued function modeled by neural networks. We establish a sufficient condition for the existence of multiple fixed points of looped neural networks based on varying input regions. Additionally, we expand our examination to include a robust version of fixed point iterations. To demonstrate the effectiveness and insights provided by our approach, we provide case studies that looped neural networks may exist $2^d$ number of robust fixed points under exponentiation or polynomial activation functions, where $d$ is the feature dimension. Furthermore, our preliminary empirical results support our theoretical findings. Our methodology enriches the toolkit available for analyzing fixed point iterations of deep neural networks and may enhance our comprehension of neural network mechanisms.

Figures (3)

• Figure 1: Example of polynomial (left) and exponential (right) functions contain at least two fixed points (red dots), and points near these fixed points will converge to them under fixed-point iteration.
• Figure 2: Example of a $2$-$d$ case of Theorem \ref{['thm:poly']}. Left: The graph of $f_1(x_1,x_2)$ (in green) and $y = x_1$ (in purple), with the red line indicating the intersection of two curves ($1$-th dimension fixed point). Right: The graph of $f_2(x_1,x_2)$ (in green) and $y = x_2$ (in purple), with the red line indicating the intersection of two curves ($2$-th dimension fixed point).
• Figure 3: Different iteration trajectory when taking different $m$ in Theorem \ref{['thm:robust_fix_point_thm_sca']}. We consider the 2-$d$ case, where the two axes are $x_1$ and $x_2$. The red square ( ■) in this figure is the initial point $x^{(0)} \in \mathbb{R}^{2}$. The red star ( ★) in this figure is the robust fixed point without noise. The $m$ represents the inverse of the noise level, i.e., noise $=1/m$. From the figure, we can see that even though the noise changes the fixed point iteration process, different trajectories will eventually converge to the robust fixed point.

Theorems & Definitions (53)

• Definition 3.1: Fixed point, ah09
• Definition 3.2: Contractive mapping, Definition 5.1.2 of ah09
• Lemma 3.3: Banach fixed-point theorem, Theorem 5.1.3 of ah09
• Lemma 3.4: Banach fixed point theorem, vector case, informal version of Lemma \ref{['lem:fix_point_thm_vec:formal']}
• Definition 3.5: Looped Neural Network
• Remark 3.6
• Theorem 4.1: General result
• proof
• Theorem 4.2: Robust Banach fixed point theorem, scalar case
• proof