Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fixed points of nonnegative neural networks

Tomasz J. Piotrowski, Renato L. G. Cavalcante, Mateusz Gabor

TL;DR

The paper develops a nonlinear Perron-Frobenius framework for fixed-point analysis of nonnegative neural networks, showing that such networks with nonnegative weights and biases induce monotone, (weakly) scalable mappings on the nonnegative cone. It introduces the asymptotic mapping $T_ extinfty$ and the nonlinear spectral radius $ ho(T_ extinfty)$ to establish existence and, under additional primitivity, uniqueness of fixed points; it also characterizes the shape of fixed-point sets (often intervals) and provides numerically accessible certificates. The results generalize to broader monotone nonnegative networks and have practical implications for autoencoders and deep equilibrium models, offering weaker, PF-based conditions than convex-analytic approaches. The work also outlines future directions on approximate fixed points, slow convergence as a design feature, Lipschitz analyses, and extensions beyond nonnegativity, highlighting the method's potential impact on implicit models and inverse problems.

Abstract

We use fixed point theory to analyze nonnegative neural networks, which we define as neural networks that map nonnegative vectors to nonnegative vectors. We first show that nonnegative neural networks with nonnegative weights and biases can be recognized as monotonic and (weakly) scalable mappings within the framework of nonlinear Perron-Frobenius theory. This fact enables us to provide conditions for the existence of fixed points of nonnegative neural networks having inputs and outputs of the same dimension, and these conditions are weaker than those recently obtained using arguments in convex analysis. Furthermore, we prove that the shape of the fixed point set of nonnegative neural networks with nonnegative weights and biases is an interval, which under mild conditions degenerates to a point. These results are then used to obtain the existence of fixed points of more general nonnegative neural networks. From a practical perspective, our results contribute to the understanding of the behavior of autoencoders, and we also offer valuable mathematical machinery for future developments in deep equilibrium models.

Fixed points of nonnegative neural networks

TL;DR

The paper develops a nonlinear Perron-Frobenius framework for fixed-point analysis of nonnegative neural networks, showing that such networks with nonnegative weights and biases induce monotone, (weakly) scalable mappings on the nonnegative cone. It introduces the asymptotic mapping and the nonlinear spectral radius to establish existence and, under additional primitivity, uniqueness of fixed points; it also characterizes the shape of fixed-point sets (often intervals) and provides numerically accessible certificates. The results generalize to broader monotone nonnegative networks and have practical implications for autoencoders and deep equilibrium models, offering weaker, PF-based conditions than convex-analytic approaches. The work also outlines future directions on approximate fixed points, slow convergence as a design feature, Lipschitz analyses, and extensions beyond nonnegativity, highlighting the method's potential impact on implicit models and inverse problems.

Abstract

We use fixed point theory to analyze nonnegative neural networks, which we define as neural networks that map nonnegative vectors to nonnegative vectors. We first show that nonnegative neural networks with nonnegative weights and biases can be recognized as monotonic and (weakly) scalable mappings within the framework of nonlinear Perron-Frobenius theory. This fact enables us to provide conditions for the existence of fixed points of nonnegative neural networks having inputs and outputs of the same dimension, and these conditions are weaker than those recently obtained using arguments in convex analysis. Furthermore, we prove that the shape of the fixed point set of nonnegative neural networks with nonnegative weights and biases is an interval, which under mild conditions degenerates to a point. These results are then used to obtain the existence of fixed points of more general nonnegative neural networks. From a practical perspective, our results contribute to the understanding of the behavior of autoencoders, and we also offer valuable mathematical machinery for future developments in deep equilibrium models.

Paper Structure

This paper contains 45 sections, 16 theorems, 30 equations, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Key concepts
  3. Fixed points of $(A_{0,1,2})$-neural networks
  4. Construction of $(A_{0,1,2})$-activation functions
  5. Construction of $(A_{0,1,2})$-neural networks
  6. Existence and uniqueness of fixed points of $(A_{0,1,2})$-neural networks
  7. Nonlinear spectral radius of an $(A_{0,1,2})$-neural network
  8. Conditions for existence of fixed points of $(A_{0,1,2})$-neural networks
  9. Comparison of the assumptions implying the existence of fixed points of $(A_{0,1,2})$-neural networks
  10. Numerically efficient methods for verifying the existence of fixed points of strongly primitive $(A_{0,1,2})$-neural networks
  11. The shape of fixed point sets of $(A_{0,1,2})$-neural networks
  12. Nonlinear Perron-Frobenius Theory
  13. Applications to neural networks
  14. Fixed points of $(A_{0,1})$-neural networks
  15. Fixed points of $(A_{0})$-neural networks
  16. ...and 30 more sections

Key Result

Lemma 8

Consider a neural network $T$ of the form in (dnn), and let $T_i\colon\mathbb{R}^{k_{i-1}}\to\mathbb{R}^{k_i}\colon\ x_{i-1}\mapsto\sigma_i(W_ix_{i-1}+b_i)$ be its $i$-th layer. Then:

Figures (9)

  • Figure 1: The unique fixed points of an autoencoder with sigmoid activation functions and nonnegative weights and biases on (a) the full MNIST dataset and (b) the ZERO dataset.
  • Figure 2: Sample fixed points of an autoencoder with hyperbolic tangent activation functions and with nonnegative weights and biases on (a) the full MNIST dataset and (b) the ZERO dataset.
  • Figure 3: Sample fixed points of an autoencoder with a primitive Jacobian, positive weights, nonnegative biases, and hyperbolic tangent activation functions on (a) the full MNIST dataset and (b) the ZERO dataset.
  • Figure 4: Sample fixed points of an autoencoder with nonnegative weights and biases and with ReLU activation functions using spectral normalization on (a) the MNIST dataset and (b) the ZERO dataset.
  • Figure 5: Sample fixed points of an autoencoder with primitive Jacobian, positive weights, nonnegative biases, and ReLU activation functions on (a) the MNIST dataset and (b) the ZERO dataset.
  • ...and 4 more figures

Theorems & Definitions (38)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Remark 6
  • Lemma 8
  • Corollary 9
  • Definition 10
  • Proposition 11
  • Definition 12
  • ...and 28 more