Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Asymptotic behavior of eigenvalues of large rank perturbations of large random matrices

Ievgenii Afanasiev, Leonid Berlyand, Mariia Kiyashko

Abstract

The paper is concerned with deformed Wigner random matrices. These matrices are closely related to Deep Neural Networks (DNNs): weight matrices of trained DNNs could be represented in the form $R + S$, where $R$ is random and $S$ is highly correlated. The spectrum of such matrices plays a key role in rigorous underpinning of the novel pruning technique based on Random Matrix Theory. In practice, the spectrum of the matrix $S$ can be rather complicated. In this paper, we develop an asymptotic analysis for the case of full rank $S$ with increasing number of outlier eigenvalues.

Asymptotic behavior of eigenvalues of large rank perturbations of large random matrices

Abstract

The paper is concerned with deformed Wigner random matrices. These matrices are closely related to Deep Neural Networks (DNNs): weight matrices of trained DNNs could be represented in the form , where is random and is highly correlated. The spectrum of such matrices plays a key role in rigorous underpinning of the novel pruning technique based on Random Matrix Theory. In practice, the spectrum of the matrix can be rather complicated. In this paper, we develop an asymptotic analysis for the case of full rank with increasing number of outlier eigenvalues.

Paper Structure

This paper contains 9 sections, 8 theorems, 131 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Formulation of the problem
  3. Motivation: weight matrices of DNNs
  4. Main results
  5. Equation for the Stieltjes transform
  6. The limiting distribution of eigenvalues outside the bulk
  7. Proof of Theorem \ref{['th:jth eval']}
  8. Numerical simulations
  9. Proof of auxiliary lemmas

Key Result

Theorem 2.1

Let $W$ be a random matrix of the form W def. Let the third and fourth moments of the entries of $R$ be finite. Let $\mu$ and $\nu$ be the ESDs of the matrices $W$ and $S$, respectively, $\mu_0$ be the limit of $\mu$ as $N \to \infty$, and let Assumptions 1--3 hold. Then a signed measure $\frac{N}{r

Figures (2)

  • Figure 1: Numeric simulation of a DNN with 3 layers. Show the dependence of the number of outlier eigenvalues of the "signal" matrix, so called spikes, on the size of the matrix.
  • Figure 2: Numerical simulations of a DNN with 3 layers. Two figures correspond to two different realizations with the same architecture.

Theorems & Definitions (15)

  • Theorem 2.1
  • Theorem 2.2
  • Proposition 3.1
  • proof
  • Lemma 3.2
  • Proposition 3.3
  • proof
  • Lemma 3.4
  • Lemma 4.1
  • proof
  • ...and 5 more