A Tighter Complexity Analysis of SparseGPT

TL;DR

The analysis of the running time of SparseGPT is improved from $O(d^{3})$ to $O(d^{\omega} + d^{2+a+o(1)} + d^{1+\omega(1,1,a)-a})$ for any $a \in [0, 1]$ for any $a \in [0, 1]$, where $\omega$ is the exponent of matrix multiplication.

Abstract

In this work, we improved the analysis of the running time of SparseGPT [Frantar, Alistarh ICML 2023] from $O(d^{3})$ to $O(d^ω + d^{2+a+o(1)} + d^{1+ω(1,1,a)-a})$ for any $a \in [0, 1]$, where $ω$ is the exponent of matrix multiplication. In particular, for the current $ω\approx 2.371$ [Alman, Duan, Williams, Xu, Xu, Zhou 2024], our running time boils down to $O(d^{2.53})$. This running time is due to the analysis of the lazy update behavior in iterative maintenance problems such as [Deng, Song, Weinstein 2022; Brand, Song, Zhou ICML 2024].

Theorems & Definitions (15)

• Theorem 1.1: Main result (Restatement of Theorem \ref{['thm:main']})
• Definition 3.1
• Definition 3.3
• Lemma 3.4: adw+24wxxz24lg24
• Theorem 4.1: Main result
• proof
• Remark 4.2
• Lemma 5.1
• proof
• Lemma 5.2