Multigrade Neural Network Approximation

TL;DR

This work develops a rigorous theoretical foundation for multigrade deep learning (MGDL), a grade-by-grade training paradigm that freezes earlier blocks and iteratively refines the approximation of a target function via residual corrections. By formulating MGDL in an operator-theoretic framework, the authors construct fixed-width networks that realize a one-step contraction on the residuals, and prove that for any target function $f \in C([0,1]^d)$ there exist affine maps with width at most $5d$ such that the MGDL approximants converge uniformly: $|R_k(\mathbf{x})| \searrow 0$ for all $\mathbf{x}$ and $\|R_k\|_{L^p([0,1]^d)} \searrow 0$ for all $p \in [1,\infty)$. The main theorem shows that a strictly decreasing residual sequence can be achieved across grades, with a constructive contraction parameter $\varepsilon$, and the auxiliary one-step contraction is realized via a network-implementable operator $S$ built from localized cutoff functions. Empirically, MGDL demonstrates grade-wise residual decay and outperforms end-to-end training on synthetic high-frequency targets, supporting its role as a stable, interpretable, and scalable alternative to standard deep learning optimization. These results connect MGDL to classical approximation theory and provide a pathway toward provable depth-based error refinement in practical architectures.

Abstract

We study multigrade deep learning (MGDL) as a principled framework for structured error refinement in deep neural networks. While the approximation power of neural networks is now relatively well understood, training very deep architectures remains challenging due to highly non-convex and often ill-conditioned optimization landscapes. In contrast, for relatively shallow networks, most notably one-hidden-layer $\texttt{ReLU}$ models, training admits convex reformulations with global guarantees, motivating learning paradigms that improve stability while scaling to depth. MGDL builds upon this insight by training deep networks grade by grade: previously learned grades are frozen, and each new residual block is trained solely to reduce the remaining approximation error, yielding an interpretable and stable hierarchical refinement process. We develop an operator-theoretic foundation for MGDL and prove that, for any continuous target function, there exists a fixed-width multigrade $\texttt{ReLU}$ scheme whose residuals decrease strictly across grades and converge uniformly to zero. To the best of our knowledge, this work provides the first rigorous theoretical guarantee that grade-wise training yields provable vanishing approximation error in deep networks. Numerical experiments further illustrate the theoretical results.

Figures (8)

• Figure 1: Overview of the MGDL framework and its grade-wise training procedure. Training proceeds grade by grade: at each grade, only the newly added block and the output map are optimized, while all previously learned blocks are frozen and serve as adaptive basis components. The network $\Phi_m$ approximates the target function via a recursive residual refinement process.
• Figure 2: 1D target function $f_1$.
• Figure 3: 2D target function $f_2$.
• Figure 4: Error curves for $f_1$ (top row) and $f_2$ (bottom row). Shaded regions indicate individual grades (G1--G4 for $f_1$ and G1--G3 for $f_2$). The horizontal axis denotes training epochs, and the vertical axis shows the base-10 logarithm of the error.
• Figure 5: Comparison of FCNN and MGDL via error curves for $f_1$ (top row) and $f_2$ (bottom row). Shaded bands indicate individual grades (G1--G4 for $f_1$ and G--G3 for $f_2$). The horizontal axis represents training epochs, and the vertical axis displays the base-10 logarithm of the error.

Theorems & Definitions (18)

• Theorem 1.1
• Corollary 1.2
• Theorem 3.1
• proof : Proof of Theorem \ref{['thm:main:mgdl']}
• Definition 4.1
• Definition 4.2
• Lemma 4.3
• proof
• Lemma 4.4
• proof