Sample complexity of data-driven tuning of model hyperparameters in neural networks with structured parameter-dependent dual function

TL;DR

The paper tackles data-driven tuning of continuous neural network hyperparameters under a task distribution, formalizing the learning problem via a dual-utility construction $u^*_{\boldsymbol{x}}(\alpha)$ defined as $u^*_{\boldsymbol{x}}(\alpha) = \max_{\boldsymbol{w}} f_{\boldsymbol{x}}(\alpha, \boldsymbol{w})$ where $f_{\boldsymbol{x}}(\alpha, \boldsymbol{w})$ is assumed to be piecewise polynomial. By developing a novel set of tools from differential and algebraic geometry, the authors bound the number of discontinuities and local extrema of the dual utility along the hyperparameter, linking these structural properties to pseudo-dimension and thus to learning guarantees for the data-driven hyperparameter choice. They provide concrete instance bounds for applications such as activation-function interpolation and graph-kernel tuning in GNNs, demonstrating that the resulting hyperparameter-tuning problems enjoy provable generalization bounds. The framework unifies piecewise-constant and piecewise-polynomial cases, offering a principled path toward extending learnability results to broader hyperparameter configurations and problem domains. Overall, the work advances theoretical understanding of hyperparameter tuning complexity in deep learning and supplies concrete, provable guarantees for practical data-driven configuration tasks.

Abstract

Modern machine learning algorithms, especially deep learning based techniques, typically involve careful hyperparameter tuning to achieve the best performance. Despite the surge of intense interest in practical techniques like Bayesian optimization and random search based approaches to automating this laborious and compute intensive task, the fundamental learning theoretic complexity of tuning hyperparameters for deep neural networks is poorly understood. Inspired by this glaring gap, we initiate the formal study of hyperparameter tuning complexity in deep learning through a recently introduced data driven setting. We assume that we have a series of deep learning tasks, and we have to tune hyperparameters to do well on average over the distribution of tasks. A major difficulty is that the utility function as a function of the hyperparameter is very volatile and furthermore, it is given implicitly by an optimization problem over the model parameters. To tackle this challenge, we introduce a new technique to characterize the discontinuities and oscillations of the utility function on any fixed problem instance as we vary the hyperparameter; our analysis relies on subtle concepts including tools from differential/algebraic geometry and constrained optimization. This can be used to show that the learning theoretic complexity of the corresponding family of utility functions is bounded. We instantiate our results and provide sample complexity bounds for concrete applications tuning a hyperparameter that interpolates neural activation functions and setting the kernel parameter in graph neural networks.

Figures (4)

• Figure 1: This figure demonstrates the oscillation property for a function $h: \mathbb{R} \rightarrow \mathbb{R}$. The oscillation of a function $h$ is defined as the maximum number of discontinuities in the function $\mathbb{I}_{\{h(\rho) \geq z\}}$, as the threshold $z$ varies. The figure shows several graphs of $\mathbb{I}_{\{h(\rho) \geq z\}}$ corresponding to different choices of the threshold $z$. We can observe that when $z = z_1$, the function $\mathbb{I}_{\{h(\rho) \geq z\}}$ exhibits the highest number of discontinuities, which is $4$. This number of discontinuities is also the maximum for any choice of $z$. Therefore, we conclude that $h$ has $4$ oscillations.
• Figure 2: A demonstration of the proof idea for \ref{['lm:piecewise-constant']}: We begin by partitioning the domain $\mathcal{A}$ of the dual utility function $u^*_{\boldsymbol{x}}(\alpha)$ into intervals. This partitioning is formed using two key points for each connected component $R$ in the partition $\mathcal{P}_{\boldsymbol{x}}$ of the domain $\mathcal{A} \times {\mathcal{W}}$ of $f_{\boldsymbol{x}}(\alpha, \boldsymbol{w})$: $\alpha_{R, \inf} = \inf_{\alpha}\{\alpha: \exists \boldsymbol{w}, (\alpha, \boldsymbol{w}) \in R\}$ and $\alpha_{R, \sup} = \sup_{\alpha}\{\alpha: \exists \boldsymbol{w}, (\alpha, \boldsymbol{w}) \in R\}$. Given that $\mathcal{P}$ contains $N$ elements, the number of such points is $\mathcal{O}(N)$. We demonstrate that the dual utility functions $u^*_{\boldsymbol{x}}$ remain constant over each interval defined by these points.
• Figure 3: A demonstration of the proof idea for Theorem \ref{['thm:learning-guarantee-piecewise-poly']} in 2D ($\boldsymbol{w} \in \mathbb{R}$). Here, the domain of $f^*_{\boldsymbol{x}}({\alpha}, \boldsymbol{w})$ is partitioned into four regions by two boundaries: a circle (blue line) and a parabola (green line). In each region $i$, the function $f_{\boldsymbol{x}}({\alpha}, \boldsymbol{w})$ is a polynomial $f_{\boldsymbol{x}, i}({\alpha}, \boldsymbol{w})$, of which the derivative curve $\frac{\partial f_{\boldsymbol{x}, i}}{\partial \boldsymbol{w}} = 0$ is demonstrated by the black dot in the plane of $({\alpha}, \boldsymbol{w})$. The value of $u^*_{\boldsymbol{x}}({\alpha})$ is demonstrated in the red line, and the red dots in the plane $({\alpha}, \boldsymbol{w})$ corresponds to the position where $f_{\boldsymbol{x}}({\alpha},\boldsymbol{w}) = u^*_{\boldsymbol{x}}({\alpha})$. We can see that it occurs in either the derivative curves or in the boundary. Our goal is to leverage this property to control the number of discontinuities and local maxima of $u^*_{\boldsymbol{x}}({\alpha})$, which can be converted to the generalization guarantee of the utility function class $\mathcal{U}$.
• Figure 4: A simplified illustration for the proof idea of Theorem \ref{['thm:learning-guarantee-piecewise-poly']} where $\boldsymbol{w} \in \mathbb{R}$. Here, our goal is to analyze the number of discontinuities and local maxima of $u^*_{\boldsymbol{x}, i}(\alpha)$. The idea is to partition the hyperparameter space $\mathcal{A}$ into intervals such that over each interval, the function $u^*_{\boldsymbol{x}, i}(\alpha)$ is the pointwise maximum of $f_{\boldsymbol{x}, i}(\alpha, \boldsymbol{w})$ along some fixed set of "monotonic curves" $\mathcal{C}$ (curves that intersect $\alpha=\alpha_0$ at most once for any $\alpha_0$). $u^*_{\boldsymbol{x}, i}(\alpha)$ is continuous over such interval; this implies that the interval endpoints contain all discontinuities of $u^*_{\boldsymbol{x}, i}(\alpha)$. In this example, over the interval $(\alpha_i, \alpha_{i + 1})$, we have $u^*_{\boldsymbol{x},i}(\alpha) = \max_{C_i}\{f_{\boldsymbol{x}, i}(\alpha, \boldsymbol{w}): (\alpha, \boldsymbol{w}) \in C_i\}$. Then, we can show that over such an interval, any local maximum of $u^*_{\boldsymbol{x}, i}(\alpha)$ is a local extremum of $f_{\boldsymbol{x}, i}(\alpha, \boldsymbol{w})$ along a monotonic curve $C \in \mathcal{C}$. Finally, we bound the number of points used for partitioning and local extrema using tools from algebraic and differential geometry.

Theorems & Definitions (92)

• Definition 1: Oscillations, balcan2021much
• Theorem 2.1: balcan2021much
• Lemma 3.1
• proof
• Lemma 3.2
• Definition 2: $B$-monotonicity
• Lemma 3.3
• proof
• Corollary 3.4
• Lemma 4.1