Provably Data-driven Multiple Hyper-parameter Tuning with Structured Loss Function

TL;DR

This work provides a principled statistical framework for provable generalization guarantees in data-driven, multi-dimensional hyperparameter tuning. By representing the inner bi-level optimization and outer validation/training objectives as polynomial first-order formulas, the authors bound the learning-theoretic complexity via the pseudo-dimension using quantifier-elimination and the Goldberg–Jerrum framework. The key contributions include general pseudo-dimension bounds for training-based tuning and validation-based tuning with multi-dimensional hyperparameters, refined bounds when explicit solution paths exist, and the first learnability results for data-driven weighted group LASSO and weighted fused LASSO. The approach offers rigorous foundations beyond grid search and Bayesian methods, with practical implications for structured regularization and other semi-algebraic objectives, and points to future work on lower bounds and broader semi- and o-minimal extensions.

Abstract

Data-driven algorithm design automates hyperparameter tuning, but its statistical foundations remain limited because model performance can depend on hyperparameters in implicit and highly non-smooth ways. Existing guarantees focus on the simple case of a one-dimensional (scalar) hyperparameter. This leaves the practically important, multi-dimensional hyperparameter tuning setting unresolved. We address this open question by establishing the first general framework for establishing generalization guarantees for tuning multi-dimensional hyperparameters in data-driven settings. Our approach strengthens the generalization guarantee framework for semi-algebraic function classes by exploiting tools from real algebraic geometry, yielding sharper, more broadly applicable guarantees. We then extend the analysis to hyperparameter tuning using the validation loss under minimal assumptions, and derive improved bounds when additional structure is available. Finally, we demonstrate the scope of the framework with new learnability results, including data-driven weighted group lasso and weighted fused lasso.

Figures (1)

• Figure 1: A simple illustration of the piecewise polynomial structure. Here, the set of boundary polynomials $\mathbb{H} = \{h_1\}$, where $h_1(z) = z_1^2 + z_2^2 - 4$. In the region $\{z \in \mathbb{R}^2 \mid h_1(z) > 0\}$ (i.e, the region outside the circle, the sign pattern $\boldsymbol{\sigma}(z) = (1) \in \{-1, 0, 1\}^1$), the function $f(z)$ admits the polynomial form $f_{(-1)}(z) = z_1 - z_2$. Similarly, in the region $\{z \in \mathbb{R}^2 \mid h_1(z) < 0\}$ inside the circle (i.e., the sign pattern $\boldsymbol{\sigma}(z) = (-1)$), we have $f(z) = f_{(-1)}(z) = z_1 + z_2$. Finally, in the region where $\{\boldsymbol{z} \in \mathbb{R}^2 \mid h_1(z) = 0\}$ (i.e., the boundary of the circle, $\boldsymbol{\sigma}(z) = (0)$), $f(z) = f_{(0)}(z) = z_1^3$. Note that all pieces are polynomials in $z$, and the complexity of the function $f(z)$ is $(1, 3, \Delta)$, where $\Delta = \max\{\textup{deg}(h_1), \textup{deg}(f_{(0)}), \textup{deg}(f_{(-1)}), \textup{deg}(f_{(1)})\} = 3$.

Theorems & Definitions (30)

• Definition 2.1: Pseudo-dimension, pollard1984convergence
• Theorem 2.2: pollard1984convergence
• Definition 2.3: First-order formula, quantified/free variables, and polynomial first-order logic renegar1992computational
• Definition 2.4: Quantifier-free formula
• Example 1
• Theorem 2.5: Quantifier elimination algorithm, basu2006algorithms
• Definition 3.1: Piecewise polynomial function balcan2025sample
• Remark 3.3: Ubiquity of structure
• Theorem 4.1: Pseudo-dimension bound
• Theorem 5.1: Pseudo-dimension -- Training loss