Relax and penalize: a new bilevel approach to mixed-binary hyperparameter optimization

TL;DR

This paper tackles mixed-binary hyperparameter optimization within a bilevel framework by replacing the hard binary constraint with a smooth concave penalty, proving that, for sufficiently small penalties, the global minimizers of the penalized problem coincide with those of the original mixed-binary problem. An iterative penalty method solves a sequence of continuous problems, guaranteeing convergence to mixed-binary local solutions under mild assumptions. The approach is demonstrated on two ML tasks: learning group-sparsity structures (group lasso) and data distillation, where it competes with or outperforms relaxation+rounding, especially in the data-distillation setting where budget constraints are critical. Overall, the method enables principled integration of mixed-binary hyperparameters into standard continuous bilevel solvers, offering theoretical guarantees and practical effectiveness.

Abstract

In recent years, bilevel approaches have become very popular to efficiently estimate high-dimensional hyperparameters of machine learning models. However, to date, binary parameters are handled by continuous relaxation and rounding strategies, which could lead to inconsistent solutions. In this context, we tackle the challenging optimization of mixed-binary hyperparameters by resorting to an equivalent continuous bilevel reformulation based on an appropriate penalty term. We propose an algorithmic framework that, under suitable assumptions, is guaranteed to provide mixed-binary solutions. Moreover, the generality of the method allows to safely use existing continuous bilevel solvers within the proposed framework. We evaluate the performance of our approach for two specific machine learning problems, i.e., the estimation of the group-sparsity structure in regression problems and the data distillation problem. The reported results show that our method is competitive with state-of-the-art approaches based on relaxation and rounding

Figures (7)

• Figure 1: Example of oracle group structure with random sizes and parameter $a=0.5$, and the corrisponding $\theta$ obtained by the relaxation and rounding before the rounding procedure (ref. as $r$), after both simple (ref. as $s$) and top-$1$ hard thresholding rounding (ref. as $top$), and the relax and penalize (ref. as $p$) methods.
• Figure 2: Hypergradient computation (reverse mode)
• Figure 3: Evolution of $\mathop{\mathrm{dist}}\nolimits_\infty (\theta^k, \Theta_{\mathrm{bin}})$ for the setting with inequal group sizes and $a = 0.3$.
• Figure 4: Evolution of $\mathop{\mathrm{dist}}\nolimits_\infty (v^k, \Theta_{\mathrm{bin}})$ for the blog dataset and budget 20%.
• Figure 5: Evolution of the quantities reported in Table \ref{['tab:table3']} for the relaxation and rounding method with simple rounding on the music dataset with distillation budget at $20\%$ of the training set size.

Theorems & Definitions (20)

• Theorem 1
• Remark 1
• Theorem 2
• proof
• Remark 2
• Theorem 3
• proof
• Remark 3
• Remark 4
• Lemma 1