Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimization

Amirmohammad Farzaneh, Sangwoo Park, Osvaldo Simeone

TL;DR

The paper addresses risk-aware hyperparameter optimization by extending Learn-Then-Test to quantile risk control ($QLTT$). $QLTT$ tests hypotheses on the $q$-th quantile of the test-risk distribution using calibrated p-values and applies family-wise error rate control to select reliable hyperparameters. It demonstrates the approach on a radio access scheduling problem, showing that the selected hyperparameters meet a target $R_q(\\lambda) \\leq \\alpha$ with high probability, and that tail risk is improved relative to standard LTT. This framework enables engineers to guarantee reliability for a specified fraction of problem instances in engineering AI deployments.

Abstract

The increasing adoption of Artificial Intelligence (AI) in engineering problems calls for the development of calibration methods capable of offering robust statistical reliability guarantees. The calibration of black box AI models is carried out via the optimization of hyperparameters dictating architecture, optimization, and/or inference configuration. Prior work has introduced learn-then-test (LTT), a calibration procedure for hyperparameter optimization (HPO) that provides statistical guarantees on average performance measures. Recognizing the importance of controlling risk-aware objectives in engineering contexts, this work introduces a variant of LTT that is designed to provide statistical guarantees on quantiles of a risk measure. We illustrate the practical advantages of this approach by applying the proposed algorithm to a radio access scheduling problem.

Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimization

TL;DR

The paper addresses risk-aware hyperparameter optimization by extending Learn-Then-Test to quantile risk control (). tests hypotheses on the -th quantile of the test-risk distribution using calibrated p-values and applies family-wise error rate control to select reliable hyperparameters. It demonstrates the approach on a radio access scheduling problem, showing that the selected hyperparameters meet a target with high probability, and that tail risk is improved relative to standard LTT. This framework enables engineers to guarantee reliability for a specified fraction of problem instances in engineering AI deployments.

Abstract

The increasing adoption of Artificial Intelligence (AI) in engineering problems calls for the development of calibration methods capable of offering robust statistical reliability guarantees. The calibration of black box AI models is carried out via the optimization of hyperparameters dictating architecture, optimization, and/or inference configuration. Prior work has introduced learn-then-test (LTT), a calibration procedure for hyperparameter optimization (HPO) that provides statistical guarantees on average performance measures. Recognizing the importance of controlling risk-aware objectives in engineering contexts, this work introduces a variant of LTT that is designed to provide statistical guarantees on quantiles of a risk measure. We illustrate the practical advantages of this approach by applying the proposed algorithm to a radio access scheduling problem.
Paper Structure (10 sections, 2 theorems, 17 equations, 2 figures, 1 algorithm)

This paper contains 10 sections, 2 theorems, 17 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation and Overview
  3. Related Work
  4. Main Contributions
  5. Learn Then Test
  6. Quantile-Based LTT
  7. Simulations and Results
  8. Problem Setting
  9. Results
  10. Conclusion

Key Result

Lemma 1

Given calibration data $\mathcal{Z} = \{Z_i\}_{i=1}^n$, for any $0<\epsilon<1$, the following is a one-sided confidence interval for the $q$-quantile risk $R_q(\lambda)$ where $\hat{R}_{q^*}(\lambda, \epsilon)$ is the $q^*$-empirical quantile of the risk, i.e., the $\lfloor n(1-q^*)\rfloor$-th smallest element of the set $\left\{ R(Z_j,\lambda)\right\}_{j=1}^n$, with and

Figures (2)

  • Figure 1: Histogram of the average packet delay $R(Z,\lambda)$ for QoS class 1 for a single run of LTT and QLTT ($q$ = 0.1).
  • Figure 2: Distribution of the average risk $R(\hat{\lambda})$ and the quantile risk $R_q(\hat{\lambda})$ for LTT and QLTT for outage rates $q = 0.2$ (a) and $q = 0.1$ (b). The distributions are obtained by running Algorithm \ref{['alg::LTT']} multiple times, obtaining different realizations of hyperparameter $\hat{\lambda}$.

Theorems & Definitions (3)

  • Lemma 1: howard2022sequential
  • Proposition 1
  • proof