Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Extensions of regret-minimization algorithm for optimal design

Youguang Chen, George Biros

TL;DR

This work extends regret-minimization approaches to optimal experimental design by incorporating an entropy regularizer, yielding a novel sample-selection objective and provable near-optimality with a tilde-$O$(d/ε^2) sample complexity and potential data-dependent improvements. It further generalizes the framework to regularized designs (ridge regression) with matching complexity guarantees, and demonstrates practical utility by selecting unlabeled subsets for image classification tasks (MNIST, CIFAR-10, ImageNet-50) where models trained on the selected samples approach or surpass baselines. The proposed algorithm proceeds via a convex relaxation of the combinatorial design problem, followed by a rounding step implemented through Follow-the-Regularized-Leader (FTRL) with closed-form updates for multiple regularizers, and rigorous bounds back the rounding quality. Empirically, Regret-min with entropy regularization often matches or outperforms alternatives across diverse datasets and design criteria, with advantages in learning-rate stability and representativeness of selected unlabeled samples for downstream classifiers.

Abstract

We explore extensions and applications of the regret minimization framework introduced by~\cite{design} for solving optimal experimental design problems. Specifically, we incorporate the entropy regularizer into this framework, leading to a novel sample selection objective and a provable sample complexity bound that guarantees a $(1+ε)$-near optimal solution. We further extend the method to handle regularized optimal design settings. As an application, we use our algorithm to select a small set of representative samples from image classification datasets without relying on label information. To evaluate the quality of the selected samples, we train a logistic regression model and compare performance against several baseline sampling strategies. Experimental results on MNIST, CIFAR-10, and a 50-class subset of ImageNet show that our approach consistently outperforms competing methods in most cases.

Extensions of regret-minimization algorithm for optimal design

TL;DR

This work extends regret-minimization approaches to optimal experimental design by incorporating an entropy regularizer, yielding a novel sample-selection objective and provable near-optimality with a tilde-(d/ε^2) sample complexity and potential data-dependent improvements. It further generalizes the framework to regularized designs (ridge regression) with matching complexity guarantees, and demonstrates practical utility by selecting unlabeled subsets for image classification tasks (MNIST, CIFAR-10, ImageNet-50) where models trained on the selected samples approach or surpass baselines. The proposed algorithm proceeds via a convex relaxation of the combinatorial design problem, followed by a rounding step implemented through Follow-the-Regularized-Leader (FTRL) with closed-form updates for multiple regularizers, and rigorous bounds back the rounding quality. Empirically, Regret-min with entropy regularization often matches or outperforms alternatives across diverse datasets and design criteria, with advantages in learning-rate stability and representativeness of selected unlabeled samples for downstream classifiers.

Abstract

We explore extensions and applications of the regret minimization framework introduced by~\cite{design} for solving optimal experimental design problems. Specifically, we incorporate the entropy regularizer into this framework, leading to a novel sample selection objective and a provable sample complexity bound that guarantees a -near optimal solution. We further extend the method to handle regularized optimal design settings. As an application, we use our algorithm to select a small set of representative samples from image classification datasets without relying on label information. To evaluate the quality of the selected samples, we train a logistic regression model and compare performance against several baseline sampling strategies. Experimental results on MNIST, CIFAR-10, and a 50-class subset of ImageNet show that our approach consistently outperforms competing methods in most cases.

Paper Structure

This paper contains 48 sections, 24 theorems, 106 equations, 10 figures, 12 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Regression-based methods.
  4. Sparse modeling.
  5. Matrix approximation.
  6. Similarity graph-based methods.
  7. Notation
  8. Notation
  9. Optimal Experimental Design via Regret Minimization
  10. Relaxed Problem
  11. Rounding via regret minimization
  12. Goal of rounding
  13. Regret minimization
  14. Choose action matrix ${\bf A}_t$ by FTRL
  15. Sample selection
  16. ...and 33 more sections

Key Result

Lemma 2.1

$f^\diamond \triangleq f\left(\sum_{i=1}^n \pi_i^\diamond {{\bf x}}_i {{\bf x}}^\top_i \right) \leq f(\sum_{i=1}^n s_i^* {{\bf x}}_i {{\bf x}}^\top_i) \triangleq f^*$.

Figures (10)

  • Figure 1: Methods for selecting representative samples without label information.
  • Figure 1: Results on synthetic dataset ($d=100$). "$\ell_{1/2}$" and "Entropy" represent using $\ell_{1/2}$ or entropy regularizer for the regret minimization algorithm, respectively. "Uniform" represents averaged result of 10 trials by selecting samples uniformly. Top row presents results for optimal design problem \ref{['eq:design']}, the bottom row presents results for regularized optimal design problem \ref{['eq:reg-ip']}. $f_\diamond$ represents objective value using the solution from the relax problem.
  • Figure 1: 40 samples selected by different sampling methods on CIFAR-10 (we use first 40 eigenvectors of the normalized graph Laplacian as features). The number above each image represents its label.
  • Figure 2: Comparison of entropy-regularizer and $\ell_{1/2}$-regularizer in Regret-min algorithm (\ref{['algo:rounding']}) for A-design on CIFAR-10. The PCA features with dimension of 100 are used. The red lines in the plot represent relative value of the objective function $\frac{f({\bf X}_S^\top {\bf X}_S)}{f^\diamond}$, ${\bf X}_S$ is the selected samples and $f^\diamond$ is the optimal value of the relaxed problem \ref{['eq:lp']}. The blue lines represent the logistic regression prediction accuracy. The dots on each line represent the optimal point.
  • Figure 2: 100 ImageNet-50 samples selected by one trial of the Random and K-means using the PCA features with dimension of 100. The number of classes collected and the corresponding logistic regression prediction accuracy are reported within the brackets.
  • ...and 5 more figures

Theorems & Definitions (43)

  • Definition 1.1: Experimental design problem
  • Lemma 2.1
  • Proposition 2.2
  • Proof 1
  • Proposition 2.3: closed forms of ${\bf A}_t$ by FTRL
  • Theorem 2.4: lower bound of $\lambda_{\min}\left(\sum_{t\in[k]} {\bf F}_t\right)$ by FTRL
  • Definition 2.5: point selection objective
  • Theorem 2.6
  • Proposition 3.1
  • Theorem 3.2: lower bound of $\lambda_{\min}\left(\sum_{t\in[k]} {\bf F}_t\right)$ by FTRL for regularized case
  • ...and 33 more