Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

High-Dimensional Importance-Weighted Information Criteria: Theory and Optimality

Yong-Syun Cao, Shinpei Imori, Ching-Kang Ing

TL;DR

This work analyzes high-dimensional regression under covariate shift and potential misspecification, focusing on adaptive model selection when target responses are unavailable. It introduces the Importance-Weighted Orthogonal Greedy Algorithm (IWOGA) together with the High-Dimensional Importance-Weighted Information Criterion (HDIWIC), and proves that IWOGA+HDIWIC achieves the minimax-optimal convergence rate $d_n^{ rac{1+2oldsymbol{\xi}}{1+oldsymbol{\xi}}}$ without requiring knowledge of the sparsity parameter $oldsymbol{\xi}$. It also shows that, under correct model specification and moderate covariate shift, OGA+HDIC attains comparable optimal rates, with data-driven iteration counts valid when the training/test distributions are not too dissimilar. Additionally, the paper extends the framework to cases with unknown weights via HDIWIC_s, establishing adaptive and optimal performance in high-dimensional misspecified settings under covariate shift.

Abstract

Imori and Ing (2025) proposed the importance-weighted orthogonal greedy algorithm (IWOGA) for model selection in high-dimensional misspecified regression models under covariate shift. To determine the number of IWOGA iterations, they introduced the high-dimensional importance-weighted information criterion (HDIWIC). They argued that the combined use of IWOGA and HDIWIC, IWOGA + HDIWIC, achieves an optimal trade-off between variance and squared bias, leading to optimal convergence rates in terms of conditional mean squared prediction error. In this article, we provide a theoretical justification for this claim by establishing the optimality of IWOGA + HDIWIC under a set of reasonable assumptions.

High-Dimensional Importance-Weighted Information Criteria: Theory and Optimality

TL;DR

This work analyzes high-dimensional regression under covariate shift and potential misspecification, focusing on adaptive model selection when target responses are unavailable. It introduces the Importance-Weighted Orthogonal Greedy Algorithm (IWOGA) together with the High-Dimensional Importance-Weighted Information Criterion (HDIWIC), and proves that IWOGA+HDIWIC achieves the minimax-optimal convergence rate without requiring knowledge of the sparsity parameter . It also shows that, under correct model specification and moderate covariate shift, OGA+HDIC attains comparable optimal rates, with data-driven iteration counts valid when the training/test distributions are not too dissimilar. Additionally, the paper extends the framework to cases with unknown weights via HDIWIC_s, establishing adaptive and optimal performance in high-dimensional misspecified settings under covariate shift.

Abstract

Imori and Ing (2025) proposed the importance-weighted orthogonal greedy algorithm (IWOGA) for model selection in high-dimensional misspecified regression models under covariate shift. To determine the number of IWOGA iterations, they introduced the high-dimensional importance-weighted information criterion (HDIWIC). They argued that the combined use of IWOGA and HDIWIC, IWOGA + HDIWIC, achieves an optimal trade-off between variance and squared bias, leading to optimal convergence rates in terms of conditional mean squared prediction error. In this article, we provide a theoretical justification for this claim by establishing the optimality of IWOGA + HDIWIC under a set of reasonable assumptions.
Paper Structure (9 sections, 17 theorems, 128 equations)

This paper contains 9 sections, 17 theorems, 128 equations.

Table of Contents

  1. Introduction
  2. Convergence rate of IWOGA + HDIWIC
  3. Framework of IWOGA under Covariate Shift: Notation and Assumptions
  4. Main results
  5. Convergence rate of OGA + HDIC
  6. Proofs of Theorems \ref{['thm:thm2']}, \ref{['thm:thm3']}, and \ref{['thm:thm5']}
  7. Some basic lemmas
  8. Proof of Theorem \ref{['thm:thm2']} and \ref{['thm:thm3']}
  9. Proof of Theorem \ref{['thm:thm5']}

Key Result

Theorem 2.1

Assume (Ccond:cond1) -- (Ccond:cond6), eq:eq51 and eq:eq52. Suppose that for some sufficiently small $M_k > 0$, and for some $M_w > 0$ and $M_\eta \geq 1/\eta + 1/2$, $b_n$ (see ing25001) satisfies Then, where $L^{(\xi)}_n(m)=m^{-(1+2\xi)}+md_n^2$. Moreover, where $\varepsilon^{te}(\bm{x}|J) = y^{te}(\bm{x}) - y^{te}( \bm{x}|J)$ and $y^{te}( \bm{x}^{te}|J)= \alpha(J)+ \boldsymbol{\beta}(J)^\to

Theorems & Definitions (18)

  • Theorem 2.1: Theorem 3 of Imori202X
  • Theorem 2.2
  • Theorem 2.3
  • Theorem 3.1: Theorem 6 of Imori202X
  • Theorem 3.2
  • Lemma 4.1
  • Lemma 4.2
  • Lemma 4.3
  • Lemma 4.4
  • Lemma 4.5
  • ...and 8 more