Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Estimating Unbounded Density Ratios: Applications in Error Control under Covariate Shift

Shuntuo Xu, Zhou Yu, Jian Huang

TL;DR

This work develops a theoretical framework for estimating density ratios under covariate shift when both the domain and range can be unbounded, by leveraging Bregman divergences with least-squares and logistic-regression losses. It establishes near-minimax estimation rates under local Hölder smoothness and sub-exponential tail assumptions, with neural networks and truncation enabling practical implementation on unbounded spaces. A key insight is that the tail behavior of r_0(X^s) can enable effective generalization from the source to the target without explicit density-ratio loss correction, and in some cases even outperforming corrected estimators. The theory is applied to nonparametric regression and conditional flow models, with simulation studies showing that source estimators generalize well across covariate shifts and can be preferable to density-ratio corrections in practice. Overall, the paper provides both rigorous error bounds and practical guidance for transfer learning tasks where densities are unbounded and shifts are present.

Abstract

The density ratio is an important metric for evaluating the relative likelihood of two probability distributions, with extensive applications in statistics and machine learning. However, existing estimation theories for density ratios often depend on stringent regularity conditions, mainly focusing on density ratio functions with bounded domains and ranges. In this paper, we study density ratio estimators using loss functions based on least squares and logistic regression. We establish upper bounds on estimation errors with standard minimax optimal rates, up to logarithmic factors. Our results accommodate density ratio functions with unbounded domains and ranges. We apply our results to nonparametric regression and conditional flow models under covariate shift and identify the tail properties of the density ratio as crucial for error control across domains affected by covariate shift. We provide sufficient conditions under which loss correction is unnecessary and demonstrate effective generalization capabilities of a source estimator to any suitable target domain. Our simulation experiments support these theoretical findings, indicating that the source estimator can outperform those derived from loss correction methods, even when the true density ratio is known.

Estimating Unbounded Density Ratios: Applications in Error Control under Covariate Shift

TL;DR

This work develops a theoretical framework for estimating density ratios under covariate shift when both the domain and range can be unbounded, by leveraging Bregman divergences with least-squares and logistic-regression losses. It establishes near-minimax estimation rates under local Hölder smoothness and sub-exponential tail assumptions, with neural networks and truncation enabling practical implementation on unbounded spaces. A key insight is that the tail behavior of r_0(X^s) can enable effective generalization from the source to the target without explicit density-ratio loss correction, and in some cases even outperforming corrected estimators. The theory is applied to nonparametric regression and conditional flow models, with simulation studies showing that source estimators generalize well across covariate shifts and can be preferable to density-ratio corrections in practice. Overall, the paper provides both rigorous error bounds and practical guidance for transfer learning tasks where densities are unbounded and shifts are present.

Abstract

The density ratio is an important metric for evaluating the relative likelihood of two probability distributions, with extensive applications in statistics and machine learning. However, existing estimation theories for density ratios often depend on stringent regularity conditions, mainly focusing on density ratio functions with bounded domains and ranges. In this paper, we study density ratio estimators using loss functions based on least squares and logistic regression. We establish upper bounds on estimation errors with standard minimax optimal rates, up to logarithmic factors. Our results accommodate density ratio functions with unbounded domains and ranges. We apply our results to nonparametric regression and conditional flow models under covariate shift and identify the tail properties of the density ratio as crucial for error control across domains affected by covariate shift. We provide sufficient conditions under which loss correction is unnecessary and demonstrate effective generalization capabilities of a source estimator to any suitable target domain. Our simulation experiments support these theoretical findings, indicating that the source estimator can outperform those derived from loss correction methods, even when the true density ratio is known.

Paper Structure

This paper contains 35 sections, 31 theorems, 325 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Bregman divergence
  4. Local Hölder class
  5. Neural networks
  6. Covariate shift
  7. Flow-based generative learning
  8. Density ratio estimation: error analysis
  9. Estimation based on least squares loss
  10. Estimation based on logistic regression loss
  11. Error control under covariate shift
  12. Nonparametric regression and conditional flow models
  13. Nonparametric regression
  14. Conditional flow models
  15. Simulation studies
  16. ...and 20 more sections

Key Result

Theorem 3.1

Assume that Then, given the hyper-parameters $L$ set to $\mathcal{O}(n^{d/(2d+4\beta_r)}\log n)$, $M$ set to $\mathcal{O}(1)$, $\underline{\delta}$ set to 0 and $\bar{\delta}=\bar{\delta}_n$ set to $(\log n)^{1+\kappa}$ with an arbitrarily fixed $\kappa\in (0, 1]$, for $n\ge 2$, we have where $c^*$ and $c^{**}$ are constants not depending on $n$.

Figures (2)

  • Figure 1: Boxplots of logarithm of mean square errors between the true density ratio values and predicted values, based on results from 100 replications.
  • Figure 2: Boxplots of mean square errors between the true conditional mean values and predicted values, based on results from 100 replications. SERS means the source estimator risk in source domain and SERT represents the source estimator risk in target domain.

Theorems & Definitions (67)

  • Definition 2.1: Bregman divergence
  • Definition 2.2: Hölder class
  • Definition 2.3: Local Hölder class
  • Remark 1
  • Theorem 3.1
  • Lemma 3.2
  • Theorem 3.3
  • Lemma 4.1
  • Remark 2
  • Corollary 4.2
  • ...and 57 more