Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Estimation of Models with Limited Data by Leveraging Shared Structure

Maryann Rui, Thibaut Horel, Munther Dahleh

TL;DR

This paper develops a three-step method which estimates the low-dimensional subspace spanned by the systems' parameters and produces refined parameter estimates within the subspace, and provides finite sample subspace estimation error guarantees.

Abstract

Modern data sets, such as those in healthcare and e-commerce, are often derived from many individuals or systems but have insufficient data from each source alone to separately estimate individual, often high-dimensional, model parameters. If there is shared structure among systems however, it may be possible to leverage data from other systems to help estimate individual parameters, which could otherwise be non-identifiable. In this paper, we assume systems share a latent low-dimensional parameter space and propose a method for recovering $d$-dimensional parameters for $N$ different linear systems, even when there are only $T<d$ observations per system. To do so, we develop a three-step algorithm which estimates the low-dimensional subspace spanned by the systems' parameters and produces refined parameter estimates within the subspace. We provide finite sample subspace estimation error guarantees for our proposed method. Finally, we experimentally validate our method on simulations with i.i.d. regression data and as well as correlated time series data.

Estimation of Models with Limited Data by Leveraging Shared Structure

TL;DR

This paper develops a three-step method which estimates the low-dimensional subspace spanned by the systems' parameters and produces refined parameter estimates within the subspace, and provides finite sample subspace estimation error guarantees.

Abstract

Modern data sets, such as those in healthcare and e-commerce, are often derived from many individuals or systems but have insufficient data from each source alone to separately estimate individual, often high-dimensional, model parameters. If there is shared structure among systems however, it may be possible to leverage data from other systems to help estimate individual parameters, which could otherwise be non-identifiable. In this paper, we assume systems share a latent low-dimensional parameter space and propose a method for recovering -dimensional parameters for different linear systems, even when there are only observations per system. To do so, we develop a three-step algorithm which estimates the low-dimensional subspace spanned by the systems' parameters and produces refined parameter estimates within the subspace. We provide finite sample subspace estimation error guarantees for our proposed method. Finally, we experimentally validate our method on simulations with i.i.d. regression data and as well as correlated time series data.
Paper Structure (25 sections, 6 theorems, 49 equations, 5 figures, 1 algorithm)

This paper contains 25 sections, 6 theorems, 49 equations, 5 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mixtures of linear regressions
  4. Low rank matrix regression
  5. Dictionary learning.
  6. Meta-learning and transfer learning.
  7. Preliminaries
  8. General notations
  9. Subspaces
  10. Random variables
  11. Model
  12. Three-Step Estimator
  13. Initial individual estimates
  14. Normalization.
  15. Comparison with dictionary learning.
  16. ...and 10 more sections

Key Result

Theorem 5.1

Let $\hat{\mathcal{B}}$ be the subspace spanned by the columns of the output $\widehat{B}$ in alg:norm with threshold level $s=\Omega(1/(\sqrt d -\sqrt{T-1}))$ if $T\leq d$, and $s =\Omega(1/(\sqrt T -\sqrt d))$ if $T>d$. Then for each $0<\delta<1$, with probability at least $1-\delta$

Figures (5)

  • Figure 1: Subspace estimation error vs. $N$ for estimates $\widehat{B}$ from (1) thresh and (2) norm. Results are shown for $d=50, r=5$, in regimes $d>T=10$, $T=d=50$, and $d<T=80$.
  • Figure 2: Comparison of $\beta_i$ estimation error of thresh as compared to oracle and naive estimators. Here, $d = 5, r = 1, T=3$.
  • Figure 3: Subspace estimation error vs. $N$ for estimates $\widehat{B}$ from (1) norm and (2) MoM, for i.i.d. data. Results are shown for $d=50, r=5$, in regimes $d>T=10$, $T=d=50$, and $d<T=80$..
  • Figure 4: Subspace estimation error vs. $T$ for estimates $\widehat{B}$ from (1) thresh, (2) norm, and (3) MoM, for i.i.d. data. Results are shown for $d=50, r=5$, for various values of $N$.
  • Figure 5: Subspace estimation error vs. $N$ for estimates $\widehat{B}$ from (1) norm and (2) MoM, for time-series data. Results are shown for $d=20, r=5$, in regimes $d>T=8$, $T=d=20$, and $d<T=36$.

Theorems & Definitions (12)

  • Theorem 5.1
  • Lemma A.1
  • proof
  • Lemma A.2
  • proof
  • Lemma A.3
  • proof
  • Lemma B.1
  • proof
  • Lemma B.2
  • ...and 2 more