Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Covariate-Elaborated Robust Partial Information Transfer with Conditional Spike-and-Slab Prior

Ruqian Zhang, Yijiao Zhang, Annie Qu, Zhongyi Zhu, Juan Shen

TL;DR

This work tackles data heterogeneity and the inefficiency of global similarity transfer in high-dimensional settings. It introduces CONCERT, a Bayesian method that uses a conditional spike-and-slab prior to enable covariate-specific partial information transfer across multiple sources, while a standard spike-and-slab handles target variable selection. A scalable variational Bayes implementation provides practical inference, with theoretical guarantees for true-posterior and VB posterior contraction, and variable/similarity selection consistency. Empirical results from simulations and real data (GTEx and Lending Club) demonstrate robust performance, showing meaningful gains when sources share partial information and mitigating negative transfer in heterogeneous settings.

Abstract

The popularity of transfer learning stems from the fact that it can borrow information from useful auxiliary datasets. Existing statistical transfer learning methods usually adopt a global similarity measure between the source data and the target data, which may lead to inefficiency when only partial information is shared. In this paper, we propose a novel Bayesian transfer learning method named ``CONCERT'' to allow robust partial information transfer for high-dimensional data analysis. A conditional spike-and-slab prior is introduced in the joint distribution of target and source parameters for information transfer. By incorporating covariate-specific priors, we can characterize partial similarities and integrate source information collaboratively to improve the performance on the target. In contrast to existing work, the CONCERT is a one-step procedure which achieves variable selection and information transfer simultaneously. We establish variable selection consistency, as well as estimation and prediction error bounds for CONCERT. Our theory demonstrates the covariate-specific benefit of transfer learning. To ensure the scalability of the algorithm, we adopt the variational Bayes framework to facilitate implementation. Extensive experiments and two real data applications showcase the validity and advantages of CONCERT over existing cutting-edge transfer learning methods.

Covariate-Elaborated Robust Partial Information Transfer with Conditional Spike-and-Slab Prior

TL;DR

This work tackles data heterogeneity and the inefficiency of global similarity transfer in high-dimensional settings. It introduces CONCERT, a Bayesian method that uses a conditional spike-and-slab prior to enable covariate-specific partial information transfer across multiple sources, while a standard spike-and-slab handles target variable selection. A scalable variational Bayes implementation provides practical inference, with theoretical guarantees for true-posterior and VB posterior contraction, and variable/similarity selection consistency. Empirical results from simulations and real data (GTEx and Lending Club) demonstrate robust performance, showing meaningful gains when sources share partial information and mitigating negative transfer in heterogeneous settings.

Abstract

The popularity of transfer learning stems from the fact that it can borrow information from useful auxiliary datasets. Existing statistical transfer learning methods usually adopt a global similarity measure between the source data and the target data, which may lead to inefficiency when only partial information is shared. In this paper, we propose a novel Bayesian transfer learning method named ``CONCERT'' to allow robust partial information transfer for high-dimensional data analysis. A conditional spike-and-slab prior is introduced in the joint distribution of target and source parameters for information transfer. By incorporating covariate-specific priors, we can characterize partial similarities and integrate source information collaboratively to improve the performance on the target. In contrast to existing work, the CONCERT is a one-step procedure which achieves variable selection and information transfer simultaneously. We establish variable selection consistency, as well as estimation and prediction error bounds for CONCERT. Our theory demonstrates the covariate-specific benefit of transfer learning. To ensure the scalability of the algorithm, we adopt the variational Bayes framework to facilitate implementation. Extensive experiments and two real data applications showcase the validity and advantages of CONCERT over existing cutting-edge transfer learning methods.
Paper Structure (22 sections, 5 theorems, 27 equations, 4 figures, 2 tables)

This paper contains 22 sections, 5 theorems, 27 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Spike-and-Slab for Variable Selection
  4. Conditional Spike-and-Slab for Similarity Selection
  5. Variational Bayes Implementation
  6. Example: Linear Regression
  7. Example: Logistic Regression
  8. Theoretical Results
  9. True Posterior Contraction
  10. Oracle theoretical results with known $T^*$
  11. Adaptive theoretical results with unknown $T^*$
  12. VB Posterior Contraction
  13. Simulation Studies
  14. Existence of Informative Set
  15. Heterogeneous Information
  16. ...and 7 more sections

Key Result

Theorem 1

Suppose $\min_{\{S:1 \leq |S|\leq L_s\}}n_S \gtrsim \log p$. Under Assumptions assump:sparsesignal- assump:betamin, the posterior probability of the true model $S^*$ satisfies that, with probability at least $1-\tilde{c}_3p^{-\tilde{c}_4}$, for some constant $\tilde{c}_1,\tilde{c}_3,\tilde{c}_4>0$, where the probability is with respect to the data-generating process.

Figures (4)

  • Figure 1: Illustration of the two cases of partial information similarity. In (a) and (c), the true variable identities are clarified with Target Positive, Source Positive, Target Negative, and Source Negative representing influential variables in the target, transferable variables in sources, non-influential variables in the target, and non-transferable variables in sources, respectively. In (b) and (d), the corresponding estimated posterior probabilities of all variables are shown in terms of significance in the target and transferability in sources.
  • Figure 2: Estimation errors with different informative set numbers $|A|$ and varying sizes $h$.
  • Figure 3: Estimation errors with different informative signal ratios and target signal sizes.
  • Figure 4: Estimation errors with different redundant signal numbers and strengths $\rho$.

Theorems & Definitions (7)

  • Remark 1
  • Remark 2
  • Theorem 1: Model selection
  • Theorem 2: Estimation
  • Theorem 3
  • Theorem 4: Model selection under VB posterior
  • Theorem 5: Estimation under VB posterior