Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Sample Sharing for Linear Regression

Hamza Cherkaoui, Hélène Halconruy, Yohan Petetin

TL;DR

This work tackles data scarcity in target regression by proposing a target-focused selective sample-sharing rule for ridge regression that borrows auxiliary samples only when they reduce target risk. The approach hinges on a transfer-gain estimator $\widehat{\Delta}(n)$ and a conservative decision rule, implemented via the Triple-S algorithm, which uses Sherman–Morrison updates to scale over potential source prefixes. The authors provide finite-sample guarantees, analyze how key factors (alignment, task difference, noise levels) influence the gain under an isotropic Gaussian design, and validate the method on synthetic and real datasets where it consistently improves over single-task training while avoiding negative transfer. The practical impact lies in a safe, data-driven mechanism to leverage related data sources without compromising the target objective, with clear pathways to extend to broader models and multiple sources.

Abstract

In many business settings, task-specific labeled data are scarce or costly to obtain, which limits supervised learning on a specific task. To address this challenge, we study sample sharing in the case of ridge regression: leveraging an auxiliary data set while explicitly protecting against negative transfer. We introduce a principled, data-driven rule that decides how many samples from an auxiliary dataset to add to the target training set. The rule is based on an estimate of the transfer gain i.e. the marginal reduction in the predictive error. Building on this estimator, we derive finite-sample guaranties: under standard conditions, the procedure borrows when it improves parameter estimation and abstains otherwise. In the Gaussian feature setting, we analyze which data set properties ensure that borrowing samples reduces the predictive error. We validate the approach in synthetic and real datasets, observing consistent gains over strong baselines and single-task training while avoiding negative transfer.

Adaptive Sample Sharing for Linear Regression

TL;DR

This work tackles data scarcity in target regression by proposing a target-focused selective sample-sharing rule for ridge regression that borrows auxiliary samples only when they reduce target risk. The approach hinges on a transfer-gain estimator and a conservative decision rule, implemented via the Triple-S algorithm, which uses Sherman–Morrison updates to scale over potential source prefixes. The authors provide finite-sample guarantees, analyze how key factors (alignment, task difference, noise levels) influence the gain under an isotropic Gaussian design, and validate the method on synthetic and real datasets where it consistently improves over single-task training while avoiding negative transfer. The practical impact lies in a safe, data-driven mechanism to leverage related data sources without compromising the target objective, with clear pathways to extend to broader models and multiple sources.

Abstract

In many business settings, task-specific labeled data are scarce or costly to obtain, which limits supervised learning on a specific task. To address this challenge, we study sample sharing in the case of ridge regression: leveraging an auxiliary data set while explicitly protecting against negative transfer. We introduce a principled, data-driven rule that decides how many samples from an auxiliary dataset to add to the target training set. The rule is based on an estimate of the transfer gain i.e. the marginal reduction in the predictive error. Building on this estimator, we derive finite-sample guaranties: under standard conditions, the procedure borrows when it improves parameter estimation and abstains otherwise. In the Gaussian feature setting, we analyze which data set properties ensure that borrowing samples reduces the predictive error. We validate the approach in synthetic and real datasets, observing consistent gains over strong baselines and single-task training while avoiding negative transfer.
Paper Structure (51 sections, 2 theorems, 49 equations, 4 figures, 1 algorithm)

This paper contains 51 sections, 2 theorems, 49 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Previous contributions
  4. Transfer Learning: General Overview
  5. Transfer Learning as Regularization for Linear Models
  6. Selective Sample Sharing Across Tasks
  7. Our contributions
  8. Preliminaries
  9. Notation
  10. Ridge regression (single task)
  11. Transfer learning via sample sharing
  12. Collaborative ridge formulation
  13. When is sample sharing beneficial?
  14. Estimating the transfer gain
  15. Our approach
  16. ...and 36 more sections

Key Result

Theorem 6.1

Assume an isotropic Gaussian design with $n\to\infty$, $d=o(n)$, fixed $n_T, n_T^{\mathrm{val}}$, and $\lambda_T,\lambda_c=O(1)$. Let $\bm{\Delta\theta}^\star:=\bm{\theta}_S^\star-\bm{\theta}_T^\star$ and $\varepsilon^2:=\|\bm{\Delta\theta}^\star\|_2^2$. Then

Figures (4)

  • Figure 1: Flowchart of the main steps of our approach.
  • Figure 2: Predictive error comparison w.r.t. the number of target samples. The solid line reports the average; while the standard deviation is encoded in transparency.
  • Figure 3: Predictive error comparison w.r.t. the number of target samples. The solid line reports the average; while the standard deviation is encoded in transparency.
  • Figure 4: Predictive error comparison (left axis) and the number of samples borrowed $n^\star$ (right axis) w.r.t. the source observation noise variance. The solid line reports the average; while the standard deviation is encoded in transparency.

Theorems & Definitions (12)

  • Definition 4.1: Transfer gain
  • Definition 4.2: Positive Transfer gain
  • Definition 4.3: Estimated transfer gain
  • Remark 4.1: Unbiased estimator of $\Delta^\star(n)$
  • proof : Sketch of proof
  • Definition 5.1: Practical transfer gain
  • Theorem 6.1: Transfer gain in the large-source isotropic Gaussian regime
  • proof : Sketch of proof
  • proof : Proof of \ref{['prop:expect_var_delta_hat']}
  • proof : Proof of \ref{['prop:lower_bound_transfer_gain']}
  • ...and 2 more