Semi-supervised learning in unmatched linear regression using an empirical likelihood approach

TL;DR

This work addresses regression when only a small set of matched data is available alongside a large unmatched dataset. It proposes the Semi-Supervised Learning Empirical Maximum Likelihood Estimator (SSLEMLE) by maximizing an empirical log-likelihood that combines both data sources, and establishes existence, consistency, and asymptotic normality with an explicit covariance that depends on the matched/unmatched ratio $\lambda$. It also derives a closed-form or semi-closed-form expression for the statistical gain from the unlabeled data in Gaussian settings and analyzes how gain varies with signal-to-noise ratio and covariate mean. The approach is validated by extensive simulations and a real-data application to the combined cycle power plant dataset, demonstrating improved estimation precision and predictive performance as the unmatched sample grows. Overall, the paper provides a rigorous framework for leveraging unmatched data to enhance inference in linear regression, with potential extensions to other regression models.

Abstract

Knowing the link between observed predictive variables and outcomes is crucial for making inference in any regression model. When this link is missing, partially or completely, classical estimation methods fail in recovering the true regression function. Deconvolution approaches have been proposed and studied in detail in the unmatched setting where the predictive variables and responses are allowed to be independent. In this work, we consider linear regression in a semi-supervised learning setting where, beside a small sample of matched data, we have access to a relatively large unmatched sample. Using maximum likelihood estimation, we show that under some mild assumptions the semi-supervised learning empirical maximum likelihood estimator (SSLEMLE) is asymptotically normal and give explicitly its asymptotic covariance matrix as a function of the ratio of the matched/unmatched sample sizes and other parameters. Furthermore, we quantify the statistical gain achieved by having the additional large unmatched sample over having only the small matched sample. To illustrate the theory, we present the results of an extensive simulation study and apply our methodology to the "combined cycle power plant" data set.

Figures (12)

• Figure 1: Statistical gain versus $\zeta$ with fixed $\rho =1$, $\eta = 1$ and $\lambda = 0.1$.
• Figure 2: Results of simulation with $\epsilon \sim \mathcal{N}(0, (0.8\sqrt{10})^2)$ and $X \sim \mathcal{N}(0,\mathbbm{1}_{3\times 3})$.
• Figure 3: Results of simulation with $\epsilon \sim \mathcal{N}(0, (0.8\sqrt{10})^2)$ and $X \sim \mathcal{N}(5 \cdot \mathbf{1}_3,\mathbbm{1}_{3\times 3})$.
• Figure 4: Results of simulation with $\epsilon \sim \mathcal{N}(0, (0.8\sqrt{10})^2)$ and $X \sim \text{U}([-\sqrt{3}, \sqrt{3}]^3)$.
• Figure 5: Results of simulation with $\epsilon \sim \mathcal{N}(0, (0.8\sqrt{10})^2)$ and $X \sim \text{U}([5-\sqrt{3}, 5+\sqrt{3}]^3)$.

Theorems & Definitions (14)

• Lemma 1
• Lemma 2
• Theorem 1
• Theorem 2
• Theorem 3
• proof : Proof of Lemma \ref{['Lemma: Existence of MLE for fixed sample case']}
• proof : Proof of Lemma \ref{['Lemma: Existence of MLE for asymptotic sample case']}
• proof : Proof of Theorem \ref{['Consis']}
• proof : Proof of Theorem \ref{['AsympNorm']}
• proof : Proof of Theorem \ref{['gain']}.