Sparse Optimization for Transfer Learning: A L0-Regularized Framework for Multi-Source Domain Adaptation
Chenqi Gong, Hu Yang
TL;DR
This work tackles transfer learning under multi-source heterogeneity with high-dimensional parameters by introducing SOTL, a framework that employs $L_0$-regularization to enforce exact sparsity and concentrate optimization on target-related parameters. By reformulating the problem into an aggregated sparse regression and using HBIC for sparsity selection, SOTL mitigates bias associated with $L_1$ penalties and reduces computational burden. Through extensive simulations and empirical testing on the Communities and Crime dataset, SOTL consistently achieves lower estimation error and faster runtimes, particularly in the presence of adversarial auxiliary data or small sample sizes. Overall, SOTL offers a robust, scalable approach for cross-domain transfer in high-dimensional settings, with practical implications for real-world multi-source learning tasks.
Abstract
This paper explores transfer learning in heterogeneous multi-source environments with distributional divergence between target and auxiliary domains. To address challenges in statistical bias and computational efficiency, we propose a Sparse Optimization for Transfer Learning (SOTL) framework based on L0-regularization. The method extends the Joint Estimation Transferred from Strata (JETS) paradigm with two key innovations: (1) L0-constrained exact sparsity for parameter space compression and complexity reduction, and (2) refining optimization focus to emphasize target parameters over redundant ones. Simulations show that SOTL significantly improves both estimation accuracy and computational speed, especially under adversarial auxiliary domain conditions. Empirical validation on the Community and Crime benchmarks demonstrates the statistical robustness of the SOTL method in cross-domain transfer.