Enhancing Variable Selection in Large-scale Logistic Regression: Leveraging Manual Labeling with Beneficial Noise

TL;DR

This work investigates variable selection in large-scale penalized logistic regression (PLR) and identifies beneficial information in label noise arising from manual labeling when the noise is tied to classification difficulty. It develops an ALASSO-PLR framework and proves that, under manual labeling modeled by posterior probabilities, the estimator retains the oracle property with asymptotic variance scaled by $\frac{m+\alpha_0}{m(1+\alpha_0)} I_{\mathcal{A}}^{-1}$; the asymptotic relative efficiency is $\text{ARE} = \frac{m(1+\alpha_0)}{m+\alpha_0}$. A partition-insensitive parallel ADMM algorithm is proposed to solve ALASSO-PLR with data distributed across multiple machines, with global convergence and a sublinear rate; experiments show higher estimation and classification accuracy on large-scale datasets when incorporating manual labels. The framework offers practical benefits for scalable high-dimensional PLR, especially in settings with missing labels or crowdsourced annotations, and provides guidance on leveraging expert-label information to enhance variable selection efficiency.

Abstract

In large-scale supervised learning, penalized logistic regression (PLR) effectively addresses the overfitting problem by introducing regularization terms yet its performance still depends on efficient variable selection strategies. This paper theoretically demonstrates that label noise stemming from manual labeling, which is solely related to classification difficulty, represents a type of beneficial noise for variable selection in PLR. This benefit is reflected in a more accurate estimation of the selected non-zero coefficients when compared with the case where only truth labels are used. Under large-scale settings, the sample size for PLR can become very large, making it infeasible to store on a single machine. In such cases, distributed computing methods are required to handle PLR model with manual labeling. This paper presents a partition-insensitive parallel algorithm founded on the ADMM (alternating direction method of multipliers) algorithm to address PLR by incorporating manual labeling. The partition insensitivity of the proposed algorithm refers to the fact that the solutions obtained by the algorithm will not change with the distributed storage of data. In addition, the algorithm has global convergence and a sublinear convergence rate. Experimental results indicate that, as compared with traditional variable selection classification techniques, the PLR with manually-labeled noisy data achieves higher estimation and classification accuracy across multiple large-scale datasets.

Figures (1)

• Figure 1: The implementation diagram of parallel algorithm.