Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Feature selection based on cluster assumption in PU learning

Motonobu Uchikoshi, Youhei Akimoto

TL;DR

This work introduces FSCPU, a feature selection method for positive-unlabeled (PU) learning that leverages a cluster-based assumption to identify feature subsets enabling strong downstream classification. The core idea is to maximize a cluster-recall/precision product $f(\phi)$ by clustering transformed data and selecting clusters likely to contain positives, with an efficient linear-time subset search and a compact genetic algorithm to optimize a binary feature mask under a cost constraint. A theoretical link to $F_1$-like performance under MCAR is established, alongside practical insights into when the method excels (sparse, clustered positives) and when it may struggle (non-clustered positives or MCAR violations). Empirical validation shows FSCPU and its MI-augmented variant perform competitively against 9–10 baselines on artificial data under the cluster assumption and offer robust downstream performance on several open datasets, highlighting potential real-world applicability and avenues for speedups and pre-checks before deployment.

Abstract

Feature selection is essential for efficient data mining and sometimes encounters the positive-unlabeled (PU) learning scenario, where only a few positive labels are available, while most data remains unlabeled. In certain real-world PU learning tasks, data subjected to adequate feature selection often form clusters with concentrated positive labels. Conventional feature selection methods that treat unlabeled data as negative may fail to capture the statistical characteristics of positive data in such scenarios, leading to suboptimal performance. To address this, we propose a novel feature selection method based on the cluster assumption in PU learning, called FSCPU. FSCPU formulates the feature selection problem as a binary optimization task, with an objective function explicitly designed to incorporate the cluster assumption in the PU learning setting. Experiments on synthetic datasets demonstrate the effectiveness of FSCPU across various data conditions. Moreover, comparisons with 10 conventional algorithms on three open datasets show that FSCPU achieves competitive performance in downstream classification tasks, even when the cluster assumption does not strictly hold.

Feature selection based on cluster assumption in PU learning

TL;DR

This work introduces FSCPU, a feature selection method for positive-unlabeled (PU) learning that leverages a cluster-based assumption to identify feature subsets enabling strong downstream classification. The core idea is to maximize a cluster-recall/precision product by clustering transformed data and selecting clusters likely to contain positives, with an efficient linear-time subset search and a compact genetic algorithm to optimize a binary feature mask under a cost constraint. A theoretical link to -like performance under MCAR is established, alongside practical insights into when the method excels (sparse, clustered positives) and when it may struggle (non-clustered positives or MCAR violations). Empirical validation shows FSCPU and its MI-augmented variant perform competitively against 9–10 baselines on artificial data under the cluster assumption and offer robust downstream performance on several open datasets, highlighting potential real-world applicability and avenues for speedups and pre-checks before deployment.

Abstract

Feature selection is essential for efficient data mining and sometimes encounters the positive-unlabeled (PU) learning scenario, where only a few positive labels are available, while most data remains unlabeled. In certain real-world PU learning tasks, data subjected to adequate feature selection often form clusters with concentrated positive labels. Conventional feature selection methods that treat unlabeled data as negative may fail to capture the statistical characteristics of positive data in such scenarios, leading to suboptimal performance. To address this, we propose a novel feature selection method based on the cluster assumption in PU learning, called FSCPU. FSCPU formulates the feature selection problem as a binary optimization task, with an objective function explicitly designed to incorporate the cluster assumption in the PU learning setting. Experiments on synthetic datasets demonstrate the effectiveness of FSCPU across various data conditions. Moreover, comparisons with 10 conventional algorithms on three open datasets show that FSCPU achieves competitive performance in downstream classification tasks, even when the cluster assumption does not strictly hold.

Paper Structure

This paper contains 30 sections, 2 theorems, 13 equations, 2 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Problem Setting
  4. Proposed Method
  5. Objective Function for Feature Selection
  6. Efficient computation
  7. Relation to $F_1$-score
  8. Theoretical Insight
  9. Compact-GA-based optimizer
  10. Experiments with artificial data
  11. Hypothesis
  12. Experimental Settings
  13. Artificial Data
  14. Baseline
  15. FSCPU with Mutual information criterion (FSCPU-MI)
  16. ...and 15 more sections

Key Result

proposition 1

Let $\mathcal{K}_{*}$ be the subset of indices that maximize eq:f. In case there is more than one optimal subset, we select the greatest subset. Then, for any pair $(k, \ell)$ such that $k \in \mathcal{K}_*$ and $\ell \notin \mathcal{K}_*$,

Figures (2)

  • Figure 1: Convergence behavior of probability parameter $\theta$ on different datasets. Red and blue lines indicate $\theta_i$ corresponding to relevant and irrelevant features, respectively. The subfigure captions describe different experimental conditions corresponding to the top four rows of Table \ref{['tab1']} (i.e., conditions = {cluster assumption, labeled rate, no. negative cluster, no. positive cluster }).
  • Figure 2: Convergence processes of stochastic parameters in experiments with artificial data (all variations). (a)-(j): FSCPU results, (k)-(t): FSCPU-MI results.

Theorems & Definitions (2)

  • proposition 1
  • proposition 2