Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Feature Selection Based on Wasserstein Distance

Fuwei Li

TL;DR

This paper introduces a Markov blanket-based feature selection algorithm and shows that the Wasserstein distance-based feature selection method effectively reduces the impact of noisy labels without relying on specific noise models.

Abstract

This paper presents a novel feature selection method leveraging the Wasserstein distance to improve feature selection in machine learning. Unlike traditional methods based on correlation or Kullback-Leibler (KL) divergence, our approach uses the Wasserstein distance to assess feature similarity, inherently capturing class relationships and making it robust to noisy labels. We introduce a Markov blanket-based feature selection algorithm and demonstrate its effectiveness. Our analysis shows that the Wasserstein distance-based feature selection method effectively reduces the impact of noisy labels without relying on specific noise models. We provide a lower bound on its effectiveness, which remains meaningful even in the presence of noise. Experimental results across multiple datasets demonstrate that our approach consistently outperforms traditional methods, particularly in noisy settings.

Feature Selection Based on Wasserstein Distance

TL;DR

This paper introduces a Markov blanket-based feature selection algorithm and shows that the Wasserstein distance-based feature selection method effectively reduces the impact of noisy labels without relying on specific noise models.

Abstract

This paper presents a novel feature selection method leveraging the Wasserstein distance to improve feature selection in machine learning. Unlike traditional methods based on correlation or Kullback-Leibler (KL) divergence, our approach uses the Wasserstein distance to assess feature similarity, inherently capturing class relationships and making it robust to noisy labels. We introduce a Markov blanket-based feature selection algorithm and demonstrate its effectiveness. Our analysis shows that the Wasserstein distance-based feature selection method effectively reduces the impact of noisy labels without relying on specific noise models. We provide a lower bound on its effectiveness, which remains meaningful even in the presence of noise. Experimental results across multiple datasets demonstrate that our approach consistently outperforms traditional methods, particularly in noisy settings.

Paper Structure

This paper contains 11 sections, 4 theorems, 34 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. The Algorithm
  3. Problem Formulation
  4. The Wasserstein Distance
  5. Markov Blanket
  6. Feature Selection Algorithm
  7. Complexity Analysis
  8. Feature Selection with Noisy Labels
  9. Feature Selection Under Noisy Labels
  10. Simulation
  11. summary and future work

Key Result

Proposition 1

$p(Y|\mathbf X_{\mathcal{G}_i},X_i) = p(Y|\mathbf X_{\mathcal{G}_i})$ iif $W [p(Y|\mathbf X_{\mathcal{G}_i},X_i),p(Y|\mathbf X_{\mathcal{G}_i})] = 0$ for every $\mathbf{X}_{\mathcal{G}_i}=\mathbf{x}_{\mathcal{G}_i}, X_i = x_i$, where $\mathcal{G}_i$ is a subset of the features, $i \notin \mathcal{G}

Figures (8)

  • Figure 1: The first column indicates the original conditional distribution for different classes. The second column indicates the conditional distribution based on feature set $\theta_1$. The third column shows the conditional distribution based on feature subset $\theta_2$. If we use KL distance as the features selection criterion, the two selected feature subset will lead to the same distance. So, there will be no differences if we select those two feature sets. However, intuitively, the second feature subset will give us more reasonable results because the husky is more similar to the alaska than the cat.
  • Figure 2: The probabilistic graphic model, the Markov Blanket of node A contains the nodes in the shaded area.
  • Figure 3: The relationships between conditional distributions before feature selection and after feature selection in the original and observed data set respectively
  • Figure 4: The three pictures from the left to the right show the hierarchic structure of the labels, the weight assigned to the path between hierarchic labels, and one example to compute the class distance.
  • Figure 5: The top-k loss with different k and different number of features.
  • ...and 3 more figures

Theorems & Definitions (8)

  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Theorem 1
  • proof
  • Theorem 2
  • proof