Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improve Cost Efficiency of Active Learning over Noisy Dataset

Zan-Kai Chong, Hiroyuki Ohsaki, Bryan Ng

TL;DR

This paper considers cases of binary classification, where acquiring a positive instance incurs a significantly higher cost compared to that of negative instances, and proposes a shifted normal distribution sampling function that samples from a wider range than typical uncertainty sampling.

Abstract

Active learning is a learning strategy whereby the machine learning algorithm actively identifies and labels data points to optimize its learning. This strategy is particularly effective in domains where an abundance of unlabeled data exists, but the cost of labeling these data points is prohibitively expensive. In this paper, we consider cases of binary classification, where acquiring a positive instance incurs a significantly higher cost compared to that of negative instances. For example, in the financial industry, such as in money-lending businesses, a defaulted loan constitutes a positive event leading to substantial financial loss. To address this issue, we propose a shifted normal distribution sampling function that samples from a wider range than typical uncertainty sampling. Our simulation underscores that our proposed sampling function limits both noisy and positive label selection, delivering between 20% and 32% improved cost efficiency over different test datasets.

Improve Cost Efficiency of Active Learning over Noisy Dataset

TL;DR

This paper considers cases of binary classification, where acquiring a positive instance incurs a significantly higher cost compared to that of negative instances, and proposes a shifted normal distribution sampling function that samples from a wider range than typical uncertainty sampling.

Abstract

Active learning is a learning strategy whereby the machine learning algorithm actively identifies and labels data points to optimize its learning. This strategy is particularly effective in domains where an abundance of unlabeled data exists, but the cost of labeling these data points is prohibitively expensive. In this paper, we consider cases of binary classification, where acquiring a positive instance incurs a significantly higher cost compared to that of negative instances. For example, in the financial industry, such as in money-lending businesses, a defaulted loan constitutes a positive event leading to substantial financial loss. To address this issue, we propose a shifted normal distribution sampling function that samples from a wider range than typical uncertainty sampling. Our simulation underscores that our proposed sampling function limits both noisy and positive label selection, delivering between 20% and 32% improved cost efficiency over different test datasets.
Paper Structure (17 sections, 2 equations, 3 figures)

This paper contains 17 sections, 2 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Random Sampling and Uncertainty Sampling
  3. Random Sampling
  4. Uncertainty Sampling
  5. General Sampling Algorithm
  6. Sampling over Uncertainty and Noise Regions
  7. Assumptions and Setup
  8. Shifted Normal Sampling: Inspired by the Uncertainty Region and Noisy Region
  9. Imbalance Labelling Cost
  10. Normal Sampling Algorithm
  11. Step 1 - Initialization
  12. Step 2 - Instance Selection
  13. Step 3 - Model Refinement
  14. Simulation Results
  15. Simulation Overview
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Optimal probability distributions for positive and negative events, exhibiting minimal overlap area, and shifted normal samping function.
  • Figure 2: Illustration of the classification dataset with parameter class_sep=0.5 indicating overlap dataset. The sub-figures represent (a) bivariate distribution, (b) model performance measured in AUC, (c) ratio of positive events, and (d) cost efficiency across each query.
  • Figure 3: Representation of the classification dataset for parameter class_sep=1.0. (a) bivariate distribution, (b) AUC, (c) positive event ratio and (d) cost efficiency.