Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robustness of AutoML on Dirty Categorical Data

Marcos L. P. Bueno, Joaquin Vanschoren

TL;DR

This work tackles the challenge of applying AutoML to datasets with dirty, high-cardinality categorical features. It introduces a pipeline that converts categorical variables into numerical representations using morphological encoders (Similarity, Min-hash, GAP) and couples them with an AutoML system (GAMA) to improve robustness and performance. Through empirical evaluation on several real-world dirty datasets, the study shows that the proposed pipeline enables AutoML to finish on all datasets and yields notable accuracy gains (e.g., Midwest from $0.50$ to $0.68$) compared to using standard encoders alone. The findings highlight the practical impact of integrating advanced categorical encoders into AutoML to better handle real-world data imperfections and expand the applicability of AutoML in domains with messy categorical information.

Abstract

The goal of automated machine learning (AutoML) is to reduce trial and error when doing machine learning (ML). Although AutoML methods for classification are able to deal with data imperfections, such as outliers, multiple scales and missing data, their behavior is less known on dirty categorical datasets. These datasets often have several categorical features with high cardinality arising from issues such as lack of curation and automated collection. Recent research has shown that ML models can benefit from morphological encoders for dirty categorical data, leading to significantly superior predictive performance. However the effects of using such encoders in AutoML methods are not known at the moment. In this paper, we propose a pipeline that transforms categorical data into numerical data so that an AutoML can handle categorical data transformed by more advanced encoding schemes. We benchmark the current robustness of AutoML methods on a set of dirty datasets and compare it with the proposed pipeline. This allows us to get insight on differences in predictive performance. We also look at the ML pipelines built by AutoMLs in order to gain insight beyond the best model as typically returned by these methods.

Robustness of AutoML on Dirty Categorical Data

TL;DR

This work tackles the challenge of applying AutoML to datasets with dirty, high-cardinality categorical features. It introduces a pipeline that converts categorical variables into numerical representations using morphological encoders (Similarity, Min-hash, GAP) and couples them with an AutoML system (GAMA) to improve robustness and performance. Through empirical evaluation on several real-world dirty datasets, the study shows that the proposed pipeline enables AutoML to finish on all datasets and yields notable accuracy gains (e.g., Midwest from to ) compared to using standard encoders alone. The findings highlight the practical impact of integrating advanced categorical encoders into AutoML to better handle real-world data imperfections and expand the applicability of AutoML in domains with messy categorical information.

Abstract

The goal of automated machine learning (AutoML) is to reduce trial and error when doing machine learning (ML). Although AutoML methods for classification are able to deal with data imperfections, such as outliers, multiple scales and missing data, their behavior is less known on dirty categorical datasets. These datasets often have several categorical features with high cardinality arising from issues such as lack of curation and automated collection. Recent research has shown that ML models can benefit from morphological encoders for dirty categorical data, leading to significantly superior predictive performance. However the effects of using such encoders in AutoML methods are not known at the moment. In this paper, we propose a pipeline that transforms categorical data into numerical data so that an AutoML can handle categorical data transformed by more advanced encoding schemes. We benchmark the current robustness of AutoML methods on a set of dirty datasets and compare it with the proposed pipeline. This allows us to get insight on differences in predictive performance. We also look at the ML pipelines built by AutoMLs in order to gain insight beyond the best model as typically returned by these methods.
Paper Structure (19 sections, 1 equation, 1 figure, 3 tables, 1 algorithm)

This paper contains 19 sections, 1 equation, 1 figure, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Simple categorical encoders
  4. One-hot encoding
  5. Ordinal encoding
  6. Encoders for high-cardinality features
  7. Target encoding
  8. Similarity encoding
  9. Min-hash encoding
  10. GAP encoding
  11. Automated machine learning
  12. Related work
  13. Proposed pipeline
  14. Experimental results
  15. Setup and data
  16. ...and 4 more sections

Figures (1)

  • Figure 1: ML pipelines built by GAMA for the Midwest dataset. Left: original GAMA; Right: GAMA+new pipeline. Note the different scales.