Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lexidate: Model Evaluation and Selection with Lexicase

Jose Guadalupe Hernandez, Anil Kumar Saini, Jason H. Moore

TL;DR

This work introduces lexidate, a lexicase-based holdout validation strategy for AutoML that trains pipelines on a learning set and scores predictions on a separate selection set to guide parent selection within TPOT2. By comparing three lexidate configurations with $10$-fold cross-validation on six OpenML classification tasks, the study demonstrates that lexidate can achieve similar accuracy to CV while often producing simpler models and delivering substantial reductions in training time. The results indicate that split choice profoundly influences performance, especially in smaller datasets, and suggest that per-example, non-aggregated evaluation pressures can be effective for evolutionary pipeline search. Overall, lexidate provides a computationally efficient alternative to CV with comparable generalization and practical benefits for AutoML workflows, while highlighting the need for data-driven split selection guidelines.

Abstract

Automated machine learning streamlines the task of finding effective machine learning pipelines by automating model training, evaluation, and selection. Traditional evaluation strategies, like cross-validation (CV), generate one value that averages the accuracy of a pipeline's predictions. This single value, however, may not fully describe the generalizability of the pipeline. Here, we present Lexicase-based Validation (lexidate), a method that uses multiple, independent prediction values for selection. Lexidate splits training data into a learning set and a selection set. Pipelines are trained on the learning set and make predictions on the selection set. The predictions are graded for correctness and used by lexicase selection to identify parent pipelines. Compared to 10-fold CV, lexicase reduces the training time. We test the effectiveness of three lexidate configurations within the Tree-based Pipeline Optimization Tool 2 (TPOT2) package on six OpenML classification tasks. In one configuration, we detected no difference in the accuracy of the final model returned from TPOT2 on most tasks compared to 10-fold CV. All configurations studied here returned similar or less complex final pipelines compared to 10-fold CV.

Lexidate: Model Evaluation and Selection with Lexicase

TL;DR

This work introduces lexidate, a lexicase-based holdout validation strategy for AutoML that trains pipelines on a learning set and scores predictions on a separate selection set to guide parent selection within TPOT2. By comparing three lexidate configurations with -fold cross-validation on six OpenML classification tasks, the study demonstrates that lexidate can achieve similar accuracy to CV while often producing simpler models and delivering substantial reductions in training time. The results indicate that split choice profoundly influences performance, especially in smaller datasets, and suggest that per-example, non-aggregated evaluation pressures can be effective for evolutionary pipeline search. Overall, lexidate provides a computationally efficient alternative to CV with comparable generalization and practical benefits for AutoML workflows, while highlighting the need for data-driven split selection guidelines.

Abstract

Automated machine learning streamlines the task of finding effective machine learning pipelines by automating model training, evaluation, and selection. Traditional evaluation strategies, like cross-validation (CV), generate one value that averages the accuracy of a pipeline's predictions. This single value, however, may not fully describe the generalizability of the pipeline. Here, we present Lexicase-based Validation (lexidate), a method that uses multiple, independent prediction values for selection. Lexidate splits training data into a learning set and a selection set. Pipelines are trained on the learning set and make predictions on the selection set. The predictions are graded for correctness and used by lexicase selection to identify parent pipelines. Compared to 10-fold CV, lexicase reduces the training time. We test the effectiveness of three lexidate configurations within the Tree-based Pipeline Optimization Tool 2 (TPOT2) package on six OpenML classification tasks. In one configuration, we detected no difference in the accuracy of the final model returned from TPOT2 on most tasks compared to 10-fold CV. All configurations studied here returned similar or less complex final pipelines compared to 10-fold CV.
Paper Structure (13 sections, 1 figure, 1 table)

This paper contains 13 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Model evaluation and selection
  3. TPOT2
  4. Lexicase selection
  5. Lexicase-based Validation Strategy
  6. Methods
  7. Evaluation strategy
  8. $10-$fold cross-validation
  9. Lexidate
  10. Model selection
  11. Experimental design
  12. Results
  13. Conclusion

Figures (1)

  • Figure 1: Raincloud plots of the accuracy values of the best models returned under different evaluation strategies. Each point in the plot corresponds to a different run.