Good practices for evaluation of machine learning systems

TL;DR

The paper tackles how evaluation protocol design determines whether ML results generalize to deployment data. It advocates a structured approach focusing on data selection, metric design, and statistical significance, arguing that poor evaluation can mislead development decisions. It introduces practical guidelines and concrete metrics (e.g., $EC$, $NCE$, $NEC$, $NTE$, $CE$, $WER$, $DTW$) and recommends bootstrapped confidence intervals to quantify uncertainty. The discussion emphasizes checking for spurious correlations, handling data scarcity with cross-validation, and distinguishing evaluation of systems versus methods. Although illustrated with speech-processing examples, the recommendations aim to generalize across domains.

Abstract

Many development decisions affect the results obtained from ML experiments: training data, features, model architecture, hyperparameters, test data, etc. Among these aspects, arguably the most important design decisions are those that involve the evaluation procedure. This procedure is what determines whether the conclusions drawn from the experiments will or will not generalize to unseen data and whether they will be relevant to the application of interest. If the data is incorrectly selected, the wrong metric is chosen for evaluation or the significance of the comparisons between models is overestimated, conclusions may be misleading or result in suboptimal development decisions. To avoid such problems, the evaluation protocol should be very carefully designed before experimentation starts. In this work we discuss the main aspects involved in the design of the evaluation protocol: data selection, metric selection, and statistical significance. This document is not meant to be an exhaustive tutorial on each of these aspects. Instead, the goal is to explain the main guidelines that should be followed in each case. We include examples taken from the speech processing field, and provide a list of common mistakes related to each aspect.

Figures (4)

• Figure 1: Data splits into three sets for training, development and evaluation.
• Figure 2: Cross-validation approach for small datasets. For each fold, all other folds are used to train the model. The resulting model is applied to the samples in the hold-out fold to obtain the outputs on those samples. Results are computed after pooling the outputs obtained on all folds. In this case, no development decisions can be made before reporting results.
• Figure 3: Data splits for small datasets when development decisions need to be made. One part of the dataset is used both for training and development using the cross-validation approach and a separate set is used for final evaluation of the model trained on that set.
• Figure 4: Data splits for very small datasets when development decisions need to be made. The full set is split into K fold. Each fold is hold-out at a time and the rest of the folds are used for training and development with an inner cross-validation loop (pink block). Development decisions are made based on those results and a model is trained pooling all those inner folds. That model is then applied to the hold-out fold. This process is repeated for all outer folds (purple block) to generate outputs which are then pooled to get the final performance estimate.