A Cross-Domain Benchmark for Active Learning

TL;DR

It is shown, that both the cross-domain character and a large amount of repetitions are crucial for sophisticated evaluation of AL research, and that the superiority of specific methods varies over the different domains, making it important to evaluate Active Learning with a cross-domain benchmark.

Abstract

Active Learning (AL) deals with identifying the most informative samples for labeling to reduce data annotation costs for supervised learning tasks. AL research suffers from the fact that lifts from literature generalize poorly and that only a small number of repetitions of experiments are conducted. To overcome these obstacles, we propose CDALBench, the first active learning benchmark which includes tasks in computer vision, natural language processing and tabular learning. Furthermore, by providing an efficient, greedy oracle, CDALBench can be evaluated with 50 runs for each experiment. We show, that both the cross-domain character and a large amount of repetitions are crucial for sophisticated evaluation of AL research. Concretely, we show that the superiority of specific methods varies over the different domains, making it important to evaluate Active Learning with a cross-domain benchmark. Additionally, we show that having a large amount of runs is crucial. With only conducting three runs as often done in the literature, the superiority of specific methods can strongly vary with the specific runs. This effect is so strong, that, depending on the seed, even a well-established method's performance can be significantly better and significantly worse than random for the same dataset.

Figures (19)

• Figure 1: Random draws from a pool of 100 runs for margin sampling on the Splice dataset with different numbers of repetitions ($\alpha=\{3,5,50\}$). Green curves are the mean performance of all 100 runs, while the samples are blue. Even with 3 or 5 repetitions, we can observe that single draws for margin sampling display below-random performance (black), while the true mean should be above random.
• Figure 2: Ranks of each AL method aggregated by domain. Horizontal bars indicate a non-significant rank difference. The significance is tested via a paired-t-test with $\alpha=0.05$.
• Figure 3: Synthetic "Honeypot" and "Diverging Sine" datasets. The optimal decision boundary is not part of the dataset and serves only as a visual guide.
• Figure 4: Performance curves per query size for normal (un-encoded) Splice
• Figure 5: Performance curves per query size for semi-supervised (encoded) Splice