Gaussian Process Assisted Meta-learning for Image Classification and Object Detection Models

TL;DR

The paper introduces GPAML, a framework that uses Gaussian process surrogates to learn how metadata balance in training data affects model accuracy, guiding data acquisition under budget constraints. By modeling an accuracy surface over metadata partitions and leveraging conic meta-learning with reference locations, GPAML proposes an acquisition rule that optimizes the proportion of data drawn from each metadata category. Applied to Spambase, MNIST, and RarePlanes, GPAML achieves competitive or superior out-of-sample performance and reduces the risk of detrimental data acquisitions, albeit with computational costs and limitations to two-category metadata. The approach offers a data-efficient strategy for planning labeled-data collection in image classification and object detection tasks where metadata about data collection conditions is available.

Abstract

Collecting operationally realistic data to inform machine learning models can be costly. Before collecting new data, it is helpful to understand where a model is deficient. For example, object detectors trained on images of rare objects may not be good at identification in poorly represented conditions. We offer a way of informing subsequent data acquisition to maximize model performance by leveraging the toolkit of computer experiments and metadata describing the circumstances under which the training data was collected (e.g., season, time of day, location). We do this by evaluating the learner as the training data is varied according to its metadata. A Gaussian process (GP) surrogate fit to that response surface can inform new data acquisitions. This meta-learning approach offers improvements to learner performance as compared to data with randomly selected metadata, which we illustrate on both classic learning examples, and on a motivating application involving the collection of aerial images in search of airplanes.

Figures (11)

• Figure 1: Process for identifying the optimal balance of metadata.
• Figure 2: Left: Sample data points $X^{(N)}$, colored by value of $Y^{(N)}$. Center: Cone resulting from all possible reference locations. Right: Center plot, zoomed in to boxed region. This shows all possible paths for the reference location $(20,20)$.
• Figure 3: Left: GP surface with cone overlaid. Center: Transects of cone plotted against model performance. Right: Integrated model performance.
• Figure 4: Multi-step GPAML, iterating over data sizes $N=N_{\mathrm{start}},\dots, N_{\mathrm{stop}}$.
• Figure 5: Left: OOS performance for each of the three methods. Proportion balances for GPAML (center left), random (center right), and random action (right) for each rep of experiment.