Active partitioning: inverting the paradigm of active learning

TL;DR

The paper addresses the challenge of learning datasets with multiple regimes by introducing active partitioning, a competition-driven partitioning method where multiple predictors vie for each data point and the winner trains on it. The resulting data-point allocations define partitions whose boundaries are captured by an SVM, enabling per-partition experts and a modular architecture. Across synthetic and real-world regression tasks, the approach reveals distinct patterns and, in several cases, substantially outperforms a monolithic model, with gains up to 54% in loss reduction. The work also outlines a path toward adaptive data collection and pattern-aware hyperparameter customization, highlighting practical benefits for structured datasets and expensive data scenarios.

Abstract

Datasets often incorporate various functional patterns related to different aspects or regimes, which are typically not equally present throughout the dataset. We propose a novel, general-purpose partitioning algorithm that utilizes competition between models to detect and separate these functional patterns. This competition is induced by multiple models iteratively submitting their predictions for the dataset, with the best prediction for each data point being rewarded with training on that data point. This reward mechanism amplifies each model's strengths and encourages specialization in different patterns. The specializations can then be translated into a partitioning scheme. The amplification of each model's strengths inverts the active learning paradigm: while active learning typically focuses the training of models on their weaknesses to minimize the number of required training data points, our concept reinforces the strengths of each model, thus specializing them. We validate our concept -- called active partitioning -- with various datasets with clearly distinct functional patterns, such as mechanical stress and strain data in a porous structure. The active partitioning algorithm produces valuable insights into the datasets' structure, which can serve various further applications. As a demonstration of one exemplary usage, we set up modular models consisting of multiple expert models, each learning a single partition, and compare their performance on more than twenty popular regression problems with single models learning all partitions simultaneously. Our results show significant improvements, with up to 54% loss reduction, confirming our partitioning algorithm's utility.

Figures (8)

• Figure 1: flow chart of the partitioning algorithm: each data pointed is assigned to the model that submitted the best prediction. All models are trained with the data points in their partition for one epoch. This process is iterated.
• Figure 2: exemplary partitioning. Figure \ref{['fig_mapping_samples']} presents the self-designed test dataset, while Figure \ref{['fig_mapping_result']} displays an exemplary partitioning result. Figure \ref{['fig_mapping_process']} illustrates the partitioning process, transitioning from networks with initial random predictions to the orange, red, and green networks each capturing distinct patterns. The process involves adding and removing networks as patterns are identified or networks deemed redundant.
• Figure 3: adding a new network (red network 12) to the competition. Regularly, a new network is trained using the data points with the poorest predictions at that time. If the new network improves the overall loss, it is added to the competition. Here, the red network 12 is the first to capture the sinusoidal pattern.
• Figure 4: dropping a network (red network 12) from the competition as it appears redundant, failing to capture any patterns uniquely. Regularly, for each model, we check how much the overall loss would increase if the network were removed. If the increase is small, the corresponding network is considered redundant and is discarded. Here, the red network's predictions were too similar to the purple network's predictions.
• Figure 5: flow chart of the modular model: each partition is learned by a separate expert model. For each data point, the SVM as a result of the partitioning algorithm decides which expert to train or to test. This way, the experts are combined to a modular model.