The Cost of Replicability in Active Learning
Rupkatha Hira, Dominik Kau, Jessica Sorrell
TL;DR
This work addresses the cost of achieving replicability in active learning, focusing on finite hypothesis classes and two standard settings: realizable and agnostic. It introduces Replicable CAL (RepliCAL) for the realizable setting and ReplicA^2 for the agnostic setting, both built on replicable statistical queries and random-thresholding techniques to ensure identical outputs across runs with high probability. The results show that, although replicability raises label complexity, the distributive gains from active learning persist, with bounds that scale with the disagreement coefficient $\Theta$ and, in the agnostic case, with the optimal error $\nu$ and target error $\varepsilon$; boosting can further mitigate dependence on the replicability parameter $\rho$. The study highlights that effective replication is compatible with substantial label savings and lays groundwork for lower bounds, broader hypothesis classes, and empirical validation in practice, bridging robustness and efficiency in machine learning systems.
Abstract
Active learning aims to reduce the required number of labeled data for machine learning algorithms by selectively querying the labels of initially unlabeled data points. Ensuring the replicability of results, where an algorithm consistently produces the same outcome across different runs, is essential for the reliability of machine learning models but often increases sample complexity. This report investigates the cost of replicability in active learning using the CAL algorithm, a classical disagreement-based active learning method. By integrating replicable statistical query subroutines and random thresholding techniques, we propose two versions of a replicable CAL algorithm. Our theoretical analysis demonstrates that while replicability does increase label complexity, the CAL algorithm can still achieve significant savings in label complexity even with the replicability constraint. These findings offer valuable insights into balancing efficiency and robustness in machine learning models.