Instance-wise Supervision-level Optimization in Active Learning

TL;DR

This paper addresses label efficiency under budget by introducing Instance-wise Supervision-level Optimization (ISO), a framework that jointly selects data instances and their supervision level (full vs. weak) using per-instance value-to-cost ratios and diversity considerations. ISO computes $v_\mathrm{f}^t({\bm x})$ and $v_\mathrm{w}^t({\bm x})$ to quantify cost-effective value and estimates expected improvements $M_\mathrm{f}^t$ and $M_\mathrm{w}^t$ while normalizing uncertainties $u_\mathrm{f}^t({\bm x})$ and $u_\mathrm{w}^t({\bm x})$ to percentiles. Batch selection then maximizes the area spanned by chosen vectors $v_\mathrm{*}({\bm x})\tilde{f}({\bm x})$ under budget constraints, enabling a diverse, high-value annotation set. Empirical results on CIFAR100 and CUB200 show ISO outperforms conventional AL and a state-of-the-art mixed-supervision method, achieving higher accuracy at lower costs and demonstrating strong cost-efficiency when weak labels are cheap. Limitations include validation only on classification tasks and the assumption of two supervision levels; future work will extend ISO to segmentation and additional supervision granularity across broader tasks.

Abstract

Active learning (AL) is a label-efficient machine learning paradigm that focuses on selectively annotating high-value instances to maximize learning efficiency. Its effectiveness can be further enhanced by incorporating weak supervision, which uses rough yet cost-effective annotations instead of exact (i.e., full) but expensive annotations. We introduce a novel AL framework, Instance-wise Supervision-Level Optimization (ISO), which not only selects the instances to annotate but also determines their optimal annotation level within a fixed annotation budget. Its optimization criterion leverages the value-to-cost ratio (VCR) of each instance while ensuring diversity among the selected instances. In classification experiments, ISO consistently outperforms traditional AL methods and surpasses a state-of-the-art AL approach that combines full and weak supervision, achieving higher accuracy at a lower overall cost. This code is available at https://github.com/matsuo-shinnosuke/ISOAL.

Figures (10)

• Figure 1: Overview of Instance-wise Supervision-level Optimization (ISO) for active learning. The proposed approach combines weak and full supervision within an active learning framework to maximize annotation efficiency under a fixed budget. First, each instance ${\bm x}$ in the unlabeled data pool is evaluated for its value-to-cost ratio ($v_\mathrm{f}({\bm x})$ for full supervision and $v_\mathrm{w}({\bm x})$ for weak supervision). Based on these values and data diversity, instances are selected within the budget constraints for weak supervision (with superclass labels) or full supervision (with exact class labels). This adaptive allocation of supervision-level enables optimal use of resources within the budget constraints in active learning.
• Figure 2: CIFAR100
• Figure 3: CUB200
• Figure 5: CIFAR100
• Figure 6: CUB200