Pool-based Active Learning as Noisy Lossy Compression: Characterizing Label Complexity via Finite Blocklength Analysis

TL;DR

This work develops an information-theoretic framework for pool-based active learning by recasting data acquisition as a noisy lossy compression problem, where pool observations pass through a fixed channel and data selection acts as the encoder. Leveraging finite blocklength analysis, it derives lower bounds on label complexity and generalization error that depend on the pool size, the learning algorithm's overfitting, and the inductive-bias mismatch to the target task, linking these to rate-distortion and tilted-information concepts. A key novelty is specializing prior finite-blocklength results to the finite pool setting, yielding bounds that explicitly reflect pool-induced constraints and offering a new lens to compare pool-based AL with i.i.d. sampling as a special case. The framework unifies information-theoretic generalization bounds and stability Theory in the context of pool-based AL, providing theoretical limits that can guide data selection strategies and illuminate the role of algorithmic inductive biases in labeling efficiency and generalization.

Abstract

This paper proposes an information-theoretic framework for analyzing the theoretical limits of pool-based active learning (AL), in which a subset of instances is selectively labeled. The proposed framework reformulates pool-based AL as a noisy lossy compression problem by mapping pool observations to noisy symbol observations, data selection to compression, and learning to decoding. This correspondence enables a unified information-theoretic analysis of data selection and learning in pool-based AL. Applying finite blocklength analysis of noisy lossy compression, we derive information-theoretic lower bounds on label complexity and generalization error that serve as theoretical limits for a given learning algorithm under its associated optimal data selection strategy. Specifically, our bounds include terms that reflect overfitting induced by the learning algorithm and the discrepancy between its inductive bias and the target task, and are closely related to established information-theoretic bounds and stability theory, which have not been previously applied to the analysis of pool-based AL. These properties yield a new theoretical perspective on pool-based AL.

Figures (1)

• Figure 3.1: Overview of the correspondence in sugiyama2026finite (black) and our extension (blue).

Theorems & Definitions (16)

• Definition 4.1
• Definition 4.2
• Theorem 4.3
• Theorem 4.4
• Theorem 4.5
• Theorem 1.1
• proof : Proof of Theorem \ref{['thm:nlc_rate_dist_MI_transform']}
• Definition 2.1: Definition B.1. in sugiyama2026finite
• Theorem 2.2: Theorem 1 in xu2017information
• Theorem 2.3: A special case of Corollary 6 in hellstrom2020generalization