BOWL: A Deceptively Simple Open World Learner

TL;DR

BOWL introduces a batch-norm-based framework that unifies out-of-distribution detection, informative data querying, and continual memory-based learning for open-world scenarios. By treating BN running statistics as Gaussian approximations of intermediate activations, it derives OoD scores, an information-density acquisition function, and a dynamic memory-update strategy to train incrementally. The approach yields strong open-world performance while substantially reducing data and computation compared with replay-based baselines, and ablation confirms each BN-based component contributes meaningfully. This simple, cohesive baseline enables effective deployment and ongoing learning of BN-equipped networks in non-stationary environments.

Abstract

Traditional machine learning excels on static benchmarks, but the real world is dynamic and seldom as carefully curated as test sets. Practical applications may generally encounter undesired inputs, are required to deal with novel information, and need to ensure operation through their full lifetime - aspects where standard deep models struggle. These three elements may have been researched individually, but their practical conjunction, i.e., open world learning, is much less consolidated. In this paper, we posit that neural networks already contain a powerful catalyst to turn them into open world learners: the batch normalization layer. Leveraging its tracked statistics, we derive effective strategies to detect in- and out-of-distribution samples, select informative data points, and update the model continuously. This, in turn, allows us to demonstrate that existing batch-normalized models can be made more robust, less prone to forgetting over time, and be trained efficiently with less data.

Figures (6)

• Figure 1: Visualization of individual BOWL components, all rooted in a common frame of batch-normalization. For adapting a deep neural network to open-world learning, we use the running mean ($\mu_{BN}$) and the running average ($\sigma_{BN}$) from the batch normalization layer. The Out of Distribution module assigns a score $\eta_1$ to the data points, used for rejecting outliers. The Active Learning module chooses top $\gamma_q$ scoring data points from the available pool $Q$ to include into further training. The Continual Learning module stores task-suitable points based on $\gamma_m$ in the memory buffer $\mathcal{M}$.
• Figure 2: Batch-norm based OOD detection is able to identify corrupted examples. A Resnet-18 model trained on Split CIFAR-10. Blue corresponds to the score on IN-distribution (CIFAR-10) and yellow corresponds to the score on OUT-of-distribution (CIFAR-10 corrupted with various forms of perturbations). The difference between the OoD and IN-distribution histograms is pronounced in the case of BOWL.
• Figure 2: Ablation analysis of BOWL on Split CIFAR-10. We report the test accuracy at the end of every timestep. "$\times$" indicates that the module is omitted and "$\checkmark$" indicates otherwise. iv represents the complete BOWL framework with all its modules present. The final performance highlights that the inclusion of each individual component is meaningful to the overall performance. This suggests that effective learning in open-world scenarios requires a wholistic approach that addresses identification of relevant samples, gauging data informativeness, and adaptive memory management. Notably, that even if the absence of OoD module initially appears to have on-par accuracy, the complete framework is required to achieve robust performance across all evaluated scenarios --- as is highlighted in Figure \ref{['fig:bowl_open_world_learning']}.
• Figure 3: BOWL adaptively selects informative examples to populate a diverse memory buffer. Left: Amount of queried samples (y-axis) across tasks ($T_1$ to $T_4$ labels) over time (batch updates, x-axis) on Split CIFAR-10 with ResNet-18, where the green bars represent the number of samples queried for the new task, and the purple curve shows the test accuracy. Right: Memory buffer composition across the tasks. GDUMB's artificial balance yields $52\%$ accuracy whereas BOWL's knowledge-driven adaptive strategy achieves $63\%$ accuracy.
• Figure 4: BOWL adapts rapidly by selecting essential samples and requires substantially less optimization steps. BOWL yields competitive accuracy ($\approx60\%$ y-axis) while using only 25% of training data and 12% of the optimization steps (x-axis) compared to baseline methods. Despite sharing the same memory buffer size (5000 samples) with other replay based contenders, BOWL demonstrates faster adaptation and better performance across all tasks ($T_1$ to $T_4$).