On the Pitfalls of Batch Normalization for End-to-End Video Learning: A Study on Surgical Workflow Analysis

TL;DR

This work interrogates Batch Normalization’s limitations in end-to-end video learning for surgical workflow analysis, revealing how BN’s batch-dependent statistics hamper online, sequential learning and can enable harmful ‘cheating’ in anticipation tasks. It shows that BN-free backbones paired with end-to-end CNN-LSTM models, along with strategies to maximize temporal context (e.g., carry-hidden training, longer sequences, and partial freezing), can surpass state-of-the-art multi-stage approaches on surgical phase recognition and instrument anticipation. The authors validate their findings across multiple surgical datasets and also reproduce the trends on natural-video data, suggesting broader applicability beyond surgery. Overall, the study advocates BN-free end-to-end learning as a simpler yet more effective paradigm for long-video tasks and highlights BN’s pitfalls as a key obstacle to realizing such models in practice.

Abstract

Batch Normalization's (BN) unique property of depending on other samples in a batch is known to cause problems in several tasks, including sequence modeling. Yet, BN-related issues are hardly studied for long video understanding, despite the ubiquitous use of BN in CNNs (Convolutional Neural Networks) for feature extraction. Especially in surgical workflow analysis, where the lack of pretrained feature extractors has led to complex, multi-stage training pipelines, limited awareness of BN issues may have hidden the benefits of training CNNs and temporal models end to end. In this paper, we analyze pitfalls of BN in video learning, including issues specific to online tasks such as a 'cheating' effect in anticipation. We observe that BN's properties create major obstacles for end-to-end learning. However, using BN-free backbones, even simple CNN-LSTMs beat the state of the art {\color{\colorrevtwo}on three surgical workflow benchmarks} by utilizing adequate end-to-end training strategies which maximize temporal context. We conclude that awareness of BN's pitfalls is crucial for effective end-to-end learning in surgical tasks. By reproducing results on natural-video datasets, we hope our insights will benefit other areas of video learning as well. Code is available at: \url{https://gitlab.com/nct_tso_public/pitfalls_bn}

Figures (9)

• Figure 1: Two surgical workflow tasks on the Cholec80 dataset twinanda2016endonet: phase recognition, a special case of temporal action segmentation, and instrument anticipation, defined as predicting the time until occurrence of an instrument within a specified horizon.
• Figure 2: Phase Recognition (Cholec80): BN-free, end-to-end CNN-LSTMs outperform 2-stage LSTMs and TCNs. The widely-used ResNet50 with BN fails. Split: 40/8/32 from TeCNO czempiel2020tecno.
• Figure 3: Phase Recognition (Cholec80) with end-to-end CNN-LSTMs: (a) While longer sequences improve BN-free models, BN in ResNet50 backbones requires careful selection of the number of sequences per batch before performance drops. (b) Carrying hidden state through entire videos at test time improves models. By also doing this during training, train-test discrepancy is removed and models improve further. In BN models, implementing this is not straightforward. (c) BN models improve by freezing backbone blocks and thus increasing training sequences while maintaining batch diversity. Yet, freezing also improves BN-free models. Explicit scores with standard deviations and additional metrics can be found in the appendix (Tables \ref{['table:acc']} & \ref{['table:f1']}).
• Figure 4: Phase recognition (Cholec80): With multi-sequence batches (4x16), train and test features cover similar distributions. In BN-based backbones, single-sequence batches (1x64) cause a shift in train and test features due to the discrepancy between batch and global statistics.
• Figure 5: Online action segmentation with end-to-end CNN-LSTMs: Results on surgical data (left) can be reproduced on natural-video datasets (right).