Is Continual Learning Ready for Real-world Challenges?

TL;DR

Is Continual Learning Ready for Real-world Challenges? addresses the gap between CL research and practical deployment in dynamic 3D environments. The authors introduce the OCL-3DSS benchmark to evaluate online continual learning for 3D semantic segmentation under long, single-pass task sequences with overlapped future data. Across ScanNet, S3DIS, and SemanticKITTI, most CL methods catastrophically forget previous tasks and fail to reach the joint offline training upper bound, even with memory replay; language-guided supervision shows promise under limited supervision. The work advocates a paradigm shift toward more realistic evaluation protocols and highlights language-based guidance and long-sequence testing as promising directions for real-world applicability.

Abstract

Despite continual learning's long and well-established academic history, its application in real-world scenarios remains rather limited. This paper contends that this gap is attributable to a misalignment between the actual challenges of continual learning and the evaluation protocols in use, rendering proposed solutions ineffective for addressing the complexities of real-world setups. We validate our hypothesis and assess progress to date, using a new 3D semantic segmentation benchmark, OCL-3DSS. We investigate various continual learning schemes from the literature by utilizing more realistic protocols that necessitate online and continual learning for dynamic, real-world scenarios (eg., in robotics and 3D vision applications). The outcomes are sobering: all considered methods perform poorly, significantly deviating from the upper bound of joint offline training. This raises questions about the applicability of existing methods in realistic settings. Our paper aims to initiate a paradigm shift, advocating for the adoption of continual learning methods through new experimental protocols that better emulate real-world conditions to facilitate breakthroughs in the field.

Figures (11)

• Figure 1: T-SNE of CLIP features on ScanNet Dai17CVPR classes, evenly distributed in 2D space.
• Figure 2: Performance variance on the task ordering in ScanNet.
• Figure 3: Learning rate in FT has minimal influence.
• Figure 4: MAS weight does not significantly impact forgetting.
• Figure 5: Overall performance of CL methods on our OCL-3DSS setup (ScanNet-v2). All CL methods except ER converge to almost zero by the $20_{th}$ task. Despite the sizable memory buffer, even ER is still far from JT. ER is more of a practical solution.