Strike a Balance in Continual Panoptic Segmentation

TL;DR

BalConpas introduces three balances to tackle continual panoptic segmentation: past-class backtrace distillation (PCBD) for selective feature distillation of past-class segments, class-proportional memory (CPM) to align replay samples with the cumulative class distribution, and balanced anti-misguidance losses (BAG) to address incomplete replay annotations without bias. The method freezes the previous model to generate pseudo-labels, backtracks to the corresponding features, and distills only the relevant features to preserve past knowledge while learning new classes. CPM uses a greedy algorithm to construct a replay set that mirrors long-term class distributions and includes a constraint to preserve earlier class representations. BAG combines two loss components to avoid background misguidance and foreground bias, enabling effective learning from replay and regular images. Evaluations on ADE20K show BalConpas achieves state-of-the-art performance across CPS, CSS, and CIS, illustrating its strong generalization and practical impact for continual segmentation tasks; code is released at the official repository.

Abstract

This study explores the emerging area of continual panoptic segmentation, highlighting three key balances. First, we introduce past-class backtrace distillation to balance the stability of existing knowledge with the adaptability to new information. This technique retraces the features associated with past classes based on the final label assignment results, performing knowledge distillation targeting these specific features from the previous model while allowing other features to flexibly adapt to new information. Additionally, we introduce a class-proportional memory strategy, which aligns the class distribution in the replay sample set with that of the historical training data. This strategy maintains a balanced class representation during replay, enhancing the utility of the limited-capacity replay sample set in recalling prior classes. Moreover, recognizing that replay samples are annotated only for the classes of their original step, we devise balanced anti-misguidance losses, which combat the impact of incomplete annotations without incurring classification bias. Building upon these innovations, we present a new method named Balanced Continual Panoptic Segmentation (BalConpas). Our evaluation on the challenging ADE20K dataset demonstrates its superior performance compared to existing state-of-the-art methods. The official code is available at https://github.com/jinpeng0528/BalConpas.

Figures (5)

• Figure 1: Illustration of our past-class backtrace distillation. It retraces features associated with output segments labeled as past classes for targeted knowledge distillation, while simultaneously allowing other features to adapt freely to new knowledge.
• Figure 2: Overview of the proposed BalConpas. Given input containing regular images and replay samples, the current and the previous models process it simultaneously. For regular images, outputs from the previous model serve as pseudo-labels, supplementing past-class annotations. After label assignment, we trace back to the features associated with output segments labeled as past classes and focus the knowledge distillation on them. Concurrently, the supervision for replay samples and regular images is managed by the first and second components of the balanced anti-misguidance losses, respectively.
• Figure 3: Qualitative comparison of BalConpas and existing methods on ADE20K for 100-10 CPS.
• Figure B1: Qualitative comparison of BalConpas and existing methods on ADE20K for 100-10 CSS.
• Figure B2: Qualitative comparison of BalConpas and existing methods on ADE20K for 50-10 CIS.