Evolving Without Ending: Unifying Multimodal Incremental Learning for Continual Panoptic Perception

TL;DR

The paper introduces Continual Panoptic Perception (CPP), a multimodal, multitask continual learning framework that jointly handles pixel-level classification, instance-level segmentation, and image-level captioning. It combines a Collaborative Cross-modal Encoder (CCE) with a malleable continual knowledge distillation (MCKD), a cross-modal bidirectional consistency constraint (CBC), and a self-supervised asymmetric pseudo-labeling (SAPL) mechanism, with a stronger CPP+ variant that adds tighter cross-modal alignment. The approach demonstrates superior performance on FineGrip, ADE20K, and COCO under overlapped continual learning settings, highlighting gains in segmentation quality, captioning BLEU scores, and cross-modal consistency. These results show the practical potential for robust, exemplar-free continual learning in complex, multimodal perception tasks, with implications for remote sensing, autonomous systems, and multimedia understanding.

Abstract

Continual learning (CL) is a great endeavour in developing intelligent perception AI systems. However, the pioneer research has predominantly focus on single-task CL, which restricts the potential in multi-task and multimodal scenarios. Beyond the well-known issue of catastrophic forgetting, the multi-task CL also brings semantic obfuscation across multimodal alignment, leading to severe model degradation during incremental training steps. In this paper, we extend CL to continual panoptic perception (CPP), integrating multimodal and multi-task CL to enhance comprehensive image perception through pixel-level, instance-level, and image-level joint interpretation. We formalize the CL task in multimodal scenarios and propose an end-to-end continual panoptic perception model. Concretely, CPP model features a collaborative cross-modal encoder (CCE) for multimodal embedding. We also propose a malleable knowledge inheritance module via contrastive feature distillation and instance distillation, addressing catastrophic forgetting from task-interactive boosting manner. Furthermore, we propose a cross-modal consistency constraint and develop CPP+, ensuring multimodal semantic alignment for model updating under multi-task incremental scenarios. Additionally, our proposed model incorporates an asymmetric pseudo-labeling manner, enabling model evolving without exemplar replay. Extensive experiments on multimodal datasets and diverse CL tasks demonstrate the superiority of the proposed model, particularly in fine-grained CL tasks.

Figures (11)

• Figure 1: Schematic illustration of the proposed method. (a) Single-task CL methods only support separate training on different tasks. (b) CPP enables a shared encoder across multimodal tasks, CPP+ integrates multimodal embedding within an end-to-end model. (c) CPP achieves class-incremental pixel classification, instance segmentation and image captioning.
• Figure 2: The framework of the proposed method. The input consists of the incremental images with corresponding multimodal annotation. The output consists of the mask predictions for both old and new classes and image captioning result with new semantics
• Figure 3: Cross-modal bi-directional consistency constraint.
• Figure 4: Self-supervised pseudo-labeling. The asymmetric task reliance indicates the pseudo labels are cross-verified by more reliable predictions from multimodal branches.
• Figure 5: Qualitative visualization before and after CL steps. Specifically, the predictions at the initial step and the final step are displayed.