CLEO: Continual Learning of Evolving Ontologies

TL;DR

CLEO tackles catastrophic forgetting in continual learning by introducing evolving ontologies, where classes can emerge, split, or be refined over time. It formalizes tasks, defines CLEO-specific evaluation, and proposes MoOn as a baseline distillation method that accounts for class evolution. Across Cityscapes, PASCAL VOC, and Mapillary Vistas, MoOn consistently outperforms traditional CIL methods by retaining prior knowledge while adapting to refined and new classes. The work demonstrates the practicality of evolving ontologies in real-world semantic segmentation and lays groundwork for robust lifelong learning in dynamic domains like autonomous driving.

Abstract

Continual learning (CL) addresses the problem of catastrophic forgetting in neural networks, which occurs when a trained model tends to overwrite previously learned information, when presented with a new task. CL aims to instill the lifelong learning characteristic of humans in intelligent systems, making them capable of learning continuously while retaining what was already learned. Current CL problems involve either learning new domains (domain-incremental) or new and previously unseen classes (class-incremental). However, general learning processes are not just limited to learning information, but also refinement of existing information. In this paper, we define CLEO - Continual Learning of Evolving Ontologies, as a new incremental learning setting under CL to tackle evolving classes. CLEO is motivated by the need for intelligent systems to adapt to real-world ontologies that change over time, such as those in autonomous driving. We use Cityscapes, PASCAL VOC, and Mapillary Vistas to define the task settings and demonstrate the applicability of CLEO. We highlight the shortcomings of existing CIL methods in adapting to CLEO and propose a baseline solution, called Modelling Ontologies (MoOn). CLEO is a promising new approach to CL that addresses the challenge of evolving ontologies in real-world applications. MoOn surpasses previous CL approaches in the context of CLEO.

Figures (8)

• Figure 1: CLEO extends class-incremental learning (CIL) by removing the constraint that incremental sets of classes must be non-overlapping. This allows for more flexible and nuanced learning, as existing classes can be refined into more specific classes.
• Figure 2: The Mapillary Vistas dataset neuhold2017mapillary provides a real-world example for an evolving ontology. This figure compares ground truth annotations for the same image in two dataset versions. It illustrates the introduction of new classes, such as traffic island and the finer categorization of existing classes such as billboard.
• Figure 3: Modelling Ontologies (MoOn) generalizes the idea of MiB MiB for all classes. During distillation, forgetting is mitigated, by grouping class-logits according to the evolution of the class ontology. In this example, the teacher network of the previous task predicts the vehicle class, while the model tries to learn bicycle. To avoid penalizing this, the predictions for vehicle and bicycle are combined before the distillation loss.
• Figure 4: Qualitative comparison on PASCAL VOC everingham2010pascalVOC-Ex3 after the final task.
• Figure 5: The official label hierarchy of Cityscapes cordts2016cityscapes.