MetaCLBench: Meta Continual Learning Benchmark on Resource-Constrained Edge Devices

TL;DR

The paper addresses catastrophic forgetting in Meta-Continual Learning (Meta-CL) for edge devices processing multimodal data (image and audio). It introduces MetaCLBench, an end-to-end benchmark that evaluates six Meta-CL methods across three architectures on five datasets, incorporating both accuracy and system-resource metrics (latency, energy, memory) on multiple edge platforms. Key findings show that pre-training followed by meta-training boosts deployment performance, while LifeLearner achieves strong accuracy with relatively small memory footprints; AIM improves accuracy but imposes significant memory and energy costs, and larger pretrained models (ViT, YAMNet) incur substantial resource demands. The authors provide practical guidelines for resource-aware method selection and release the benchmark framework to facilitate reproducible, fair evaluations for real-world edge scenarios.

Abstract

Meta-Continual Learning (Meta-CL) has emerged as a promising approach to minimize manual labeling efforts and system resource requirements by enabling Continual Learning (CL) with limited labeled samples. However, while existing methods have shown success in image-based tasks, their effectiveness remains unexplored for sequential time-series data from sensor systems, particularly audio inputs. To address this gap, we conduct a comprehensive benchmark study evaluating six representative Meta-CL approaches using three network architectures on five datasets from both image and audio modalities. We develop MetaCLBench, an end-to-end Meta-CL benchmark framework for edge devices to evaluate system overheads and investigate trade-offs among performance, computational costs, and memory requirements across various Meta-CL methods. Our results reveal that while many Meta-CL methods enable to learn new classes for both image and audio modalities, they impose significant computational and memory costs on edge devices. Also, we find that pre-training and meta-training procedures based on source data before deployment improve Meta-CL performance. Finally, to facilitate further research, we provide practical guidelines for researchers and machine learning practitioners implementing Meta-CL on resource-constrained environments and make our benchmark framework and tools publicly available, enabling fair evaluation across both accuracy and system-level metrics.

Figures (9)

• Figure 1: The framework overview. Testing trade-offs between performance and system resources across three devices with five datasets and six Meta-CL methods using three model architectures
• Figure 2: The illustration of the Meta-CL methods evaluated in our benchmark framework.
• Figure 3: The analysis of Meta-CL Methods for the audio datasets
• Figure 4: The accuracy of ESC-50, Urbansound8k, GSCv2 datasets using the 3-Layer CNN architecture.
• Figure 5: The accuracy of ESC-50, Urbansound8k, GSCv2 datasets using the YAMNet architecture.