Advancing Brain Imaging Analysis Step-by-step via Progressive Self-paced Learning

TL;DR

Brain-imaging deep learning faces heterogeneity and limited samples, leading to overfitting and poor generalization. The authors propose Progressive Self-Paced Distillation (PSPD), a decoupled curriculum framework that combines a past-model teacher with a current-model student and adaptive pacing via self-paced regularizers, integrating paced curriculum learning and distillation. The method optimizes $L = \frac{1}{N} \sum_i L_{CE}(y_i, p^S(x_i)) + \gamma L_{KL}(p^T(x_i), p^S(x_i))$ with pace parameters $\lambda_w$ and $\lambda_\varphi$ evolving as $\lambda_w = \lambda_{w,0} + \alpha_w t$ and $\lambda_\varphi = \lambda_{\varphi,0} + \alpha_{\varphi} t$, enabling dynamic curriculum refinement. Experiments on the ADNI dataset across three CNN backbones show PSPD improves accuracy, AUC, and calibration (lower ECE/NLL), indicating stronger generalization for brain-imaging tasks and practical potential, with public code to follow.

Abstract

Recent advancements in deep learning have shifted the development of brain imaging analysis. However, several challenges remain, such as heterogeneity, individual variations, and the contradiction between the high dimensionality and small size of brain imaging datasets. These issues complicate the learning process, preventing models from capturing intrinsic, meaningful patterns and potentially leading to suboptimal performance due to biases and overfitting. Curriculum learning (CL) presents a promising solution by organizing training examples from simple to complex, mimicking the human learning process, and potentially fostering the development of more robust and accurate models. Despite its potential, the inherent limitations posed by small initial training datasets present significant challenges, including overfitting and poor generalization. In this paper, we introduce the Progressive Self-Paced Distillation (PSPD) framework, employing an adaptive and progressive pacing and distillation mechanism. This allows for dynamic curriculum adjustments based on the states of both past and present models. The past model serves as a teacher, guiding the current model with gradually refined curriculum knowledge and helping prevent the loss of previously acquired knowledge. We validate PSPD's efficacy and adaptability across various convolutional neural networks using the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, underscoring its superiority in enhancing model performance and generalization capabilities. The source code for this approach will be released at https://github.com/Hrychen7/PSPD.

Figures (3)

• Figure 1: The architecture of the proposed PSPD, where the teacher model is derived from the models trained at previous epochs. We first establish a decoupled curriculum setting that considers both the current and past states of the model, and then train the model based on this setting for learning and distillation.
• Figure 2: The learning curve of the validation accuracy.
• Figure 3: The results of different curriculum approaches and ablation studies.