Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Structured Output Regularization: a framework for few-shot transfer learning

Nicolas Ewen, Jairo Diaz-Rodriguez, Kelly Ramsay

TL;DR

Structured Output Regularization (SOR) addresses the challenge of adapting pretrained models with very limited labeled data by freezing internal structures and regulating their outputs rather than unfreezing weights. By inserting trainable beta weights between frozen blocks and applying $L_1$ penalties on these weights together with group-lasso penalties on block inputs, SOR enables targeted pruning and task-tailored adaptation with a minimal parameter budget. The authors provide implementations for convolutional layers, DenseNet blocks, and EfficientNet blocks, and demonstrate competitive few-shot performance on three medical-imaging tasks against established baselines, while enabling data-driven pruning. This approach offers a practical, data-efficient transfer-learning paradigm with broad applicability to CNN-based architectures and potential extensions to transformers, contributing to more parameter-efficient and robust domain adaptation in medical imaging and beyond.

Abstract

Traditional transfer learning typically reuses large pre-trained networks by freezing some of their weights and adding task-specific layers. While this approach is computationally efficient, it limits the model's ability to adapt to domain-specific features and can still lead to overfitting with very limited data. To address these limitations, we propose Structured Output Regularization (SOR), a simple yet effective framework that freezes the internal network structures (e.g., convolutional filters) while using a combination of group lasso and $L_1$ penalties. This framework tailors the model to specific data with minimal additional parameters and is easily applicable to various network components, such as convolutional filters or various blocks in neural networks enabling broad applicability for transfer learning tasks. We evaluate SOR on three few shot medical imaging classification tasks and we achieve competitive results using DenseNet121, and EfficientNetB4 bases compared to established benchmarks.

Structured Output Regularization: a framework for few-shot transfer learning

TL;DR

Structured Output Regularization (SOR) addresses the challenge of adapting pretrained models with very limited labeled data by freezing internal structures and regulating their outputs rather than unfreezing weights. By inserting trainable beta weights between frozen blocks and applying penalties on these weights together with group-lasso penalties on block inputs, SOR enables targeted pruning and task-tailored adaptation with a minimal parameter budget. The authors provide implementations for convolutional layers, DenseNet blocks, and EfficientNet blocks, and demonstrate competitive few-shot performance on three medical-imaging tasks against established baselines, while enabling data-driven pruning. This approach offers a practical, data-efficient transfer-learning paradigm with broad applicability to CNN-based architectures and potential extensions to transformers, contributing to more parameter-efficient and robust domain adaptation in medical imaging and beyond.

Abstract

Traditional transfer learning typically reuses large pre-trained networks by freezing some of their weights and adding task-specific layers. While this approach is computationally efficient, it limits the model's ability to adapt to domain-specific features and can still lead to overfitting with very limited data. To address these limitations, we propose Structured Output Regularization (SOR), a simple yet effective framework that freezes the internal network structures (e.g., convolutional filters) while using a combination of group lasso and penalties. This framework tailors the model to specific data with minimal additional parameters and is easily applicable to various network components, such as convolutional filters or various blocks in neural networks enabling broad applicability for transfer learning tasks. We evaluate SOR on three few shot medical imaging classification tasks and we achieve competitive results using DenseNet121, and EfficientNetB4 bases compared to established benchmarks.

Paper Structure

This paper contains 27 sections, 6 equations, 4 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Summary of Contributions.
  3. Related work
  4. Background
  5. Convolutional neural networks
  6. Problem
  7. Preliminary notation
  8. Assumptions and Conditions
  9. Structured Output Regularization
  10. Advantages of SOR
  11. Implementation details
  12. Experiments
  13. Noise and Box experiments
  14. Few Shot Medical imaging
  15. Datasets
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Outline of SOR. We start with a mostly frozen model (top), then add new weights $\beta$ and penalties (middle), resulting in a regularized model (bottom).
  • Figure 2: An example of the notation showing a block, $f_i$, a subfunction of a block, $f_{i1}$, and it's output $a_{i1}$.
  • Figure 3: Simulated results on the Noise and Box dataset with training size of 100. The accuracy chart shows the average performance of the models, while the reduced chart shows the fraction of outputs that were removed for different values of $\lambda$.
  • Figure 4: Before: an image of a baby's head. After: the single channel heatmap.

Theorems & Definitions (8)

  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Remark 5
  • Example 1: DenseNet
  • Example 2: EfficientNet
  • Remark 6