Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parameter-Efficient Active Learning for Foundational models

Athmanarayanan Lakshmi Narayanan, Ranganath Krishnan, Amrutha Machireddy, Mahesh Subedar

TL;DR

This work tackles data-annotation efficiency for foundation-model-based few-shot transfer in out-of-distribution image tasks. It introduces Parameter-Efficient Active Learning (PEAL), which uses Low-Rank Adaptation (LoRa) adapters integrated into a DINOv2 backbone to enable parameter-efficient fine-tuning within active learning. By evaluating entropy-based (uncertainty) and Featdist-based (diversity) sampling, PEAL demonstrates improved transfer-learning performance and labeling efficiency over linear probing across Histology, APTOS, and EuroSAT, while using only a tiny fraction of trainable parameters ($0.03\%$). The findings highlight practical benefits for domain-specific data annotation and point to avenues for extending PEAL to other vision tasks and modalities.

Abstract

Foundational vision transformer models have shown impressive few shot performance on many vision tasks. This research presents a novel investigation into the application of parameter efficient fine-tuning methods within an active learning (AL) framework, to advance the sampling selection process in extremely budget constrained classification tasks. The focus on image datasets, known for their out-of-distribution characteristics, adds a layer of complexity and relevance to our study. Through a detailed evaluation, we illustrate the improved AL performance on these challenging datasets, highlighting the strategic advantage of merging parameter efficient fine tuning methods with foundation models. This contributes to the broader discourse on optimizing AL strategies, presenting a promising avenue for future exploration in leveraging foundation models for efficient and effective data annotation in specialized domains.

Parameter-Efficient Active Learning for Foundational models

TL;DR

This work tackles data-annotation efficiency for foundation-model-based few-shot transfer in out-of-distribution image tasks. It introduces Parameter-Efficient Active Learning (PEAL), which uses Low-Rank Adaptation (LoRa) adapters integrated into a DINOv2 backbone to enable parameter-efficient fine-tuning within active learning. By evaluating entropy-based (uncertainty) and Featdist-based (diversity) sampling, PEAL demonstrates improved transfer-learning performance and labeling efficiency over linear probing across Histology, APTOS, and EuroSAT, while using only a tiny fraction of trainable parameters (). The findings highlight practical benefits for domain-specific data annotation and point to avenues for extending PEAL to other vision tasks and modalities.

Abstract

Foundational vision transformer models have shown impressive few shot performance on many vision tasks. This research presents a novel investigation into the application of parameter efficient fine-tuning methods within an active learning (AL) framework, to advance the sampling selection process in extremely budget constrained classification tasks. The focus on image datasets, known for their out-of-distribution characteristics, adds a layer of complexity and relevance to our study. Through a detailed evaluation, we illustrate the improved AL performance on these challenging datasets, highlighting the strategic advantage of merging parameter efficient fine tuning methods with foundation models. This contributes to the broader discourse on optimizing AL strategies, presenting a promising avenue for future exploration in leveraging foundation models for efficient and effective data annotation in specialized domains.
Paper Structure (13 sections, 1 equation, 2 figures, 2 tables, 1 algorithm)

This paper contains 13 sections, 1 equation, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Linear Probing
  5. Parameter efficient fine tuning
  6. Selection Strategies
  7. Uncertainty-based sampling
  8. Diversity-based sampling
  9. Experiments and Discussion
  10. Model Architecture and Training parameters
  11. Datasets
  12. Results
  13. Conclusion

Figures (2)

  • Figure 1: Comparing Linear probing and PEAL on unknown sample selection ratio (higher is better) at initial active learning cycles. The selection strategy needs to select more incorrect samples (model unknowns) than correct samples (model knowns) for effective data annotation and model learning.
  • Figure 2: Active learning with DINOv2 foundation model on various datasets. The plots compare accuracy as a function of amount of labeled data obtained from different sample selection strategies with linear probing versus PEAL.