Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Integrating kNN with Foundation Models for Adaptable and Privacy-Aware Image Classification

Sebastian Doerrich, Tobias Archut, Francesco Di Salvo, Christian Ledig

TL;DR

This work integrates the k-Nearest Neighbor (k-NN) classifier with a vision-based foundation model, pre-trained self-supervised on natural images, enhancing interpretability and adaptability and exhibits great promise for bridging the gap between foundation models’ performance and challenges tied to data privacy.

Abstract

Traditional deep learning models implicity encode knowledge limiting their transparency and ability to adapt to data changes. Yet, this adaptability is vital for addressing user data privacy concerns. We address this limitation by storing embeddings of the underlying training data independently of the model weights, enabling dynamic data modifications without retraining. Specifically, our approach integrates the $k$-Nearest Neighbor ($k$-NN) classifier with a vision-based foundation model, pre-trained self-supervised on natural images, enhancing interpretability and adaptability. We share open-source implementations of a previously unpublished baseline method as well as our performance-improving contributions. Quantitative experiments confirm improved classification across established benchmark datasets and the method's applicability to distinct medical image classification tasks. Additionally, we assess the method's robustness in continual learning and data removal scenarios. The approach exhibits great promise for bridging the gap between foundation models' performance and challenges tied to data privacy. The source code is available at https://github.com/TobArc/privacy-aware-image-classification-with-kNN.

Integrating kNN with Foundation Models for Adaptable and Privacy-Aware Image Classification

TL;DR

This work integrates the k-Nearest Neighbor (k-NN) classifier with a vision-based foundation model, pre-trained self-supervised on natural images, enhancing interpretability and adaptability and exhibits great promise for bridging the gap between foundation models’ performance and challenges tied to data privacy.

Abstract

Traditional deep learning models implicity encode knowledge limiting their transparency and ability to adapt to data changes. Yet, this adaptability is vital for addressing user data privacy concerns. We address this limitation by storing embeddings of the underlying training data independently of the model weights, enabling dynamic data modifications without retraining. Specifically, our approach integrates the -Nearest Neighbor (-NN) classifier with a vision-based foundation model, pre-trained self-supervised on natural images, enhancing interpretability and adaptability. We share open-source implementations of a previously unpublished baseline method as well as our performance-improving contributions. Quantitative experiments confirm improved classification across established benchmark datasets and the method's applicability to distinct medical image classification tasks. Additionally, we assess the method's robustness in continual learning and data removal scenarios. The approach exhibits great promise for bridging the gap between foundation models' performance and challenges tied to data privacy. The source code is available at https://github.com/TobArc/privacy-aware-image-classification-with-kNN.
Paper Structure (12 sections, 3 figures, 3 tables)

This paper contains 12 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Overview
  5. Backbone architecture
  6. Knowledge storage
  7. Experiments and Results
  8. Backbone choice for our method
  9. Adaptation for medical image analysis
  10. Continual learning and incremental forgetting
  11. Discussion and Conclusion
  12. Compliance with ethical standards

Figures (3)

  • Figure 1: During pretraining (1), the image encoder is trained to extract representative features. The knowledge-storing phase (2) utilizes the pre-trained (now frozen) encoder to extract and store task-relevant knowledge from the training data. During inference (3), that knowledge allows the classification of query images through majority voting on the top-$k$ similar embeddings.
  • Figure 2: Visualization of the method's ability for diverse continual learning tasks.
  • Figure 3: Illustration of our method's classification consistency despite the continuous diminishing of the support set.