Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CacheFL: Privacy-Preserving and Efficient Federated Cache Model Fine-Tuning for Vision-Language Models

Mengjun Yi, Hanwen Zhang, Hui Dou, Jian Zhao, Furao Shen

TL;DR

CacheFL tackles the privacy and efficiency barriers of fine-tuning large vision-language models in federated settings by replacing full-model updates with a lightweight cache model. It builds a class-balanced synthetic cache from DALL·E-generated data and uses FedAvg-style aggregation to fine-tune only the cache, effectively leveraging pre-trained CLIP knowledge while reducing communication and computation. A theoretical convergence analysis under standard FL assumptions is provided, along with an extensive empirical evaluation on 11 datasets showing superior accuracy and faster convergence, even under extreme non-IID distributions. The approach preserves privacy by avoiding raw local data exposure and demonstrates practical impact for privacy-preserving, scalable fine-tuning of vision-language systems.

Abstract

Large pre-trained Vision-Language Models (VLMs), such as Contrastive Language-Image Pre-training (CLIP), have exhibited remarkable zero-shot performance across various image classification tasks. Fine-tuning these models on domain-specific datasets further enhances their effectiveness for downstream applications. However, fine-tuning in cloud environments raises significant concerns regarding data security and privacy. Federated Learning (FL) offers a decentralized solution by enabling model training across local clients without centralizing sensitive data, but the high communication and computation costs of transmitting full pre-trained models during training limit its scalability. Additionally, non-Independent and Identically Distributed (non-IID) data across local clients can negatively impact model convergence and performance. To address these challenges, we propose CacheFL, a novel federated learning method that replaces traditional full model fine-tuning with lightweight cache model fine-tuning. The cache model is initialized using a class-balanced dataset generated by a generative pre-trained model, effectively mitigating the impact of non-IID data. This cache model is then distributed to local clients for fine-tuning, and the updated parameters from each client are aggregated on the server and redistributed. With the updated cache model, the classification performance of CLIP is improved after just a few epochs. By limiting the training and communication to the cache model, CacheFL significantly reduces resource demands while ensuring data privacy and security. Extensive experiments conducted on ImageNet and 10 additional datasets demonstrate that CacheFL outperforms traditional approaches in terms of classification accuracy, resource efficiency, and privacy preservation.

CacheFL: Privacy-Preserving and Efficient Federated Cache Model Fine-Tuning for Vision-Language Models

TL;DR

CacheFL tackles the privacy and efficiency barriers of fine-tuning large vision-language models in federated settings by replacing full-model updates with a lightweight cache model. It builds a class-balanced synthetic cache from DALL·E-generated data and uses FedAvg-style aggregation to fine-tune only the cache, effectively leveraging pre-trained CLIP knowledge while reducing communication and computation. A theoretical convergence analysis under standard FL assumptions is provided, along with an extensive empirical evaluation on 11 datasets showing superior accuracy and faster convergence, even under extreme non-IID distributions. The approach preserves privacy by avoiding raw local data exposure and demonstrates practical impact for privacy-preserving, scalable fine-tuning of vision-language systems.

Abstract

Large pre-trained Vision-Language Models (VLMs), such as Contrastive Language-Image Pre-training (CLIP), have exhibited remarkable zero-shot performance across various image classification tasks. Fine-tuning these models on domain-specific datasets further enhances their effectiveness for downstream applications. However, fine-tuning in cloud environments raises significant concerns regarding data security and privacy. Federated Learning (FL) offers a decentralized solution by enabling model training across local clients without centralizing sensitive data, but the high communication and computation costs of transmitting full pre-trained models during training limit its scalability. Additionally, non-Independent and Identically Distributed (non-IID) data across local clients can negatively impact model convergence and performance. To address these challenges, we propose CacheFL, a novel federated learning method that replaces traditional full model fine-tuning with lightweight cache model fine-tuning. The cache model is initialized using a class-balanced dataset generated by a generative pre-trained model, effectively mitigating the impact of non-IID data. This cache model is then distributed to local clients for fine-tuning, and the updated parameters from each client are aggregated on the server and redistributed. With the updated cache model, the classification performance of CLIP is improved after just a few epochs. By limiting the training and communication to the cache model, CacheFL significantly reduces resource demands while ensuring data privacy and security. Extensive experiments conducted on ImageNet and 10 additional datasets demonstrate that CacheFL outperforms traditional approaches in terms of classification accuracy, resource efficiency, and privacy preservation.
Paper Structure (17 sections, 6 theorems, 47 equations, 8 figures, 3 tables, 1 algorithm)

This paper contains 17 sections, 6 theorems, 47 equations, 8 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Models
  4. Federated Learning
  5. Methodology
  6. Class-Balanced Synthetic Dataset Generation
  7. Cache Model Construction
  8. Federated Training
  9. Theoretical Analysis
  10. Definition of CacheFL
  11. Assumption and Lemmas
  12. The Convergence of CacheFL
  13. Experiments
  14. Experimental Settings
  15. Overall Performance
  16. ...and 2 more sections

Key Result

Lemma 1

When Assumption assumption:1 and assumption:2 holds. If $\eta_t\leq \frac{1}{4L}$, we have where $\Gamma = F^* - \sum_{k=1}^{N} p_k F_k^* \geq 0$.

Figures (8)

  • Figure 1: Performance comparison on the Extreme non-IID setting. Each axis denotes an accuracy value for the corresponding dataset (refer to Table \ref{['tab:results']}). CacheFL surpasses state-of-the-art methods on 11 diverse image classification datasets.
  • Figure 2: The Structure of CLIP. Each backbone contains an image encoder and a text encoder. Each encoder extracts feature representations from its respective modality. The model is trained to maximize the cosine similarity of the correct image-text pairs while minimizing the incorrect pairs.
  • Figure 3: Class-balanced Synthetic Dataset Generation. We adopt DALL·E to generate $K$ synthetic images for $N$ categories separately.
  • Figure 4: The workflow of CacheFL Federated Training. Each client includes a cache model (with the visual feature weights being trainable and the one-hot vector weights remaining frozen) and an out-of-the-box CLIP (with the backbone remaining frozen). The server aggregates the updates to the visual feature weights from multiple clients, and then transmits the updated parameters back to each client for further local processing.
  • Figure 5: Convergence Speed. The y-axis shows test accuracy (%), and the x-axis shows communication rounds. Across 4 datasets (ImageNet, Stanford Cars, SUN397, and UCF101) and data settings (iid, dir, pat), CacheFL consistently outperforms all other methods in terms of final accuracy and convergence speed.
  • ...and 3 more figures

Theorems & Definitions (12)

  • Lemma 1: Results of one step SGD
  • proof
  • Lemma 2: Bounding the variance
  • proof
  • Lemma 3: Bounding the divergence of $\{c_t^k\}$
  • proof
  • Lemma 4: Unbiased sampling scheme
  • proof
  • Lemma 5: Bounding the variance of $\bar{c}_t$
  • proof
  • ...and 2 more