F$^3$OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics

TL;DR

F3OCUS tackles the challenge of federated fine-tuning of Vision-Language Foundation Models under client resource constraints by combining LNTK-based client-layer importance with server-side, data-free multi-objective meta-heuristics to optimize layer updates across heterogeneous clients. The approach formalizes the FL objective and masks layer updates, then derives per-layer importance from the LNTK principal eigenvalue $\lambda_1^l$ to guide client selections, while proving convergence bounds (Theorem 1) for the LNTK-based scheme. It introduces a bi-objective server optimization balancing aggregate importance and inter-client layer diversity, solved via five meta-heuristics to approximate Pareto-optimal configurations without server data. Empirically, Ultra-MedVQA is released and F3OCUS demonstrates superior performance over state-of-the-art PEFT and pruning baselines across six Vision-Language tasks and four VLMs, with substantial gains in accuracy and reductions in communication and computation. Overall, F3OCUS provides a practical, theoretically grounded framework for privacy-preserving, resource-aware federated fine-tuning of large Vision-Language models in medical and other sensitive domains.

Abstract

Effective training of large Vision-Language Models (VLMs) on resource-constrained client devices in Federated Learning (FL) requires the usage of parameter-efficient fine-tuning (PEFT) strategies. To this end, we demonstrate the impact of two factors \textit{viz.}, client-specific layer importance score that selects the most important VLM layers for fine-tuning and inter-client layer diversity score that encourages diverse layer selection across clients for optimal VLM layer selection. We first theoretically motivate and leverage the principal eigenvalue magnitude of layerwise Neural Tangent Kernels and show its effectiveness as client-specific layer importance score. Next, we propose a novel layer updating strategy dubbed F$^3$OCUS that jointly optimizes the layer importance and diversity factors by employing a data-free, multi-objective, meta-heuristic optimization on the server. We explore 5 different meta-heuristic algorithms and compare their effectiveness for selecting model layers and adapter layers towards PEFT-FL. Furthermore, we release a new MedVQA-FL dataset involving overall 707,962 VQA triplets and 9 modality-specific clients and utilize it to train and evaluate our method. Overall, we conduct more than 10,000 client-level experiments on 6 Vision-Language FL task settings involving 58 medical image datasets and 4 different VLM architectures of varying sizes to demonstrate the effectiveness of the proposed method.

Figures (9)

• Figure 1: Distinction of our approach from prior works. (a) illustrates vanilla layer-selection process, which only selects parameter subsets based on the local client data without considering the requirements of the other clients. (b) depicts our approach, F3OCUS, which refines the client-specific layer selection by jointly maximizing overall client-specific importance score and layer selection diversity score across clients.
• Figure 2: Loss convergence of layer selection methods. The gap between the client-specific NTK and FOCUS demonstrates the importance of our multi-objective meta-heuristic optimization.
• Figure 3: Visualization of principal eigenvalue magnitudes of LNTK (see $\S 4.1$) for computing layer importance score of LLaVA-1.5-7b
• Figure 4: Overview of our layer selection strategy, F3OCUS. Each client sends layer importance scores based on the principal eigenvalue of LNTK to the server. The server refines client-specific layer selection by maximizing the cumulative client-specific importance scores while simultaneously minimizing the variance of the histogram of layer selections across clients. It sends the revised layer ranks back.
• Figure 5: Visualization of layer ranks of LLaVA-1.5 across rounds in different clients based on LNTK. Darker color implies higher rank.