A Framework for Double-Blind Federated Adaptation of Foundation Models

TL;DR

BlindFed presents a double-blind federated approach for adapting foundation models without any data or model leakage between data owners and the learning service provider. It combines an FHE-friendly FM redesign, offline knowledge distillation, and online encrypted adaptation with MPC-based secure aggregation, augmented by privacy boosts such as sample-level permutation and stochastic block sampling. Empirical results on multiple image datasets show competitive accuracy relative to baselines while illustrating substantial privacy and practicality trade-offs, notably high communication and server computation costs. This framework offers a foundational step toward privacy-preserving FM adaptation in cross-silo settings, with clear avenues for reducing overhead and strengthening security in future work.

Abstract

Foundation models (FMs) excel in zero-shot tasks but benefit from task-specific adaptation. However, privacy concerns prevent data sharing among multiple data owners, and proprietary restrictions prevent the learning service provider (LSP) from sharing the FM. In this work, we propose BlindFed, a framework enabling collaborative FM adaptation while protecting both parties: data owners do not access the FM or each other's data, and the LSP does not see sensitive task data. BlindFed relies on fully homomorphic encryption (FHE) and consists of three key innovations: (i) FHE-friendly architectural modifications via polynomial approximations and low-rank adapters, (ii) a two-stage split learning approach combining offline knowledge distillation and online encrypted inference for adapter training without backpropagation through the FM, and (iii) a privacy-boosting scheme using sample permutations and stochastic block sampling to mitigate model extraction attacks. Empirical results on four image classification datasets demonstrate the practical feasibility of the BlindFed framework, albeit at a high communication cost and large computational complexity for the LSP.

Figures (14)

• Figure 1: Conceptual illustration of BlindFed framework for double-blind federated adaptation of a foundation model.
• Figure 2: Overview of the proposed BlindFed framework for double-blind federated adaptation. The framework consists of three main components: (1) FHE-friendly architecture redesign, where the original foundation model (FM) is modified by approximating non-linear operations; (2) offline distillation, where the approximated blocks are fine-tuned via knowledge distillation using an auxiliary dataset; and (3) online adaptation, where clients interact the FHE-enabled FM under homomorphic encryption, performing local updates on the parallel adapter and classification head.
• Figure 3: FHE-friendly architecture redesign. Each transformer block's non-linear operations -- GELU activations, Softmax attention, and the division step in LayerNorm -- are replaced with low-degree polynomial approximations (denoted "Quad" for GELU and "ASoftmax" for Softmax). A lightweight parallel adapter and classification head are then trained on the client side.
• Figure 4: Illustration of the parallel adapter design.
• Figure 5: Stochastic block sampling strategy.

Theorems & Definitions (3)

• Proposition 1
• proof : Proof of Proposition \ref{['lemma: 1']}
• proof : Proof of Proposition \ref{['lemma: 1']}