Lorica: A Synergistic Fine-Tuning Framework for Advancing Personalized Adversarial Robustness

TL;DR

Lorica addresses the challenge of delivering personalized adversarial robustness for edge-deployed pre-trained models under federated, non-IID data while minimizing communication and computation. It introduces a two-phase framework: Phase 1 uses LoRA-FA for local adversarial fine-tuning and ball-tree aggregation to produce a generalized backbone plus personalized classifiers, aided by a MoE-inspired adaptive loss. Phase 2 employs a forward-gating network to selectively fine-tune layers, balancing benign accuracy with adversarial robustness. Empirical results show up to 68× communication efficiency and substantial gains in both adversarial robustness (up to 29.9%) and benign accuracy (up to 52.2%), demonstrating Lorica’s practical viability for edge deployments and large-scale federated settings.

Abstract

The growing use of large pre-trained models in edge computing has made model inference on mobile clients both feasible and popular. Yet these devices remain vulnerable to adversarial attacks, threatening model robustness and security. Federated adversarial training (FAT) offers a promising solution by enhancing robustness while preserving client privacy. However, FAT often yields a generalized global model that struggles with heterogeneous client data, leading to limited personalization and significant communication overhead. In this paper, we propose \textit{Lorica}, a personalized synergistic adversarial training framework that delivers customized defense models through a two-phase process. In Phase 1, \textit{Lorica} applies LoRA-FA for local adversarial fine-tuning, enabling personalized robustness while reducing communication by uploading only LoRA-FA parameters. In Phase 2, a forward-gating selection strategy improves benign accuracy, further refining the personalized model. This yields tailored defense models that effectively balance robustness and accuracy. Extensive experiments on benchmark datasets demonstrate that \textit{Lorica} can achieve up to 68$\times$ improvements in communication efficiency compared to state-of-the-art algorithms, while achieving up to 29.9\% and 52.2\% enhancements in adversarial robustness and benign accuracy, respectively.

Figures (20)

• Figure 1: Challenges of adversarial training in multi-client scenario. (1) Left: Local adversarial training performs well on abundant classes but fails on rare or unseen ones. (2) Right: Federated adversarial training improves generalization across clients but lacks personalization for unique client data distributions.
• Figure 2: Performance of personalized LoRA-FA in adversarial training under heterogeneous environments
• Figure 3: PCA embedding of the LoRA-FA after federated adversarial training
• Figure 4: Comparison of aggregation methods for adversarial training in heterogeneous environments
• Figure 5: Comparison of aggregation methods for adversarial training in heterogeneous environments