R-MTLLMF: Resilient Multi-Task Large Language Model Fusion at the Wireless Edge

TL;DR

This work tackles adversarial noise in wireless-edge multi-task fine-tuning by analyzing how cross-task interference scales with wireless MSE through weight disentanglement error. It proposes R-MTLLMF, an AI-driven resilience framework that freezes sensitive LLM embeddings and applies few-shot realignment to stabilize task-vector aggregation, achieving near-baseline performance under ideal channels and substantial gains over unprotected MTMF under hostile conditions. The authors derive worst-case noise-covariance solutions for sum-rate and strongest-user scenarios and validate results on ViT-LLMs across eight datasets, highlighting the limits of AI resilience and the potential need for complementary physical-layer protections. Overall, the work demonstrates a promising hybrid AI-physics approach to robust, edge-enabled MTLLM fusion with clear directions for future integration of PHY safeguards.

Abstract

Multi-task large language models (MTLLMs) are important for many applications at the wireless edge, where users demand specialized models to handle multiple tasks efficiently. However, training MTLLMs is complex and exhaustive, particularly when tasks are subject to change. Recently, the concept of model fusion via task vectors has emerged as an efficient approach for combining fine-tuning parameters to produce an MTLLM. In this paper, the problem of enabling edge users to collaboratively craft such MTLMs via tasks vectors is studied, under the assumption of worst-case adversarial attacks. To this end, first the influence of adversarial noise to multi-task model fusion is investigated and a relationship between the so-called weight disentanglement error and the mean squared error (MSE) is derived. Using hypothesis testing, it is directly shown that the MSE increases interference between task vectors, thereby rendering model fusion ineffective. Then, a novel resilient MTLLM fusion (R-MTLLMF) is proposed, which leverages insights about the LLM architecture and fine-tuning process to safeguard task vector aggregation under adversarial noise by realigning the MTLLM. The proposed R-MTLLMF is then compared for both worst-case and ideal transmission scenarios to study the impact of the wireless channel. Extensive model fusion experiments with vision LLMs demonstrate R-MTLLMF's effectiveness, achieving close-to-baseline performance across eight different tasks in ideal noise scenarios and significantly outperforming unprotected model fusion in worst-case scenarios. The results further advocate for additional physical layer protection for a holistic approach to resilience, from both a wireless and LLM perspective.

Figures (5)

• Figure 1: MTMF system model where users collaboratively construct a MTLLM. R-MTLLMF safeguards task vector aggregation by realigning the perturbed model and freezing sensitive LLM parameters that do not change significantly.
• Figure 2: Results for MTMF with 8 tasks. R-MTLLMF achieves close-to-baseline performance for the ideal transmission case, while having degraded performance under worst-case noise for sum rate (SR) and strongest user (SU).
• Figure 3: Cosine similarities between task vectors for R-MTLLMF (ideal case) as a measure for weight disentanglement. All $\hat{\tau}_q$ have small similarity scores, indicating less cross-task interference and increased orthogonality.
• Figure 4: Cosine similarities between task vectors for the worst-case (SR). All $\hat{\tau}_q$ have increased similarity scores around 0.5, indicating less orthogonality, more weight entanglement, and increased cross-task interference.
• Figure 5: Ablation results for the ideal transmission case indicate that best performance is only achieved when all R-MTLLMF modules work together. Few-shot realignment with 10 samples is enough for satisfactory performance.