Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models

TL;DR

This work tackles privacy constraints in vision–language model alignment by replacing parameter sharing with preference-based signals. It introduces MoR, a mixture-of-rewards framework where each client trains a local reward model and a lightweight routing network is learned via FL to fuse diverse rewards, allowing GRPO to optimize a base VLM with routed feedback. Empirical results on three VQA benchmarks show MoR outperforms federated baselines in generalization, robustness, and cross-client adaptability, while maintaining scalable privacy-preserving training. The approach enables effective alignment across heterogeneous clients and paves the way for adaptive, dynamic federated settings with minimal data exposure.

Abstract

VLMs have broad potential in privacy-sensitive domains such as healthcare and finance, yet strict data-sharing constraints render centralized training infeasible. FL mitigates this issue by enabling decentralized training, but practical deployments face challenges due to client heterogeneity in computational resources, application requirements, and model architectures. We argue that while replacing data with model parameters characterizes the present of FL, replacing parameters with preferences represents a more scalable and privacy-preserving future. Motivated by this perspective, we propose MoR, a federated alignment framework based on GRPO with Mixture-of-Rewards for heterogeneous VLMs. MoR initializes a visual foundation model as a KL-regularized reference, while each client locally trains a reward model from local preference annotations, capturing specific evaluation signals without exposing raw data. To reconcile heterogeneous rewards, we introduce a routing-based fusion mechanism that adaptively aggregates client reward signals. Finally, the server performs GRPO with this mixed reward to optimize the base VLM. Experiments on three public VQA benchmarks demonstrate that MoR consistently outperforms federated alignment baselines in generalization, robustness, and cross-client adaptability. Our approach provides a scalable solution for privacy-preserving alignment of heterogeneous VLMs under federated settings.

Figures (3)

• Figure 1: Visualization of multimodal data distributions across different task domains. The embeddings are extracted from the joint vision-language space and projected onto a 2D plane using UMAP mcinnes2018umap. Distinct clusters for Detail, Medical, and General OCR demonstrate significant domain shifts.
• Figure 2: Overview of our proposed federated framework architecture. The framework is composed of three interconnected stages: (A) Decentralized Reward Model Training: Individual clients independently train local reward models (RMs) on their private, domain-specific datasets. (B) Federated Router Training: A global routing mechanism is collaboratively trained across clients to learn how to dispatch inputs to the most capable local RM. (C) GRPO Policy Alignment: The final multimodal policy is optimized using Group Relative Policy Optimization (GRPO), guided by the high-quality reward signals selected by the trained router.
• Figure 3: Computation Efficiency Analysis. We compare one epoch training time (in hours) of our method against individual reward models (RMs) and the ensemble baseline (Avg RM) in the heterogeneous setting.