AutoRank: MCDA Based Rank Personalization for LoRA-Enabled Distributed Learning

TL;DR

AutoRank tackles the challenge of per-participant rank tuning in LoRA-enabled distributed learning under double-imbalanced, non-IID data. It defines data-complexity metrics, and employs TOPSIS with CRITIC-based weights to compute a per-client rank score $C_i$, subsequently deriving $r_i$ via min–max normalization to assign adaptive local ranks. The approach is implemented in Python/TensorFlow and evaluated on MNIST, FMNIST, CIFAR-10, and CINIC-10, showing faster convergence and higher global accuracy than state-of-the-art baselines across highly heterogeneous settings. By enabling flexible, MCDA-driven rank personalization, AutoRank offers a scalable solution for efficient, personalized distributed learning in large-scale systems.

Abstract

As data volumes expand rapidly, distributed machine learning has become essential for addressing the growing computational demands of modern AI systems. However, training models in distributed environments is challenging with participants hold skew, Non-Independent-Identically distributed (Non-IID) data. Low-Rank Adaptation (LoRA) offers a promising solution to this problem by personalizing low-rank updates rather than optimizing the entire model, LoRA-enabled distributed learning minimizes computational and maximize personalization for each participant. Enabling more robust and efficient training in distributed learning settings, especially in large-scale, heterogeneous systems. Despite the strengths of current state-of-the-art methods, they often require manual configuration of the initial rank, which is increasingly impractical as the number of participants grows. This manual tuning is not only time-consuming but also prone to suboptimal configurations. To address this limitation, we propose AutoRank, an adaptive rank-setting algorithm inspired by the bias-variance trade-off. AutoRank leverages the MCDA method TOPSIS to dynamically assign local ranks based on the complexity of each participant's data. By evaluating data distribution and complexity through our proposed data complexity metrics, AutoRank provides fine-grained adjustments to the rank of each participant's local LoRA model. This adaptive approach effectively mitigates the challenges of double-imbalanced, non-IID data. Experimental results demonstrate that AutoRank significantly reduces computational overhead, enhances model performance, and accelerates convergence in highly heterogeneous federated learning environments. Through its strong adaptability, AutoRank offers a scalable and flexible solution for distributed machine learning.

Figures (11)

• Figure 1: Comparison of standard distributed learning with LoRA-enabled distributed learning. In the LoRA-enabled setup, each participant trains two low-rank models instead of full parameter matrices, reducing communication and computation overhead.
• Figure 2: Fig. \ref{['fig:noniid']} compares model convergence under different data distributions. Fig. \ref{['fig:bias_variance_tradeoff']} illustrates the bias-variance trade-off, showing how increasing model complexity reduces bias but increases variance and good or bad of different model complexity choices.
• Figure 3: Relationship between complexity levels and learning difficulty metric. The left panel shows participant training loss across 5 epochs for different complexity levels (C1 to C5, from simplest to most complex). The right panel displays the information entropy corresponding to each complexity level, highlighting the increasing complexity from C1 to C5.
• Figure 4: Process of AutoRank.
• Figure 5: Data distribution in each participant in our experiment, we can see the client 9 owns all labels and much more data samples compared with other clients.