TuckA: Hierarchical Compact Tensor Experts for Efficient Fine-Tuning

TL;DR

The paper tackles the efficiency bottleneck of fine-tuning large foundation models by proposing TuckA, a Tucker-decomposition–driven tensor adapter that ensembles many small experts in a compact 3D tensor. It introduces a hierarchicalMoE design and a batch-level routing mechanism to efficiently select and combine experts, along with data-aware initialization to prevent load imbalance. The approach achieves state-of-the-art parameter-efficiency across NLP (GLUE with DeBERTa-v3-base), vision (ViT in few-shot settings), and mathematical reasoning (Llama2-7B on MetaMathQA) while maintaining or improving accuracy with far fewer trainable parameters than existing PEFT methods. This makes fine-tuning scalable to large models under limited compute, enabling broader practical deployment in real-world tasks. Theoretical and empirical analyses support the routing and initialization choices, and ablations highlight the benefits of hierarchical expert groups and data-aware initialization.

Abstract

Efficiently fine-tuning pre-trained models for downstream tasks is a key challenge in the era of foundation models. Parameter-efficient fine-tuning (PEFT) presents a promising solution, achieving performance comparable to full fine-tuning by updating only a small number of adaptation weights per layer. Traditional PEFT methods typically rely on a single expert, where the adaptation weight is a low-rank matrix. However, for complex tasks, the data's inherent diversity poses a significant challenge for such models, as a single adaptation weight cannot adequately capture the features of all samples. To address this limitation, we explore how to integrate multiple small adaptation experts into a compact structure to defeat a large adapter. Specifically, we propose Tucker Adaptation (TuckA), a method with four key properties: (i) We use Tucker decomposition to create a compact 3D tensor where each slice naturally serves as an expert. The low-rank nature of this decomposition ensures that the number of parameters scales efficiently as more experts are added. (ii) We introduce a hierarchical strategy that organizes these experts into groups at different granularities, allowing the model to capture both local and global data patterns. (iii) We develop an efficient batch-level routing mechanism, which reduces the router's parameter size by a factor of $L$ compared to routing at every adapted layer (where $L$ is the number of adapted layers) (iv) We propose data-aware initialization to achieve loss-free expert load balancing based on theoretical analysis. Extensive experiments on benchmarks in natural language understanding, image classification, and mathematical reasoning speak to the efficacy of TuckA, offering a new and effective solution to the PEFT problem.

Figures (9)

• Figure 1: (a) The performance and parameter efficiency tradeoff of TuckA and other PEFT methods on the CoLA benchmark, where the top-left is ideal. (b) Parameter scaling under different configurations of TuckA's hierarchical multi-expert framework. The plot shows the number of trainable parameters as a function of expert numbers when adapting a single linear layer in DeBERTa-v3-base model.
• Figure 2: An overview of the proposed TuckA architecture. It is primarily composed of hierarchical experts constructed through Tucker decomposition and a batch-level routing mechanism, which determines how the experts are combined to produce the adaptation weight.
• Figure 3: UMAP visualization of expert centroids and input embeddings. The figure compares the distribution of expert centroids (red diamonds) and CIFAR-100 input embeddings (blue circles) in ViT-base model. (a), (b) Kaiming initialization. (c), (d) Our proposed data aware initialization (DAI).
• Figure 4: Comparison of expert activation on CIFAR-100. The bar charts show the total number of activations for each expert (E1, E2, E3) during the fine-tuning process. (a) With Kaiming initialization. (b) Our proposed data-aware initialization.
• Figure 5: Tradeoff between Performance and Parameter Efficiency. The ideal position is the top-left corner, representing high performance with high parameter efficiency. (a) A comparison of TuckA with other PEFT methods on the CoLA benchmark. (b) Impact of expert numbers and rank on CIFAR-100 dataset under few-shot setting.

Theorems & Definitions (16)

• Theorem 1: Imbalanced Load, Informal
• proof : proof sketch
• Theorem 2
• Definition 1: sub-gaussian
• Definition 2: sub-exponential
• proof
• Lemma 1: Gaussian Concentration
• proof
• Lemma 2: Bernstein
• proof