A Multi-Encoder Frozen-Decoder Approach for Fine-Tuning Large Language Models
Kaustubh D. Dhole
TL;DR
The paper investigates a multi-encoder frozen-decoder approach for parameter-efficient fine-tuning of large language models, aiming to reduce training and deployment overhead while mitigating catastrophic forgetting across multilingual and multi-task settings. By evaluating AlexaTM variants with frozen and trainable decoders on diverse NLP tasks, the authors show that decoder freezing is effective for natural-language outputs and multilingual transfer, especially when using larger frozen decoders to compensate for structured and QA tasks. Key findings include substantial gains in non-English languages under freezing in the MASSIVE benchmark, and the necessity of decoder scaling to maintain performance on structured tasks; cross-task interference is observed in mixed-task setups. The work offers practical guidance for deploying task groups with common output formats and highlights directions for future research, such as freezing encoders and improving multi-task design to balance cross-task effects.
Abstract
Among parameter-efficient fine-tuning methods, freezing has emerged as a popular strategy for speeding up training, reducing catastrophic forgetting, and improving downstream performance. We investigate the impact of freezing the decoder in a multi-task setup comprising diverse natural language tasks, aiming to reduce deployment overhead and enhance portability to novel tasks. Our experiments, conducted by fine-tuning both individual and multi-task setups on the AlexaTM model, reveal that freezing decoders is highly effective for tasks with natural language outputs and mitigates catastrophic forgetting in multilingual tasks. However, we find that pairing frozen decoders with a larger model can effectively maintain or even enhance performance in structured and QA tasks, making it a viable strategy for a broader range of task types.