AdvFusion: Adapter-based Knowledge Transfer for Code Summarization on Code Language Models

TL;DR

AdvFusion introduces a principled adversarial extension to AdapterFusion for multilingual parameter-efficient fine-tuning of Code-LMs. By first learning from non-target language adapters and then incorporating the target adapter, it promotes cross-language knowledge transfer for code summarization and method name prediction. Empirical results across CodeBERT, GraphCodeBERT, and CodeT5+ show AdvFusion achieving substantial gains over AdapterFusion and LoRA on several languages, while reducing training time by about 44% on CodeBERT. The work highlights when multilingual adapter approaches are beneficial and provides open-source code to facilitate replication and extension to other tasks and architectures.

Abstract

Programming languages can benefit from one another by utilizing a pre-trained model for software engineering tasks such as code summarization and method name prediction. While full fine-tuning of Code Language Models (Code-LMs) has been explored for multilingual knowledge transfer, research on Parameter Efficient Fine-Tuning (PEFT) for this purpose is limited. AdapterFusion, a PEFT architecture, aims to enhance task performance by leveraging information from multiple languages but primarily focuses on the target language. To address this, we propose AdvFusion, a novel PEFT-based approach that effectively learns from other languages before adapting to the target task. Evaluated on code summarization and method name prediction, AdvFusion outperforms AdapterFusion by up to 1.7 points and surpasses LoRA with gains of 1.99, 1.26, and 2.16 for Ruby, JavaScript, and Go, respectively. We open-source our scripts for replication purposes.

Figures (6)

• Figure 1: Internal structure of AdapterFusion.
• Figure 2: The attention contribution from each programming language at each layer when we feed the Ruby test dataset to the fine-tuned AdapterFusion model.
• Figure 3: The adversarial training phase of AdvFusion.
• Figure 4: The fine-tuning phase of AdvFusion.
• Figure 5: The attention contribution from each programming language at each layer when we feed the Ruby test dataset to the fine-tuned AdvFusion model.