Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Analysis of AdvFusion: Adapter-based Multilingual Learning for Code Large Language Models

Amirreza Esmaeili, Fahd Seddik, Yongyi Ji, Fatemeh Fard, Fuxiang Chen

TL;DR

The paper extends AdvFusion to Code-LLMs and evaluates it across three software-engineering tasks, revealing that cross-language knowledge transfer via adversarial fusion yields task- and model-dependent gains. It systematically compares AdvFusion to AdapterFusion, LoRA, Compacter, and TaskAdapter, showing that advantages are not uniform: AdvFusion often helps code generation but can underperform for commit messages and translation unless augmented (e.g., with Compacter). The study provides practical guidance on when to prefer adapters vs fusion methods, highlights language-specific effects, and demonstrates substantial efficiency benefits from PEFT. Overall, results stress the importance of task, language, and model context when selecting a PEFT strategy for Code-LLMs, and they point to promising directions like Compacter-enabled AdvFusion and broader robustness evaluations.

Abstract

Programming languages can benefit from one another by utilizing a language model for software engineering tasks. Full fine-tuning and Parameter Efficient Fine-Tuning (PEFT) of Code Language Models (Code-LMs) has been explored for multilingual knowledge transfer. AdapterFusion is a PEFT architecture that aims to enhance task performance by leveraging information from multiple programming languages, but primarily focuses on the target programming language. In our previous work, we proposed AdvFusion, a novel PEFT-based approach that effectively learns from other programming languages before adapting to the target task. Though previous experiments showed that AdvFusion outperformed AdapterFusion and LoRA, it was applied on pre-trained Code-LMs and was limited to only two tasks, code summarization and method name prediction. In this study, we expanded our work and investigated AdvFusion on Code Large Language Models (Code-LLMs), considering three new tasks: code generation, code translation, and commit message generation. We observed that different Code-LLMs/tasks exhibit different characteristics. In code generation, AdvFusion outperformed AdapterFusion but not other PEFT methods (LoRA, Compacter, and TaskAdapter). In commit message generation, AdapterFusion performed better than AdvFusion, and contrary to code generation, we found that the other PEFT methods do not have better performance. In code translation, AdvFusion performed worse than AdapterFusion overall, with the performance gap marginally widening as the model size increases. However, consistent with code generation, other PEFT methods showed better performance.

Analysis of AdvFusion: Adapter-based Multilingual Learning for Code Large Language Models

TL;DR

The paper extends AdvFusion to Code-LLMs and evaluates it across three software-engineering tasks, revealing that cross-language knowledge transfer via adversarial fusion yields task- and model-dependent gains. It systematically compares AdvFusion to AdapterFusion, LoRA, Compacter, and TaskAdapter, showing that advantages are not uniform: AdvFusion often helps code generation but can underperform for commit messages and translation unless augmented (e.g., with Compacter). The study provides practical guidance on when to prefer adapters vs fusion methods, highlights language-specific effects, and demonstrates substantial efficiency benefits from PEFT. Overall, results stress the importance of task, language, and model context when selecting a PEFT strategy for Code-LLMs, and they point to promising directions like Compacter-enabled AdvFusion and broader robustness evaluations.

Abstract

Programming languages can benefit from one another by utilizing a language model for software engineering tasks. Full fine-tuning and Parameter Efficient Fine-Tuning (PEFT) of Code Language Models (Code-LMs) has been explored for multilingual knowledge transfer. AdapterFusion is a PEFT architecture that aims to enhance task performance by leveraging information from multiple programming languages, but primarily focuses on the target programming language. In our previous work, we proposed AdvFusion, a novel PEFT-based approach that effectively learns from other programming languages before adapting to the target task. Though previous experiments showed that AdvFusion outperformed AdapterFusion and LoRA, it was applied on pre-trained Code-LMs and was limited to only two tasks, code summarization and method name prediction. In this study, we expanded our work and investigated AdvFusion on Code Large Language Models (Code-LLMs), considering three new tasks: code generation, code translation, and commit message generation. We observed that different Code-LLMs/tasks exhibit different characteristics. In code generation, AdvFusion outperformed AdapterFusion but not other PEFT methods (LoRA, Compacter, and TaskAdapter). In commit message generation, AdapterFusion performed better than AdvFusion, and contrary to code generation, we found that the other PEFT methods do not have better performance. In code translation, AdvFusion performed worse than AdapterFusion overall, with the performance gap marginally widening as the model size increases. However, consistent with code generation, other PEFT methods showed better performance.

Paper Structure

This paper contains 57 sections, 6 equations, 10 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Adapters
  4. Task Adapters
  5. Language Adapters
  6. AdapterFusion
  7. Adversarial Fusion Adapter
  8. Architecture
  9. Learning Algorithm
  10. AdvFusion on Code-Language Models
  11. Setup
  12. Backbone Models
  13. Tasks and Metrics
  14. Baselines
  15. Training Details
  16. ...and 42 more sections

Figures (10)

  • Figure 1: Internal structure of AdapterFusion.
  • Figure 2: The adversarial training phase of AdvFusion.
  • Figure 3: The fine-tuning phase of AdvFusion.
  • Figure 4: The attention contribution from each programming language at each layer when we feed the Ruby test dataset to the fine-tuned AdapterFusion model.
  • Figure 5: The attention contribution from each programming language at each layer when we feed the Ruby test dataset to the fine-tuned AdvFusion model.
  • ...and 5 more figures