Parameter-Efficient Multi-Task Fine-Tuning in Code-Related Tasks

TL;DR

The paper assesses whether parameter-efficient fine-tuning with QLoRA can support effective multi-task learning across code generation, translation, and summarization. Using Qwen2.5-Coder at three scales, it compares Multi-task QLoRA against Single-task QLoRA and Multi-task full fine-tuning, analyzing both functional correctness and non-functional code quality with a rich set of metrics and human-aligned judgments. Key findings show that Multi-task QLoRA achieves competitive correctness while dramatically reducing resource usage, with larger models delivering better cross-task transfer and maintaining or improving code quality; translation results, however, are more nuanced and depend on direction and scale. The work provides a rigorous, reproducible framework and highlights the nuanced trade-offs between efficiency and performance in multi-task fine-tuning for code-oriented AI systems, offering guidance for deploying scalable, multi-task capable coding assistants.

Abstract

Large Language Models (LLMs) have proven highly effective in automating software engineering tasks, bridging natural language and code semantics to achieve notable results in code generation and summarization. However, their scale incurs substantial computational costs, making full fine-tuning impractical. Parameter-Efficient Fine-Tuning (PEFT) methods like QLoRA enable efficient specialization with lower resource demands. Recent studies show QLoRA-optimized Large Code Models (LCMs) perform strongly across diverse tasks, yet it remains unclear whether this effectiveness persists when a single model is QLoRA fine-tuned for multiple code-related tasks. The interaction between Multi-task fine-tuning and QLoRA optimization, and how transfer learning affects correctness and quality of generated artifacts, remains largely unexplored. We investigate Multi-task QLoRA fine-tuning across three representative tasks: code generation, translation, and summarization. We evaluate functional correctness through execution-based and similarity-based metrics, complemented by comprehensive code quality analysis--an aspect largely overlooked in prior work. Our findings show that Multi-task QLoRA effectively leverages transfer learning, achieving competitive or superior performance relative to both Single-task QLoRA and Multi-task full fine-tuning. Larger models demonstrate more consistent balance between correctness and quality, whereas smaller models preserve functionality but exhibit a higher incidence of quality-related issues.

Figures (6)

• Figure 1: Zero-shot prompt used for LLM-as-a-judge evaluation of code summaries.
• Figure 2: Qualitative comparison of Python code generated by ST-QLoRA and MT-QLoRA for an interface verification routine. MT-QLoRA (right, green checkmark) produces a passing solution while ST-QLoRA (middle, red X) fails functional testing.
• Figure 3: Qualitative comparison of C#$\rightarrow$Java translations generated by ST-QLoRA and MT-QLoRA for a representative instance where MT-QLoRA achieves exact match (EM$=1$) while ST-QLoRA fails (EM$=0$).
• Figure 4: Qualitative comparison of Python code summaries generated by ST-QLoRA and MT-QLoRA. MT-QLoRA produces a higher quality summary with better content adequacy, while ST-QLoRA generates a less accurate summary that misses key details.
• Figure 5: Boxplots for Java code summarization across three quality dimensions. Distribution compares Human-written summaries against models at three scales (0.5B, 1.5B, 3B) trained with Full Fine-Tuning (FFT, solid) or QLoRA (faded) in SingleTask and MultiTask configurations, including parameter-matched variants and task-pair combinations. Red diamonds indicate mean scores, blue lines indicate medians, background shading distinguishes model sizes.