Multi-task Code LLMs: Data Mix or Model Merge?

TL;DR

The paper investigates how to construct compact multi-task code LLMs by comparing data mixing (joint fine-tuning on blended tasks) versus model merging (combining task-specific specialists) across two code-focused families and multiple scales. It reports a clear scale-dependent pattern: small models benefit from data-mixed SFT with limited degradation, while larger models gain from post-hoc merging of specialists, preserving or even enhancing performance on code generation and summarization benchmarks. A weight-space diagnostic based on layer-wise weight shifts and Pearson correlations links these outcomes to internal representations, enabling predictive guidance on strategy choice. Practically, the work provides actionable guidelines for resource-constrained deployment, enabling modular, multi-task code intelligence with minimal performance loss.

Abstract

Recent research advocates deploying smaller, specialized code LLMs in agentic frameworks alongside frontier models, sparking interest in efficient strategies for multi-task learning that balance performance, constraints, and costs. We compare two approaches for creating small, multi-task code LLMs: data mixing versus model merging. We conduct extensive experiments across two model families (Qwen Coder and DeepSeek Coder) at two scales (2B and 7B parameters), fine-tuning them for code generation and code summarization tasks. Our evaluation on HumanEval, MBPP, and CodeXGlue benchmarks reveals that model merging achieves the best overall performance at larger scale across model families, retaining 96% of specialized model performance on code generation tasks while maintaining summarization capabilities. Notably, merged models can even surpass individually fine-tuned models, with our best configuration of Qwen Coder 2.5 7B model achieving 92.7% Pass@1 on HumanEval compared to 90.9% for its task-specific fine-tuned equivalent. At a smaller scale we find instead data mixing to be a preferred strategy. We further introduce a weight analysis technique to understand how different tasks affect model parameters and their implications for merging strategies. The results suggest that careful merging and mixing strategies can effectively combine task-specific capabilities without significant performance degradation, making them ideal for resource-constrained deployment scenarios.

Figures (4)

• Figure 1: Given a base code model and multiple downstream tasks, we investigate which strategy is optimal between model merging and data mixing to obtain a tuned multi-task model, based on model size and across model families.
• Figure 2: Layerwise L$_2$ distance from the base model for each fine-tuned or merged variant, across four model architectures. Each subplot corresponds to one base model and parameter scale: (a) Qwen 1.5B, (b) Qwen 7B, (c) DeepSeek 1.3B, and (d) DeepSeek 7B. Within each subplot, curves are shown for the two single-task fine-tunes (CodeGen and CodeSum), the multi-task data-mix fine-tune, and the four merged models (Linear, TIES, DARE, DELLA). Lower values indicate that the post-trained model's weights remain closer to the original base model at that layer.
• Figure 3: Layerwise Pearson correlation between the weight updates of code-generation vs. code-summarization fine-tuned models for (a) DeepSeek and (b) Qwen families. Smaller models (Qwen 1.5B, DeepSeek 1.3B) show consistently higher correlation coefficients across layers, whereas larger models (7B) exhibit lower correlation.
• Figure 4: Layer-wise L2 norm weight differences from base models across fine-tuning and merging approaches. Each boxplot shows parameter deviations aggregated across transformer layers for Qwen and DeepSeek models. Three supervised fine-tuning (SFT) variants are compared: CodeGen (code generation), CodeSum (code summarization), and Data-Mixture, alongside four merging strategies: Linear, TIES, DARE, and DELLA. Data-mixture SFT produces the largest weight shifts, while merged models exhibit substantially smaller deviations. Code summarization SFT shows minimal parameter changes, particularly in 7B models, suggesting this task requires fewer weight modifications from the base model.