On the Compression of Language Models for Code: An Empirical Study on CodeBERT

TL;DR

This study systematically evaluates three LM compression strategies—knowledge distillation, quantization, and pruning—on CodeBERT fine-tuned for vulnerability detection, code summarization, and code search. It assesses efficiency (inference time, model size) and effectiveness (task-specific metrics) to characterize cross-task trade-offs. The results show that quantization is most effective for reducing model size with minimal impact on accuracy but can slow inference, while knowledge distillation offers broad efficiency gains at the cost of reduced effectiveness, and pruning yields limited or hardware-dependent benefits. The findings provide actionable guidance for practitioners choosing compression strategies based on task and hardware constraints, and highlight avenues for future research in automatic strategy selection and broader LM evaluations with reproducible replication materials.

Abstract

Language models have proven successful across a wide range of software engineering tasks, but their significant computational costs often hinder their practical adoption. To address this challenge, researchers have begun applying various compression strategies to improve the efficiency of language models for code. These strategies aim to optimize inference latency and memory usage, though often at the cost of reduced model effectiveness. However, there is still a significant gap in understanding how these strategies influence the efficiency and effectiveness of language models for code. Here, we empirically investigate the impact of three well-known compression strategies -- knowledge distillation, quantization, and pruning -- across three different classes of software engineering tasks: vulnerability detection, code summarization, and code search. Our findings reveal that the impact of these strategies varies greatly depending on the task and the specific compression method employed. Practitioners and researchers can use these insights to make informed decisions when selecting the most appropriate compression strategy, balancing both efficiency and effectiveness based on their specific needs.

Figures (2)

• Figure 1: Experimental Methodology
• Figure 2: Trade-off between effectiveness (y-axis) and efficiency (x-axis) metrics for each of the SE tasks.