Towards Better Code Understanding in Decoder-Only Models with Contrastive Learning

TL;DR

Decoder-only code-generation models excel at generation but underperform on code understanding tasks. The authors introduce CL4D, a contrastive-learning–based continued pre-training framework that adapts decoder-only models to code understanding by extracting discriminative representations (via last-token or averaged token embeddings) and training with in-batch and hard negatives using a dual-encoder setup. The approach achieves competitive or superior results to encoder-only baselines on code search and clone detection, with larger decoder-only models showing stronger gains and substantial zero-shot improvements. This work demonstrates the feasibility of unifying code understanding and generation within a single decoder-only backbone, potentially reducing the need to train encoder-only models from scratch.

Abstract

Recent advances in large-scale code generation models have led to remarkable progress in producing high-quality code. These models are trained in a self-supervised manner on extensive unlabeled code corpora using a decoder-only architecture. However, despite their generative strength, decoder-only models often exhibit limited performance on code understanding tasks such as code search and clone detection, primarily due to their generation-oriented training objectives. While training large encoder-only models from scratch on massive code datasets can improve understanding ability but remains computationally expensive and time-consuming. In this paper, we explore a more efficient alternative by transferring knowledge from pre-trained decoder-only code generation models to code understanding tasks. We investigate how decoder-only architectures can be effectively adapted to learn discriminative and semantically meaningful code representations. To this end, we propose CL4D, a contrastive learning framework tailored to strengthen the representation capabilities of decoder-only models. Extensive experiments on multiple benchmark datasets demonstrate that CL4D achieves competitive or superior performance compared to existing methods on representative code understanding tasks, including code search and clone detection. Further analysis reveals that CL4D substantially improves the semantic alignment of code representations by reducing the distance between semantically similar code snippets. These findings highlight the feasibility of leveraging decoder-only models as a unified backbone for both code generation and understanding.

Figures (4)

• Figure 1: Timeline of code-large models. For visual clarity, only the largest size of representative models is plotted. The orange arrow indicate the development trends of encoder-only code understanding models, while the green arrow represent the trends of decoder-only code generation models.
• Figure 2: Overview of CL4D.
• Figure 3: Case study on the code search task in the zero-shot setting.
• Figure 4: Visualization of code representations. The first row shows the encoding results of CodeGPT, while the second row illustrates the encoding results of CodeGPT+CL4D.