Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Pooling

Jin Qin, Zihan Liao, Ziyin Zhang, Hang Yu, Peng Di, Rui Wang

TL;DR

C2LLM introduces a cross-attention pooling module (PMA) that sits on top of code-focused LLM backbones to produce flexible, high-quality code embeddings without sacrificing the causal pretraining of the LLM. By attending to all tokens with a single learnable query, PMA overcomes the information bottleneck of EOS or the limitations of mean pooling, while decoupling the final embedding dimension from the LLM hidden size. On public data, the 7B variant achieves a top score on the MTEB-Code benchmark (average $80.75$), and the 0.5B variant sets a new frontier for compute-efficient retrieval with an average of $75.46$, demonstrating robust performance across complex code reasoning tasks. These results suggest C2LLM’s embeddings are well-suited for scalable code search and autonomous code agents, and the work promotes open, multilingual expansion of CodeFuse Embeddings.

Abstract

We present C2LLM - Contrastive Code Large Language Models, a family of code embedding models in both 0.5B and 7B sizes. Building upon Qwen-2.5-Coder backbones, C2LLM adopts a Pooling by Multihead Attention (PMA) module for generating sequence embedding from token embeddings, effectively 1) utilizing the LLM's causal representations acquired during pretraining, while also 2) being able to aggregate information from all tokens in the sequence, breaking the information bottleneck in EOS-based sequence embeddings, and 3) supporting flexible adaptation of embedding dimension, serving as an alternative to MRL. Trained on three million publicly available data, C2LLM models set new records on MTEB-Code among models of similar sizes, with C2LLM-7B ranking 1st on the overall leaderboard.

C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Pooling

TL;DR

C2LLM introduces a cross-attention pooling module (PMA) that sits on top of code-focused LLM backbones to produce flexible, high-quality code embeddings without sacrificing the causal pretraining of the LLM. By attending to all tokens with a single learnable query, PMA overcomes the information bottleneck of EOS or the limitations of mean pooling, while decoupling the final embedding dimension from the LLM hidden size. On public data, the 7B variant achieves a top score on the MTEB-Code benchmark (average ), and the 0.5B variant sets a new frontier for compute-efficient retrieval with an average of , demonstrating robust performance across complex code reasoning tasks. These results suggest C2LLM’s embeddings are well-suited for scalable code search and autonomous code agents, and the work promotes open, multilingual expansion of CodeFuse Embeddings.

Abstract

We present C2LLM - Contrastive Code Large Language Models, a family of code embedding models in both 0.5B and 7B sizes. Building upon Qwen-2.5-Coder backbones, C2LLM adopts a Pooling by Multihead Attention (PMA) module for generating sequence embedding from token embeddings, effectively 1) utilizing the LLM's causal representations acquired during pretraining, while also 2) being able to aggregate information from all tokens in the sequence, breaking the information bottleneck in EOS-based sequence embeddings, and 3) supporting flexible adaptation of embedding dimension, serving as an alternative to MRL. Trained on three million publicly available data, C2LLM models set new records on MTEB-Code among models of similar sizes, with C2LLM-7B ranking 1st on the overall leaderboard.
Paper Structure (8 sections, 6 equations, 2 figures, 2 tables)

This paper contains 8 sections, 6 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model Architecture: Introducing Pooling by Multihead Attention into Embedding Models
  4. Takeaway
  5. Experiments
  6. Training Settings
  7. Results
  8. Conclusion

Figures (2)

  • Figure 1: MTEB-Code leaderboard. C2LLM-7B ranks 1st among all models, surpasssing the best closed-source models, while C2LLM-0.5B ranks 1st among models with less than 1B parameters, and 6th overall.
  • Figure 2: C2LLM Model architecture, comprising an LLM followed by a PMA (Pooling by Multihead Attention) module. PMA is a single layer of cross attention with one learnable query and takes the LLM's last hidden states as KV, serving both to pool over the input sequence and to provide support for flexible embedding dimension. Multi-head mechanism is omitted in the illustration.