Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Text-to-Code Generation with Modality-relative Pre-training

Fenia Christopoulou, Guchun Zhang, Gerasimos Lampouras

TL;DR

This paper tackles text-to-code generation by acknowledging that natural language and code tokens have distinct semantics and should occupy modality-specific embedding spaces. It introduces modality-relative continual pre-training (MRPT) built on top of modality-agnostic pre-training, with partial (PES) and full (FES) embedding separation and three objectives (code-clm, corrupt-code-clm, prefix-code-clm) to tailor representations for the downstream task. Through experiments on Python data with PyCodeGPT and PanGu-Coder, it demonstrates consistent zero-shot improvements on HumanEval and MBPP, and introduces the incremental pass@$k$ metric to better capture partial code solutions and completions. The work provides qualitative evidence via embedding visualizations and discusses generalization limitations, signaling practical implications for designing Code LMs that respect modality distinctions and offering avenues for more efficient and scalable modality-aware pre-training.

Abstract

Large pre-trained language models have recently been expanded and applied to programming language tasks with great success, often through further pre-training of a strictly-natural language model--where training sequences typically contain both natural and (linearised) programming language. Such approaches effectively map both modalities of the sequence into the same embedding space. However, programming language keywords (e.g. "while") often have very strictly defined semantics. As such, transfer learning from their natural language usage may not necessarily be beneficial to their code application and vise versa. Assuming an already pre-trained language model, in this work we investigate how sequence tokens can be adapted and represented differently, depending on which modality they belong to, and to the ultimate benefit of the downstream task. We experiment with separating embedding spaces between modalities during further model pre-training with modality-relative training objectives. We focus on text-to-code generation and observe consistent improvements across two backbone models and two test sets, measuring pass@$k$ and a novel incremental variation.

Text-to-Code Generation with Modality-relative Pre-training

TL;DR

This paper tackles text-to-code generation by acknowledging that natural language and code tokens have distinct semantics and should occupy modality-specific embedding spaces. It introduces modality-relative continual pre-training (MRPT) built on top of modality-agnostic pre-training, with partial (PES) and full (FES) embedding separation and three objectives (code-clm, corrupt-code-clm, prefix-code-clm) to tailor representations for the downstream task. Through experiments on Python data with PyCodeGPT and PanGu-Coder, it demonstrates consistent zero-shot improvements on HumanEval and MBPP, and introduces the incremental pass@ metric to better capture partial code solutions and completions. The work provides qualitative evidence via embedding visualizations and discusses generalization limitations, signaling practical implications for designing Code LMs that respect modality distinctions and offering avenues for more efficient and scalable modality-aware pre-training.

Abstract

Large pre-trained language models have recently been expanded and applied to programming language tasks with great success, often through further pre-training of a strictly-natural language model--where training sequences typically contain both natural and (linearised) programming language. Such approaches effectively map both modalities of the sequence into the same embedding space. However, programming language keywords (e.g. "while") often have very strictly defined semantics. As such, transfer learning from their natural language usage may not necessarily be beneficial to their code application and vise versa. Assuming an already pre-trained language model, in this work we investigate how sequence tokens can be adapted and represented differently, depending on which modality they belong to, and to the ultimate benefit of the downstream task. We experiment with separating embedding spaces between modalities during further model pre-training with modality-relative training objectives. We focus on text-to-code generation and observe consistent improvements across two backbone models and two test sets, measuring pass@ and a novel incremental variation.
Paper Structure (34 sections, 2 equations, 5 figures, 6 tables)

This paper contains 34 sections, 2 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Modality-agnostic Pre-Training
  4. Modality-relative Continual Pre-Training
  5. Space Separation
  6. Partial Embedding Separation (PES)
  7. Full Embedding Separation (FES)
  8. Training Objectives
  9. code-clm: Causal Language Modelling on Code
  10. corrupt-code-clm: Corrupted Docstring
  11. prefix-code-clm: Bidirectional Attention on Docstring
  12. Experimental Settings
  13. Data Formulation
  14. Inference
  15. Evaluation
  16. ...and 19 more sections

Figures (5)

  • Figure 1: Overview of modality-agnostic/relative pre-training and training objectives.
  • Figure 2: text-code clm, code-clm, corrupt-code-clm and prefix-code-clm pre-training objectives.
  • Figure 3: T-SNE plots of the top 20 neighbours of the tokens open (top) and join (bottom) on different embedding spaces for PanGu-Coder 350M parameter model with partial embedding space separation (PES).
  • Figure 4: Input data formats during modality-agnostic (MAPT) and modality-relative pre-training (MRPT).
  • Figure 5: T-SNE plots of the top 20 neighbours of the tokens def, when, except and split (top to bottom) on different embedding spaces.