On-Policy Optimization with Group Equivalent Preference for Multi-Programming Language Understanding

TL;DR

The paper tackles the cross-language performance gap in code generation by training large language models on code translation tasks. It introduces OORL, a hybrid reinforcement learning framework that combines on-policy code-translation optimization using a binary rule-based reward with GEPO, a group-based off-policy preference optimization that leverages intermediate representations to capture functional equivalence. GEPO enforces equivalence within groups of IRs and ties preference signals to policy probabilities, enabling nuanced guidance beyond pairwise comparisons. Empirical results on CrossPLEval and MultiPL-E show substantial improvements, including strong gains in low-resource languages, demonstrating the approach’s potential for broad multilingual code understanding and generation.

Abstract

Large language models (LLMs) achieve remarkable performance in code generation tasks. However, a significant performance disparity persists between popular programming languages (e.g., Python, C++) and others. To address this capability gap, we leverage the code translation task to train LLMs, thereby facilitating the transfer of coding proficiency across diverse programming languages. Moreover, we introduce OORL for training, a novel reinforcement learning (RL) framework that integrates on-policy and off-policy strategies. Within OORL, on-policy RL is applied during code translation, guided by a rule-based reward signal derived from unit tests. Complementing this coarse-grained rule-based reward, we propose Group Equivalent Preference Optimization (GEPO), a novel preference optimization method. Specifically, GEPO trains the LLM using intermediate representations (IRs) groups. LLMs can be guided to discern IRs equivalent to the source code from inequivalent ones, while also utilizing signals about the mutual equivalence between IRs within the group. This process allows LLMs to capture nuanced aspects of code functionality. By employing OORL for training with code translation tasks, LLMs improve their recognition of code functionality and their understanding of the relationships between code implemented in different languages. Extensive experiments demonstrate that our OORL for LLMs training with code translation tasks achieves significant performance improvements on code benchmarks across multiple programming languages.

Figures (3)

• Figure 1: Overview of the OORL integrating on-policy RL with the rule-based reward and on-policy preference optimization (GEPO).
• Figure 2: Overview of the equivalent and equivalent IRs groups for the training process with GEPO. The equivalent IRs group is constructed from different LLVM optimization levels (e.g., -Oz, -O3) of the source code. The inequivalent IRs group is augmented from the equivalent IRs group.
• Figure 3: Code Translation Performance of LLMs on CrossPLEval.