Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Aligning CodeLLMs with Direct Preference Optimization

Yibo Miao, Bofei Gao, Shanghaoran Quan, Junyang Lin, Daoguang Zan, Jiaheng Liu, Jian Yang, Tianyu Liu, Zhijie Deng

TL;DR

This work identifies that the commonly used PPO algorithm may be suboptimal for the alignment of CodeLLM because the involved reward rules are routinely coarse-grained and potentially flawed, and advocates addressing this using the DPO algorithm, which can render the model rank data automatically and give rise to a fine-grained rewarding pattern more robust than human intervention.

Abstract

The last year has witnessed the rapid progress of large language models (LLMs) across diverse domains. Among them, CodeLLMs have garnered particular attention because they can not only assist in completing various programming tasks but also represent the decision-making and logical reasoning capabilities of LLMs. However, current CodeLLMs mainly focus on pre-training and supervised fine-tuning scenarios, leaving the alignment stage, which is important for post-training LLMs, under-explored. This work first identifies that the commonly used PPO algorithm may be suboptimal for the alignment of CodeLLM because the involved reward rules are routinely coarse-grained and potentially flawed. We then advocate addressing this using the DPO algorithm. Based on only preference data pairs, DPO can render the model rank data automatically, giving rise to a fine-grained rewarding pattern more robust than human intervention. We also contribute a pipeline for collecting preference pairs for DPO on CodeLLMs. Studies show that our method significantly improves the performance of existing CodeLLMs on benchmarks such as MBPP and HumanEval.

Aligning CodeLLMs with Direct Preference Optimization

TL;DR

This work identifies that the commonly used PPO algorithm may be suboptimal for the alignment of CodeLLM because the involved reward rules are routinely coarse-grained and potentially flawed, and advocates addressing this using the DPO algorithm, which can render the model rank data automatically and give rise to a fine-grained rewarding pattern more robust than human intervention.

Abstract

The last year has witnessed the rapid progress of large language models (LLMs) across diverse domains. Among them, CodeLLMs have garnered particular attention because they can not only assist in completing various programming tasks but also represent the decision-making and logical reasoning capabilities of LLMs. However, current CodeLLMs mainly focus on pre-training and supervised fine-tuning scenarios, leaving the alignment stage, which is important for post-training LLMs, under-explored. This work first identifies that the commonly used PPO algorithm may be suboptimal for the alignment of CodeLLM because the involved reward rules are routinely coarse-grained and potentially flawed. We then advocate addressing this using the DPO algorithm. Based on only preference data pairs, DPO can render the model rank data automatically, giving rise to a fine-grained rewarding pattern more robust than human intervention. We also contribute a pipeline for collecting preference pairs for DPO on CodeLLMs. Studies show that our method significantly improves the performance of existing CodeLLMs on benchmarks such as MBPP and HumanEval.

Paper Structure

This paper contains 15 sections, 4 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Issues of PPO for CodeLLMs
  4. DPO for CodeLLMs
  5. Experiments
  6. Experimental Details
  7. Main Results
  8. On-policy DPO vs. Off-policy DPO
  9. Related Works
  10. CodeLLMs
  11. Execution Feedback from Code Executor
  12. Conclusion
  13. Limitations
  14. The Underlying Reward of DPO
  15. Hyperparameter Setting

Figures (2)

  • Figure 1: Two cases for illustration of the reward difference of different responses given by the DPO reward and rule-based reward. When given a code question, response 1 and response 2 are two responses that have logic errors but the two responses are not the same. Reward difference means the reward of response 1 minus that of response 2. The rule-based reward assigns the same reward to different responses while DPO recognizes the reward difference between the different responses.
  • Figure 2: The pipeline of using execution feedback from the code executor to construct preference dataset.