Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code

Hyungjoo Chae, Taeyoon Kwon, Seungjun Moon, Yongho Song, Dongjin Kang, Kai Tzu-iunn Ong, Beong-woo Kwak, Seonghyeon Bae, Seung-won Hwang, Jinyoung Yeo

TL;DR

Coffee-Gym presents an end-to-end RL environment for learning natural language feedback in code editing by coupling a human-grounded Coffee dataset with a unit-test–driven CoffeeEval reward. The approach addresses data scarcity and reward reliability by using pairwise feedback and synthetic test cases to train open-source feedback models that can rival closed-source baselines in editing tasks. Empirical results show CoffeeEval provides a faithful reward signal and that PPO-CoffeeEval, together with Coffee data, yields the best performance among training strategies. This work enables open-source progress in code-editing assistance and provides publicly available data and models to drive further research. The combination of SFT, RL, and unit-test evaluation demonstrates practical potential for improving NL feedback quality and, consequently, code correctness in real-world editing workflows.

Abstract

This paper presents Coffee-Gym, a comprehensive RL environment for training models that provide feedback on code editing. Coffee-Gym includes two major components: (1) Coffee, a dataset containing humans' code edit traces for coding questions and machine-written feedback for editing erroneous code; (2) CoffeeEval, a reward function that faithfully reflects the helpfulness of feedback by assessing the performance of the revised code in unit tests. With them, Coffee-Gym addresses the unavailability of high-quality datasets for training feedback models with RL, and provides more accurate rewards than the SOTA reward model (i.e., GPT-4). By applying Coffee-Gym, we elicit feedback models that outperform baselines in enhancing open-source code LLMs' code editing, making them comparable with closed-source LLMs. We make the dataset and the model checkpoint publicly available.

Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code

TL;DR

Coffee-Gym presents an end-to-end RL environment for learning natural language feedback in code editing by coupling a human-grounded Coffee dataset with a unit-test–driven CoffeeEval reward. The approach addresses data scarcity and reward reliability by using pairwise feedback and synthetic test cases to train open-source feedback models that can rival closed-source baselines in editing tasks. Empirical results show CoffeeEval provides a faithful reward signal and that PPO-CoffeeEval, together with Coffee data, yields the best performance among training strategies. This work enables open-source progress in code-editing assistance and provides publicly available data and models to drive further research. The combination of SFT, RL, and unit-test evaluation demonstrates practical potential for improving NL feedback quality and, consequently, code correctness in real-world editing workflows.

Abstract

This paper presents Coffee-Gym, a comprehensive RL environment for training models that provide feedback on code editing. Coffee-Gym includes two major components: (1) Coffee, a dataset containing humans' code edit traces for coding questions and machine-written feedback for editing erroneous code; (2) CoffeeEval, a reward function that faithfully reflects the helpfulness of feedback by assessing the performance of the revised code in unit tests. With them, Coffee-Gym addresses the unavailability of high-quality datasets for training feedback models with RL, and provides more accurate rewards than the SOTA reward model (i.e., GPT-4). By applying Coffee-Gym, we elicit feedback models that outperform baselines in enhancing open-source code LLMs' code editing, making them comparable with closed-source LLMs. We make the dataset and the model checkpoint publicly available.
Paper Structure (96 sections, 6 equations, 15 figures, 6 tables)

This paper contains 96 sections, 6 equations, 15 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Task Definition and Problem Statement
  3. Code Editing with Natural Language Feedback
  4. Learning Feedback Models
  5. Supervised fine-tuning.
  6. Reinforcement learning.
  7. Problem Statement
  8. Dataset.
  9. Reward model.
  10. Constructing Coffee-Gym
  11. Coffee: Human-written Code Edit Traces with Annotated Pairwise Feedback
  12. Collecting Code Edit Traces from Human Programmers
  13. Annotating Pairwise Feedback Data
  14. Augmenting Synthetic Test Cases
  15. CoffeeEval: Unit-test-driven Feedback Evaluation
  16. ...and 81 more sections

Figures (15)

  • Figure 1: A motivating example (Top) and Pass@1 accuracy in HumanEvalFix (Bottom). We compare the feedback from our model and various other models, both paired with DeepSeekCoder-7B as the code editor. SFT denotes the model trained on Code-Feedback zheng2024opencodeinterpreter using the same backbone model as ours.
  • Figure 2: Comparison between Coffee-Gym and the previous approach.
  • Figure 3: Overview of the data collection process of Coffee.
  • Figure 4: Example and statistics of Coffee.
  • Figure 5: Analysis results of Coffee. Experiment details are in Appendix \ref{['appendix:error_code_analysis']}.
  • ...and 10 more figures