FunPRM: Function-as-Step Process Reward Model with Meta Reward Correction for Code Generation

TL;DR

FunPRM tackles the challenge of improving code-generation with large language models by introducing a function-level Process Reward Model (PRM) and a meta-learning reward-correction mechanism. By organizing code into modular functions, FunPRM defines clear PRM steps and leverages unit-test-based final rewards to denoise noisy partial-solution rewards through a bi-level optimization process. Empirical results on LiveCodeBench and BigCodeBench show consistent gains across multiple base LLMs and establish state-of-the-art performance on LiveCodeBench when paired with strong base models, along with improved code readability and reusability. The work demonstrates practical benefits for test-time scaling in coding tasks and discusses safety considerations for deploying such systems in real-world development contexts.

Abstract

Code generation is a core application of large language models (LLMs), yet LLMs still frequently fail on complex programming tasks. Given its success in mathematical reasoning, test-time scaling approaches such as Process Reward Model (PRM)-based Best-of-N selection offer a promising way to improve performance. However, existing PRMs remain ineffective for code generation due to the lack of meaningful step decomposition in code and the noise of Monte Carlo-estimated partial-solution correctness scores (rewards). To address these challenges, we propose FunPRM. FunPRM prompts LLMs to encourage modular code generation organized into functions, with functions treated as PRM reasoning steps. Furthermore, FunPRM introduces a novel meta-learning-based reward correction mechanism that leverages clean final-solution rewards obtained via a unit-test-based evaluation system to purify noisy partial-solution rewards. Experiments on LiveCodeBench and BigCodeBench demonstrate that FunPRM consistently outperforms existing test-time scaling methods across five base LLMs, notably achieving state-of-the-art performance on LiveCodeBench when combined with O4-mini. Furthermore, FunPRM produces code that is more readable and reusable for developers.

Figures (13)

• Figure 1: Comparison of generated code between the baseline method and FunPRM. FunPRM prompts LLMs to encourage them generate modular code organized into multiple functions with accompanying docstrings. These functions serve as reasoning steps for the process reward model while simultaneously improving code readability for human developers.
• Figure 2: Leaderboard results on LiveCodeBench (2025-02-01--present) for FunPRM and other LLMs. FunPRM achieves state-of-the-art performance when using OpenAI O4-mini (High) as the base LLM.
• Figure 3: Chain-of-Function system prompt. The prompt encourages function-level logic decomposition, top-down function organization, and descriptive docstrings to produce clearly defined PRM reasoning steps in generated code.
• Figure 4: Meta-learning-based reward correction framework in FunPRM. Noisy partial-solution rewards (correctness scores) are initialized via Monte Carlo sampling. The PRM parameters are first updated using these noisy partial-solution rewards through a one-step gradient descent. The updated PRM is then evaluated on clean final-solution reward data to compute a meta loss, which is used to optimize the partial-solution rewards from the previous stage.
• Figure 5: Domain generalization results of FunPRM on HumanEval+ and MBPP+. FunPRM consistently improves pass@1 over the base Qwen2.5-7B-Coder model, indicating improved code generation quality across both benchmarks.