Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges

Zi-Han Wang, Lam Nguyen, Zhengyang Zhao, Mengyue Yang, Chengwei Qin, Yujiu Yang, Linyi Yang

TL;DR

CreativeBench, a benchmark for evaluating machine creativity in code generation, grounded in a classical cognitive framework, is introduced and EvoRePE, a plug-and-play inference-time steering strategy that internalizes evolutionary search patterns to consistently enhance machine creativity is proposed.

Abstract

The saturation of high-quality pre-training data has shifted research focus toward evolutionary systems capable of continuously generating novel artifacts, leading to the success of AlphaEvolve. However, the progress of such systems is hindered by the lack of rigorous, quantitative evaluation. To tackle this challenge, we introduce CreativeBench, a benchmark for evaluating machine creativity in code generation, grounded in a classical cognitive framework. Comprising two subsets -- CreativeBench-Combo and CreativeBench-Explore -- the benchmark targets combinatorial and exploratory creativity through an automated pipeline utilizing reverse engineering and self-play. By leveraging executable code, CreativeBench objectively distinguishes creativity from hallucination via a unified metric defined as the product of quality and novelty. Our analysis of state-of-the-art models reveals distinct behaviors: (1) scaling significantly improves combinatorial creativity but yields diminishing returns for exploration; (2) larger models exhibit ``convergence-by-scaling,'' becoming more correct but less divergent; and (3) reasoning capabilities primarily benefit constrained exploration rather than combination. Finally, we propose EvoRePE, a plug-and-play inference-time steering strategy that internalizes evolutionary search patterns to consistently enhance machine creativity.

CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges

TL;DR

CreativeBench, a benchmark for evaluating machine creativity in code generation, grounded in a classical cognitive framework, is introduced and EvoRePE, a plug-and-play inference-time steering strategy that internalizes evolutionary search patterns to consistently enhance machine creativity is proposed.

Abstract

The saturation of high-quality pre-training data has shifted research focus toward evolutionary systems capable of continuously generating novel artifacts, leading to the success of AlphaEvolve. However, the progress of such systems is hindered by the lack of rigorous, quantitative evaluation. To tackle this challenge, we introduce CreativeBench, a benchmark for evaluating machine creativity in code generation, grounded in a classical cognitive framework. Comprising two subsets -- CreativeBench-Combo and CreativeBench-Explore -- the benchmark targets combinatorial and exploratory creativity through an automated pipeline utilizing reverse engineering and self-play. By leveraging executable code, CreativeBench objectively distinguishes creativity from hallucination via a unified metric defined as the product of quality and novelty. Our analysis of state-of-the-art models reveals distinct behaviors: (1) scaling significantly improves combinatorial creativity but yields diminishing returns for exploration; (2) larger models exhibit ``convergence-by-scaling,'' becoming more correct but less divergent; and (3) reasoning capabilities primarily benefit constrained exploration rather than combination. Finally, we propose EvoRePE, a plug-and-play inference-time steering strategy that internalizes evolutionary search patterns to consistently enhance machine creativity.
Paper Structure (108 sections, 12 equations, 12 figures, 21 tables, 2 algorithms)

This paper contains 108 sections, 12 equations, 12 figures, 21 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Machine Creativity Evaluation.
  4. Evolutionary Algorithm.
  5. Representation Engineering.
  6. CreativeBench
  7. Overview
  8. CreativeBench-Combo Dataset
  9. Solution Fusion.
  10. Test Function Generation.
  11. Problem Synthesis.
  12. CreativeBench-Explore Dataset
  13. Dynamic Constraint Stacking.
  14. Refinement and Termination.
  15. Data Filtering
  16. ...and 93 more sections

Figures (12)

  • Figure 1: The demonstration of two types of machine creativity considered in CreativeBench.
  • Figure 2: Overview of our framework.(Left) We introduce CreativeBench, built via an automated reverse engineering and self-play pipeline. (Middle) We evaluate evolutionary systems using a unified Creativity Score, defined as the Quality (Pass@1) and Novelty (embedding + n-gram distance). (Right) Based on our analysis, we propose the EvoRePE strategy to steer models toward more creative solutions at inference time.
  • Figure 3: Performance of foundation models on CreativeBench.The left and right columns correspond to the Combinatorial (CreativeBench-Combo) and Exploratory (CreativeBench-Explore) subsets, respectively.
  • Figure 4: Scaling analysis of the Qwen2.5-Instruct model family on CreativeBench.
  • Figure 5: Impact of reasoning mode on CreativeBench.
  • ...and 7 more figures