BikeBench: A Bicycle Design Benchmark for Generative Models with Objectives and Constraints

TL;DR

BikeBench introduces a first-of-its-kind, constrained engineering design benchmark for generative models, focusing on parametric bicycle designs that map to CAD representations and multiphysics evaluators. It combines 10 objectives and 40 constraints across geometric, structural, aerodynamic, ergonomic, usability, and aesthetic criteria, using four core metrics (design quality, constraint violation, similarity to data, and diversity) and a standardized evaluation procedure. Across baselines, LLMs, tabular generative models, and optimization-augmented approaches, optimization-augmented GenAI methods deliver the strongest balance of performance, while pure LLMs and simple tabular models show clear room for improvement. The results highlight the value of hybrid methods and multi-objective optimization in constrained engineering design, and the authors advocate for broader, multimodal, and cross-domain benchmarks to accelerate progress in AI-assisted engineering design.

Abstract

We introduce BikeBench, an engineering design benchmark for evaluating generative models on problems with multiple real-world objectives and constraints. As generative AI's reach continues to grow, evaluating its capability to understand physical laws, human guidelines, and hard constraints grows increasingly important. Engineering product design lies at the intersection of these difficult tasks, providing new challenges for AI capabilities. BikeBench evaluates AI models' capabilities to generate bicycle designs that not only resemble the dataset, but meet specific performance objectives and constraints. To do so, BikeBench quantifies a variety of human-centered and multiphysics performance characteristics, such as aerodynamics, ergonomics, structural mechanics, human-rated usability, and similarity to subjective text or image prompts. Supporting the benchmark are several datasets of simulation results, a dataset of 10,000 human-rated bicycle assessments, and a synthetically generated dataset of 1.6M designs, each with a parametric, CAD/XML, SVG, and PNG representation. BikeBench is uniquely configured to evaluate tabular generative models, large language models (LLMs), design optimization, and hybrid algorithms side-by-side. Our experiments indicate that LLMs and tabular generative models fall short of hybrid GenAI+optimization algorithms in design quality, constraint satisfaction, and similarity scores, suggesting significant room for improvement. We hope that BikeBench, a first-of-its-kind benchmark, will help catalyze progress in generative AI for constrained multi-objective engineering design problems. We provide code, data, an interactive leaderboard, and other resources at https://github.com/Lyleregenwetter/BikeBench.

Figures (11)

• Figure 1: BikeBench is a set of evaluators, metrics, and datasets---built to benchmark generative models' capability to synthesize parametric bicycle designs satisfying a variety of objectives and constraints.
• Figure 2: Overview of the synthetic data generation pipeline.
• Figure 3: High-level overview of the benchmarking procedure.
• Figure 4: Example scorecard. The scorecard shows key model performance metrics at the top, objective score distributions in the middle, and constraint violation rates at the bottom.
• Figure 5: Scorecards for unconditional benchmarking results (part 1)