Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IndustryCode: A Benchmark for Industry Code Generation

Puyu Zeng, Zhaoxi Wang, Zhixu Duan, Liang Feng, Shaobo Wang, Cunxiang Wang, Jinghang Wang, Bing Zhao, Hu Wei, Linfeng Zhang

Abstract

Code generation and comprehension by Large Language Models (LLMs) have emerged as core drivers of industrial intelligence and decision optimization, finding widespread application in fields such as finance, automation, and aerospace. Although recent advancements have demonstrated the remarkable potential of LLMs in general code generation, existing benchmarks are mainly confined to single domains and languages. Consequently, they fail to effectively evaluate the generalization capabilities required for real-world industrial applications or to reflect the coding proficiency demanded by complex industrial scenarios. To bridge this gap, we introduce IndustryCode, the first comprehensive benchmark designed to span multiple industrial domains and programming languages. IndustryCode comprises 579 sub-problems derived from 125 primary industrial challenges, accompanied by rigorous problem descriptions and test cases. It covers a wide range of fields, including finance, automation, aerospace, and remote sensing-and incorporates diverse programming languages such as MATLAB, Python, C++, and Stata. In our evaluation, the top-performing model, Claude 4.5 Opus, achieved an overall accuracy of 68.1% on sub-problems and 42.5% main problems. The benchmark dataset and automated evaluation code will be made publicly available upon acceptance.

IndustryCode: A Benchmark for Industry Code Generation

Abstract

Code generation and comprehension by Large Language Models (LLMs) have emerged as core drivers of industrial intelligence and decision optimization, finding widespread application in fields such as finance, automation, and aerospace. Although recent advancements have demonstrated the remarkable potential of LLMs in general code generation, existing benchmarks are mainly confined to single domains and languages. Consequently, they fail to effectively evaluate the generalization capabilities required for real-world industrial applications or to reflect the coding proficiency demanded by complex industrial scenarios. To bridge this gap, we introduce IndustryCode, the first comprehensive benchmark designed to span multiple industrial domains and programming languages. IndustryCode comprises 579 sub-problems derived from 125 primary industrial challenges, accompanied by rigorous problem descriptions and test cases. It covers a wide range of fields, including finance, automation, aerospace, and remote sensing-and incorporates diverse programming languages such as MATLAB, Python, C++, and Stata. In our evaluation, the top-performing model, Claude 4.5 Opus, achieved an overall accuracy of 68.1% on sub-problems and 42.5% main problems. The benchmark dataset and automated evaluation code will be made publicly available upon acceptance.

Paper Structure

This paper contains 39 sections, 29 figures, 4 tables.

Table of Contents

  1. Introduction
  2. IndustryCode
  3. Diverse and Realistic Industrial Coding Problems
  4. A Main Problem with Multiple Sub-problems
  5. Data Annotation
  6. Problem Selection
  7. Manual Revision and Difficulty Enhancement
  8. Evaluation System Design:
  9. Iterative Verification:
  10. Experiments
  11. Evaluated Models
  12. Main Results
  13. Error Analysis and Failure Modes
  14. Impact of Explicit Reasoning(Thinking Mode)
  15. Domain-Specific Performance Analysis
  16. ...and 24 more sections

Figures (29)

  • Figure 1: Hierarchical decomposition of an IndustryCode task. A complex Main Problem is factorized into multiple modular Sub-problems to simulate real-world development workflows. Each component includes detailed functional requirements, necessary library dependencies, and precise function signatures.
  • Figure 2: Task distribution across programming languages in IndustryCode. The pie charts illustrate the proportional composition of the dataset. (a) Breakdown of sub-problems by languages. (b) Breakdown of main problems by languages.
  • Figure 3: Data Annotation flowchart
  • Figure 4: Performance comparison on main problems and sub-problems.The observed trends indicate that strong foundational capabilities in sub-problems generally translate to better performance in main problems.
  • Figure 5: Distribution of failure cases in IndustryCode.
  • ...and 24 more figures