Work in Progress: AI-Powered Engineering-Bridging Theory and Practice
Oz Levy, Ilya Dikman, Natan Levy, Michael Winokur
TL;DR
The paper addresses whether generative AI can automate and improve requirements analysis in systems engineering under INCOSE standards. It evaluates AI across seven INCOSE criteria using NLP/ML pipelines and compares AI assessments with experienced engineers on a DR TOOL RFID-based dataset. Contributions include an empirical comparison of AI models (GPT-4b3, Claude Sonnetb4, Llamab5) against experts using Cohen's Kappa, and methods for automated functional/non-functional classification and test-generation aligned with ISO/IEC 25030, along with education and responsible AI discussions. The work informs practical tool development and engineering education, while highlighting limitations like hallucinations and the need for human oversight.
Abstract
This paper explores how generative AI can help automate and improve key steps in systems engineering. It examines AI's ability to analyze system requirements based on INCOSE's "good requirement" criteria, identifying well-formed and poorly written requirements. The AI does not just classify requirements but also explains why some do not meet the standards. By comparing AI assessments with those of experienced engineers, the study evaluates the accuracy and reliability of AI in identifying quality issues. Additionally, it explores AI's ability to classify functional and non-functional requirements and generate test specifications based on these classifications. Through both quantitative and qualitative analysis, the research aims to assess AI's potential to streamline engineering processes and improve learning outcomes. It also highlights the challenges and limitations of AI, ensuring its safe and ethical use in professional and academic settings.