Morescient GAI for Software Engineering (Extended Version)
Marcus Kessel, Colin Atkinson
TL;DR
This paper argues that trustworthiness and utility of Generative AI for software engineering can be substantially enhanced by morescient models trained on both static syntax and de facto run-time semantics. It introduces three data structures—sequence sheets, stimulus-response matrices (SRMs), and stimulus-response hypercubes (SRHs)—and the Large-Scale Software Observatorium (LASSO) platform to collect, organize, and analyze execution observations at scale. It then outlines an open, continually evolving SRH dataset and a comprehensive roadmap for data curation, training, augmented generation, prompting, test-driven experimentation, and AI-driven decision-making, all within an open-science framework. The envisioned approach aims to unify benchmarks, enable scalable observation collection, and accelerate the deployment of morescient GAI in mainstream software engineering, potentially transforming how software is synthesized, tested, and reasoned about in practice.
Abstract
The ability of Generative AI (GAI) technology to automatically check, synthesize and modify software engineering artifacts promises to revolutionize all aspects of software engineering. Using GAI for software engineering tasks is consequently one of the most rapidly expanding fields of software engineering research, with over a hundred LLM-based code models having been published since 2021. However, the overwhelming majority of existing code models share a major weakness - they are exclusively trained on the syntactic facet of software, significantly lowering their trustworthiness in tasks dependent on software semantics. To address this problem, a new class of "Morescient" GAI is needed that is "aware" of (i.e., trained on) both the semantic and static facets of software. This, in turn, will require a new generation of software observation platforms capable of generating large quantities of execution observations in a structured and readily analyzable way. In this paper, we present a vision and roadmap for how such "Morescient" GAI models can be engineered, evolved and disseminated according to the principles of open science.