Foundation Model Engineering: Engineering Foundation Models Just as Engineering Software

TL;DR

Foundation Models are expanding in capability but complexity risks a development crisis unless data and model management are systematized. The authors propose Foundation Model Engineering (FME), a framework of declarative, automated interfaces for data and model management, complemented by data-wrangling with weak supervision and a Git-like, collaborative workflow for models. Key contributions include a data-management architecture emphasizing labeling, cleaning, access control, and auditing, and a model-management architecture featuring distributed versioning, Fisher-information-guided updates, and modular mixture-of-experts with automated finetuning and deployment. The work outlines three research pillars—Declarative FM Engineering, Fine-grained Data Management, and Automating Model Management—along with practical implications for tooling, governance, and scalable, responsible FM development.

Abstract

By treating data and models as the source code, Foundation Models (FMs) become a new type of software. Mirroring the concept of software crisis, the increasing complexity of FMs making FM crisis a tangible concern in the coming decade, appealing for new theories and methodologies from the field of software engineering. In this paper, we outline our vision of introducing Foundation Model (FM) engineering, a strategic response to the anticipated FM crisis with principled engineering methodologies. FM engineering aims to mitigate potential issues in FM development and application through the introduction of declarative, automated, and unified programming interfaces for both data and model management, reducing the complexities involved in working with FMs by providing a more structured and intuitive process for developers. Through the establishment of FM engineering, we aim to provide a robust, automated, and extensible framework that addresses the imminent challenges, and discovering new research opportunities for the software engineering field.

Figures (6)

• Figure 1: An analogy between the development of traditional software and foundation models. In traditional software development, source code is manually written by human developers. The source code (e.g., cpp files) is compiled into an executable binary to perform specific tasks. In foundation model development, the "source code" typically consists of two main components: 1) the dataset, which outlines the desired behavior, and 2) the neural network architecture, which provides a basic structure for the program, though many specifications (such as the weights) remain to be determined. Through the training process, the dataset is "compiled" into the final foundation model, which is similar to the compiled binary of traditional software.
• Figure 2: (a) Increasing complexity of FMs (Figure \ref{['subfig::model_complexity']} borrowed from model_size_trend). The model size (represented by number of parameters) grows exponentially with respect to time since the invention of Transformer. (b) Diverse customization demand. The staggering array of models, datasets, and AI applications available on Hugging Face hugging_face, featuring 254,871 models with Transformer architecture, 127,462 datasets, and over 170,000 AI app demos, vividly illustrates the diverse customization demand of LLMs.
• Figure 3: Overview of Foundation Model (FM) Engineering.
• Figure 4: Overview of Data Management.
• Figure 5: Overview of Model Management.