Generative AI Systems: A Systems-based Perspective on Generative AI

TL;DR

This paper reframes Generative AI from a sole focus on LLMs to Generative AI Systems (GenAISys), arguing for a modular, systems-level perspective that integrates data encoders, a central GenAI model, and a retrieval/storage layer to enable multimodal processing and tool use. It outlines a practical GenAISys architecture, discusses end-to-end training challenges, and surveys concrete instantiations such as RAGs, Speech2Txt, and Large Vision Models to illustrate how the framework maps to real systems. The authors advocate a compositionality-driven design approach, drawing connections to refinement, world models, safety specifications, and verifiability, and they highlight future opportunities in design patterns, tooling, and formal methods. The work aims to guide solution architects, practitioners, and researchers toward building trustworthy, scalable GenAISys with broad applicability across domains such as medicine, manufacturing, and education.

Abstract

Large Language Models (LLMs) have revolutionized AI systems by enabling communication with machines using natural language. Recent developments in Generative AI (GenAI) like Vision-Language Models (GPT-4V) and Gemini have shown great promise in using LLMs as multimodal systems. This new research line results in building Generative AI systems, GenAISys for short, that are capable of multimodal processing and content creation, as well as decision-making. GenAISys use natural language as a communication means and modality encoders as I/O interfaces for processing various data sources. They are also equipped with databases and external specialized tools, communicating with the system through a module for information retrieval and storage. This paper aims to explore and state new research directions in Generative AI Systems, including how to design GenAISys (compositionality, reliability, verifiability), build and train them, and what can be learned from the system-based perspective. Cross-disciplinary approaches are needed to answer open questions about the inner workings of GenAI systems.

Figures (8)

• Figure 1: An example of a Deep Learning system: An environment connected with a neural network consisting of three layers.
• Figure 2: An example of a learning system with a DL model: A neural network (three layers) is connected with a data source, a loss function and an optimizer.
• Figure 3: Diagrams for LLMs: A. An unconditional LLM. B. A conditional LLM.
• Figure 4: A general architecture of a Generative AI system with encoders for various modalities, a retrieval/storage module for accessing external tools and databases, and a central generative AI model producing new content (output). The snowflake icon represents that a module is "frozen" (i.e., already trained).
• Figure 5: An example of a learning system with GenAISys: Data is taken from the environment and it is used to first pre-train a DE (the first dotted subsystem) and subsequently a GeM is fine-tuned (the second dotted subsystem). Eventually, both trained subsystems are used for inference (the dashed subsystem).