Towards a Middleware for Large Language Models

TL;DR

This paper discusses the vision for a forward-looking middleware system architecture that facilitates the deployment and adoption of LLMs in enterprises, even for advanced use cases in which it foresee LLMs to serve as gateways to a complete application ecosystem and, to some degree, absorb functionality traditionally attributed to the middleware.

Abstract

Large language models have gained widespread popularity for their ability to process natural language inputs and generate insights derived from their training data, nearing the qualities of true artificial intelligence. This advancement has prompted enterprises worldwide to integrate LLMs into their services. So far, this effort is dominated by commercial cloud-based solutions like OpenAI's ChatGPT and Microsoft Azure. As the technology matures, however, there is a strong incentive for independence from major cloud providers through self-hosting "LLM as a Service", driven by privacy, cost, and customization needs. In practice, hosting LLMs independently presents significant challenges due to their complexity and integration issues with existing systems. In this paper, we discuss our vision for a forward-looking middleware system architecture that facilitates the deployment and adoption of LLMs in enterprises, even for advanced use cases in which we foresee LLMs to serve as gateways to a complete application ecosystem and, to some degree, absorb functionality traditionally attributed to the middleware.

Figures (12)

• Figure 1: Retrieval-augmented LLM deployment with multiple model variants and session cache
• Figure 2: LLM as a microservice.
• Figure 3: Baseline use-case. The typical scenario for using an LLM is when no external services are considered. In this case, the middleware typically involves keeping track of registered users, caching information about the current or previous sessions, and scheduling jobs. Additionally, the middleware can also involve components that handle observability and explainability.
• Figure 4: User Registry. To access the framework, every user has an AUTH key and Access Certificate associated. The AUTH key is intended to enable a programmatic way to grant or restrict access to the middleware. The Access Certificate essentially contains all access roles associated with services, for instance, a user might have access to Service 1, and Service 2, both of which have to be present in the Access Certificate.
• Figure 5: LLM as a gateway.