A system architecture for AI as a platform

Disclaimer. I am not an AI/LLM domain expert. I am interested to build a model to abstract an AI platform as a software system.

AI as a Platform (AIaaP)¹ feels very close to what Software as a Service (SaaS) was more than a decade ago. A space where the pace of changes is very high, challenges are emerging day by day, and opportunities grow by the hour.

As for any platform, an AI Platform must also have a system architecture that helps to:

1. abstract the platform in logical functionalities (system components)
2. reveal where functional boundaries end and cross-functional² requirements begin

We start with (1) which will help us think about (2). In this post, we present a system architecture in a simple form to abstract how an AI platform operates.

An AI Platform has 3 major system components:

1. Model Pre-Training
2. Model Post-Training
3. Model Deployment

Model Pre-Training

In the pre-training stage, the focus is on tokenization³, embeddings⁴, and transformers⁵ with attention architecture⁶.

In summary:

• A training loop⁷ runs iterations on a model with input and parameters.
• The outcome is to have a trained set of model hyper-parameters and weights $W$.
• This is stored as a .pt or a .bin file in storage.
• The training loop fully runs on GPU infrastructure.

Model Post-Training

Post-training takes the output of pre-training and focuses on optimization of the model parameters. This stage runs test/validation data through the model and applies fine-tuning techniques, e.g. DPO or RLHF, to derive a better version of model parameters.

In summary:

• In iterations,
  • test data is benchmarked with model parameters, and/or,
  • model output is evaluated against benchmarks for scoring and ranking, and/or,
  • human feedback is collected per the chosen strategy (e.g. RLHF pipelines), and/or,
  • advanced techniques such as DPO or RLHF are applied, and/or,
• The outcome is to have a next optimized version of model parameters and weights $W'$.
• The outcome is stored as .pt or a .gguf.
• The post-training loop fully runs on GPU infrastructure.
• Other steps such as quantization to reduce space and keep capability are applied on the way to deployment, sometimes off the training loop (e.g. post-training quantization on CPU).

Model Deployment

This stage is also known as “model inference” or “model serving”. Intuitively, it is loading the model in a runtime engine to interface and interact for prediction of next token in real time.

We can refer to the runtime engine as Model OS¹. As with an OS, it runs on a compute infra (GPU). It manages memory used by model parameters. It manages resources such as KV caches and paging for PagedAttention. vLLM⁸ is a popular choice as a Model OS.

Model OS is core but not sufficient to complete the system. It requires a bridge into a service architecture whose core functionality goes beyond prediction of the next token.

We refer to this as Model Services¹. It is a bridge architecture between the model and its consumers/users. It has components that focus on software system functionalities (e.g. Auth(z)). Others act as middleware between the consumer and the model (e.g. RAG). It operates on different compute infra (CPU).

Model Services also lays out the building blocks for the platform’s extensions and ecosystem, including APIs and SDKs for integration and MCP⁹ support, either on the server side or client side.

Here is my take on the system architecture for an AI Platform:

While Model Pre-Training and Post-Training stages are fascinating domains, I am more interested to dive deeper in Model OS and Model Services boundaries. They represent an emerging area of challenging problems. The space where AI as a model is transformed into a service experience. A service experience is commonly proved to be where cross-functional requirements start to push the system to its limits.

There are open questions that are challenging; a few that have been my thinking focus lately:

• How can context and knowledge be shared across multi-tenant requests without leaking one tenant’s data into another’s? If a privacy-safe sharing mechanism exists, it opens the door to caching responses across tenants, avoiding a meaningful share of recomputation and its cost.
• What kind of scalability¹⁰ applies when GPU and CPU infrastructure must scale together? Do they need to scale in lockstep (one unit of GPU capacity always paired with one unit of CPU capacity), in some fixed ratio, or can they scale independently and heterogeneously based on load?
• Within the Model Services bridge, which component is most likely to become the bottleneck or single point of failure as load and tenant count grow?

These may already have answers or working solutions. Though with fast pace of this emerging domain, the challenges also shift and it’s important to have a solid foundation to build for a scalable architecture and approach.

In SaaS era, principles of distributed systems and software architecture came to offer building blocks; e.g. tenant isolation, autoscaling, and system instrumentation were foundations for similar context of above questions. The same principles should be applicable, maybe with variations and adaptation, to evolve AI as a platform. I am thinking and writing lately in this space, and interested to share and collaborate.

@misc{nobakht2026system,
  author = {Nobakht, Behrooz},
  title = {A system architecture for AI as a platform},
  howpublished = {\url{https://nobeh.com/blog/2026/07/a-system-architecture-for-ai-platform/}},
  year = {2026},
  note = {Accessed: 2026-09-09}
}