Chapter 9 — Where Does AI Run?

AI also runs on physical computing machinery

The word "AI" sounds abstract. But a model's Training and Inference ultimately take place on a physical computer. That means AI is not entirely free from constraints such as:

• processor
• memory
• storage
• network
• power
• cooling

This chapter looks at where AI runs, from the broad perspective of:

Local
Cloud / Server
Data Center

CPU

A CPU is a general-purpose processor. It's used in many kinds of processing, including the operating system and applications. A CPU can also be involved in an AI system. However, a fixed division of labor — CPU only for control, GPU only for AI — does not apply to every system. Depending on the AI workload, processing may happen on the CPU alone.

GPUs and Accelerators

In AI, especially in the computation of large neural networks, accelerators such as GPUs or TPUs are sometimes used. These kinds of processors have designs suited to processing many computations in parallel. However, AI = always GPU does not hold. The computational resources required differ depending on model size, the nature of the processing, software, and hardware.

Memory is also needed

Running a model also requires memory to hold model weights and data being processed. Larger models generally need more computational resources and memory. That said, this book does not cover hardware sizing along the lines of "this many parameters always means this many GB." Required resources depend on multiple conditions, including model format, precision, runtime, and hardware.

Local LLM

An AI model does not necessarily run on a remote cloud. Using a compatible model and runtime, there are examples of running a generative AI model on a local device such as a PC or mobile device. Conceptually:

your own device
 ├─ CPU / GPU / accelerator
 ├─ memory
 ├─ runtime
 └─ model

This is broadly called Local LLM or on-device inference. However:

Local = always offline
Local = always safe
Local = higher-performing than Cloud

does not hold. Local execution being possible, and the entire system not using the network at all, are also not the same thing.

Running on the Cloud

In an AI service, the model may run on a server separate from the user's own device. The user sends a request over the network, Inference happens on the server side, and the result is returned. Conceptually:

User Device
    ↓ network
AI Service / Server
    ↓
Model Inference
    ↓
Result
    ↓ network
User Device

This diagram exists to help you understand the difference from Local execution.

Not necessarily just one machine

Large AI workloads are not necessarily processed on a single computer alone — configurations that combine multiple accelerators or servers also exist. In that case, compute, memory, and network need to be coordinated across multiple machines. That said, not every system uses the same cluster configuration. Even where a vendor's reference architecture exists, that's evidence that "such a configuration exists" — not proof of a structure common to all AI systems.

Data Center

An environment that operates many servers needs supporting infrastructure beyond the computers themselves — for example:

• electrical power
• network
• cooling
• physical space

A data center running AI workloads also involves this kind of infrastructure.

Why cooling is needed

A computer consumes energy when it computes, and part of that becomes heat. A data center running a large amount of IT equipment needs to handle that heat appropriately, which is why airflow management and cooling equipment matter. That said, it doesn't mean that every data center uses the same cooling method — the specific design varies by facility and equipment.

"Cloud" doesn't mean vanishing from the physical world

The word "Cloud" can make it feel as if computers exist in some abstract space. But a Cloud service, too, ultimately runs on top of physical infrastructure.

Cloud
 !=
no physical hardware exists

Even when AI is provided as a remote service, compute, network, and power, among other things, are needed behind it.

Local and Cloud aren't a simple better/worse comparison

You cannot classify Local as good and Cloud as bad, or the reverse. The comparison involves many conditions, including:

• model size
• hardware
• latency
• network
• cost
• operations
• security/privacy requirements
• required capability

The point of this chapter is not to teach you a selection method, but to understand that:

Inference in AI also requires real, physical computing resources.

Chapter summary

AI computation
    ↓
physical compute
    ├─ CPU
    ├─ GPU / accelerator
    ├─ memory
    ├─ network
    └─ supporting infrastructure

• AI also runs on a physical computer
• CPUs, GPUs, and TPUs can all be involved in AI workloads
• a GPU is not necessarily required for AI
• there are examples of running a model on a local device
• Local execution and Cloud execution can involve different configurations
• large-scale environments may also use configurations with multiple servers or accelerators
• a Data Center also requires things such as power, network, and cooling
• Cloud also runs on top of physical infrastructure
• don't generalize a specific vendor's reference architecture to every AI system

The next chapter organizes what limitations and unresolved problems remain in current AI, which runs using these computational resources.