Provisioning for Intelligence: A Reference Architecture for Enterprise AI/ML Infrastructure at Scale

Abstract

Enterprises adopting machine learning at scale frequently discover that their existing cloud platform, however mature, is the wrong shape for the workload. General-purpose cloud architecture is optimized around abundant, cheap, elastic commodity compute. AI workloads run on scarce, expensive accelerators, are dominated by the movement and lineage of large datasets, and split cleanly into a training world and a serving world with almost opposite operational characteristics. Treating AI infrastructure as ordinary cloud with graphics processors bolted on produces stranded capacity, runaway cost, and models that cannot be reproduced or safely retired. This article presents a five-layer reference architecture for enterprise AI/ML infrastructure (accelerated compute, data foundation, ML platform, serving, and a cross-cutting governance plane) and argues that each layer inverts an assumption that general-purpose cloud architecture takes for granted. I describe how accelerators should be pooled and scheduled rather than allocated per team, why data gravity constrains where the lower layers can live, how the model lifecycle is properly understood as a closed loop rather than a pipeline with an end, and why serving must be split into latency-bound and throughput-bound paths with opposing cost profiles. The reference architecture is provider-neutral and intended as a durable design vocabulary rather than a product recipe.