What to look out for in an AI training site.
Training is one synchronised job across thousands of GPUs. It needs a large, contiguous, dense, well-networked block of power. It tolerates distance from users, but not being split across sites.
Everything about a training site follows from one fact: the job is synchronised. Thousands of GPUs compute, exchange gradients, and compute again, in lockstep. That lockstep sets the size of the power block, the network, the layout, and even how the grid sees the load.
Because the job does not talk to users, the site does not need to be near them: remote regions with cheap power and room to grow suit it well.
Density and liquid cooling
Training racks are the densest in the building, and each GPU generation has raised the figure. NVIDIA describes GB300 NVL72 as fully liquid-cooled, so direct-to-chip liquid is the baseline for a training hall. Ask about the site’s density roadmap, not just today’s number.
- Heat rejection
- Capacity to reject the rack’s heat to liquid, with the CDU and facility loop installed today.
- Warm water
- Warm-water loops cut chiller energy, provided the plant is sized for the higher flow they need.
- Water
- Availability and quality, and what happens in a drought year.
Fabric and network
800 Gb/s per GPU over Quantum-X800 InfiniBand or Spectrum-X Ethernet. Run NCCL tests at acceptance, so cabling and configuration faults surface before the first training job.
- Fibre pathways sized for the east-west fabric, not just the uplinks.
- Spine and leaf placement that keeps cable runs within the reach of the optics you plan to use.
Contiguity and layout
Keep a cluster in one hall with adjacent expansion space. Splitting a training job across halls or sites adds latency to every synchronisation step, and every step waits for the slowest link.
Power quality and load swings
Synchronised compute and communication phases make the whole cluster’s load swing at once. Checkpointing, pause and resume, and ramp-up add spikes. This stresses grid frequency and voltage and complicates reserve planning.
- What a site can offer
- Rack or facility energy storage; ramp-rate limits agreed with the utility; GPU power capping.
- Inside the rack
- Power shelves with built-in energy storage can smooth swings inside the rack. It helps, but it does not remove the need to size for peak.
- The question to ask
- How much load swing does the interconnection allow, and what happens when the cluster exceeds it?
The redundancy trade-off
Training can checkpoint and restart, so N+1 power is often acceptable where inference wants 2N. But at liquid-cooled densities a cooling loss leaves very little time to react. CDU and pump redundancy, leak detection and fluid handling matter more than generator count.
Storage and checkpoints
Checkpoint writes are brief, sharp spikes in both power and storage traffic. Plan for high-throughput storage close to the cluster, and a network that carries checkpoint traffic without starving the fabric.
Training and inference at a glance
| AI inference site | AI training site | |
|---|---|---|
| Rack density | Lower | Highest, and rising with each GPU generation |
| Facility size | Smaller sites, many of them | Very large sites, few of them |
| Location | Premium metros near cloud availability zones | Remote regions with cheap, abundant power |
| Latency | Decisive: close to the users served | Largely irrelevant |
| Power redundancy | 2N, or at least N+1 | N+1 often acceptable; cooling redundancy matters more |
| Cooling | Air or rear-door usually enough | Direct-to-chip liquid is the baseline |
| Growth | Small steps, across many regions | One contiguous block, expanded in place |
| Load shape | Bursty, follows users, daily cycle | Synchronised swings across the whole cluster |
Questions for this kind of site
- What contiguous power is energised today, and what is the written date for the rest?
- What rack density is installed with liquid today, and what is the roadmap?
- Can the whole cluster sit in one hall, with expansion space next to it?
- What load ramp rate does the utility interconnection allow? Is there energy storage on site?
- What is the CDU and pump redundancy, and how is a leak detected and isolated?
- What are the facility water supply temperature, flow and quality?
- What do the generator and UPS cover, and does that include the liquid loop?
- Can we run NCCL and fabric tests before acceptance?
Questions for every GPU colocation deal
- Committed kW per rack at delivery, and the growth path after it
- Cooling installed at that density today, not planned
- The power redundancy model, and how power is billed
- Carrier neutrality and the cost of a cross-connect
- Proven GPU deployments on site, and remote hands who have done them
- Contract term, exit terms and overage terms
- Full-term cost per usable kW, not the headline rate
- Where the data sits and which compliance regimes apply
- Deployment date, and a plan for the years after it
Sources
Show 3 sourcesHide sources
Facts and figures on these pages come only from NVIDIA product pages and documentation, as of November 2026; everything else is general guidance. Confirm every number with the operator before you commit.
Have a requirement in mind? Send it to FPX and we check the site file against this guide for you.
FIND COLOCATION →