What to look out for in an AI inference site.
Inference serves live users, so the site has to behave like production hosting: close to the people it serves, always on, and able to grow in small steps across regions.
Training and inference want different buildings. Inference runs in many smaller sites in premium metros near cloud availability zones; training runs in a few very large sites in remote regions with cheap power.
An inference site is judged the way a production web platform is judged: by latency to users, by uptime, by how cheaply and quickly it can add capacity, and by whether user data can legally live there.
Uptime and resilience
Uptime requirements are strict and latency SLAs are real. Look for Tier III or IV design, 2N or at least N+1 power, generators with fuel on site, and SLAs in writing with credits attached.
- Ask for the maintenance history of the UPS and generator plant, not just the design tier.
- Ask how many times the site has run on generator in the past year, and why.
Power, density and cooling
Inference racks are usually less dense than training racks. Air cooling or rear-door heat exchangers are usually enough; direct liquid only comes in when rack-scale systems such as GB300 NVL72, which NVIDIA describes as fully liquid-cooled, are deployed for inference. That makes far more existing inventory usable for inference than for training.
Load profile, and how power is billed
Inference power runs in short, highly variable bursts that follow user requests, with strong daily patterns. That affects how power should be billed, whether committed, peak or blended, and how much headroom to buy. A contract priced on peak demand penalises exactly the shape inference produces.
- Get the billing model in writing and model it against your own traffic curve.
- Ask what happens when you exceed committed power for an hour, a day, a month.
Growth and operations
Inference fleets grow in small steps and across regions. Look for contract flexibility, short add-on lead times, and remote hands with GPU experience: swapping a failed GPU server at three in the morning is a different job from rebooting a web server.
- Expansion rights to adjacent racks or cages, with lead times stated.
- Remote hands SLAs that name GPU-specific tasks.
- Spare parts handling and RMA logistics on site.
Compliance and data residency
User data has to live where the law allows. Data residency and sovereignty for the markets you serve are site-selection criteria, not afterthoughts. Ask about certifications (SOC 2, ISO 27001 and sector-specific ones) and about where backups and logs physically sit.
Inference and training at a glance
| AI inference site | AI training site | |
|---|---|---|
| Rack density | Lower | Highest, and rising with each GPU generation |
| Facility size | Smaller sites, many of them | Very large sites, few of them |
| Location | Premium metros near cloud availability zones | Remote regions with cheap, abundant power |
| Latency | Decisive: close to the users served | Largely irrelevant |
| Power redundancy | 2N, or at least N+1 | N+1 often acceptable; cooling redundancy matters more |
| Cooling | Air or rear-door usually enough | Direct-to-chip liquid is the baseline |
| Growth | Small steps, across many regions | One contiguous block, expanded in place |
| Load shape | Bursty, follows users, daily cycle | Synchronised swings across the whole cluster |
Questions for this kind of site
- What is the measured round-trip latency from this site to our top three markets?
- Which carriers, cloud on-ramps and exchanges are in the building, and what does a cross-connect cost per month?
- What is the design tier, and when did the site last run on generator?
- How is power billed: committed, peak or blended? Show us the formula.
- What density can you support on air, and what above it?
- What are the lead times and terms for adding racks in this site and in your other regions?
- Does remote hands cover GPU server swaps, and what is the SLA?
- Where do our data, backups and logs physically sit, and which certifications apply?
Questions for every GPU colocation deal
- Committed kW per rack at delivery, and the growth path after it
- Cooling installed at that density today, not planned
- The power redundancy model, and how power is billed
- Carrier neutrality and the cost of a cross-connect
- Proven GPU deployments on site, and remote hands who have done them
- Contract term, exit terms and overage terms
- Full-term cost per usable kW, not the headline rate
- Where the data sits and which compliance regimes apply
- Deployment date, and a plan for the years after it
Sources
Show 1 sourceHide sources
Facts and figures on these pages come only from NVIDIA product pages and documentation, as of November 2026; everything else is general guidance. Confirm every number with the operator before you commit.
Have a requirement in mind? Send it to FPX and we check the site file against this guide for you.
FIND COLOCATION →