What AI hosting actually costs, and when renting a GPU is worth it
AI infrastructure is where the biggest cloud billing mistakes are being made right now, because idle GPUs cost the same as busy ones and nothing warns you. Here is how the two models actually compare.
| Model | Typical cost | Best when |
|---|---|---|
| Hosted API, per token | Cents per thousand tokens | Bursty or unpredictable usage; you want zero cost when idle |
| Serverless GPU, per second | $0.0005 - $0.002 / sec | Intermittent heavy jobs; you accept a cold start |
| Rented GPU, hourly | $1.20 - $3.50 / hour | Sustained load above roughly 8-12 hours a day |
| Reserved GPU, monthly | $700 - $2,000 / month | Continuous production inference with steady demand |
Where AI bills actually go wrong
- A GPU left running after an experiment finished - the single most common one
- Sending far more context than the task needs, on every single call
- No spending cap, so a loop or a retry storm bills silently until someone notices
- Paying for a large model where a small one would do the job
- Re-computing the same answer instead of caching it
Controls worth putting in before anything else
- A hard monthly spending cap, set at the provider
- Per-feature cost tracking, so you know which feature costs what
- Alerts on daily spend, not just monthly - a monthly alert arrives too late
- Automatic shutdown of idle GPU instances
Common questions
Is it cheaper to run our own model?
Only at sustained volume. Below roughly 8 to 12 hours of continuous GPU work a day, a hosted API almost always wins, because you pay nothing when idle and someone else handles the operational work.
What is the most common AI cost mistake?
An idle GPU. It bills identically whether it is serving traffic or sitting at zero percent utilisation, and nothing tells you.
Affiliate disclosure. Some links on this page are affiliate links, which means we may earn a commission if you sign up. It does not change what we recommend or the order things appear in, and we only recommend providers we run workloads on ourselves.
Also worth reading
Want this checked against your actual bill?
We will read your current setup and spend and tell you where the money is going, whether or not you hire us.