1. The bill that scales with success
Every new user makes your LLM bill bigger. That is the deal you sign with a per-token API: the cost structure is variable, forever, and it grows exactly as fast as the thing you wanted to grow. Self-hosting flips it. You stop paying per token and start paying per hour of GPU time, which stays flat no matter how hard you push the hardware.
One honest sentence before anything else: below a few million tokens a day, the API is cheaper and simpler, and you should keep it. This article is for everyone past that line. All figures below are estimates for orientation, rounded aggressively.
2. Pick the model
Open-weights models in 2026 are good enough for most production workloads: classification, extraction, internal copilots, agents. You are not choosing between frontier or nothing; you are choosing the smallest model that passes your own evaluation on your own prompts.
Our worked example for the rest of this article: Qwen3-32B under vLLM on a single H100 80GB node, which public benchmarks put at roughly 2,300-2,400 output tokens per second at healthy batch sizes. The bigger option on the same card is gpt-oss-120B, a mixture-of-experts model with about 5B active parameters per token; it fits and serves, at roughly 1,000 tokens per second.
| Model | Hardware | Sustained throughput (est.) |
|---|---|---|
| Qwen3-32B | 1× H100 80GB | ~2,350 tok/s ≈ 8.5M tok/hour |
| gpt-oss-120B (MoE, ~5B active) | 1× H100 80GB | ~1,000 tok/s ≈ 3.6M tok/hour |
3. What you are actually paying for
Three facts explain the whole cost structure. First, the model's weights must fit in GPU memory; that alone decides how many cards you need before a single token is generated. Second, generating tokens is mostly a memory-bandwidth problem, not a compute problem: the GPU spends its time moving weights around, not multiplying. Third, and this is the fact that makes the economics work at all: a GPU serving one user is almost idle. Serving many users at once, so that one read of the weights produces tokens for the whole batch, is what makes a GPU pay for itself.
That is the whole theory section. For a much deeper engineering treatment of inference math at serious scale, see Cline's write-up on self-hosting LLMs; we deliberately stay at back-of-the-envelope depth here.
4. The one-line cost model
The entire cost model fits in one line:
Plug in the worked example. An H100 80GB rents for about $2.50/hour on committed terms (on-demand runs $3-3.50, spot lower; market-dependent). vLLM sustains about 8.5M tokens/hour on Qwen3-32B at healthy batch sizes. So: $2.50 ÷ 8.5 ≈ $0.29 per 1M tokens at full load, which you will not have. Typical GPT-4-class API pricing blends to roughly $5-15 per 1M tokens, with flagship models near $10. The raw gap is around 30x; the rest of this article is about how much of that gap survives contact with reality.
5. Utilization is the whole game
Traffic has peaks and troughs. A GPU you rent for the peak sits half-idle the rest of the day, and every idle hour drags your real cost per million tokens back toward the API's price. At 25% utilization our $0.29 becomes roughly $1.18. Still cheap, but a 4x haircut that the one-line model never showed you.
The honest architecture: size the GPUs for the base load and let the peaks spill over to a public API behind one gateway that routes by load. You get flat-rate economics for the traffic you always have, and pay per token only for the traffic you sometimes have. The bill you should model is the hybrid bill, not the fantasy bill.
6. The monthly bill
Two concrete workloads, with managed operations priced in rather than hidden behind a plus sign. Assumptions: blended API rate $10 per 1M tokens (flagship-class); a private node is one H100 plus its share of 24/7 operations; one node comfortably carries ~80M tokens/day of a 30B-class model at realistic utilization.
| Scenario | At 30M tok/day | At 100M tok/day |
|---|---|---|
| Public API, blended $10/1M | ~$9,000 | ~$30,000 |
| Private: GPU nodes + managed operations | ~$8,500 | ~$11,300 |
Read the table honestly. At 30M tokens a day you are sitting almost exactly on the breakeven: the private bill matches the API bill, and what you are buying is data control, flat costs, and freedom from rate limits rather than savings. At 100M tokens a day the private bill is roughly 60% lower, and it keeps flattening as volume grows because extra nodes cost far less than the first one. And one thing the table undersells: privacy comes free. Every prompt stays on hardware you control, at either volume.
7. When to do it, and when not to
Self-host when at least one of these is true: you sustain multiple millions of tokens per day and the trend is up; or privacy and compliance pressure means prompts cannot leave your perimeter. Both together make the decision easy.
Below that: stay on the API and revisit quarterly. Open models and GPU prices both move fast enough that last quarter's no can be this quarter's yes.
This is exactly the modeling we run as a free cost assessment at DigitalCare: against your real workload, with the honest answer either way, including "keep the API." An anonymized production result of this exact playbook is in our private LLM case study.
