Self-Hosting vs. API: The Real Break-Even Math

Search for “self-hosting vs API break-even” and you’ll find the same claim repeated across a dozen blog posts: somewhere around 10 to 50 million tokens a day, running your own GPUs starts beating what you’d pay OpenAI, Anthropic, or an open-weight API provider. It’s a specific-sounding number. It’s also, as far as I can tell after actually tracing it back, not derived from anything. Nobody cites a methodology. Nobody shows their work. It’s just… repeated.

So let’s do the math ourselves, from numbers that actually have a source attached. The honest version turns out to be more interesting than the rule of thumb, because the variable that decides the outcome isn’t the one most people are arguing about.

The number that makes self-hosting look obviously worth it

Start with the appealing case, because it’s real. SemiAnalysis runs a benchmarking platform called InferenceX that measures actual serving throughput for DeepSeek R1 (671B parameters, FP8 precision) on different GPUs. As of their published comparison, at a target of 42 tokens/second per user, an Nvidia B200 pushes about 4,792 tokens/sec per GPU, working out to roughly $0.11 per million tokens. An H100 at the same latency target manages 266 tokens/sec per GPU, landing around $1.36 per million tokens.

Separately, a Hacker News thread from earlier this year walked through the economics of serving DeepSeek-V3 (671B) on a 16xH200 cluster using vLLM’s “wide expert-parallelism” technique, a serving optimization for mixture-of-experts models that’s been maturing through 2026. Commenters got 2.2k tokens/sec per H200 GPU. Do the depreciation math on roughly $750,000 of hardware over three years, and you land at $0.22 to $0.30 per million tokens, plus another few cents for electricity.

Now compare that to what you’d actually pay an API provider for a comparable open-weight model, as of July 2026: DeepSeek V3 runs $0.25 to $0.89 per million tokens blended across providers like Hyperbolic, DeepInfra, and Novita. Llama 3.3 70B sits around $0.88-0.90/M on Together and Fireworks. Qwen3’s smaller mixture-of-experts variant goes for as little as $0.12-0.50/M on DeepInfra, while the flagship Qwen3.7 Max runs $1.25-3.75/M. Kimi K2’s family spans roughly $0.60-3.50/M depending on version. GLM 5.2, which independent analyst Martin Alderson described in a July 2026 post as the first open-weight model to genuinely reach GPT/Opus-tier quality, prices around $4.40/M.

Line those up against the $0.11-0.30/M self-hosted numbers and self-hosting looks like a 2x to 10x win. That’s a real spread, from real benchmarks, not a marketing claim. If you stopped reading here, you’d go buy GPUs.

The number that actually decides it

Here’s the thing about every one of those self-hosted numbers: they assume the GPU is busy. Constantly. That assumption is doing almost all the work, and it’s the part that quietly falls apart in production.

A GPU sitting at 10% utilization doesn’t cost 10% of the busy price. It costs the same per hour whether it’s serving requests or idling, which means your effective cost-per-token at 10% utilization is roughly 10x worse than the benchmark number. An API, by contrast, scales to zero the moment you stop calling it. You’re never paying for idle capacity you don’t own.

This is why the widely repeated “50%+ utilization needed to break even” framing shows up across so many practitioner writeups, even though (and this is worth being honest about) nobody seems to have published an actual measured utilization number from a real production fleet. It’s a consensus built on shared intuition, not on data anyone’s shown. And the intuition lines up with what’s easy to observe anecdotally: most real workloads are bursty and business-hours-weighted, running more like 30-40% actual utilization rather than the 50%+ the math needs.

There’s also an asymmetry in how mistakes play out. If you over-provisioned API capacity, you just… stop calling the API next month. If you built a GPU cluster sized for volume that didn’t show up, you’re stuck depreciating hardware you can’t easily resell, for volume you don’t have. One direction of error is reversible. The other isn’t. That asymmetry alone should raise the bar for how confident you need to be in your volume forecast before you commit to owning hardware.

None of this shows up in a GPU-hours calculator. It shows up three months later, in a bill for hardware that spent most of its life waiting for a request.

The costs that don’t show up on the invoice either

Utilization is the big one, but it’s not the only thing missing from the spreadsheet. Running your own inference stack means someone owns:

Engineering time. Estimates from teams who’ve done this land around 10-20 hours a month of ops overhead once things are stable, and that’s before an incident. At a loaded engineer cost of $75-150/hour, that’s $750-3,000 a month of recurring labor an API bill doesn’t have.

Model churn. Open-weight releases move fast: DeepSeek went V3 to V3.1 to V3.2, Llama went 3.3 to 4, Qwen went 3 to 3.5. Each new release is a re-benchmarking and re-validation cycle if you want to actually capture the quality and efficiency gains. An API provider absorbs that upgrade cost invisibly; you just get better numbers on their pricing page.

No automatic failover. A single self-hosted cluster is a single point of failure. A 2am hardware fault is your outage. Providers who split traffic across multiple upstreams don’t have that problem by construction.

Several practitioner writeups converge on a rough multiplier here: take the sticker GPU rental price and multiply by 1.3-2x for a realistic total cost of ownership, or 3-5x once you’ve priced in redundancy and churn. I want to flag clearly that this multiplier is directional, not measured, the same way the utilization threshold is. Nobody in this space has published a rigorous, reproducible TCO model that holds across model sizes and deployment styles. If someone tells you they have the definitive number, ask to see the derivation.

A twist worth sitting with: the API markup might already be thin

One more thing surfaced in that Hacker News thread that I think gets missed in the “self-host to escape the API markup” argument: a commenter pointed out that DeepSeek’s own API pricing lands remarkably close to the independent, from-scratch depreciation math for self-hosting the same model. If that’s right, DeepSeek isn’t pocketing a huge margin on its own API. It’s pricing close to its real compute cost.

That matters because it means the “get around the middleman markup” argument has a lot less room to work with against the cheapest open-weight API tiers than people assume. It still holds against frontier-model pricing, where margins are structurally higher. But against DeepInfra, Fireworks, Together, and friends, who are all competing hard on price for the same open-weight models, the gap you’re trying to capture by self-hosting may already be mostly competed away.

There’s also a live, unresolved question hanging over all of this: is current rock-bottom open-weight API pricing sustainable, or is it a subsidized land-grab that rises later? Nobody has a confident answer as of July 2026. If it’s subsidized and prices climb, a break-even calculation done today gets more favorable to self-hosting over time. If it’s already near cost, as the DeepSeek observation suggests, it probably stays roughly where it is.

Meanwhile the GPU side of the equation is just as unsettled: some sources describe H100 pricing falling steadily as commodity supply grows, others describe a concurrent HBM memory shortage holding spot prices 3-5x higher in certain markets, expected to persist into 2027. One tracked data point: B200 spot pricing reportedly jumped 114% over six weeks earlier this year, driven by demand spikes around new model releases. A break-even number calculated in July could be stale by September, in either direction.

So when does self-hosting actually make sense?

Pulling this together honestly: self-hosting an open-weight model tends to pencil out when you have genuinely high and predictable sustained volume, when open-weight quality is good enough for your use case (a gap that’s closing, per GLM 5.2’s showing), when your team has real capacity to own ops and upgrade churn, and when your real comparison point is frontier-model pricing rather than the already-thin open-weight API tier.

It tends not to pencil out when your volume is bursty or still growing, when you’re implicitly comparing against the cheapest API tier rather than frontier pricing, or when you haven’t actually measured your own utilization and are hoping it’ll land above 50%.

The honest summary of “the real math” is less a threshold and more a question: do you actually know your utilization number? Most teams doing this calculation don’t, and that unknown matters more than whether H100s are $2/hour or $6/hour this month.

Where this connects to the routing problem

There’s a version of this whole debate that treats it as binary: own the hardware, or pay full price to an API. But the thing that actually kills the self-hosting math, the utilization risk of a single company’s dedicated cluster sitting idle outside peak hours, is a problem of scale, not a problem of hardware ownership per se. A shared pool of capacity, serving demand across many customers instead of one, doesn’t have the same idle-hours problem a single team’s cluster does, because someone else’s traffic fills the gaps.

That’s part of what AJNT’s routing layer is built around: a cost-ordered fallback ladder for every model request that can end on GPU capacity AJNT operates itself, provisioned and torn down by a margin-driven autoscaler that shops multiple GPU marketplaces against live demand. The utilization risk doesn’t disappear, it just moves from your balance sheet to a shared pool sized against aggregate demand instead of one team’s forecast, which is a fundamentally easier problem to manage well.

There’s a second, less obvious way the math above gets easier to move. A lot of what actually drives API spend for agent-style workloads isn’t the model price per token at all, it’s how much of the conversation history gets re-sent and re-processed on every single turn. Compressing that resent context before it crosses the wire shrinks the token bill on the API side of this comparison before you ever get to the self-hosting question. Nobody in the self-hosting research seems to have modeled that lever explicitly yet, probably because it’s a gateway-level fix rather than a hosting decision, but for any team running long, tool-heavy agent sessions, it’s worth working out before assuming the only two options are “own GPUs” or “pay the sticker API price.”