A private LLM can cost three to five times more than the GPU bill on your spreadsheet suggests. That gap is where most teams get burned. As of May 27, 2026, deciding whether to move off OpenAI or Anthropic and run your own model has become less about model quality and more about workload economics, data control, and how good your platform team actually is. This piece lays out when private LLM deployment pays off, when it doesn’t, and the hybrid pattern that’s quietly winning in production.
Quick answer. Self-host when your data can’t legally leave your boundary, your token volume is steady and large (often tens of millions per month or more), or compliance rules out third-party processing. Keep the API for hard reasoning, long context, and spiky traffic. For most teams, the right move is a routing layer that splits work between both.
Four ways to deploy a private LLM, not just two
The cloud-versus-on-prem framing misses the real choices. There are four patterns, and they sit on a spectrum of control.
- Public API access (OpenAI, Anthropic): fastest setup, weakest data control. Prompts and outputs traverse provider infrastructure even with enterprise no-training agreements.
- Managed private cloud (Azure OpenAI, AWS Bedrock): traffic stays inside your VPC or VNet via PrivateLink, with IAM, customer-managed keys, and audit logs.
- Self-hosted in your own cloud account or data centre: full control of weights, serving stack, and logs. Full operational burden too.
- Air-gapped on-prem: nothing leaves the building. Reserved for ITAR, classified, SCIF, and similar work.
Most “should we go private?” debates collapse the middle two options into one. They shouldn’t. The jump from public API to managed private cloud is mostly a contract and networking exercise. The jump to self-hosted is an entirely different commitment.
When self-hosting actually pays off
Cost is where the spreadsheet lies. Multiple analyses across the research material converge on the same finding: real total cost of ownership for a self-hosted model runs three to five times the bare GPU rental rate once you fold in engineering, DevOps, observability, patching, model refresh, and depreciation.
The break-even point is workload-specific. Against premium APIs (GPT-4o, Claude-class), some enterprise analyses put break-even around 5 to 10 million tokens per month, or roughly $20K per month in API spend. Against budget APIs, that pushes out to 50 million tokens per month or higher. For pure cost advantage with no compliance pressure, one analysis puts the bar past 10 billion tokens per month.
GPU usage is the variable that wrecks naive math. A cloud GPU running at 10% load costs roughly 10x more per token than one kept busy. Below about 30% load, serverless GPU inference tends to be cheaper than dedicated instances. Above that, dedicated wins.
Serverless has its own trap, though. Cold starts and idle retention eat billable time. One real example in the research showed 113 minutes of billed time for 8.2 minutes of actual execution on a 25B model. That’s the kind of detail that doesn’t appear in vendor decks and absolutely matters when you cost out a private LLM deployment.
Does on-premise LLM hosting actually make it faster?
Sometimes. Keeping inference close to the application cuts network latency and removes queueing at a third-party provider. That matters for agent loops, real-time document workflows, and anything running on a LAN. But a private model is only fast if the serving stack is tuned. Projects like vLLM, LMCache, and the vLLM production stack bring prefix-aware routing, KV-cache sharing, autoscaling, and fault tolerance into the picture, and the difference between a default deployment and a tuned one is large.
Here’s the part the research makes hard to ignore: routing beats replacement. The vLLM Semantic Router reports a 10.2 point accuracy improvement, 47.1% lower latency, and 48.5% lower token consumption versus direct inference on its benchmark. The win comes from sending easy prompts to a small fast model and reserving the heavyweight model for prompts that need it.
“80.7% of real-world LLM inference workloads can be shifted to small local models, with routers achieving 77.1% energy, 67.1% compute, and 60.2% cost savings compared with cloud-only deployments.”
TrafficBench, OpenReview, 2025/2026
If most of your traffic is already shaped like classification, extraction, or summarisation, you don’t need to host a frontier-class model to get a latency and cost win. You need a small local model and a router that knows when to use it.
Compliance and data sovereignty: the non-negotiable case
For some industries, the decision isn’t about cost. It’s about whether your data can legally touch a third-party model at all.
CMMC Level 2 requires the 110 controls in NIST SP 800-171, covering audit and accountability, access control, identification and authentication, incident response, and system protection. AI systems that process Controlled Unclassified Information have to meet the same controls a human user would, including authenticated access, FIPS encryption, and tamper-evident audit trails. Self-attestation isn’t enough under CMMC final-rule processes. Third-party assessment is required.
Healthcare and pharma sit in similar territory. Architecture guidance for those sectors favours on-prem or VPC-isolated deployments because PHI exposure has to be tightly controlled and auditable. Retrieval-augmented generation patterns help by limiting raw PHI exposure to the model itself.
One nuance worth stating clearly: compliance does not automatically mean fully self-hosted. Azure OpenAI Government and AWS Bedrock GovCloud already carry a lot of regulated workloads. Air-gapping is for the highest sensitivity cases, classified, SCIF, ITAR. If your compliance team can write a credible exception for a regulated managed service, that’s almost always cheaper and faster than building an air gap.
When should you stop using public APIs?
Concrete decision rule. Stop using public APIs when one or more of these are true:
- Your data cannot leave your boundary: PHI, CUI, ITAR, classified, trade secret, or data residency constraints a public API contract can’t satisfy.
- You need controls a public API can’t offer: customer-managed keys, SIEM-integrated logs, change control, third-party audit, private endpoints.
- Your token volume is steady and large enough to amortise infrastructure, typically tens of millions per month or more.
- Local execution gives a measurable latency win your application can actually exploit.
- You can staff a real LLMOps function: autoscaling, observability, safety monitoring, evaluation, prompt and model versioning, fallback routing.
Keep using public APIs when:
- Volume is low or spiky.
- You don’t have specialist platform engineers.
- Your workload is mostly frontier reasoning or long-context analysis.
- You need access to the newest model the week it ships.
- A regulated managed service already meets your compliance bar.
That second list catches more organisations than the first. If you’re a 40-person company without a dedicated MLOps team, the API is almost certainly cheaper than any private LLM you’d build, and the answer doesn’t change just because someone in the room is nervous about privacy.
The routing layer is the real strategic asset
Here’s the part most “private LLM vs API” articles miss. The organisations doing this well in 2026 aren’t choosing between API and self-hosted. They’re building a routing control plane that spans both.
The vLLM Semantic Router has matured into a production pattern. It scores each prompt against cost, privacy, and capability boundaries, with built-in jailbreak and sensitive-data leakage detection. It runs alongside the vLLM production stack and can sit in an Envoy external processor. Red Hat has folded similar routing patterns into its open-source AI work, and the underlying research on multi-signal routing is now on arXiv.
A workable hybrid architecture looks like this:
- Small local model for classification, extraction, summarisation, and routine support.
- Mid-sized private model for sensitive internal reasoning that has to stay in your boundary.
- Frontier API for hard reasoning, long-context analysis, and tasks that need live external data.
- Policy and routing layer that evaluates each prompt’s sensitivity, complexity, and urgency before dispatch.
This is the architecture I’d point any enterprise toward when they ask the original question. It keeps frontier quality where you need it, holds sensitive traffic inside your boundary, and lets you swap model providers without rewriting application code. The lock-in risk drops too, which matters more as model pricing keeps shifting.
How public API, private cloud, and self-hosted LLM options compare
| Criterion | Public API | Managed private cloud | Self-hosted / on-prem |
|---|---|---|---|
| Data control | Medium | High | Highest |
| Frontier model access | Fastest | Strong, region-dependent | Slower, often delayed |
| Cost at low volume | Best | Often acceptable | Usually worst |
| Cost at high steady volume | Becomes expensive | Better for some | Best if GPUs stay busy |
| Compliance fit | Lower-risk workloads | Most regulated workloads | Classified, ITAR, air-gap |
| Engineering overhead | Lowest | Moderate | Highest |
| Best for | Pilots, broad productivity | Regulated enterprises | Sovereign, very high volume |
The table reads cleanly, but in real engagements the answer is rarely “one column wins everything.” It’s usually “column 1 for these workloads, column 2 for those, column 3 for the rest.”
How to decide for your own workload
Don’t start with the model. Start with the traffic.
Run a week of logging through whatever you’re using now and tag each request by sensitivity, complexity, and volume. You’ll usually find that 70 to 80 percent of calls are extraction, classification, or summarisation work a small local model could handle. Maybe 10 percent needs genuine reasoning. A small slice is genuinely sensitive.
Then put a gateway in front and split the traffic. Keep the frontier API where it earns its place. Move the rest privately as the volume and compliance case justifies. The competitive edge from here isn’t owning the GPUs. It’s knowing which prompt goes where.
