Self-hosted LLM vs API: sovereignty and honest costs
The self-hosted LLM vs API question arrives at our door dressed as a technical decision, and it almost never is one. The short answer: most companies should use commercial APIs, and the minority who genuinely need self-hosting can point to the exact regulation, contract clause or data classification that forces it. If nobody in the room can name that document, the GPUs you are about to buy are an expensive way to feel sovereign.
That answer deserves its reasoning, because both camps in this debate routinely lie to themselves. The API camp underestimates how real the constraints of regulated data are. The self-hosting camp underestimates everything else.
What self-hosting actually buys
Run an open-weight model on your own infrastructure and some real things become true. Your data never leaves machines you control, which turns certain compliance conversations from negotiation into demonstration. Residency stops being a promise in a provider’s terms and becomes a fact of your network diagram. Nothing gets deprecated on someone else’s schedule: the model you validated in January behaves identically in December. And at very high, steady volume, the marginal cost of a request approaches your electricity bill rather than a per-token price.
Open-weight models are also genuinely good now. For classification, extraction, summarization and retrieval-backed answering, the gap to the frontier is irrelevant for many workloads. The capability argument against self-hosting is weaker than it was two years ago; the operational argument is not.
What it costs beyond the hardware
The GPU invoice is the visible cost. The invisible ones decide the outcome. Someone must run inference serving, scaling, patching, model upgrades and security, which is a specialty, not a side duty for whoever runs your Kubernetes cluster, and people who do it well are scarce and priced accordingly. Utilization is the quiet killer: an API provider pools demand across thousands of customers, while your GPUs sit idle at night and depreciate around the clock. Most mid-size workloads are bursty, which is precisely the shape that makes owned hardware uneconomical.
Then there is drift: API prices keep falling and hosted models keep improving with no effort on your part, so a self-hosting business case has to beat a competitor that gets better every quarter for free. And for the hardest tasks, the frontier gap still exists. Teams that self-host everything usually end up quietly routing their most difficult work back to an API, which is an architecture worth designing on purpose rather than arriving at in embarrassment. Model choice per task is its own decision, one we argue matters less than teams fear precisely because switching should stay cheap.
Who genuinely needs it
Real cases exist, and we build for them. Health data under hosting regimes like the French HDS certification, where the list of acceptable hosts is short and audited. Defense and public-sector workloads with formal clearance or air-gap requirements. Client contracts that flatly forbid data leaving named infrastructure. Data whose cross-border movement is legally constrained. If you operate under one of these, self-hosting part of your stack is not ideology, it is the entry price, and the constraint belongs in the architecture from the first design meeting, not as a retrofit.
Even then, scope it. The mandate covers specific data in specific workflows. Route those to the self-hosted model and let everything else, the marketing drafts and the meeting summaries, use an API. Blanket self-hosting because ten percent of your data is sensitive is paying the full toll on every road.
Who just likes the idea
The other motivation we meet is sovereignty as identity: a discomfort with dependence on American providers, felt sincerely and priced rarely. The discomfort deserves respect, and it also deserves to meet the middle options before it meets a purchase order. Major providers now offer EU-region processing with contractual data-use commitments. European providers such as Mistral offer strong models under EU jurisdiction, via API or deployed privately. Open-weight models can run in your own cloud tenancy, giving most of the control without racking servers. For the large majority of companies whose actual requirement is GDPR compliance plus client reassurance, one of these satisfies it. GDPR, on its own, does not require self-hosting anything.
The self-hosted LLM vs API decision, in one meeting
Write down the binding requirement with legal in the room, with a citation to the regulation or contract that creates it. Classify your AI workloads against it. Route the constrained minority to self-hosted or EU-jurisdiction options, the rest to APIs through an abstraction layer that keeps the routing changeable. Revisit yearly, because both the models and your obligations move. That is the whole method, and it fits on one page.
We have deployed on every point of this spectrum, from API-only builds to healthcare systems on certified French infrastructure, so we hold no stake in either camp’s mythology. The meeting above works best with the actual contracts and workloads on the table; hold it before the purchase order, not after.