Self Hosted AI: Why Firms Are Pulling Models Back In House

For two years the default answer to almost any corporate AI question was an API key. Sign up, send your prompts to a data centre somewhere in Virginia, pay per token, move on. That reflex is starting to break down. Self hosted AI, meaning models that run on hardware a company actually controls, has moved from hobbyist curiosity to a real line item in serious IT budgets, and the reasons have very little to do with ideology.

The bill arrived

Token pricing looks trivial during a pilot. A team builds a document summariser, spends forty dollars in a month, and everyone agrees the economics are wonderful. Then the summariser gets wired into a workflow that chews through eleven thousand contracts a week, and the invoice comes back with an extra digit. Inference costs scale almost linearly with usage, which surprises finance teams accustomed to software licences that flatten out once you buy enough seats. A mid-sized company running a handful of always-on AI workflows can reach the point where a rack of GPUs pays for itself inside eighteen months.

Hardware has met them halfway. Consumer graphics cards with twenty four gigabytes of memory now run models that would have demanded a small cluster in 2023, and quantisation has cut memory requirements again on top of that. Plenty of engineers now run a capable model locally on a single workstation as a matter of routine. The underlying architecture, and why parameter count alone stopped predicting quality, is covered clearly in the Wikipedia entry on the large language model.

Data that cannot leave the building

Cost gets the meeting started. Compliance closes it. Hospitals, law firms, defence contractors and anyone processing European personal data face a question that no vendor assurance fully answers: where did that text physically go, and who could have read it. A self hosted model turns that question into a network diagram instead of a contract negotiation. For regulated industries this is often the only version of the argument that matters, and it explains why ai for business automation in legal departments has skewed towards on-premise deployments faster than in marketing or sales.

Governance frameworks have caught up enough to make this defensible rather than merely cautious. The voluntary risk framework published by the US National Institute of Standards and Technology gives teams a vocabulary for documenting what a model does, what it was trained on and how failures get caught. The NIST AI Risk Management Framework is not law anywhere, but auditors have started asking for it by name, and mapping a self hosted stack against it is considerably easier than mapping a black box you rent.

Firms pulling models back in-house are making a bet about cost, control and data that will look either prescient or expensive within two years. Either way it is one of the clearer tech trends to track, because infrastructure decisions reveal what companies actually believe rather than what they say. Capital expenditure is a form of confession.

Open models closed the quality gap

The uncomfortable fact for closed vendors is that the best open source llm releases of the past year land close enough to frontier commercial models for most business tasks. Summarisation, extraction, classification, routing, drafting: these are not tasks that need the absolute cutting edge. They need consistency, latency you control and a model that will still exist in the same form next quarter. Anyone who has had a production prompt quietly break because a hosted model was updated overnight understands why version pinning has become a selling point rather than a technicality.

Fine tuning is where the money actually is

Running someone else's weights on your own metal is a cost decision. Adapting those weights to your domain is a competitive one. Supervised fine tuning on a few thousand well-chosen examples reliably outperforms a much larger general model on narrow tasks, and it bakes your terminology, formatting and house style into the model rather than into an ever-growing system prompt. The practical mechanics of building that training set, and the point at which fine tuning beats prompt engineering, are laid out well in PoliLingua's discussion of domain specific supervised fine tuning. Companies with a decade of internal documents are sitting on a training corpus their competitors cannot buy.

The costs that stay off the slide deck

None of this is free. Someone has to patch the inference server, monitor GPU temperatures, handle the driver update that breaks everything on a Friday, and explain to the board why the model got slower after a library upgrade. That is a real job, usually a real hire, and it rarely appears in the spreadsheet that justified the hardware. Capacity planning is harder too, because a self hosted cluster cannot absorb a sudden tenfold spike the way a hyperscaler can. Most mature deployments end up hybrid: sensitive and high-volume work stays in house, overflow and experimental work goes to an API.

Who should stay on the API

If your AI usage is spiky, exploratory, or measured in hundreds of requests a day rather than millions, self hosting is a distraction. The same is true if you have no one on staff who enjoys Linux. The honest test is whether you can name the specific workflow, the specific volume and the specific data-residency requirement driving the decision. Teams that can answer all three tend to be glad they moved. Teams chasing a headline about open weights tend to end up with an expensive space heater in a cupboard.