All Playbooks
Field Manual7 min readUpdated November 8, 2026

Should Your Business Run AI Locally?

Open models are now good enough to take seriously, but local AI rarely wins on cost alone. Here is when running it yourself makes sense, what it costs and the security risks that come with it.

Published 8 November 2026. Prices and licences checked on this date. Both change quickly, and hardware prices especially so.

Running AI on your own hardware has an obvious appeal: no per-token bills, no data leaving the building and no dependence on someone else’s roadmap. The second of those can be a strong reason. The first usually isn’t.

For most small businesses, a hosted model will be cheaper and simpler. Local AI earns its place when control over the data matters more than cost, or when a large, steady workload makes the hardware pay for itself.

What has changed

Licensing is much less of an obstacle. Google released Gemma 4 under Apache 2.0, with native function calling. Google’s announcement Alibaba’s Qwen3.6-35B-A3B and Qwen3.8-27B, and Mistral’s Small 4 and Devstral Small 2, also list Apache 2.0, while GLM-5.2 lists MIT. These permissive licences suit commercial use, but check the exact model you plan to deploy, because terms vary across the field.

The quality gap has narrowed too, though how much depends on the measure. Stanford’s 2026 AI Index put the top closed model 3.3% ahead of the top open model on Arena scores, up from 0.5% in August 2024. The AI Index Epoch AI estimated a lag of about three months on its general index in late 2025, and six to seven months for cyber capabilities in October 2026. Epoch’s general estimate These numbers measure different things, so they can’t be combined into one figure.

There’s also a catch: the open models at the top of those charts are often not the ones you can run on a desk.

What actually fits on what

  • A good GPU or a well-equipped Mac: Mistral says Devstral Small 2, at 24 billion parameters, can run on a single RTX 4090 or a Mac with 32 GB of memory. Mistral’s model card
  • A dedicated AI desktop: NVIDIA rates DGX Spark for models of up to 100 billion parameters with 64 GB of memory, and up to 200 billion with 128 GB. NVIDIA’s specifications
  • Server territory: GLM-5.2 has 753 billion parameters and is documented for serving frameworks rather than office machines. The GLM-5.2 card
  • Small devices: Gemma 4 ranges from small E2B and E4B models to a 31B version, so simple tasks may not need new hardware at all. The Gemma 4 model card

Fitting in memory is not the same as being fast. One published analysis assumes about 28 tokens per second on a single stream from its Mac cluster, which is fine for one person and limiting for a team. DecodeTheFuture’s assumptions

Hardware has also become more expensive. In the UK, Mac Studio starts at £2,499 with M5 Max and £5,499 with M5 Ultra. Apple’s UK store NVIDIA’s 128 GB DGX Spark rose to $6,950 in October, nearly 75% more than a year earlier, with memory prices blamed. The 64 GB version is expected to retail at around $4,999. The Register’s report

The cost maths most articles get wrong

The usual comparison sets a local model against an expensive frontier API. That flatters local AI twice: the frontier model may be doing better work, and it’s not your only alternative. The fairer comparison is the same open model, hosted by someone else.

As listed on OpenRouter today, several providers offer Qwen3.6-35B-A3B at around $0.10 per million input tokens and $0.90 per million output tokens. Current provider prices Anthropic lists Sonnet 5.5 at $2 and $10, and Opus 5.5 at $4 and $20. Anthropic’s pricing

Now do the sums. At those rates, 10 million output tokens a month from the hosted open model costs about $9. One analysis estimates that a used three-GPU rig costs about $10 per million tokens at that volume, once the hardware is spread over three years, and only beats hosted open models above roughly a billion tokens a month. Capital & Compute’s estimate

Another compared a four-Mac cluster with APIs on one heavy workload: 300 million input and 20 million output tokens a month. The cluster came to about $1,425 a month, including four hours of administration, against about $199 for the same open model through its API, $475 for Sonnet 5.5 and $907 for Opus 5.5. The full comparison

The picture does change against pricier models. Digital Applied, a consultancy that sells this kind of advice, found that a $9,000 workstation could break even against an earlier premium model at about 26% daily use, but never against commodity APIs near $0.88. Its break-even model All of these rest on the authors’ assumptions, so run the numbers with your own tokens, power costs and staff time.

Privacy is the real reason, with limits

If prompts contain client files, health records or unreleased financials, keeping them on your own machines removes a third party from the chain. Check that this is what’s actually happening, though. Ollama says it doesn’t see your prompts when models run locally, but it also offers cloud models whose prompts it processes on its servers. Setting OLLAMA_NO_CLOUD=1 turns those off. Ollama’s documentation

Local doesn’t remove your legal duties either. If you process personal data, you still need a lawful basis, security and a way to honour people’s rights. The UK regulator’s AI guidance covers these areas and is under review after the Data (Use and Access) Act. The ICO’s guidance In the EU, data protection authorities say a model trained on personal data can’t always be treated as anonymous, and a company deploying one may need to assess how it was developed. EDPB Opinion 28/2024 This is general information, not legal advice.

Hosted doesn’t have to mean careless, but read the routing details. When we checked, OpenRouter listed Qwen3.6-35B-A3B as unavailable through its EU in-region routing, which matters if your data must stay in the EU. The provider page

Local can be less secure, not more

Moving a model in-house makes its security your job, and exposed servers show how that can go wrong.

Ollama binds to 127.0.0.1 by default, so it starts out reachable only from the machine itself. Exposing it is a deliberate configuration change. Ollama’s documentation Plenty of servers are exposed anyway. Intruder found 1,652 Ollama APIs answering without authentication, out of more than 5,200 it identified. Intruder’s research LeakIX counted 12,269 exposed instances in February, and Cisco found 1,139 in 2025. LeakIX Cisco The counts differ because the methods do, but the pattern is consistent.

The stakes rose in May, when Cyera disclosed a critical flaw it called Bleeding Llama. It could let an unauthenticated attacker leak Ollama’s process memory, including prompts, system prompts and environment variables. Cyera’s disclosure Versions before 0.17.1 are affected. The CVE record

If you do run a local model on a network, keep it on a private network, put authentication in front of it, patch it like any other server and log who uses it.

When local is the right call

  • The data can’t leave. A contract, regulator or client requires it, and you’ve checked that hosted options with the right terms won’t do.
  • The connection can’t be relied on. Think factories, ships, field work or remote sites.
  • Volume is very high and steady. Classification or extraction running around the clock, where a smaller model is good enough.
  • Response time matters at the edge. The model must run on or near the device itself.

It’s probably the wrong call if your usage is light or bursty, if the work needs the strongest available reasoning, or if nobody will own patching and monitoring.

Many businesses will end up with a mix, sending each task to the cheapest option that’s allowed to handle it.

Illustrative only: this is a hypothetical routing policy, not a recommendation for any specific business or a real product’s configuration.

{
  "default_route": "hosted_open_model",
  "routes": [
    {
      "when": "contains_client_confidential_data",
      "use": "local_model",
      "cloud_fallback": false
    },
    {
      "when": "high_volume_classification",
      "use": "local_model",
      "cloud_fallback": true
    },
    {
      "when": "complex_reasoning_no_personal_data",
      "use": "frontier_api"
    }
  ],
  "log": [
    "route",
    "model",
    "tokens",
    "human_correction"
  ]
}

Notice cloud_fallback: false on the confidential route. If the local model fails, that work should stop and alert someone, not quietly go to the cloud. Rules like this belong in the agent’s harness, enforced by software rather than left to the model’s judgment.

Test it before you buy anything

  • Collect real tasks. Gather 50 to 100 examples of the actual job, with sensitive details removed.
  • Try the open model hosted first. This tests the model’s quality with no hardware purchase.
  • Compare it with what you use now. Measure how often the output is accepted, how much correction it needs, and the tokens and time used.
  • Try it locally. Use a machine you already own, with a runner such as Ollama or LM Studio. LM Studio’s terms cover internal business use, but check them against your situation.
  • Then cost the hardware. Include administration, power and security, and compare it with the hosted price at your measured volume.

If the numbers don’t justify the purchase, you’ve spent a few hours instead of a few thousand pounds. If they do, you’ll be buying on evidence.

When a regulated client genuinely needs on-premise AI, it’s usually a custom project rather than a box you can buy. If you’re weighing that up, tell us about the workload.