Local AI: When Does Running a Model In-House Make Sense?

Short answer: Local AI, meaning the model runs on your company's own servers, makes sense when three conditions are met together: keeping data inside rests on a written rule (law or a customer contract), the work is narrow and repetitive (classifying documents, extracting fields, summarising), and it is clear who will operate it (your own team or a partner). If either of the first two is missing, the business versions of cloud AI usually give better results at lower cost.

This article is for information only and is not legal advice. Review your own situation with your data protection officer or legal counsel.

---

What Does "Local" Mean? Three Options

When people say "our data must not leave", they often mean three different setups. The difference is where the data is processed:

This article is about the third option, also called an on-premise or local LLM. But you can only decide with all three side by side.

The two providers are in different positions:

• Claude: Anthropic does not offer Claude models for self-hosting on a company's own servers. So in an on-premise scenario, Claude is not a direct option. Regional control depends on the channel: on Anthropic's own API there are two options today (US only or global). On Amazon Bedrock and Google Cloud, the endpoint you use decides the region¹. On Amazon Bedrock, Claude is available in several European regions, including Frankfurt, Paris, Stockholm and Ireland².
• OpenAI released two open-weight models called gpt-oss in August 2025. They are licensed under Apache 2.0 and can be downloaded and run on your own hardware³. For its cloud products, OpenAI also offers data residency in Europe for eligible ChatGPT Enterprise, Edu and API customers⁴.

---

What Changes Under GDPR?

As we explain in our article on AI at work and GDPR, the key questions are the contract with the provider and transfers outside the EU. GDPR's rules on international transfers apply when personal data leaves the EU or EEA⁵.

For the three options, this means:

• Option 1: Data may be processed outside the EU. You need a data processing agreement (DPA) and a valid transfer mechanism, usually Standard Contractual Clauses.
• Option 2: Processing stays in the region you choose. This reduces transfer questions, but the provider is still a processor, so the DPA and the provider's list of sub-processors still matter.
• Option 3: Data stays on your servers, and no external AI provider processes it. GDPR's other obligations apply as in any setup; security of processing is first among them⁶.

---

When Does It Make Sense?

1. Data can't leave, and this is written down. If a customer contract clearly says data may not be given to third-party services, or sector rules require it, options 1 and 2 are off the table. The key word is "written". "Our data doesn't leave the building" can be a preference, and preferences can be reviewed.

2. The work is narrow and repetitive. For classifying incoming requests, extracting specific fields from documents or summarising standard texts, open-weight models are usually good enough. If you process hundreds of similar documents every day, the cost comparison with pay-per-use cloud services can also change.

3. Responsibility for operations is clear. A local model is not software you install and forget. Hardware, model updates, security patches, access rights and monitoring are ongoing work. The team doing this does not have to be in-house. For a single use case, hiring a specialist can cost more than giving setup and operations to a partner. What matters is that someone owns it from day one. A model without an owner starts going out of date the day it is installed.

---

When Doesn't It Make Sense?

1. The rule is really a habit. Does the "data must not leave" rule come from law, a contract or an unwritten preference? We cover this in our article on where to start with AI. Investing in a local model before the source is clear can mean paying for a constraint that doesn't exist.

2. Your data isn't classified yet. If it isn't defined which data is sensitive, "keep everything local" is the easiest answer but the most expensive one. The right order is to build a data inventory and decide which data can go into which system before you choose a model.

3. You need the best reasoning. Open-weight models are improving fast, but they are not at the level of the providers' latest cloud models. According to OpenAI's own announcement, gpt-oss-120b performs at the level of its o4-mini model on core reasoning benchmarks⁷. These are OpenAI's own test results; independent results can differ by task. For long contract analysis, multi-step planning or complex question answering, the gap shows in the results.

4. Volume is low or uneven. gpt-oss-120b is designed to fit on a single 80 GB data centre GPU (for example an NVIDIA H100). The smaller gpt-oss-20b can run with 16 GB of memory³. A practical rule: with low and uneven volume, pay-per-use cloud is usually better. With high, predictable and repetitive volume, the fixed cost of a local setup can pay off over time.

---

5 Questions Before You Decide

1. Where is the rule that stops data from leaving written down?
2. Is the work narrow and repetitive, or does it need broad reasoning?
3. How many times a month is this work done?
4. Who will operate the model, updates and security: your own team or a partner?
5. Is it defined which data can go into which system?

Until the first two are answered, talking about hardware is too early. Model choice comes last: first the data inventory and classification, then where AI actually pays off. Our AI Discovery service starts with this question: together, we find which process AI creates value in, and which setup is right for that process, without your company data leaving your control.

---

Sources

Numbers match the markers in the text. Provider regions and products change often; information as of 13 October 2026.

1. Anthropic, Data residency. On the Claude API, where inference runs is set with "inference_geo", and today there are only two values: "global" (default) and "us" (US only). On Amazon Bedrock and Google Cloud, the region is determined by the endpoint or inference profile. ↩
2. AWS, Amazon Bedrock model availability by region. Claude models are available from Frankfurt, Zurich, Stockholm, Milan, Spain, Ireland, London and Paris (checked 13 October 2026). ↩
3. OpenAI, gpt-oss-120b model card (Hugging Face). License: Apache 2.0. gpt-oss-120b is designed to "fit into a single 80GB GPU (like NVIDIA H100 or AMD MI300X)"; gpt-oss-20b to "run within 16GB of memory". According to the model card, 120b is "for production, general purpose, high reasoning use cases", 20b "for lower latency, and local or specialized use cases". ↩ ↩
4. OpenAI, Expanding data residency access to business customers worldwide. Data residency in Europe and other regions for eligible ChatGPT Enterprise, Edu and API Platform customers. ↩
5. GDPR, Chapter V (Articles 44–49) on transfers of personal data to third countries; Article 28 on processors and the required contract. ↩
6. GDPR, Article 32: the controller and the processor shall implement appropriate technical and organisational measures to ensure a level of security appropriate to the risk. ↩
7. OpenAI's gpt-oss announcement (5 August 2025), as reported by InfoQ: gpt-oss-120b matches or exceeds o4-mini on core reasoning benchmarks; gpt-oss-20b performs at the level of o3-mini. These are OpenAI's own results; independent measurements can vary by task. ↩