On-premise AI, EU cloud or hyperscaler: how much sovereignty your data really needs
On 10 June 2025, Microsoft France's director of public and legal affairs sat before a French Senate inquiry and was asked, under oath, whether he could guarantee that French citizens' data would never reach US authorities without France's consent. He said he couldn't. He added that it had never happened.
Read that, and the obvious next question is whether your own AI now has to run on-premise. My answer as a CTO, for most companies with 50–500 employees, is no. On-premise AI pays off for one or two classes of data, rarely for everything. If you insist on processing every prompt in your own server room, you buy yourself a patching problem and a permanent claim on staff time that never shows up in the hardware quote. And you end up protecting website copy that's going to be published anyway.
Pushback comes from two sides. One camp thinks a hyperscaler's EU region plus the Data Privacy Framework is enough. The other thinks only your own server counts as sovereign. Both mistake where a data center stands for who controls the data.
This is a technical assessment based on our understanding of the industry, as of October 2026, and not legal advice.
What the CLOUD Act actually says
The CLOUD Act of March 2018 added one short section to US law, 18 U.S.C. § 2713. Providers of communication and cloud services must hand over data in their possession, custody or control, whether that data sits inside or outside the United States. The operative word is control. A data center in Frankfurt changes nothing as long as a US company controls how it's run.
Then there's Section 702 of FISA, which allows surveillance of non-US persons outside the US through US providers. Its statutory authority lapsed in June 2026 after Congress failed to agree on an extension. The Brennan Center notes that surveillance continues under the existing certifications until March 2027 regardless. What happens after that is open.
Transfers of personal data to the US have relied on the EU-US Data Privacy Framework since July 2023. The EU General Court dismissed French MP Philippe Latombe's challenge on 3 September 2025 (T-553/23). Latombe appealed, and case C-703/25 P is now before the Court of Justice. The Court struck down both predecessors, Safe Harbor in 2015 and Privacy Shield in 2020. I wouldn't build an architecture that only holds as long as the third agreement does.
This assessment of the Data Privacy Framework reflects our understanding of the industry as of October 2026 and is not legal advice. Whether a specific transfer to the US is permissible is a question for your data protection adviser.
In fairness, the Senate testimony had a second half. According to Microsoft, no such request had affected a European company in the previous three years. The committee chair pointed out that the transparency reports behind that claim aren't externally audited. So the risk is real, but small. Small isn't zero. For a newsletter draft it doesn't matter. For the notes from a performance review it does.
Four options and what each one protects you from
A US hyperscaler's EU region, meaning an AI service from Microsoft, Google or AWS running in a European data center, meets data residency requirements and keeps latency low. It doesn't protect you from § 2713. In return you get the strongest proprietary models, security patches you never have to apply yourself, and certifications no SME could afford on its own.
EU-sovereign clouds come in two flavors. European providers with no US parent host open-weight or in-house models, and the US companies build their own structures. AWS launched its European Sovereign Cloud in Brandenburg in January 2026, with a new European parent company, three German GmbHs, and operations staffed exclusively by EU residents. As far as I know, nobody has yet tested whether a US court would treat a structure like that as outside the group's control. With a provider that has no US entity at all, the question doesn't arise.
A dedicated server at a European hosting company, running an open-weight model you operate yourself, moves control to you. For SMEs, hosting for knowledge preservation should follow the sensitivity of the conversations and the team's ability to maintain the environment. Root access, model version and deletion are yours. The host provides power, network and hardware.
On-premise means the hardware sits in your building. It's the only option that can run without an internet connection, and the only one where nobody but you has physical access. It's also the option where every vulnerability in the operating system, the GPU driver and the inference server is your problem, on a Friday night just as much as during the vacation of the one person who understands the system. Here, sovereignty is mostly work.
One technical detail tends to get lost in sales calls. Customer-managed keys protect data at rest. During processing, the model has to see the plaintext, and that happens on the hardware of whoever runs the service.
Which data class justifies which option
That's why I don't decide per tool. I decide per data class, and five classes cover most companies. The mapping below is my working rule.
- Public content, or content meant for publication, such as website copy and job ads, can go into any of the four options, usually most cheaply into the hyperscaler region.
- Internal content without personal data, such as manuals and process descriptions, belongs in the hyperscaler region with a data processing agreement and training contractually excluded, or better still in an EU-sovereign cloud.
- Personal customer and project data, such as CRM notes and meeting minutes with names, belongs in an EU-sovereign cloud or on a dedicated server, and in a hyperscaler region only with standard contractual clauses and a documented risk assessment.
- Employee data and special categories under Art. 9 GDPR, such as appraisal conversations, job applications and health information, belong on a dedicated server with an EU provider or on-premise.
- Trade secrets the business depends on, export-controlled data and critical-infrastructure data belong on-premise, if in doubt with no outside connection at all.
Test this on your own data. Ask three people from different departments to assign their last ten AI prompts to a class. Every prompt that lands in class four or five points to a use case that needs a separate environment. You don't need one for the rest.
Meetings are where the sensitive classes appear most easily, because the same conversation covers a quote for a client and a colleague's sick leave. What that means for AI note-takers is covered in the post on AI note-takers and the GDPR, and the employment-law side in the post on documenting employee conversations. We had to make this call ourselves for RECO, and since it's our product, please read this with appropriate skepticism. Conversation documentation generally falls into class three or four, so we process on EU servers with a dedicated server in Germany and offer an on-premise version for class five.
What on-premise AI really costs to run
Most calculations online compare hardware with API prices. What they leave out is staff time. I'm deliberately not giving you sample prices here, since they'd be out of date within a year. Instead, here's the scheme I would use. The inputs come from your own quotes and your payroll.
Start with model size, because it determines the hardware. The GPU memory needed for the weights is the parameter count times the bytes per parameter. At 4-bit quantization that's half a byte, so a model in the 70-billion-parameter class needs about 35 GB for the weights alone. On top of that you need headroom for the context of every concurrent request. Only with that number can you compare GPU quotes meaningfully.
Then add up these items per option and per month:
- Hardware: purchase price from the quote divided by the months of depreciation, or the monthly rent for a rented server.
- Power and cooling: average power draw in kW times operating hours per month times your price per kWh. Measure the average in a test run; the data sheet gives the peak load.
- Backup, monitoring and spare parts, as quoted by your service provider.
- Staff time: the share of a position needed for drivers, the inference server, model updates, permissions and same-day response to critical vulnerabilities, times the annual employer cost of that position, divided by twelve.
- For the hyperscaler: list price per user from the current price list times the number of users, plus admin time.
With a rented server, the hardware upkeep goes away but the software upkeep doesn't. Set the staff share lower than for on-premise, but never at zero. You buy the hardware once and depreciate it, while staff time comes due every month for as long as the system runs. That's why it can swing the whole calculation.
And a share of a position isn't a human being. If that one person is out, your AI for class four stops until someone else is up to speed. If you need on-premise, get a maintenance contract with a service provider and a backup person in place before the first model goes live.
On quality, my assessment is that open-weight models of this size are good enough for summarizing, classifying and searching your own documents. For long, multi-step reasoning, the big proprietary models are ahead.
The exit is what people forget
To me, sovereignty also means being able to leave. Before you sign, ask whether you get your data, prompt templates and search indexes back in an open format. Ask whether the model your workflows depend on today will still be offered beyond the term of your contract. Proprietary model versions get retired, while the weights of an open-weight model stay with you.
Open-weight doesn't automatically mean open source, though. Meta's Llama models come under their own license with conditions, while some Mistral models are released under Apache 2.0. Read the license before a model ends up in a customer-facing process. I described the general data check for new tools in the post on AI tools for the second half of 2026. If you plan to let agents loose on your own systems, settle the hosting question before the pilot. The post on AI agents in mid-sized companies explains why.
Where I would start
With a list, not a hardware quote. One line per AI use case, and next to it the highest data class that appears in it. For most companies of this size, I'd expect two pillars, a widely used cloud service for classes one and two and a small, controlled environment for HR and conversation data. One server for everything is rarely the right answer, and neither are ten tools with no classification at all.
Sources
- 18 U.S.C. § 2713, added by the CLOUD Act (Pub. L. 115-141, 23 March 2018), Cornell LII
- French Senate, inquiry committee on public procurement, Microsoft hearing of 10 June 2025 (minutes)
- Appeal Latombe v Commission, Case C-703/25 P, Official Journal of the EU C/2025/6610
- WilmerHale: European Court of Justice to Review Challenge to EU-U.S. Data Privacy Framework (1 December 2025)
- Brennan Center for Justice: Section 702 of FISA, 2026 Resource Page
- EFF: Victory! 702 has Expired! (12 June 2026)
- Amazon: AWS launches AWS European Sovereign Cloud (press release, January 2026)
Frequently asked questions
Can Microsoft Copilot with the EU Data Boundary be used in line with the GDPR?+
For many use cases, yes, provided there is a data processing agreement, the transfer is covered by the Data Privacy Framework or standard contractual clauses, and you have documented how the tool is used. The EU Data Boundary governs where data is stored and processed. It doesn't change the fact that a US company controls the service, so § 2713 still applies. For data classes one to three I consider that acceptable. For employee data I would run a data protection impact assessment and lean towards a different environment. Not legal advice.
What hardware do I need to self-host an LLM?+
As a rule of thumb, multiply the parameter count by the bytes per parameter. An 8-billion-parameter model needs about 4 GB for its weights at 4-bit quantization, a 70-billion-parameter model about 35 GB. On top of that comes memory for the context of every running request, and that grows quickly with long documents and many concurrent users. Before you buy, rent a GPU server for a trial period and measure, with your real documents, at how many parallel requests the response time starts to climb.
Is a European cloud provider with a US subsidiary covered by the CLOUD Act?+
Section 2713 applies to providers subject to US law and to data under their control. As I understand it, what matters is whether the US entity actually has access to your data or can legally compel it, and that question isn't fully settled. Group structure and admin rights are what count; where the data center stands makes no difference. Ask the provider to confirm in writing which entities have administrator access and whether it has a US subsidiary or branch. Not legal advice.
Related articles
In a 30 minute software demo we show you how RECO secures and transfers knowledge in your company. Request a demo now and we'll get back to you with a proposed time.
