What is the core risk described in AI Vendor Risk: Are We Building a Death Star to Blow Up Our Own Planet?

The article describes a scenario where enterprises adopting commercial AI models diligently send prompts, documents, and workflows to vendors, effectively handing over proprietary thinking at scale. No single decision is wrong, but collectively this builds a structural flaw analogous to a Death Star's exhaust port, a vulnerability nobody stopped to question because doing so would have slowed a project everyone was proud of.

How does this relate to the earlier article on AI First Mover Finality?

This piece follows on from AI First Mover Finality: Why Reaction Becomes Impossible, which argued AI collapses decision latency so whoever fires first wins completely. This article extends that argument by asking what happens if the weapon built to fire first has its own design flaw, and how long first mover advantage lasts if rivals can simply buy the same technology and build their own on a normal timeline.

What is Satya Nadella's Reverse Information Paradox?

Satya Nadella raised this concern in a post on X, arguing that enterprises using AI effectively pay for intelligence twice: once with money, and again by revealing the proprietary knowledge necessary to make the AI useful. He calls this the Reverse Information Paradox.

Could an AI vendor actually use client data as a competitive weapon?

The article argues the question is not whether a vendor could theoretically turn the stream of proprietary client data into a competitive weapon, since that is clearly possible, but rather what it would actually take in storage and compute terms to do so, and whether the industry is currently building that capability one enthusiastic integration at a time.

Why does the article compare AI vendor dependence to a Death Star?

The comparison illustrates a kind of institutional failure where every engineer is competent, every requirement met, and every review board approves, yet the finished project becomes a weapon pointed at its own builders. The flaw is a design choice nobody questioned, mirroring how enterprises enthusiastically adopt AI without examining the systemic vulnerability it creates.

13 Jul 2026 Artificial Intelligence

AI Vendor Risk: Are We Building a Death Star to Blow Up Our Own Planet?

1. The question worth asking properly

There is a particular kind of institutional failure where nobody does anything wrong at any individual step, and yet the finished project turns out to be a weapon pointed at the people who built it. Every engineer on that project was competent, every requirement was met, and every review board signed off, so the flaw was not a mistake anyone made. It was a design choice nobody questioned, because questioning it would have meant slowing down a project everyone was proud of.

This is a follow on from an earlier piece, AI First Mover Finality: Why Reaction Becomes Impossible, which argued that AI collapses decision latency to something close to zero, so that whoever fires first wins completely, the same way a weapon travelling at light speed arrives at the same instant as the warning of its arrival, leaving no window in which the other side can react at all. That argument is correct as far as it goes, and it is worth reading first. But it leaves two questions sitting unanswered underneath the physics. First, what if the weapon you just finished building, the one that lets you fire first, has a flaw in its own design that only becomes visible once it is operational. Second, and more importantly, how long does firing first actually stay decisive if the technology that let you build the thing is available to buy, meaning your rival is not stuck reacting; they are building their own on a normal commercial timeline. Finality that only lasts until the other side finishes construction is not finality; it is a head start.

That is roughly the shape of the question I want to work through here. Every enterprise sending prompts, documents, and workflows through a commercial AI model is handing that vendor a stream of proprietary thinking, diligently, at scale, with genuine enthusiasm for the productivity gain, the same way a very capable engineering corps diligently builds a very capable piece of infrastructure without stopping to ask what happens if the thing they are building has a structural weakness baked into its own design. The question is not whether a vendor could theoretically turn that stream into a competitive weapon against its own clients, since it clearly could. The question is what it would actually take in storage and compute terms to do it, and whether the industry is currently building the exhaust port that makes it possible, one enthusiastic integration at a time.

This is not a hypothetical concern dreamed up by paranoid security teams. Satya Nadella raised the same alarm directly, in a long post on X that is worth reading in full. He argues that enterprises using AI pay for intelligence twice, once with money and again with the proprietary knowledge they must reveal to make the intelligence useful. His framing is the Reverse Information Paradox, and it deserves a section of its own before we get to the technical mechanics, not least because the word he chooses for the leak is more fitting than he may have intended. You can read his original post here.

The strategic mistake enterprises are making underneath all of this is confusing access to intelligence with ownership of advantage. Every competitor can now rent the same frontier model, which means the model itself was never going to be the moat. The durable advantage sits in the proprietary context, evaluations, corrections, memory, and workflows wrapped around that rented model, and if those things accumulate outside your organisation rather than inside it, you are not merely buying intelligence from a vendor; you are helping the vendor manufacture the intelligence your competitors will rent next. Part of why this is hard to see coming is that there is no forensic trail for it, since you can prove nobody opened a specific file by checking an access log, but it is far harder to prove a model did not quietly get better at your exact problem because of six months of your own team’s corrections, and that asymmetry is the real subject of this piece.

2. Nadella’s Reverse Information Paradox

Nadella builds his argument on Kenneth Arrow’s classic Information Paradox, where a seller of information cannot prove its value without disclosing it, at which point the buyer already has it for free. Nadella argues AI creates the opposite problem for buyers, who must disclose their own proprietary knowledge simply to get the tool to work well.

His central mechanism is what he calls exhaust. Models learn continuously from the prompts employees write, the tools agents use, and especially the corrections people make when a model gets something wrong, and every correction is distilled into institutional know how that a competitor could never buy and that leaks almost imperceptibly, trace by trace, correction by correction, eval by eval. That is a different threat model than someone stealing a document. It is the slow transfer of judgment, process, and domain expertise, compounding with every interaction.

It is worth sitting with the word he picked. The most famous fictional design flaw in engineering history was a two metre wide exhaust port that ran straight into the reactor core of an enormous, beautifully engineered battle station, built by an empire that had every technical resource in the galaxy and still lost the whole thing to a single small ship that found the one unguarded vent. Nadella is describing something structurally identical. A vendor builds an extraordinarily capable system, an enterprise plugs its most valuable thinking into it every single day, and the exhaust, his word, not mine, is the port nobody sealed because sealing it would have meant slowing down adoption everyone was excited about.

He also points out an asymmetry in the current market structure. AI vendors claim fair use rights to train on public data, then write contracts that restrict customers from distilling their models, while reserving the right to learn from customer usage themselves. If learning only flows in one direction, the economic value of that learning concentrates with the infrastructure owner rather than the enterprise that generated it.

His proposed response is five priorities for enterprises: control over enterprise memory, evaluations, feedback, and institutional context; capability, meaning private environments where models can be customised without exposing proprietary knowledge; choice, meaning decoupling the orchestration layer from any single model; cost efficiency across models and workflows; and continuous ownership of the outputs used to fine tune systems. That last principle, own your evals, memory, and adapted weights, is the practical answer to a question every enterprise technology leader is now asking: what remains defensible once every competitor can rent the same underlying model.

3. Why don’t we already worry about this with S3, GitHub, Office 365, or SAP

This is the right question to press on, because the honest answer is that we already accept nearly identical structural exposure in several other places, and naming why we tolerate it there but not in AI is more useful than pretending AI is uniquely dangerous.

Could Amazon read everything sitting in your S3 buckets and copy your product? Technically, someone with the right internal access could open any object in any bucket. AWS’s own shared responsibility documentation states plainly that AWS operates the infrastructure layer while the customer configures access and encryption, and AWS states its systems are designed to prevent remote access by AWS personnel to customer data unless specifically requested by the customer or required by law. That is a policy and access control commitment, the same category of protection as a data processing agreement with an AI vendor, not a law of physics. Nobody treats this as an open question in practice, mostly because S3 has run at planetary scale for close to twenty years without a credible public case of Amazon mining a customer bucket to build a competing product, and because a leak like that would be a company ending scandal for a business whose entire value proposition rests on being trusted with everyone’s data. Worth noting though, AWS has a real history of watching which open source projects run profitably on its platform, then launching a managed competing service, Elasticsearch and OpenSearch, MongoDB and DocumentDB, Redis and MemoryDB. That is not reading anyone’s private bucket contents; it is reading public usage patterns and commercial success, but it is the same underlying instinct Nadella is warning about, a platform learning what works well on top of it and building a competitor from that knowledge.

GitHub is the most topical comparison, because it is happening right now. GitHub’s own privacy statement update, effective the twenty fourth of April 2026, draws a specific and fairly narrow line, private repository content sitting at rest is not used for training, but interaction data, which GitHub itself defines as inputs, outputs, code snippets, and associated context generated during an active Copilot session, can be used for training unless a Copilot Free, Pro, or Pro+ user opts out, while Copilot Business and Enterprise customers remain contractually excluded regardless. That distinction between content at rest and content in an active session is closer to the AI vendor risk than the S3 example, because it is not about someone reading data at rest; it is about live interaction data flowing into a training pipeline, exactly the mechanism Nadella calls exhaust. It drew real pushback in developer forums, noticeably less than Zoom faced in 2023 for a similar move, largely because the change was scoped to individual and small team tiers rather than enterprise contracts, and because GitHub held the line on data already sitting in a repository untouched.

Office 365 carries the same structural exposure through a different door. Administrators and Microsoft support staff have legitimate technical access to tenant content for support and legal hold purposes, and an entire product category, Advanced eDiscovery, exists because that access is real. Enterprise agreements state Microsoft will not use tenant content to train its models without a separate agreement, and large customers negotiate that explicitly, but the technical capability to read a tenant’s email and documents is not meaningfully different from the technical capability an AI vendor has over your prompts. What holds it in check is decades of enterprise contract precedent, audit rights, and a customer base large enough to walk away, not a technical wall that makes the access impossible.

SAP is the clearest case of all, and worth taking seriously given Capitec runs on it. A core banking or ERP platform does not see prompts describing your business; it sees the business itself, every transaction, every margin, every product line’s actual profitability, at a granularity no AI chat interface will ever approach through inference alone. SAP could, in principle, know exactly which customer segments are most profitable at every bank running its software, with more precision than any AI vendor could infer from prompts. Nobody frames this as an existential IP risk, because SAP’s business model has never been about becoming a bank. Its economics come from selling and supporting software, not from extracting and reselling the insight sitting on top of it.

Two more comparisons sharpen the point further, because they sit at opposite ends of how a vendor can choose to handle this exposure. Snowflake sits directly on top of a customer’s raw data warehouse, arguably a more complete view of the business than any AI prompt log, yet Snowflake’s own terms of service explicitly bar the company from using its technology to build any substantially similar cloud based service for a third party, and separately prohibit customers from reverse engineering Snowflake’s own technology, a mutual non extraction clause written directly into the contract rather than left to trust. Snowflake also draws a hard contractual line between Usage Data, which it can use to improve its product, and Customer Data, which it may not share with third parties unless aggregated and anonymised beyond identification. That distinction, spelled out in the contract rather than a blog post, is closer to what Nadella is asking AI vendors to commit to than most AI vendor agreements currently offer.

Palantir sits at the other end and is worth naming because Nadella quotes its chief executive directly in making his argument. Palantir has built its entire commercial pitch around the opposite promise to the one implicit in most AI vendor relationships, that the customer owns the means of production, meaning their compute, their models, their data stack, and the value derived from it, rather than having that value quietly transferred to whoever hosts the platform. Whether or not that promise is fully realised in every Palantir deployment is a separate question, but the fact that owning your own data and model layer is now a marketable feature, one Nadella leans on explicitly, tells you the market already knows this is the real fault line.

So the honest answer to why AI feels different is not that the underlying technical exposure is somehow larger, since in several of the cases above it is smaller, and the real difference is behavioural and structural. Infrastructure and enterprise software vendors have spent decades building a norm, reinforced by contract law, audit rights, and reputational consequence, that the provider’s business is renting capability rather than extracting and reselling a specific customer’s competitive intelligence. AI vendors are newer, their business models are less settled, and the same company is frequently both your tool provider and a direct or adjacent competitor in ways SAP and AWS mostly are not. There is also the forensic trail problem raised at the start of this piece, and it applies here with particular force, since every comparison above still leaves you able to check an access log, while no equivalent log exists for whether a model quietly absorbed your operational intelligence.

4. Three different threats hiding inside one question

It helps to separate what Nadella is describing from two further threats that are easy to lump in with it, because the defenses differ, and the first of the three, aggregate learning, does not need your specific document at all. It needs the pattern across thousands of customers like you, so that if every bank sends similar fraud detection prompts, similar contextual authorisation logic, similar mule network heuristics, the vendor’s model quietly becomes excellent at exactly the problem you are trying to solve better than your competitors, and every one of your competitors gets access to a model shaped by your own operational intelligence, without anyone having to leak a single document. The learning happens at the level of statistical pattern, not verbatim content.

The second threat is direct extraction. This is the classic security research problem of getting a trained model to regurgitate specific training examples verbatim, including things that were never meant to be public. This is a real and measured phenomenon, not speculation. Researchers have shown that large language models memorize examples from their training data, and that larger and more capable models are more vulnerable to this kind of extraction. A widely cited study found that among six hundred thousand generated samples, extraction attacks recovered memorized training text in roughly zero point one percent of cases, and this is likely an extremely loose lower bound. Follow up work on production chat models found something more unsettling still. Aligned models like ChatGPT initially appear far more private than raw base models, but the researchers developed an attack showing this is not really true, and using it, the aligned model emitted memorized training data at a far higher rate than previously measured.

So the honest answer to “could a vendor reverse engineer client IP” splits into two honest answers, before we even get to the third threat covered in the next section. Direct verbatim extraction of a specific client’s specific secrets is possible but narrow, noisy, and detectable if you know to look for it. Aggregate extraction of competitive knowledge is technically possible wherever customer interaction data is retained and permitted to influence model improvement, and unlike direct extraction, it requires no malicious intent; it is simply the predictable consequence of allowing proprietary corrections and workflows to become training signal. Enterprise agreements often explicitly exclude customer data from training, which is precisely why the contract terms in section nine matter more than anything else in this piece.

5. The third threat, when context escapes into orchestration

Both threats above are about knowledge transfer, your competitive edge ending up baked into a model somewhere outside your control. There is a third threat that is not about knowledge transfer at all, and it shows up the moment your organisational context stops being retrieved and summarised back to you, and starts being acted on. This third risk is structurally different enough to deserve its own separate treatment elsewhere, but it belongs here too, because enterprises frequently assume that solving data retention also solves AI risk, and it does not.

A model that pulls your own documents, policies, and prior decisions back into a chat response is doing what one sharp colleague I was messaging about this piece called information production, genuinely low risk in isolation, since a human reads the output and decides what to do next. The moment that same context feeds into an agent that takes action on its own, adjusting a credit limit, waiving a fee, blocking a transaction, calling an internal API, it becomes orchestration, and orchestration is a different risk category, one industry analysts covering agentic AI in 2026 draw the same line around, noting that a model trustworthy for question answering can introduce very different risks once it starts making autonomous decisions inside enterprise systems, with separate reporting finding that a majority of organisations have already encountered risky agent behaviour once agents moved from answering questions to acting on them.

What makes this dangerous is speed rather than malice. In a conventional generative tool, a wrong answer gets noticed and corrected by a human before it does damage. In an agentic system, the same error can trigger an action that propagates through a workflow before anyone reviews it, which is the same physics argument this piece opened with, once a system can decide and act faster than governance can observe, review does not happen before the consequence does. Your own institutional judgment, the fraud heuristics, the credit exceptions, the pricing logic your team spent years refining, is exactly what makes an agent good at your specific problem, which means it is also exactly what turns a plausible sounding mistake into an executed one, and zero data retention does nothing to prevent it, since a vendor can retain nothing at all and still hand an agent enough live context to act before any privacy commitment is even relevant. The controls that matter here are scope and gating rather than retention, covered as their own item in section nine.

6. What it would actually take to build one

Every good story about a planet destroying weapon eventually gets asked the boring logistical question. How big was the budget, how long did construction take, how many people signed off on the plans without reading past page one. This is that question, asked of the actual pipeline rather than the movie.

Assume a vendor wanted to do this on purpose, systematically, at scale, and here is roughly what the pipeline requires, starting with storage. Raw prompt and completion logs at enterprise scale are text, and text compresses well, so the storage burden is smaller than most people assume. A large bank sending several million prompts a month, each averaging a few thousand tokens of context including documents and code, produces something on the order of tens of terabytes a year of raw log data before compression, and a few terabytes after. Retaining that indefinitely across thousands of enterprise clients pushes a vendor into petabyte scale within a few years, which is unremarkable for any hyperscaler already running exabyte scale storage for other workloads, so storage was never really the constraint. Any of the major cloud providers can retain this volume of data as a rounding error against their existing footprint, which is exactly why the meaningful defense is contractual and architectural rather than a hope that nobody has enough disk space.

Compute for the aggregate learning path is also modest, because this path does not require retraining a frontier model from scratch. It requires curating a fine tuning dataset from the accumulated logs, filtering and deduplicating it, and running supervised fine tuning or preference optimisation on top of an existing base model. Fine tuning runs on the order of hours to a few days on a modest cluster of high end GPUs, depending on dataset size and target model size, several orders of magnitude cheaper than pretraining. Vendors already operate this exact pipeline for legitimate purposes such as improving safety behaviour and response quality, which is precisely what makes the aggregate learning threat hard to distinguish from normal product improvement.

Compute for the direct extraction path is different again, and it does not require the vendor to do anything beyond what any external researcher can already do with API access. The published extraction studies used ordinary inference calls at scale, generating on the order of hundreds of billions of tokens of output and scanning it for matches against known or suspected training data, work that ran on standard inference infrastructure rather than specialised training clusters. One study found that memorization grows nearly linearly even after generating several hundred billion tokens of output, meaning the attacker’s real cost is inference volume and search effort rather than any exotic compute requirement.

Put plainly, none of this requires anything close to the compute needed to train a frontier model, and nobody needs to build a second Death Star to exploit the first one. The barrier to a vendor doing this was never technical capacity; it was always policy, contract terms, and the reputational cost of getting caught.

7. This has already happened outside AI, which tells you how it plays out

Section three asked why AI draws more suspicion than S3, GitHub, Office 365, or SAP. Amazon and Zoom are the two cases where the suspicion turned out to be justified, and both are worth walking through, because they show what happens once trust actually breaks rather than merely being theoretically breakable.

If this pattern sounds familiar, it is because Amazon has been through almost exactly this fight, minus the AI framing. A Wall Street Journal investigation found that Amazon executives had access to seller data that was used to discover bestselling items worth competing against, with executives developing workarounds to internal restrictions to reach seller specific reports. Amazon had told Congress under oath that it does not use individual seller data to compete with third party sellers, while the investigation found the opposite among more than twenty former employees interviewed, and the mechanism behind it was mundane, since Amazon simply watched what sold well on its own marketplace, then launched a private label competitor priced against that intelligence.

The AI version of this is structurally identical, just one layer more abstract. Instead of watching what sells, the vendor watches what your engineers ask, what your analysts correct, and what your domain experts had to teach the model before it got useful, and the output is not a competing product on a shelf; it is a better foundation model, trained in part on the accumulated correction effort of every client who used it before you, sold back to your direct competitor next quarter.

The Zoom episode from 2023 is the other useful precedent, because it shows how fast this becomes a public trust problem rather than a quiet contractual one. Zoom’s terms of service, updated in March 2023, had customers grant a perpetual, worldwide, royalty free license covering machine learning and artificial intelligence training and testing. The backlash was significant enough that Zoom rewrote the relevant sections to state it would not use audio, video, or chat customer content to train its AI models without consent. The lesson was not that Zoom had a secret plan to strip mine customer meetings; it was that the default posture of most AI vendor contracts, before anyone pushes back, is broad enough to allow it.

8. Everyone eventually gets the bomb

The Death Star explains the architectural vulnerability, a flaw baked into how the thing works. What it does not explain is why the advantage of building it first cannot stay exclusive, and there is an older, more literal story that answers that half of the question directly. In July 1945 the United States tested the first nuclear weapon and had, for a brief window, the only one in existence. Rather than assume the monopoly was durable, the US tried something close to what a cautious enterprise tries with a vendor today, the Baruch Plan, offering in 1946 to dismantle its own arsenal once an international authority could verify nobody else could build one either. The Soviet Union rejected it, insisting the US disarm first, and the talks collapsed by the end of the decade, because whatever assurance a cautious first mover tries to negotiate, the other side has every incentive to hold out for a better deal while it finishes building its own.

The Soviet Union tested its own weapon on the twenty ninth of August 1949, four years after Hiroshima, far sooner than American planners expected, and the monopoly did not last a full presidential term. Within another decade the UK, France, and China all had their own, and the pattern has not stopped since, nine states possess nuclear weapons today, with roughly one new entrant joining every decade or so despite the Non Proliferation Treaty, the Comprehensive Test Ban, and sanctions regimes explicitly built to prevent exactly this. Prevention, as a strategy for stopping a transformative capability from spreading once someone has an incentive to acquire it, has a poor track record over eighty years, not for lack of resolve, but because capability that valuable eventually gets built or bought by whoever wants it enough. That pushes back directly on the first mover finality argument this piece opened with. The physics of firing first is real, once a light speed weapon is away there is no reacting to it, but four years is not finality; it is a head start, and every proliferation event since has followed the same shape, with the window between leader and follower getting shorter, not longer.

Eisenhower’s Atoms for Peace program in 1953 makes the sharper version of the point. It shared civilian nuclear expertise internationally as a goodwill gesture, training thousands of foreign scientists, a number of whom later worked on secret weapons programs back home. That same generous, commercially sensible decision to share a dual use capability broadly is what every AI vendor does when it sells the same frontier model to you and your direct competitor in the same sales quarter, and nobody has to steal anything to make that happen, since the capability is handed out as the ordinary business model of the vendor, with your competitor standing as a customer in exactly the same position you are.

What actually changed the nuclear story after 1949 was not a renewed attempt to keep the technology contained. It was the slow build, through the IAEA from 1957 and the Non Proliferation Treaty from 1968, of a verification regime that never pretended to stop the underlying capability from spreading, and instead made it detectable and costly when material was diverted from declared peaceful use into a weapons program. Safeguards inspectors do not prevent a country from having nuclear material; they make it very hard to divert that material to something undeclared without anyone noticing, and that is the right ambition for an enterprise dealing with an AI vendor too. You are not going to be the only customer with access to the model, and pretending otherwise repeats the Baruch Plan’s mistake of expecting a rival to accept a permanent disadvantage voluntarily. What you can build instead is a safeguards regime of your own, contractual and technical controls that do not stop the vendor’s underlying capability from existing or from being sold to everyone else, but that make it detectable and costly if your specific data, corrections, and competitive edge get diverted into a model that ends up in a competitor’s hands. That is a meaningfully more achievable goal than trying to keep the technology to yourself, and it is the actual shape of every defense in the next section.

9. Sealing the exhaust port

Given the mechanics above, here is what genuinely reduces the risk, roughly in order of how much it actually matters rather than how good it sounds in a vendor pitch deck. None of these prevent the vendor’s underlying capability from existing or from being sold to your competitor next quarter, since the nuclear parallel says plainly that battle is already lost. What they can do is function as your own safeguards regime, making it detectable and costly if your specific exhaust, rather than the general capability everyone is buying, ends up somewhere it shouldn’t.

9.1 Contractual zero data retention, verified, not assumed. Zero data retention means prompts, completions, and associated metadata are not stored, logged, or used for any purpose beyond the immediate call, with data deleted within seconds rather than retained for the industry standard thirty day window. The distinction that matters is between a vendor saying it will not train on your data and a vendor saying it will not retain your data at all. These are different commitments, and the more stringent retention guarantee is the one that actually removes the aggregate learning risk, since a vendor cannot learn from data it never keeps. Ask for this in writing, ask whether it is enforced at the infrastructure level rather than only contractually, and ask whether every subprocessor in the chain, including the underlying model provider if your vendor is a wrapper, is bound by the same terms.

9.2 Treat metadata as exposure too. Even under a strong data processing agreement, request timing, frequency, prompt length, and access patterns typically remain outside contractual protection, and for some workloads that pattern alone is enough to violate confidentiality, such as a bank’s query volume around a specific fraud typology or a law firm’s call frequency on a specific case. Budget for this in your threat model even after the contract is signed.

9.3 Architectural isolation for the workloads that matter most. The most resilient version of this control is not trusting the vendor’s promise at all. Samsung’s response after an internal leak incident was to build its own family of internal models running entirely on its own infrastructure, so the data simply never leaves the perimeter, with external tools like ChatGPT permitted only under executive approval in specific divisions. For Capitec, the equivalent is not abandoning frontier models; it is routing the fraud architecture, mule network detection logic, and contextual authorisation work, the genuinely differentiating IP, through a private inference layer or a properly isolated enterprise tenant, while general purpose work can sit on standard commercial terms.

9.4 A stateless proxy you control, sitting between your systems and the vendor. A trust layer inside your own perimeter can intercept traffic, scrub identifying details before they leave your network, and log only metadata rather than full content, so you retain an audit trail without the vendor ever seeing the raw material at all. This shifts the burden of proof from the vendor’s promise to your own infrastructure, which is a much stronger position to negotiate from.

9.5 Own the layer that actually compounds. This is Nadella’s point, and it is correct regardless of what you think of the framing. The durable asset is not the model you rent, since every competitor can rent the same one. The durable asset is the evaluation suite, the fine tuned adapters, the memory layer, and the corrections your own people generate, kept inside your own infrastructure rather than handed to the vendor as free training signal. Build the muscle to capture that exhaust yourself before you hand it over. This is the same principle that separates Snowflake’s contractual non extraction clause from Palantir’s own the means of production pitch, discussed in section three, from the default posture of most AI vendor agreements. The vendors that treat this as a real commitment write it into the contract in the same specific terms they use for uptime or liability. The ones that treat it as a talking point leave it to a paragraph on a trust page that can be revised without renegotiating anything you signed.

9.6 Read the actual contract, not the marketing page. Default settings across every major provider retain prompt content for a period measured in weeks for abuse monitoring even when training is contractually excluded, and feedback mechanisms such as a thumbs up on a response can explicitly opt a specific conversation back into the training pool. Disable feedback mechanisms on sensitive workspaces. Get the data processing addendum reviewed by someone who understands both the legal language and the technical architecture behind it, because the two do not always say the same thing.

9.7 Scope and gate what an agent is authorised to do, not just what it can see. This is the answer to the third threat from section five, and it sits outside every retention control above, because a zero data retention agreement says nothing about what an agent with live organisational context is permitted to actually do while that context is in its working memory. Give agents narrow, explicit permissions for actions rather than broad access modelled on a human role, require a human or a rules based check before anything consequential executes, credit changes, fee waivers, transaction blocks, and treat ownership of the orchestration layer itself as the real control point, since that is where your context turns into action rather than into a paragraph you can review afterward.

9.8 Seed and monitor vendor learning canaries. Every control above reduces exposure, but none of them tells you whether proprietary knowledge has actually escaped, which is the one gap a genuine safeguards regime cannot leave open. Maintain synthetic evaluation cases, terminology, and correction patterns that are unique to your organisation and unlikely to exist anywhere else, expose different canary sets to different vendors or environments, and periodically test whether those specific patterns turn up in models or services where they have no business appearing. This is not cryptographic proof of misuse, and false positives should be expected, but a safeguards regime with no detection mechanism is just a policy statement, and this is the piece that makes the analogy to IAEA inspectors actually hold.

10. Are we building it or are we guarding it

Nothing in this analysis requires exotic compute or heroic storage engineering on the vendor’s side. A vendor with a normal commercial API and a normal fine tuning pipeline already has everything technically necessary to learn, in aggregate, from the accumulated corrections and workflows of every client who sends it traffic, and that is not a conspiracy theory; it is the default architecture of the industry unless a client actively negotiates against it, and Nadella saying so publicly from inside Microsoft is itself a signal of how seriously this is now being taken at the top of the industry, whatever his commercial motives for saying it.

The direct extraction threat, a vendor deliberately reverse engineering one specific client’s specific secrets out of a trained model, is real, published, and measurable, but it is the noisier and more detectable of the two risks. The quieter risk is the one worth losing sleep over: your competitive edge, learned one correction at a time, showing up as a small unexplained improvement in a rival’s AI tool eighteen months from now, with no single moment you could point to as the leak.

So, we know exactly what we built, and nobody adopting AI at this pace is unsure whether it is transformative; that was never the open question. The open question is the one this whole piece has been circling. We are hoping it stays pointed at competitors and slower moving industries, the way the US spent 1945 to 1949 hoping its monopoly was durable, and we are hoping the mechanism that makes the thing work, the exhaust, does not run a line back to our own reactor core the way a two metre vent ran back into a battle station’s own power source. History answers the first hope plainly; your competitor gets a version of this too, on a sales cycle rather than a spy’s timeline, because the vendor’s business model requires selling it to them. The second hope is the one still worth acting on, because whether the specific edge you built, your evals, your corrections, your fine tuned layer, ends up staying yours or ends up diverted into whatever the vendor sells next door is not decided by history; it is decided by whether you built a safeguards regime or just assumed the vent was sealed.

The strategic question, in the end, is not whether your enterprise should use frontier AI models, since refusing to use them simply guarantees that competitors compound faster than you do. The question is where the learning accumulates. If your employees correct the model, your workflows teach the agent, your evaluations expose what good actually looks like, and your institutional memory supplies the context, then the system in front of you is becoming more valuable with every single interaction, and someone owns that compounding asset whether you’ve decided to claim it or not. Guard against the mechanism, not just the headline, with zero retention in writing and verified, metadata treated as sensitive, architectural isolation for the workloads that actually differentiate you, scoped and gated agent permissions, canaries that tell you whether anything has actually escaped, and a habit of owning the evaluation and correction layer rather than gifting it to whoever is hosting the model this quarter. Build the thing, since everyone else is going to have one too, and just make sure the compounding asset it creates along the way ends up belonging to you.