The copy was for the dashboard

Notion's stolen AI chats existed so admins could see how their teams used AI. They sat in an inference vendor's storage, outside its zero-retention default, during a month when intruders were breaking into AI platforms for the keys.

There is a tab in Notion's AI settings called Insights, and at the moment it says it is paused. When it worked, it was there to tell a workspace owner how their team used AI: the kinds of things people asked for, how many conversations they had, which use cases were taking off. It is a reasonable thing to want. If you are paying for a tool across a whole company, you want to know whether anyone is using it and what for, and the honest way to find out is to look at the conversations, or at least to have something look at them on your behalf and sort them into piles.

That sorting had to happen somewhere. Notion's own notice says it used an outside company, Fireworks AI, "for classification jobs to help customers understand their own AI usage through a feature in Notion called Insights," and also as a general inference provider to serve open-weight models inside Notion AI. So to build the dashboard, some of the chats people had with Notion AI were copied to a second company's systems, where they could be read by a model and labelled. Nobody who typed into Notion AI was thinking about that copy. Most people would not have known it existed. This week it turned out to be the copy that got stolen.

Notion now says that a breach at Fireworks exposed Notion AI chat transcripts "used for the Insights feature," affecting fewer than 1% of Notion workspaces. It has switched Insights off, stopped serving open-weight models through Fireworks, disconnected from it entirely, and told affected customers to treat the data "as if it were exposed." On 11 November it revoked Notion API keys that its investigators had found pasted into the stolen messages themselves. That last detail is the one I keep coming back to. People were pasting live keys into a chat box, which is what people do, and the chat box was quietly keeping a copy for a report about how people use the chat box.

I want to walk through how this happened, because the sequence is more interesting than the headline, and because the headline undersells it. This was not one vendor having a bad week. It was one stop on a month-long run of break-ins at AI developer platforms by people who mostly wanted something else entirely: working keys to frontier models, which they could resell by the token. Notion's chats were, as far as anyone can tell, collateral. That should worry you more, not less.

Two break-ins, one disclosure at a time

Fireworks is one of the big independent inference companies. You bring it a model, usually an open-weight one, or you pick one off its shelf, and it runs it fast and cheaply on its own GPUs. According to The Record, it has more than 10,000 customers. A lot of products you have used this year run some of their AI through a company like this rather than directly through a frontier lab, because it is faster or cheaper or lets them use an open model. You would not know from the outside.

The first incident started on 6 October. At nine in the morning Pacific time, Fireworks spotted someone using internal Fireworks credentials who should not have had them. In a bulletin on 9 October, it said the intruder had reached environment variables used in evaluation jobs in a limited number of customer accounts, and that those variables included "API keys and tokens stored as secrets." In plain terms, when customers set up jobs on Fireworks to test their models, some of them had handed Fireworks the keys to other services so the jobs could run, and those keys were taken. At the time, Fireworks said it had seen no sign that customer models, datasets, training data, inference traffic or personal information had been touched. It revoked the stolen internal credentials, rotated tokens, and killed the affected keys, including some the attacker had created for themselves.

Then, on 23 October, something else happened. A third-party cloud provider, which Fireworks has not named, that hosted Fireworks' research and development machines had its own security incident. The next day, someone used that opening to get onto Fireworks' development machines and "run an unauthorized workload that led to additional customer data impact." Fireworks says it detected that the same day, closed what it knew about, and has not seen the intruder successfully back on the customer platform since 25 October.

What that second break-in reached, Fireworks did not say publicly until 9 November, more than two weeks later. Its update listed three kinds of data: resource metadata, such as the names and descriptions of things customers had set up; billing information, including billing and shipping addresses, invoices, payment history, balances and credit usage, though not card numbers, which Fireworks does not store; and, "for certain customers, a subset of incomplete model weights and datasets." It added the line that has since been read very closely: "We have not detected any impact to Inference, Training SDK and Nexus, which are all zero data retention (ZDR) by default."

How Notion found out

Notion has published its side as a dated table, and it is worth reading as a sequence, because it shows what being downstream of a breach actually feels like.

On the evening of 2 November, Fireworks told Notion that "a small set of data had been accessed by malicious actors." Notion looked, and found that the dataset was Notion AI synthetic testing data, made-up conversations used to test the product, with no customer data in it. Even so, by the morning of 3 November Notion had cut its systems off from Fireworks completely. On 4 November it deleted everything it had in Fireworks.

On 6 November Fireworks came back and said that "malicious actors may have accessed potential customer data." On 8 November it confirmed the final dataset. That was the one with the Insights transcripts in it. On 9 November Notion started contacting the customers it could identify. Notion then published its notice, and on 11 November it revoked the API keys it had found in the stolen chats.

So Notion did the cautious thing early, on the basis of a breach that looked harmless, and it still could not do anything about data that had already left. Disconnecting a vendor stops the next copy being made. It does nothing about the copy that is already in someone else's hands. By the time Fireworks knew what had gone, the break-in that took it was two weeks old.

Notion has tried to soften the impact in one specific way. It says that for "the vast majority" of affected workspaces, the transcripts were "unattributable," which it defines carefully: the chats were not labelled with who wrote them or which workspace they came from, and they did not include the results of tool calls, meaning the bits of documents and data Notion AI fetches to answer a question. "Beyond what's in the chat transcript itself," it says, "there's no information tying the chat back to workspaces, users, or companies."

That is true and it is also a strange comfort. TechCrunch reported that Notion described the transcripts as including "any input to or output from the model that was written to the chat." Whatever someone typed is in there. If they pasted in a meeting summary with a client's name at the top, the chat is attributable to that client whether or not the file has a label on it. If they pasted in an API key, as some evidently did, the key does not need to know whose it is to work. On Reddit, one Notion user summed up the mood under the company's reassurance that fewer than 1% of workspaces were affected: "looks like all of us are part of that <1%."

Zero retention, mostly

The line about zero data retention has drawn the most attention, and for good reason, because "ZDR by default" is precisely the kind of phrase that ends up in a procurement spreadsheet and is never looked at again.

VentureBeat's Louis Columbus did the useful work here, and his reading is that Fireworks' statements and Fireworks' documentation are both accurate and say different things. For ordinary inference, the documentation says prompts and outputs "exist only in volatile memory for the duration of the request." That is the default, and that is what most people mean when they hear zero retention: you send a question, you get an answer, and nothing is written down. But the same policy page says "a few features do store customer content, such as stored responses, batch jobs, and training." Batch jobs are when you hand over a big pile of inputs to be processed later, which is exactly the shape of "classify a lot of chat transcripts into use cases."

There is also a stricter setting, a Zero Data Retention policy that works by "rejecting every request and job that would persist or log customer content." That one is available on Enterprise accounts, and Fireworks' own documentation lists its default as "Off by default." Columbus also noticed that when Fireworks CEO Lin Qiao repeated the line about no impact on her X account, in an "October Incident Update" post that has been seen around 185,000 times, her wording was slightly different. It reads "Inference (excluding batch inference)." Fireworks has since disabled the self-serve dataset uploads and downloads used by batch inference and dataset-based training while the investigation continues.

I want to be careful here, because neither company has said which Fireworks feature held Notion's transcripts, and nothing in the disclosures shows that switching on the stricter policy would have stopped the break-in. What the disclosures do show is the general shape of the gap. Zero retention describes the live path, the request that comes in and the answer that goes out. It does not describe the side paths: the batch job, the eval run, the dataset somebody uploaded so a model could be tested or tuned against it. Those are the paths that store things, because storing things is their job, and those are the paths the intruders reached. The first break-in hit evaluation jobs. The second hit datasets and half-finished model weights. Neither of them hit the thing that the marketing promise was about.

What the intruders actually wanted

If you read only the Notion side, you would think this was a story about someone hunting for corporate secrets in chat logs. Fireworks says something different. "The evidence indicates links to a cybercriminal threat actor group, which is financially motivated and aimed at selling tokens from unauthorized access to Large Language Models (LLMs) to illicit routing gateways." When Cantina Security's chief executive, Hari Mulackal, raised concerns about the stolen weights, Qiao answered, as The Register reported: "From what we see attackers are not after weights. Most downloads were partial and stopped."

The phrase for what they were after is LLMjacking. On 2 November, the same day Fireworks first told Notion anything, Google's Threat Intelligence Group and Mandiant published a long post called "The Rise of LLMJacking." It describes intruders stealing cloud credentials and API keys and using them to run frontier models on someone else's bill. Unlike crypto-mining on stolen servers, which builds up slowly, this can run up charges very fast: Google says it saw fraudulent charges above $100,000 in a single day, and at some organisations as much as $1 million before anyone shut it off. It describes a division of labour, with one group breaking in and another running what it calls "LLM Mixers," illicit gateways that pool stolen access and resell it to underground subscribers. It names one of them, Jet Night, which sold an 'unlimited' plan for about $60 a month.

Wiz, the cloud security company, had already put a name to the month. On 30 October it published "Black October for AI Developer Platforms," listing nine AI developer platforms that had disclosed incidents in October alone: Fireworks, Andromeda, Zeabur, Lightning AI, OpenCode, Railway, Warp, CommandCode and Northflank. It has been adding to the list since. Brev, which is owned by Nvidia, and Trigger.dev were reported on 2 November; Chainstack on 3 November; kernel.sh and Blacksmith in the first week of November; Tensorlake on 8 November, linked to "an upstream provider"; NEAR AI on 9 November, after someone used a remote code execution bug in SGLang, a popular open-source serving engine, to get onto an inference server. Zeabur, a deployment platform, has said it is shutting down because of its incident.

Wiz is clear that there is no single bug tying these together. Some started with stolen credentials, one with privilege escalation in build infrastructure, one with a vendor dependency, one with a legacy system. What ties them together is what the intruders went looking for once they were inside. "All public reports highlight targeted exploitation, explicitly hunting stored environment variables, with a focus on those granting AI access." Fireworks, OpenCode and Warp all reported signs of what Wiz calls an "agent swarm" attack, meaning many automated agents working the intrusion at once. OpenCode and CommandCode said the attackers were trying to collect material for distillation, the practice of training one model on another model's outputs.

Wiz could not see inside most of these breaches, but it could see what happened to two enterprise keys afterwards, keys that had been heavily used from the infrastructure of one of the breached vendors, and the detail is almost dull in its efficiency. Within 30 minutes of the credentials being taken inside the vendor, both keys got the same first call: a single request to Amazon Bedrock from a home internet address, part of a one-minute sweep testing stolen keys from several organisations. The request asked for Claude Fable 5.1, failed, and fell back to Opus 5.5, which worked. For the next few days the keys saw a trickle of 14 to 120 calls a day, in lockstep about a minute apart, which Wiz reads as a script checking that the keys were still alive.

Then they were sold, or used, or both. In the first case, four days after that first test, a pool of rotating addresses tried one call per model across the catalogue, a script mapped the account's quotas, and 22 seconds later bulk traffic started. Over 30 hours, before the key was shut off, it made about 800,000 calls, about 90% of them successful, three quarters of them to Opus models. In the second case, the key was locked down more tightly, and the first reseller pool that got it spent four hours failing against its restrictions. Then a checker worked out which model names it would accept, the same script turned up, down to its version string, and over 13 hours a single address made about 1.7 million calls, more than a million of which succeeded. At list price, Wiz estimates those two keys alone ran up more than $100,000.

The most telling part is what the traffic looked like. A typical call sent a handful of fresh input tokens and got a few hundred back, which is what you see when a coding agent is working through a long session with most of its context cached. A long tail of calls carried tens of thousands of fresh tokens, consistent with many different people behind a reseller. In other words, the stolen access was most likely being sold on to people running coding agents, at a discount, on somebody else's bill. Wiz found that an operator who appears to call themselves "momo" had access to both keys, and tied them to an address that hosts a site advertising cheap frontier model access for payment in USDT, a cryptocurrency stablecoin. In early October the same address hosted a page titled, in Chinese, "ARTEX autonomous penetration testing console."

Why the chats came along

Put those two stories next to each other and the Notion leak looks less like a targeted theft and more like what happens when a burglar goes through a house looking for car keys and takes the filing cabinet because it was next to the door. Fireworks' second intrusion ran an unauthorised workload on development machines that could reach storage. Some of what was in storage was customer datasets, half-trained weights and billing records. Some of it, for Notion, was a pile of chat transcripts that existed so a dashboard could tell an admin what people were asking.

None of this is an argument that Insights was a bad feature or that Notion was careless. By its own account, it moved fast, and it moved early on information that looked harmless. It is an argument about where copies come from. Every AI product now has at least three places your words go: the live request, which is often genuinely not stored; the product's own logs; and a growing number of side jobs that turn conversations into something else, such as evaluations, classifications, fine-tuning sets, analytics and quality reviews. The side jobs are where the copies pile up, and they are increasingly run by someone other than the company whose name is on the chat box. A retention promise that covers only the first of those places does not tell you very much.

Meanwhile, the attackers have found something in AI infrastructure that is worth more than the data: the keys themselves, which turn straight into money because there is now a working market for frontier model access by the token. That changes who gets targeted. A platform that holds customers' AI keys in environment variables is now a bank vault with the sign on the door, and the attackers are, at least according to three of the victims, using swarms of agents to work through the vaults faster. The data around the keys, your chats, your datasets and your half-finished models, comes along for the ride.

What I'd ask

If you run a company that uses an AI tool, the useful question is no longer "do you retain my data?" That gets answered with "zero retention by default," which is true. The questions that would have caught this one are narrower and duller. Which of your features store what my people type, even briefly, and why? Which of those copies go to another company, and which company? Is the strict no-storage setting on for my account, or just available? Fireworks' own documentation notes that "an absent zeroDataRetention means the policy is off," and that turning it on "doesn't delete anything already stored." When a breach happens two companies away, how long will it take you to find out what was in it?

And if you are a person who uses these tools at work, the advice is older than any of this. The chat box is not a password manager. If you have pasted a key into one this year, rotate it. If you have pasted a client's notes into one, assume there is a copy somewhere you have never heard of, made for a reason that probably made sense at the time.

Notion's Insights tab is still paused. When it comes back, I hope it comes back telling admins not only what their teams asked the AI, but where, along the way, each of those questions was kept.