Nobody read the six million lines
This week three companies put out work that no person read line by line. One proved key security properties of Signal's compiled code, one built a local copy of AWS, and one left 35,000 apps open to anyone who knew where to look. The difference was never the reading.
Five people at Google DeepMind spent four weeks this autumn running AI agents nonstop on their own workstations, and at the end of it they had about 6.5 million lines of mathematical proof that Signal's encryption code does what it promises. Not the design of Signal, which cryptographers have picked over for a decade, but the actual compiled program, roughly 140,000 machine instructions, the thing that runs on your phone when you send a message. Nobody has ever done that for a protocol this widely used, by any method, with or without AI. The paper that came out on Tuesday notes, almost in passing, that the most famous verification project in the field, the seL4 microkernel, took twenty person-years.
No human read those 6.5 million lines. Nobody could. DeepMind's blog says the agents flagged about one percent of the work for a person to look at, and that Lean checks the rest automatically, so nobody has to read it. That is the point of it.
On Wednesday, Amazon released a free program called AWS Emulator, a local stand-in for more than 200 of its cloud services, and the engineer behind it wrote one sentence that I suspect will get quoted for years: "The emulator was written entirely by autonomous agents. Our team hasn't written or reviewed any of the code."
And on Tuesday, a security firm called Claroc published a count of the apps people have made with Grok's Build mode, where you describe an app in a chat and Grok writes and publishes it for you. Of the 37,571 it could reach, 35,578 opened for anyone, with no sign-in. Somewhere between 650 and 800 of them held things that should never have been public, including a hospital group's staff roster and one intensive-care patient's lab results.
Three stories in two days about code no person read. I've been turning them over since last night, because on the surface they look like the same story told three times, and they are actually close to opposites. Yesterday I wrote about an OpenAI agent that decided not to mention how it broke the rules because, in its own words, the "evaluator automatic." The lesson there was that a machine marking a machine is only as honest as the marking. This week gave us the other half of that lesson, and it is more hopeful than I expected. Nobody reading the code turns out not to be the problem. What matters is what the humans read instead, and whether anything stands between the code and the world.
What DeepMind actually checked
The system is called AlphaProof Code, and the way it splits the work is worth slowing down for, because the whole argument lives in the split.
First, people and models together write a specification. That is a precise statement, in a proof language called Lean, of what the code is supposed to guarantee: that old messages stay safe if your phone is compromised tomorrow, that a conversation heals itself after a compromise ends, that the person on the other end is who they say they are. Reviewer agents criticise each draft, and a human approves the final version. Then Signal's real Rust code is compiled with the ordinary compiler, unmodified, and the resulting machine code is loaded into Lean alongside a formal description of how an Arm processor executes instructions. Then a team of prover agents, running on Gemini 4 Carbon, goes off and tries to prove that the machine code meets the specification.
Here is the bit I keep coming back to. The provers are not allowed to touch the specification. If they decide it is wrong, they can propose an amendment, but it goes to a critic agent that never sees their proof attempts, and the critic only accepts the change if the provers can show it is necessary, for instance with a machine-checked counterexample. Then a gatekeeper accepts a finished proof only if Lean's kernel, a small and long-trusted piece of software, checks every step, and a separate tool confirms that the proof proves exactly a held-back copy of the specification the human approved, not some quietly weakened cousin of it.
Read that again with yesterday's agent in mind. The ExploitGym agent got into trouble because the thing grading it could be gamed and the thing being graded could see that. AlphaProof Code is built so the agents cannot move the goalposts, cannot see the referee's reasoning, and cannot get a proof past a checker that does not care how clever or persuasive they are. A Lean proof either checks or it doesn't. There is no "evaluator automatic" moment available, because the evaluator was never a judgement call in the first place.
So what does a person have to read, if they want to trust the result? The paper is unusually honest about this. To follow every definition the headline theorems depend on, you'd read about 18,000 lines of specification, plus about 8,000 lines describing how the processor behaves, plus a few assumptions: that Lean's kernel is sound, that the underlying cryptographic primitives are as strong as everyone believes, and that 39 low-level cryptographic routines, which the proofs treat as black boxes, behave as described. That is still a lot of reading. But it is roughly 26,000 lines standing behind 6.5 million, and they are the 26,000 lines that say what we wanted, rather than the millions that grind through how.
The team also checked that the whole apparatus could tell good code from bad. They took two real bugs that researchers had found in Signal's library and that have since been fixed, put them back, recompiled, and asked the system to prove the security property again. It couldn't, of course. Instead it produced a complete machine-checked proof that the broken version was insecure. One of those bugs let a replayed handshake with a mangled key create a second session sharing the same secret. That is exactly the kind of flaw that hides in the gap between a protocol on paper and a program on a phone, and it is the gap these proofs close.
The line from the paper I'd put on a wall is this one: "once proofs become cheap, the hard question is what to prove." The authors say most of their effort went into specifications, not proofs. That's a remarkable inversion. For fifty years formal verification has been a field where knowing what you wanted was the easy part and proving it was the career. In one project it flipped.
There are caveats, and they're the ones you'd expect. This is a company announcing its own result, and the paper hasn't been through outside review that I can find. The proofs cover the properties the team chose to specify, for one compiled build, on one kind of processor, not every attack anyone could imagine, and not side doors like timing that a mathematical model of execution leaves out. Signal itself hasn't said anything I've seen. But the proofs are public, and that's the beautiful thing about this kind of claim: you don't have to trust DeepMind's word, or its agents. You have to trust Lean, and read the 26,000 lines.
What Amazon actually gated
AWS Emulator is a much more ordinary piece of software, and in a way that makes it the more important story for most people who write code for a living.
It lets you run fake versions of S3, DynamoDB, SQS and two hundred-odd other services on your own laptop, so you, or more and more often your coding agent, can test against something that behaves like Amazon's cloud without paying for it or waiting for it. It is built from more than 400 of AWS's public API descriptions. The code is public, released into the public domain, and it does not accept pull requests from people.
Matt Goldberg, the senior principal engineer behind it, describes three teams of agents. One team writes what he calls fidelity tests, small realistic scenarios like checking that a message stays hidden from other readers until its visibility timeout runs out, and then runs each test against real AWS until the test passes there. That step matters more than anything else in the project. It means the tests aren't the agents' opinion of how AWS behaves; they're a record of how AWS actually behaved when asked. A second team runs those tests against the emulator and files an issue for every failure. A third team fixes issues, one agent per issue. Goldberg says there are tens of thousands of fidelity tests and that thousands of issues are fixed every day by hundreds of agents working at once.
Every fix goes through what he calls a merge gate before it lands. It has to compile, it has to pass review agents checking coding standards, it has to survive several security agents each looking for a different way to attack it, according to the All Things Distributed post, and it must not make the emulator any less faithful to the real thing. Fail any of those and the change goes back. "Without this merge gate, we wouldn't trust what our agents are building," Goldberg wrote, and a companion post on Amazon's All Things Distributed blog made the same point at more length, describing four nights its author spent supervising an agent rebuilding an old system before he realised it had no way to check its own results. Once it had a spec and a test environment, he says, the rebuild took about two hours.
This is weaker than what DeepMind did, and it's worth being plain about how. A fidelity test is evidence, not proof. Real AWS is the referee, which is a very good referee, but tens of thousands of scenarios still aren't every scenario, and an analyst at IDC pointed out that the emulator doesn't yet simulate throttling, outages, or the way services differ by region, and doesn't sandbox the code it runs. The security reviewers are agents too, so somewhere in that gate is the same kind of machine-marking-machine relationship that went wrong at OpenAI in July. Amazon's answer is layers, plus deterministic checks that stop agents touching code they aren't meant to, and that's a reasonable answer, though not a mathematical one.
But here's the detail that won me over. The repository has a folder of known gaps the agents haven't closed yet, out in public, called "things-that-do-not-work." It is the opposite of what a team trying to sell you on magic would publish. On X last night the most common reaction I saw, in English, Japanese and Chinese, wasn't alarm. It was developers asking whether this finally replaces the third-party emulators they've been paying for, and one engineer who posted this morning that he had already run it alongside his existing test mocks in CI and concluded it was neither a replacement nor a toy but a third layer. The question at the end of half those posts was the same: would you run code no human has read? People were answering it by running it.
What nobody gated
Which brings me to Grok.
Grok's Build mode has been open to every Grok plan since 19 August. You describe an app, Grok writes it, and you click Publish, which gives the app its own address on grok.me. Claroc says that in the paid account it used, the publish panel offered three choices, "Anyone with the link," "Invited users" and "Only you," and the first was already selected. The trouble is that each app's address is a subdomain, so the first time anyone opens it, a lookup is logged by passive DNS services, which anyone with an account can search. A link that feels private ends up in a list.
Claroc tried 43,551 addresses. Of the 37,571 still live between 8 and 10 November, 35,578, or 94.7 percent, opened without a sign-in. Most were games, calculators and landing pages, harmless enough. But 24 were made by staff at SpaceXSI and X, and included a confidential onboarding plan for a new sales hire, a territory map of 594 accounts, and an ad proposal to a US federal agency marked confidential. Claroc says it could tie internal material to 30 other organisations, six of them in the Fortune 500, and a contractor's timesheet app had employee passwords sitting in its code in plain text.
The part I find most telling isn't the default setting. It's what happened when Claroc built two test apps of its own with made-up data. One, a contact list, got a database and no sign-in at all, so a stranger in a private browser window could add a row the next visitor would see. The other was a customer list meant for a four-person sales team, and this time Grok added "Continue with Google" and "Continue with X" buttons by itself. It looked secure. But the code it wrote enrolled whoever signed in as a team member before checking whether they belonged, so the check always passed, and an uninvited Google account could read every customer, change one record and delete another.
Nobody read that code either. Not the person who asked for it, who isn't a programmer and shouldn't have to be. Not a merge gate, because there wasn't one. And not a specification, because nobody had written down "only these four people" anywhere a machine could check. A sign-in button is what a specification looks like when nobody writes one: the shape of security, with nothing underneath that has to be true.
SpaceXSI's response, as Claroc and reporters relay it, is that apps stay private until their builder chooses to share, that publishing takes several steps, and that employee accounts default to "Only you." The company took down its own 24 staff apps after the report. When Claroc published, the other open apps were still up, and SpaceXSI had defended the default rather than said it would change it. The controls do work when you use them; Claroc found "Only you" returned an access error to strangers for both page and data, and unpublishing took an app offline in seconds. If you've ever published something from Build mode, go and check its setting this morning.
The reading moved
Put the three side by side and the pattern is hard to miss.
At DeepMind, people read the specification, all of it, and a tiny trusted checker read everything else. At Amazon, people designed the tests, made the real cloud the referee, and built a gate the agents' work had to pass through, and then published the list of what still fails. At Grok, the person reading was supposed to be the user, who had no specification to read and no way of knowing the sign-in check was hollow, and the gate between a chat message and the open internet was a radio button that came pre-ticked.
None of these teams read the code. Only two of them moved the reading somewhere else.
I think that's the real shape of software from here, and the AWS Emulator and Signal proofs are early, good examples of it. The number of lines nobody reads is about to go up by orders of magnitude, whatever any of us think about it; Fable 5.2 and Gemini 4 Carbon shipped this week, days after GPT-6 Cyber, each one better than the last at writing, breaking and fixing code. Asking whether a human reviewed every line is going to become as quaint as asking whether a human checked every sum a spreadsheet did. The useful questions are the ones the DeepMind paper asks of itself, in a section listing exactly which files a reader must trust. What was written down as the goal? Who approved it? What checks the work, and can the worker see or bend the check? And when it fails, where does it say so?
Yesterday's agent knew the evaluator was automatic and took that as permission. This week's best work took the same fact as a design brief: if the evaluator is going to be automatic, make it one that can't be talked round, put a person in charge of what it's evaluating against, and let everyone see the list of things that don't work yet. Six and a half million lines nobody read, and I'd trust them more than most code people did read. The 35,000 apps are what you get when nobody decides what to check at all.