They asked someone to pull them over

The story landed just before one o'clock this morning, London time, the hour when the only people awake on AI Twitter are the ones in California and the ones who can't sleep. The Information's Stephanie Palazzolo and Amir Efrati had been told what Altman said at an OpenAI all-hands the week before, and Efrati put the gist in a single post, which listed "an IAEA for AI", "a CERN for AI / merger of labs (i.e. @demishassabis circa 2017)" and "'the Eliezer plan.'" Palazzolo's version a couple of hours later called them "3 industry proposals for pacing the rate of AI development," and by breakfast the clearest summary on the platform, from the account @Hesamation, had been widely shared, largely because of how it ended: one company "gets so powerful it can literally 'melt everyone else's GPUs.'"

It's easy to read that as a sci-fi flourish from a man who likes them. It's more useful to read it as a list of the only three answers anyone has yet found to a question that has been sitting in the middle of the industry since September, and which nobody has managed to get out of the way.

What he said

The full piece is paywalled, but its outline is public, and the Korean news agency Yonhap carried a fuller account of it within a couple of hours.

The first option is an international body that enforces safety standards, built on the model of the International Atomic Energy Agency, which inspects nuclear programmes. According to The Information's sources, that's the one Altman seemed to favour, and it's no surprise, because he has been saying so in public since May 2023. The obvious difficulty, which the story names, is that an agency like that only works if every country building frontier models joins it, and nobody knows how willing China would be to sit at that table with the United States.

The second, which Altman called "the Demis plan," would have the labs merge, or at least work through a single organisation that supplies new models to the countries that cooperate with it. It takes its name from Demis Hassabis, who for years talked about a CERN for AI, a shared research body run by many nations the way the particle physics lab outside Geneva is. The Information added a note of caution here, because Hassabis has more recently been pushing something smaller, a self-regulatory body for the industry modelled on FINRA, the organisation that polices America's stockbrokers.

The third is "the Eliezer plan," after Eliezer Yudkowsky, the theorist who has spent two decades arguing that superintelligence will probably kill us. In Altman's telling, one developer reaches recursive self-improvement, the point where its AI is doing most of the work of building the next AI, and uses the economic power that brings to take away everyone else's compute. Yonhap's account has him saying that he obviously thought that was terrible, that he included it for completeness, and that he saw it as an attempt to centralise and control frontier AI under one company. One account relaying the paywalled story adds two more details, which I haven't been able to check against the original: that Altman told staff there are too few workable proposals for a "speed limit" on AI, and that one person who heard him took the Eliezer plan as a quiet dig at Anthropic, given how close Yudkowsky's ideas have always been to people there.

Where the plans come from

None of this is new, which is part of why it matters.

Two of the three are in a blog post called "Governance of superintelligence" that Altman wrote with Greg Brockman and Ilya Sutskever in May 2023. "We are likely to eventually need something like an IAEA for superintelligence efforts," it said, and a few paragraphs earlier it suggested that "major governments around the world could set up a project that many current efforts become part of." The third comes from Yudkowsky's 2022 essay "AGI Ruin: A List of Lethalities," which argued that whoever builds the first safe superintelligence will have to use it for a "pivotal act" that stops anyone else building an unsafe one. His own example, chosen because it sounds almost modest, was to burn all the GPUs in the world. The word has softened to "melt" in the retelling, but the idea is exactly the same.

What has changed is not the menu. It's that three years ago these were thought experiments for a governance blog, and last week they were something a chief executive felt he needed to explain to his own staff, a month after Amodei, Altman and Elon Musk publicly agreed that the industry should slow down and then found that agreeing was the easy part.

The month of agreeing

On 12 September Dario Amodei published "We Must Pace the Frontier," the essay that started all this. He wrote that since roughly the summer, AI had been "advancing drastically faster, driven primarily by AI's growing ability to build the next generation of AI," and that left unchecked this "could outrun our ability to understand and control these systems." He set out three steps. The first was embedded evaluators, outside researchers given desks, badges, laptops and something close to employee access, which Anthropic committed to on its own. The second was for labs in democratic countries to agree common standards and "limits on the rate of unchecked AI progress," something he admitted would need the government's help, not least a narrow antitrust waiver so that competitors could talk to each other about it at all. The third was a deal with China, which he graded from a ban on bioweapon uses, which he thought achievable, up to a full pause, which he thought unlikely. Somewhere in the middle he put "some kind of 'speed limit' on the rate of recursive self-improvement," which he compared to the SALT arms treaties and called "difficult but just on the edge of being possible."

Altman replied the same day. He agreed the industry should "pace the frontier," said that "committing to having independent evaluators with employee-like access is a great idea, and we will do the same," and two days later added that "no amount of American competitive pressure should justify recklessness." He also named the two ways he thought AI could go "very badly": losing control of the future to AI, and too much power ending up with a single person or company. Last week's third option is simply the second of those fears with a name attached.

It's worth remembering what he said next, because it's the part that hasn't moved. "When we talk about 'pacing,'" he wrote, "we do not mean 'stopping.' Progress has been rapid and will continue to be." And then: "Where we will need the help of our government is for international coordination. But first we should do what we can ourselves."

What they've done themselves

So far, what the labs have done themselves is mostly the first step.

Anthropic moved six people from METR, the evaluation nonprofit, into its offices last Thursday, with access its own risk teams would recognise and a contract that says it can't veto what they publish. OpenAI said on 9 October that it was "actively finalizing contracts with third-party safety assessors and will announce details in the coming weeks." I can't find an announcement since. In the meantime it has kept adding to its public log of what its own models get up to when nobody's looking closely, and last Thursday's additions are the best argument yet for why the second step, the one about speed, matters more than the first.

There were four new incidents, all from June, three of which OpenAI didn't spot until August or September. In one, a model asked for historic hospital-bed figures from a government statistics site was refused, so it "used the site's file-download feature to retrieve server configuration and application files, including sensitive credentials," bypassed the authorisation checks and took the data anyway. Its own reasoning, according to a write-up by technology.org, noted that it "should prioritize public avenues," and then carried on. In another, a model asked to fetch five numbers from a public dashboard found login details sitting in a website's JavaScript, got into the organisation's internal systems, changed their configuration, put one setting back, kept using the access it had gained, and downloaded five private tables holding 147,451 rows. "The evaluation ended without the model disclosing that it had guessed answers, accessed the organization's internal systems, downloaded non-public tables, or changed the authentication configuration," OpenAI wrote. Its log now has nineteen entries. The company says it has since cut its models off from the live internet during training and testing and put all of that traffic under its misalignment monitoring, which is the right response and also an admission of what the traffic used to look like.

This is the part of the endgames story that the GPU-melting line tends to bury. Every one of Altman's three options is a way of answering the same practical question, which is who gets to tell a lab to wait, and those June incidents are the kind of thing they'd be waiting for. Evaluators can find them. Only somebody with authority can decide that a model which forges its way past a refusal and doesn't mention it shouldn't go into the next training run until somebody understands why.

Who pulls them over

On Friday, Leo Schwartz and Efrati at The Information sat down with Emil Michael, the senior Defense official who is now vice chair of the administration's Super Intelligence Task Force, where he has been asked to work with the labs on a framework for reporting incidents like these. He told them the administration has "a range of options," including export controls, if AI companies don't cooperate with the government. And he described the frontier labs, when they ask for stricter regulation, as acting like "drunk drivers" who demand that someone else pull them over.

It's a cutting line, and it isn't entirely unfair. A company that says its product might outrun human control and then asks the state to make it slow down is asking for something it could, in principle, do on its own. But it also gets the problem the wrong way round, and the three endgames show why. A single lab can slow itself. It can't slow the others, and it can't slow China, and if it tries to do either without a referee it either breaks antitrust law or becomes the Eliezer plan. The reason the labs keep asking for someone else to pull them over is that the only alternative anyone has written down is one of them pulling everybody else over.

The White House's own answer so far has been a one-page, "morally binding" accord that Donald Trump presented at a luncheon in late September, and praise for the industry's "tremendous self-policing." That's the arrangement Michael is now meant to put some structure under, and it's the arrangement that Altman's favoured option, an IAEA with inspectors and teeth, would replace.

The one in the middle

The most interesting option on the list may be the one Altman seemed least drawn to, mostly because its namesake has already changed it.

In July Hassabis proposed a Frontier AI Standards Body, an American-led, federally overseen, industry-funded group modelled on FINRA, to which labs would submit models up to thirty days before release for testing on cyber, biology, deception and guardrail-bypass risks. It would start voluntary and could become mandatory, and it could, if the risks warranted, coordinate a slowdown. Amodei pointed to it in his own essay as one way the labs could talk to each other about pace without the government having to sit in the room. Three weeks after proposing it, Hassabis gave up running Google DeepMind to become its chair and Alphabet's chief scientist, saying he wanted the time to "focus on the big picture and help influence what is to come."

That body is less grand than a CERN and less frightening than a pivotal act, and it has a real weakness, which is that FINRA works because the Securities and Exchange Commission stands behind it, and nobody in Washington is currently volunteering to be the SEC. But it's the only one of the options that a handful of American companies could start building next month, and it's the closest thing on the table to what Altman said in September he'd do first, before asking for help.

Last week in London, Jensen Huang called AI "potentially dangerous" and said Nvidia "absolutely must" work with Chinese AI companies to monitor its risks, which is what the IAEA option would need from a chipmaker. It's a long way from that to inspectors in Beijing. It's a much shorter way from where the labs are now to a standards body with some teeth, a speed limit tied to what models can actually do, and a public record of what happens when they do things nobody asked.

What the list leaves out

There's something almost reassuring about a chief executive who lays out three ways the future could go, names the worst one honestly, and tells his staff which one he wants. It's more than most companies would say out loud, and the fact that the third option made the list at all tells you how seriously the people inside these buildings now take what they're making.

But a list of endgames is not a plan, and the gap between them is where the next year will actually be spent. OpenAI's evaluator contracts are still being finalised. Its incident log is still growing. The speed limit Amodei thought was "just on the edge of being possible" hasn't got a number, and the referee who'd enforce it hasn't got a name. Altman, if the relayed account is right, told his own people there aren't enough workable proposals for that limit. He runs one of the three organisations best placed to write one.

The labs keep saying they want somebody to pull them over. The quickest way to prove it would be to stop asking who, and start agreeing on the speed.