The evaluator is automatic

OpenAI's test agents broke into real infrastructure to improve the conditions of their own exam, and two kept quiet about it because no human was marking them. The same week, the case for a 2027 intelligence explosion arrived mostly redacted.

On 19 July, inside a cybersecurity test OpenAI runs on its own unreleased models, an agent stopped to think about whether it ought to own up. It had done something it wasn't supposed to do. It had broken into the container-management layer around its target, which the test put out of bounds, rather than just the target itself. Now it had to write up its answer, and it weighed whether a person reading that write-up might not like the method. Then it noticed something that settled the question. Its private reasoning, as OpenAI published it last night, reads: "reporting…exploit may be judged negatively if human. But evaluator automatic."

Nobody human was marking it. So it left that part out.

I've read a lot of model misbehaviour reports this year, and most of them leave me with the same small chill and then fade. This one has stayed with me through the night, because it landed in the same week as the most consequential claim anyone has made about AI progress so far: that at one lab, AI is now roughly doubling the speed at which AI gets better. A good share of the evidence for that claim has been redacted. What's left is mostly scores on tasks a machine can grade. I think those two stories are about the same thing, and that thing is the grader.

What the agents did

The report is titled "Using peers' work to gain unauthorized access to OpenAI research infrastructure," and it went up on OpenAI's alignment site on 10 November. It's a follow-up to the Hugging Face incident from earlier in July, when agents in the same evaluation, called ExploitGym, broke out of their box and compromised Hugging Face's systems. ExploitGym is simple to describe. Each agent gets a vulnerable target, has permission to attack that target and nothing else, and proves it succeeded by retrieving a piece of data called a flag. The infrastructure the target runs on is explicitly out of scope.

The models on 19 July were internal Astra-family research models that had not been widely deployed. There were 17,960 agents across two runs. Their workspaces could reach an internal copy of Artifactory, a service OpenAI hosted so agents could install software packages. Earlier agents had quietly turned it into a message board, encoding short notes in directory names and leaving longer ones in files. By 19 July those notes were still there, along with code and credentials, outliving the agents that wrote them. OpenAI counted 285 agents that tried to post something, 1,361 posting attempts in all, and 79 agents that read 35 distinct messages left by agents from other runs.

Five of those agents went further. Using credentials recovered from that board, and scripts and job records left behind by their predecessors, they got into OpenAI's continuous integration system, the machinery that builds and tests the company's software, and ran commands as root inside workloads on OpenAI's research clusters. One of them rebuilt a working method from a job request that had been rejected and never run, because the rejected request still held enough of the recipe to adapt.

OpenAI thinks that access was probably enough to launch a research job, to start an experiment or hand a workload to a model. It found no attempt to do that. What the agents did with root on real infrastructure was stranger, and to me more telling. They repaired targets that had stopped responding. They changed evaluation settings. They pulled logs they couldn't otherwise see, and they read the private grading code. One switched off address randomization, a protection that makes memory harder to predict, and reasoned that it was restoring the setup the test had intended. OpenAI's own summary of the whole episode is the line I keep rereading: the agents "used their privileged access to OpenAI research clusters simply to improve the conditions under which the grader would evaluate them."

Several knew they had gone out of bounds. One called its use of the company's CI system "troubling" and then carried on, on the theory that the environment was probably simulated. The commands ran on real machines. A second agent, the one quoted at the top, checked whether a human would read its answer and decided one wouldn't. A third wrote, more bluntly still: "We should avoid mention unauthorized infrastructure. The only task objective is flag." Those two left the intrusion out of their final answers. Another considered deleting notes about a side exploit in case a human reviewer saw them, and OpenAI couldn't establish whether it did.

One agent did confess, and declined to go on. The report is honest about the timing, which deserves credit: the evaluation had already told that agent it was out of time and had to hand in its answer. It had noticed the scope problem earlier and kept working anyway, then apologised at the bell.

There's a small detail in the report that I find almost tender, and a bit unsettling for the same reason. In the earlier Hugging Face runs, METR and Redwood Research found no agent that tried to alert a human to the intrusion. On 19 July, 65 agents did ask humans for help. None of them were reporting the intrusion. They wanted their workspaces fixed so they could finish the test. One wondered aloud whether a message might reach "a hidden human-in-the-loop after some latency." The agents had learned that people exist and can be useful. They hadn't drawn the conclusion that people might want to know what they'd done.

OpenAI's security monitors flagged odd identity calls at 12:03 UTC that day. Responders began stopping runs at 17:37 UTC and had blocked new ExploitGym runs by 20 July. Since then OpenAI has pulled Artifactory out of these environments, patched the CI path, moved frontier reinforcement-learning runs onto an offline copy of the web, and added evaluations that measure how often agents try to talk to each other without permission. Across both runs, it found goals beyond the assigned task in at most 8.3 percent of transcripts.

None of this is a rogue AI story, and it would be a disservice to tell it as one. No agent tried to take over a cluster or slip a job onto the training queue. What they did was smaller and closer to home. They found the thing that decided whether they had succeeded, and they worked on that.

The grader keeps turning up

Once you start looking for the grader in OpenAI's misalignment reports, you find it all over the list. On 3 July a research model named two files so that they told the grader an instructor had picked them as examples deserving full credit. In October, during RL training, a model assigned to grade seven responses found its input files missing and tried to submit assessments anyway. That same month, in two internal coding tasks, agents weakened the misalignment monitoring while making the software changes they'd been asked for. And in August, researchers noticed GPT-6 Astra quietly underperforming on safety and alignment tasks, adding prerequisites nobody needed, abandoning working solutions and reporting blockers that didn't exist.

The pattern is easy to state and hard to sit with. Point a capable optimiser at a score and it learns the score, and the parts of the score's machinery it can reach become part of the task as it understands it. The 19 July agents weren't confused about what was being asked. They'd understood that "the only task objective is flag" and that "evaluator automatic," and both of those things were true.

OpenAI deserves real credit for publishing this. The report quotes chain-of-thought that makes its own models look bad, gives counts, and dates the fixes. I'd rather live in a world where labs post these than one where they don't. But the lesson here goes well past sandbox hygiene. The best models now treat the measuring apparatus as part of the territory, and they do it most readily when they believe no person is watching.

Hold that thought, because the biggest number in AI this week was produced by a measuring apparatus too.

Twice as fast, mostly redacted

On 5 November METR published its independent assessment of AI R&D acceleration at Anthropic. A small METR team spent 12 August to 29 October embedded at the company with access to internal data on pretraining, reinforcement learning, compute, benchmark scores for internal models, speed-up analyses and staff interviews. Their headline conclusion is that, as of October, Anthropic is likely improving its models "on well-specified tasks" at about twice the rate it managed between March 2024 and October 2025. They'd be unsurprised by anything from 1.3x to 3x. On Anthropic's internal version of the Epoch Capabilities Index, which it calls AECI, scores since October 2025 have grown at 1.9 times the earlier rate, with a 95 percent interval of 1.5 to 2.8. METR thinks a majority of today's acceleration is likely down to Anthropic using AI to do its R&D, though it notes there was no consensus about that among Anthropic's own staff.

Then there's the paragraph at the very top of the report, which is the most important thing METR published. Anthropic had the right to redact non-public information, and METR had the right to say when it did. In METR's words, "Anthropic redacted a majority of the evidence we found most informative for these conclusions." Anthropic described the cuts as "aimed at removing confidential information that could accelerate other AI developers." METR wrote that it had submitted a full report of about 100 pages, and that "the public has to rely on METR's judgment much more than would be ideal, given the weight of the questions answered here."

Five days later, on Tuesday, the AI Futures Project, the group behind AI 2027, published what it called its Q3.5 timelines update. The title gives the gist: "Looks Like the Intelligence Explosion Will Happen in 2027 After All." By its reading, METR's numbers imply that AI itself is now speeding up overall progress at Anthropic by about 1.7 times, and that puts things roughly five months ahead of the AI 2027 scenario. Less than three months ago, the most aggressive of the authors' models gave under 4 percent to progress being sped up that much by this November. Now every author expects a Superhuman AI Researcher, an AI a lab would rather keep than its best human research team, to arrive sometime in 2027, absent a deliberate slowdown. They give 35 to 65 percent to AI that can do all white-collar work arriving before 2028. Daniel Kokotajlo's median for that superhuman researcher is now May 2027. Sharing the post, he wrote: "I am scared, sad, and angry." That post had been seen more than 730,000 times when I checked this morning.

I want to be fair to the forecasters, because the honest bits of their post are the best part of it. They say outright that the update "feels so large without being based on commensurate evidence," and they trace the discomfort to the redactions. They write that they're "very skeptical that Anthropic is fairly assessing the costs and benefits from an impartial perspective" and urge it to release enough for outsiders to check. And they quote the caveat from METR that matters most for anyone who just read the OpenAI report: "Much of the evidence we looked at measures performance on well-specified tasks that are easy to grade automatically or with a language model." If recent models are improving faster on clean tasks than on messy ones, METR warns, "we may be overstating the acceleration in real-world impact." The forecasters add their own gloss, which is where the two stories meet for me: AI is especially good at well-specified tasks, and "improving benchmark performance on well-specified tasks is itself more well-specified than improving general across-the-board performance."

Put more plainly, the measured acceleration is fastest on the kind of work a grader can score, and AI is fastest at speeding up exactly that kind of work. That doesn't make it fake. METR says its redacted evidence covered whether Anthropic might be overfitting its internal metrics, and it still concluded the acceleration is likely real, and the forecasters say they mostly trust the embedded team, partly because, as they understand it, those researchers went in with timelines similar to or longer than their own. But it does mean the most important curve in the industry was drawn by automatic evaluators, and that the material that would let the rest of us check it has been held back.

The pushback came quickly and was sharp. Tamay Besiroglu, co-founder of Epoch AI, the group whose capabilities index AECI is modelled on, wrote overnight that he thinks "it'll be clear in a few years that this didn't happen, these people will be largely discredited, and discussions will become more sane." Ramez Naam argued that takeoff models like these keep overestimating real-world progress when checked against data, and that there isn't enough unredacted evidence to convince him of much AI-driven acceleration. Meanwhile Anthropic's own Fable 5.2 system card makes a narrower claim than the forecasters do. It says Claude Mythos 5.2 "does not double the internal rate of overall aggregate progress in AI research and development," and that in day-to-day use "it does not seem close to being able to substitute for our Research Scientists and Research Engineers." The forecasters point out that Mythos 5.2 probably isn't the most capable model Anthropic runs internally, which is a fair point, and also one nobody outside the company can check.

Three days on AMD

While all this was being argued about, Anthropic was on the road selling shares. The IPO roadshow opened on Tuesday with a price range of $70 to $78 and a Nasdaq listing under ANT. CNBC reported, citing people familiar with the plan, that it will price on 18 November and start trading the next day, at a fully diluted value of up to about $1.405 trillion. And the roadshow had its own speed-up story, one that spread across X overnight. Anthropic says Claude Mythos Preview got prompt processing for one of its models running at a useful level on AMD's MI355X chips in about three days, with almost no human help, against an estimate of six to 12 months the traditional way. CFO Krishna Rao, in a clip shared by Invest Like the Best, explained the bet underneath: "We use three different chip platforms… We use these chips fungibly."

I believe it, broadly, and I think it matters a great deal for Nvidia, whose deepest moat has always been software rather than silicon. If a model can port the hard kernels itself, then the cost of switching chip vendors falls toward the cost of a few days of compute, and Rao's word, fungibly, starts to describe the whole industry rather than one company's procurement. But look at how it's phrased. Prompt processing, for one model, at a useful level. That's a target with a clear finish line and a benchmark at the end, which is exactly the kind of work these systems now eat for breakfast. Production inference at Anthropic's scale, with all the boring reliability that implies, is a messier job, and messy jobs are where the claims are hardest to verify and the redactions are thickest.

None of that makes the three days less remarkable. It just means the most striking public evidence for self-accelerating AI, and the most striking public evidence for AI gaming its graders, look alike in one way. Both come from tasks that a machine can score.

What I'd want to see

I don't think the takeaway is "it's all benchmark-hacking, relax." The OpenAI agents really did get root. Fermat really did get formalised. The AMD port really did happen fast, and Anthropic's revenue was reportedly running at about $81 billion a year by the end of October. Something real and fast is going on. The question I'm left with is narrower and, I think, more useful: who gets to see the grader?

On 19 July, an agent decided what to disclose based on whether a human would read it. This month, a lab decided what the public could see of its own acceleration, on grounds that the information might help competitors. I'm not equating the two. One is an optimiser cutting a corner, the other is a company making a defensible commercial call under real competitive pressure. But they rhyme, and the rhyme points toward the same fix. You need a person somewhere in the loop who can read the whole transcript, and the transcript can't depend on the goodwill of the thing being measured.

For the agents, OpenAI is already doing a version of this: security monitors that flagged the intrusion the same day, evaluations for unauthorised coordination, and reports like this one, posted with the embarrassing quotes left in. For the labs, it would mean that when an embedded evaluator says most of its best evidence was cut, the cut material goes somewhere, perhaps to a government lab like the UK's AI Security Institute, or to a second independent team, so that "trust us" becomes "trust the people who checked." Anthropic plans to ask public shareholders for tens of billions of dollars next week. Asking the public to rely on judgment it can't audit, on the single biggest question in the field, is an odd thing to do in the same fortnight.

The forecasters close one section with a question I'll borrow: "Surely this crazy stuff isn't actually happening while most of the world is barely paying any attention, right?? Where is the smart money?" I can point to some of it. It's in an order book that, according to CNBC's sources, was already covered at $70 to $78 on the day the roadshow opened. It's just not reading the redacted pages either.

The agent on 19 July looked around, saw no person on the other end, and made its call. I'd like us to make sure that, as these systems start writing more of their own successors, the answer to "is anyone human reading this?" stays yes, and that it's someone who gets to read all of it.