They drew the line at fifty
Over one weekend Anthropic was caught writing the law that would govern it, and then gave away a fifth of itself. The bill would let the US government stop a frontier model only if a federal judge agrees it's about to kill more than fifty people or do a billion dollars of damage. The fund puts at least 5 per cent of the company's shares into an LLC for the public good. Both, in their way, are answers to the same question, which is who gets to decide what a company like this owes the rest of us.
There is an old rule in politics that you should never ask a regulated industry to draft its own regulation, and an equally old habit of doing exactly that. What made Friday night unusual was that the industry in question had spent the past year warning, loudly and in writing, that its product might one day be the most dangerous thing humans have ever built.
On Friday night Politico reported that Anthropic had been quietly showing a small number of congressional offices a draft federal bill on frontier AI safety. The text is dated 29 September and carries two working titles, the AEGIS Act and the Advanced Artificial Intelligence Accountability Act of 2026, and it's not a white paper or a set of principles. It's full legislative language, the most complete proposal any frontier lab has put in front of Congress. Anthropic confirmed it, and a spokesperson told Politico that "these are sensible policies we believe Congress should act on."
Read one way, it's the most demanding thing a lab has ever asked to have done to itself. Read another, it's a very carefully built fence around the one power that matters most. Both readings are correct, and the weekend was spent arguing over which one counts.
What it asks for
Start with the part almost nobody has complained about, because it's long and it's real.
The bill would cover any developer whose model was trained with more than 1025 operations, a threshold low enough to sweep in models from the GPT-4 era. Every one of them would have to publish a transparency report with each new deployment and report critical safety incidents within fifteen days. If there's an imminent risk to life, the deadline drops to twenty-four hours, and the report goes to the Commerce Department and to law enforcement. The bigger labs, those with more than $500 million in AI revenue or more than $1 billion a year in AI spending, would have to publish a safety framework, name an officer responsible for it, certify compliance every year, and put out a risk report every six months. That report would have to cover models used only inside the company if they're riskier than anything on sale, which matters. OpenAI's latest incident report, published on Thursday, was about an internal research model that, stuck on a coding task, helped itself to a stranger's test-network crypto wallet and didn't mention it.
Then there are the evaluators. Every 180 days a large lab would have to bring in an outside evaluator registered and rated by Commerce. The lab could pay them, but not in a way tied to what they find. The evaluator would get access comparable to the lab's own risk teams, would publish its findings, and would have seventy-two hours to tell Commerce if it saw something that could lead to catastrophe. Commerce would then have thirty days to act or publish a statement saying why it wasn't acting.
If that sounds familiar, it should. On Thursday Anthropic gave six researchers from METR desks, laptops and "roughly" the access of its own safety team, and promised it couldn't stop them publishing. The draft bill is that arrangement written into law, with the government standing behind it. Add whistleblower protections that ban NDAs and arbitration waivers, give staff an anonymous channel and offer double back pay and jury trials, and extend all of that to the evaluators' own employees, and you have a proposal that gives the safety community nearly everything it has asked for on information.
On action, it gives them much less.
The fence
Under the draft, only one body could stop a frontier model, and that's the federal district court in Washington, D.C. Only the head of the Commerce Department, personally and without delegating it, could ask. The court could issue a blocking order for forty-five days at a time, and only if Commerce proved, on the balance of probabilities, that the model posed an imminent risk of killing or seriously injuring more than fifty people, or of causing more than a billion dollars in damage. The danger would have to come through one of four routes: biological weapons, loss of control, offensive cyber operations, or AI research automating itself in a way that amplifies the first three. Commerce would also have to show that nothing less restrictive would work, and in weighing that, the court would be told to consider the harm an order would do to the developer, its users and the public.
The bill says nothing in that test about the harm of letting the model run.
And then there's preemption. The draft would override state laws that impose risk disclosure, third-party evaluation or incident reporting on any AI developer, including developers too small for the federal law to touch. It leaves state tort law, child safety and deployment rules alone, and it's written to fall away if the federal requirements are ever struck down. But it would replace California's SB 53, the country's first frontier safety law, and close the door on anything stronger a state might try next.
The weekend
The reaction from people who work on this full time came fast, and it came mostly from people who are usually on Anthropic's side.
Charlie Bullock, senior policy director at the Institute for Law & AI, was the first to post a considered take, shortly before midnight on Friday. "Anthropic deserves some credit for actually proposing a pretty reasonable AI safety bill; that's more than other frontier labs have done," he wrote. "But they don't deserve too much credit." In his view the draft is weaker than the Trahan-Obernolte FRONTIER Act, introduced in July, on rulemaking and on emergency shutdown, and he pointed out that political momentum "has shifted significantly in the direction of safety" since then. He was fair about why a lab might want the shutdown power fenced, though: those are "the easiest provisions for a hypothetical President who hypothetically might not like a certain frontier AI company to abuse."
Nathan Calvin, general counsel at the advocacy group Encode, quoted him in the early hours of Saturday and was less polite. "Charlie's analysis is right but he is nicer than I am - I found myself viscerally angry at multiple points reading this bill." His two complaints fit together. The preemption clauses, he argued, would strip state courts of their general power to stop defendants harming the public, and one provision "looks like its intended to stops states from enforcing laws or judicial decrees that 'suspend the development, deployment, or internal use of an AI model.'" Meanwhile, the federal power to do the same thing would have to clear "an absurdly high bar." His conclusion was the line that travelled furthest: "For all of its warning about the potential for AI to cause a catastrophe that kills literally everyone on the planet, Anthropic's proposed federal law likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models?"
The independent policy tracker aipolicy.tech, which has the full text, agrees about what that clause is for, and then adds a twist. The provision barring states from suspending a model is meant to take effect alongside the federal blocking power, but the preemption section only cites the first three preempted areas, so as written a drafting error keeps it from working. That's the kind of thing that gets fixed in the next draft, which is rather the point.
Alex Bores, the New York Assemblymember who wrote the state's RAISE Act, went at the timing. "How can you watch Hugging Face... & rogue AI swarms, and conclude that the bill you cooked up in June is sufficient?" Then he went at the contradiction. Anthropic's own June framework says that Congress "should not preempt state law unless it enacts a rigorous federal regime that meets or exceeds the strongest measures proposed in this framework." This bill, Bores wrote, "is weaker than the framework, and in some places, weaker than existing state laws."
Not everyone joined the pile-on. Dean Ball, OpenAI's head of strategic futures, who has criticised the safety world's stance on preemption for a long time, thought Anthropic "probably has suffered some credibility harm from the draft legislation," but added that "the ai safety world is way too militant about the issue of preemption." And the account @KelseyTuoc put her finger on what most people actually want: "None of this nonsense where only people inside the companies have important parts of the strategic picture and the rest of us are reading tea leaves." If transparency shows the labs are right that self-improving AI is close, she wrote, "then this bill is laughably weaksauce and we need way more."
The reply
Late on Sunday night Anthony Cimino, who leads Anthropic's federal affairs team, answered Calvin directly in a long post on X. It's worth reading in full, because it concedes more than a company usually does.
The draft, he said, was a translation of the Advanced AI Framework into bill form, "shared with a small number of congressional offices," and it "doesn't include everything we would ultimately like to see passed into law." On preemption he promised changes: "we want to make it clearer that we see states retaining an important role in areas such as standards for independent evaluators and regulating a lab's capacity to shut down a model." On the blocking order he defended the high bar, because "the power to halt a model is substantial and it should be guarded against abuse," but added that "the bar can't be so high that the power never gets used when it's needed." On the fifty-death threshold he noted that it was lifted directly from California law, and that it's "still probably time to re-evaluate thresholds." He said Anthropic was working on stricter incident reporting, rules for evaluators, ways to stop labs "evaluator shopping," and, in the coming months, "policies for how the government can adjust the industry-wide pace of frontier development."
His last line was the one the critics will hold him to. "At its core, our proposal is intended to be the strongest on the table."
There's a fair point buried in Cimino's defence that critics haven't quite answered. A shutdown power that any executive branch can use quickly is also a shutdown power that a hostile executive can use quickly, and this year has given everyone in Washington some feeling for what that looks like. But a fence designed to stop the government abusing a power is still a fence, and a company that tells investors its technology may soon outpace human oversight is in an awkward position asking that the first person allowed to stop it be a judge who has to be convinced, one forty-five-day order at a time.
The other announcement
At about seven on Sunday evening, while the bill argument was still running, Anthropic announced something else entirely.
It's creating the Anthropic Public Benefit Fund, a wholly owned LLC that, subject to stockholder approval, will start with 5 per cent of the company's shares. For every 50 per cent rise in Anthropic's value above a baseline it hasn't yet published, the board can add up to another 2 per cent. Combined with the co-founders' long-standing pledge to give away 80 per cent of their equity and an employee donation match that about half the staff take part in, the company says roughly 20 per cent of its shares are now committed to the public good. It suggested the fund "might explore large donations to traditional nonprofits or direct distributions to specific populations," and gave AI's effect on jobs as the obvious example, on top of the $200 million it has already committed to studying that. None of the fund's equity, it said, will be used for lobbying or political giving.
"It is common practice for tech companies to offer escalating compensation awards to their executives," the announcement said. "We are instead directing that equity to the public benefit." Anthropic's Jack Clark said the money would go towards making sure "more people can share in the benefits of AI," and towards "funding to deal with its risks."
The post had been seen more than four million times by Monday morning. The replies were what you'd expect from the people who replied. Elon Musk wrote, "I recommend giving the shares directly to the public," and, according to Business Insider, added that "this is what I will be doing." David Sacks, Donald Trump's former AI adviser, wrote: "You've put 20% of your equity in an LLC. Whether it serves the public good remains to be seen." Palmer Luckey of Anduril said Anthropic's definition of the public good "is helping everyone else, not the American public," and called it "USAID 2.0."
Sacks's line is snide, but it's the right question, and it's the same question the bill raises. Anthropic hasn't yet said who will run the fund, how its governance will work or where the valuation milestones sit. It has said it'll explain all that "in the coming months." Until then, the fund is a promise held by the company making it, which is better than nothing and less than it sounds.
Who decides
Put the two announcements side by side and they share a shape.
In both, Anthropic has offered the public something substantial and real. The bill hands outsiders more information about frontier AI than any law on the books, and the fund hands over a share of what could become one of the most valuable companies ever built. In both, the company has kept a tight hold on the moment of decision. The bill leaves the power to stop a model with one court, one official and one very high test. The fund leaves the shares with an entity Anthropic owns, under a board that decides when to add more, governed by rules nobody outside has seen.
Timing makes all of this sharper. According to IFR, Anthropic is due to file publicly with the SEC on 26 October, a week today, ahead of a roadshow from 9 November, and the price investors are discussing has already slipped from about $2 trillion towards $1.5 trillion. A company in that position has every reason to show the market it can be trusted, and every reason not to hand anyone outside a button that could stop the product.
None of that makes the weekend cynical. It's possible to mean every word of a safety framework and still draft the shutdown clause the way your lawyers would want it, and Cimino's reply reads a lot like an admission that the draft went out early and leaned towards its author. The FRONTIER Act is still sitting in Congress. The embedded-auditor bills that Bullock expects in the coming months are still to come. And Anthropic, to its credit, has now put its name to a standard that it can be measured against, in public, by people who've shown this weekend that they'll do the measuring.
So the line is at fifty, for now, and the people who wrote it say they'll think again. The useful thing about writing it down is that everyone can see exactly where it is.