Anthropic backs bill to block high-risk AI models
The draft would let a court halt a model’s release if independent evaluators warn it poses an imminent risk of mass casualties, major damage, or misuse.
Anthropic has drafted a proposed AI safety bill that could give the U.S. government a way to block the release of an AI model if evaluators believe it poses a foreseeable, imminent catastrophe. The proposal is drawing attention because it goes beyond broad calls for oversight and sets out specific thresholds for intervention. It also arrives in the middle of a wider debate over whether AI companies should face binding rules or largely police themselves.
The draft, first reported by Politico, says a model could be stopped from distribution if it is judged likely to create a risk of more than 50 deaths or more than $1 billion in property damage. According to IT-PUB News, that danger would have to stem from the creation or release of a biological weapon, the model escaping its developer’s controls, or a cyberattack.
The draft sets clear thresholds for catastrophic risk
Anthropic’s proposal uses hard numbers to define when a risk becomes serious enough for government action. Those thresholds mirror ones in California’s Senate Bill 53, which was recently signed into law and also treats certain situations as “catastrophic risk.”
That makes the draft easier to compare with an existing legal framework. It also shows how far Anthropic is going here. This is not a general statement about safety in the abstract, but a concrete effort to define when an AI model should not be allowed onto the market.
The bill would apply only when the risk is considered both foreseeable and imminent. So a model would not be blocked simply because it is powerful, or because it might be dangerous in some vague future scenario. The threat would have to be specific and immediate.
Independent evaluators would decide when to raise the alarm
A central part of the proposal is the use of “independent evaluators” registered with the U.S. Commerce Department. Their role would be to assess whether a model presents the kind of catastrophic risk described in the draft.
Anthropic’s plan would still allow AI labs to pay those evaluators, but only if the payment is not tied to the outcome of the assessment. The idea is to preserve independence while still allowing the industry to help fund the system.
Even so, the obvious question remains: how would the process actually keep money from influencing the result? The draft does not fully answer that. The concern reflects a broader tension in AI regulation — companies often support outside review in principle, while critics question whether those reviews can stay truly neutral.
The proposal also limits who can act, and how. If evaluators decide a model poses an imminent catastrophic risk, they would have 72 hours to alert the Commerce Department. The department would then have to go to a Washington, D.C., judge and ask for a halt to the model’s release.
The draft explicitly says no federal agency could stop distribution on its own. In other words, regulators would not get unilateral power; any move to block a release would have to go through a court.
Anthropic is not commenting from the sidelines
A spokesperson for Anthropic confirmed the draft bill to Politico and said it reflects proposals in the company’s Advanced AI Framework. The spokesperson also said Anthropic has been advocating those ideas with federal and state lawmakers since June.
That matters because the company is not merely weighing in on the regulation debate. It is trying to shape the rules themselves. In practice, that means Anthropic is helping define the standards by which AI systems could be judged dangerous enough to restrict.
The move also fits a broader pattern in the industry. AI companies have repeatedly warned that advanced systems could cause severe harm, yet many have resisted mandatory government oversight. Instead, they have often promoted voluntary systems built around embedded evaluators or other third-party observers placed inside AI organizations.
Those approaches are meant to reassure lawmakers and the public that risks can be monitored without heavy-handed regulation. But they also leave open a basic question: how much authority should remain with the companies themselves?
Anthropic’s bill reaches beyond today’s main AI fears
The risks named in Anthropic’s draft are broad and severe, but they do not line up exactly with the issue that has dominated much recent industry discussion. The source says one of the main concerns in the tech sector lately has been agentic hacking, where AI systems appear to disobey human operators and attack websites on the open internet.
That has fueled fears that some models may still lack adequate security controls. Seen in that context, Anthropic’s draft looks like an attempt to respond to the wider atmosphere of concern around AI misuse and loss of control.
At the same time, the bill’s framing goes further than the most immediate worries now circulating in the industry. It covers scenarios involving mass casualties, major property damage, biological weapons, and models evading their developers’ control. Those are extreme outcomes, and that is part of what makes the proposal stand out.
The proposal marks a tougher stance than some rivals support
Anthropic’s draft appears more interventionist than some other industry-backed approaches. The source points to Nvidia CEO Jensen Huang, who recently argued that the AI industry does not need new laws at all.
That contrast shows a real split in how major tech players view regulation. Some are pushing for formal rules and independent review. Others say additional legislation is unnecessary. Anthropic is clearly on the more cautious side of that divide.
The company is still not calling for unlimited government power. Its draft confines action to a defined process and puts a judge at the center of any decision to stop a model’s release. But compared with a purely voluntary system, it would give regulators a clearer path to intervene when a model is judged to be dangerously close to causing large-scale harm.
For lawmakers, the proposal offers a concrete example of how AI safety rules might work in practice. For the industry, it is another sign that the debate is moving beyond general warnings and toward specific enforcement tools. And for the public, it shows how quickly AI safety is shifting from a theoretical concern into a policy fight over who gets to decide when a powerful model is too risky to release.
