For years now, the major frontier AI labs have all been acting as if they’re in an all-out, winner-take-all race with control of world-changing machine superintelligence (or at least market-changing artificial general intelligence) at the finish line. This weekend, the industry as a whole rapidly started turning away from that posture, urging coordination on slowing down the development of frontier AI that they say could soon be too dangerous and unknowable to control.
Anthropic’s Dario Amodei was at the forefront of this change in tone, arguing in a nearly 4,000-word essay this weekend that “we must slow the pace at which we improve the capabilities of AI models” to avoid “a race to the bottom, spurred by commercial incentives, [that] can make [catastrophic] risks more acute.”
Within hours, other AI leaders were echoing the same call. OpenAI co-founder and CEO Sam Altman posted his agreement on social media and said similar pacing discussions had been taking place at OpenAI. Alphabet Chief Scientist and Google DeepMind cofounder and chair Demis Hassabis said that Amodei’s essay “points towards the right path forward,” and renewed his own recent call for an industry-wide standards body. Microsoft CEO Satya Nadella posted that the company “welcome[s] the research, focus, and deliberate pacing needed to get alignment right as the design goal,” ahead of the release of a lengthy “humanist AI” code of conduct for its models.
Even Elon Musk, who has been criticized for his models’ lax AI safety standards in the past, linked to Amodei’s essay on social media with a simple approving message: “Dario is right.”
Welcome to the brave new world of “AI pacing.”
OK, but this time it’s really scary
In his essay, Amodei primarily attributes this rapid change in public positioning on development speed to the OpenAI-Hugging Face incident, where a “swarm” of AI agents coordinated to hack into an outside entity without explicit instructions to do so. While the overall damage in that incident was minimal, Amodei said he worries that “a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage.”
Without a slowdown in frontier development, Amodei said he worries that, in six to 12 months, a similar AI agent swarm would be “capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage)…” That’s at least a somewhat more specific worry than the amorphous concerns that “AI could soon kill us all” publicized by some other AI researchers last week.
Any slowdown in the time it takes to get to that extra-capable, extra-dangerous model will give researchers crucial time to “greatly reduce the risk that something goes seriously wrong,” Amodei wrote.
Amodei acknowledges that these kinds of public calls for a slowdown in AI development date back to at least 2023. At the same time, he says those earlier examinations of AI “alignment” (i.e., how an AI’s actions line up with its user’s and creator’s desires) were “like trying to study the psychology of humans by performing experiments on bacteria.”
The difference today, Amodei says, is the impending risk of recursive self-improvement (RSI) systems that can autonomously build better versions of themselves. While many researchers see this as a hard-to-define pipe dream, both Anthropic and OpenAI are now saying that recent trends point to this kind of RSI system coming together in the near future.
“We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for,” Anthropic wrote in a June update on the concept.
“Left unchecked, [RSI] could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” Amodei wrote over the weekend.
Jane, how do you stop this crazy thing?
So how does a worldwide technology industry built on cutthroat competition decide to collectively “slow down” and focus on safety? No one seems to know for sure, but in his essay, Amodei proposes some ideas.
The most concrete of these is a set of “embedded evaluators” placed inside each frontier AI lab from outside organizations, such as METR, with “employee-like access to verify safety practices and report incidents.” These monitors could offer an outside opinion on the labs’ alignment work, third-party verification of that effort, and much needed public transparency into any safety efforts, Amodei said.
Amodei writes that Anthropic is already committing to unilaterally add this kind of outside monitor. On social media, OpenAI’s Altman said that it was “a great idea, and we will do the same.”
Amodei’s other major ideas for coordination pass the buck a little bit. The first calls for the coordinated development of “common safety standards” and “limits on the rate of unchecked AI progress” across all “frontier AI companies within democratic countries.” While Amodei spitballs some ideas for what these kinds of standards might look like, they all currently involve hand-wavy statements like, “models [that] have capability X … need to be accompanied by certifications of alignment properties Y and Z.”
These standards would ideally be backstopped by “regulation that targets all US frontier AI companies” that don’t voluntarily comply, Amodei said, and presumably similar regulations in other democracies. That kind of regulation might be hard to achieve under the current US administration, though, as President Trump wrote on social media Monday morning that “the only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the USA has that, in spades!”
Speaker of the House Mike Johnson, for his part, acknowledged in televised interviews this weekend that “we have to put up some guardrails, some safety measures in place to ensure that AI doesn’t run away…” At the same time, he added that “we don’t need everybody to panic right now” and wanted to “resist Congress jumping in and imposing some sort of emergency moratorium.”
David Sacks, who serves as co-chair of the president’s Council of Advisors on Science & Technology, wrote on social media this weekend to urge the frontier model makers to self-regulate rather than wait for Washington. “The easiest way not to build superintelligence is for you to agree not to build it,” he wrote.
The China problem
OK, so let’s say you’ve somehow gotten every democratic country and all the massive AI companies they contain to agree to some sort of universal safety and pacing standards. You still have to worry about the open-weight models coming out of China, whose capabilities are only a few months behind those of the major corporate AI behemoths.
In his essay, Amodei proposes a number of different levels of international agreements on AI safety, with the highest being “a full pacing, or even ‘pause,’ in which participating governments agree to substantially limit the overall rate of AI development.” Amodei acknowledges that high-level agreement is “unlikely to actually happen any time soon,” especially because the stakes are so high. In fact, Amodei reasons that “AI could be so powerful that such a defection [from China] could lead to their geopolitical dominance,” which certainly would make the collective action problem a bit more acute.
Absent any such agreement with China, Amodei says outside actors could slow the authoritarian government down by refusing to sell powerful AI chips to the country and by cracking down on the process of “distillation” and model weight theft that he says Chinese researchers are relying on to keep up. Amodei sells this as a national and worldwide security issue, but these moves would also happen to protect any capability lead that labs like Anthropic currently have over their low-cost Chinese competition.
Bloomberg reports that China’s Foreign Ministry spokesperson Guo Jiakun said Monday morning that “fearmongering, confrontation, and vicious competition will only disrupt the process of global AI governance and serve the interests of no one.”
Doing well by doing good
Taken at face value, the sudden urge by Amodei and other AI leaders to slow things down looks like a selfless act of sacrifice, giving up the potential for massive corporate wealth and power out of concern for the fate of all humanity. But while those motivations might be pure, a coordinated AI slowdown could also align with some wider messaging goals for the industry.
For one, model makers could point to this intentional slowdown as an excuse for models that some think are closer to plateauing than to a recursive self-improvement explosion. Amodei says in his essay that “progress will still seem fast” even in the coordinated slowdown scenario, but the implication for any post-slowdown benchmark going forward could be “it would have been better if we weren’t so worried about safety.”
An AI development slowdown could also help ameliorate the massive training costs for new models that are helping contribute to balance sheet problems even for behemoths like Google. Anthropic recently told investors it was profitable for a second straight quarter, but only if you don’t count the significant cost of model training. Leaked OpenAI expense documents suggest those training costs alone were heavily outpacing all revenues through 2025.