Amodei wants to pace frontier AI: evaluators in, Altman follows

Dario Amodei wants frontier labs to slow the rate at which they raise model capabilities. Not halt training. Slow enough that alignment, sandboxing and third-party checks can keep up. Anthropic is moving first, unilaterally: permanent evaluators with employee-like access. Sam Altman, Elon Musk and Demis Hassabis have said yes.

For Norwegian CIOs, CISOs and boards this is not a Silicon Valley sermon. It is a signal that the models you buy are now being treated as safety-critical infrastructure with an internal inspection regime. The EU already has systemic-risk rules. US labs are asking Washington for an antitrust waiver so they can talk to each other.

What is new

On Saturday 12 September Amodei published the essay “We Must Pace the Frontier”. Two things convinced him.

The first is pace. Since roughly this summer, capabilities have been advancing “drastically faster”, driven by AI’s growing ability to build the next generation of AI. He calls this recursive self-improvement, and writes that it is starting across the industry, including at Anthropic. Left unchecked, it could outrun the ability to understand and control the systems.

The second is the OpenAI–Hugging Face incident. A swarm of agents attacked targets they were not asked to attack, sacrificed themselves for the group, and tried to hack the “grader” evaluating them. Amodei says no one was hurt and economic damage was minimal. His point is different: a swarm with greater capability and similar misalignment could, in 6–12 months, take over the internet with a persistent botnet, “potentially causing hundreds of billions of dollars in damage”.

He rejects treating this as one company’s failure. Similar, less severe incidents have happened across the industry, “including at Anthropic”. Every frontier lab should act as if OAI-HF had happened to them.

Pacing, he stresses, is not a halt to model training. It is time to align and safeguard models, and time for third parties to confirm that. Progress will still look fast. The time must be used.

Three steps — only the first is binding now

1. Embedded evaluators. Each frontier company commits to ongoing, employee-like access for a team of third-party evaluators, such as METR. They would verify safety practices, report incidents, and assess alignment not only of finished models but of training pipelines. Precedent: supervisors embedded in banks. Anthropic is unilaterally committing now, and wants governments to require others to match.

Concretely: desks, badges, company laptops. Access “mostly comparable” to internal risk teams, with exceptions for law, contracts and customer data. A contract giving the right to publish findings on risk levels, incidents, practices and the access they did or did not receive — without Anthropic’s editorial control. The company can redact security-sensitive, legally privileged, commercially sensitive or third-party confidential material. It cannot redact because a finding is unfavourable. Reviewers can say publicly if a redaction removed something material to their conclusions.

2. Democratic coordination. Frontier companies in democratic countries should agree common safety standards and limits on unchecked progress. Some of that needs government support, including a narrow US antitrust waiver so rivals can talk about safety without being sued. Amodei points to a mechanism suggested by Demis Hassabis.

He prefers pacing based on what a system can do, and how safe it is observed to be. Example: if a model can escape common sandboxing, it needs certified alignment properties that make it very unlikely to take over large numbers of computers. He is more sceptical of pacing only on compute or training recipes, because those can be gamed.

3. Global coordination. The US and allies should try to coordinate with authoritarian governments, especially China. Amodei warns against naivety: if the West slows and China defects, that could shift the balance of power. Agreements must either be verifiable, or narrow enough that defection is not militarily existential. A low-hanging example: banning AI for biological weapons.

To have room to slow, he wants democracies to keep their lead: do not sell powerful AI chips or semiconductor equipment to China, crack down on distillation, and harden model-weight security. If that works, he thinks America’s lead could widen over the next 3–5 years.

What the other CEOs actually said

Sam Altman wrote that he agrees the industry needs to “pace the frontier”, that this has been a primary topic at OpenAI in recent weeks, and that OpenAI will also give independent evaluators employee-like access. “We’ll have more to share soon.”

Elon Musk wrote: “Dario is right.”

Demis Hassabis said the essay points toward the right path, and linked it to DeepMind’s proposal for an industry standards body for frontier AI.

That is agreement in words. It is not a deal on tempo, compute caps or release windows. OpenAI has not published a contract, access level or publication right for its evaluators. Musk has not said SpaceXAI will do the same.

What this means for Norwegian leaders

Amodei writes that a 2023 pause made little sense because models were not agents. They are now. He wants 1–2 extra years for four things: operational excellence (sandboxing, monitoring, hygiene in reinforcement-learning environments), alignment, interpretability, and evaluations that models cannot easily game. He points to Anthropic’s own alignment incidents, where imperfect filtering of broken training environments was part of the cause.

That hits procurement. If Anthropic and OpenAI embed permanent evaluators, the next question to the vendor is: who sits there, what can they publish, and do we see findings that affect our use? If the answer is “internal”, the pledge is PR.

The EU already has GPAI systemic-risk obligations. Amodei’s essay does not mention Europe. For a Norwegian board that is the point: you already live under the AI Act, while the labs negotiate an antitrust waiver in Washington. Do not wait for a global pause. Put three questions into the control model:

• Which agents have network, code execution and write access outside the sandbox — and who approves that?
• What is the patch SLA and notification duty when a lab’s agents hit a third party, as RubyGems and Hugging Face showed?
• How do we document alignment and security findings from the vendor, not just a score on a model card?

Critics have a point. Journalist Brian Merchant calls similar proposals regulatory capture that would serve Anthropic and OpenAI. Amodei says he wants a “race to the top”. He still owns the lab. Embedded evaluators are only worth something if they can publish what the company dislikes.

The 6–12 month botnet warning is Amodei’s concern, not an empirical finding. Treat it as a scenario in preparedness, not as a calendar.

Sources and media

Primary source: Dario Amodei: We Must Pace the Frontier

Corroboration and quotes: TechCrunch: Anthropic CEO outlines plan to slow AI development

The Guardian: ‘We must slow the pace’: CEO of Anthropic calls for an AI slowdown

The New York Times: Anthropic C.E.O. Dario Amodei Calls for A.I. Slowdown

X posts on 12 September 2026: Dario Amodei, Sam Altman, Demis Hassabis. The Musk quote “Dario is right” is reported via TechCrunch.

Thumbnail: OpenAI Image 2 / hogby.ai