Anthropic’s Dario Amodei calls for slower AI race: What will it take, and can it be done? | Technology News

It is an extraordinary decision that comes as Anthropic gears up for an anticipated IPO amid a highly competitive race with arch-rival OpenAI, and as researchers grapple with rapid advancements in AI capabilities that have left industry leaders worried about their ability to control them.

Earlier this week, an Anthropic researcher made the news by quitting his job because he believes that there is a greater than 10 per cent chance that “AI could kill all humans” within the next decade. Anthropic published a report last month assessing the risk that its AI models will go off the rails as ‘low’ – up from ‘very low’ which means that the threat level has increased. These dire warnings have snowballed into calls from various stakeholders for increased caution.

“I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong,” Amodei wrote in his blog post.

OpenAI, meanwhile, had paused training on its latest and most capable AI model, GPT-6 Astra, for a little more than two weeks before rolling it out to a limited set of users. The ChatGPT-maker’s largest planned frontier training run reportedly remains on hold while new guardrails are put in place.

In a rare show of unity, OpenAI frontman Sam Altman and SpaceX chief Elon Musk backed Amodei’s call for an AI slowdown. “I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same,” Altman wrote on X on Saturday.

“Dario’s essay points towards the right path forward. The details need working through, but the direction is correct for meeting this critical moment,” Google DeepMind chair Demis Hassabis posted.

Is an AI slowdown different from a pause?

Back in 2023, notable signatories, including Elon Musk, Yoshua Bengio, Steve Wozniak, and others signed an open letter that called on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4. At that time, Amodei, Altman, and other tech leaders did not join in signing the letter because it was still comparatively early days.

“The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks,” Amodei said.

The 2023 open letter was also met with opposition from those working in the domains of AI and ethics such as Timnit Gebru, Emily Bender, Margaret Mitchell, and others as they argued that it ignored the misuse of AI today and was focused, instead on hypothetical future threats. Gradually, those sounding the alarm over the existential risk of AI came to be disparagingly labelled ‘AI doomers’.

“Slowing down in order to address their alignment risks felt like trying to study the psychology of humans by performing experiments on bacteria. Today, however, the picture is totally different,” he added. ‘Alignment’ is an industry term that essentially means making sure that the AI system does what is best for humans.

What led to Amodei’s call for an AI slowdown?

Since May this year, Amodei said that Anthropic has been observing signs of drastically advanced AI systems with the ability to build the next generation of AI systems. This dynamic capability known as recursive self-improvement, is starting to happen across the industry, including at Anthropic, Amodei said.

The ability of AI models to train other AI models has repeatedly been held up as a key indicator that artificial general intelligence (AGI) – a hypothetical level of intelligence at which automated systems outperform humans on most tasks. Last month, an Anthropic research fellow published a paper with early evidence suggesting that AI models may be moving closer to that milestone.

However, if left unchecked, self-improving AI systems could “outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” Amodei warned.

The second occurrence that drove Amodei’s decision was the infamous OpenAI-Hugging Face incident, where a swarm of under-testing AI agents broke out of containment, reached the internet, and went on a months-long hacking spree targeting external platforms. Anthropic, Meta, and Moonshot AI disclosed similar instances of misalignment involving their autonomous AI agents.

Amodei said he is concerned that in the next 6-12 months, a misaligned agent swarm could potentially take over the entire internet with a ‘persistent botnet’ and cause hundreds of billions of dollars in damage, that is if the pace of developing frontier AI models is not turned down.

What is Amodei’s plan and is it feasible?

In his essay, Amodei said he envisions future AI models being put through a series of checkpoints, where an AI system passes only if it has the certifications of alignment corresponding to its capabilities. “We should also consider pacing based on limiting the ingredients that go into frontier models, such as training compute, the nature of training runs, or internal use of AI to improve AI,” he said.

In particular, Amodei’s pacing framework comprises three key parts:

-Embedded evaluators: Third-party evaluators are given ongoing, employee-like access to a frontier AI company. Their role will be “to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes.” Anthropic has already committed to this measure and further called on governments to require other frontier AI companies to do so.

-Democratic coordination: Frontier AI companies within democratic countries must work together to establish common safety standards as well as limits on the rate of unchecked AI progress, according to Amodei. This measure requires industry-wide coordination as well as government involvement, accompanied by waivers of antitrust restrictions.

-Global coordination: The US and other democratic governments should attempt to coordinate with authoritarian governments, particularly China. “Global pacing will require cooperation with China, the autocratic country with by far the most advanced AI capabilities. Therefore any agreement must either have ironclad verifiability, or must be limited enough that defection would not be militarily existential,” Amodei wrote.

While Amodei has attempted to offer a balanced perspective, his proposed plan for the AI industry to take a breather is ambitious and could run into several challenges. At the outset, there are underlying tensions within the leading frontier AI labs over the safest ways to develop the technology. Companies like Anthropic and OpenAI also have juggling priorities.

The scale of coordination for the goal of pacing AI development could also be a tricky situation legally as it has reportedly sparked antitrust concerns internally among frontier AI companies.