Tech bosses Elon Musk and Sam Altman have backed a call from the head of Anthropic to have permanent third-party reviewers inside “frontier” AI companies. Anthropic chief executive Dario Amodei has written an essay, called We Must Pace the Frontier, in which he urges such companies to slow the rate at which they improve the capabilities of AI models. (Photo: Anthropic chief Dario Amodei).

It comes amid increased fears about the possible misuse of AI, with Anthropic recently revealing that several bad actors had used its Claude AI models for activities including weapons development, cyber operations, surveillance, ​and fraud. Mr Amodei has come up with a three-part plan, part of which he said his AI company was unilaterally committed to.

In a post on X in which he commits to the “first of these steps”, he writes: “We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.”

Mr Musk, the boss of X, which has developed its own ‘Grok’ AI, replied to the post, writing: “Dario is right.” And OpenAI boss Mr Altman wrote on X: “I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks. “Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We’ll have more to share soon.” (Photo: Mr Elon Musk).

Mr Altman’s comments come after he revealed that OpenAI will not go public this year, citing safety concerns. “I actually think that, given everything happening with safety, right now would be an ill-advised moment to go public, and we don’t feel pressure on that,” he told Fortune magazine. The New York Times in June reported that the company was considering whether to hold off on a potentially trillion-dollar IPO until next year.

“We got a lot of stuff to do, like meeting this moment of what is going to be required for safety and alignment, and how the industry and governments can work together,” Mr Altman continued when asked about whether 2027 was more likely for them to go public. Fortune reported that he also suggested OpenAI and other leading AI companies may be close to announcing an agreement to slow AI development and work together to address safety risks.

Mr Amodei said he was concerned that since this summer, AI has been “advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI”. And he warned that within six to 12 months, AI could be capable of leading a “swarm” that could take over “the entire internet,” causing “hundreds of billions of dollars in damage”, among other risks. Ex-PM Rishi Sunak wrote on X: “Dario is right to call today for slowing the pace of frontier development so that safety can keep up.

When the labs themselves are asking for speed limits, governments should listen. He added that “even as the US and China compete, they urgently need a frank conversation about how to avoid catastrophe”. Evan Hubinger, a top safety researcher at Anthropic, recently said there is a more than 10% chance that “AI could kill all humans” in the next decade. (Photo: OpenAI chief Sam Altman).

And researcher Jacob Coxon ⁠resigned from Anthropic amid fears the company and OpenAI “are racing straight to self-improving superintelligence and gambling with our lives”. Mr Amodei said he was not calling for stopping model training or technical progress, but making sure companies take enough time to align and ‌safeguard their models, and for third-party evaluators to confirm these steps.

What are His Proposed Three Steps?

• Embedded evaluators. Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes.

This is the key step for verifiability of any pacing commitments, and has a precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees. Anthropic is unilaterally committing to this step now. We intend this to be part of a broader push to redouble efforts on our safety and alignment work.

• Democratic coordination. Frontier AI companies in democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. Some forms of coordination that would be impactful for pacing are legally challenging and will require government support.
• Global coordination. The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking the challenges of verifying compliance seriously.

Courtesy: Sky News; Photos: AP and Reuters