This weekend current and former employees of US AI laboratories and their leaders issued dire warnings of a near-term apocalypse at the hands of AI. Out-of-control AI development could mean ‘we could all die in the immediate future’ said one former lab employee. This was followed by a proposal from Dario Amodei, the CEO of AI company Anthropic, to slow down AI development to ensure the technology is safe. The proposal met with approval from the leaders of other labs, including Sam Altman and Elon Musk.
No observer of AI would argue against a greater focus on safety in AI development and deployment. But the renewed attention on AI’s long-term risks must not become too hyperopic, at the cost of tackling the present-day impacts of the technology. It must not be a blunt instrument used to crush competition to current US AI leaders. And it will not be possible to address without some kind of US–China agreement.
Why the new warnings?
The path to catastrophe described by Amodei and others runs through a technique referred to as recursive self-improvement (RSI) – an approach to training AI models that is beginning to roll out, if still in its early stages.
The risk is that RSI, which involves AI training AI, could lead to AI development of such speed that at some point a tipping point will be reached, where the AIs are capable enough to improve themselves without and beyond human oversight.
Whether this alone is sufficient to create AI models that threaten human control is less clear. But what this technology can do has evolved so quickly in the past few years that dismissing the possibility outright looks foolhardy.
Experts have been quick to point out that some of the claims made over the weekend do not wholly stand up. On cybersecurity in particular, the risk of an AI model shutting down the entire Internet within a year was dismissed by the founder of the UK’s National Cyber Security Centre as ‘a thought experiment masquerading as an evidence-based warning’. But for the most part the message that AI poses near-term existential risk to humanity has led the news cycle largely uncontested.
Present day impacts
Some risks emerging from AI are immediate, and certain. Over the next couple of years, increasingly capable machine learning technologies will be developed and deployed in ways that dramatically reshape economies, societies, governments, battlefields, classrooms and so on. Most recently it was mathematicians who had their Lee Sedol moment: the uneasy feeling when machines begin to outperform them on tasks that were previously limited to human ingenuity. Most everyone will have theirs soon enough.
Few understand the immediate impacts better than the AI companies themselves: Anthropic’s excellent recent report outlines the misuse of its products for espionage, cyberattacks, information operations and weapons development. Most major labs maintain significant research capacity into the likely impact of AI on the global economy. This a technological revolution which is reshaping the face of global society and present-day consequences cannot be left ungoverned.
The present-day impacts are real. The short-term risks are significant.
Transparency, and the longer term
The longer tail of risks – whether through gradual human disempowerment as machines take on greater responsibilities or something more sudden – are both plausible and sufficiently threatening to demand attention.
Efforts led by experts like Stuart Russell on international agreement on clear and verifiable red lines to prevent catastrophic outcomes are gathering pace. Without them, the first opportunity to pass enforceable global governance of this technology may be missed.
Transparency from AI labs will also be vital. The kind outlined in Amodei’s proposal – a third party embedded in AI labs with employee-like privileges, tasked to evaluate the AI firms’ safeguarding measures – has been a demand from AI governance advocates for years. That Anthropic have offered this unilaterally, with some other labs offering to follow suit, is unquestionably a step forward.
If such a third party is to be created, it is absolutely essential that it is drawn from beyond the silicon valley bubble. The UK’s AI Safety Institute looks like one of the world’s most plausible candidate institutions. National models for AI oversight that resemble the Food and Drug Administration or international models based on the International Energy Agency are also credible.
Amodei’s suggestion that US AI labs ‘work together to set standards’ and coordinate a slowdown brought a stinging rebuke from the former White House AI Czar David Sacks: ‘Stop pretending antitrust law has to be suspended so you can form a cartel’, he wrote on X. ‘Stop pretending you need a regulatory approval process that supersedes product liability’.
‘Demanding your preferred regulatory framework as the price… will look like blackmail of the public and the political system’.
This mirrors fears that the enormous pressures on these companies to meet expectations on revenue could lead them to attempt to widen their moat – that is, entrench their competitive advantage at the expense of others.
Any such effort should be rejected: competition to achieve performance and reliability will be essential to advancing near-term AI safety. And it could plausibly decelerate development, as it undercuts the commercial incentive to pour massive capital into training ever-more-expensive frontier models.
Any proposed policy that would inhibit competition – such as bans on open source AI models, the relaxation of product liability laws, or bans on the use of current models already shown to be safe – should sound alarm bells. The UK must certainly not ban open-source or open-weight AI models.
Finally, no effort to maintain human control of AI development will be complete without some kind of understanding between the AI superpowers, the US and China – perhaps along the red lines proposed by Professor Russell.
But the current narrative, so heavily predicated on maintaining US advantage over China, makes this more difficult. Amodei states that democracies must maintain a lead over China, and accuses Beijing of a less rigorous approach to AI safety. And US President Donald Trump presents AI as a zero-sum game, choosing instead to focus on winning the AI race: ‘whoever wins AI, wins’.
By comparison, President Xi Jinping has at least spoken publicly of the need for human control of AI. And this week, China’s spy agency issued its first warning about the national security risk presented by AI. However, Xi’s push to advance China’s vision of AI governance, largely in competition with Washington, will undoubtedly fuel the sense of an AI arms race gathering pace, which will hardly act as a brake on the speed of development.
AI will be on the table when Trump and Xi meet in DC later this month. Some kind of progress is not impossible. But while leading voices in both countries are doubling down on the need to outcompete one another, it will require a monumental effort to broker an agreement.