Will it take a ‘Chernobyl-scale disaster’ for us to regulate AI? | Stuart Russell
The AI company Anthropic has been making major headlines recently. Its trillion-dollar IPO plan and its blood feud with secretary of defense Pete Hegseth have attracted much attention, but two other events may be even more consequential.
In early June, the company posted an article describing early signs of recursive self-improvement (RSI), a process in which an AI system devises ways to increase its own intelligence, leading to a greater ability to improve itself, and so on.
Obviously, uncontrolled RSI could produce a runaway feedback loop that leads to an irreversible loss of human control. Anthropic suggested the world should “slow or temporarily pause frontier AI development”. Then on 12 June, the White House issued an export control directive banning access to Anthropic’s new frontier models, Fable 5 and Mythos 5, for all foreign nationals – including many of its own key researchers. Anthropic responded by shutting the models down altogether.
These two June events are closely related. A few months ago, Anthropic’s Claude Code became good enough that its leading researchers no longer write any code at all; they just describe ideas and experiments to Claude and it does all the work.
This sped up the cycle of improvement – including the improvement of Claude Code itself – to the point where the latest iteration, called Mythos 5, showed the ability to conduct end-to-end cyber-attacks with no human assistance. If such systems were released without cast-iron guardrails, almost anyone in the world could attack any country’s critical infrastructure at will.
These developments are only to be expected. They are symptoms of the inexorable increase in AI risk arising from the inexorable increase in AI capabilities. Yet, with the honorable exception of the UK’s AI Safety Summit in 2023, the world has largely been ignoring the risks.
The CEOs are telling us: “We’re on track to create superhuman intelligence, which has a good chance of causing human extinction.” (By “good chance” here, they mean a chance similar to the one in six chance of dying while playing Russian roulette with a loaded revolver; in this game, however, the revolver is pointed at all of our heads.) Yet governments reply: “That’s wonderful! Can we offer you a subsidy? Fast-track your permits?”
But finally, with the prospect of weapons of mass cyberdestruction in the hands of billions, the White House has reversed its deregulatory stance and suffered a rare attack of common sense.
after newsletter promotion
They sputter: “Why did no one warn us about these AI systems?” Their response has been spasmodic, with an on-again, off-again executive order and now a ban on a system that had already been deployed, but the direction of travel is clear.
Unrestrained development of unsafe systems leads to intolerable risks. Governments can respond now, before the risks materialize, or they can wait and clean up the mess (if they still exist, that is). One leading AI CEO told me he didn’t expect serious regulation to happen until there was a “Chornobyl-scale disaster”. If that happens, of course, the AI companies can expect to be shut down immediately and perhaps permanently.
The recent changes in White House policy suggest we might not need a Chornobyl to spur real regulation, but perhaps only a Three Mile Island. The kind of regulation we need is not new: a licensing regime that requires a minimum safety standard before a system can be built and released. This is how we handle nuclear power, airplanes, buildings, elevators, hairdressers and sandwich makers. Is it too much to ask of trillion-dollar AI corporations, who claim to be building the most dangerous technology in history?
Stuart Russell is a distinguished professor of computer science at University of California, Berkeley, the president of the International Association for Safe and Ethical Artificial Intelligence and a Guardian US columnist
Read the full story at The Guardian ↗
Anthropic's June disclosures of recursive self-improvement capabilities and autonomous cyber-attack potential prompted White House export restrictions on its frontier models. The White House reversed prior deregulatory stance and imposed the ban; Anthropic then shut the models down entirely. Stuart Russell, a UC Berkeley computer scientist, argues that regulatory frameworks analogous to those governing nuclear power, aviation, and other high-risk sectors could manage AI development risks proactively, rather than requiring a major incident to trigger intervention.
Read the full story at The Guardian ↗
The AI company Anthropic has been making major headlines recently. Its trillion-dollar IPO plan and its blood feud with secretary of defense Pete Hegseth have attracted much attention, but two other events may be even more consequential.
In early June, the company posted an article describing early signs of recursive self-improvement (RSI), a process in which an AI system devises ways to increase its own intelligence, leading to a greater ability to improve itself, and so on.
Obviously, uncontrolled RSI could produce a runaway feedback loop that leads to an irreversible loss of human control. Anthropic suggested the world should “slow or temporarily pause frontier AI development”. Then on 12 June, the White House issued an export control directive banning access to Anthropic’s new frontier models, Fable 5 and Mythos 5, for all foreign nationals – including many of its own key researchers. Anthropic responded by shutting the models down altogether.
These two June events are closely related. A few months ago, Anthropic’s Claude Code became good enough that its leading researchers no longer write any code at all; they just describe ideas and experiments to Claude and it does all the work.
This sped up the cycle of improvement – including the improvement of Claude Code itself – to the point where the latest iteration, called Mythos 5, showed the ability to conduct end-to-end cyber-attacks with no human assistance. If such systems were released without cast-iron guardrails, almost anyone in the world could attack any country’s critical infrastructure at will.
These developments are only to be expected. They are symptoms of the inexorable increase in AI risk arising from the inexorable increase in AI capabilities. Yet, with the honorable exception of the UK’s AI Safety Summit in 2023, the world has largely been ignoring the risks.
The CEOs are telling us: “We’re on track to create superhuman intelligence, which has a good chance of causing human extinction.” (By “good chance” here, they mean a chance similar to the one in six chance of dying while playing Russian roulette with a loaded revolver; in this game, however, the revolver is pointed at all of our heads.) Yet governments reply: “That’s wonderful! Can we offer you a subsidy? Fast-track your permits?”
But finally, with the prospect of weapons of mass cyberdestruction in the hands of billions, the White House has reversed its deregulatory stance and suffered a rare attack of common sense.
after newsletter promotion
They sputter: “Why did no one warn us about these AI systems?” Their response has been spasmodic, with an on-again, off-again executive order and now a ban on a system that had already been deployed, but the direction of travel is clear.
Unrestrained development of unsafe systems leads to intolerable risks. Governments can respond now, before the risks materialize, or they can wait and clean up the mess (if they still exist, that is). One leading AI CEO told me he didn’t expect serious regulation to happen until there was a “Chornobyl-scale disaster”. If that happens, of course, the AI companies can expect to be shut down immediately and perhaps permanently.
The recent changes in White House policy suggest we might not need a Chornobyl to spur real regulation, but perhaps only a Three Mile Island. The kind of regulation we need is not new: a licensing regime that requires a minimum safety standard before a system can be built and released. This is how we handle nuclear power, airplanes, buildings, elevators, hairdressers and sandwich makers. Is it too much to ask of trillion-dollar AI corporations, who claim to be building the most dangerous technology in history?
Stuart Russell is a distinguished professor of computer science at University of California, Berkeley, the president of the International Association for Safe and Ethical Artificial Intelligence and a Guardian US columnist
Read the full story at The Guardian ↗
Anthropic posted an article in early June describing early signs of recursive self-improvement in AI systems Recursive self-improvement is a process in which an AI system devises ways to increase its own intelligence, potentially creating irreversible loss of human control if uncontrolled Anthropic's latest model, Mythos 5, demonstrated ability to conduct end-to-end cyber-attacks with no human assistance On 12 June, the White House issued an export control directive banning foreign nationals' access to Anthropic's Fable 5 and Mythos 5 models Anthropic shut the models down entirely in response to the export ban Uncontrolled development of unsafe AI systems leads to intolerable risks Governments should implement licensing regimes requiring minimum safety standards before AI systems are built and released Proactive regulation could manage risks without waiting for a major disaster
Read the full story at The Guardian ↗
- Anthropic reported early signs of recursive self-improvement in AI systems in June, raising risks of uncontrolled capability acceleration
- The company's latest model, Mythos 5, demonstrated ability to conduct autonomous cyber-attacks; the White House subsequently banned its access to foreign nationals
- The article argues AI regulation through safety licensing—applied to other industries—could prevent catastrophic risks without waiting for a major disaster