AI models are evolving rapidly — so fast that they may soon be able to develop themselves, without human intervention. That’s why Anthropic is warning the AI sector: a “brake pedal” needs to be created, or companies risk losing control over their creations.
AI systems capable of improving themselves, known as “full recursive self-improvement,” could bring great benefits to science and health, but they also pose great risks to humanity, according to a blog post written by Marina Favaro, leader of The Anthropic Institute, and Jack Clark, co-founder of Anthropic.
“Full recursive self-improvement may also increase the risks of humans losing control over AI systems,” they wrote. “If systems are capable of fully building their own successors, the ways we protect them, monitor them, and shape their behavior become much more important.”
The industry is much closer to self-improving AI than previously expected, they warn. Therefore, technology companies should consider slowing down or pausing the development of cutting-edge AI to give researchers time to better understand the potential social harms that self-improving AI could cause.
Researchers should also create a way for humans to intervene if things get out of control, they said.
Appearing on CNN on Thursday night, Clark asked the sector to provide itself with a “brake pedal”.
“When I look at the car we’re driving, all I have is an accelerator pedal. I don’t have a brake pedal, and certainly at some point in the future we might want that option,” he told Anderson Cooper in an interview.
Cooper cited the example of science fiction films where AI went rogue to kill humans and asked if that was what worried Clark.
“Yes, we read and watch science fiction here too, so it’s not something that escapes us,” Clark replied. “How do you maintain control over teams of scientists that are much, much larger and much faster than you’ve ever had before?”
The inability to validate, verify and trust AI behavior poses a major risk to its development, he added.
The warning comes after Anthropic filed to hold an IPO (initial public offering) of its shares, a move that could raise tens of billions of dollars from investors to accelerate the construction of data centers and computers needed for AI.
SpaceX, which in addition to its rocket and satellite businesses also operates in the AI sector, is expected to hold next week what would be the largest initial public offering on record, raising US$75 billion.
Having big AI companies like Anthropic, SpaceX, and OpenAI work together to create brakes might seem antithetical to a competitive business with billions if not trillions of dollars at stake. But such cooperation is possible, Clark insisted.
“We’ve done this before. At the height of the Cold War, amid situations of great tension between rival countries, they found ways to stabilize aspects of the nuclear arms race,” Clark told CNN. “All of this has been done before in other domains, and perhaps this is something we need to do in the AI domain.”
