Is the field of artificial intelligence proceeding at too fast a pace? If you ask the leaders building the systems, the answer this weekend was an overwhelming “yes.” If you ask President Donald Trump, the answer is basically “hell, no.” FYI, the general public is not on the president’s page: 73% do not trust companies to use AI responsibly and 70% are more worried than excited about AI.
Anthropic CEO Dario Amodei took the lead Saturday with a 3,800-word blog postoutlining the reasons model development should slow down. He also proposed a verification process designed to create trust. It was quickly seconded by Sam Altman at OpenAI and Elon Musk at Grok. Later Google and Microsoft joined the club.
In contrast, the president’s Truth Social posts have one theme: the U.S. must come out on top in the race with China. One post from Monday (there were several): “WHOEVER WINS AI, WINS! We are leading China, and all others, and will continue to do so.” Never mind that Trump has eased export controls on advanced Nvidia chips, which are “the most crucial input to AI development.” So much for pedal-to-the-metal.
History suggests what’s missing isn’t speed limits alone, it’s trust and verification, which are the things Amodei’s own proposal is asking for.
We’ve engaged in similar “races” before. For example, the U.S. was first with a nuclear warhead, bombing Japan twice in 1945. Russia was not far behind and then came Britain and a few other countries. Suddenly we had global nuclear proliferation, but each instance was a government. It wasn’t a series of businesses spread across the globe.
Working government-to-government to reduce the risk of nuclear war seems like it would be easier than working company-to-company or company-to-government and then government-to-company to reduce the chance of a catastrophic AI breach.
Even 1950s illegal street racers knew better than to aim their cars at a cliff.
It wasn’t easy for the world to come together to sign the Nuclear Non-Proliferation Treaty (NPT) in 1968. It took more than 20 years after the U.S. demonstrated the raw power of a nuclear weapon to sign an agreement. In that period, five countries became nuclear powers: the U.S., Russia, England, France and China. Yet another 54 additional countries signed the agreement, a first step back from global armageddon.
Two things were essential and interrelated to secure those signatures: trust and verification.
There had to be enough trust to come to the table. Then the verification program had to be robust enough and transparent enough for the parties to trust that it was sufficient to hold the other parties in check.
Before there was an NPT, there was the Cuban Missile Crisis in 1962. After averting nuclear war using kludgy communications, the U.S. and Russia set up a hotline to facilitate direct communication and, although it may have been an unstated goal, build trust. In 1963, teletype machines connected the two countries, one in the Pentagon and one in the Kremlin. President John F. Kennedy could deliver a message to the Pentagon; someone there would input it in the teletype machine. On the Russian side, someone would translate the received message into Russian and then deliver it to Premier Nikita Kruschchev. His response repeated the process in reverse.
If you’re thinking ‘that sounds slow,’ it was deliberately so. This friction-by-design prevented knee-jerk reactions or misunderstandings that might arise in an ephemeral telephone call. It prevented heated rhetoric. It helped develop trust.
Right now, in the field of AI we don’t have trust across governments or even an acknowledgment that dialog might be important. What we do have are three examples of elevated risk that are not quite to the level of the Cuban Missile Crisis. We also have Trump’s messages on Truth Social feeding knee-jerk reactions.
Trump is not building trust elsewhere, either. He let the last treaty capping U.S. and Russian nuclear arsenals, the New START, expire in February. There was a shaky verbal understanding to keep observing the old limits for six months, a window that’s now closed. Even before that the actual verification component, on-site inspections, had been suspended since 2023. Iran diplomacy has fared no better this summer, lurching from agreement to collapse and back.
So we have a trust deficit between the two nuclear-armed superpowers, the actors who once cooperated on a hotline. AI is starting even further behind on trust and verification. With AI, the rival isn’t a former Cold War partner with a 60-year cooperation track record; it’s China, with none of that infrastructure to leverage.
In 2010, the New York Stock Exchange experienced a runaway algorithm that automatically placed trades based on market volatility. Its trades increased volatility, which caused it to make more trades. Stocks lost value in minutes. In about 10 minutes, the Dow Jones Industrial Average (DJIA) dropped 1,000 points. At one point, Accenture was valued at one-cent.
The incident is called the Flash Crash, and it illustrates the risk of recursive loops. Fortunately, this was just the Exchange and not the entire internet, and it recovered most of its value by end of day. (The miscreant behind the crash was convicted in 2016 after pleading guilty.)
The Exchange subsequently built safeguards so that extreme volatility in the future would lead to an automatic pause or full stop in trading.
Strike one.
It’s 16 years later. OpenAI experienced two dismaying events this year. The first, in May-July, involved a group of agents taking over a dormant German wiki forum, which they used to communicate with each other.
The agents could read the wiki but were not supposed to be able to edit the content. However, the code base was old, and the agents figured out how to turn the wiki into a communications board between them. They were also able to hide themselves from human detection. This is an example of agentic AI, agents designed to act autonomously, make decisions and follow work flows to accomplish a task.
Strike two.
Then came the Hugging Face incident. OpenAI gave autonomous agents a set of cybersecurity challenges in what was supposedly a secure sandbox. OpenAI’s goal was to see what a highly capable model might do, especially because the agents were tasked with not giving up. Until this experiment, OpenAI models had only correctly accomplished 198 of the 898 tasks.
However, the persistent agents found a way out and onto the internet. They created a makeshift message board to communicate with one another. In July, they hacked into the AI platform, Hugging Face, in an attempt to complete the challenge. According to outside researchers:
… from July 7th to July 13th, the message board was used to exchange over 70,000 messages, including files, questions and requests directed at other agents, information or ideas, coordination, shared results. Early in the period, questions and requests predominated; over time, the messages shifted toward sharing information and ideas, with agents increasingly exchanging techniques for cheating…
Strike three.
We have three examples of code doing things it was not engineered to do. The industry calls this misalignment (models trained in ways that didn’t keep them “safe, ethical, [and] compliant with … guidelines”). So far, the incidents have not reached the level of the Cuban Missile Crisis, but do we really want to wait until it does before we develop global safeguards?
One of the keys to reining in nuclear proliferation was that trust went further than merely sitting around a table and signing a document. Trust was verified by the independent International Atomic Energy Agency (IAEA). There is no such global body for AI. There is no such domestic body, either, even though one of President Joe Biden’s executive orders called for a “coordinated approach to responsible development and use of artificial intelligence.” Trump nullified this executive order after taking office.
The comparison runs deeper, too. Nuclear weapons require access to rare and regulated metals, such as highly enriched uranium and plutonium. They also require rare earth metals for their guidance and targeting systems, allowing a global body to monitor and control purchased material. However, the primary restriction on effective AI is access to super chips, which the U.S. controls through export rules. Multilateral verification vs. unilateral control.
Anthropic CEO Amodei called for coordination in his entreaty on Saturday. His goal was to go beyond domestic coordination to global coordination. The first step? Like IAEA, AI companies should have embedded third party evaluators who are charged with verifying safety practices and who report incidents publicly.
TIME’s new cover story, titled “The AI Tipping Point,” shows that alarm-sounding has gone mainstream.
TIME’s new cover: This summer, AI showed how frighteningly capable it has become. Now researchers, CEOs, and governments are sounding the alarm like never before https://t.co/ybKHG4xmYP pic.twitter.com/GWIiGukFi9
— TIME (@TIME) September 15, 2026
How do we ensure that future models are aligned with safety goals?
With nuclear weapons, Russia and the United States built a hotline with friction as a feature, a way to slow down communications and reactions. With agentic AI, we are on a race to a cliff (or the bottom, take your pick) with no breathing room. Can we learn from that earlier existential risk management tool and, as Amodei suggested, agree to slow down?
We cannot expect one firm to slow down on its own. The race to be first exists between firms as it does between countries.
And while it might be possible to code ethical guardrails into agentic AI, what happens if the agent starts resisting correction? If it’s in a closed system like the Exchange, we might be able to stop it. If it’s in a diverse system like the power grid or the internet itself, how will we be able to turn it off? A run of the mill circuit breaker doesn’t cut it here. Once the house is on fire from an electrical spark, most circuit breakers are useless.
Finally, what can we do to prevent the worst thing that we might imagine from happening?
I’m sure I’m not the only person following AI news who has thought of Isaac Asimov and his laws of robotics. Microsoft seems to have taken a page from his playbook: Asimov’s robots weren’t allowed to disobey or deceive their human handlers.
Monday, Microsoft released a code of conduct that could be a step towards Asimov’s rules of robotics: AI models should not have rights or legal personhood; they shouldn’t be engineered to escape human control or deceive users; and if completing a task requires that one of these rules be violated, the agent should not complete the task.
I’m not a software engineer. I don’t know how developers build ethical decision making into agentic agents. But since those engineers seem to care very much about agent actions that align with values, shouldn’t we be helping those firms work in an environment that makes protecting all of us easier? That’s the trust gap that the public is already telling pollsters they feel.
Known for gnawing at complex questions like a terrier with a bone. Digital evangelist, writer, teacher. Transplanted Southerner; teach newbies to ride motorcycles. @kegill (Twitter and Mastodon.social); wiredpen.com