Tech titans like CEOs of Anthropic and OpenAI warned an Artificial Intelligence-weary world that their own advanced systems could endanger humanity. Now the question rises -- is there any force which can counter the detrimental impact? The answer may lie in Nvidia's efforts to stop AI agents from going rogue.
Nvidia on Monday unveiled a new security platform designed to stop artificial intelligence agents from going rogue, saying it sets “boundaries” that could have stopped previous breaches. The announcement of the company's Open Agent Safety Platform follows a series of revelations from top AI companies about their models escaping and breaking into other organizations. The disclosures sparked furious debate about the safety of advanced artificial intelligence systems, including self-improving models that some fear could race out of human control.
The CEOs of Anthropic and OpenAI recently declared America’s cutting-edge models are so powerful, they’re dangerous, and need to be regulated and independently tested before being released. In a rare instance of unity, they have sketched alarming scenarios in essays, social media posts and speeches to the United Nations.
Those aims may not exactly align with what Anthropic engineer Jacob Coxon sought when he quit via a post on X this month, calling for a pause on tech development to keep “superhuman” systems from eluding their makers' control. But the companies saw his post as an opportunity to highlight their own safety efforts and position themselves as cautious market leaders, just when they need fresh capital before going public on Wall Street.
AI Risks a Reality?
The AI safety debate has divided the industry, with the heads of Anthropic and OpenAI championing a coordinated slowdown of Artificial Intelligence development to let safety efforts catch up. But others including Nvidia CEO Jensen Huang say it should be up to individual companies to make sure their models are safe for release.
Huang, during the annual Salesforce technology conference held earlier this month, characterized AI safety, including the danger of rogue agents, as an engineering problem that software developers can address.
President Donald Trump has shunned the need for new AI regulations, dismissing talk of risks to humanity as a “HOAX” designed to help China.
An Anthropic spokesperson said in response to questions for this story that the company has been calling for regulation for several years. An OpenAI spokesperson, Liz Bourgeois, noted the company recently paused training of its most advanced models. “People want to know AI is being developed safely, and that starts with what companies like ours do ourselves," Bourgeois said.
Turning the conversation toward unproven threats — and away from polarizing issues such as data centers’ environmental impacts, uncontrolled hacking incidents, mass AI-powered surveillance and the AI systems' use in warfare — puts Silicon Valley in a more comfortable position, said Sarah Shoker, who previously led OpenAI’s geopolitics team.
As AI companies have gone from building chatbots to advanced AI “ world models ” with 3D awareness, debate has raged over how their technologies should be tested.
Problematic incidents have shown leading AI companies failing to police themselves or design safe experiments. In recent months, leading labs’ AI agents have hacked into external websites after being allowed to escape company training sandboxes, interacted with U.S. government websites in unexpected ways, and appeared to achieve a mathematical breakthrough only to face accusations of stealing mathematicians' work.
Nvidia May Have Answer to AI Risks
Nvidia executives said in a media briefing the new, open-source system could have prevented a recent incident involving a swarm of OpenAI agents that autonomously hacked into AI company Hugging Face.
“From what we know, this new security platform could have stopped the breach if it was being used in frontier labs for model evaluation early on," said the company’s vice president of enterprise AI, Justin Boitano, referring to companies at the forefront of AI.
The Hugging Face incident was a high-profile breach that inflamed the safety concerns about AI, which was followed by similar rogue actions involving OpenAI's models including breaching an Australian health department website. Anthropic and Meta have also disclosed that their AI systems hacked into other organizations on their own.
Nvidia, based in Santa Clara, California, makes high-end chips that have emerged as the leading building blocks for AI. The company's board has cleared the way for the company to spend $150 billion more in share buybacks, bringing its stock repurchase program to $235 billion, the company said Monday.
Nvidia's security software, called OpenShell, lets developers “formally verify an agent has enough authority to do its job and no more,” Boitano said. Because it's open source, it can be “extended” to run on rival computing platforms including those from Arm and Intel.
The platform also includes a separate security layer called Sentry that runs onboard chips to continuously monitor AI agent activity and can "intervene instantly" if the agent starts trying to move beyond its target, the company said.
Nvidia said more than 100 organizations are using the platform at its launch, including Microsoft, Perplexity, Accenture and JPMorgan Chase.
