Nvidia and a host of tech giants on Monday launched a new AI safety initiative focused on open models, as the fallout from a cyber attack committed by a rogue OpenAI model continues.
Last week, it emerged that the target of the attack, startup Hugging Face, was unable to use leading U.S. frontier models to defend itself, with guardrails not distinguishing between aggressor and defender. Instead, it turned to a self-hosted, open weight Chinese model, which was not bound by those same restrictions.
In the wake of U.S. lawmakers increasingly weighing how to curb growing adoption of Chinese AI models, the most advanced of which are open weight, tech giants have launched an initiative aimed at building and sharing open AI tools.
Open models can be downloaded, modified and self-hosted, in contrast to closed models — including frontier systems built by Anthropic and OpenAI — which can only be accessed through specific infrastructure.
“The Open Secure AI Alliance will work to remediate and disclose vulnerabilities using open technologies,” Nvidia said in a statement. “The recent Hugging Face security incident delivered a clear reminder: cyber defenders need open, frontier agentic systems for self-defense.”
Alongside Nvidia, other members of the alliance include Microsoft, SpaceX, Palantir, and dozens of other tech companies from the U.S. and Europe.
The push to curb Chinese AI
There are growing calls for measures to limit access to models built by Chinese AI companies, which have been accused of campaigns to extract information from U.S. rivals’ systems, known as “distillation” — when one model extracts knowledge from a better-trained model.
Last week, Treasury Secretary Scott Bessent threatened sanctions on Chinese companies that commit distillation attacks against U.S. companies.
“There is a real possibility the US government does impose restrictions on Chinese models,” Chris McGuire, senior fellow for China and emerging technologies, at think tank Council on Foreign Relations, told CNBC.
That could include a ban on transactions involving the models, such as purchasing tokens via an API or U.S. companies hosting the model on the cloud and charging customers for inference, he said.
“In Washington this is not a debate about open-source vs closed-source, it is a debate about whether or not to tolerate Chinese IP theft,” McGuire said. “Any actions would be focused on Chinese companies, not the open-source ecosystem.”
But with the majority of the most capable open source models being built by Chinese companies, there are concerns over restrictions.
Last week, Nvidia, Microsoft, Meta, Palantir and more than 20 other companies released a letter urging policymakers to avoid “premature restrictions” on open-weight AI models that would “stifle competition or drive innovation overseas.”
The OpenAI-Hugging Face incident showed a “practical truth,” said Nvidia. “When defenders cannot inspect, adapt and run advanced AI on their own infrastructure, their ability to respond is constrained at exactly the moment speed matters most.”
