When Are Open-Weight Models Too Dangerous to Release? The US and China Both Lack Answers

Summary

• Free to download: Chinese AI models Kimi K3 and GLM 5.3 are just months behind American frontier models in cyber capabilities, and are freely available on the internet.
• Missing rules: Neither China nor the US has set standards that developers must meet before publishing the weights of frontier AI models.
• Two to tango: China is unlikely to restrict future releases on safety grounds, and the US has so far exempted open-weight models from its pre-release evaluation framework.
• Recommendations: The US should reverse the exemption, and start discussions with China on what constitutes a red line for open-weight model releases.

‍

In July 2026, Chinese company Moonshot released Kimi K3. At 2.8 trillion parameters, Kimi K3 was the largest open-weight model at launch and arguably the most generally capable Chinese model ever released. Barely a month later, a second Chinese company, Z.ai, released GLM 5.3 – a model with general capabilities almost as strong as Kimi K3.

Both are open-weight models: their weights are available on the internet for anyone with sufficient computing resources to download, run, and modify.

As Chinese open-weight models display increasingly alarming cyber capabilities, they present a problem. Neither the US nor China has set a threshold for what counts as too dangerous to release. Without that threshold, nobody can say whether or not Kimi K3 and GLM 5.3 have already crossed it.

How dangerous are these models?

The UK AI Security Institute and the US Center for Advancing Innovation and Standards for Super Intelligence (CAISSI, formerly CAISI) together evaluated Kimi K3’s offensive cyber abilities. They found that in one of ten attempts the model completed a simulated 32-step attack on a corporate network, a task that would take human experts around 20 hours. 

Moonshot also conducted its own evaluation. Kimi K3 found 16 previously unknown software vulnerabilities and built working exploits for 14 of 36 known vulnerabilities. Yet Moonshot has still named no threshold for keeping the weights of future models unpublished.

On biological capabilities, independent evaluator SecureBio found that Kimi K3 was roughly comparable to closed models from about eight months earlier. When prompted with questions designed to elicit dangerous biological information, the model declined to answer only 26.9% of the time. By comparison, top-performing closed-weight models refused over 90%. More importantly, such refusals matter little in open models, since safeguards are easy to remove once the weights are downloaded.

What about GLM 5.3? On 17 September, CAISSI found that GLM 5.3 had even greater cyber capabilities than Kimi K3 and lagged the US frontier by about 4 months. Z.ai completed its own evaluation for cyber capabilities, and delayed publishing its weights out of cybersecurity concerns. Anthropic’s Frontier Red Team reported that GLM 5.3 could build end-to-end exploits at close to the rate of Mythos Preview.

Will China restrict open models? 

While China has benefited from leading the world in open-weight models, it is also vulnerable to the same risks of misuse. In September, China’s Minister of State Security acknowledged that AI models like Claude Mythos are “sharply lowering the technical threshold and cost of carrying out cyberattacks, and posing serious risks to China's critical information infrastructure”. If Chinese AI companies keep publishing model weights, China will be as exposed as anyone to non-state actors using these models without safeguards.

Given what evaluations tell us and the current trend in capabilities, the safest course of action would be to stop releasing the weights of models more powerful than these. The only actor who can enforce this on the Chinese side is the Chinese government. 

It is true that China regulates AI more prescriptively than most jurisdictions, including through mandatory security assessments, algorithm registration and content rules for public-facing services. However, these obligations govern how AI is deployed in China, not whether the underlying model is too capable to release openly. 

Beijing's AI Safety Governance Framework 3.0, released on 14 September, acknowledges the downstream risks of open models but does not establish a clear, pre-release capability threshold. The AI-focused article added to the Cybersecurity Law in October 2025 and the draft cybercrime law circulated in January 2026 also attach duties to deployed services rather than models before release. 

May 2026 discussions among Chinese legal experts favored a tiered approach that preserves open-source development while tightening controls on the most advanced systems. However, their concerns center on protecting sensitive training data and China’s strategic position rather than catastrophic misuse. 

Chinese authorities in the Ministry of Commerce (MOFCOM) have reportedly discussed restricting overseas access to the country's most advanced models, limiting foreign investment, and treating leaks of AI technology as a national security offense. It’s not currently clear whether these proposals would prevent future releases of open-weight models. Either way, the reported discussions indicate that China’s focus would be on industrial competition, rather than how capabilities might spread or the potential for misuse. 

If China won’t restrict open models, what should other governments do?

In an ideal world, Washington would press for pre-release testing at the next iteration of bilateral AI talks, currently planned for November. 

However, in August, the White House finalized a non-public framework for voluntary pre-release evaluations of AI models, which reportedly excludes open-weight models. Washington cannot credibly ask Beijing for pre-release testing of open models when it has just declined to do so domestically.

If anything, that is a reason for the US to set some release norms for open weights of its own. While there has been clear industry pressure in the US against restrictions on open models, US officials have reportedly said that bringing open models into scope is still in consideration. If sincere, this needs to be expedited. 

The next best option would be to closely monitor models once they are released and track safety incidents. The US-China agreement in September to establish a bilateral channel for AI incidents could become a route for sharing early-warning signals on open-weight models, though so far details of this agreement are scarce. 

Middle powers, meanwhile, should not wait for Washington and Beijing to act. Governments with the technical and institutional capacity should evaluate these models to better understand what is being deployed. A wider set of evaluators would broaden the credibility of the evidence and help establish a common basis for understanding what these models can do.

At the same time, no amount of evaluation can prevent AI misuse once model weights have been published. The US and China need to move toward agreement on the categories of capability to measure, the results to report and, crucially, what those results should trigger. Whether Kimi K3 and GLM 5.3 were too dangerous to publish is a question no one is currently prepared to answer.