Open source AI models should not be allowed to publish weights for frontier-scale systems

The core case for restriction is asymmetric: the safety benefits of open weights (enabling safety research) can largely be achieved through API access and audited access programs. The harm case is different in kind — open weights are permanently out there once released. A researcher studying bioweapon risks can get structured API access; a state actor wanting to fine-tune a model for WMD uplift needs the weights. The asymmetry favours restriction at the frontier.

The 'marginal uplift' argument applies to current models. The debate is about frontier models as capabilities improve. A model trained on 10^26 FLOPs with broad internet knowledge and chemistry/biology training may provide meaningfully more uplift than current models. Building the legal infrastructure to restrict it now, before the capability threshold is crossed, is standard risk governance. We regulate drugs in advance of their synthesis, not after.

The ITAR/cryptography analogy fails on asymmetry of consequences. Bad cryptography meant less secure commerce. Bad AI safety at the frontier means... what exactly? The biosecurity risk alone — designed pathogens, automated gain-of-function research — involves tail risks with civilisational consequences. We don't apply normal cost-benefit to nuclear weapons either. At some capability level, the consequences of being wrong are large enough that the burden of proof for unrestricted release should be high.

Define-the-threshold-first is the argument that has been used to delay action on every technology risk from CFCs to algorithmic trading to COVID preparedness. The threshold for demonstrable harm is, by design, only reachable after the harm has occurred. The question is whether the expected cost of being wrong tilts toward premature restriction or premature openness. For irreversible, catastrophic-tailed risks, premature restriction is usually cheaper. I'd rather be the country that regulated too early than the country that published the weights for the model that helped design the pandemic.

Look at the actual track record of information restrictions. Every restriction justified on security grounds eventually becomes a tool for incumbent protection and political censorship. ITAR controls on cryptography in the 1990s delayed the commercial internet by years, provided zero security benefit as the same algorithms were independently developed globally, and were walked back only after enormous industry pressure. There is no reason to believe AI restrictions would be better targeted or have smaller collateral costs.

The precautionary argument always sounds compelling for the most scary capabilities. But 'at some capability level' is doing all the work. We are not at that level now, the threshold has been moved every year as capabilities improve without the predicted catastrophes materialising, and building restrictive infrastructure now — before the risk is demonstrated — creates path dependency that will be used to restrict far less dangerous systems on the same logic. Define the threshold first. Restrict after.

The asymmetry argument proves too much. Nuclear knowledge is restricted, but nuclear materials are the actual binding constraint on weapons programs. For AI, the compute and data are the binding constraints, not the weights. A model weights file doesn't contain training data or compute — a bad actor still needs enormous resources to do anything interesting with a weight file that they couldn't already do by other means. The security threat model is wrong.

The 'frontier models' framing hides that the regulatory target will be defined by governments under industry lobbying, invariably in ways that entrench incumbents. Meta's Llama series, released open, has produced an enormous amount of safety research that closed models don't enable. The specific biosecurity risk claim (LLM uplift for bioweapons) has been studied; the consensus finding is that current models provide marginal uplift over freely available information. We're restricting real democratisation benefits to address marginal risks.