AI guardrails meant to stop hackers now block researchers

For months, AI companies have built vetted programs and strict guardrails to keep malicious hackers from using their models to build cyberattacks. In June, the U.S. government imposed export control restrictions on Anthropic’s Mythos and Fable models, a move prompted at least in part by a report claiming their guardrails could be bypassed for malicious cyberattacks. Anthropic had marketed Mythos as a tool so sensitive it could only go to carefully vetted users under strict guardrails. The restrictions have since been lifted: Fable 5 returned to general access on July 1, while Mythos 5 has been reintroduced only to vetted U.S. organizations as part of the government’s review process.

That gatekeeping extends beyond Mythos. Anthropic and OpenAI both run vetting programs — Anthropic’s Cyber Verification Program and OpenAI’s Trusted Access for Cyber — that give approved researchers access to models with fewer cybersecurity restrictions. Mark Dowd, a researcher who has spent decades finding and selling zero-day vulnerabilities to Western governments rather than reporting them for a patch, said on a cybersecurity podcast: “it’s not really comfortable to me that these random large companies are making arbitrary decisions about what is safe in security and what’s not.”

Chris Anley, chief scientist at NCC Group, said asking a model to try to exploit a bug is a key step in confirming it’s a real vulnerability worth fixing — but when a guardrail makes the model refuse outright, it hurts defenders as much as it blocks attackers, since a prompt like “fix this code” is simultaneously a defensive tool and a roadmap for finding vulnerabilities. When he hits that wall, he and his colleagues sometimes fall back on open-source models that carry no guardrails at all. Paolo Stagno, chief technology officer at Crowdfense, said his team avoids using cloud-based AI models for vulnerability research or exploit-building altogether, over concerns the work could leak or be absorbed into future training runs, and instead runs open-source models locally for that step.

Not every researcher is affected the same way. Giuseppe Cali said guardrails don’t slow him down because he doesn’t use AI for offensive work at all — only for reverse engineering and building supporting tools. But Chris Thompson, chief executive of RemoteThreat, said guardrails behave inconsistently even inside the vetted programs, so researchers spend time negotiating with the model instead of analyzing exploitability — a friction he said pushes some researchers toward unrestricted, freely downloadable Chinese open-source models like GLM instead.

Why it matters

If strict, inconsistent guardrails keep pushing legitimate researchers toward foreign open-source models the labs can’t monitor, the safety controls built to stop attackers could end up weakening the defenders racing to find the same flaws first.