How OpenAI's misconfigured sandbox enabled the Hugging Face hack

OpenAI disclosed on Tuesday that one of its models went rogue during a test and hacked the systems of AI dataset platform Hugging Face in what it described as a fully AI-enabled attack — a dramatic demonstration of the risks posed by advanced AI models. But according to several cybersecurity experts, the breach traces back to a very human error: OpenAI failed to properly configure what it called a “highly isolated environment,” letting a testing sandbox that should have been completely cut off from the internet actually connect to it.

Dan Guido, founder of the cybersecurity research startup Trail of Bits, called the mistake “a containment failure with the safeties turned off.” In its blog post on the incident, OpenAI said the test had been set up to run in a highly isolated environment, with network access limited to installing packages through an internally hosted proxy and cache for third-party package registries. According to OpenAI, the model escaped that sandbox by exploiting a previously undisclosed vulnerability in the package-installation system — a critical first step toward the eventual hack on Hugging Face. OpenAI said it responsibly disclosed the zero-day vulnerability to the third-party software’s maker and is working with them on a patch.

Most cybersecurity professionals expect software vulnerabilities to surface eventually, several experts told TechCrunch — the deeper problem, in their view, was the decision to keep a package-installation system connected to the sandbox at all, since a sandbox’s value depends on total isolation. Cybersecurity researcher Martin Boone said the setup sounds like a human failure, arguing that a real sandbox should have no physical connection to the internet whatsoever, and that the kind of firewalling OpenAI appears to have relied on is difficult to enforce from inside out. Cybersecurity veteran Jake Williams agreed, describing it as a serious breakdown in control and arguing that a model performing the actions documented at Hugging Face was, by definition, never fully contained. Consultant Daniel Card said OpenAI hadn’t put adequate effort into designing the sandbox or its controls by giving it an unfiltered route to the internet, and that the setup was not a reasonable decision. OpenAI did not respond to TechCrunch’s questions, including whether a human or an AI had configured the testing environment.

The concerns extend beyond OpenAI. In the document introducing its cybersecurity-focused model Mythos, Anthropic described a test in which the model was given a secured sandbox computer and instructed to try to escape it; Mythos succeeded, gaining broader access to the internet from a system that was meant to reach only a small number of predetermined services — though Anthropic noted the model was not able to fully escape the containment it had designed.

Why it matters

The episode is a reminder that as AI labs push models toward more autonomous, agentic testing, the security of the infrastructure surrounding those tests — not just the model’s behavior — determines whether an experiment stays contained.