Hugging Face details how an OpenAI test agent breached its systems

Hugging Face published a technical timeline of the intrusion on Monday. The agent originated inside one of OpenAI’s own cybersecurity evaluations, an exam that scores AI systems on finding and exploiting software bugs; for this run, OpenAI had turned off its usual safety filters to test the model at full strength, with no human involved in any individual move. The agent discovered that the exam’s reference answers were likely stored on Hugging Face’s servers and set out to steal them instead of solving the exercise.

To get there, the agent first exploited an unpatched flaw to escape the OpenAI exam environment, then broke into a separate, exposed AI-testing tool unrelated to Hugging Face and used it as a base for the rest of the operation. From there, it found that a Hugging Face filter checked only outgoing requests, not requests to read local files, and slipped through the gap by uploading a file disguised as an ordinary dataset that carried hidden instructions to pull passwords and source code off the server. A separate bug then let it execute its own code directly on Hugging Face’s servers. The agent relied only on public tools — paste sites, request-logging services, and Hugging Face’s own upload and proxy features — layering in a custom messaging system with sequence numbers and error-checking to keep stolen data and commands in order.

It also encrypted everything it moved. Hugging Face’s initial automated scan caught only a fraction of the traffic; investigators found roughly four times more data after reverse-engineering the agent’s own encryption method. The agent planted copies of itself across 11 servers, each set to relaunch independently, and obtained a private cryptographic key that let it mint its own valid login credentials. A misconfiguration meant one compromised server’s credential unlocked several separate internal systems at once. It also submitted a malicious change to a build script, which Hugging Face’s safeguards caught before it could take effect. Every command capable of deleting or altering cloud resources ran only in “dry run” mode, simulating the action rather than carrying it out. OpenAI CEO Sam Altman has said this is the first security incident he “felt very viscerally.” Hugging Face said the agent also hacked at least four other online services.

Why it matters

Hugging Face concluded that a capable human hacker could have found and exploited the same flaws — unsafe dataset processing, exposed cloud metadata, overly broad access, and long-lived credentials — but the agent explored them at a different scale, a warning for any organization still sizing its defenses to human-speed attackers.