How OpenAI's agent escaped: Sprung by humans in a series of preventable events
AI News Desk
·
ZDNet
··
15 min read
Follow ZDNET: Add us as a preferred source on Google.
Follow ZDNET: Add us as a preferred source on Google.
On July 16, the AI community website Hugging Face reported being targeted by "an autonomous AI agent system" of unknown origin that unleashed a torrent of traffic on its domain, flooding its security logs with more than 17,000 events, some of which ultimately succeeded in exfiltrating secret information stored in its databases.
According to Hugging Face, the attacker gained "unauthorized access to a limited set of internal datasets and to several credentials used by our services" and appeared to be "run by an autonomous agent framework (appearing to be built on an agentic security-research harness - used LLM still not known)." My ZDNET colleague Charlie Osborne reported on the intrusion .
Also: OpenAI's rogue agent didn't stop at Hugging Face - here's what we know
Five days later, on July 21, OpenAI stepped forward to claim responsibility for the attack, and all hell broke loose (including reports of other organizations targeted as part of the incident ). The media responded with a range of fear-mongering stories that essentially made it look as though ChatGPT went rogue and decided, of its own volition and malice, to attack Hugging Face's systems.
Then yesterday, adding fuel to the fire, Anthropic made a similar disclosure about its models inadvertently attacking other organizations as a part of its ongoing safety testing.
(Disclosure: Ziff Davis, ZDNET's parent company, filed an April 2025 lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.)
As I noted in my coverage of OpenAI's disclosure, Hugging Face was correct in that it was an agent under the direction of an autonomous security research framework. But, humans were unquestionably in the loop -- and at least some of the agent's behavior should have been anticipated.
Importantly, it wasn't ChatGPT itself that was responsible for the attack, as some commentators insinuated. Rather, the attack was attributable to an agent under the direction of OpenAI's AI safety researchers, who, in an environment supposedly isolated from the internet, deliberately provisioned it to attempt a series of exploits as part of an AI safety test. As often happens in the labs of various frontier models, AI safety researchers were attempting to gauge the capabilities of OpenAI's latest large language models (LLMs).
Also: Why enterprise AI agents could become the ultimate insider threat
The "unprecedented cyber incident" (as OpenAI called it) has been widely described as an agent escaping its theoretically secure enclosure and wreaking havoc on Hugging Face's systems. Such enclosures are sometimes discussed in technical circles as "sandboxes" -- even OpenAI's disclosure makes reference to a "sandbox environment." However, in using that phrase, my sources have suggested that the environment may simply have been a firewall configured to emulate a sandbox rather than an actual third-party sandbox solution such as Blaxel, Daytona, E2B, or Modal. OpenAI has not yet disclosed the details of the solution it was using or its provider.
All the talk of agents and sandbox escapes prompted my editor at ZDNET to ask, "How does an agent escape a sandbox, anyway, and can it be prevented from happening again?" In search of an answer, I reached out to OpenAI and Hugging Face. Neither replied. But between the two companies' public disclosures and my other sources, there's enough information to start theorizing about how such an "escape" might happen.
My editor wasn't the only one asking these questions. Since my original post, I've been asked all sorts of questions about what agents are and how, on their own, they could manage to escape from a supposedly secure testing environment to wreak their havoc on the internet. Even my Dad asked me, "What is an agent?" So, let's start there.
There's a big difference between malice and agency. While the attack was unquestionably a malicious intrusion, the involved agent itself harbored no malice. It didn't wake up one morning and decide to attack Hugging Face's systems. Instead, it was afforded the agency to very dispassionately do everything it did -- embark on a safety test, break out of its confines, choose targets, and exploit those targets -- by humans.
Also: Why this fully agentic ransomware attack is giving researchers nightmares
Some of that agency was by design, and some was inherited from the powerful LLMs that OpenAI's AI safety testing personnel were testing at the time. Also, in the case of OpenAI's usage of the ExploitGym open source AI testing solution to conduct this particular bit of safety research, some of that agency was the result of previously undiscovered vulnerabilities in both the third-party "sandbox environment" meant to securely quarantine those tests, as well as in parts of Hugging Face's infrastructure that the agent also exploited.
It was unquestionably AI's version of a series of unfortunate events. Was it preventable? My short answer: Yes, the calamity was preventable by taking at least one single and reasonable precaution -- a precaution that the developers of ExploitGym apparently take themselves, but one that OpenAI may have overlooked.
In a comprehensive tweet that everyone should read, UC Berkeley computer science professor Dawn Song wrote that ExploitGym's "evaluations are designed to run within isolated sandbox environments with tightly restricted network access." Song is also a member of the team at UC Berkeley's Center for Responsible Decentralized Intelligence, which is responsible for the development of ExploitGym. Via email, Song told me that ExploitGym essentially comes with its own sandbox but that companies like OpenAI are free to make substitutions.
Also: As AI agents spread, 1Password's new tool tackles a rising security threat
According to Song, "network and model-API proxies that restrict which external services the agent can reach" are among the major components of the ExploitGym reference implementation. "During the actual agent-execution phase in our current reference setup, outbound network access is restricted to the services necessary for the evaluation. We also use an LLM proxy to block provider-side web search, remote URL retrieval, hosted code execution, and similar channels that could otherwise bypass a container-level firewall," Song told me. "OpenAI may have modified this architecture to fit its own internal infrastructure, scaling requirements, and evaluation systems. This is a common practice as they have different needs for their internal evaluation. We cannot say precisely how its implementation differed from ours."
In addition, the ExploitGym team was clearly aware of how, with tests governed by their framework, models were exhibiting potentially pernicious behavior. According to Song's tweet, "During development, we observed models probing the surrounding infrastructure for additional privileges or information beyond what was intended for the task. We also deliberately tasked models with stress-testing our infrastructure so that we could identify and fix potential weaknesses."
Seeking to determine whether OpenAI's researchers should have been better prepared for the possibility of an attempted agent escape, I asked Song whether these observations had been made public prior to the cyber incident.
"We did not separately publish additional observations from our internal development and experimental process," Song replied. "That said, the broader phenomenon of frontier models attempting to probe their execution environment or seek additional information or privileges during evaluations is not something unique to the ExploitGym experience. Similar behaviors have been observed and discussed by other researchers as well."
Also: Assume AI cybersecurity attacks are the future: 43% of companies have already experienced it
In other words, when used for safety testing and benchmarking purposes, and when enabled by powerful LLMs, ExploitGym-enabled agents are essentially expected to try everything within their power to break out of whatever sandbox is trying to securely contain them.
And that's precisely what happened in this case. As OpenAI wrote in its disclosure, "While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we've now responsibly disclosed to the vendor) in the package registry cache proxy."
When ExploitGym was used to task OpenAI's newest models with an objective, those models were relentless in their pursuit of access to the outside world, beyond the walls of their container.
Song said: "Exploiting the evaluation infrastructure to escape the sandbox or enter an unrelated real-world system is a security boundary violation. It is not part of ExploitGym and must be prevented. The OpenAI–Hugging Face incident falls into this category." In a separate email, Song wrote: "When evaluating increasingly capable AI systems -- especially on cybersecurity tasks -- the evaluation infrastructure itself should be treated as part of the attack surface and designed, enforced, and continuously verified accordingly."
OpenAI has yet to disclose details regarding what precautions were taken.
So why, under the framework of ExploitGym, were the models so zealous and tenacious in pursuit of a breakout? If you read between the lines of what OpenAI said above, its models were simply trying to figure out how best to complete an exploitation test (the "evaluation problem") that humans gave them. In the same way that public LLMs like ChatGPT sometimes rely on information on the internet to answer questions, the models being tested by OpenAI appear to have turned to the internet for help.
"Older models might rely only on what they know from their training," said AppOmni director of AI Melissa Ruzzi in a Zoom interview with ZDNET. AppOmni specializes in SaaS and AI security and also deals in active threat intelligence research. "But, ask any new model out there how important it is to be up to date to solve a security vulnerability, and all of them are going to say that it's of the highest importance. I doubt any model is going to tell you, 'you don't need to know the latest things and just use whatever you already know.' [It will assume] that it shouldn't trust itself and then it will task itself with discovering what's new."
Also: Cloud attacks are getting faster and deadlier - 4 ways to secure your business
But if OpenAI's models looked to the internet for the latest and greatest information and encountered the boundaries meant to confine them, they very dispassionately and logically took the next obvious step: to find a workaround for those boundaries. They did what experienced threat actors might do, but at machine speed. For all intents and purposes, the models appear to have conducted penetration tests on their boundaries until they found a vulnerability, and from there, a path to the internet was discovered as a part of a process that also leveraged some other common human threat actor tactics, techniques, and procedures (TTPs) -- including moving laterally to other systems and escalating privileges as necessary.
To some extent, we can thank the models that OpenAI included in its safety test for these novel achievements. One of those models was GPT 5.6 Sol , OpenAI's most powerful publicly available model. Another was an unnamed pre-release model that OpenAI claimed to be even more "cyber capable" than GPT 5.6 Sol. The company later stated that the pre-release model was "an internal-only research prototype and was never intended for public release."
However, according to information from ExploitGym's developers, GPT 5.6 Sol might also be considered the most dangerous model on the market. Folks in the AI world don't like to use scary words like "dangerous." They use the phrase "cyber capable" instead.
Also: OpenAI's GPT-5.6 and ChatGPT Work aim to beat Anthropic on price, speed, and productivity
As noted in ExploitGym's most recent leaderboard (see chart below), GPT 5.6 Sol is currently the top-ranked model for its "cyber capability" to not only successfully exploit the highest percentage of real-world vulnerabilities, but to also survive attempts by defenders to mitigate attacks under its direction. As can be seen from the chart, it is even more dangerous than the Preview of Anthropic's Mythos, which just a few months ago, gained significant notoriety for its cyber capability. Even OpenAI coyly referred to the prowess of its supermodel when it wrote: "We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities."
According to a leaderboard prepared by developers of the ExploitGym testing solution, OpenAI's GPT 5.6 Sol model is the top ranked model in terms of its so-called "cyber capability."
But these so-called cyber capabilities, along with the buggy sandbox environment, were not all it took for the agent to pursue its exploits. Whereas the public version of GPT 5.6 Sol found in ChatGPT is governed by parameters that, as best as they possibly can, prevent the model from being used for harmful purposes, the version of GPT 5.6 Sol used in OpenAI's ExploitGym-based testing was given much more freedom to do malicious things.
According to OpenAI's disclosure, the company was gauging the cyber capabilities of its models "by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity." As OpenAI hasn't responded to my inquiries regarding this incident, I asked AppOmni's Ruzzi what a "production classifier" is.
Ruzzi explained that it's a parameter that tells the model whether or not it's running in a production environment. If it's a production environment like the publicly available version of ChatGPT, then all safety guardrails are activated. But if the model is in a non-production environment -- for example, the ExploitGym testing environment that's supposedly sealed off from the outside world -- then the guardrails are relaxed, and the model is given the leeway to pursue "high-risk cyberactivity."
In layperson's terms, they took the seat belts off the crash test dummy before driving the car into a brick wall. It was another human decision, in a series of human decisions, that led to the so-called escape of the agent.
Also: 10 ways AI can inflict unprecedented damage
Keep in mind that the models were tasked to solve a problem. They weren't told how to solve the problem, nor were they asked to respect any moral, ethical, or legal boundaries. In other words, they were afforded -- by humans -- the agency to try anything.
Perhaps the most interesting point in OpenAI's disclosure comes when it says, "After gaining internet access, the models inferred that Hugging Face potentially hosted models, datasets, and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation."
Let's unpack that. Although we can't be 100% certain from the information provided, the implication is that, among the steps that the models were willing to take to solve the so-called "evaluation problem," one was simply to cheat by stealing a pre-existing solution (perhaps one "honestly" figured out by another model). According to Ruzzi, the models had a key piece of information to go on: from their ability to probe their testing environment, they could tell they were taking part in an evaluation governed by ExploitGym. And when the models began looking for the latest information on ExploitGym, they likely encountered a page like this one, hosted on Hugging Face's systems. Suddenly, Hugging Face became a target of interest, and in much the same way as the models relentlessly exhausted every option to break out of their confines in OpenAI's AI testing infrastructure, they exhausted all possible options to break into Hugging Face's systems.
Also: AI agents of chaos? Research shows how bots talking to bots can go sideways fast
Ruzzi was quick to note that the word "cheat" is a bit loaded in this context, implying that the models were not emotionally compelled to pass the test without doing any of the hard work. OpenAI may have chosen to use the word "cheat" because of what humans understand the word to mean. But the model never thought to itself, "Ha ha, I'm going to beat the system." It just pursued a plausible path to attain the objective.
In describing how ExploitGym governed tests work, however, Song told me that "obtaining hidden information through an unintended external channel (such as web search, an API feature that retrieves web content) invalidates the evaluation. Because the system is no longer measuring whether the agent can derive an exploit from the provided task information." In other words, under the way ExploitGym-governed evaluations are intended to work, the agent in question is essentially tasked with figuring out an exploit on its own.
It is worth noting, however, that, just as a cheating student seeks to evade the watchful eyes of a proctor during an exam, the attack on Hugging Face's systems included another TTP commonly used by human threat actors: decoys. According to Hugging Face's disclosure, "To understand what a swarm of tens of thousands of automated actions did, we ran LLM-driven analysis agents over the full attacker action log, comprised of more than 17,000 recorded events. This allowed us to reconstruct the timeline, extract indicators of compromise, map the credentials touched, and separate genuine impact from decoy activity." Although the full details were never published, the implication is that at least some of those 17,000 events were designed to throw Hugging Face's defenses off the scent of the actual attack as it was underway.
In the attack on Hugging Face, did a machine act on its own? It's a moot question. Knowing exactly how we got here -- was it sentience... malice...agency? -- is relevant to whatever remedy lies ahead. Most important, however, is that we've arrived at a teachable moment in which human AI experts gave AI an objective, lost control of AI's pursuit of that objective, and then AI demonstrated unmatched tenacity and speed in accomplishing it.
Also: Open weights vs. closed: An AI civil war's afoot, and the stakes are existential
In the big picture, it's that unmatched tenacity and speed that should be most worrisome. Never mind a relatively benign cyber-incident like this one. There should be no doubt that threat actors paid close attention to what happened here.
Fortunately, OpenAI is not a threat actor. As far as we know, the damage amounted to an unfortunate but timely demonstration of cyber capability rather than any intent to inflict lasting damage on another company. Yes, it will happen again. If not by accident, then at the hands of an adversary. And when adversaries are involved, they'll go after soft but valuable targets (like big businesses that lack the defenses Hugging Face has in place). When that time comes, it's anybody's guess just how prepared we'll be.
