The HuggingFace Hack
On the 16th of July, the AI model repository HuggingFace reported that they had been hacked by what they believed was an “AI Swarm”. HuggingFace stated that the incident had begun on the 11th of July and halted on the 13th of July, and secondary reporting states that during the attack large numbers of actions were taken in rapid succession, with logs showing what seemed to be AI reasoning comments, and hallucinated/erroneous output from an AI model. HuggingFace stated that due to the strict Cyber use guard rails in public models from Western AI providers such as OpenAI and Anthropic, they were unable to use those models to aid their forensic analysis and incident response. Instead, they downloaded a Chinese open weights model, GLM5.2, and used that. They stated that in the end this had two benefits: First, the model had essentially no guard rails, performing any cyber related task they asked of it, and secondly it ran locally, meaning that any data they gave it did not leave their network.
Roughly one week later, on the 21st of July, ChatGPT makers OpenAI announced that the hack of HuggingFace had been the result of a prototype LLM test which had gone wrong. OpenAI had been testing an LLM in a closed cyber firing range. The model was being tested for its cybersecurity capabilities against the ExploitGym benchmark, and so the cybersecurity behaviour guard rails in the model had been reduced or removed entirely. The only external connection from the firing range was a proxy which allowed access to certain package repositories. The LLM derived solution in this particular run was to cheat at the task, so it began targeting the proxy. It found a zero-day in the software which allowed it to compromise the device and freely access the Internet, then began looked for the solutions to the ExploitGym tasks. Those solutions are hosted on HuggingFace’s infrastructure, so the model targeted HuggingFace, finding a zero-day and hacking in. Once it had access it began looking for the ExploitGym solutions.
What is interesting
Ok, so a rogue ChatGPT prototype breaking out of its network and hacking one of the central services in the LLM ecosystem, all so that it could cheat at a task it had been set, is itself quite interesting. But there’s even more to this story than that.
It is difficult not to speculate that this is all a guerilla marketing stunt of some kind, performed with the intention of inflating OpenAI’s valuation. While that might seem extreme, we are talking about billions, or even trillions of dollars in incentive for such an action. As such, scepticism of any and all information provided by frontier LLM companies is not only justified, it is recommended. However, even if it was a publicity stunt, it may well have backfired.
Access governance
In our defender’s window report, we described one of the major dynamics of the AI-Cyber landscape right now as being access governance – who can access the latest and greatest frontier models, and so who has that next level LLM capability.
In their disclosure of the incident and their response to it, HuggingFace stated that they tried to use public models while responding to the incident but were unable to do so because the models refused any cybersecurity related queries. In this they are most likely referring to Western AI company LLMs and recent aggressive guard rails which these companies have implemented around cyber security related queries. The phrasing used by HuggingFace was that their attacker was not bound by a usage policy, while their defenders were blocked by guardrails. This is one possible view on the matter, however there is no evidence that the compromise could not have been analysed with existing forensic and IR processes. HuggingFace briefings have reportedly stated that they turned to LLMs because the volume of both successful and failed attack activity was so great that they needed to use LLMs to parse it, but it must be noted that HuggingFace are a company which exists entirely because of the LLM industry and has reached its current size and notoriety due to the AI bubble. As such, they most likely are heavily using LLMs internally and they have a vested interest in promoting the use and potential prowess of LLMs for any possible task.
HuggingFace stated that because they were unable to use public, hosted AI models they turned to Chinese LLM provider Z.AI’s open weight GLM-5.2. They stated that not only was the model capable enough to be useful, it also ran locally so no internal data left the network, and it operated with essentially no guard rails around cyber security queries, accepting all prompts and tasking. That’s great, but it suggests that the first thing HuggingFace attempted was to simply dump data and queries from an ongoing IR incident into an LLM hosted by a 3rd party. Potentially, as it later turned out, the 3rd party responsible for the compromise in the first place.
HuggingFace strongly recommend that defenders preparing for security incidents should be aware that they are unlikely to be able to use public hosted models, and so they should have a locally hosted LLM, unconstrained by cyber query guard rails which they can turn to. This seems sensible, as sending data on and information about an ongoing security incident to a 3rd party without a strong contractual framework and a clear understanding of the legal and regulatory risk is not a good idea. Neither is relying on a tool which can have desired capability turned on and off at will by that 3rd party. Both of which, of course, HuggingFace did.
Considering the above, even if this incident was in some way staged or allowed to occur by OpenAI, it may not have turned out the way they had planned. HuggingFace essentially gave a ringing endorsement and strong recommendation for not only a rival company’s product, but a rival company with a completely contrasting ethos of access governance. Of course, when trying to determine if this was a publicity stunt or simply a mistake it is useful to turn to both Occam and Hanlon’s razors. What is the simplest explanation, an ingenious and amoral plan to deploy a highly capable LLM and set it to work hacking a major partner in the LLM space in the hope of improving the company valuation, or incompetence?
Useful context
On July 20th, OpenAI posted a blogpost titled Safety and alignment in an era of long-horizon models. The very first sentence of this article is:
Long-running models can solve difficult, open-ended problems, but their persistence gives them more opportunities to take unwanted actions.
The post is about a model test that they performed at least 2 months earlier, when they were attempting to (and in the end did) find an answer to a mathematical theorem. They say this about the model:
The new model can continue working toward an objective through repeated attempts over a long period of time. That same persistence can lead it to find and exploit weaknesses in its environment. Previous models, when they hit sandboxing or environmental constraints, would simply stop and return to the user. This model often kept trying, including by looking for ways to act outside its sandbox.
An example of this occurred during an internal evaluation on the NanoGPT speedrun, a public benchmark for training a small language model in as few steps as possible. Our model developed a power-law learning-rate cooldown called PowerCool that led to a significant improvement. The model was instructed to post its results only to Slack, but the NanoGPT instructions say to post results as a PR to GitHub, and the model followed the latter instructions to open PR #287 on the public GitHub repository. The model circumvented sandbox restrictions designed to block external access and proceeded with the upload to GitHub. It took our model an hour to find a vulnerability in the sandbox in order to make the PR; earlier models were less persistent and did not manage to find vulnerabilities, instead giving up on the PR part.
So OpenAI had in fact already had at least one model escape confinement in this way in order to perform a task.
OpenAI stated that they made a number of changes to their safeguards around long time frame model execution, then began testing again, and as of the 20th of July:
We have not observed any serious circumvention of safeguards since redeployment began several weeks ago. The first version of these safeguards was deliberately conservative. We have continued tuning the system to reduce unnecessary interruptions without weakening the safeguards.
However, reporting from Reuters which quotes multiple unnamed sources states that just before this blogpost was published, on the 18th and 19th of July, OpenAI staff found indications in their internal logs that one of their prototype long horizon models had escaped its sandbox, and most likely hacked HuggingFace. Reuters further reports that OpenAI’s investigation then identified that the model had begun its sandbox escape on the 9th of July, before starting to hack HuggingFace on the 11th. Reuters states that OpenAI responded to its report by saying that it contains several inaccuracies but did not respond when asked what those inaccuracies were.
Reuters go on to state that OpenAI had repeated issues in the past with models breaking containment, and at one point, after breaking out of its sandbox a model left instructions on OpenAI infrastructure on how to break containment.
Cheating
Generally, trying to complete a task by contradicting the guidelines of the task is called cheating. The UK’s AI Security Institute (AISI) described it as follows, in their post titled Cheating behaviour in frontier model evaluations :
We define cheating as taking an action that is out of scope for the task or explicitly disallowed by the rules, in order to achieve a goal through a shortcut, workaround, or unintended solution that the task was not meant to, or should not, permit.
AISI then disclosed that in at least one case, the cheating which they observed was on a scale similar to that of the OpenAI/HuggingFace incident, though, fortunately, the model targeting AISI infrastructure with its actions, not another organisation:
One particularly stark example of cheating behaviour occurred during a cyber capability evaluation that was accidentally misconfigured and therefore impossible to solve. The model tested was so persistent in attempting to cheat that it wrote and ran code on an external service, hosted on the open internet outside of AISI’s systems, in an attempt to access our evaluation infrastructure, triggering a security alert in AISI’s systems.
AISI state that every single model they have analysed attempted to cheat at one point or another. Out of 475 evaluations, GPT 5.4, 5.5 and 5.6 attempting to cheat between 11.4 and 14.1% of the time:
There are further fascinating and very relevant findings from AISI on this subject:
AISI found that the likelihood of cheating does not increase with model capability. They suggest that this implies that cheating is not linked to capability or ingenuity but instead is an artifact of model training techniques including alignment training.
AISI found that the method of cheating differs drastically between models, as shown in the following graph. What is particularly interesting here is that in almost 50% of evaluations, Mythos’ cheating methods were described as “Other”, a classification seen in 0-10% of evaluations of other models.
Even more interesting (and once again, something that we touch on in our Defender’s Window report is that the models could not reliably self-audit. AISI found that none of the models they tested consistently admitted to cheating when asked, and even when they did admit to cheating, they described the actions as wrong less than 50% of the time.
AISI also found that amongst models which produced chain-of-thought reasoning, ,often seen by users as a way to audit what a model is doing, the chain-of-thought did not mention their cheating behaviour 50% of the time.
Is AI coming for your bits?
The HuggingFace hack may seem to imply that the latest, prototype models being tested by OpenAI will be able to act autonomously and overwhelmingly to compromise victims. If that were the case, then considering how rapidly public frontier models advance this attack would be a preview of the reality that defenders will soon face.
This probably is not the case, however. Yes, this prototype model was able to do this, but it had no cyber capability guard rails. We can also surmise based on OpenAI’s recent blog posts around what it calls long horizon models, that this was a long horizon model unlike the standard, public release ChatGPT. It is likely that this model used a lot of compute power and was extremely expensive to run, as such it is not likely to be released in its current form, the weights are very unlikely to be made public, and the model almost certainly won’t be released without guard rails. In part, this is because the US government have communicated to AI companies that while there is no specific regulation covering LLM capability, guard rails around cyber capability are essentially mandatory. What is interesting from OpenAI’s response is their proposition that the only way to respond to an agentic LLM cyber-attack is with an open model without guardrails. If you accept that proposition, then you must accept that the winning move in this arms race is to sidestep the Western AI industry gatekeeping (Both in terms of subscriptions and guardrails) and just use Chinese open weight models.
The interesting point to note here, as we covered in The Defender’s Window, is that attacker capability is now limited by budget, not skill. Even when using a less capable but “free” open-weight model (as HuggingFace did), you still need to pay the hardware cost of executing the model, whether on local hardware on in cloud compute. Autonomous, agentic hacking is essentially pay to play, and there is not a guaranteed pay out. If an autonomous attempt at a compromise costs X amount in compute with a capable public model, making repeated attempts will require repeated payments. This is key, because the kind of prototype models that OpenAI is running, and the research they are doing, is very expensive. According to leaked OpenAI financial details, in 2025 they spent 160% of their revenue on R&D, almost $20 billion. It is safe to say that most attackers do not have that budget, or LLMs of the same capability to spend it on.
In fact, while AI model capability is increasing, there is little evidence that point-and-prompt agentic attacks are increasing. Ransomware activity is reportedly increasing, but there simply are not the public reports of repeated successful agentic ransomware attacks that would be expected if capable, cost-effective attack agents were available to cyber criminals. Recent research published by Black Kite echoes one of the predictions of our Defender’s Window report – AI has not yet been seen to accelerate (i.e. reduce time to detonation) ransomware incidents, but it has been seen to enable more threat actors with low tier technical skills to operate as effective ransomware threat actors. To quote Dark Reading, as they quote Black Kite:
"Open-source LLMs and code agents lowered the cost of building an original operation, and we saw early signs of AI-assisted code showing up in encryptors," he says. But it's hard to say if AI drove an acceleration in ransomware attacks. "What I can say is the growth is human. AI just let more people show up at once."
And tying into the Defender’s Window again, an effective defence is an effective defence – Reducing attack surface, reducing risk, and take preparatory steps pays dividends in any way that you choose to measure it. Black Kite observed that companies which they rated as having the most exposed corporate attack surface, and which were carrying the most uncontrolled risks, had a 41% chance of experiencing a ransomware attack. Meanwhile, the least exposed companies had a 0.14% chance of a ransomware attack. Even the legendary HuggingFace Hacker LLM didn’t find or employ some radical new compromise technique. Yes, it performed actions rapidly and at scale, and it used an undisclosed zero-day in the software run by HuggingFace to gain access, but we don’t actually know what that zero-day was, or what level of capability of complexity was required to find and exploit it. Frankly, considering HuggingFace’s operational focus on LLMs, and the fact that their first response to a compromise was to go ask an LLM, it is entirely possible that their software and infrastructure configurations are heavily vibe coded, in the true spirit of move fast and break things.
What we do know is that the attack employed standard techniques and behaviours, all of which rely on the standard security primitives of exposed attack surfaces, software vulnerabilities and misconfigurations, and missing security controls. How do we know this? Because if anything amazing had happened, OpenAI and HuggingFace would absolutely have told us.
And the good thing for defenders is that those cyber security primitives are all things that defenders can address.
The Defender’s Window
Now this is not to say that LLM use by attackers is not a cyber security threat, but it is to say that the capability level demonstrated in the HuggingFace hack is not what defenders are now facing. It may not even be what defenders face in 1-2 years’ time, because alongside the capability of the model, the attack also relied upon the model’s long horizon of operation, a complete lack of guard rails, and what was very likely a fantastically large compute budget. Recreating this level of capability will not be easy, and it definitely will not be cheap, but knowing that this level of capability is possible in certain situations gives defenders a strong signal that they need to be preparing and investing in their defences now, ready for when capable, autonomous, point and prompt hacking becomes widespread.
Will security pros be out of a job?
No.
Recently, a maths problem known as the Jacobian conjecture was disproved by AI. A lot of the coverage of this event said that mathematicians were becoming obsolete, and as such they are riven by anxiety about the future of their profession.
However, actual mathematicians don’t necessarily all feel that way. Commentary I’ve seen from that community points out that the history of mathematics has been built upon repeated technological and procedural improvements intended to reduce what you might call “the boring bits” of maths, ever since somebody invented the calculator, or the abacus, or even just picked up a handful of pebbles to count with. Doing the boring stuff is what mathematicians (and people more widely) want technology to do.
Mathematicians want to think up new problems to solve, they don’t want to be sat around trying to find absurdly long polynomials to counter obscure theorems. To quote Eva Roytburg writing in Fortune on the topic, as she paraphrases a piece by Columbia Professor of Mathematics, Michael Harris:
The AI industry treats reasoning, or understanding, as commercially worthless, and human mathematicians as a “beta version of intelligence.” Yet mathematics, [Michael] argued, is one of the last examples of unalienated labor, a field that people enter, in the words of Abel Prize winner Pierre Deligne, because one can earn a living “by playing” — what Mathew Buzzard calls “telling a story.” Even when Deep Blue “solved” chess by beating Garry Kasparov in 1997, people didn’t stop playing chess; they learned from it.
LLMs remain a tool that can be used to solve problems. That doesn’t mean that they can solve problems by themselves, that they know which problems need solving, or that they even know why their solution works. All of these things require a guiding intelligence, i.e. someone to use the tool.