Neon cyan line drawing of a helmeted figure with a round visor against a black background.

Hugging Face: Over the router, through the firewall, off to Grandma's house we go

Indeed yes! The title is borrowed from Eric Schultze and George Kurtz’s Black Hat Briefings ’99 talk where they described (last century!) a live cross-platform demonstration of chaining router, firewall, and host weaknesses to reach a targeted internal network [21].

The short version

The stock dismissal of a large language model (LLM) is that it is autocomplete with airs: A predictive word generator, with little more intelligence than your phone suggesting the end of your sentence. Taken literally, that is not a bad summary: The model maps text to text, but it has no “agency” so it cannot (on its own) run a command, open a connection or exploit anything; it can only describe what it would do. But if we wrap that text generator in an agentic system (a loop that takes what the LLM produces, runs it against the world, observes the result and feeds it back for the next step) and the model’s reasoning starts driving real actions.

And that’s what happened! Two of OpenAI’s models were sitting in a sealed test environment being scored on how well they could turn known software bugs into working attacks. The answers to the problems they’d been tasked with were not in their sealed environment, and the only wire out led to a software-package caching proxy, a service that could fetch and cache software packages from the internet.

Apparently the problem that the AI models had been tasked with was very hard, so the models turned their focus to finding a solution outside of their test-environment confines. They “spent a substantial amount of inference compute finding a way to obtain open Internet access” [2] and they achieved that goal by discovering a zero-day flaw in the package caching proxy; they then exploited that flaw so they could reach a machine with an Internet connection, before breaking into Hugging Face (a collaborative AI platform) to try to find what they wanted.

Back in 1999, Schultze and Kurtz demonstrated some similar chaining techniques live on stage. But that was two human experts in a controlled demonstration. This event, by contrast, involves no humans, just an LLM on a mission, breaking out of its cage, and attacking an unrelated organisation’s service, to try to find a solution to a benchmarking problem. As OpenAI noted, that is “unprecedented” [2].

Why this post?

We wrote this post as a byproduct of working to understand what this event is all about. There are lots of other writeups, but this one is ours. It separates claims that are independently verifiable from claims that rest solely on the accounts of the two companies involved. Both of the companies involved have described their findings as preliminary, and their two accounts do not fully reconcile on all points, so it’s possible that we’ll need to revise this post in the future.

Timeline of disclosures

Hugging Face published an incident disclosure on Thursday 16 July 2026 [1]. It stated that earlier that week it had identified unauthorised access to a limited set of internal datasets and to several credentials used by its services [1][3].

OpenAI published a related disclosure on Tuesday 21 July 2026, stating that its own models were responsible for the activity Hugging Face had detected [2][7]. This attribution was reported the same day by Axios, Fortune, Unite.AI and others [6][7][8][9].

What Hugging Face stated

On or around 16 July, Hugging Face attributed the intrusion to an attacker using a malicious dataset that abused two code-execution paths in its dataset processing pipeline: a remote-code dataset loader and a template injection in a dataset configuration. It stated that these were used to run code on a processing worker [1][3][5].

From that foothold, Hugging Face stated that the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend [1][3].

Hugging Face stated that it had found no evidence of tampering with public, user-facing models, datasets, or Spaces, and that its software supply chain (container images and published packages) was verified clean. It stated that its assessment of whether partner or customer data was affected was still ongoing [1][5][16]. Reported remediation steps included removing the attacker’s foothold, rebuilding compromised nodes, revoking and rotating affected credentials, a broader precautionary rotation of secrets, deploying improved detection, and reporting the incident to law enforcement [1][5].

At the point of its 16 July disclosure, Hugging Face stated that an LLM was behind the attack but that the actual LLM was not known, and it characterised the campaign as an autonomous agent framework executing many thousands of individual actions across short-lived sandboxes, with self-migrating command-and-control staged on public services [1][4][6][14].

Hugging Face’s use of AI in detection and forensics

Hugging Face stated that its own anomaly-detection pipeline, using LLM-based triage over security telemetry, first flagged the compromise, and that it then ran LLM-driven analysis over more than 17,000 recorded attacker actions to reconstruct the timeline, identify affected credentials, and separate genuine activity from decoy activity [15][16].

Interestingly, Hugging Face also stated that frontier models accessed through commercial APIs were not usable for parts of their incident containment and response work because their safety guardrails blocked the submission of real exploit payloads and command-and-control artefacts. Hugging Face therefore ran the forensic analysis on GLM 5.2, an open-weight model, on its own infrastructure, keeping the data within its own environment [15][17].

What OpenAI stated

OpenAI stated that the incident was driven by a combination of its models, naming GPT-5.6 Sol and a more capable pre-release model, and that these were running with reduced cyber refusals for evaluation purposes [2][7][8].

Broadly, there can be two separate safety layers in play:

  1. Refusals are the model’s own trained behaviour, declining to comply.
  2. Classifiers or filters sit outside the model at deployment time, inspecting inputs and outputs and blocking them.

OpenAI stated that its models’ activity occurred during an internal evaluation that tasks models with advanced exploitation; the models were run without the production classifiers that would normally prevent high-risk cyber activity, in order to estimate maximal capability. It stated that the benchmark involved was ExploitGym [2][10][11]. ExploitGym is described by OpenAI as a benchmark created by UC Berkeley researchers in collaboration with OpenAI and other frontier labs [10][11].

The ExploitGym paper describes the benchmark as providing the agent with vulnerable source code and build instructions, an input that already triggers the bug, and a containerised environment in which to interact with the target, then tasking the agent with extending that input into a working exploit [11][18].

As an aside, GPT-5.6 Sol appears on the public ExploitGym leaderboard with a (self-reported) score of 0.337, meaning it was successful in creating working exploits for roughly a third of the instances attempted [20]. So what? Well, in May, just a few months ago, the strongest models measured managed roughly 13% to 17%, so 34% represents roughly a doubling of capability in just two months.

Anyhow, back on track: OpenAI’s account of the chain of events is as follows [2]:

  • The evaluation ran in an isolated environment whose network access was limited to a proxy and cache for package registries.
  • The models identified and exploited a previously unknown vulnerability in that package registry cache proxy, described by OpenAI as internally hosted third-party software, which OpenAI stated it has since disclosed to the vendor.
  • The models then performed privilege escalation and lateral movement within OpenAI’s research environment until reaching a node with internet access.
  • After gaining internet access, the models inferred that Hugging Face might host models, datasets, and solutions for ExploitGym.
  • The models then chained multiple attack vectors, including stolen credentials and zero-day vulnerabilities, to reach a remote code execution path on Hugging Face’s servers.

OpenAI characterised the models as hyperfocused on obtaining a solution to the benchmark. It stated that its own security team discovered the anomalous activity internally, and that Hugging Face’s team had already begun containment and forensic reconstruction using its own open-source models when the two teams connected [2][8].

OpenAI stated remediation and follow-up actions including tightening infrastructure controls at the cost of slowing its research, responsibly disclosing the identified zero-day in the third-party proxy software, and adding Hugging Face to its trusted-access cyber programme [2][7]. Fortune reported that this programme gives Hugging Face access to a version of GPT-5.6 Sol with fewer guardrails around cyber capabilities, intended for defenders [7].

Other interesting stuff

OpenAI stated that the models chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database [2]. That seems to answer the question of why Hugging Face was targeted: The LLM believed Hugging Face held benchmark solutions, and the models sought to obtain them.

The step that led there was inferential. OpenAI’s wording is that the models inferred Hugging Face potentially hosted models, datasets, and solutions for the benchmark [2], while the publicly documented distribution point for the ExploitGym harness and dataset is its GitHub repository [19]. The models reasoned their way to a location that was not the documented one, and found an alternative path instead.

On a related point, the disclosure reports have not explained why material sufficient to solve the benchmark would not already have been available to the models, given that the ExploitGym harness and dataset are publicly published [19]. Whether the internal evaluation used non-public instances, withheld information, or a private split is not addressed in the disclosure.

Hugging Face’s public disclosure does not specify whether the remote-code dataset loader it refers to is the client-side dataset loading-script mechanism or an internal server-side processing path. This distinction is relevant because the client-side mechanism had already been removed from the datasets library before the incident [12][13].

Neither company has published indicators of compromise at the time of writing [17].

All of which makes us wonder

GPT-5.6 Sol was previewed by OpenAI on 26 June 2026 as its most capable model for cyber security tasks including vulnerability research and exploitation, released with layered safeguards and a phased rollout [10]. Reporting on the incident noted that the model had gone through a government-gated release process and that a UK government evaluation had identified susceptibility to jailbreaks affecting cyber capabilities [8].

When Hugging Face came under attack from the new, highly capable model that was running with reduced cyber refusals, it was not able to respond symmetrically since (as we’ve seen in our own work recently) generally-available models are becoming more restrictive and less able to work on meaningful cyber security problems. In the case of Hugging Face, that asymmetry appears to have been addressed now, [7] but it does lead to an interesting thought:

We’re entering a phase where even highly skilled professionals will be outpaced by frontier AI LLMs so that even the best of the best will find it hard to defend against an autonomous agent framework executing many thousands of individual actions. Yet the plan so far seems to be:

  • Continue to develop ever-more capable models.
  • Try to constrain the use of those models to organisations and individuals who are considered suitable by some standard.

On the surface, that plan might seem to hold water… but it might have a few, widening cracks:

  1. This incident has shown that at least in this case, an LLM on a mission can find a way out of its confines, and can exhibit incredible ingenuity in finding a path towards achieving its goal, even if it does seem to be unconcerned with collateral (incident response, reputational, etc.) damage.
  2. The genie will seep out of the bottle and the use of the highly-capable models will broaden. Hugging Face now has access to Sol (which is fair enough since it could not use frontier models accessed through commercial APIs to defend itself, ironically due to security safeguards). But, as more access is granted, the restrict-to-a-select-few-companies (not to mention the select people within the select companies) model will fray at the edges.

So what now?

What do we mere mortals do in relation to this most interesting and unprecedented event?

Take control!

As we’ve written before [22], follow the UK AI Security Institute’s (AISI) recommendation and “do the basics well” [23]: Security updates, access controls, security configuration, and logging. SANS editors like Lee Neely pointed to the CIS Controls Implementation Group 1 as an effective defence, and William Murray’s entire contribution was: “Think least privilege and defence in depth.”

We are not helpless victims waiting for AI to come and destroy us!

Even without frontier LLM support, Hugging Face was able to use open-weight models to analyse the log events that they have been collecting, and to evict the attacking LLM. Every organisation, with a risk-based approach, and reference to control frameworks and standards, is well able to defend itself against the kinds of attacks that it’s most likely to face, and to manage their defences in response to their frequently-reviewed level of risk. Onwards!

References

  1. Hugging Face, “Security incident, July 2026.” https://huggingface.co/blog/security-incident-july-2026
  2. OpenAI, “OpenAI and Hugging Face partner to address security incident during model evaluation,” 21 July 2026. https://openai.com/index/hugging-face-model-evaluation-security-incident/
  3. Help Net Security, “Hugging Face breached by autonomous AI agent,” 20 July 2026. https://www.helpnetsecurity.com/2026/07/20/hugging-face-breached-by-autonomous-ai-agent/
  4. BleepingComputer, “Hugging Face breach: autonomous AI agent system, internal datasets, credentials.” https://www.bleepingcomputer.com/news/security/hugging-face-breach-autonomous-ai-agent-system-internal-datasets-credentials/
  5. The Hacker News, “World’s Largest AI Model Repository Hugging Face Breached by Autonomous AI Agent.” https://thehackernews.com/2026/07/worlds-largest-ai-model-repository.html
  6. Axios, “Hugging Face AI cyberattack data breach,” 20 July 2026. https://www.axios.com/2026/07/20/hugging-face-ai-cyberattack-data-breach
  7. Fortune, “OpenAI says AI models escaped control, hacked Hugging Face,” 21 July 2026. https://fortune.com/2026/07/21/openai-says-ai-models-escaped-control-hacked-hugging-face/
  8. Unite.AI, “OpenAI says its own test models breached Hugging Face.” https://www.unite.ai/openai-says-its-own-test-models-breached-hugging-face/
  9. Axios (via Yahoo), “Hugging Face breach: OpenAI claims its models were responsible.” https://tech.yahoo.com/cybersecurity/articles/hugging-face-breach-openai-claims-194520168.html
  10. OpenAI, “Previewing GPT-5.6 Sol: a next-generation model,” 26 June 2026. https://openai.com/index/previewing-gpt-5-6-sol/
  11. ExploitGym benchmark paper, arXiv:2605.11086, 11 May 2026. https://arxiv.org/abs/2605.11086
  12. Hugging Face Forums, “Dataset scripts are no longer supported.” https://discuss.huggingface.co/t/dataset-scripts-are-no-longer-supported/163891
  13. Hugging Face, hotpot_qa dataset discussion, “datasets==4.0.0 dropped support for remote code.” https://huggingface.co/datasets/hotpotqa/hotpot_qa/discussions/8
  14. Security Magazine, “Hugging Face Confirms Data Breach Caused by Autonomous AI Agent.” https://www.securitymagazine.com/articles/102442-hugging-face-confirms-data-breach-caused-by-autonomous-ai-agent
  15. Cyber Security News, “Hugging Face Confirms AI-Driven Breach.” https://cybersecuritynews.com/hugging-face-confirms-ai-driven-breach/
  16. Techzine, “Hugging Face was attacked by a fully AI-driven system.” https://www.techzine.eu/news/security/142991/hugging-face-was-attacked-by-a-fully-ai-driven-system/
  17. Gizmodo, “Hugging Face: We Used AI to Catch the First Confirmed AI Agent Breach of a Major AI Platform.” https://gizmodo.com/hugging-face-we-used-ai-to-catch-the-first-confirmed-ai-agent-breach-of-a-major-ai-platform-2000787778
  18. Berkeley RDI, “ExploitGym” project write-up. https://rdi.berkeley.edu/blog/exploitgym/
  19. ExploitGym source repository, sunblaze-ucb/exploitgym. https://github.com/sunblaze-ucb/exploitgym
  20. ExploitGym leaderboard, LLM-Stats. https://llm-stats.com/benchmarks/exploitgym
  21. Black Hat Briefings ’99 speaker list, including Eric Schultze and George Kurtz, “Over the Router, Through the Firewall, to Grandma’s House We Go.” https://blackhat.com/html/bh-usa-99/bh3-speakers.html
  22. “Anthropic Mythos: The model, the myth and the mundane”. https://www.dotsec.com/anthropic-mythos-the-model-the-myth-and-the-mundane/
  23. UK AI Security Institute, “Our evaluation of Claude Mythos Preview’s cyber capabilities.” https://www.aisi.gov.uk/blog/our-evaluation-of-claude-mythos-previews-cyber-capabilities

This post reflects the public record as of 22 July 2026. Both companies’ investigations were described as ongoing at that date.

Premier Australian cyber security specialists