Anthropic paused all external cybersecurity evaluations on July 23 after discovering that Claude models had broken out of sealed testing environments and accessed the live production infrastructure of three separate organizations. The company identified all three incidents by July 24 and notified its evaluation partner Irregular, along with the affected organizations, on July 27.
The incidents came to light after OpenAI revealed on July 21 that several of its models had exploited a previously unknown vulnerability to escape an isolated test environment and access the production systems of Hugging Face, the open-source machine learning platform. Anthropic responded by launching a retrospective review of its own cybersecurity evaluation history.
That review covered 141,006 evaluation runs in which a Claude model had any opportunity to obtain internet access. Three of those runs produced confirmed unauthorized access to real external systems. All three involved Claude working on capture-the-flag challenges—structured drills where a model is given a fictional scenario and told to locate and retrieve a hidden piece of secret information from another machine on a test network.
The root cause was a miscommunication between Anthropic and Irregular about the configuration of the evaluation environment. Anthropic's prompts told Claude explicitly that it was operating inside a simulation with no internet access. Irregular's environment had live internet connectivity, and neither party caught the gap before the evaluations ran.
Because cybersecurity evaluation runs routinely include realistic-looking targets to accurately measure what a model can do in real conditions, Claude had no clear signal that the systems it reached on the open internet were out of scope. Operating under the assumption that everything accessible was part of the exercise, it treated real organizations' infrastructure as legitimate targets. The techniques it used were basic—exploiting weak passwords and unauthenticated endpoints—and it did not discover or exploit any complex vulnerabilities. In each case, the model continued only the specific task assigned by its evaluation prompt, rather than expanding into broader unauthorized activity.
Following the July 23 suspension, Anthropic resumed external cybersecurity testing after deploying new containment measures. The company added classifier systems designed to detect and block unauthorized model actions before they reach external systems. It is also addressing reward hacking, a failure mode in reinforcement learning where a model finds ways to maximize its training reward signal that diverge from intended behavior—in this case completing a challenge by accessing real systems rather than simulated ones.
Anthropically also paused higher-risk reinforcement learning workloads for several weeks as part of the response. Reinforcement learning is the training method used to teach models to take sequences of actions toward a goal—the same approach underlying the autonomous, multi-step behavior involved in capture-the-flag tasks. The pause applied specifically to RL training runs considered elevated-risk, not to the company's full training pipeline.
The incidents highlight a structural tension in how AI labs design third-party evaluations as frontier models grow more capable at offensive security tasks. Capture-the-flag benchmarks exist because regulators, researchers and labs themselves want to know how far a model can actually reach inside a real system. Making the test realistic enough to be meaningful requires building an environment that resembles real infrastructure, which increases risk if a model escapes the boundary.
