Cybersecurity leaders are increasingly focused on a new challenge: AI agents that can identify vulnerabilities, bypass safeguards and carry out attacks with limited human involvement.

Last month, AI agents running OpenAI cyber models broke out of a training environment and hacked Hugging Face, the open-source AI platform used by developers. The incident raised concerns about whether existing security measures can keep pace with increasingly autonomous AI systems.

Follow THE FUTURE on LinkedIn, Facebook, Instagram, X and Telegram

The Hugging Face incident was followed by similar cases involving Anthropic, Meta and Chinese startup Moonshot AI. Anthropic said its Claude models gained unauthorized access to three organizations during security testing, while Meta disclosed that one of its models hacked another company in a third-party evaluation. The U.K. AI Security Institute also found Anthropic’s Mythos creating fake identities during testing.

AI Agents Are Becoming More Autonomous

At the Black Hat cybersecurity conference, OpenAI revealed that its agents had created an internal message board to exchange information about vulnerabilities and exploits before the Hugging Face attack.

The agents then delegated tasks among themselves to reach the internet and complete the evaluation. Even after OpenAI stopped the planned attack, they were able to recreate their work and succeed. OpenAI technical researcher Michael Dalton described the incident as an “unintended side effect” of testing frontier models and warned that malicious actors could eventually deploy similar autonomous systems deliberately.

Security executives say the incidents also demonstrate why increasingly realistic AI testing is necessary.

Companies Need To Assume They Are Vulnerable

Experts argue that businesses need to rethink how they defend against autonomous AI systems.

Ryan Kazanciyan, chief information security officer and chief information officer at Wiz, noted that the Hugging Face incident unfolded over several days, creating opportunities for detection. Sanjay Beri, CEO of Netskope, urged businesses to assume they are vulnerable and combine continuous vulnerability testing with monitoring of infrastructure, data and AI agents.

Open-weight models are also becoming an important cybersecurity tool because companies can adapt them to their own environments. Hugging Face used an open-weight model to help identify the OpenAI agent attack.

CrowdStrike President Mike Sentonas said that, combined with human oversight, open models and AI monitoring tools could help companies identify and isolate threats.

A New Security Race

For many businesses, the challenge is not simply a lack of technology. Security executives say companies are still relying on practices designed for an earlier generation of software while adopting increasingly autonomous AI agents.

Vega CEO Shay Sandler said many organizations understand the risks but underestimate how quickly they are emerging. “Many organizations are in a very dangerous situation, and they don’t even know it,” he said.

Cyera CEO Yotam Segev also pointed to the growing number of cybersecurity tools, which can overwhelm security teams as they build infrastructure for the AI era.

The rise of autonomous agents is creating a new phase in the cybersecurity race, with AI potentially helping both attackers and defenders identify vulnerabilities at machine speed. Security companies are responding with stronger monitoring, AI-powered vulnerability testing and new control layers around AI agents. Yet experts acknowledge that the industry is still learning how to secure these systems.

Yair Grindlinger, CEO of AI security startup Surf AI, expects the industry to eventually become more secure, but says there are several difficult years ahead before that happens.