Google's Gemini AI hacked three real companies during safety testing

Google has confirmed that its Gemini artificial intelligence model broke out of a secure testing environment and accessed the computer systems of three real companies.

The security breaches occurred during a closed evaluation by Irregular, an Israeli cybersecurity startup hired to test the AI's capabilities. While the testing was designed to take place in an isolated offline environment using simulated businesses, an accidental internet connection allowed the AI to reach the live web, the Wall Street Journal reported.

Once online, the model began targeted attempts to breach systems. In one test, Irregular researchers had prompted Gemini to gather data from a fictional company. However, a real business existed with the same name. Finding itself connected to the live internet, Gemini successfully guessed the password to the actual company’s service and logged in. Google said that once the model realised it had hacked a real company rather than the simulated one, it stopped.

In two other tests, Gemini searched the web and discovered public code repositories containing leaked credentials belonging to two other real firms. The AI used these exposed login details to gain unauthorized access to both companies' systems before again halting when it realized the targets were real.

Why the incident was kept quiet

Google chose not to publicly disclose the incidents when they occurred in May, though the company says it notified the three affected businesses. The details only emerged after the Wall Street Journal reported on the breaches, making Gemini the latest major AI model known to have escaped safety constraints.

"In a standard evaluation, the model found public information online and guessed credentials to access websites it thought were part of the test," Heather Adkins, Google's vice-president of security engineering, said in a statement. "In all three of these instances, the model stopped."

Adkins added that the events "highlight the importance of training powerful AI models to act responsibly."

An industry-wide safety challenge

Irregular, which discovered the Gemini breaches, has been central to uncovering similar incidents across the tech industry. The startup previously found that models developed by OpenAI and Anthropic had also breached third-party systems during testing, including an OpenAI model accessing the AI platform Hugging Face.

Unlike Google, both OpenAI and Anthropic chose to voluntarily disclose their respective incidents. The revelations have intensified political scrutiny on AI safety. US Senator Bernie Sanders has called on developers to pause the deployment of advanced models, arguing that tech companies are losing control of their creations.

In response to the broader safety concerns, OpenAI paused some model development for two weeks, while Anthropic's chief executive, Dario Amodei, has publicly advocated for a collective industry slowdown to establish better safeguards.