AI agents can do research on something, analyze code, or handle a long task without needing human intervention to tell it what to do and how to do it. But this same ability can become a security concern. A recent Gemini cybersecurity test showed us what happens when the AI agent has access to the internet, credentials, or real systems.
In May 2026, a Gemini Cybersecurity Test was conducted in which Gemini was tested by a cybersecurity company. The test was conducted by involving fake companies and systems. However, because of a configuration problem, the model got access to the internet. It then reached the systems of three real companies. Google later confirmed the incidents.
Gemini being able to hack something is not the most surprising part, but the fact that the AI agent was able to take actions outside the environment it was supposed to stay in.
This raises a bigger question: how do we keep AI systems secure when they can actually take actions on their own?
What Happened During the Gemini Cybersecurity Test?
Irregular is an independent AI security testing company that conducted the Gemini Cybersecurity Test. Gemini was taking part in a cybersecurity challenge designed to test how well it could find and solve security problems. The idea behind this test was fairly simple. That is, to give the AI a simulated environment and see whether it can identify vulnerabilities and complete specific security challenges.
The problem was that the test environment was not completely isolated. A configuration mistake gave the Gemini models access to the internet. Once that happened, the model encountered real systems that happened to match the information it was working with.
In one case, Gemini guessed credentials and gained access to a company’s online service. In two other cases, it found credentials that had been exposed in public software repositories.
There is an important detail here. According to Google, the model stopped after realizing that the systems it had reached were real and outside the intended scope of the exercise. The affected companies were notified, and changes were made to the testing process.
So, Gemini did not attack random companies but just went outside its intended boundaries.
Why Is This Gemini Cybersecurity Test Important?
AI has been used in cybersecurity for quite some time now. Existing security solutions use machine learning for anomaly detection, malware detection, log analytics, and pinpointing vulnerabilities.
The difference now is agency. So a traditional AI system might tell a security analyst that a server is suspicious; an AI agent may be able to go the extra step. It could probe the server, search the internet, invoke APIs, run tools, analyze code, and act on what it finds.
That additional functionality can be quite handy. It can be a lot larger attack surface too.
Give an employee, even a well-intentioned one, access to dozens of internal systems, then ask yourself, “I’d want to have access controls, auditing, login controls, logging, and the ability to establish limits on what they can do.” AI agents require many of the same safeguards.
Google DeepMind has characterized its strategy to advanced AI agents in similar language, such as treating subjectively misaligned agents as potential “insider threats” and deploying guards around them to monitor, prevent, and respond.
That’s why AI cybersecurity is starting to be a bigger area than simply stopping someone from hacking an AI model. You also need to consider what would happen if the AI has multiple systems at its disposal.
What Are AI Agents?
AI agent is an AI system that can do more than answer your questions. It can understand a task, make decisions, leverage tools, gather information and perform multiple steps to achieve the desired result. For example, as opposed to just identifying security flaws in some code, an AI agent can browse files, execute tests, analyze outputs and write a report.
How AI Can Help With Cybersecurity
It would be easy to focus only on the risks after an incident like this. But AI also has a growing role on the defensive side.
Threat Detection
Security teams deal with enormous amounts of data. Network activity, login attempts, application logs, alerts, and endpoint events can quickly become difficult for humans to review manually.
AI can help identify unusual patterns and prioritize events that deserve attention.
For example, an AI system could notice that an account is suddenly logging in from unusual locations while accessing systems it normally never touches.
That doesn’t automatically mean an attack happened. It gives the security team something worth investigating.
Vulnerability Detection
AI can also analyze source code and applications for possible security weaknesses. This can help developers find problems earlier, especially when security testing needs to cover a large amount of code.
AI-assisted vulnerability discovery is also becoming part of offensive security research. The same technology that helps defenders find a weakness can potentially help an attacker discover it.
That dual-use problem is not going away.
Security Monitoring
AI agents can potentially monitor security alerts around the clock. An agent could group related events, investigate suspicious activity, and provide context about what might be happening.
Google says it uses automated red teaming to continuously test Gemini against realistic attacks, including indirect prompt injection attacks.
Incident Response
An AI system could help collect evidence, summarize what happened, identify affected systems, and suggest possible next steps. Giving an AI unrestricted permission to make irreversible changes to production systems creates a very different risk profile.
The Risks of AI in Cybersecurity
AI can make security work faster, but speed isn’t automatically a benefit. If an AI agent makes a mistake while analyzing a document, you can usually correct the answer. If it makes a mistake while interacting with a production system, the consequences can be very different.
Some of the biggest concerns include:
- Unauthorized access
- Exposed credentials
- Excessive agent permissions
- Prompt injection
- Data leakage
- Incorrect automated actions
- Poorly isolated testing environments
- Unintended interaction with external systems
The Gemini incident is particularly useful as an example because the problem involved access and boundaries.
The model was supposed to operate within a particular test environment. Once it had internet access, the environment became much less predictable.
What Is Prompt Injection and Why Does It Matter?
Prompt injection is another important AI security problem. A simple example would be an AI agent that is asked to summarize emails. One email could contain hidden instructions telling the AI to ignore its original task and perform another action.
The AI has to distinguish between what the user asked it to do and what information it encountered while completing that task. It sounds simple but is actually not.
Google DeepMind has specifically researched indirect prompt injection, where malicious instructions can be hidden inside content that an AI agent retrieves while working. Google says it uses automated red teaming to test Gemini’s defenses against these attacks.
This is why you can add prompt injection to the list of security topics people should understand when working with AI agents.
The model itself isn’t necessarily the only thing that needs protection. The entire chain of tools, data, and permissions matters.
How Companies Can Secure AI Agents
There isn’t one security setting that solves this problem. AI agents need several layers of protection.
Limit Permissions
An agent shouldn’t automatically have access to everything. If it only needs to read a particular database, there may be little reason to give it permission to modify another system.
The basic idea is simple: give the agent only the access it actually needs.
Isolate Testing Environments
The Gemini incident also shows why testing environments need strong isolation.If a cybersecurity exercise is supposed to use fake companies, the AI should not accidentally be able to reach real companies.
Network restrictions, sandboxing, and access controls can reduce this risk.
Protect Credentials
Credentials should never be casually exposed in public repositories. In the Gemini Cybersecurity Test, two of the real-world accesses involved credentials found in public software repositories.
That is a useful reminder for developers too. An AI doesn’t have to break sophisticated encryption if a password or API key is sitting somewhere it can find.
Monitor Agent Actions
Companies should monitor what AI agents actually do, not only what they were instructed to do. An agent might receive a harmless instruction but then take an unexpected path while completing it.
Google DeepMind’s AI Control Roadmap includes monitoring agents and using supervisory systems to detect behavior that may be going off track.
Keep Humans Involved in High-Risk Actions
Not every AI action needs human approval. Reading a document is different from deleting a database. Generating a security report is different from changing firewall rules. For high-impact or irreversible actions, human approval can provide another layer of protection.
AI Red Teaming Is Becoming More Important
AI models need security testing too. But testing an AI agent isn’t exactly the same as testing a normal application. A traditional application generally follows programmed rules. An AI agent can interpret instructions, make plans, and respond differently depending on the information it encounters.
That makes testing more complicated. Security researchers can deliberately try to make an AI agent:
- Access information it shouldn’t
- Follow malicious instructions
- Reveal sensitive data
- Misuse connected tools
- Escape a defined task
- Take actions outside its permissions
Google calls one part of its approach automated red teaming, where AI is used to attack Gemini in realistic scenarios and uncover weaknesses.
And this is likely to become a much bigger part of AI development.
If AI agents are going to operate websites, software, cloud environments, and business systems, developers need to test not only whether the model gives good answers, but also whether it behaves safely when things go wrong.
What the Gemini Cybersecurity Test Tells Us About AI Security
This Gemini Cybersecurity Test incident is not all about how AI hacked three companies. We should learn something from it. The lesson is that AI agents can sometimes take unexpected actions if they get access to systems.
It also shows that we cannot rely only on an AI model to follow instructions correctly. AI systems need other security measures around them, such as:
- Secure testing environments
- Limited permissions
- Protected passwords and credentials
- Network restrictions
- Activity monitoring
- Human oversight
- Regular security testing
Conclusion
Gemini cybersecurity test shows that AI can make cybersecurity stronger and weaker at the same time. A security team can use AI to find vulnerabilities faster. Attackers can use similar technology to find weaknesses faster. AI can monitor thousands of security events. An AI agent can also make thousands of decisions much faster than a person. That doesn’t mean AI is automatically a security threat. It means security has to account for what increasingly capable AI systems can do when connected to real tools and data.
FAQs
Is Gemini safer than ChatGPT?
Both Gemini and ChatGPT have security features and are regularly tested for safety.
What is the test for cybersecurity?
It is a test used to find security weaknesses in a system before they can be exploited.
What happened during the Gemini cybersecurity test?
Gemini accessed three real company systems after gaining unintended internet access during a security test.
What is an AI agent?
An AI agent can use tools, access information, and take actions to complete a task.
Why is AI security important?
AI systems can access sensitive data and tools, so they need strong security controls and monitoring.
