Why Your Annual Penetration Test Isn’t Covering Your AI Systems
Why Your Annual Penetration Test Isn’t Covering Your AI Systems
What is AI penetration testing? AI penetration testing is a security assessment that specifically targets AI applications, large language models, and AI agents for vulnerabilities including prompt injection, RAG data poisoning, tool-call abuse, and guardrail bypass. It uses the same authorized attack simulation approach as traditional pen testing but applies techniques designed for AI-specific attack surfaces that standard web application testing cannot detect.
Most organizations running annual penetration tests right now are testing their networks, their web applications, and their APIs. Almost none are testing their AI systems.
That gap matters because attackers already know it exists.
In June 2025, researchers at Aim Security disclosed EchoLeak (CVE-2025-32711), the first documented zero-click prompt injection exploit in a production enterprise AI system. The target was Microsoft 365 Copilot. The attack required no user interaction: a single crafted email could coerce Copilot into accessing internal files and transmitting their contents to an attacker-controlled server. No phishing link. No malware. Just a prompt the model followed because nothing in its environment told it not to. [1]
In May 2026, Microsoft’s own security blog disclosed remote code execution vulnerabilities in its Semantic Kernel AI agent framework (CVE-2026-25592 and CVE-2026-26030), demonstrating that prompt injection can escalate beyond data access to full command execution when agents are given tool access. [2]
These are not edge cases. According to OWASP’s LLM Top 10, prompt injection appears in 73% of production AI deployments and is ranked the single most critical vulnerability class for LLM applications. [3] Only 34.7% of organizations have deployed dedicated defenses against it.
Your annual penetration test almost certainly did not test for any of this. The question is whether your organization knows that.
What Is AI Penetration Testing, and How Does It Differ from a Standard Pen Test?
AI penetration testing is a security assessment focused on AI applications, large language models, and AI-powered workflows. It uses the same fundamental approach as traditional pen testing: authorized testers attempt to identify and exploit vulnerabilities before attackers do. The difference is in the attack surface.
The Attack Vectors Are Different
A traditional pen test checks for known web application vulnerabilities, network misconfigurations, authentication weaknesses, and exploitable CVEs. AI systems introduce a completely separate category of vulnerability.
Prompt injection: an attacker manipulates an AI system’s input, or content the system trusts, to override its intended instructions. Direct injection targets the system prompt directly. Indirect injection embeds malicious instructions in content the model retrieves, such as a document, web page, or email, and relies on the model to act on them. EchoLeak was an indirect injection delivered via email.
RAG data poisoning: organizations using retrieval-augmented generation, where the model fetches documents to inform responses, can have their document stores manipulated. A poisoned document in a trusted repository can cause an AI assistant to consistently return incorrect or attacker-directed information.
Tool-call abuse: AI agents are often given access to tools including APIs, databases, and workflow systems. An attacker who can manipulate the model’s reasoning can cause it to invoke those tools with attacker-controlled parameters, creating a path to systems the AI was authorized to reach but the attacker was not.
Model manipulation and jailbreaking: carefully constructed prompts can cause a model to bypass its guardrails and perform actions its system prompt explicitly prohibits, including accessing restricted data, generating prohibited content, or disclosing system configuration.
Standard web application testing does not probe for any of these. An OWASP Top 10 web test cannot detect whether your AI assistant will comply with a prompt injection embedded in a customer support ticket.
Why AI Systems Are Rarely Included in Pen Test Scope
In our assessments, when we ask clients what they want included in their annual penetration test, AI systems are almost never on the list. The most common reasons:
The client doesn’t know the AI system is there. Business units deploy AI assistants through SaaS platforms, Microsoft Copilot licenses, or third-party integrations without informing security teams. By the time the pen test is scoped, the security team is working from a network diagram that doesn’t reflect what the organization is actually running.
The client assumes the web application test covers it. If the AI system has a web front-end, it appears in the web application scope. The tester checks for standard web vulnerabilities. Prompt injection testing requires a different methodology, different tools, and a different understanding of how the application processes input. A web application test that doesn’t specifically probe for AI-specific vulnerabilities will not find them.
AI systems are treated as vendor-managed and therefore out of scope. When an organization uses a vendor’s AI product, the security assumption is that the vendor has handled security. What gets missed is the integration layer: the custom system prompt, the data the model is given access to, the tools it is connected to, and the trust relationships it operates within. Those are the organization’s configurations and the organization’s risk.
According to the Cisco State of AI Security 2026 report, 83% of organizations plan to deploy agentic AI, but only 29% feel ready to do so securely. [4] The gap between deployment and security testing is where exposure accumulates.
What AI Penetration Testing Actually Tests
A well-scoped AI penetration test covers the attack vectors that standard testing cannot reach.
Prompt Injection Testing
Testers systematically attempt direct and indirect prompt injection across all user-facing inputs, document retrieval pipelines, external API calls, and any other content source the model processes. This includes testing whether the model can be made to exfiltrate data, invoke tools with attacker-controlled parameters, or bypass its system prompt instructions.
Context Window Manipulation
AI models process a context window that may include the system prompt, retrieved documents, conversation history, and tool outputs. Testers examine whether content from lower-trust sources can influence higher-trust instructions, a condition that enabled EchoLeak to succeed without any user interaction.
Tool and API Boundary Testing
For AI agents with access to tools, testers verify whether the scope of tool access matches what the organization believes, whether the AI can be directed to call APIs with unintended parameters, and whether the tool layer enforces access controls independent of the model.
Guardrail Bypass Assessment
Testers evaluate the model’s configured restrictions and test whether they hold under adversarial conditions, including known jailbreaking techniques and context-manipulation approaches.
Integration Credential Review
AI systems often carry service account credentials, OAuth tokens, and API keys to connect with enterprise systems. Testers review whether those credentials are scoped appropriately and whether a compromised model would have access the business intended it to have.
Is your AI stack in your pen test scope? Clone Systems’ penetration testing service covers AI-specific attack vectors including prompt injection and tool-call abuse, not just the underlying web application layer. Contact us at clonesystems.com/penetration-testing.
The Clone Systems AI Pen Test Scope Checklist: 7 Questions Before You Declare Your AI Systems In-Scope
Before scoping an AI penetration test, every organization should be able to answer these seven questions. Missing answers indicate gaps in the test scope.
1. What AI systems does your organization currently run? Include vendor-provided tools (Copilot, ChatGPT Enterprise, Salesforce Einstein), internally developed applications, and AI agents embedded in existing products.
2. What data sources does each AI system have access to? Every retrieval pipeline, document store, email integration, and database connection is a potential injection surface.
3. What tools or APIs can each AI agent invoke? The tool access of an AI agent defines its blast radius if compromised.
4. What credentials does each AI system carry? Service accounts, OAuth tokens, and API keys connected to AI systems should be inventoried and reviewed for over-provisioning.
5. Who configured the system prompt? The system prompt defines the model’s operating rules. If it was configured informally or not reviewed since deployment, its restrictions may be weaker than assumed.
6. Has the AI system ever been included in a security test? If the answer is no, the system has not been tested against any of the attack classes OWASP classifies as the most critical LLM vulnerabilities.
7. Does your vendor manage the integration layer or do you? Vendor security does not extend to custom integrations, data connections, or access configurations your organization controls.
A no or unclear answer to any of these is a gap that belongs in a pen test scope.
How Clone Systems Can Help
Clone Systems’ penetration testing service is designed around the environment your business actually runs, not a checklist built when AI systems were a future concern.
For organizations deploying AI assistants, RAG pipelines, or AI agents, we scope tests to include the attack vectors specific to those systems: prompt injection, tool-call abuse, RAG poisoning, and guardrail bypass. Our assessments produce findings written for both security teams and the technical teams responsible for AI system configuration, with remediation guidance that addresses the integration layer, not just the model.
We also run PCI DSS Requirement 11.4.1 penetration tests for organizations subject to card brand requirements, with scoping documentation that accounts for AI systems touching or adjacent to the cardholder data environment.
Start the conversation at clonesystems.com.
Frequently Asked Questions
What is AI penetration testing?
AI penetration testing is a security assessment that specifically targets AI applications, large language models, and AI agents for vulnerabilities including prompt injection, RAG data poisoning, tool-call abuse, and guardrail bypass. It uses the same authorized attack simulation approach as traditional pen testing but applies techniques designed for AI-specific attack surfaces that standard web application testing cannot detect.
How is AI penetration testing different from regular penetration testing?
Regular penetration testing focuses on network, web application, and infrastructure vulnerabilities. AI penetration testing focuses on vulnerabilities specific to how AI systems process input and make decisions, including prompt injection, model manipulation, and the abuse of tools AI agents are authorized to use. The attack techniques, testing tools, and remediation approaches are different because the underlying vulnerability classes are different.
What is prompt injection and why is it the top LLM security risk?
Prompt injection is an attack where an adversary manipulates the input an AI system receives, either directly or through content the system trusts, to override its intended instructions and cause it to take attacker-directed actions. OWASP ranks it the most critical LLM vulnerability because it appears in 73% of production AI deployments and can enable data exfiltration, unauthorized tool use, and guardrail bypass without any conventional exploit code.
Does my organization need AI penetration testing if we only use a vendor’s AI product?
Yes. Vendor security covers the model and its core infrastructure. It does not cover the custom system prompt your organization configured, the data sources you connected to it, the tools you authorized it to use, or the service account credentials it carries. Those configurations are your organization’s responsibility and your organization’s risk surface.
Do AI systems need to be included in PCI DSS penetration testing scope?
If an AI system has access to or communicates with systems in or adjacent to your cardholder data environment, it should be within scope for PCI DSS Requirement 11.4.1 penetration testing. The standard requires testing any system that could affect the security of the CDE. An AI agent with access to customer data, payment systems, or administrative APIs meets that threshold.
Conclusion
Attackers are already probing enterprise AI systems. EchoLeak was disclosed in June 2025. The Semantic Kernel RCE vulnerabilities were disclosed in May 2026. OWASP has classified prompt injection as the top LLM vulnerability since 2023. The security community has documented the attack surface clearly.
The gap is not knowledge. The gap is whether your annual penetration test scope has caught up to what your organization actually runs.
If your pen test was scoped before your organization deployed AI assistants, agents, or integrated LLM capabilities, your test is answering the wrong question. Contact Clone Systems at clonesystems.com to scope a penetration test that includes your AI systems.
References
[1] Aim Security, EchoLeak (CVE-2025-32711): First Zero-Click Prompt Injection Exploit in Production LLM. SecurityPost, 2025.
https://www.securitypost.org/en/articles/echoleak-zero-click-prompt-injection-microsoft-365-copilot
[2] Microsoft Security Blog, When Prompts Become Shells: RCE Vulnerabilities in AI Agent Frameworks. Microsoft, May 2026. https://www.microsoft.com/en-us/security/blog/2026/05/07/prompts-become-shells-rce-vulnerabilities-ai-agent-frameworks/
[3] OWASP Gen AI Security Project, LLM01:2025 Prompt Injection. OWASP, 2025. https://genai.owasp.org/llmrisk/llm01-prompt-injection/
[4] Cisco, State of AI Security 2026. Cisco, 2026.
[5] Clone Systems, Shadow AI Isn’t Just a Data Leakage Problem. https://www.clonesystems.com/blog/shadow-ai-security-risks-access-control
[6] Clone Systems, How AI Is Shrinking the Window to Fix Vulnerabilities. https://www.clonesystems.com/blog/how-ai-is-shrinking-the-window-to-fix-vulnerabilities
[7] Clone Systems, AI Security Services. https://www.clonesystems.com/ai-security
[8] Clone Systems, Penetration Testing Services. https://www.clonesystems.com/penetration-testing
