Menu

Star on GitHub · 3.1k
All posts
Research

How we do agentic web testing

A short, first-person look at how CyberStrike splits a web assessment across specialist agents, keeps the reasoning with the AI, and uses a deterministic signal layer so evidence is never lost.

Many agents, one mind: how CyberStrike tests

I want to explain, plainly, how agentic testing works in CyberStrike, because “AI does the pentest” is a slogan, not an architecture. The interesting part is not that a model reads a response. It is how we split the work, where we keep the judgment, and how we make sure a finding surfaces no matter which part of the run tripped over it.

1. One big tester does not scale

Our first version was a single AI tester with a very long prompt doing everything in order: recon, injection, access control, the lot. Two problems showed up fast. The prompt grew into hundreds of lines of payload lists the model had to carry every time, and evidence elicited in one step died when that step ended. If a marker was injected on one page and only rendered on another, nobody was looking.

2. Split the run into specialists

So we broke the tester into agents aligned to the OWASP WSTG categories: recon, misconfiguration, injection, access control, auth and session, server-side request, business logic. Each one has a narrow job and a thin prompt. The orchestrator routes work between them and handles chaining across endpoints. Narrow agents reason better than one agent trying to hold the whole methodology in its head.

3. Tools own mechanics, the AI owns judgment

This is the rule I care about most. Tools handle the mechanical parts: payload libraries, encoding, sending, comparing responses, detecting an error signature. The agent decides what to test, whether a result is real, and what to chain next. Roughly a quarter of each agent is mechanics, three quarters is reasoning. Keeping the judgment with the model, not baked into a tool, is what keeps false positives down.

4. Every request goes through one governed path

No tool opens its own socket or shells out to curl. Every request travels through a single replay transport that is byte-exact and governed, with timeouts, retries, a budget and a circuit breaker. It is a safety guarantee and a clean seam at the same time: one place to reason about what we send and how hard.

5. A deterministic layer that never loses evidence

The load-bearing piece is not an agent at all. It is a layer of plain code that scans every response, whether captured by the proxy or elicited by a tool, and runs passive detectors plus a canary registry. When we inject a marker, we record it. When any later response comes back, we scan it against every marker we have registered. A marker injected on page A that shows up on page B fires a stored-reflection signal and routes itself to the right specialist. That is how second-order and stored injection stop being invisible, and it works regardless of which agent or what order triggered it.

6. Coverage you can actually see

All 97 WSTG tests are mapped to an owner and a mechanism in a matrix. A gap is visible, not silent. The browser-dependent client-side checks are flagged as a later phase rather than quietly dropped.

The honest summary: agentic testing does not replace the tester. It automates the repetitive breadth and makes the evidence impossible to lose, so the human spends their time on scope, judgment and sign-off instead of copy-pasting payloads.

Keep reading

More from on call

View all posts
How we run Navy on Navy
newsJun 2, 2025

How we run Navy on Navy

We monitor our own checks, alerts and status pages with Navy. Here is what that setup looks like and what it has taught us.

Response time percentiles, explained
researchMay 30, 2025

Response time percentiles, explained

Averages hide the requests your users actually notice. Here is how to read p50, p95 and p99 response times.

Monitors as code with Terraform
researchMay 29, 2025

Monitors as code with Terraform

Define your monitors, alerts and status pages next to the infrastructure they watch, with the Navy Terraform provider.

Get the next post in your inbox

One email a month with new posts, product updates and the odd postmortem worth reading. Unsubscribe in one click.