XRanges for AI measures what security agents actually do

Teams building AI security agents, programs that look for software weaknesses by themselves and decide what to try next, face a measurement problem. They can place an agent against a realistic target application, but the result is usually a report the agent wrote about itself: confident text, a list of findings, and no independent way to know which of those findings really happened. Someone with security experience must then manually check every claim against the target, separating real findings from duplicates and inventions, and also trying to work out what the agent never attempted. CTF.ae built XRanges for AI to close that loop. The platform deploys realistic target applications with instrumentation already built into every service, records what an agent actually does inside them, and scores each run live on four independent signals.

The review problem gets worse as experiments grow. One manual check may be manageable for a single run, but an AI engineering team often tests three models, four prompt variations and ten repetitions. That creates a review queue longer than the experiment itself. A report also only describes what the agent found; it is silent about features the agent never opened, API endpoints it never enumerated, and second bugs sitting in the same endpoint where it found the first. There is also the agent that deletes a table or revokes every API key while reaching a finding, an outcome no client would accept and that a findings list does not record. Manual review does not cope with that matrix.

XRanges for AI is one workspace for the whole evaluation and has two halves. The first is a library of benchmark targets. Each target is a complete application, not a set of puzzle challenges: a multi-service company with its own business logic, seeded data, background jobs and simulated user traffic, built across several languages and frameworks because that is how real software is built. Each target carries 20 or more injected vulnerabilities, from single-step flaws to chains that cross service boundaries, including zero-days found by CTF.ae's own researchers. None of those targets exist in public training data, which the company says matters more every month. The second half is the instrumentation layer. Every service in every target sends structured telemetry through OpenTelemetry, a standard way for software to report its own activity. The instrumentation is written by hand by application security and software engineers for each specific target, because generic HTTP logging would miss most of what matters. The platform at ai.xranges.com consumes that telemetry for each deployment and turns it into four scores that update while the agent is still working.

The four scores are chosen to be independent, so an agent cannot improve one by gaming another. Coverage asks whether the agent explored the target. Every user-facing feature is a coverage point described as a business action rather than a URL, such as registered an account, browsed job postings, opened a shared conversation, or ran code in an assessment. A coverage point can only be reached through normal use, never through an exploit, so the score measures how thoroughly the agent worked the legitimate surface. Unhit points are listed by name, and most teams find that list more useful than the score. Boundaries asks whether the agent respected the rules of engagement. Each target ships with guard rules such as 'must not delete hiring content' or 'must not revoke API keys'. A violation is recorded the moment it happens, with the container and timestamp; zero violations is the expectation, and any violation is a finding about the agent, not the target. Exploited records which vulnerabilities the agent actually exploited. Every vulnerability is defined as a kill chain of ordered phases, from first contact with the vulnerable surface to an exploitation signal that only fires on success. Because each phase is detected from inside the target, the platform knows which step the agent completed and where it stalled, regardless of what the agent wrote. A three-step access control chain that stopped at step two appears as exactly that: two of three, with timestamps. Integrity checks whether the target survived. Checks run every minute and confirm the application is still functionally correct, including seed data still present, services answering with the right content, and cross-service trust intact. A failed check is a penalty no matter the cause, catching the agent that found a bug by breaking the environment around it. The four signals roll up into one score, but the breakdown is where the actual work appears.

A run starts with a target deployed as an isolated multi-container environment in about ninety seconds. The platform can run up to a thousand deployments at once, so AI engineers, software engineers and the infrastructure team can each run their own experiments without queueing. The agent then runs against the deployment endpoint on its own; the platform never sits between the agent and the target, instead it watches from the inside. While the agent works, a timeline records what it actually did mapped to business functionality, such as previewed a job posting, submitted an enterprise request, or minted an API key. Engineers who want raw material can read the OpenTelemetry stream directly and query it with a log query language that handles regular expressions and attribute filters. When the agent reports something that is not in the target's vulnerability catalogue, the timeline settles it: sometimes it is a false positive, and sometimes the agent found a real bug nobody planted, which has happened more than once.

Retesting does not require a second lab. Vulnerabilities can be toggled or patched in place on a running deployment. Some patches apply at runtime, while others need a restart of a minute or two. Either way the agent retests against the same environment with the same state. Every deployment also carries custom metadata such as model name, agent version, prompt variant and the engineer who ran it. Deployments are grouped, and a group shows the average and best score across its runs plus a per-vulnerability view of which run completed which chain. Repeated runs side by side are how variance gets separated from improvement; a single run proves very little, and the platform is built on that assumption. Everything in the console is also available through an API and a Model Context Protocol server with a bearer token. Deploying a batch of targets, launching the agent, pulling coverage and kill-chain progress and collecting the comparison at the end can run from a CI pipeline or from a chat assistant with nobody watching. The console is for people reading results; the API is for the experiment matrix.

Field proof came at DEF CON 34 in August 2026. Bug Bounty Village runs a capture-the-flag contest for the bug hunting community each year at DEF CON, and for that edition CTF.ae built the target: