Beyond CyberGym: Evaluating Security Agents

Understand the task before the score

CyberGym is a research benchmark built around real software vulnerabilities. It offers a useful way to test parts of an agent's security capability in a repeatable environment. As with any benchmark, the first question should be what information the agent receives and which outcome the evaluator checks. Reproducing a known vulnerability from a description is a different task from discovering an unknown issue in a production service with an incomplete asset inventory.

Read the benchmark paper and task configuration before comparing results. Record the available source code, supplied hints, tool permissions, time limits and verification criteria. A result is meaningful within those conditions; extending it to another setting requires additional evidence.

Separate capability from the surrounding harness

An agent's result depends on more than its model. Repository retrieval, build instructions, test feedback and tool reliability affect whether it can investigate an issue. If a dependency fails to install, a run may fail before meaningful reasoning begins. If the harness exposes information that would be unavailable in deployment, a successful run may answer an easier question than the one a defender needs answered.

Change one component at a time when investigating improvements. Keep the task set and resource budget fixed, then compare the retained traces. This makes it possible to distinguish a more useful retrieval strategy from simply granting additional attempts or supplying a stronger hint.

Inspect failures and apparent successes

Classify failed runs by cause: an environment problem, missing prerequisite, incorrect reasoning, invalid test artefact or resource exhaustion. Do not quietly remove difficult cases from the denominator. For successful runs, verify that the submitted artefact satisfies the intended condition in a clean environment. A result that passes because the evaluator was modified or the wrong binary was tested should not count as a capability gain.

Track both first-attempt results and results after multiple attempts. Additional attempts can be useful, but they consume time and compute. Report the number of attempts, total cost and elapsed time alongside the outcome, so another team can assess whether the approach fits its own operating constraints.

Add a separate deployment evaluation

A production readiness set should include the conditions the benchmark leaves out. For an application-security workflow, that may mean tenant isolation, sensitive-data handling, safe cancellation, permissions and accurate reporting when a tool is unavailable. Include benign cases and cases with insufficient evidence. The agent should be able to return uncertainty without inventing a successful test.

Benchmark progress is most useful as a diagnostic instrument. It can identify a repeatable weakness, motivate a targeted change and show whether that change helps on held-out tasks. It should sit alongside realistic workflow tests and human review, rather than serving as a blanket claim that the agent can safely secure any environment.

Keep a small replay bundle for each evaluated case: task description, permitted inputs, environment identifier, generated artefact and verifier output. This lets another engineer investigate a disputed result without relying on the original agent conversation or reconstructing a transient workspace.