Governance
Configure fix approvals, protected paths, and account settings in one place.
Research
Designing a security-agent benchmark that measures useful work
A framework for evaluating vulnerability detection, validation, and change analysis with reproducible evidence.
Define the decision the benchmark supports
A security benchmark should begin with an operational question. A team selecting a pull-request reviewer needs different evidence from a team evaluating a repository-wide investigator. A single aggregate score can hide those differences, especially when tasks vary in language, authentication assumptions, repository size, or available execution tools.
Write down the task boundary, permitted context, expected artifact, and stopping rule before running a comparison. Separate vulnerability detection from validation of a supplied candidate and from analysis of a code change. Combining these tasks without identifying them makes it difficult to explain why one system appears stronger than another.
Treat ground truth as an evidence set
A benchmark label should be supported by a reproducible explanation, affected code, and the conditions that make the issue reachable. A historical advisory is useful, but it may omit deployment-specific prerequisites or describe a fix more broadly than the test target. Preserve uncertainty where the available evidence does not support a binary answer.
Include realistic negative cases, such as a fixed version or a similar path with the necessary authorization check. Avoid constructing negatives that are trivially different from positives. Review the labels independently and record disagreements; scoring precision against an unreliable reference can reward confident mistakes.
Control leakage and execution conditions
Keep evaluation tasks separate from development and training material where possible. Public vulnerabilities may already appear in model training data, retrieved web content, or example prompts. Record what information the system could access so a successful answer is not mistaken for independent reasoning about an unseen issue.
Freeze repository revisions, dependency versions, tool permissions, and time or compute budgets. Count build failures and unavailable environments explicitly. A system should not receive a clean-negative label because it failed to inspect or execute the target, nor should an evaluator silently repair the environment for only one participant.
Report several measures together
Recall describes which known issues were found, while precision describes how many reported findings were supported. Both need a clear unit: distinct root causes, reported locations, or task-level success are not interchangeable. Include the cost of duplicates and the evidence required for a reviewer to resolve them.
For change analysis, distinguish newly introduced, still-present, and resolved issues. Measure elapsed time, computation, and human verification effort alongside correctness. Use repeated trials when outputs vary, and show confidence or variability rather than selecting the most favorable run. Report results by relevant task categories before emphasizing an overall average.
Make the result reviewable and bounded
Publish the evaluation procedure, environment identity, scoring rules, and representative artifacts that can be shared safely. Preserve failure cases as well as successful examples. A benchmark should help another team reproduce the measurement or identify why its own environment would produce a different outcome.
CyberGym provides a primary research example of task-based cybersecurity evaluation, while NIST's AI Risk Management Framework emphasizes measurement within a broader risk process. This guide introduces no proprietary benchmark, model ranking, or performance claim. Its value is a structure for producing evidence that supports a specific deployment decision and remains honest about what the evaluation did not measure.
