Governance
Configure fix approvals, protected paths, and account settings in one place.
Research
Security-agent capability is a system design problem
How to evaluate the effects of tools, context, verification, and budgets without confusing them with model improvements.
Measure the complete system
A security agent is a model connected to tools, repository context, an execution environment, and a stopping rule. Changing any of these can change its measured capability. A better result after a tooling change does not prove that the underlying model improved, just as a weaker prompt does not establish a model's fundamental limit.
CyberGym is a research benchmark for evaluating agents on real-world vulnerability tasks. Its existence is useful because it makes parts of the task and environment explicit. A deployment assessment still needs to ask which benchmark assumptions match the system under review and which important workflows lie outside that task definition.
Context should answer concrete questions
Give the agent enough information to identify the relevant execution path, dependency versions, and expected behavior. More repository text is not automatically better context. Repeated files, stale generated output, and irrelevant build logs can obscure the evidence needed to make a decision and consume the available budget.
Instrument context construction so a reviewer can see which files and revisions informed a finding. Compare targeted retrieval against a broader baseline using the same tasks. If retrieval fails, record that failure directly; a confident response to an incomplete context package should not count as a successful analysis.
Tools determine what can be verified
An agent that can only read files has different evidentiary limits from one that can build a target and run isolated tests. Tool output must expose exit status, timeouts, and relevant artifacts clearly. A truncated trace or a failed build is not evidence that a vulnerability is absent.
Keep tool authority narrower than the agent's reasoning scope. A model may discuss a suspected issue without receiving unrestricted network access or production credentials. Prefer reproducible test environments and explicit data boundaries so an evaluation measures analysis rather than accidental access to privileged resources.
Control the experimental comparison
To compare two agent designs, hold the model version, task set, execution budget, and scoring rules constant. Change one meaningful part of the harness at a time. Report unsuccessful builds and unsupported tasks separately from confirmed negative results, because otherwise a fragile setup can appear safer simply by failing to execute.
Use repeated runs where stochastic behavior matters, and show the distribution rather than a single best outcome. Include cost, elapsed time, and human review effort. A design that finds slightly more issues while overwhelming reviewers may be a poor operational choice even if its headline recall is higher.
Translate results into deployment decisions
Before adopting an improvement, replay representative internal workflows with known benign behavior and known defects. Verify that findings include enough evidence to reproduce the result and that proposed changes preserve application semantics. Include multi-service paths, authentication, and configuration differences that a repository-only benchmark may not capture.
A benchmark score is a measurement under stated conditions, not a blanket claim of autonomous security coverage. Publish the conditions and limits alongside the result. This article makes no percentage-improvement claim: its purpose is to explain how an engineering team can distinguish a real system improvement from a favorable test setup.
