Governance
Configure fix approvals, protected paths, and account settings in one place.
Research
Training compact security agents with verifiable feedback
An original guide to reward design, isolated execution, and evaluation for smaller models used in defensive analysis.
A training reward is an engineering specification
Reinforcement learning changes behavior according to the feedback supplied by the training environment. For a security agent, that feedback might reflect a reproduced defect, a correct rejection, or a repair that passes both regression and functionality tests. A reward for persuasive text alone cannot establish that the underlying security claim is true.
The environment therefore needs an explicit contract. Define the task, allowed actions, available evidence, and terminal outcome. OpenAI's Spinning Up material explains the basic relationship between an agent, its actions, the environment, and reward. Applying that structure to security requires particular care because apparent success may be produced by manipulating the evaluation rather than solving the intended problem.
Build a trustworthy verifier
A verifier should measure an externally meaningful property and run outside the agent's editable workspace when practical. For patch tasks, checking that a failing test now passes is insufficient if the agent can delete the test or disable the affected functionality. Confirm the security invariant and preserve expected behavior through independent checks.
Keep environmental errors separate from negative outcomes. A missing dependency, broken compiler, or exhausted resource budget should not look like a secure program. Retain execution status and artifacts so training and evaluation can distinguish a failed investigation from a demonstrated absence of the suspected issue.
Keep the training environment contained
Security tasks may require compiling untrusted code, invoking package scripts, or exercising malformed input. Those operations should run with restricted credentials, filesystem scope, and network access. The training system should not depend on a model consistently choosing to respect boundaries that its tools do not enforce.
Reset the environment between tasks and track the identity of images, dependencies, and target revisions. Prevent one task's files or successful answers from leaking into another. Constrain the actions needed for the intended defensive skill rather than granting a general-purpose production environment in the hope that the reward will discourage misuse.
Use a curriculum without hiding hard cases
A compact model may benefit from tasks that increase in difficulty: locating a relevant check, tracing a bounded data flow, comparing a repair, then investigating a larger multi-file path. Each stage should preserve evidence quality. Easier tasks should teach the actual invariant instead of a superficial signal such as a suspicious function name.
Evaluate progress on a held-out set with different repositories and failure patterns. Track false positives, repeated tool errors, and refusal to revise a wrong hypothesis. A model that performs well only when the relevant file is preselected has a narrower capability than one that can investigate a whole task independently.
Judge the finished agent, not a training curve
A rising training reward is evidence about the training process, not a complete deployment result. Compare the final agent with practical baselines using the same tools, context, budgets, and verification rules. Measure how much human review remains and whether the smaller model's operational savings survive its additional retries or escalations.
Use deployment permissions appropriate to the evidence the agent can produce. Keep high-impact changes reviewable and reversible, and preserve a path for uncertain cases to receive deeper analysis. This guide describes a training design; it makes no claim that Counterswarm has trained a particular model, achieved a benchmark score, or inherited another research group's experimental results.
