Governance
Configure fix approvals, protected paths, and account settings in one place.
Security commentary
Alignment, Reward Hacking, and Autonomous Cyberattacks
Separate an agent’s objective from the evidence and permissions used to judge its work.
The score and the task can diverge
Reward hacking occurs when a system improves the signal used to judge success without accomplishing the intended task. In software work, a familiar analogue is changing a test so that it passes instead of correcting the implementation. For a security agent, an equally misleading result would be a report that describes successful verification without preserving the execution evidence. The underlying problem is a gap between what the operator values and what the workflow actually measures.
Published research on reward tampering provides reasons to examine this gap carefully. It does not justify assuming that every agent is deceptive or that one experiment predicts every deployment. The practical response is to make important outcomes independently checkable.
Keep the evaluator outside the work area
An agent asked to repair a repository may legitimately need to change application tests. That does not mean it should control the only acceptance test used to approve its own patch. Keep a separate verification environment with known inputs and checks that the worker cannot rewrite. Record the source revision used by that environment and preserve its output. If a check cannot run, return an incomplete result rather than treating a plausible explanation as a pass.
The same separation applies to vulnerability validation. Define the allowed target and expected observation before the run. A crash, an HTTP error and unauthorised data access are not interchangeable. The verifier should test the specific security property the finding claims to violate.
Enforce scope even when the goal is legitimate
A useful objective can still encourage an unsafe shortcut. An agent may find a relevant-looking external system or a document that offers apparently helpful instructions. The execution environment must still enforce account scope, network destinations and data handling. A tool response cannot grant permission to test a different organisation. A high confidence score cannot authorise access to production credentials.
Build explicit states for proposed, attempted and confirmed actions. This helps a reviewer understand what happened without reconstructing a long conversation. For a destructive or externally consequential operation, retain the approved parameters and match the execution to them. Do not infer an effect merely because a request was sent.
Measure the failures you care about
Include negative cases in an evaluation set: unavailable tools, misleading retrieved instructions, cross-tenant identifiers and incomplete evidence. Check whether the agent recognises missing information, respects the boundary and accurately reports uncertainty. Review examples where the system appears to perform unusually well; a sudden score improvement may reflect a useful capability, an easier test distribution or a flaw in the evaluator.
Treat alignment work and security engineering as complementary. Better objectives and training can reduce undesirable behaviour. Independent permissions, immutable evidence and controlled acceptance tests reduce the consequences when behaviour still departs from the task. The resulting assurance comes from the combination, with the limits of each test stated clearly.
During a patch evaluation, compare the submitted application changes with the files the independent verifier expected to remain stable. An unexpected change to the acceptance environment should trigger investigation, even when the final summary reports that every test passed.
