Marius CiepluchandClaude Opus 5.5
d4467f4236
Maintainer 2026-10-04: the safety net is for unseen code and the A/B tests exist to prove the approach generalises; infrastructure work had drifted from that. Runs are designed for AX on kube-ax, kind is the fallback. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1.7 KiB
Brief: does the safety net work on repositories it has never seen?
Maintainer, 2026-10-04: "we want to spread the safety net across repos in general, unseen code. that was the purpose of the AB tests to see if our approach is generalizeable." And: "we should run our tests / runs in kube. now we can speed it up and design the workloads for ax directly. kind is only a fallback."
Why this spec exists
The A/B experiments so far (exp-003, exp-004, exp-005) score the decision engine on repository-held-out folds of known fix pairs. That measures whether the vulnerable side of a pair scores above its fixed side. It does not measure what a safety net on an arbitrary repository lives on:
- on an ordinary push, does it stay quiet? (false-alarm rate)
- on a push that introduces a real vulnerability, does it say so? (catch rate)
Populations v1 and v2 measured the engine scan on fresh cases once (1 of 11, 0 of 17) and are spent. Nothing since has measured push-level behaviour on unseen repositories.
What this spec delivers
- A frozen pool of repositories never used anywhere in development, with two kinds of change each: real vulnerability-introducing commits and ordinary commits.
- A push-level metric: each change replayed through the hook exactly as a user would run it, and through other arms (decision engine, engine evidence) where an experiment asks.
- Execution designed for the kube-ax cluster from the start: one push replay is one AX Task; kind and the VM are fallbacks.
- A baseline for today's hook, then a table of arms scored on the same pool.
Not in scope
Changing detection. This spec only builds the measurement. Improvements are separate, registered experiments judged by it.