marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-21 05:59:29 +00:00
Code Issues Packages Projects Releases Wiki Activity

steering: roadmap re-centred on generalisation to unseen repositories (G0-G5); specs updated; unseen-repo-evaluation brief and requirements (approved)

Browse Source 
Maintainer 2026-10-04: the safety net is for unseen code and the A/B tests exist
to prove the approach generalises; infrastructure work had drifted from that.
Runs are designed for AX on kube-ax, kind is the fallback.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-04 17:34:57 +02:00
co-authored by Claude Opus 5.5
parent ee422b63e0
commit d4467f4236
11 changed files with 371 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/unseen-repo-evaluation/brief.md
+32
View File
@@ -0,0 +1,32 @@
# Brief: does the safety net work on repositories it has never seen?
Maintainer, 2026-10-04: "we want to spread the safety net across repos in general, unseen code. that was the purpose
of the AB tests to see if our approach is generalizeable." And: "we should run our tests / runs in kube. now we can
speed it up and design the workloads for ax directly. kind is only a fallback."
## Why this spec exists
The A/B experiments so far (exp-003, exp-004, exp-005) score the decision engine on repository-held-out folds of known
fix pairs. That measures whether the vulnerable side of a pair scores above its fixed side. It does not measure what a
safety net on an arbitrary repository lives on:
- on an ordinary push, does it stay quiet? (false-alarm rate)
- on a push that introduces a real vulnerability, does it say so? (catch rate)
Populations v1 and v2 measured the engine scan on fresh cases once (1 of 11, 0 of 17) and are spent. Nothing since has
measured push-level behaviour on unseen repositories.
## What this spec delivers
1. A frozen pool of repositories never used anywhere in development, with two kinds of change each: real
vulnerability-introducing commits and ordinary commits.
2. A push-level metric: each change replayed through the hook exactly as a user would run it, and through other
arms (decision engine, engine evidence) where an experiment asks.
3. Execution designed for the kube-ax cluster from the start: one push replay is one AX Task; kind and the VM are
fallbacks.
4. A baseline for today's hook, then a table of arms scored on the same pool.
## Not in scope
Changing detection. This spec only builds the measurement. Improvements are separate, registered experiments judged
by it.