marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-18 20:49:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(plane): scoring parity with the reference -- sites, ranges, triage cost

Browse Source 
agree matched declared sites through benchmarks/independent/evaluate.py,
which the runner image does not carry, so site matching would have been
"not assessed". The generator now resolves the fix's old-side ranges
(evaluate.hunks, line for line) into case.json, with the declared sites in
the set and the reference's cost, and agree.matches_declared reproduces
matches_v2; a parity test runs both over the recorded outputs of the 15
validation cases (200 probes, identical verdicts) and agree on those outputs
returns the reference's 16 of 20 and $0.022.

agree reports cost_per_candidate (plane spend) and
cost_per_candidate_with_recorded_triage; Requirement 6.2 is judged on the
latter (design.md). --runner-image re-pins the templates from ops/ax/up.sh's
pin file; the committed manifests are regenerated at the current digest.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-29 19:49:28 +02:00
co-authored by Claude Opus 5.5
parent ac296ee3a4
commit 56b1717246
42 changed files with 591 additions and 174 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plane/tasks/verify.yaml
+1 -1
View File
@@ -10,7 +10,7 @@ metadata:
annotations:
openultrasast.io/model: deepseek-flash
spec:
image: "localhost:5001/ousast-runner@sha256:318cbe019c1155b43fe59e47368d99e78f6f6011b7c822292251de53cf567bf3"
image: "localhost:5001/ousast-runner@sha256:055ef8581911fbe9c3a3eb97b78b8a7206723460238aed52ce25b3d04756b3f4"
command: ["verify"]
resources:
requests: