marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-17 20:19:29 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(plane): scoring parity with the reference -- sites, ranges, triage cost

Browse Source 
agree matched declared sites through benchmarks/independent/evaluate.py,
which the runner image does not carry, so site matching would have been
"not assessed". The generator now resolves the fix's old-side ranges
(evaluate.hunks, line for line) into case.json, with the declared sites in
the set and the reference's cost, and agree.matches_declared reproduces
matches_v2; a parity test runs both over the recorded outputs of the 15
validation cases (200 probes, identical verdicts) and agree on those outputs
returns the reference's 16 of 20 and $0.022.

agree reports cost_per_candidate (plane spend) and
cost_per_candidate_with_recorded_triage; Requirement 6.2 is judged on the
latter (design.md). --runner-image re-pins the templates from ops/ax/up.sh's
pin file; the committed manifests are regenerated at the current digest.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-29 19:49:28 +02:00
co-authored by Claude Opus 5.5
parent ac296ee3a4
commit 56b1717246
42 changed files with 591 additions and 174 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/ai-service-plane/design.md
+11
View File
@@ -133,6 +133,17 @@ artifacts for the increment; the `classify` and `triage` tasks are later specs.
Reads the pass outputs, emits `agreed.json` and `disputed.json` by the same rule as `verify_batched.agreement`,
plus the per-candidate cost, turns and the declared-site match (`evaluate.matches_v2`) for the measurement.
The rule is reproduced in the package (`agree.matches_declared`, parity-tested against `matches_v2` on the recorded
reference outputs) and the generator writes the fix ranges it matches against into each case's `case.json`, so
the runner image carries no benchmark tree.
**Measurement.** Requirement 6.2's cost is judged on `cost_per_candidate_with_recorded_triage`: the plane's two
verify passes plus the recorded triage cost of the case, over the candidates before triage, summed across the run.
The reference's $0.022 (`verifier-batched-check-2026-09-29.json`) paid for per-file triage and the hunts while the
plane applies the recorded triage without a model call, so `cost_per_candidate` (plane spend alone) is reported but
not judged; the triage share is derived per file from the recorded spend minus the priced hunt usage (about $0.087
of the $1.01) and labelled as derived in `case.json`. Sites are judged as `declared_sites_matched_agreed` against
`declared_sites_in_set`, the reference's 16 of 20.
## Data Models