marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-21 05:59:29 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(plane): tie-break pass on disputed candidates; 2-of-3 agreement

Browse Source 
The design revisit after plane-increment-1.json (14/20 agreed, 19/20 in either
pass): a third verify pass on the candidates passes a and b dispute, and agree
decides 2-of-3 where pass c asked (the a/b rule elsewhere; c alone never decides).

- verify: optional OUSAST_INPUT_ONLY restricts the hunt to the named candidates
  (agree's disputed.json); an empty list is done with 0 units and no model client.
- agree: optional OUSAST_INPUT_PASS_C; metrics keep every key and add tiebreak
  {asked, flipped_to_agreed, flipped_to_rejected, rule}; usd_total, hunts and
  declared-site metrics are on the final decision.
- generate: per case <case>-vc and <case>-final appended to validation-46; the
  first 60 entries stay byte-identical, so a rerun keeps their done state.
- Offline replay over the recorded outputs with an empty pass c reproduces the
  recorded agree summaries exactly (14/20 agreed, 23 agreed, 10 disputed).
- Spec: Requirement 6.4, design Measurement, task 8.1.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-30 08:19:46 +02:00
co-authored by Claude Opus 5.5
parent c5e6ed3221
commit 6e6bcfc40c
11 changed files with 1197 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/ai-service-plane/design.md
+5
View File
@@ -206,6 +206,11 @@ plane applies the recorded triage without a model call, so `cost_per_candidate`
not judged; the triage share is derived per file from the recorded spend minus the priced hunt usage (about $0.087
of the $1.01) and labelled as derived in `case.json`. Sites are judged as `declared_sites_matched_agreed` against
`declared_sites_in_set`, the reference's 16 of 20.
The tie-break (Requirement 6.4) appends per case `<case>-vc`, `verify` pass c restricted by `OUSAST_INPUT_ONLY`
to the agree task's `disputed.json` (an empty list is `done` with no model call), and `<case>-final`, `agree` over
passes a, b and c, which decides 2-of-3 where pass c asked and keeps the a/b rule elsewhere, so pass c alone never
decides and `metrics.tiebreak` counts the flips; the first 60 entries stay byte-identical, so a rerun keeps their
`done` state, executes only the 30 new tasks, and pass c's cost adds to `usd_total` for the 6.2 judgement.
## Data Models