marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-20 21:49:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(plane): tie-break pass on disputed candidates; 2-of-3 agreement

Browse Source 
The design revisit after plane-increment-1.json (14/20 agreed, 19/20 in either
pass): a third verify pass on the candidates passes a and b dispute, and agree
decides 2-of-3 where pass c asked (the a/b rule elsewhere; c alone never decides).

- verify: optional OUSAST_INPUT_ONLY restricts the hunt to the named candidates
  (agree's disputed.json); an empty list is done with 0 units and no model client.
- agree: optional OUSAST_INPUT_PASS_C; metrics keep every key and add tiebreak
  {asked, flipped_to_agreed, flipped_to_rejected, rule}; usd_total, hunts and
  declared-site metrics are on the final decision.
- generate: per case <case>-vc and <case>-final appended to validation-46; the
  first 60 entries stay byte-identical, so a rerun keeps their done state.
- Offline replay over the recorded outputs with an empty pass c reproduces the
  recorded agree summaries exactly (14/20 agreed, 23 agreed, 10 disputed).
- Spec: Requirement 6.4, design Measurement, task 8.1.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-30 08:19:46 +02:00
co-authored by Claude Opus 5.5
parent c5e6ed3221
commit 6e6bcfc40c
11 changed files with 1197 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/ai-service-plane/tasks.md
+9
View File
@@ -130,6 +130,15 @@ mypy are clean, and the commit is gated on pytest's own exit code. Script-level
if not met, revise the design before task 9.
- _Requirements: 6.1, 6.2, 6.3_
- [ ] 8.1 Tie-break pass and re-measurement
- `verify` takes `OUSAST_INPUT_ONLY` (pass c on the disputed candidates only); `agree` takes
`OUSAST_INPUT_PASS_C` and decides 2-of-3 where pass c asked, the a/b rule elsewhere (`metrics.tiebreak`).
- `validation-46` gains `<case>-vc` and `<case>-final` per case, appended so the first 60 entries keep their
state; an offline replay with an empty pass c reproduces the recorded 14/20.
- Rerun `validation-46` on ax (only the new tasks run), re-evaluate the 6.2 gate on the final decision and record
`benchmarks/independent/plane-increment-2.json` next to the first.
- _Requirements: 6.2, 6.4_
- [ ] 9. Handover
- `ops/README.md`: how to run a Run locally and what an ax deployment would need (Kubernetes, Agent
Substrate, the runner image, the provider extension); memory note updated; scripts that the tasks replaced