mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(plane): tie-break pass on disputed candidates; 2-of-3 agreement
The design revisit after plane-increment-1.json (14/20 agreed, 19/20 in either
pass): a third verify pass on the candidates passes a and b dispute, and agree
decides 2-of-3 where pass c asked (the a/b rule elsewhere; c alone never decides).
- verify: optional OUSAST_INPUT_ONLY restricts the hunt to the named candidates
(agree's disputed.json); an empty list is done with 0 units and no model client.
- agree: optional OUSAST_INPUT_PASS_C; metrics keep every key and add tiebreak
{asked, flipped_to_agreed, flipped_to_rejected, rule}; usd_total, hunts and
declared-site metrics are on the final decision.
- generate: per case <case>-vc and <case>-final appended to validation-46; the
first 60 entries stay byte-identical, so a rerun keeps their done state.
- Offline replay over the recorded outputs with an empty pass c reproduces the
recorded agree summaries exactly (14/20 agreed, 23 agreed, 10 disputed).
- Spec: Requirement 6.4, design Measurement, task 8.1.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
c5e6ed3221
commit
6e6bcfc40c
@@ -206,6 +206,11 @@ plane applies the recorded triage without a model call, so `cost_per_candidate`
|
||||
not judged; the triage share is derived per file from the recorded spend minus the priced hunt usage (about $0.087
|
||||
of the $1.01) and labelled as derived in `case.json`. Sites are judged as `declared_sites_matched_agreed` against
|
||||
`declared_sites_in_set`, the reference's 16 of 20.
|
||||
The tie-break (Requirement 6.4) appends per case `<case>-vc`, `verify` pass c restricted by `OUSAST_INPUT_ONLY`
|
||||
to the agree task's `disputed.json` (an empty list is `done` with no model call), and `<case>-final`, `agree` over
|
||||
passes a, b and c, which decides 2-of-3 where pass c asked and keeps the a/b rule elsewhere, so pass c alone never
|
||||
decides and `metrics.tiebreak` counts the flips; the first 60 entries stay byte-identical, so a rerun keeps their
|
||||
`done` state, executes only the 30 new tasks, and pass c's cost adds to `usd_total` for the 6.2 judgement.
|
||||
|
||||
## Data Models
|
||||
|
||||
|
||||
@@ -132,6 +132,9 @@ before the remaining concerns are converted.
|
||||
current $0.022 for two passes; otherwise the increment is reported as not met and the design revisited.
|
||||
3. The development transcript measure (`benchmarks/dev/token_report.py`) is run at the end of the increment and
|
||||
recorded next to the result.
|
||||
4. After the first measurement (plane-increment-1.json: 14/20 agreed, 19/20 in either pass), agreement is decided
|
||||
2-of-3 with a third pass run only on candidates passes a and b dispute; the gate of 6.2 is re-evaluated on that
|
||||
decision, and the result is recorded as plane-increment-2.json next to the first.
|
||||
|
||||
### Requirement 7: Routing is a declared concern of the plane
|
||||
|
||||
|
||||
@@ -130,6 +130,15 @@ mypy are clean, and the commit is gated on pytest's own exit code. Script-level
|
||||
if not met, revise the design before task 9.
|
||||
- _Requirements: 6.1, 6.2, 6.3_
|
||||
|
||||
- [ ] 8.1 Tie-break pass and re-measurement
|
||||
- `verify` takes `OUSAST_INPUT_ONLY` (pass c on the disputed candidates only); `agree` takes
|
||||
`OUSAST_INPUT_PASS_C` and decides 2-of-3 where pass c asked, the a/b rule elsewhere (`metrics.tiebreak`).
|
||||
- `validation-46` gains `<case>-vc` and `<case>-final` per case, appended so the first 60 entries keep their
|
||||
state; an offline replay with an empty pass c reproduces the recorded 14/20.
|
||||
- Rerun `validation-46` on ax (only the new tasks run), re-evaluate the 6.2 gate on the final decision and record
|
||||
`benchmarks/independent/plane-increment-2.json` next to the first.
|
||||
- _Requirements: 6.2, 6.4_
|
||||
|
||||
- [ ] 9. Handover
|
||||
- `ops/README.md`: how to run a Run locally and what an ax deployment would need (Kubernetes, Agent
|
||||
Substrate, the runner image, the provider extension); memory note updated; scripts that the tasks replaced
|
||||
|
||||
Reference in New Issue
Block a user