feat(plane): tie-break pass on disputed candidates; 2-of-3 agreement

The design revisit after plane-increment-1.json (14/20 agreed, 19/20 in either
pass): a third verify pass on the candidates passes a and b dispute, and agree
decides 2-of-3 where pass c asked (the a/b rule elsewhere; c alone never decides).

- verify: optional OUSAST_INPUT_ONLY restricts the hunt to the named candidates
  (agree's disputed.json); an empty list is done with 0 units and no model client.
- agree: optional OUSAST_INPUT_PASS_C; metrics keep every key and add tiebreak
  {asked, flipped_to_agreed, flipped_to_rejected, rule}; usd_total, hunts and
  declared-site metrics are on the final decision.
- generate: per case <case>-vc and <case>-final appended to validation-46; the
  first 60 entries stay byte-identical, so a rerun keeps their done state.
- Offline replay over the recorded outputs with an empty pass c reproduces the
  recorded agree summaries exactly (14/20 agreed, 23 agreed, 10 disputed).
- Spec: Requirement 6.4, design Measurement, task 8.1.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-30 08:19:46 +02:00
co-authored by Claude Opus 5.5
parent c5e6ed3221
commit 6e6bcfc40c
11 changed files with 1197 additions and 19 deletions
+5
View File
@@ -206,6 +206,11 @@ plane applies the recorded triage without a model call, so `cost_per_candidate`
not judged; the triage share is derived per file from the recorded spend minus the priced hunt usage (about $0.087
of the $1.01) and labelled as derived in `case.json`. Sites are judged as `declared_sites_matched_agreed` against
`declared_sites_in_set`, the reference's 16 of 20.
The tie-break (Requirement 6.4) appends per case `<case>-vc`, `verify` pass c restricted by `OUSAST_INPUT_ONLY`
to the agree task's `disputed.json` (an empty list is `done` with no model call), and `<case>-final`, `agree` over
passes a, b and c, which decides 2-of-3 where pass c asked and keeps the a/b rule elsewhere, so pass c alone never
decides and `metrics.tiebreak` counts the flips; the first 60 entries stay byte-identical, so a rerun keeps their
`done` state, executes only the 30 new tasks, and pass c's cost adds to `usd_total` for the 6.2 judgement.
## Data Models
@@ -132,6 +132,9 @@ before the remaining concerns are converted.
current $0.022 for two passes; otherwise the increment is reported as not met and the design revisited.
3. The development transcript measure (`benchmarks/dev/token_report.py`) is run at the end of the increment and
recorded next to the result.
4. After the first measurement (plane-increment-1.json: 14/20 agreed, 19/20 in either pass), agreement is decided
2-of-3 with a third pass run only on candidates passes a and b dispute; the gate of 6.2 is re-evaluated on that
decision, and the result is recorded as plane-increment-2.json next to the first.
### Requirement 7: Routing is a declared concern of the plane
+9
View File
@@ -130,6 +130,15 @@ mypy are clean, and the commit is gated on pytest's own exit code. Script-level
if not met, revise the design before task 9.
- _Requirements: 6.1, 6.2, 6.3_
- [ ] 8.1 Tie-break pass and re-measurement
- `verify` takes `OUSAST_INPUT_ONLY` (pass c on the disputed candidates only); `agree` takes
`OUSAST_INPUT_PASS_C` and decides 2-of-3 where pass c asked, the a/b rule elsewhere (`metrics.tiebreak`).
- `validation-46` gains `<case>-vc` and `<case>-final` per case, appended so the first 60 entries keep their
state; an offline replay with an empty pass c reproduces the recorded 14/20.
- Rerun `validation-46` on ax (only the new tasks run), re-evaluate the 6.2 gate on the final decision and record
`benchmarks/independent/plane-increment-2.json` next to the first.
- _Requirements: 6.2, 6.4_
- [ ] 9. Handover
- `ops/README.md`: how to run a Run locally and what an ax deployment would need (Kubernetes, Agent
Substrate, the runner image, the provider extension); memory note updated; scripts that the tasks replaced