Marius CiepluchandClaude Fable 5.1
adb70b6b98
Legacy inventory 2, item R2. tests/test_harnessx_removal_equality.py loaded
benchmarks/archive/harnessx_removal_equality.py by path and checked that its comparison sees a
difference. The spec is implementation-complete and the proof is the committed pair of records
(2026-09-30-harnessx-removal-{baseline,equality}); the archive README says these scripts are
reference only and a record is re-run at the commit it names, not with today's package. A unit test
that keeps the archived copy importable contradicted that and would break the next time cli or
learn.cli move. The archive README row now says the test was dropped. test_removed_plane_references.py
(the Req 2.4 name search) is a different test and stays.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
3.3 KiB
Archived benchmark scripts
Reference-only scripts, moved here with git mv on 2026-10-02 (legacy cleanup) so their history
follows them. Nothing in src/ imports them and no current workflow runs them. They stay because
committed records cite them and their numbers: to re-run a record, check out the commit the record
names rather than running these copies against today's package.
The old paths still appear inside committed records and in some docstrings. paths.md maps each old path to the new one. The records themselves are not rewritten.
| script | what it was | records that cite it | last recorded run |
|---|---|---|---|
verify_batched.py |
Stage-2 verifier, second design: triage per file, verify per file, report what two passes agree on. Replaced by the plane's verify + agree tasks. |
independent/results-v2-model-pipeline.json, independent/verifier-batched-check-2026-09-29.json, independent/plane-increment-2.json (parity baseline) |
072a102 (named in results-v2-model-pipeline.json) |
score_batched.py |
Exploratory scorer for the batched verifier: sites anchored to the candidate, matched by evaluate.matches_v2. |
independent/verifier-batched-check-2026-09-29.json, independent/plane-increment-2.json; tests/test_plane_scoring_parity.py still loads it as the reference |
added in 6acd354 (2026-09-29); the parity test replays it on hosts with the recorded scans |
verify_sinks.py |
The first stage-2 verifier, one candidate at a time. verify_batched.py imports ACCOUNT_ERRORS and _Recording from it. |
independent/verifier-check-2026-09-29.json |
the known-answer check after 041d871 |
model_sinks.py |
Stage-1 source-only model classification of sinks. Moved into the package as plane/tasks/roles.py. |
independent/selection-v2-model-sinks.json, measurements/2026-10-01-decision-engine-harvest/record.json, both 2026-10-01-decision-engine-leak-audit records |
05d5699 (named in selection-v2-model-sinks.json) |
sink_candidates.py |
Diagnostic: vocabulary-declared sink candidates per case. Its declaration patterns live on in plane/tasks/repo_facts.py. |
independent/results-v2-model-pipeline.json ("vocabulary sink-first: 2/17") |
no commit recorded |
hunt.py |
The v2 tool-hunter run over the population (protocol hunter-protocol-v2.md, including its FAMILY_WORDS table). Replaced by the plane's verify / agree tasks. |
independent/hunter-protocol-v2.md, independent/results-v2-hunter.json (a label source in learn/sources.toml) |
6bfb05f, the commit that introduced the protocol, which pins the runner |
harnessx_removal_equality.py |
One-off equality proof for harnessx-removal (record and compare deterministic outputs). The spec is closed; its unit test was dropped on 2026-10-02 so the archived copy is reference only like the rest. |
measurements/2026-09-30-harnessx-removal-baseline/, measurements/2026-09-30-harnessx-removal-equality/ |
baseline ca5caf7, candidate b9afb66 |
The six independent scripts still read the population, evaluate.py and the results from
benchmarks/independent/. Their HERE points there, so outputs land where they always did.
evaluate.py and freeze.py are current and were not moved.
verify_sinks.py --help fails with an argparse % formatting error. That bug was already there
before the move. The script still imports cleanly.