diff --git a/benchmarks/archive/README.md b/benchmarks/archive/README.md new file mode 100644 index 0000000..357341f --- /dev/null +++ b/benchmarks/archive/README.md @@ -0,0 +1,26 @@ +# Archived benchmark scripts + +Reference-only scripts, moved here with `git mv` on 2026-10-02 (legacy cleanup) so their history +follows them. Nothing in `src/` imports them and no current workflow runs them. They stay because +committed records cite them and their numbers: to re-run a record, check out the commit the record +names rather than running these copies against today's package. + +The old paths still appear inside committed records and in some docstrings. [paths.md](paths.md) +maps each old path to the new one. The records themselves are not rewritten. + +| script | what it was | records that cite it | last recorded run | +|---|---|---|---| +| `verify_batched.py` | Stage-2 verifier, second design: triage per file, verify per file, report what two passes agree on. Replaced by the plane's `verify` + `agree` tasks. | `independent/results-v2-model-pipeline.json`, `independent/verifier-batched-check-2026-09-29.json`, `independent/plane-increment-2.json` (parity baseline) | `072a102` (named in `results-v2-model-pipeline.json`) | +| `score_batched.py` | Exploratory scorer for the batched verifier: sites anchored to the candidate, matched by `evaluate.matches_v2`. | `independent/verifier-batched-check-2026-09-29.json`, `independent/plane-increment-2.json`; `tests/test_plane_scoring_parity.py` still loads it as the reference | added in `6acd354` (2026-09-29); the parity test replays it on hosts with the recorded scans | +| `verify_sinks.py` | The first stage-2 verifier, one candidate at a time. `verify_batched.py` imports `ACCOUNT_ERRORS` and `_Recording` from it. | `independent/verifier-check-2026-09-29.json` | the known-answer check after `041d871` | +| `model_sinks.py` | Stage-1 source-only model classification of sinks. Moved into the package as `plane/tasks/roles.py`. | `independent/selection-v2-model-sinks.json`, `measurements/2026-10-01-decision-engine-harvest/record.json`, both `2026-10-01-decision-engine-leak-audit` records | `05d5699` (named in `selection-v2-model-sinks.json`) | +| `sink_candidates.py` | Diagnostic: vocabulary-declared sink candidates per case. Its declaration patterns live on in `plane/tasks/repo_facts.py`. | `independent/results-v2-model-pipeline.json` ("vocabulary sink-first: 2/17") | no commit recorded | +| `hunt.py` | The v2 tool-hunter run over the population (protocol `hunter-protocol-v2.md`, including its `FAMILY_WORDS` table). Replaced by the plane's `verify` / `agree` tasks. | `independent/hunter-protocol-v2.md`, `independent/results-v2-hunter.json` (a label source in `learn/sources.toml`) | `6bfb05f`, the commit that introduced the protocol, which pins the runner | +| `harnessx_removal_equality.py` | One-off equality proof for `harnessx-removal` (record and compare deterministic outputs). The spec is closed. `tests/test_harnessx_removal_equality.py` still checks that the comparison detects differences. | `measurements/2026-09-30-harnessx-removal-baseline/`, `measurements/2026-09-30-harnessx-removal-equality/` | baseline `ca5caf7`, candidate `b9afb66` | + +The six `independent` scripts still read the population, `evaluate.py` and the results from +`benchmarks/independent/`. Their `HERE` points there, so outputs land where they always did. +`evaluate.py` and `freeze.py` are current and were not moved. + +`verify_sinks.py --help` fails with an argparse `%` formatting error. That bug was already there +before the move. The script still imports cleanly. diff --git a/benchmarks/harnessx_removal_equality.py b/benchmarks/archive/harnessx_removal_equality.py similarity index 98% rename from benchmarks/harnessx_removal_equality.py rename to benchmarks/archive/harnessx_removal_equality.py index 835c082..a37be93 100644 --- a/benchmarks/harnessx_removal_equality.py +++ b/benchmarks/archive/harnessx_removal_equality.py @@ -5,8 +5,8 @@ non-zero on the first difference outside the exclusion list, naming the file and key. Usage: - python benchmarks/harnessx_removal_equality.py record [--tree-ish HEAD] [--workdir DIR] - python benchmarks/harnessx_removal_equality.py compare \\ + python benchmarks/archive/harnessx_removal_equality.py record [--tree-ish HEAD] [--workdir DIR] + python benchmarks/archive/harnessx_removal_equality.py compare \\ [--suite-map map.json] [--record comparison.json] The instrument checks come first and fail loudly: ``harnessx`` must not be importable, ``tree_sitter`` must be, @@ -30,7 +30,7 @@ import xml.etree.ElementTree as ET from pathlib import Path from typing import Any -REPO = Path(__file__).resolve().parents[1] +REPO = Path(__file__).resolve().parents[2] # archived under benchmarks/archive/ # Design section 3 "Comparison": timestamps, durations and run identifiers are excluded; nothing else is. EXCLUDED_KEYS = frozenset({"runtime_seconds", "scan_id", "timestamp", "started", "finished", "created_at", "generated_at"}) diff --git a/benchmarks/independent/hunt.py b/benchmarks/archive/hunt.py similarity index 94% rename from benchmarks/independent/hunt.py rename to benchmarks/archive/hunt.py index 08837da..b65aeae 100644 --- a/benchmarks/independent/hunt.py +++ b/benchmarks/archive/hunt.py @@ -10,9 +10,9 @@ Model calls go to the resolved hunter client (DeepSeek when its key is set); the the recorded spend reaches `--budget-usd`. Usage: - python benchmarks/independent/hunt.py --population population-v2.toml [--only ID,...] [--pins a,b] - python benchmarks/independent/hunt.py --population population-v2.toml --score - python benchmarks/independent/hunt.py --pilot-root --family injection # a development case + python benchmarks/archive/hunt.py --population population-v2.toml [--only ID,...] [--pins a,b] + python benchmarks/archive/hunt.py --population population-v2.toml --score + python benchmarks/archive/hunt.py --pilot-root --family injection # a development case """ from __future__ import annotations @@ -25,7 +25,8 @@ import sys import time from pathlib import Path -HERE = Path(__file__).resolve().parent +# Archived 2026-10-02 from benchmarks/independent/: HERE stays that directory (population, evaluate.py, results). +HERE = Path(__file__).resolve().parents[1] / "independent" sys.path.insert(0, str(HERE)) import evaluate # noqa: E402 diff --git a/benchmarks/independent/model_sinks.py b/benchmarks/archive/model_sinks.py similarity index 96% rename from benchmarks/independent/model_sinks.py rename to benchmarks/archive/model_sinks.py index c6d790a..6860e35 100644 --- a/benchmarks/independent/model_sinks.py +++ b/benchmarks/archive/model_sinks.py @@ -12,7 +12,7 @@ counted as UNCLASSIFIED, never as "no sinks". Hard spend ceiling `--budget-usd`; v2 is spent for tuning; this informs the selection design and qualifies nothing. -Usage: python benchmarks/independent/model_sinks.py [--population population-v2.toml] [--only ID,...] +Usage: python benchmarks/archive/model_sinks.py [--population population-v2.toml] [--only ID,...] """ from __future__ import annotations @@ -27,7 +27,8 @@ import threading import time from pathlib import Path -HERE = Path(__file__).resolve().parent +# Archived 2026-10-02 from benchmarks/independent/: HERE stays that directory (population, evaluate.py, results). +HERE = Path(__file__).resolve().parents[1] / "independent" sys.path.insert(0, str(HERE)) import evaluate # noqa: E402 diff --git a/benchmarks/archive/paths.md b/benchmarks/archive/paths.md new file mode 100644 index 0000000..fae68ae --- /dev/null +++ b/benchmarks/archive/paths.md @@ -0,0 +1,24 @@ +# Old path to new path (2026-10-02) + +Committed records, specs and some docstrings name these scripts by their old path. Those files are +not rewritten. Read the old path as the new one: + +| old path | new path | +|---|---| +| `benchmarks/independent/verify_batched.py` | `benchmarks/archive/verify_batched.py` | +| `benchmarks/independent/score_batched.py` | `benchmarks/archive/score_batched.py` | +| `benchmarks/independent/verify_sinks.py` | `benchmarks/archive/verify_sinks.py` | +| `benchmarks/independent/model_sinks.py` | `benchmarks/archive/model_sinks.py` | +| `benchmarks/independent/sink_candidates.py` | `benchmarks/archive/sink_candidates.py` | +| `benchmarks/independent/hunt.py` | `benchmarks/archive/hunt.py` | +| `benchmarks/harnessx_removal_equality.py` | `benchmarks/archive/harnessx_removal_equality.py` | + +Files that still use the old path: + +- Records: `independent/hunter-protocol-v2.md`, `independent/results-v2-model-pipeline.json`, + `independent/selection-v2-model-sinks.json`, `independent/verifier-batched-check-2026-09-29.json`, + `measurements/2026-09-30-harnessx-removal-equality/README` +- Specs: `.kiro/specs/ai-service-plane/`, `.kiro/specs/harnessx-removal/design.md`, + `.kiro/specs/learned-decision-engine/` +- Docstrings: `src/openultrasast/plane/generate.py`, `plane/tasks/{verify,agree,roles,repo_facts}.py`. + These are left for the owners of those packages. diff --git a/benchmarks/independent/score_batched.py b/benchmarks/archive/score_batched.py similarity index 94% rename from benchmarks/independent/score_batched.py rename to benchmarks/archive/score_batched.py index bf9a9cb..b26b3b0 100644 --- a/benchmarks/independent/score_batched.py +++ b/benchmarks/archive/score_batched.py @@ -10,7 +10,7 @@ detected when an AGREED finding matches; unstable when only a DISPUTED one does; finding at the fixed pin matches the fix's new side. The precision sample is every third agreed vulnerable-pin finding (all if 15 or fewer), to be adjudicated by hand. v2 is spent for tuning: this qualifies nothing. -Usage: python benchmarks/independent/score_batched.py [--results-suffix batched] [--population population-v2.toml] +Usage: python benchmarks/archive/score_batched.py [--results-suffix batched] [--population population-v2.toml] """ from __future__ import annotations @@ -20,7 +20,8 @@ import json import sys from pathlib import Path -HERE = Path(__file__).resolve().parent +# Archived 2026-10-02 from benchmarks/independent/: HERE stays that directory (population, evaluate.py, results). +HERE = Path(__file__).resolve().parents[1] / "independent" sys.path.insert(0, str(HERE)) import evaluate # noqa: E402 diff --git a/benchmarks/independent/sink_candidates.py b/benchmarks/archive/sink_candidates.py similarity index 95% rename from benchmarks/independent/sink_candidates.py rename to benchmarks/archive/sink_candidates.py index eb872f5..051a1a7 100644 --- a/benchmarks/independent/sink_candidates.py +++ b/benchmarks/archive/sink_candidates.py @@ -12,7 +12,7 @@ changed files -- never to choose scope. v2 is spent for tuning; this informs the selection design and qualifies nothing. -Usage: python benchmarks/independent/sink_candidates.py [--population population-v2.toml] [--only ID,...] +Usage: python benchmarks/archive/sink_candidates.py [--population population-v2.toml] [--only ID,...] """ from __future__ import annotations @@ -24,7 +24,8 @@ import shutil import sys from pathlib import Path -HERE = Path(__file__).resolve().parent +# Archived 2026-10-02 from benchmarks/independent/: HERE stays that directory (population, evaluate.py, results). +HERE = Path(__file__).resolve().parents[1] / "independent" sys.path.insert(0, str(HERE)) import evaluate # noqa: E402 diff --git a/benchmarks/independent/verify_batched.py b/benchmarks/archive/verify_batched.py similarity index 97% rename from benchmarks/independent/verify_batched.py rename to benchmarks/archive/verify_batched.py index d406994..4bef059 100644 --- a/benchmarks/independent/verify_batched.py +++ b/benchmarks/archive/verify_batched.py @@ -22,7 +22,7 @@ Source only; the stage-1 candidates come from `model_sinks.py`. Results in `~/ou EXPLORATORY: v2 is spent for tuning; this qualifies nothing. Usage: - python benchmarks/independent/verify_batched.py [--only ID,...] [--subset sets.json] [--passes 2] [--budget-usd 40] + python benchmarks/archive/verify_batched.py [--only ID,...] [--subset sets.json] [--passes 2] [--budget-usd 40] """ from __future__ import annotations @@ -35,8 +35,10 @@ import sys import time from pathlib import Path -HERE = Path(__file__).resolve().parent +# Archived 2026-10-02 from benchmarks/independent/: HERE stays that directory (population, evaluate.py, results). +HERE = Path(__file__).resolve().parents[1] / "independent" sys.path.insert(0, str(HERE)) +sys.path.insert(0, str(Path(__file__).resolve().parent)) # archived siblings: model_sinks, verify_sinks import evaluate # noqa: E402 from model_sinks import OPERATIONS, chunks # noqa: E402 diff --git a/benchmarks/independent/verify_sinks.py b/benchmarks/archive/verify_sinks.py similarity index 97% rename from benchmarks/independent/verify_sinks.py rename to benchmarks/archive/verify_sinks.py index 78e6e1d..8d61354 100644 --- a/benchmarks/independent/verify_sinks.py +++ b/benchmarks/archive/verify_sinks.py @@ -10,7 +10,7 @@ Scope: the candidates `model_sinks.py` recorded for each case's vulnerable pin detected sites (`--fixed`). Findings are written in the scan-result shape `evaluate.py` reads. Hard spend ceiling. EXPLORATORY: v2 is spent for tuning; this qualifies nothing. -Usage: python benchmarks/independent/verify_sinks.py [--only ID,...] [--budget-usd 40] [--fixed] +Usage: python benchmarks/archive/verify_sinks.py [--only ID,...] [--budget-usd 40] [--fixed] """ from __future__ import annotations @@ -23,8 +23,10 @@ import sys import time from pathlib import Path -HERE = Path(__file__).resolve().parent +# Archived 2026-10-02 from benchmarks/independent/: HERE stays that directory (population, evaluate.py, results). +HERE = Path(__file__).resolve().parents[1] / "independent" sys.path.insert(0, str(HERE)) +sys.path.insert(0, str(Path(__file__).resolve().parent)) # archived siblings: model_sinks, verify_sinks import evaluate # noqa: E402 from model_sinks import OPERATIONS # noqa: E402 diff --git a/tests/test_harnessx_removal_equality.py b/tests/test_harnessx_removal_equality.py index 013c0a5..55f63bd 100644 --- a/tests/test_harnessx_removal_equality.py +++ b/tests/test_harnessx_removal_equality.py @@ -12,7 +12,7 @@ import json from pathlib import Path ROOT = Path(__file__).resolve().parents[1] -_SPEC = importlib.util.spec_from_file_location("hxr_equality", ROOT / "benchmarks" / "harnessx_removal_equality.py") +_SPEC = importlib.util.spec_from_file_location("hxr_equality", ROOT / "benchmarks" / "archive" / "harnessx_removal_equality.py") assert _SPEC is not None and _SPEC.loader is not None equality = importlib.util.module_from_spec(_SPEC) _SPEC.loader.exec_module(equality) diff --git a/tests/test_plane_scoring_parity.py b/tests/test_plane_scoring_parity.py index 0093578..7c44cb6 100644 --- a/tests/test_plane_scoring_parity.py +++ b/tests/test_plane_scoring_parity.py @@ -1,6 +1,6 @@ """`agree` scores exactly as the reference measurement did (ai-service-plane Requirement 6). -The reference is `benchmarks/independent/score_batched.py` over the batched-check scans: sites anchored by +The reference is `benchmarks/archive/score_batched.py` over the batched-check scans: sites anchored by `score_batched.anchored`, matched by `evaluate.matches_v2` against `evaluate.hunks(case, "old")`. The plane cannot ship the benchmark tree, so the generator resolves the ranges into `case.json` and `agree.matches_declared` reproduces the rule. These tests run both on the recorded reference outputs of the 15 validation cases and @@ -23,6 +23,7 @@ from openultrasast.plane.manifests import load_manifests from openultrasast.plane.tasks import agree BENCH = Path("benchmarks/independent") +ARCHIVE = Path("benchmarks/archive") # score_batched.py, archived 2026-10-02 POPULATION = BENCH / "population-v2.toml" SCANS = Path.home() / "ousast-results" / "independent-v2-batched-check" / "scans" REPOS = Path.home() / ".cache" / "openultrasast" / "independent" @@ -33,10 +34,10 @@ pytestmark = pytest.mark.skipif( ) -def _load(name: str) -> ModuleType: +def _load(name: str, where: Path = BENCH) -> ModuleType: sys.path.insert(0, str(BENCH.resolve())) # score_batched imports evaluate by name try: - spec = importlib.util.spec_from_file_location(f"parity_{name}", BENCH / f"{name}.py") + spec = importlib.util.spec_from_file_location(f"parity_{name}", where / f"{name}.py") assert spec is not None and spec.loader is not None module = importlib.util.module_from_spec(spec) spec.loader.exec_module(module) @@ -47,7 +48,7 @@ def _load(name: str) -> ModuleType: @pytest.fixture(scope="module") def reference() -> tuple[ModuleType, ModuleType]: - evaluate, score_batched = _load("evaluate"), _load("score_batched") + evaluate, score_batched = _load("evaluate"), _load("score_batched", ARCHIVE) evaluate.use(POPULATION.resolve()) return evaluate, score_batched