From d36a68724f10a948730ed7190d15bd54ee2c0177 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Sat, 5 Sep 2026 10:52:23 +0200 Subject: [PATCH] docs(pair-corpus-honesty): record the round 13 feature approval Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG --- .kiro/specs/pair-corpus-honesty/spec.json | 4 ++-- .kiro/specs/pair-corpus-honesty/tasks.md | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/.kiro/specs/pair-corpus-honesty/spec.json b/.kiro/specs/pair-corpus-honesty/spec.json index 2572282..d2a53fc 100644 --- a/.kiro/specs/pair-corpus-honesty/spec.json +++ b/.kiro/specs/pair-corpus-honesty/spec.json @@ -1,7 +1,7 @@ { "feature_name": "pair-corpus-honesty", "created_at": "2026-09-04T16:12:17Z", - "updated_at": "2026-09-05T12:00:00Z", + "updated_at": "2026-09-05T12:30:00Z", "language": "en", "phase": "implementation", "supersedes": [ @@ -27,5 +27,5 @@ ], "completed": true, "completed_at": "2026-09-05T12:00:00Z", - "review_rounds": 11 + "review_rounds": 13 } diff --git a/.kiro/specs/pair-corpus-honesty/tasks.md b/.kiro/specs/pair-corpus-honesty/tasks.md index 95749e6..2c4eaec 100644 --- a/.kiro/specs/pair-corpus-honesty/tasks.md +++ b/.kiro/specs/pair-corpus-honesty/tasks.md @@ -200,7 +200,8 @@ - Review round 9 (2026-09-05) REJECTED 8.3 on two findings, both fixed RED-first (2 failing tests captured: warm cache + network off scored the pointer pair; `evaluate_profiles` kept an `agent` pointer pair): `_pointer_ready` consults `network_allowed` before the cache, so a warm cache only spares the fetch under `--pointers`/env and never turns a default run into a pointer run; `evolve.evaluate_profiles` applies `select_vendored` before the tier filter (design amendment: "tests and gates call `select_vendored`"; ProfileGate is the gate the design names, so the one-line filter is in boundary). Suggestions applied: fetch failures carry a `detail`; the harvest module load is memoized; pointer excerpt headers get `function`/`mechanism` from the expected row; `tests/conftest.py` autouse fixture clears `OPENULTRASAST_PAIRS_NETWORK` and points the cache at a temp dir for every test; full-catalog on-disk assertions use `select_vendored`. - Review round 10 (2026-09-05) APPROVED 8.3 (verify-completion: `pytest -q` 481 passed / 4 skipped; extra-free 468 / 17 inside a no-network namespace; ruff/format/mypy clean; three gates PASS; reviewer reproduced both RED tests by mutation, the warm-cache/network-off skip on the real cache, and that the default run creates no cache and touches no network). Non-blocking: top-level README/docs mention `--pointers` only in benchmarks/pairs/README.md (ride with 8.4); the cwd-relative harvest library path follows the DEFAULT_CATALOG convention. - Task 8.4 (2026-09-05, RED first: 4 failing tests captured — `build(pointers=)` unknown in the vibe-py builder, `merge_recipes` missing from the harvest library, the agent-vfc writer rendered booleans as strings, no pointer rows in the default catalog; then green): `harvest.merge_recipes` appends new recipes by name and never rewrites existing rows; `vibe-py/build_recipes.py --pointers` keeps the LLM-generated unlicensed repositories as `vendored = false`, `agent`, `review_tier = "seeded"` rows and appends them to recipes.toml (80 pairs over the 40 repositories; the 36 pre-existing recipes are byte-for-byte identical after reload; unlicensed human repositories stay skipped); `agent-vfc/build_recipes.py --from ... --pointers [--cache]` inverts the license rule, harvests each unlicensed candidate into the cache to derive its function label with `catalog_gen.derive_function`, dedupes against known commits/repos, and appends `title` pointer rows (30 added from a fresh 662-candidate search; 2 candidates failed to harvest, 10 had no named function; the 39 pre-existing recipes are identical). Catalogs regenerated: vibe-py 115 rows (35 vendored + 80 pointer), agent-vfc 59 rows (29 vendored + 30 pointer); training manifests unchanged (pointer rows carry no vendored files). Evidence for the observable, cache in the scratchpad: `pairs --slice vibe-py --pointers --json` scored 115 pairs, per_profile human 35 (13 correct, Youden +0.371) and agent 80 (0 correct, Youden 0.000; the LLM-repo baseline), all 80 pointer pairs fetched, no fetch failure; `pairs --slice agent-vfc --pointers --json` 1/59, Youden 0.000, unresolved 2; the default `pairs --slice vibe-py --json` scores 35 pairs and records 80 `pointer_pair_skipped`; `~/.cache/openultrasast` was never created. README/docs name the nightly `--pointers` command. Suite and gates unchanged (see round 11). -- Review round 12 (2026-09-05) verified every closure record, the runtime smoke, gates byte-identical to the pre-spec commit, and Req 1–10 coverage, and REJECTED the feature on one finding: `tests/test_pair_harvest.py` asserted redaction against a contiguous Google API key literal (introduced in the first task commit; the repository's push protection rejects such literals). Decision and fix: the key is now assembled at runtime (`"AIza" + "SyDEMO..."`, the `tests/test_redaction.py` convention) and, because the thirteen-commit series was local and unpushed, it was rewritten in place with a scripted `git rebase -x` so the literal exists in no commit (`git log -p` count 0); the temporary backup branch was deleted after the suite (485 / 472 passed), ruff and mypy were re-run green. Commit ids after 6cd842e changed; messages and trailers are unchanged. +- Review round 12 (2026-09-05) verified every closure record, the runtime smoke, gates byte-identical to the pre-spec commit, and Req 1–10 coverage, and REJECTED the feature on one finding: `tests/test_pair_harvest.py` asserted redaction against a contiguous Google API key literal (introduced in the first task commit; the repository's push protection rejects such literals). Decision and fix: the key is now assembled at runtime (`"AIza" + "SyDEMO..."`, the `tests/test_redaction.py` convention) and, because the thirteen-commit series was local and unpushed, it was rewritten in place with a scripted `git rebase -x` so the literal exists in no commit (`git log -p` count 0); the temporary backup branch was deleted after the suite (485 / 472 passed), ruff and mypy were re-run green. 6cd842e and every later commit were replaced (the first task commit is now 071dea4); messages and trailers are unchanged. +- Review round 13 (2026-09-05) APPROVED the feature (FEATURE_GO): the key literal is absent from every commit tree of the rewritten series, trailers intact, no backup ref, suite 485 / 472 passed, statics clean, gate outputs byte-identical to the pre-spec archive, default-catalog smoke and network-off skips reproduced; Req 1–10 coverage carried from round 12 because `src/` is byte-identical to that state. Dangling pre-rewrite objects purged locally afterwards. - ESCALATION (task 5.1 observable) — CLOSED 2026-09-05 by the design amendment (Req 10) and task 8.4: the 40 LLM-generated Real-Vuln repositories have no license, so they are `vendored = false` pointer rows (80 pairs, `agent`, `seeded`) scored by `pairs --slice vibe-py --pointers`; `per_profile` reports human 35 and agent 80; Req 5.4 still holds (nothing unlicensed is vendored). The vendored vibe-py rows remain human-only by construction. - ESCALATION (Req 5.3) — CLOSED 2026-09-05 by Req 9 (review tiers) via task 8.1: the agent-vfc rows load at `review_tier = "title"` (29 vendored + 30 pointer), are scored and reported, and never gate the improve loop; `catalog-candidates.toml` is retired and `REVIEW.md` is the queue a human works through by setting `reviewer` in recipes.toml, which promotes a row to `reviewed`. - Review round 11 (2026-09-05) APPROVED 8.4 (verify-completion: `pytest -q` 485 passed / 4 skipped, also inside a no-network namespace; extra-free 472 / 17; ruff/format/mypy clean; gate outputs byte-identical to HEAD and to the pre-spec commit; reviewer reproduced RED on a HEAD worktree, 9 mutations, the `--pointers` fetch of evicted cache entries, and byte-identical catalog regeneration offline). Feature verdict was REJECTED on records only (stale escalations, roadmap, spec.json), fixed in the closing commit. Carry-forwards for the follow-on specs: offline unit test for `agent-vfc/build_recipes.py build(pointers=True)`; `sink` not propagated onto pointer catalog rows; the suite writes `.harnessx` under `$HOME` (pre-existing); `merge_recipes` lives in the harvest library because HarvestLibrary already owns recipe-file handling (design dependency SliceCatalogs -> HarvestLibrary).