diff --git a/.kiro/specs/pair-corpus-honesty/tasks.md b/.kiro/specs/pair-corpus-honesty/tasks.md index f53d6ac..95749e6 100644 --- a/.kiro/specs/pair-corpus-honesty/tasks.md +++ b/.kiro/specs/pair-corpus-honesty/tasks.md @@ -200,6 +200,7 @@ - Review round 9 (2026-09-05) REJECTED 8.3 on two findings, both fixed RED-first (2 failing tests captured: warm cache + network off scored the pointer pair; `evaluate_profiles` kept an `agent` pointer pair): `_pointer_ready` consults `network_allowed` before the cache, so a warm cache only spares the fetch under `--pointers`/env and never turns a default run into a pointer run; `evolve.evaluate_profiles` applies `select_vendored` before the tier filter (design amendment: "tests and gates call `select_vendored`"; ProfileGate is the gate the design names, so the one-line filter is in boundary). Suggestions applied: fetch failures carry a `detail`; the harvest module load is memoized; pointer excerpt headers get `function`/`mechanism` from the expected row; `tests/conftest.py` autouse fixture clears `OPENULTRASAST_PAIRS_NETWORK` and points the cache at a temp dir for every test; full-catalog on-disk assertions use `select_vendored`. - Review round 10 (2026-09-05) APPROVED 8.3 (verify-completion: `pytest -q` 481 passed / 4 skipped; extra-free 468 / 17 inside a no-network namespace; ruff/format/mypy clean; three gates PASS; reviewer reproduced both RED tests by mutation, the warm-cache/network-off skip on the real cache, and that the default run creates no cache and touches no network). Non-blocking: top-level README/docs mention `--pointers` only in benchmarks/pairs/README.md (ride with 8.4); the cwd-relative harvest library path follows the DEFAULT_CATALOG convention. - Task 8.4 (2026-09-05, RED first: 4 failing tests captured — `build(pointers=)` unknown in the vibe-py builder, `merge_recipes` missing from the harvest library, the agent-vfc writer rendered booleans as strings, no pointer rows in the default catalog; then green): `harvest.merge_recipes` appends new recipes by name and never rewrites existing rows; `vibe-py/build_recipes.py --pointers` keeps the LLM-generated unlicensed repositories as `vendored = false`, `agent`, `review_tier = "seeded"` rows and appends them to recipes.toml (80 pairs over the 40 repositories; the 36 pre-existing recipes are byte-for-byte identical after reload; unlicensed human repositories stay skipped); `agent-vfc/build_recipes.py --from ... --pointers [--cache]` inverts the license rule, harvests each unlicensed candidate into the cache to derive its function label with `catalog_gen.derive_function`, dedupes against known commits/repos, and appends `title` pointer rows (30 added from a fresh 662-candidate search; 2 candidates failed to harvest, 10 had no named function; the 39 pre-existing recipes are identical). Catalogs regenerated: vibe-py 115 rows (35 vendored + 80 pointer), agent-vfc 59 rows (29 vendored + 30 pointer); training manifests unchanged (pointer rows carry no vendored files). Evidence for the observable, cache in the scratchpad: `pairs --slice vibe-py --pointers --json` scored 115 pairs, per_profile human 35 (13 correct, Youden +0.371) and agent 80 (0 correct, Youden 0.000; the LLM-repo baseline), all 80 pointer pairs fetched, no fetch failure; `pairs --slice agent-vfc --pointers --json` 1/59, Youden 0.000, unresolved 2; the default `pairs --slice vibe-py --json` scores 35 pairs and records 80 `pointer_pair_skipped`; `~/.cache/openultrasast` was never created. README/docs name the nightly `--pointers` command. Suite and gates unchanged (see round 11). +- Review round 12 (2026-09-05) verified every closure record, the runtime smoke, gates byte-identical to the pre-spec commit, and Req 1–10 coverage, and REJECTED the feature on one finding: `tests/test_pair_harvest.py` asserted redaction against a contiguous Google API key literal (introduced in the first task commit; the repository's push protection rejects such literals). Decision and fix: the key is now assembled at runtime (`"AIza" + "SyDEMO..."`, the `tests/test_redaction.py` convention) and, because the thirteen-commit series was local and unpushed, it was rewritten in place with a scripted `git rebase -x` so the literal exists in no commit (`git log -p` count 0); the temporary backup branch was deleted after the suite (485 / 472 passed), ruff and mypy were re-run green. Commit ids after 6cd842e changed; messages and trailers are unchanged. - ESCALATION (task 5.1 observable) — CLOSED 2026-09-05 by the design amendment (Req 10) and task 8.4: the 40 LLM-generated Real-Vuln repositories have no license, so they are `vendored = false` pointer rows (80 pairs, `agent`, `seeded`) scored by `pairs --slice vibe-py --pointers`; `per_profile` reports human 35 and agent 80; Req 5.4 still holds (nothing unlicensed is vendored). The vendored vibe-py rows remain human-only by construction. - ESCALATION (Req 5.3) — CLOSED 2026-09-05 by Req 9 (review tiers) via task 8.1: the agent-vfc rows load at `review_tier = "title"` (29 vendored + 30 pointer), are scored and reported, and never gate the improve loop; `catalog-candidates.toml` is retired and `REVIEW.md` is the queue a human works through by setting `reviewer` in recipes.toml, which promotes a row to `reviewed`. - Review round 11 (2026-09-05) APPROVED 8.4 (verify-completion: `pytest -q` 485 passed / 4 skipped, also inside a no-network namespace; extra-free 472 / 17; ruff/format/mypy clean; gate outputs byte-identical to HEAD and to the pre-spec commit; reviewer reproduced RED on a HEAD worktree, 9 mutations, the `--pointers` fetch of evicted cache entries, and byte-identical catalog regeneration offline). Feature verdict was REJECTED on records only (stale escalations, roadmap, spec.json), fixed in the closing commit. Carry-forwards for the follow-on specs: offline unit test for `agent-vfc/build_recipes.py build(pointers=True)`; `sink` not propagated onto pointer catalog rows; the suite writes `.harnessx` under `$HOME` (pre-existing); `merge_recipes` lives in the harvest library because HarvestLibrary already owns recipe-file handling (design dependency SliceCatalogs -> HarvestLibrary). @@ -209,9 +210,9 @@ - 2.4 baseline after the scorer change is in the roadmap (sast 4/11 → achievable 4/10; vfc 0/176 under function matching). Regex findings get an enclosing function from a parse so function-labeled rows can match the inventory column. - 2.5 hunter scorer: `pairs --hunter [--hunter-model]`; client resolution prefers HarnessX (`resolve_hunter_client(prefer_harnessx=...)`, best-effort adapter, offline-untested) then any OpenAI-compatible endpoint via `OPENROUTER_API_KEY` + `OPENROUTER_BASE_URL` (DeepSeek documented in `.env.example`). Live run not executed in this session; scripted-client test covers the scorer. - 3.x harvest: shared `benchmarks/pairs/harvest.py` (comment-blind `name`, `line_range`, `hunk` with next-function and whole-small-file fallbacks, `enclosing` with `fix_path`/`fix_line`), `vfc/harvest.py` is a shim; `catalog_gen.py --slice` regenerates catalogs and derives `function` for hunk-harvested pairs from the excerpt diff. 3.3: curl re-harvested; parse failures fell from 50 files to 36 (remaining are `#if`-split control flow and similar grammar limits, visible in `loss.parse_failed_files`; engine tolerance belongs to `overlay-ir-completeness`). -- 5.1 vibe-py: 35 pairs from the 15 licensed human-authored Real-Vuln repos. The 40 LLM-generated repos (`kolega-ai-dev/vc-*`) have no license file and were NOT vendored (Req 5.4); `build_recipes.py --allow-unlicensed` exists for local inspection only. So the slice currently has `human` provenance only; agent provenance comes from `agent-vfc`. +- 5.1 vibe-py: 35 pairs from the 15 licensed human-authored Real-Vuln repos. The 40 LLM-generated repos (`kolega-ai-dev/vc-*`) have no license file and were NOT vendored (Req 5.4); `build_recipes.py --allow-unlicensed` exists for local inspection only. So the slice currently has `human` provenance only; agent provenance comes from `agent-vfc`. _(dated note; superseded by task 8.4: the LLM repositories are pointer rows now)_ - 5.2 vfc-js: 15 pairs from upstream fix commits after dedupe by fix commit and module/repo attribution check (SecBench.js pointer only; 10 unlicensed upstreams, 3 duplicate fixes, 8 misattributed modules skipped; 2 excerpts without a named function not cataloged). `sink_location` is kept as metadata, not as the expected sink. -- 5.3 agent-vfc: 29 candidate pairs in `catalog-candidates.toml` (not loaded), all `reviewer = "pending"`; `REVIEW.md` is the review sheet. Mechanism/CWE were assigned from commit titles by `build_recipes.py` and must be confirmed by a human. 631 candidates were searched; 202 skipped for missing license; the loaded `catalog.toml` has 0 rows until review. +- 5.3 agent-vfc: 29 candidate pairs in `catalog-candidates.toml` (not loaded), all `reviewer = "pending"`; `REVIEW.md` is the review sheet. Mechanism/CWE were assigned from commit titles by `build_recipes.py` and must be confirmed by a human. 631 candidates were searched; 202 skipped for missing license; the loaded `catalog.toml` has 0 rows until review. _(dated note; superseded by task 8.1: rows load at tier `title`, the candidates file is retired)_ - 6.1 provenance: `src/openultrasast/provenance.py`, manifest `provenance` block, stage `provenance` in `_run_scan`. - 6.3 profile gate: `run_round(pair_cases=..., profile_tolerance, min_holdout_pairs)`; CLI `improve --pair-catalog/--no-pair-gate/--profile-tolerance/--min-holdout-pairs`; default on with the full catalog holdout split (slower rounds). - Measured after round-2 remediation (2026-09-04): sast 4/11 (achievable 4/10), Youden +0.273; vibe-py 13/35 pair-correct, Youden +0.371 (overlay) vs inventory 7/35, 0 weak labels; vfc-js 0/15, Youden −0.200 (2 parse-failed files, 0 weak labels; the earlier 5/32 was inflated by first-function labels and duplicate rows); agent-vfc 0 loaded rows, 29 candidates awaiting review. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index df24f77..fd9d609 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -64,7 +64,7 @@ sast: java-sqli moved MISS→LEAK (coverage counted on both sides; the fixed twi ## Direct Implementation Candidates -- [ ] Record inventory vs overlay Youden side-by-side in `ousast pairs --slice sast` JSON -- folded into pair-corpus-honesty task 2.3. +- [x] Record inventory vs overlay Youden side-by-side in `ousast pairs --slice sast` JSON -- folded into pair-corpus-honesty task 2.3. Done (`scorers` block, 2026-09-05). - [x] Add Firefox/gecko-dev, Chromium, OpenSSL pointers to `benchmarks/pairs/datasets.toml`. ## Specs (dependency order)