From 27f9ec6a92198f640e6a9a23d9619a3308d89653 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Sat, 5 Sep 2026 10:38:50 +0200 Subject: [PATCH] docs(pair-corpus-honesty): close the spec; escalations resolved, roadmap carries the measured state - tasks.md: all tasks checked; the two escalations are closed (5.1 by the Req 10 pointer rows, 5.3 by the Req 9 review tiers); round 11 recorded with the carry-forwards for corpus-seeded-mechanisms and overlay-ir-completeness. - roadmap: measured state after tiers and pointer slices (vibe-py 115, agent-vfc 59, gates byte-identical to pre-spec); pair-corpus-honesty marked completed. - spec.json: completed 2026-09-05 after eleven review rounds. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG --- .kiro/specs/pair-corpus-honesty/spec.json | 7 +++++-- .kiro/specs/pair-corpus-honesty/tasks.md | 11 ++++++----- .kiro/steering/roadmap.md | 4 +++- 3 files changed, 14 insertions(+), 8 deletions(-) diff --git a/.kiro/specs/pair-corpus-honesty/spec.json b/.kiro/specs/pair-corpus-honesty/spec.json index 9a174a0..2572282 100644 --- a/.kiro/specs/pair-corpus-honesty/spec.json +++ b/.kiro/specs/pair-corpus-honesty/spec.json @@ -1,7 +1,7 @@ { "feature_name": "pair-corpus-honesty", "created_at": "2026-09-04T16:12:17Z", - "updated_at": "2026-09-05T09:00:00Z", + "updated_at": "2026-09-05T12:00:00Z", "language": "en", "phase": "implementation", "supersedes": [ @@ -24,5 +24,8 @@ "ready_for_implementation": true, "amendments": [ "2026-09-05 review tiers (Req 9) and pointer slices (Req 10), approved by the maintainer in session" - ] + ], + "completed": true, + "completed_at": "2026-09-05T12:00:00Z", + "review_rounds": 11 } diff --git a/.kiro/specs/pair-corpus-honesty/tasks.md b/.kiro/specs/pair-corpus-honesty/tasks.md index 4713c38..f53d6ac 100644 --- a/.kiro/specs/pair-corpus-honesty/tasks.md +++ b/.kiro/specs/pair-corpus-honesty/tasks.md @@ -89,7 +89,7 @@ - _Boundary: SliceCatalogs_ - [x] 5. Integration: new slices -- [x] 5.1 Add the vibe-py slice from Real-Vuln-Benchmark _(escalated: the "at least one agent provenance" observable cannot be met without vendoring unlicensed LLM-generated repos; see Implementation Notes)_ +- [x] 5.1 Add the vibe-py slice from Real-Vuln-Benchmark _(the "at least one agent provenance" observable is met by the 80 `agent`/`seeded` pointer rows of task 8.4, Req 10; nothing unlicensed is vendored)_ - Recipes carry pinned commit, file, line range, function, primary and acceptable CWEs, authorship, and vulnerable or trap flag; harvest uses line-range mode; traps become fixed-side twins; provenance from authorship. - README states the educational and CTF caveat and the license. - Observable: the slice loads offline with tens of pairs, at least one agent and one human provenance, and scores with per-profile metrics. @@ -155,7 +155,7 @@ - _Requirements: 8.3_ - _Depends: 7.1_ -- [ ] 8. Amendment: review tiers and pointer slices +- [x] 8. Amendment: review tiers and pointer slices - [x] 8.1 Add review tiers to the label schema and the payload - `review_tier` on every pair with per-slice defaults; `per_tier` metrics in the payload; `title` rows load and replace the candidates file; `reviewed` requires a reviewer name. - Observable: agent-vfc rows load with tier `title`; `pairs --json` reports `per_tier`; a `reviewed` row without a reviewer is rejected by the loader. @@ -176,7 +176,7 @@ - _Boundary: Scorer, HarvestLibrary_ - _Depends: 8.1_ -- [ ] 8.4 Add the LLM-generated Real-Vuln repositories and unlicensed agent fixes as pointer pairs +- [x] 8.4 Add the LLM-generated Real-Vuln repositories and unlicensed agent fixes as pointer pairs - vibe-py gains the 40 LLM-generated repositories as `agent`, `seeded`, `vendored = false` rows built by `build_recipes.py --pointers`; agent-vfc gains the license-less candidates as `title`, `vendored = false` rows. - Observable: `pairs --slice vibe-py --pointers --json` scores both provenances; the default suite and gates are unchanged. - _Requirements: 10.6, 5.1_ @@ -200,8 +200,9 @@ - Review round 9 (2026-09-05) REJECTED 8.3 on two findings, both fixed RED-first (2 failing tests captured: warm cache + network off scored the pointer pair; `evaluate_profiles` kept an `agent` pointer pair): `_pointer_ready` consults `network_allowed` before the cache, so a warm cache only spares the fetch under `--pointers`/env and never turns a default run into a pointer run; `evolve.evaluate_profiles` applies `select_vendored` before the tier filter (design amendment: "tests and gates call `select_vendored`"; ProfileGate is the gate the design names, so the one-line filter is in boundary). Suggestions applied: fetch failures carry a `detail`; the harvest module load is memoized; pointer excerpt headers get `function`/`mechanism` from the expected row; `tests/conftest.py` autouse fixture clears `OPENULTRASAST_PAIRS_NETWORK` and points the cache at a temp dir for every test; full-catalog on-disk assertions use `select_vendored`. - Review round 10 (2026-09-05) APPROVED 8.3 (verify-completion: `pytest -q` 481 passed / 4 skipped; extra-free 468 / 17 inside a no-network namespace; ruff/format/mypy clean; three gates PASS; reviewer reproduced both RED tests by mutation, the warm-cache/network-off skip on the real cache, and that the default run creates no cache and touches no network). Non-blocking: top-level README/docs mention `--pointers` only in benchmarks/pairs/README.md (ride with 8.4); the cwd-relative harvest library path follows the DEFAULT_CATALOG convention. - Task 8.4 (2026-09-05, RED first: 4 failing tests captured — `build(pointers=)` unknown in the vibe-py builder, `merge_recipes` missing from the harvest library, the agent-vfc writer rendered booleans as strings, no pointer rows in the default catalog; then green): `harvest.merge_recipes` appends new recipes by name and never rewrites existing rows; `vibe-py/build_recipes.py --pointers` keeps the LLM-generated unlicensed repositories as `vendored = false`, `agent`, `review_tier = "seeded"` rows and appends them to recipes.toml (80 pairs over the 40 repositories; the 36 pre-existing recipes are byte-for-byte identical after reload; unlicensed human repositories stay skipped); `agent-vfc/build_recipes.py --from ... --pointers [--cache]` inverts the license rule, harvests each unlicensed candidate into the cache to derive its function label with `catalog_gen.derive_function`, dedupes against known commits/repos, and appends `title` pointer rows (30 added from a fresh 662-candidate search; 2 candidates failed to harvest, 10 had no named function; the 39 pre-existing recipes are identical). Catalogs regenerated: vibe-py 115 rows (35 vendored + 80 pointer), agent-vfc 59 rows (29 vendored + 30 pointer); training manifests unchanged (pointer rows carry no vendored files). Evidence for the observable, cache in the scratchpad: `pairs --slice vibe-py --pointers --json` scored 115 pairs, per_profile human 35 (13 correct, Youden +0.371) and agent 80 (0 correct, Youden 0.000; the LLM-repo baseline), all 80 pointer pairs fetched, no fetch failure; `pairs --slice agent-vfc --pointers --json` 1/59, Youden 0.000, unresolved 2; the default `pairs --slice vibe-py --json` scores 35 pairs and records 80 `pointer_pair_skipped`; `~/.cache/openultrasast` was never created. README/docs name the nightly `--pointers` command. Suite and gates unchanged (see round 11). -- ESCALATION (task 5.1 observable): vibe-py has `human` provenance only because the 40 LLM-generated Real-Vuln repos have no license (Req 5.4 wins). Spec owner decision needed: accept human-only vibe-py, or obtain a license statement upstream. -- ESCALATION (Req 5.3): agent-vfc rows are unreviewed. They are NOT in the loaded catalog; `benchmarks/pairs/agent-vfc/catalog-candidates.toml` and `REVIEW.md` list them for a human reviewer. Task 5.3 stays open until reviewed. +- ESCALATION (task 5.1 observable) — CLOSED 2026-09-05 by the design amendment (Req 10) and task 8.4: the 40 LLM-generated Real-Vuln repositories have no license, so they are `vendored = false` pointer rows (80 pairs, `agent`, `seeded`) scored by `pairs --slice vibe-py --pointers`; `per_profile` reports human 35 and agent 80; Req 5.4 still holds (nothing unlicensed is vendored). The vendored vibe-py rows remain human-only by construction. +- ESCALATION (Req 5.3) — CLOSED 2026-09-05 by Req 9 (review tiers) via task 8.1: the agent-vfc rows load at `review_tier = "title"` (29 vendored + 30 pointer), are scored and reported, and never gate the improve loop; `catalog-candidates.toml` is retired and `REVIEW.md` is the queue a human works through by setting `reviewer` in recipes.toml, which promotes a row to `reviewed`. +- Review round 11 (2026-09-05) APPROVED 8.4 (verify-completion: `pytest -q` 485 passed / 4 skipped, also inside a no-network namespace; extra-free 472 / 17; ruff/format/mypy clean; gate outputs byte-identical to HEAD and to the pre-spec commit; reviewer reproduced RED on a HEAD worktree, 9 mutations, the `--pointers` fetch of evicted cache entries, and byte-identical catalog regeneration offline). Feature verdict was REJECTED on records only (stale escalations, roadmap, spec.json), fixed in the closing commit. Carry-forwards for the follow-on specs: offline unit test for `agent-vfc/build_recipes.py build(pointers=True)`; `sink` not propagated onto pointer catalog rows; the suite writes `.harnessx` under `$HOME` (pre-existing); `merge_recipes` lives in the harvest library because HarvestLibrary already owns recipe-file handling (design dependency SliceCatalogs -> HarvestLibrary). - 2026-09-04 execution evidence (manual mode, main context): full suite `uv run pytest` 440 passed / 4 skipped; `python -m openultrasast.gate` PASS; `map_gate` PASS; `pair_gate` PASS (local 3/3); `ousast --help` boots. Validation commands come from `.github/workflows/ci.yml`. - Scorer (2.1–2.3): coverage-with-source counts both ways; function-scoped match; CWE-only rows are `weak_label`; `scorers[slice]` has overlay+inventory for every overlay slice, `hunter` with a client; payload gains `per_profile`, `per_mechanism`, `achievable`, `known_limit`, `loss`, `degradations`. `OverlayRecord.function` is the one additive overlay field; `_flows_for` parses once and keeps function ranges. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index fd39796..df24f77 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -41,6 +41,8 @@ Overlay column equals inventory on vfc (no C memory sink fact). 25 curl pairs pa New slices (pair-corpus-honesty, 2026-09-05, after the debug round: shared declarator naming, containment matching, sink-location labels): vibe-py (35, human, Real-Vuln licensed repos) 13/35 pair-correct, Youden +37.1% overlay vs 7/35 inventory; vfc-js (17) 5/17, Youden +23.5%; agent-vfc (29, agent, tier `title`) 0/29, Youden 0.0: no excerpt carries a rule-known sink and the overlay has no TypeScript grammar (18 rows); the unreviewed-tier baseline, never a gate. vfc parse-failed files 50 → 36 after the harvest fix. +Measured 2026-09-05 after review tiers (Req 9) and pointer slices (Req 10): vibe-py 115 = 35 vendored human (13/35, Youden +37.1%) + 80 `agent` pointer pairs over the 40 LLM-generated Real-Vuln repositories (0/80, Youden 0.0; the LLM-repo baseline, scored by the nightly `pairs --slice vibe-py --pointers` only, never in tests or gates); agent-vfc 59 = 29 vendored + 30 pointer, tier `title`, 1/59, Youden 0.0, 2 unresolved labels; sast, vfc-js, vfc unchanged; detection/map/local-pair gate outputs byte-identical to the pre-spec commit. Only `seeded`/`reviewed` pairs gate the improve loop. Carry-forwards: `language_unsupported` loss counter and multi-line C/C++ declarator rescue (overlay-ir-completeness), `profiles_ungated` reporting and an offline test for the agent-vfc pointer builder (corpus-seeded-mechanisms). + sast: java-sqli moved MISS→LEAK (coverage counted on both sides; the fixed twin leaks via parameter-as-source). vfc: recall fell to zero because every label now names a function and no C detection lands inside it; that is the honest baseline the engine specs measure against. ## Constraints @@ -69,7 +71,7 @@ sast: java-sqli moved MISS→LEAK (coverage counted on both sides; the fixed twi - [x] tree-sitter-overlay-extra -- Optional extra: tree-sitter + grammars, one CST walker, `quick` stays zero-dep. Dependencies: none - [x] real-world-vfc-slice -- Harvest reviewed function-level vuln/fix pairs from large projects (Firefox/gecko-dev, Chromium, OpenSSL, …) as an honesty slice. Seed is testable (`ousast pairs --slice vfc`). Dependencies: none. **Closed 2026-09-04, superseded by pair-corpus-honesty on the corpus/scorer plane.** -- [ ] pair-corpus-honesty -- Scorer counts coverage and matches by function; labels carry mechanism/provenance/split/known_limit; harvest anchors at the declarator with line-range and hunk modes; sast hygiene; new slices `vibe-py` (Real-Vuln-Benchmark), `vfc-js` (SecBench.js fix commits), `agent-vfc` (agent-trailer fixes); deterministic provenance fingerprint; per-profile improve acceptance. Dependencies: none (supersedes real-world-vfc-slice) +- [x] pair-corpus-honesty -- Scorer counts coverage and matches by function; labels carry mechanism/provenance/split/known_limit; harvest anchors at the declarator with line-range and hunk modes; sast hygiene; new slices `vibe-py` (Real-Vuln-Benchmark), `vfc-js` (SecBench.js fix commits), `agent-vfc` (agent-trailer fixes); deterministic provenance fingerprint; per-profile improve acceptance; review tiers and pointer slices (amendment). Dependencies: none (supersedes real-world-vfc-slice). **Completed 2026-09-05 (11 review rounds).** - [ ] corpus-seeded-mechanisms -- Every seeded or reviewed pair becomes a mechanism record (source shape, sink shape, guard the fix added); structural variant search over the scanned tree proposes matches as `variant` findings that the overlay and sandbox verify; leave-one-out recall on the pair corpus is the corpus's own detection rate; the improve loop gains a recall-raising `mechanisms` lever under the per-profile holdout gate. Principle: recall first under a false-positive ceiling, per profile. Dependencies: pair-corpus-honesty (tiers, pointer slices) - [ ] reachability-flow-model -- Name-resolved call graph over `FunctionIR`; per-function taint summaries composed along the graph into path records; graph reachability drives rank, map, worth-fixing; hunter gets `path_context` and hunter findings with a locatable sink become sandbox candidates; closed-kind fact oracle with candidate cache; `facts` improve lever under the per-profile gate; fix points ranked by reachable risk removed. Dependencies: pair-corpus-honesty - [ ] overlay-ir-completeness -- Shared CST walker completeness (C `init_declarator`, lvalue paths for field stores, per-grammar literal kinds, local ERROR-node tolerance, multi-line C/C++ declarators the walker names `` and `functions.py`'s one-line rescue cannot name: `chromium-cve-2022-2162`, `openssl-cve-2025-69421`; a Scorer-side multi-line-aware rescue is the cheaper alternative) plus Python parity with stdlib ast (f-string constness, augmented assignment, kwargs); injection-regime facts: container/transformer taint, `weak_literals` consulted, parameter sources off for injection. Measured on sast and vibe-py. Dependencies: pair-corpus-honesty