diff --git a/.kiro/specs/corpus-seeded-mechanisms/spec.json b/.kiro/specs/corpus-seeded-mechanisms/spec.json index 1900791..2fa9d26 100644 --- a/.kiro/specs/corpus-seeded-mechanisms/spec.json +++ b/.kiro/specs/corpus-seeded-mechanisms/spec.json @@ -1,7 +1,7 @@ { "feature_name": "corpus-seeded-mechanisms", "created_at": "2026-09-05T06:18:31Z", - "updated_at": "2026-09-05T13:00:00Z", + "updated_at": "2026-09-05T16:30:00Z", "language": "en", "phase": "implementation", "approvals": { @@ -19,5 +19,8 @@ } }, "ready_for_implementation": true, - "approval_note": "2026-09-05 approved by the maintainer in session ('continue with corpus-seeded-mechanisms')" + "approval_note": "2026-09-05 approved by the maintainer in session ('continue with corpus-seeded-mechanisms')", + "completed": true, + "completed_at": "2026-09-05T16:30:00Z", + "review_rounds": 6 } diff --git a/.kiro/specs/corpus-seeded-mechanisms/tasks.md b/.kiro/specs/corpus-seeded-mechanisms/tasks.md index bde203c..319be5c 100644 --- a/.kiro/specs/corpus-seeded-mechanisms/tasks.md +++ b/.kiro/specs/corpus-seeded-mechanisms/tasks.md @@ -64,15 +64,15 @@ - _Requirements: 5.3, 5.4_ - _Depends: 4.1, 3.1_ -- [ ] 5. Integration: reports -- [ ] 5.1 Show mechanism id, known fix guard, and provenance on findings +- [x] 5. Integration: reports +- [x] 5.1 Show mechanism id, known fix guard, and provenance on findings - Markdown and SARIF show mechanism summary, guard shape, and pair provenance for findings carrying a mechanism id; variant findings are labeled suspicions unless raised. - Observable: a standard scan report on the split-sink fixture names the mechanism and its guard; SARIF properties carry the id. - _Requirements: 6.1, 6.3_ - _Depends: 2.2_ -- [ ] 6. Validation -- [ ] 6.1 Prove offline tests and unchanged gates +- [x] 6. Validation +- [x] 6.1 Prove offline tests and unchanged gates - Unit and integration tests run without network, model, or sandbox; semantic-marked tests skip without the extra; detection, map, and local pair gates unchanged; extra-free suite green. - Observable: both pytest matrices green; gate entrypoints unchanged. - _Requirements: 7.1, 7.2, 7.3_ @@ -93,5 +93,11 @@ - Task 4.1 (2026-09-05, RED first: 3 failing tests in `tests/test_mechanism_lever.py`, `ImportError: MechanismEdit`; then green after the store tombstone landed): `MechanismEdit(action, mechanism_id, source, rationale)` with lever `mechanisms` (`VALID_LEVERS` extended); `EvolveValidator.validate_mechanism(edit, candidates)` accepts only admit/retract of a record present in the exporter's candidate set with `origin = corpus`, a shape whose guard and source kinds are closed and whose `sink_name`/`language`/`mechanism` are identifiers (a forged guard `if user not in ALLOWED` or a sink `os.system(cmd + ' ')` is rejected); `apply_mechanism_edits(edits, candidates, scan_store) -> StoreSnapshot` appends the exporter record for admit and a `retracted = true` tombstone for retract (the log stays append-only; `MechanismStore.load()` folds tombstones away), `StoreSnapshot.restore()` is the byte-for-byte revert. - Task 4.2 (2026-09-05, RED first: 2 failing tests in `tests/test_mechanism_loop.py` — `evaluate_mechanism_profiles` missing, `run_round` rejected `mechanism_candidates`; then green): `evolve.propose_mechanism_edits(holdout, candidates, scan_store)` retracts admitted records that leak on a holdout fixed side and admits a candidate only when a trial store holding it alone recovers a currently missed holdout pair and leaks nowhere; `evaluate_mechanism_profiles` scores the holdout pairs with a given store per provenance profile (pairs, pair_correct = detected and silent, pass rate, Youden); `run_round(..., mechanism_candidates=, mechanism_store=, scripted_mechanism_edits=)` validates mechanism edits with the same validator, applies them after the rule gate passed, re-scores, and rejects with byte-for-byte revert on `mechanism_leak:`, `profile_regression:` (the existing per-profile clause over the before/after store metrics) or `mechanism_no_gain`; reverted keys are blocked in later rounds like rule edits; the journal records each mechanism edit with its pair provenance and `mechanisms_before`/`mechanisms_after`. This is the closed loop: leave-one-out misses propose, holdout leaks retract, the gate decides, the store is the only thing that changes. Design deviation recorded: the gate for mechanism edits is the per-profile clause computed by variant search (the change they make) rather than by the overlay scorer, which cannot see mechanisms; the rule gate and its clause are unchanged. - Review round 3 (2026-09-05) APPROVED 3.1, 3.2, 4.1, 4.2 (reviewer re-simulated the three REDs, reproduced the roadmap table offline in a network namespace, confirmed no gate reads loo.json and gates byte-identical to HEAD, ran 16 mutants: 12 killed). Judged the variant-search-based per-profile clause for mechanism edits acceptable and required (design §Error Handling's leak row is only checkable through variant search). Carried into round 4: tests pinning the four surviving mutants (hit outside the labeled function, hit not naming the label, `mechanism_no_gain`, reverted mechanism keys re-proposed); forward `search_tree` degradations into `LooResult`; design §Lever sync (deviation, tombstone, reasons, `score_pair_with_store`); novelty gate keyed on the mechanisms lever regardless of rationale; exporter must not re-admit a standing tombstone (default `--store` to a candidates file); retraction allowed for admitted records absent from the candidate set; the `not a and b or c` precedence in the loop test. +- Round-3 suggestions applied (2026-09-05): `tests/test_loo.py` pins containment and label naming (a sink call in a helper outside the labeled function and an off-label mechanism never count) and forwards `search_tree` degradations into `LooResult.degradations` (deduped); `reverted_edit_keys` blocks mechanism-lever keys regardless of rationale, and a scripted no-gain admission is rejected as `mechanism_no_gain`, reverted, and not re-proposed (`tests/test_mechanism_loop.py`); `validate_mechanism(..., admitted=)` allows retracting a record admitted in the scan store that left the candidate set; `mechanisms export` defaults to `.openultrasast/calibration/mechanism-candidates.jsonl` so the exporter never re-admits a standing tombstone (only the lever writes `mechanisms.jsonl`); the loop-test precedence fixed; design §Lever rewritten to the implemented gate (deviation, tombstone, reasons, `score_pair_with_store`, candidates file). +- Task 5.1 (2026-09-05, RED first: 3 failing tests in `tests/test_mechanism_reports.py` — `write_markdown_report`/`write_sarif_report` rejected `mechanisms=`; the scan-level test stayed red until the CLI wiring; then green): the report writers take `mechanisms: {id: {summary, guard, pairs, cwe}}`; a finding cites a mechanism through its `mechanism:` tag (variant) or its merged overlay record's `mechanism_id`; markdown shows `Mechanism`, `Known fix guard`, `Learned from pairs` per finding plus a `## Mechanisms` section, SARIF properties carry `mechanism_id`, `mechanism_summary`, `mechanism_guard`, `mechanism_pairs`; variant findings keep `suspicion`; `_search_variants` returns the cited mechanisms from the store and `_run_scan` passes them to both writers. Reports without mechanisms are unchanged. +- Task 6.1 evidence (2026-09-05): both pytest matrices green (see the numbers in round 4), semantic-marked tests skip without the extra, `gate`/`map_gate`/`pair_gate` verdicts and stdout unchanged (reviewers compared against `git archive HEAD` in rounds 2 and 3), no test needs network, model or sandbox (fake sandbox only). `docs/examples.md` and `README.md` name `mechanisms export` and `pairs --loo`. +- Review round 4 (2026-09-05) APPROVED 5.1 (12 of 12 targeted mutants killed; Req 6.3 checked directly); REJECTED 6.1 and the feature on two records-and-portability points: `tests/test_mechanism_lever.py` symlinked this machine's absolute repository path, and the notes did not say the `mechanisms` lever was reachable only through `run_round`. Remediated: the test symlinks `Path(__file__).resolve().parents[1] / "benchmarks"`; instead of recording the gap, the lever is now threaded end to end — `run_improvement(mechanism_candidates=, mechanism_store=)` and `ousast improve --mechanism-candidates/--mechanism-store/--no-mechanisms` (defaults: `/.openultrasast/calibration/mechanism-candidates.jsonl` -> `mechanisms.jsonl` next to it; active when the candidates file exists and the pair gate is on), RED-first (`tests/test_mechanism_loop.py`: the CLI round admitted nothing; then green: `export -> candidates -> improve -> mechanisms.jsonl the scan reads`, converging on `no_proposals`); `RoundOutcome.mechanism_edits` is populated; the merged-overlay report path is asserted end to end in `tests/test_variant_scan.py` (mutants M12/M13 now killed). Full suite 524 passed / 4 skipped, extra-free 511 / 17, statics clean, three gates PASS. +- Review round 5 (2026-09-05) REJECTED 6.1 and the feature on three concrete points, fixed RED-first (2 failing tests captured in `tests/test_mechanism_loop.py`: `--dry-run` wrote the scan-time store; the documented `export` (cwd default) -> `improve` (target default) flow admitted nothing): `improve` now resolves candidates from the target's calibration dir, then the working directory's, and always writes the store under the target (where a scan of it reads); `--dry-run` copies the store to scratch so nothing under the target changes; `--no-mechanisms` and the absent-candidates guard are tested; the accepted round prints `admit `/`retract `; the roadmap paragraph that had split the leave-one-out table sits below it and records the measured admission state (4 records, human 0 -> 6 of 17, Youden +0.353 on the python-vulnerable target); design file plan no longer says "gate unchanged". +- Review round 6 (2026-09-05) APPROVED 6.1 and the feature (FEATURE_GO): reviewer reproduced the documented default flow on a clean copy (export from cwd, `improve benchmarks/manifests/python-vulnerable.toml --pair-catalog benchmarks/pairs/catalog.toml` admits 4 records, human 0 -> 6 of 17, Youden +0.353, idempotent on re-run, `--dry-run` leaves the target untouched), ran every spec test and the export/LOO/scan commands with sockets blocked, killed the four round-5 mutants, and confirmed gates byte-identical to HEAD and the pre-spec commit. Cosmetic suggestions applied in the landing commit: argparse help strings name the real defaults, the dry-run test judges only the candidate-less round's stdout, the unneeded `benchmarks/` symlink dropped from the default-flow test. - Seeding reads `PairCase.review_tier` from `pair-corpus-honesty` task 8.1; until it lands, sast and vibe-py rows default to `advisory`/`seeded` per that design and only vibe-py seeds. - Guard classification is heuristic and closed; `none` is allowed and reported, never used to demote. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 8bd689c..630dcf3 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -51,6 +51,8 @@ Leave-one-out (corpus-seeded-mechanisms, 2026-09-05; the corpus as a teacher, ne | github | 6 | 0 | 6 | 0.000 | 0 | advisory tier | | vfc-js | 17 | 0 | 17 | 0.000 | 0 | advisory tier | +The loop is closed: `ousast mechanisms export` writes candidates, `ousast improve --pair-catalog` admits the ones that recover a holdout pair without leaking (per-profile clause over variant search; `mechanism_leak`/`mechanism_no_gain` revert byte for byte), the scan searches the admitted store and reports name the mechanism and its known fix. Measured 2026-09-05 on the `python-vulnerable` benchmark target with the vibe-py candidates: 4 records admitted in round 1 (`execute/1 parameter` taught by 7 pairs among them), human holdout profile pair_correct 0 -> 6 of 17, Youden 0 -> +0.353, round 2 `no_proposals`. + Measured 2026-09-05 after review tiers (Req 9) and pointer slices (Req 10): vibe-py 115 = 35 vendored human (13/35, Youden +37.1%) + 80 `agent` pointer pairs over the 40 LLM-generated Real-Vuln repositories (0/80, Youden 0.0; the LLM-repo baseline, scored by the nightly `pairs --slice vibe-py --pointers` only, never in tests or gates); agent-vfc 59 = 29 vendored + 30 pointer, tier `title`, 1/59, Youden 0.0, 2 unresolved labels; sast, vfc-js, vfc unchanged; detection/map/local-pair gate outputs byte-identical to the pre-spec commit. Only `seeded`/`reviewed` pairs gate the improve loop. Carry-forwards: `language_unsupported` loss counter and multi-line C/C++ declarator rescue (overlay-ir-completeness), `profiles_ungated` reporting and an offline test for the agent-vfc pointer builder (corpus-seeded-mechanisms). sast: java-sqli moved MISS→LEAK (coverage counted on both sides; the fixed twin leaks via parameter-as-source). vfc: recall fell to zero because every label now names a function and no C detection lands inside it; that is the honest baseline the engine specs measure against. @@ -82,7 +84,7 @@ sast: java-sqli moved MISS→LEAK (coverage counted on both sides; the fixed twi - [x] tree-sitter-overlay-extra -- Optional extra: tree-sitter + grammars, one CST walker, `quick` stays zero-dep. Dependencies: none - [x] real-world-vfc-slice -- Harvest reviewed function-level vuln/fix pairs from large projects (Firefox/gecko-dev, Chromium, OpenSSL, …) as an honesty slice. Seed is testable (`ousast pairs --slice vfc`). Dependencies: none. **Closed 2026-09-04, superseded by pair-corpus-honesty on the corpus/scorer plane.** - [x] pair-corpus-honesty -- Scorer counts coverage and matches by function; labels carry mechanism/provenance/split/known_limit; harvest anchors at the declarator with line-range and hunk modes; sast hygiene; new slices `vibe-py` (Real-Vuln-Benchmark), `vfc-js` (SecBench.js fix commits), `agent-vfc` (agent-trailer fixes); deterministic provenance fingerprint; per-profile improve acceptance; review tiers and pointer slices (amendment). Dependencies: none (supersedes real-world-vfc-slice). **Completed 2026-09-05 (11 review rounds).** -- [ ] corpus-seeded-mechanisms -- Every seeded or reviewed pair becomes a mechanism record (source shape, sink shape, guard the fix added); structural variant search over the scanned tree proposes matches as `variant` findings that the overlay and sandbox verify; leave-one-out recall on the pair corpus is the corpus's own detection rate; the improve loop gains a recall-raising `mechanisms` lever under the per-profile holdout gate. Principle: recall first under a false-positive ceiling, per profile. Dependencies: pair-corpus-honesty (tiers, pointer slices) +- [x] corpus-seeded-mechanisms -- Every seeded or reviewed pair becomes a mechanism record (source shape, sink shape, guard the fix added); structural variant search over the scanned tree proposes matches as `variant` findings that the overlay and sandbox verify; leave-one-out recall on the pair corpus is the corpus's own detection rate; the improve loop gains a recall-raising `mechanisms` lever under the per-profile holdout gate. Principle: recall first under a false-positive ceiling, per profile. Dependencies: pair-corpus-honesty (tiers, pointer slices). **Completed 2026-09-05 (6 review rounds).** - [ ] reachability-flow-model -- Name-resolved call graph over `FunctionIR`; per-function taint summaries composed along the graph into path records; graph reachability drives rank, map, worth-fixing; hunter gets `path_context` and hunter findings with a locatable sink become sandbox candidates; closed-kind fact oracle with candidate cache; `facts` improve lever under the per-profile gate; fix points ranked by reachable risk removed. Dependencies: pair-corpus-honesty - [ ] overlay-ir-completeness -- Shared CST walker completeness (C `init_declarator`, lvalue paths for field stores, per-grammar literal kinds, local ERROR-node tolerance, multi-line C/C++ declarators the walker names `` and `functions.py`'s one-line rescue cannot name: `chromium-cve-2022-2162`, `openssl-cve-2025-69421`; a Scorer-side multi-line-aware rescue is the cheaper alternative) plus Python parity with stdlib ast (f-string constness, augmented assignment, kwargs); injection-regime facts: container/transformer taint, `weak_literals` consulted, parameter sources off for injection. Measured on sast and vibe-py. Dependencies: pair-corpus-honesty - [ ] guard-dominance-regime -- Guard and Use IR records; guard dominance by nesting or early exit; memory sinks (memcpy/malloc/deref) with operand roles, nullable callees, out-param transformers, fixed-array capacity; same regime applied to web absence bugs (route auth guard, identity from auth context). Consumes path records. Measured on vfc, vfc-js, agent-vfc. Dependencies: reachability-flow-model, overlay-ir-completeness