diff --git a/.kiro/specs/contributor-scan/requirements.md b/.kiro/specs/contributor-scan/requirements.md index 933cec2..95132a6 100644 --- a/.kiro/specs/contributor-scan/requirements.md +++ b/.kiro/specs/contributor-scan/requirements.md @@ -95,6 +95,10 @@ in the corpus exercises it. 4. The measurement shall state whether the known vulnerability was found, and at which rung. 5. The system shall not require the whole repository to fit in one CPG if that proves infeasible; the fallback shall be recorded and measured rather than assumed. +6. At least one pinned checkout shall carry a known vulnerability that a family can decide in principle, so + that a scan producing nothing is distinguishable from an engine that cannot see. *(Added 2026-09-09. + Criterion 4.1 is satisfied by libpng, whose CVE is out of family and which no version can fix, because + the library never calls a sink any family models: the letter was met and the intent was not.)* ### Requirement 5: Output a contributor can act on @@ -107,6 +111,17 @@ in the corpus exercises it. reader does not treat it as established. 3. The system shall write findings to the writable output location, never into the repository under analysis. 4. The system shall summarise a scan in a form readable without opening a JSON file. +5. The report shall be proportionate to the evidence behind a finding: a rule whose only evidence is a text + match shall be summarised once it exceeds a stated number of sites, and a finding a proposer reasoned + about shall keep its detail whatever its rung. Nothing shall be dropped from `findings.json`, which + remains the record a baseline is compared against. *(Added 2026-09-09. libpng emitted 198 findings, 163 of + them a pattern asserting `memcpy` and `strcpy` are PRESENT in a library that uses them correctly + throughout, and no criterion here forbade it.)* +6. Where a family's abstraction cannot decide a weakness class, the system shall state that limit in the + report, and no scan shall present an absence of findings as evidence of absence. *(Added 2026-09-09. + `c/memory` scored zero true positives across 89k lines of C for a structural reason — it models string + functions and libpng overflows through arithmetic. A silent scan reading as a clean bill of health is + worse than noise because it is quiet, and it is the one failure mode the rung ladder does not guard.)* ### Requirement 6: Packaging is part of the product @@ -197,55 +212,22 @@ tables and queries can be checked for regressions that pair scoring cannot see. --- -## Proposed amendments from group 2 (drafted 2026-09-09, NOT approved) +## Amendment record — group 2 (approved 2026-09-09) -Group 2 was specified as the phase that can reshape the rest, and it did. These are the changes it argues for -at the requirements level rather than the task level. They are recorded here unapproved; the tasks already -merged into group 2 implement what was within the approved requirements, and these are the parts that are not. +Group 2 was specified as the phase that can reshape the rest, and it did. Three amendments were drafted from +what it measured and **approved by the maintainer on 2026-09-09**; they are folded into the numbered +requirements above and marked there with the evidence that produced them. -### Amendment A — Req 5 should state what a report may repeat +| amendment | folded into | evidence | +| --- | --- | --- | +| A — what a report may repeat | Req 5.5 | libpng: 198 findings, 163 a pattern match; task 2.11 fixed that half | +| B — a family that cannot decide must say so | Req 5.6 | `c/memory`: 0 true positives on 89k lines of C | +| C — a checkout that can demonstrate detection | Req 4.6 | libpng satisfies 4.1 and cannot demonstrate detection | -**Why.** libpng emitted 198 findings, 163 of them a static pattern asserting that `memcpy` and `strcpy` are -*present* in a library that uses them correctly throughout. Req 5 says the output must be one a contributor -can act on; it does not say anything about volume, so nothing in the spec was violated by a report no -contributor would read. - -**Proposed criterion.** *The report shall be proportionate to the evidence behind a finding: a rule whose only -evidence is a text match shall be summarised once it exceeds a stated number of sites, and a finding a -proposer reasoned about shall keep its detail whatever its rung. Nothing shall be dropped from -`findings.json`, which remains the record a baseline is compared against.* - -Implemented in task 2.11 for pattern matches. **Not implemented, and the reason this is an amendment rather -than a closed task:** the same argument applies to the model layer's own suspicion band, which is 38 of vampi's -60 findings — an enumerated candidate that the graph could not decide and an LLM affirmed. That is the same -epistemic class as a pattern match with an opinion attached, and treating it as a reasoned claim is what -leaves 51 sections on a 520-line application. - -### Amendment B — a family that cannot decide a class shall say so - -**Why.** `c/memory` produced zero true positives across an 89k-line C library and three distinct classes of -false one. The reason is structural: it models string functions, and libpng overflows through arithmetic. A -silent scan currently reads as a clean bill of health, and the stated goal is that an LLM will help a -developer dive in — so it will fluently explain a silence it has no basis for. - -**Proposed criterion.** *Where a family's abstraction cannot decide a weakness class, the system shall state -the limit in the report rather than be silent about it, and no scan shall present the absence of findings as -evidence of absence.* - -This is worse than noise because it is quiet, and it is the one failure mode the rung ladder does not already -guard against. - -### Amendment C — Req 4 should require a checkout that can produce a true positive - -**Why.** Req 4.1 asks for "at least one known-vulnerable repository checkout at a vulnerable commit", and -libpng satisfies it while being incapable of demonstrating detection: its CVE is out of family, and no -version of it can be otherwise, because the library never calls a sink any family models. The requirement is -met and the intent is not. - -**Proposed criterion.** *At least one pinned checkout shall carry a known vulnerability that a family can -decide in principle, so that a scan producing nothing is distinguishable from an engine that cannot see.* - -vampi satisfies this for Python. Nothing satisfies it for C, and 2.15 exists to find out whether anything can. +**A is only half implemented, deliberately.** Task 2.11 made pattern matches proportionate. The same argument +applies to the model layer's own suspicion band — 38 of vampi's 60 findings — which is an enumerated +candidate the graph could not decide and an LLM affirmed: a pattern match with an opinion attached, not a +reasoned claim. Task 2.17 carries that half. ### What group 2 confirmed rather than changed diff --git a/.kiro/specs/contributor-scan/spec.json b/.kiro/specs/contributor-scan/spec.json index 81b0d59..4ec5260 100644 --- a/.kiro/specs/contributor-scan/spec.json +++ b/.kiro/specs/contributor-scan/spec.json @@ -1,7 +1,7 @@ { "feature_name": "contributor-scan", "created_at": "2026-09-08T20:00:00Z", - "updated_at": "2026-09-09T12:00:00Z", + "updated_at": "2026-09-09T12:30:00Z", "language": "en", "phase": "implementation", "approvals": { @@ -22,5 +22,5 @@ "depends_on": [ "model-grounded-detection" ], - "approval_note": "2026-09-08 brief drafted after the maintainer asked what the spec state was and whether anything was forgotten. It was: model-grounded-detection is 16/16 complete and its arbiter is measured at 86.7% pair-correct / 6.7% leak, but the model layer has ZERO production callers -- ousast scan does not use it, so the feature is complete and unshipped at once. That is a scope gap in the predecessor (it had no task saying 'make this reachable', and its byte-identical-gates requirement is satisfied precisely because nothing is wired). This spec covers scan integration, whole-repo scanning against known vulnerable repositories, the deferred memory-family decision, and records the pipeline, ops and corpus work delivered outside any spec. Not approved; requirements not generated. ; 2026-09-08 requirements generated at the maintainer's direction ('we follow the spec and build integration afterwards'). ; 2026-09-09 requirements approved and design generated ('let us continue with these'): the model layer joins as one stage inside MAP beside the obligations checker, one CPG per scan, regions from entry points, a total budget rather than a per-region cap. Phase 2 (the first run on a pinned real repository) can invalidate the rest and comes before shipping. ; 2026-09-09 tasks approved by the maintainer ('approve the tasks and start group 1'); implementing group 1 (regions, driver, MAP stage). ; 2026-09-09 group 2 complete except 2.8/2.9/2.13-2.16, and it reshaped the phase as intended. Nine engine defects found by measurement and none by review, gates byte-identical through every fix -- which says the pair corpus could not have caught any of them. Three requirements amendments are DRAFTED AND UNAPPROVED at the end of requirements.md: (A) Req 5 should state what a report may repeat, because libpng emitted 198 findings with nothing in the spec forbidding it, and the model layer's own suspicion band is still 38 of vampi's 60; (B) a family that cannot decide a class must say so, because c/memory scored zero true positives on 89k lines of C for a structural reason and a silent scan reads as a clean bill of health; (C) Req 4.1 is satisfied by a checkout that cannot demonstrate detection at all, so it should require one that can. Separately, two task groups appended to tasks.md without passing requirements or design have been LIFTED OUT to the new finding-feedback-loop spec, where they belong." + "approval_note": "2026-09-08 brief drafted after the maintainer asked what the spec state was and whether anything was forgotten. It was: model-grounded-detection is 16/16 complete and its arbiter is measured at 86.7% pair-correct / 6.7% leak, but the model layer has ZERO production callers -- ousast scan does not use it, so the feature is complete and unshipped at once. That is a scope gap in the predecessor (it had no task saying 'make this reachable', and its byte-identical-gates requirement is satisfied precisely because nothing is wired). This spec covers scan integration, whole-repo scanning against known vulnerable repositories, the deferred memory-family decision, and records the pipeline, ops and corpus work delivered outside any spec. Not approved; requirements not generated. ; 2026-09-08 requirements generated at the maintainer's direction ('we follow the spec and build integration afterwards'). ; 2026-09-09 requirements approved and design generated ('let us continue with these'): the model layer joins as one stage inside MAP beside the obligations checker, one CPG per scan, regions from entry points, a total budget rather than a per-region cap. Phase 2 (the first run on a pinned real repository) can invalidate the rest and comes before shipping. ; 2026-09-09 tasks approved by the maintainer ('approve the tasks and start group 1'); implementing group 1 (regions, driver, MAP stage). ; 2026-09-09 group 2 complete except 2.8/2.9/2.13-2.16, and it reshaped the phase as intended. Nine engine defects found by measurement and none by review, gates byte-identical through every fix -- which says the pair corpus could not have caught any of them. Three requirements amendments are DRAFTED AND UNAPPROVED at the end of requirements.md: (A) Req 5 should state what a report may repeat, because libpng emitted 198 findings with nothing in the spec forbidding it, and the model layer's own suspicion band is still 38 of vampi's 60; (B) a family that cannot decide a class must say so, because c/memory scored zero true positives on 89k lines of C for a structural reason and a silent scan reads as a clean bill of health; (C) Req 4.1 is satisfied by a checkout that cannot demonstrate detection at all, so it should require one that can. Separately, two task groups appended to tasks.md without passing requirements or design have been LIFTED OUT to the new finding-feedback-loop spec, where they belong. ; 2026-09-09 the three group-2 amendments APPROVED by the maintainer and folded into the numbered requirements: Req 5.5 (a report proportionate to the evidence behind a finding), Req 5.6 (a family that cannot decide a class states the limit, and no scan presents absence of findings as evidence of absence), Req 4.6 (at least one checkout carrying a vulnerability a family can decide in principle). Req 5.5 is half implemented by task 2.11 -- pattern matches -- and task 2.17 was added for the model layer's own suspicion band, which is 38 of vampi's 60 findings and is an enumerated candidate the graph could not decide with a judge's opinion attached rather than a reasoned claim." } diff --git a/.kiro/specs/contributor-scan/tasks.md b/.kiro/specs/contributor-scan/tasks.md index 07ea3c5..9d67296 100644 --- a/.kiro/specs/contributor-scan/tasks.md +++ b/.kiro/specs/contributor-scan/tasks.md @@ -307,6 +307,21 @@ - _Requirements: 3.1, 8.2_ - _Depends: 2.15_ +- [ ] 2.17 The model's own suspicion band is a pattern match with an opinion attached + - **Req 5.5's other half.** Task 2.11 made pattern matches proportionate and left this untouched, so vampi + still emits 61 sections for a 520-line application -- 38 of them model suspicions, roughly one per + fourteen lines of code. + - The epistemic case for treating them as reasoned claims does not survive inspection. A model suspicion is + what the enumerator proposed at a site the graph could NOT decide, affirmed by a judge with no coverage + there. That is a pattern match with an opinion attached, and it is weaker evidence than the obligations + checker's output, which comes from a structural analysis rather than a guess. + - So group them by family the way 2.11 groups pattern rules, and keep obligation findings and everything + at `model_corroborated` or above in full. Expect vampi to fall from 61 sections to roughly 23. + - Measure both repositories, and measure what it costs: if a real finding only ever appeared at model + suspicion, this hides it, and that is the number that decides whether the rule is right. + - _Requirements: 5.1, 5.2, 5.5_ + - _Depends: 2.11_ + ## Group 3 — Ship it - [ ] 3.1 Contributor output diff --git a/.kiro/specs/finding-feedback-loop/requirements.md b/.kiro/specs/finding-feedback-loop/requirements.md new file mode 100644 index 0000000..28bed51 --- /dev/null +++ b/.kiro/specs/finding-feedback-loop/requirements.md @@ -0,0 +1,136 @@ +# Requirements Document: finding-feedback-loop + +## Introduction + +Nine engine defects were found in one session against two repositories. **Three were the same bug in three +matchers** — a spec token matched as a substring rather than a word: `resolve` inside `resolveUrl` in the +sanitizer matcher, `user` inside `users` in the discharger matcher, `gets` inside `fgets` in the sink matcher. +Each was found by a measurement, fixed locally, and left no trace; the third survived until an 89k-line C +library made it visible as 26 identical false positives. + +Nothing in the system remembers that a finding was wrong. There is no artifact saying *this site, at this +rung, is not a defect, and here is why* — so a class recurs, and each recurrence costs another measurement. + +Two audiences want that artifact for different reasons. A maintainer wants the *propagation* automated: two of +the three matcher bugs needed no judgement at all, only the fix carried across. A developer wants to dismiss a +finding in their own repository, have it stay dismissed, and have the reason reviewable beside the code — a +tool that cannot be told it is wrong is one they stop running. + +This feature is bounded by a rule the project has held from the start and a leak that shows why. An optimiser +may evolve **the question** — the judge and residual prompts, which are how we ask. It may never author **the +facts** — tables, rulesets, the declared policy file, which are what we claim. On 2026-09-05 the improve lever +admitted candidate shapes because they "recover a currently missed holdout pair"; five of eleven were taught by +the holdout pairs they then recovered, and a holdout Youden of **−0.059 was reported as +0.059**. + +Context: `brief.md`, `.kiro/specs/contributor-scan/` (Req 8 on overfitting, Req 10 on baselines), the memory +`closed-loop-train-on-test-leak`, and the measurements +`2026-09-09-repo-vampi-authz-precision.json` and `2026-09-09-repo-libpng-bounds.json`. + +## Requirements + +### Requirement 1: A dismissal the developer owns + +**Objective:** As a developer, I want to record that a finding is not a defect in my repository, so it stays +recorded and my colleagues can see why. + +#### Acceptance Criteria + +1. The system shall read dismissals from a declarative file in the **target** repository, and shall never + write to it — the same rule the declared policy file already follows. +2. A dismissal shall carry the site it concerns, the rung it was dismissed at, a reason, and an author. +3. The system shall reject a dismissal that carries no reason, because a suppression list without reasons is + a graveyard nobody can audit. +4. The system shall report a dismissed finding as dismissed, with its reason, rather than omitting it — so a + stale dismissal is visible rather than silent. +5. When a dismissal no longer matches any finding, the system shall say so, because a dismissal that has + stopped applying is either a fixed bug or a moved target and both are worth knowing. + +### Requirement 2: A counter-example ledger the maintainer can group + +**Objective:** As a maintainer, I want every rejected finding recorded with what produced it, so a recurring +cause is visible as a cause rather than as five separate bugs. + +#### Acceptance Criteria + +1. When a finding is contradicted — by a dismissal, by the judge with coverage, or by a declared contract — + the system shall record the site, the rung, the witness, the spec tokens that produced it, and the commit. +2. Each ledger entry shall be runnable as a regression case: this site, at this commit, shall not be reported + at that rung. +3. The ledger's regression cases shall run offline in CI, so that re-introducing a fixed cause fails a test + rather than a repository scan four weeks later. +4. The system shall group ledger entries by what they share — spec token, matcher clause, family, witness + shape — and report the groups by size. +5. The ledger shall be committed evidence, not a local artifact. + +### Requirement 3: A model may propose a cause, and only propose it + +**Objective:** As a maintainer, I want the model to do the part that needs judgement — naming what a group of +counter-examples has in common — without it deciding anything. + +#### Acceptance Criteria + +1. Given a group of counter-examples, the system shall ask a model for a hypothesis: what these share, and + what change would remove all of them. +2. The model's output shall be written to a review queue, and shall never be written to a fact table, a + ruleset, or the declared policy file. +3. An adoption shall require a human commit, exactly as any other change to those files does. +4. The system shall be validated by reverting the three known matcher fixes and measuring how many of their + groups it recovers and what it proposes. A model that cannot rediscover a cause already known shall not be + trusted to propose one that is not. + +### Requirement 4: Dismissals are a labelled evaluation set + +**Objective:** As a maintainer, I want a repository's dismissals and its known vulnerabilities to form one +evaluation set, so an optimiser has both classes rather than only the negatives. + +#### Acceptance Criteria + +1. The system shall build a per-repository evaluation set from a `benchmarks/repos/` recipe's known + vulnerabilities and that repository's dismissals. +2. The evaluation set shall declare a train/holdout split. +3. The system shall not read the holdout during optimisation, and a test shall assert it. +4. The system shall report the size of both classes, because an evaluation set of negatives alone can only + teach silence. + +### Requirement 5: Evolution targets the question, never the facts + +**Objective:** As a maintainer, I want the prompts to improve against real rejections without the fact tables +ever being authored by a model. + +#### Acceptance Criteria + +1. The system shall support reflective evolution over the judge and residual questions only. +2. The system shall not modify sources, sinks, sanitizers, obligations, dischargers, rulesets or the declared + policy file by any automated means. +3. An evolved prompt shall be scored on the train split alone. +4. An evolved prompt shall be adopted only when it affirms fewer dismissed sites **and** loses no known + vulnerability. +5. The three gates shall stay byte-identical across an adoption, or the adoption is a regression wearing a + fix's clothes. + +### Requirement 6: An adoption reports what it cost + +**Objective:** As a maintainer, I want to see what an adoption traded, because a false-positive reducer that +quietly loses recall is worse than none. + +#### Acceptance Criteria + +1. Every adoption shall record precision and recall before and after, on the pinned repositories and on the + pair corpus. +2. Every adoption shall record which counter-examples it removed and which it did not. +3. A net-negative adoption shall be reverted automatically rather than argued about. +4. The system shall record what a target repository's evolved state learned and what it was measured against, + so per-repository tuning is legible rather than a black box. + +### Requirement 7: The loop inherits the corpus's blind spot + +**Objective:** As a maintainer, I want this loop measured on the evidence that can actually see its failures. + +#### Acceptance Criteria + +1. The evaluation set shall include whole repositories, not pairs alone: the pair corpus could not see any of + the nine defects that motivate this feature, and the gates stayed byte-identical through every fix. +2. The system shall state how many repositories its evidence base contains, because two — one of which has + never produced a true positive — is very little to tune against. +3. Where an adoption is measured on one repository only, the system shall label it as such rather than + reporting it as a general improvement. diff --git a/.kiro/specs/finding-feedback-loop/spec.json b/.kiro/specs/finding-feedback-loop/spec.json index 91296ee..5ff8639 100644 --- a/.kiro/specs/finding-feedback-loop/spec.json +++ b/.kiro/specs/finding-feedback-loop/spec.json @@ -1,15 +1,26 @@ { "feature_name": "finding-feedback-loop", "created_at": "2026-09-09T12:00:00Z", - "updated_at": "2026-09-09T12:00:00Z", + "updated_at": "2026-09-09T12:30:00Z", "language": "en", - "phase": "brief", + "phase": "requirements", "approvals": { - "requirements": { "generated": false, "approved": false }, - "design": { "generated": false, "approved": false }, - "tasks": { "generated": false, "approved": false } + "requirements": { + "generated": true, + "approved": false + }, + "design": { + "generated": false, + "approved": false + }, + "tasks": { + "generated": false, + "approved": false + } }, "ready_for_implementation": false, - "depends_on": ["contributor-scan"], - "approval_note": "2026-09-09 brief drafted. Scope lifted out of contributor-scan, where two task groups (a counter-example ledger and per-repository evolutionary prompting) had been appended directly to tasks.md without passing requirements or design -- a workflow error, and they are a separate feature rather than a refinement of scan integration. Prompted by the maintainer asking for GEPA/SIMBA-style evolution with declarative persistent per-repository state so developers can commit dismissals and evolve the harness. The design boundary is that an optimiser may evolve the QUESTION (judge and residual prompts) and never the FACTS (tables, rulesets, declared policy), because the closed-loop leak measured 2026-09-05 turned a -0.059 holdout Youden into +0.059 by teaching from the pairs it then recovered. Not approved; requirements not generated." + "depends_on": [ + "contributor-scan" + ], + "approval_note": "2026-09-09 brief drafted. Scope lifted out of contributor-scan, where two task groups (a counter-example ledger and per-repository evolutionary prompting) had been appended directly to tasks.md without passing requirements or design -- a workflow error, and they are a separate feature rather than a refinement of scan integration. Prompted by the maintainer asking for GEPA/SIMBA-style evolution with declarative persistent per-repository state so developers can commit dismissals and evolve the harness. The design boundary is that an optimiser may evolve the QUESTION (judge and residual prompts) and never the FACTS (tables, rulesets, declared policy), because the closed-loop leak measured 2026-09-05 turned a -0.059 holdout Youden into +0.059 by teaching from the pairs it then recovered. Not approved; requirements not generated. ; 2026-09-09 requirements generated at the maintainer's direction ('approve the amendments and generate the requirements for finding-feedback-loop'). Seven requirements: a dismissal file the developer owns and the tool never writes; a counter-example ledger whose entries are runnable regression cases grouped by shared cause; a model that proposes a cause into a review queue and decides nothing, validated by reverting the three known matcher fixes; dismissals plus recipe vulnerabilities as one split evaluation set with the holdout unread; evolution scoped to the judge and residual questions with the fact tables off limits by any automated means; an adoption that reports what it traded and reverts itself when net-negative; and the loop inheriting the corpus blind spot, so its evidence base is repositories and it says how few of them there are. Not approved." }