From 14ca0a96ce381deeabd37a1da1cd6bf303a1f62c Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Tue, 8 Sep 2026 19:42:53 +0200 Subject: [PATCH] docs(contributor-scan): requirements, including the corpus asks I first missed 10 requirements, 46 criteria. A NEW spec rather than an amendment to model-grounded-detection, which is complete and closed. Req 1-8 cover the gap that prompted it: the model layer has zero production callers, so ousast scan must use it (1), regions must come from entry points rather than a labelled function (2), ranking and budget must work at repository scale where the enumerator's per-region cap of eight is meaningless (3), the performance envelope must be measured on a real checkout (4), output must be actionable (5), packaging must be built and verified rather than merely defined (6), the two deferred decisions must close (7), and the honesty disciplines carry forward (8). Req 9 and 10 were added after the maintainer asked whether their requests had been captured. They had not been -- I folded the corpus asks implicitly into "run against a known-vulnerable repository", which is not the same thing. Now explicit: a PHP/WordPress slice (php2cpg works once an interpreter is present, which the image provides, and plugin CVEs are overwhelmingly the families the model arbitrates well); real-world C library CVEs -- libpng, image libraries, OpenSSH, libpam, kernel modules -- gated on the memory decision, because until it closes they would join 173 pairs already idle; more vibe-code pairs, since that slice is where the stated audience works and it is only 35 pairs; and regression tuning over a pinned large-repository baseline, which is the thing pair scoring structurally cannot see. Req 9.4 carries today's lesson: verify on a sample that the model can READ a slice before vendoring it. The OWASP harvest found 238 pairs and vendored 40 for exactly that reason. Not approved. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG --- .kiro/specs/contributor-scan/requirements.md | 196 +++++++++++++++++++ .kiro/specs/contributor-scan/spec.json | 8 +- 2 files changed, 200 insertions(+), 4 deletions(-) create mode 100644 .kiro/specs/contributor-scan/requirements.md diff --git a/.kiro/specs/contributor-scan/requirements.md b/.kiro/specs/contributor-scan/requirements.md new file mode 100644 index 0000000..666b915 --- /dev/null +++ b/.kiro/specs/contributor-scan/requirements.md @@ -0,0 +1,196 @@ +# Requirements Document: contributor-scan + +## Introduction + +`model-grounded-detection` built an arbiter and measured it: 41.3% of the corpus reaches `model_entailed` +against the flat IR's 2.2%, and the wired pipeline scores 86.7% pair-correct at 6.7% leak against the LLM +hunter's 76.7% / 12.3%, at a fifth of the model calls. All 16 of its tasks are complete. + +**And the model layer has zero production callers.** `ousast scan` does not use it. Every number above came +from a measurement harness, not from the tool. This feature closes that gap and answers the question the +maintainer actually posed: *a real world tool that helps developers when they want to contribute.* + +That framing changes the work. A contributor has no labelled function and no declared weakness class — they +have a repository they are about to touch. Everything measured so far is function-level pairs, which +exercise none of the questions a whole repository asks: which regions to look at, how to rank thousands of +candidates, what to spend, and whether a CPG build on a 100k-line tree finishes at all. + +Context: `brief.md`, `.kiro/specs/model-grounded-detection/`, and the committed measurements +`2026-09-08-pipeline-vibe-py.json`, `2026-09-08-model-entailment-ceiling-cpg.json`. + +## Boundary Context + +- **In scope**: wiring `model/pipeline` into `ousast scan`; region selection from entry points; ranking and + budget at repository scale; the performance envelope; running against known-vulnerable checkouts; + contributor-facing output; the packaging already built (image, compose, shell wrappers, pyinfra); the + deferred memory-family decision; retiring `evidence_level`. +- **Out of scope**: new arbiters or families (the three forms exist); prompt optimisation of any kind; + building a sandbox (Req 10 of the predecessor stands — adopt, never build); changing the gates. +- **Carried forward, not re-litigated**: the evidence ladder, the taxonomy, the fact tables, and the finding + that the model supplies precision while the LLM supplies recall. + +## Requirements + +### Requirement 1: The model layer reaches a user + +**Objective:** As a developer, I want `ousast scan` to use the model layer, so that the arbiter this project +built is something I can run rather than something a harness demonstrates. + +#### Acceptance Criteria + +1. When a scan runs with a CPG engine available, the system shall arbitrate its candidate findings through + `model/pipeline` and attach a rung to each. +2. The system shall report model findings alongside the existing pattern and overlay findings, without + removing either. +3. When no CPG engine is available, the scan shall behave exactly as it does today and record + `cpg_unavailable` — the model layer is additive, never a precondition. +4. The detection, map and local pair gates shall remain byte-identical to the committed baseline. +5. The system shall expose the model layer's per-rung counts in the scan's own output, so a user can see how + much of the result the model arbitrated rather than inferring it. + +### Requirement 2: Regions come from the code, not from a label + +**Objective:** As a developer scanning a repository, I want the tool to decide what to look at, because I +cannot tell it which function is interesting. + +#### Acceptance Criteria + +1. The system shall select regions from entry points and reachability rather than from a declared function + name. +2. If a region has no resolvable entry point, then the system shall fall back to file-level regions and + record that it did. +3. The system shall route each region to the families its shape admits, rather than attempting every family + on every region. +4. The system shall not require a `function` label anywhere on the scan path; that label belongs to pair + scoring alone. + +### Requirement 3: Ranking and budget at repository scale + +**Objective:** As a developer, I want the most useful findings first and a bounded cost, because a repository +offers thousands of candidates and my attention and budget are finite. + +#### Acceptance Criteria + +1. The system shall order findings by rung first, so what the model established outranks what only the LLM + proposed. +2. The system shall apply a total budget across a scan for model calls, not merely the per-region cap of + eight the enumerator uses today. +3. When the budget is exhausted, the system shall stop asking and report how many candidates went unjudged, + rather than silently truncating. +4. The system shall spend the budget on the highest-ranked candidates first. +5. The system shall report cost and elapsed time for the run. + +### Requirement 4: It runs on a real repository + +**Objective:** As a maintainer, I want the performance envelope measured on real checkouts, because nothing +in the corpus exercises it. + +#### Acceptance Criteria + +1. The system shall be measured against at least one known-vulnerable repository checkout at a vulnerable + commit, not an excerpt. +2. The measurement shall record CPG build time, scan wall-clock, peak memory, findings by rung, and cost. +3. If a repository cannot be processed within its resource envelope, then the system shall record the reason + and the stage it failed at, rather than failing opaquely. +4. The measurement shall state whether the known vulnerability was found, and at which rung. +5. The system shall not require the whole repository to fit in one CPG if that proves infeasible; the + fallback shall be recorded and measured rather than assumed. + +### Requirement 5: Output a contributor can act on + +**Objective:** As a developer, I want to know what is risky and why, in terms I can check. + +#### Acceptance Criteria + +1. Every finding shall carry its rung, its witness, and the file and line it concerns. +2. The system shall state plainly what a `suspicion` means — that the model could not confirm it — so a + reader does not treat it as established. +3. The system shall write findings to the writable output location, never into the repository under analysis. +4. The system shall summarise a scan in a form readable without opening a JSON file. + +### Requirement 6: Packaging is part of the product + +**Objective:** As a developer with nothing installed, I want to run this, because an analyser I cannot start +is not a tool. + +#### Acceptance Criteria + +1. The system shall run from `docker compose` with no host dependency beyond Docker, and the image shall be + **built and verified**, not merely defined. +2. The shell wrappers shall make the container invisible for ordinary use, and shall prefer a native install + when one exists. +3. The repository under analysis shall be mounted read-only, and findings shall go to the one writable mount. +4. The network shall be off by default and required only for the optional LLM judge. +5. The engine version shall be pinned and checksum-verified in every install path. + +### Requirement 7: Close the two deferred decisions + +**Objective:** As a maintainer, I want the outstanding decisions closed rather than carried, because each one +silently caps what the corpus can tell us. + +#### Acceptance Criteria + +1. The system shall either reach the `memory` family through an adopted execution tier, or record explicitly + that C memory-safety is out of scope — 173 of the 176 real-world CVE pairs currently sit idle behind a + stub. +2. The system shall retire `evidence_level` in favour of `rung`, or record why both are kept. +3. Each decision shall be recorded with its reason where a later reader will find it. + +### Requirement 8: Honesty disciplines carry forward + +**Objective:** As a maintainer, I want the reporting disciplines that survived the predecessor to survive +this one, because each exists due to a specific past error. + +#### Acceptance Criteria + +1. The system shall report per-slice numbers with the real-world slice deciding, and refuse a bare aggregate. +2. The system shall publish the overfitting gap beside any headline number, and record it as absent rather + than zero when a slice is missing. +3. The system shall pass every prompt and persisted artifact through redaction. +4. The core install shall remain zero-dependency, with the CPG engine, the LLM endpoint and any execution + tier optional and degrading to a recorded reason. +5. Any new family-specific code path shall be checked for sibling paths that need the same change, before it + is measured — five bugs in the predecessor shared exactly that shape. + +### Requirement 9: Grow the corpus where the engine can use it + +**Objective:** As a maintainer, I want more real-world and more vibe-code bugs, so the engine is tuned against +what developers actually contribute to rather than against 30 pairs of one slice. + +#### Acceptance Criteria + +1. The system shall add a **PHP/WordPress** slice covering plugin and core weaknesses. WordPress plugin CVEs + are overwhelmingly injection, XSS and authorization bypass — the families the model layer already + arbitrates well — and `php2cpg` works once a PHP interpreter is present, which the shipped image now + provides. +2. The system shall add **real-world C library CVEs** beyond the existing curl and OpenSSL rows — libpng and + the image libraries, OpenSSH, libpam, and vulnerable kernel modules were named. These are gated on Req 7.1: + until the memory-family decision is closed they would join 173 pairs already sitting idle, so the decision + comes first and the harvest follows it. +3. The system shall add further **vibe-code** pairs — agent-written and human-written web application bugs — + because that slice is where the tool's stated audience actually works, and it is currently 35 pairs. +4. Before any slice is vendored at scale, the system shall verify on a sample that the model can **read** its + pairs, and shall vendor only the categories that a measurement shows it can arbitrate. The OWASP harvest + found 238 candidate pairs and vendored 40 for exactly this reason; the other 198 would have grown the + corpus while making every number less interpretable. +5. Every vendored pair shall carry a stated licence, and no vendored excerpt or test shall contain a + contiguous key-shaped literal. +6. Each new slice shall report as its own row and shall never be blended into a headline, and a synthetic + slice shall say in its catalog that it measures breadth rather than real-world performance. +7. The system shall record, per slice, how many pairs the model can arbitrate — so corpus growth is measured + by what it teaches rather than by how many rows it adds. + +### Requirement 10: Regression tuning over a large codebase + +**Objective:** As a maintainer, I want a repeatable run over a large real codebase, so changes to the fact +tables and queries can be checked for regressions that pair scoring cannot see. + +#### Acceptance Criteria + +1. The system shall support a repeatable scan over a pinned checkout of a large repository, recorded as a + committed baseline. +2. When the fact tables or queries change, the system shall report the delta against that baseline — + findings gained, findings lost, and rung movements. +3. The system shall treat an unexplained loss of a previously-entailed finding as a regression to be + investigated, not a number to be accepted. +4. The baseline shall record the engine version, because a CPG engine change moves every verdict it produces. diff --git a/.kiro/specs/contributor-scan/spec.json b/.kiro/specs/contributor-scan/spec.json index 6926345..32a7e9c 100644 --- a/.kiro/specs/contributor-scan/spec.json +++ b/.kiro/specs/contributor-scan/spec.json @@ -1,12 +1,12 @@ { "feature_name": "contributor-scan", "created_at": "2026-09-08T20:00:00Z", - "updated_at": "2026-09-08T20:00:00Z", + "updated_at": "2026-09-08T20:30:00Z", "language": "en", - "phase": "brief", + "phase": "requirements", "approvals": { "requirements": { - "generated": false, + "generated": true, "approved": false }, "design": { @@ -22,5 +22,5 @@ "depends_on": [ "model-grounded-detection" ], - "approval_note": "2026-09-08 brief drafted after the maintainer asked what the spec state was and whether anything was forgotten. It was: model-grounded-detection is 16/16 complete and its arbiter is measured at 86.7% pair-correct / 6.7% leak, but the model layer has ZERO production callers -- ousast scan does not use it, so the feature is complete and unshipped at once. That is a scope gap in the predecessor (it had no task saying 'make this reachable', and its byte-identical-gates requirement is satisfied precisely because nothing is wired). This spec covers scan integration, whole-repo scanning against known vulnerable repositories, the deferred memory-family decision, and records the pipeline, ops and corpus work delivered outside any spec. Not approved; requirements not generated." + "approval_note": "2026-09-08 brief drafted after the maintainer asked what the spec state was and whether anything was forgotten. It was: model-grounded-detection is 16/16 complete and its arbiter is measured at 86.7% pair-correct / 6.7% leak, but the model layer has ZERO production callers -- ousast scan does not use it, so the feature is complete and unshipped at once. That is a scope gap in the predecessor (it had no task saying 'make this reachable', and its byte-identical-gates requirement is satisfied precisely because nothing is wired). This spec covers scan integration, whole-repo scanning against known vulnerable repositories, the deferred memory-family decision, and records the pipeline, ops and corpus work delivered outside any spec. Not approved; requirements not generated. ; 2026-09-08 requirements generated at the maintainer's direction ('we follow the spec and build integration afterwards')." }