diff --git a/.kiro/specs/contributor-scan/design.md b/.kiro/specs/contributor-scan/design.md new file mode 100644 index 0000000..f57149d --- /dev/null +++ b/.kiro/specs/contributor-scan/design.md @@ -0,0 +1,219 @@ +# Design Document: contributor-scan + +## Overview + +`model-grounded-detection` built an arbiter and measured it at 90.0% pair-correct / 6.7% leak on vibe-py. It +has **zero production callers**. This feature makes it reachable, and makes it work on a repository rather +than on a labelled function. + +The shape is deliberately conservative: the model layer joins the existing scan as **one more stage inside +MAP**, beside the obligations checker it most resembles, and adds findings rather than replacing any. The +pattern and overlay paths keep working exactly as they do today, the gates stay byte-identical, and a machine +without a CPG engine scans exactly as it does now. Nothing in this design is allowed to make the tool worse +for someone who does not have Joern. + +### Goals + +- A finding with a rung reaches a user through `ousast scan`. +- Regions come from entry points, not from a label a contributor cannot supply. +- Ranking and budget hold at repository scale. +- The performance envelope is measured on a real checkout, not asserted. + +### Non-Goals + +- New arbiters or families — the three forms exist and are measured. +- Any optimiser, prompt search, or K-run averaging. The predecessor deleted that machinery; this does not + reintroduce it by another name. +- Building a sandbox. Req 10 of the predecessor stands: adopt, never build. +- Changing the gates, the taxonomy, or the fact-table schema. + +## Boundary Commitments + +### This Spec Owns + +- `model/regions.py` — entry points and file fallbacks to regions, with the family routing for each. +- `model/scan.py` — the repository-level driver: rank, budget, dispatch to `model/pipeline`, collect. +- The `Stage.MAP` wiring in `cli.py` and the scan's model-layer payload and report section. +- `benchmarks/repos/` — pinned known-vulnerable checkouts and their baselines. +- The decisions in Req 7 and their record. + +### Out of Boundary + +- `model/pipeline`, `taint`, `dominance`, `config_value`, `ladder`, `specs`, `candidates` — built, measured, + used unchanged. If this feature needs to change one, that is a signal the change belongs in its own commit + with its own measurement. +- `mapping.analyze_entry_points`, the overlay, the obligations checker, the gates — untouched. + +### Allowed Dependencies + +- Core: standard library. `dependencies = []` unchanged. +- Optional and capability-detected, each degrading to a recorded reason: the CPG engine (Joern), the LLM + endpoint, any execution tier. + +### Revalidation Triggers + +- Any gate output moving (Req 1.4). +- A scan on a machine without Joern behaving differently from today (Req 1.3). +- The repository baseline losing a previously-entailed finding (Req 10.3). + +## Architecture + +### Existing Architecture Analysis + +`ousast scan` runs `policy_load → ruleset_load → preprocess → entry_point_mapping → rank → calibrate → +STATIC → MAP → REGRESS → report`. Inside MAP it already builds a complexity map, runs the overlay, and — the +closest analogue to what we are adding — runs `_check_obligations(root, targets, entries, facts, settings, +runtime, hunter_model)`, which takes entry points, produces findings, and merges them by re-ranking. + +The model layer slots in exactly there, with the same shape and the same failure discipline. `entry_points` +is already computed at line ~249 and carries what region selection needs: `path`, `line`, `end_line`, +`function_name`, `access_level`, `trust_boundary`. + +### Boundary Map + +```mermaid +flowchart TD + EP[entry_point_mapping
existing, untouched] --> REG[model/regions
regions + families] + REG --> RANK[model/scan
rank, budget] + RANK --> CPG[cpg.build once per repo] + CPG --> PIPE[model/pipeline.scan_region
existing, unchanged] + PIPE --> F[findings with rungs] + F --> MERGE[merge beside pattern/overlay/obligation findings] + CPG -. absent .-> SKIP[stage skipped, cpg_unavailable recorded] + RANK -. budget spent .-> REPORT[unjudged count reported] +``` + +### Key Decisions + +| Decision | Choice | Why | +|---|---|---| +| Where it runs | a stage inside `Stage.MAP` | it needs entry points, which MAP has; it mirrors the obligations checker exactly | +| CPG lifetime | **one build per scan**, reused across regions | a build is 5–30s on an excerpt and the dominant cost at repo scale; per-region builds would make this unusable | +| Region source | entry points, file fallback | Req 2; a contributor has no labelled function | +| Family routing | by region shape and language, from the specs that exist | Req 2.3; asking every family of every region is the unbounded version | +| Budget | one total per scan, spent highest-rank-first | Req 3.2; the enumerator's per-region cap of 8 is meaningless across thousands of regions | +| Ordering | rung first, then rank | Req 3.1; what the model established outranks what the LLM proposed | +| Merge | additive | Req 1.2; removing an existing finding is a product change nobody asked for | + +## File Structure Plan + +### New + +``` +src/openultrasast/model/ +├── regions.py # EntryPointRecord|FileTarget -> Region + the families that region admits +└── scan.py # repo driver: rank regions, hold the budget, drive pipeline, collect findings + +benchmarks/repos/ +├── README.md # what a pinned checkout is for and how a baseline is regenerated +└── .toml # pinned url + commit + the known CVE and where it lives +``` + +### Changed + +- `cli.py` — a `model` stage inside MAP, mirroring `_check_obligations`; its payload into the manifest and a + report section. +- `model/report.py` — a per-rung summary block for a scan (it currently reports per-slice for corpus runs). +- `config.py` — a `ModelConfig`-style block: enabled, budget, max regions. Zero-dep, defaults conservative. + +### Unchanged and load-bearing + +`model/pipeline`, `taint`, `dominance`, `config_value`, `specs`, `candidates`, `ladder`; `mapping`; the three +gates; `redaction`. + +## Components and Interfaces + +### model/regions.py + +```python +@dataclass(frozen=True) +class ScanRegion: + """Where to look, and what to look for there.""" + path: str + function: str | None # None for a file-level fallback region + families: tuple[str, ...] # only the families this region's shape and language admit + rank: float # from the entry point's access level and trust boundary + source: str # "entry_point" | "file_fallback" + +def regions_for(entries, targets, *, taxonomy, language_of) -> tuple[ScanRegion, ...] +``` + +Ranking uses what `EntryPointRecord` already carries — an externally-reachable entry point at a trust +boundary outranks an internal one — rather than inventing a new signal. + +### model/scan.py + +```python +@dataclass(frozen=True) +class ScanBudget: + """A total for the run, not a per-region cap.""" + max_model_calls: int = 200 + max_regions: int = 500 + +@dataclass(frozen=True) +class ModelScanResult: + findings: tuple[ModelFinding, ...] + by_rung: Mapping[str, int] + regions_scanned: int + regions_unjudged: int # Req 3.3: reported, never silently truncated + model_calls: int + cost_usd: float + degradations: tuple[Mapping[str, object], ...] + +def scan_repository(root, regions, *, backend, specs, client, model, budget) -> ModelScanResult +``` + +One CPG for the run; regions consumed in rank order; the budget decremented per model call and checked before +each; everything past exhaustion counted into `regions_unjudged`. + +## Error Handling + +| Condition | Behaviour | Recorded | +|---|---|---| +| No CPG engine | stage skipped, scan otherwise identical to today | `cpg_unavailable` | +| CPG build fails on the repo | stage skipped, other stages proceed | `cpg_build_failed` + stage | +| A region's query times out | that region yields nothing, the scan continues | `cpg_query_timeout` | +| Budget exhausted | stop asking; count the rest | `budget_exhausted` + `regions_unjudged` | +| No LLM endpoint | entailment still reported; suspicion band unasked | `learning_endpoint_unavailable` | + +## Requirements Traceability + +| Criteria | Delivered by | +|---|---| +| 1.1, 1.2 | the `model` stage in MAP; findings merged additively | +| 1.3 | capability probe; stage skipped with `cpg_unavailable` | +| 1.4 | the committed gate baseline, re-run before and after | +| 1.5 | `ModelScanResult.by_rung` into the manifest and report | +| 2.1, 2.2 | `regions_for` over `entry_points`, file fallback recorded in `ScanRegion.source` | +| 2.3 | `ScanRegion.families` from region shape and language | +| 2.4 | no `function` label on the scan path; `ScanRegion.function` may be None | +| 3.1 | ordering by rung, then rank | +| 3.2, 3.4 | `ScanBudget` total, regions consumed in rank order | +| 3.3 | `regions_unjudged` + `budget_exhausted` | +| 3.5 | `cost_usd`, elapsed in the manifest | +| 4.1–4.5 | `benchmarks/repos/` pinned checkouts; a measurement recording build time, wall-clock, peak memory, rungs, cost, and whether the known CVE was found | +| 5.1, 5.2 | rung + witness + path:line on every finding; the report states what `suspicion` means | +| 5.3 | findings to the writable output, never the analysed tree | +| 5.4 | `model/report.py` scan summary | +| 6.1–6.5 | the image built and verified; wrappers; read-only mount; network off; pinned + checksummed engine | +| 7.1, 7.2, 7.3 | the two decisions recorded in this spec's notes with their reasons | +| 8.1, 8.2 | `model/report.py` per-slice + overfitting gap, already built | +| 8.3, 8.4 | redaction on every prompt; capability detection throughout | +| 8.5 | the sibling-audit step in task 1.1 | +| 9.1–9.7 | the corpus tasks, each gated on a read-check before vendoring | +| 10.1–10.4 | `benchmarks/repos/` baselines, delta reporting, engine version recorded | + +## Migration and Phases + +1. **Wire it (Req 1, 2).** Regions, the MAP stage, findings with rungs reaching a report. Gate-protected. + Nothing else matters until a user can see a rung. +2. **Scale it (Req 3, 4).** Budget and ranking, then the first run against a pinned known-vulnerable + checkout — the measurement that has never been taken and can reshape the rest. +3. **Ship it (Req 5, 6).** Contributor output; build and verify the image that is currently only defined. +4. **Close the deferred (Req 7).** The memory decision; retire `evidence_level`. +5. **Grow the corpus (Req 9).** PHP/WordPress first — the families the model arbitrates best. C CVEs only + after 7.1. +6. **Regression baselines (Req 10).** + +Phase 2 is the one that can invalidate the rest: if a CPG build on a real repository is infeasible, the +region/budget design changes shape and it is better to learn that before phases 3–6 are built on it. diff --git a/.kiro/specs/contributor-scan/spec.json b/.kiro/specs/contributor-scan/spec.json index 32a7e9c..7cac816 100644 --- a/.kiro/specs/contributor-scan/spec.json +++ b/.kiro/specs/contributor-scan/spec.json @@ -1,20 +1,20 @@ { "feature_name": "contributor-scan", "created_at": "2026-09-08T20:00:00Z", - "updated_at": "2026-09-08T20:30:00Z", + "updated_at": "2026-09-09T09:15:00Z", "language": "en", - "phase": "requirements", + "phase": "tasks", "approvals": { "requirements": { "generated": true, - "approved": false + "approved": true }, "design": { - "generated": false, - "approved": false + "generated": true, + "approved": true }, "tasks": { - "generated": false, + "generated": true, "approved": false } }, @@ -22,5 +22,5 @@ "depends_on": [ "model-grounded-detection" ], - "approval_note": "2026-09-08 brief drafted after the maintainer asked what the spec state was and whether anything was forgotten. It was: model-grounded-detection is 16/16 complete and its arbiter is measured at 86.7% pair-correct / 6.7% leak, but the model layer has ZERO production callers -- ousast scan does not use it, so the feature is complete and unshipped at once. That is a scope gap in the predecessor (it had no task saying 'make this reachable', and its byte-identical-gates requirement is satisfied precisely because nothing is wired). This spec covers scan integration, whole-repo scanning against known vulnerable repositories, the deferred memory-family decision, and records the pipeline, ops and corpus work delivered outside any spec. Not approved; requirements not generated. ; 2026-09-08 requirements generated at the maintainer's direction ('we follow the spec and build integration afterwards')." + "approval_note": "2026-09-08 brief drafted after the maintainer asked what the spec state was and whether anything was forgotten. It was: model-grounded-detection is 16/16 complete and its arbiter is measured at 86.7% pair-correct / 6.7% leak, but the model layer has ZERO production callers -- ousast scan does not use it, so the feature is complete and unshipped at once. That is a scope gap in the predecessor (it had no task saying 'make this reachable', and its byte-identical-gates requirement is satisfied precisely because nothing is wired). This spec covers scan integration, whole-repo scanning against known vulnerable repositories, the deferred memory-family decision, and records the pipeline, ops and corpus work delivered outside any spec. Not approved; requirements not generated. ; 2026-09-08 requirements generated at the maintainer's direction ('we follow the spec and build integration afterwards'). ; 2026-09-09 requirements approved and design generated ('let us continue with these'): the model layer joins as one stage inside MAP beside the obligations checker, one CPG per scan, regions from entry points, a total budget rather than a per-region cap. Phase 2 (the first run on a pinned real repository) can invalidate the rest and comes before shipping." } diff --git a/.kiro/specs/contributor-scan/tasks.md b/.kiro/specs/contributor-scan/tasks.md new file mode 100644 index 0000000..fca74e1 --- /dev/null +++ b/.kiro/specs/contributor-scan/tasks.md @@ -0,0 +1,157 @@ +# Implementation Plan: contributor-scan + +## Guidance for the implementing agent + +- **The predecessor's lesson, in one line.** Five bugs in `model-grounded-detection` shared one shape: a + taint-specific assumption applied to every family, each caught by a *measurement* rather than a check. + Before you finish any change to a routing rule, a matching rule, or a per-family question, **grep for its + siblings and fix them in the same commit.** The most expensive instance cost a 19-minute run that returned + an identical number, and only the identity revealed the bug. +- **Run the whole CI list before pushing**, not the parts you remember: `ruff format --check .`, + `ruff check .`, `mypy src/openultrasast`, `pytest`, the three gates, `compileall src tests`, and + `pytest -m semantic`. CI was red for most of a session because `ruff check` passing was taken as evidence + that `ruff format --check` would. +- **The gate baseline is the safety net.** `benchmarks/measurements/2026-09-06-gate-baseline.txt`. Run all + three gates before and after every task. The model layer is additive: if a gate moves, you changed + something that was never meant to change. +- **RED first**, one commit per task, selective `git add`. +- **Zero-dependency core.** Copy the capability-probe pattern in `semantic/extra.py` and `cpg/capability.py`. +- **Do not modify `model/pipeline`, `taint`, `dominance`, `config_value`, `specs` or `candidates`.** They are + built and measured. If a change looks necessary, it belongs in its own commit with its own measurement. + +## Group 1 — Wire it (gate-protected) + +- [ ] 1.1 Regions from entry points + - `model/regions.py`: `ScanRegion(path, function, families, rank, source)` and + `regions_for(entries, targets, *, taxonomy, language_of)`. Rank from what `EntryPointRecord` already + carries (`access_level`, `trust_boundary`) — do not invent a new signal. File-level fallback when a target + has no entry point, with `source="file_fallback"`. Families from region shape and language, via the + existing `taint_specs`/`dominance_specs`/`config_specs`. + - **Sibling audit (Req 8.5):** family routing now exists in three places — `ceiling.py`'s harness, + `pipeline._arbitrate`, and here. Check all three agree before finishing. + - Observable: a region is produced per entry point; a target with no entry point yields a file region marked + as such; no region carries a family whose spec is absent for its language. + - _Requirements: 2.1, 2.2, 2.3, 2.4, 8.5_ + +- [ ] 1.2 The repository driver + - `model/scan.py`: `ScanBudget`, `ModelScanResult`, `scan_repository(...)`. **One CPG per run**, reused + across regions. Regions in rank order; budget decremented per model call and checked before each; the + remainder counted into `regions_unjudged`. Findings ordered by rung, then rank. + - Observable: one `backend.build` call for many regions; a budget of N yields at most N model calls and a + non-zero `regions_unjudged`; ordering puts `model_entailed` above `suspicion`; a failed CPG build yields + an empty result with `cpg_build_failed`, never an exception. + - Redaction (Req 8.3) is inherited from `model/pipeline`, which redacts every prompt; a test asserts the + driver adds no unredacted path of its own. + - _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 8.3_ + - _Depends: 1.1_ + +- [ ] 1.3 The MAP stage + - Wire `scan_repository` into `cli.py` inside `Stage.MAP`, mirroring `_check_obligations`: takes + `entry_points` and `targets`, returns findings merged **additively** beside the pattern, overlay and + obligation findings. A `[model]` config block (enabled, budget, max regions), zero-dep, conservative + defaults. Payload into the manifest with `by_rung`, cost and elapsed. + - Observable: a scan on a repo with Joern present reports findings carrying rungs; the same scan with + `OPENULTRASAST_JOERN_PROBE=off` is identical to today's output plus a `cpg_unavailable` degradation; + **the three gates are byte-identical**. + - The `[model]` block is capability-detected like every other optional plane (Req 8.4): core stays + `dependencies = []`, and the engine, endpoint and execution tier each degrade to a recorded reason. + - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 8.4_ + - _Depends: 1.2_ + +## Group 2 — Scale it (the phase that can reshape the rest) + +- [ ] 2.1 A pinned known-vulnerable checkout + - `benchmarks/repos/.toml`: url, commit, the known CVE, the file and function it lives in, licence. + Start with one small enough to iterate on. Fetch is explicit and offline-by-default, like the pointer + pairs. + - Observable: the recipe resolves to a checkout; CI never fetches it. + - _Requirements: 4.1_ + - _Depends: 1.3_ + +- [ ] 2.2 The first repository measurement + - Run the full scan against that checkout and commit the artifact: CPG build time, scan wall-clock, peak + memory, findings by rung, cost, `regions_unjudged`, and **whether the known CVE was found and at which + rung**. If the repository cannot be processed in its envelope, record the stage it failed at. + - This is the go/no-go for the shape of the rest: if one CPG per repository is infeasible, the region and + budget design changes and phases 3–6 should not be built on it first. + - Observable: the artifact exists with every field; the reading states go or no-go and why. + - _Requirements: 4.2, 4.3, 4.4, 4.5_ + - _Depends: 2.1_ + +## Group 3 — Ship it + +- [ ] 3.1 Contributor output + - `model/report.py`: a scan summary — counts per rung, the top findings with witness and `path:line`, and a + plain statement of what `suspicion` means (the model could not confirm it). Findings to the writable + output only. + - Observable: a scan produces a summary readable without opening JSON; every finding shows rung, witness + and location; nothing is written into the analysed tree. + - Keeps the disciplines already built into `model/report.py` (Req 8.1, 8.2): per-slice rows with the + real-world slice deciding, a refused bare aggregate, and the overfitting gap published beside any headline + — absent rather than zero when a slice is missing. + - _Requirements: 5.1, 5.2, 5.3, 5.4, 8.1, 8.2_ + - _Depends: 1.3_ + +- [ ] 3.2 Build and verify the image + - `docker compose build`, then run a real scan through it and through the shell wrapper. The Dockerfile and + compose exist and are syntax-valid; **the image has never been built**, so nothing about it is proven. + - Observable: the image builds; `TARGET= docker compose run --rm ousast scan /target` produces the + same findings as a native run; the target mount is read-only; the network is off by default. + - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_ + - _Depends: 3.1_ + +## Group 4 — Close the deferred decisions + +- [ ] 4.1 The memory-family decision + - Either reach `memory` through an adopted execution tier, or record explicitly that C memory-safety is out + of scope. 173 of the 176 real-world CVE pairs sit idle behind a stub; the decision gates Req 9.2. + - Observable: the decision and its reason are recorded where a later reader finds them; if out of scope, + the corpus says so per slice. + - _Requirements: 7.1, 7.3_ + +- [ ] 4.2 Retire `evidence_level` + - It duplicates `rung` in intent. Retire it, or record why both are kept. Note the legacy value + `static_corroboration` comes from the flat-IR overlay, which Req 4.4 of the predecessor demotes to the + suspicion band — so a straight rename would overstate it. + - Observable: one vocabulary, or a recorded reason for two; SARIF and JSON consumers considered. + - _Requirements: 7.2, 7.3_ + +## Group 5 — Grow the corpus + +- [ ] 5.1 PHP/WordPress slice + - Plugin and core weaknesses. `php2cpg` needs a PHP interpreter, which the image provides. **Validate on a + sample that the model can read the pairs before vendoring** — the OWASP harvest found 238 and vendored 40 + for exactly this reason. + - Observable: a sample measurement precedes vendoring; only categories the model can arbitrate are + vendored; the slice reports as its own row; every pair states a licence. + - _Requirements: 9.1, 9.4, 9.5, 9.6, 9.7_ + - _Depends: 3.2_ + +- [ ] 5.2 More vibe-code pairs + - The slice where the stated audience works, currently 35 pairs. Same read-check discipline. + - _Requirements: 9.3, 9.4, 9.5, 9.6, 9.7_ + +- [ ] 5.3 Real-world C library CVEs + - libpng and the image libraries, OpenSSH, libpam, kernel modules. **Gated on 4.1**: until the memory + decision closes these would join 173 idle pairs. + - _Requirements: 9.2, 9.4, 9.5, 9.6_ + - _Depends: 4.1_ + +## Group 6 — Regression baselines + +- [ ] 6.1 Repository baseline and delta + - A committed baseline per pinned checkout; a delta report on fact-table or query changes showing findings + gained, lost and rung movements. An unexplained loss of a previously-entailed finding is a regression to + investigate, not a number to accept. The baseline records the **engine version**, because a CPG engine + change moves every verdict. + - _Requirements: 10.1, 10.2, 10.3, 10.4_ + - _Depends: 2.2_ + +## Task-plan review notes + +- Group 1 must land and pass review before group 2: measuring an unwired pipeline is what produced a + session's worth of harness-only numbers. +- **Task 2.2 is a go/no-go.** If one CPG per repository is infeasible, stop and redesign the region/budget + model rather than building groups 3–6 on an assumption. +- Group 5 is deliberately last. The corpus is not the binding constraint today — reachability is — and 198 of + 238 OWASP pairs were left unvendored precisely because volume without readability makes every number worse.