marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-19 21:19:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: plain-English pass over the active specs

Browse Source 
Ran the humanizer rules over 79 prose files in the active specs (brief,
design, requirements, research and narrative docs). Removed em dashes,
significance inflation, forced rule-of-three, bold-label list items and
filler. Skipped tasks.md checklists and the retired specs (harnessx-*,
openrouter-sast-harness, unseen-repo-evaluation).

Every number, path, command, hash and identifier was preserved: checked
each file against HEAD for drift in numbers, inline code and code fences,
zero drift across all 79 files.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-12 18:16:41 +02:00
co-authored by Claude Opus 4.8
parent b08f372022
commit 39a1873e5b
79 changed files with 1103 additions and 1103 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/real-world-vfc-slice/brief.md
+9 -9
View File
@@ -2,11 +2,11 @@
## Problem
Operators care whether the harness fires on a real buggy snapshot and stays quiet after the patch — Firefox, Chromium, OpenSSL — not whether OWASP BenchmarkTest00074 is a TP. The labeled SAST slice is 11 textbook files. Overlay Youden there is **worse** than inventory (+9.09% → 0%). We cannot tell if the tool got better on messy product code.
Operators care whether the harness fires on a real buggy snapshot and stays quiet after the patch (Firefox, Chromium, OpenSSL), not whether OWASP BenchmarkTest00074 is a TP. The labeled SAST slice is 11 textbook files. Overlay Youden there is **worse** than inventory (+9.09% → 0%). We cannot tell if the tool got better on messy product code.
## Current State
Slices today: `local` (CI 100%), `github` (6 vendored functions: SVEN, hutool, node-serialize), `sast` (OWASP/Juliet). `datasets.toml` already points at CVEfixes, MoreFixes, CWE-Bench-Java, Vul4J. Nothing is harvested from gecko-dev / Firefox. Whole mozilla-central is not in-tree (and should not be). FixFox (Zenodo) is embargoed until 2030. Firefox security commit messages are often scrubbed of “security” words, so grep-the-log is a bad miner.
Slices today: `local` (CI 100%), `github` (6 vendored functions: SVEN, hutool, node-serialize), `sast` (OWASP/Juliet). `datasets.toml` already points at CVEfixes, MoreFixes, CWE-Bench-Java, Vul4J. Nothing is harvested from gecko-dev / Firefox. Whole mozilla-central is not in-tree (and should not be). FixFox (Zenodo) is embargoed until 2030. Firefox security commit messages are often scrubbed of "security" words, so grep-the-log is a bad miner.
Measured on the current 11-pair **sast** slice (2026-09-04):
@@ -27,8 +27,8 @@ Do not clone Firefox. Filter public VFC databases for `gecko-dev` / `mozilla-fir
## Scope
- **In**: harvest script or documented procedure; catalog slice; provenance headers; overlay-vs-inventory scoreboard on that slice; dataset.toml pointers for Firefox/Chromium.
- **Out**: Vendoring mozilla-central; live Bugzilla scraping; making Youden a merge gate; buffer-size semantics; tree-sitter packaging (owned by `tree-sitter-overlay-extra`).
- In: harvest script or documented procedure; catalog slice; provenance headers; overlay-vs-inventory scoreboard on that slice; dataset.toml pointers for Firefox/Chromium.
- Out: Vendoring mozilla-central; live Bugzilla scraping; making Youden a merge gate; buffer-size semantics; tree-sitter packaging (owned by `tree-sitter-overlay-extra`).
## Boundary Candidates
@@ -41,17 +41,17 @@ Do not clone Firefox. Filter public VFC databases for `gecko-dev` / `mozilla-fir
## Upstream / Downstream
- **Upstream**: `pairs.py` catalog format; `propose-adjudicate-prove` overlay scoring
- **Downstream**: improve-loop signals from real misses; later memory-safety spec can reuse the same slice
- Upstream: `pairs.py` catalog format; `propose-adjudicate-prove` overlay scoring
- Downstream: improve-loop signals from real misses; later memory-safety spec can reuse the same slice
## Existing Spec Touchpoints
- **Extends**: pair catalog only (new slice). Does not change smoke gate.
- **Adjacent**: `tree-sitter-overlay-extra` (C/Java overlay on this slice once the extra exists)
- Extends: pair catalog only (new slice). Does not change smoke gate.
- Adjacent: `tree-sitter-overlay-extra` (C/Java overlay on this slice once the extra exists)
## Constraints
Isolated functions, same `relpath` vuln/fixed. License in the header. No 16 GB dumps in git. Firefox pairs only when a public git commit pair exists — MFSA text without a commit is not a pair.
Isolated functions, same `relpath` vuln/fixed. License in the header. No 16 GB dumps in git. Firefox pairs only when a public git commit pair exists; MFSA text without a commit is not a pair.
## Closure (2026-09-04)
.kiro/specs/real-world-vfc-slice/design.md
+8 -8
View File
@@ -132,12 +132,12 @@ benchmarks/pairs/
### Modified Files
- `src/openultrasast/pairs.py` — concatenate vfc catalog; overlay path for `vfc`; `scorers` on the eval result; `DEFAULT_VFC_CATALOG`.
- `src/openultrasast/cli.py` — `--slice` choice `vfc`.
- `src/openultrasast/pair_gate.py` — comment that vfc is honesty-only; fail condition stays local.
- `tests/test_pair_corpus.py` — vfc load, provenance, CLI, not in `LANGUAGE_MANIFESTS`, pair_gate still passes.
- `benchmarks/pairs/datasets.toml` — project pointers.
- `benchmarks/pairs/README.md` — slice list and harvest pointer.
- `src/openultrasast/pairs.py`: concatenate vfc catalog; overlay path for `vfc`; `scorers` on the eval result; `DEFAULT_VFC_CATALOG`.
- `src/openultrasast/cli.py`: `--slice` choice `vfc`.
- `src/openultrasast/pair_gate.py`: comment that vfc is honesty-only; fail condition stays local.
- `tests/test_pair_corpus.py`: vfc load, provenance, CLI, not in `LANGUAGE_MANIFESTS`, pair_gate still passes.
- `benchmarks/pairs/datasets.toml`: project pointers.
- `benchmarks/pairs/README.md`: slice list and harvest pointer.
## System Flows
@@ -203,8 +203,8 @@ Harvest (maintainer, not CI): recipe → fetch parent blob and fix blob → extr
- Runnable set stays small; recipes may list more than CI runs.
**Dependencies**
- Inbound: pair loader — concatenate file (P0)
- Outbound: evaluate_pair — cases (P0)
- Inbound: pair loader: concatenate file (P0)
- Outbound: evaluate_pair: cases (P0)
**Contracts**: State [x]
.kiro/specs/real-world-vfc-slice/requirements.md
+3 -3
View File
@@ -8,9 +8,9 @@ AppSec engineers cannot tell whether OpenUltraSAST improved on real product code
## Boundary Context
- **In scope**: A new pair slice of reviewed isolated function pairs from OpenSSL, Firefox (gecko-dev / mozilla-central / mozilla-firefox/firefox), and Chromium; provenance on every vendored file; overlay-versus-inventory scoreboard on that slice; a harvest procedure that does not clone megarepos; offline runnable catalog; testability of catalog load and slice evaluation.
- **Out of scope**: Cloning mozilla-central or Chromium into the repository or CI; FixFox (embargoed until 2030); live Bugzilla scraping; making overlay or vfc Youden a merge or smoke gate; buffer-size / CWE-121 semantics; inter-file taint; parser packaging; changing pattern inventory rules so the seed pairs pass.
- **Adjacent expectations**: Existing pair evaluation still scores `local` as the 100% CI gate and `github` / `sast` as honesty dashboards. Overlay dispositions, taint, and the stage-1 detection gate stay owned by their current specs. Dataset pointers may list Firefox/Chromium/OpenSSL sources; the dumps themselves are not vendored.
- In scope: A new pair slice of reviewed isolated function pairs from OpenSSL, Firefox (gecko-dev / mozilla-central / mozilla-firefox/firefox), and Chromium; provenance on every vendored file; overlay-versus-inventory scoreboard on that slice; a harvest procedure that does not clone megarepos; offline runnable catalog; testability of catalog load and slice evaluation.
- Out of scope: Cloning mozilla-central or Chromium into the repository or CI; FixFox (embargoed until 2030); live Bugzilla scraping; making overlay or vfc Youden a merge or smoke gate; buffer-size / CWE-121 semantics; inter-file taint; parser packaging; changing pattern inventory rules so the seed pairs pass.
- Adjacent expectations: Existing pair evaluation still scores `local` as the 100% CI gate and `github` / `sast` as honesty dashboards. Overlay dispositions, taint, and the stage-1 detection gate stay owned by their current specs. Dataset pointers may list Firefox/Chromium/OpenSSL sources; the dumps themselves are not vendored.
## Requirements
.kiro/specs/real-world-vfc-slice/research.md
+41 -41
View File
@@ -2,49 +2,49 @@
## Summary
- **Feature**: `real-world-vfc-slice`
- **Discovery Scope**: Extension of the existing pair catalog and pair eval CLI
- **Key Findings**:
- Feature: `real-world-vfc-slice`
- Discovery Scope: Extension of the existing pair catalog and pair eval CLI
- Key Findings:
- Pair eval already concatenates `benchmarks/pairs/catalog.toml` with `sast/catalog.toml`. A third file `vfc/catalog.toml` is the smallest seam.
- CLI `--slice` choices are a closed set (`all`, `local`, `github`, `sast`). `vfc` must be added there and in tests that assert slice names.
- `sast` already scores overlay with inventory fallback when nothing is promoted or demoted. `vfc` should use that path as the primary scorer and add a second inventory-only figure for the honesty dashboard.
- `pair_gate` evaluates the full catalog but fails CI only on `local` 100%. Adding vfc pairs does not change the fail condition if names stay out of `LANGUAGE_MANIFESTS`.
- Seed commits with public VCS snapshots exist: OpenSSL Heartbleed (`tls1_process_heartbeat`), Firefox libmar (`mar_insert_item`, CVE-2020-15667), Chromium FileReader (`ArrayBufferResult`, CVE-2019-5786). All three keep a sink-shaped call after the fix, so overlay/inventory may LEAK — that is the dashboard, not a 100% gate.
- Seed commits with public VCS snapshots exist: OpenSSL Heartbleed (`tls1_process_heartbeat`), Firefox libmar (`mar_insert_item`, CVE-2020-15667), Chromium FileReader (`ArrayBufferResult`, CVE-2019-5786). All three keep a sink-shaped call after the fix, so overlay/inventory may LEAK; that is the dashboard, not a 100% gate.
- mozilla-central / gecko-dev must not be cloned. Phabricator and GitHub raw blobs are enough to extract one function. FixFox stays embargoed.
## Research Log
### Existing pair catalog and scoring
- **Context**: Brownfield extension; must not invent a second eval engine.
- **Sources Consulted**: `src/openultrasast/pairs.py`, `cli.py`, `pair_gate.py`, `tests/test_pair_corpus.py`, `benchmarks/pairs/catalog.toml`, `benchmarks/pairs/sast/catalog.toml`, `benchmarks/pairs/README.md`
- **Findings**:
- Context: Brownfield extension; must not invent a second eval engine.
- Sources Consulted: `src/openultrasast/pairs.py`, `cli.py`, `pair_gate.py`, `tests/test_pair_corpus.py`, `benchmarks/pairs/catalog.toml`, `benchmarks/pairs/sast/catalog.toml`, `benchmarks/pairs/README.md`
- Findings:
- `PairCase.slice` is a free string in the catalog loader; the closed set lives only in argparse and tests.
- Overlay pair eval is gated on `case.slice == "sast"`.
- GitHub pairs already use provenance comments plus catalog fields (`repo`, `commit`, `commit_url`, `cve`, `license`).
- Pair names must not appear in `LANGUAGE_MANIFESTS`.
- **Implications**: Add `vfc` as a concatenated catalog; extend the overlay branch to `sast` **or** `vfc`; keep the gate local-only.
- Implications: Add `vfc` as a concatenated catalog; extend the overlay branch to `sast` **or** `vfc`; keep the gate local-only.
### Harvestable public commits (OpenSSL, Firefox, Chromium)
- **Context**: User asked for those three projects and a labeled corpus that can actually be tested.
- **Sources Consulted**:
- Context: User asked for those three projects and a labeled corpus that can actually be tested.
- Sources Consulted:
- OpenSSL `96db9023b881d7cd9f379b0c154650d6c108e9a3` (parent `0d7717fc9c83dafab8153cbd5e2180e6e04cc802`), `ssl/t1_lib.c` `tls1_process_heartbeat`, CVE-2014-0160
- Firefox hg `b79b6cc78248eea7fda10bfb76aa273c19c9fa65` (parent `0c0f777161a9499dd149853ff62d356f75d16c2a`), `modules/libmar/src/mar_read.c` `mar_insert_item`, CVE-2020-15667 / Bug 1653371; Phabricator raw diff
- Chromium `ba9748e78ec7e9c0d594e7edf7b2c07ea2a90449` (parent `9f3fccdba567f30ea39b08f57d8d5b49cd832b4a`), `FileReaderLoader::ArrayBufferResult`, CVE-2019-5786
- Rejected: CVE-2019-11730 (SOP/file URI policy, not a sink pair); FixFox (embargoed to 2030); MFSA text without a commit
- **Findings**:
- Findings:
- Heartbleed fix adds a record-length check; `memcpy(bp, pl, payload)` remains. Inventory `c-unsafe-memory-copy` will fire on both sides (LEAK) until CWE-121 semantics exist.
- Firefox fix is `int namelen` → `uint32_t namelen`; `memcpy(item->name, name, namelen + 1)` remains. Same LEAK class.
- Chromium fix copies the ArrayBuffer on partial results (UAF). No memcpy/eval sink; expected MISS on inventory and overlay. Still a labeled pair.
- GitHub raw and Phabricator `?diff=1` return the function text without cloning.
- **Implications**: Seed three reviewed pairs. Tests assert presence and that the slice **runs**, not that pair_correct is 100%. Memory-safety pairs are allowed as known leaks (Req 6.3).
- Implications: Seed three reviewed pairs. Tests assert presence and that the slice **runs**, not that pair_correct is 100%. Memory-safety pairs are allowed as known leaks (Req 6.3).
### Dataset pointers versus vendored dumps
- **Context**: `datasets.toml` already lists CVEfixes, MoreFixes, SVEN, CWE-Bench-Java. Roadmap asked for Firefox/Chromium/OpenSSL pointers.
- **Findings**: Pointers are metadata only. CI must stay offline on vendored excerpts.
- **Implications**: Add dataset rows for OpenSSL, gecko-dev / mozilla-firefox/firefox, and Chromium. Do not fetch dumps in tests.
- Context: `datasets.toml` already lists CVEfixes, MoreFixes, SVEN, CWE-Bench-Java. Roadmap asked for Firefox/Chromium/OpenSSL pointers.
- Findings: Pointers are metadata only. CI must stay offline on vendored excerpts.
- Implications: Add dataset rows for OpenSSL, gecko-dev / mozilla-firefox/firefox, and Chromium. Do not fetch dumps in tests.
## Architecture Pattern Evaluation
@@ -59,48 +59,48 @@
### Decision: Concatenated `vfc` catalog, overlay primary, inventory sidecar
- **Context**: Need a labeled corpus that is offline, isolated, and honest about overlay versus regex.
- **Alternatives Considered**:
1. Inventory-only like `github` — misses the overlay question the user is evolving.
2. Overlay-only like `sast` without a sidecar — cannot see whether overlay helped or hurt.
3. Dual scoreboard with overlay primary and inventory sidecar — **selected**.
- **Selected Approach**: `slice = "vfc"` uses the overlay pair path (inventory fallback when unadjudicated). `PairEvalResult.scorers["vfc"]` holds both `overlay` and `inventory` metrics. Pair gate still fails only on `local`.
- **Rationale**: Reuses `sast` overlay machinery; satisfies Req 4 without making Youden a merge gate.
- **Trade-offs**: Pair gate runtime grows by three small overlay scans. Acceptable.
- **Follow-up**: Do not retune memcpy rules to force these pairs to PASS.
- Context: Need a labeled corpus that is offline, isolated, and honest about overlay versus regex.
- Alternatives Considered:
1. Inventory-only like `github`: misses the overlay question the user is evolving.
2. Overlay-only like `sast` without a sidecar: cannot see whether overlay helped or hurt.
3. Dual scoreboard with overlay primary and inventory sidecar: **selected**.
- Selected Approach: `slice = "vfc"` uses the overlay pair path (inventory fallback when unadjudicated). `PairEvalResult.scorers["vfc"]` holds both `overlay` and `inventory` metrics. Pair gate still fails only on `local`.
- Rationale: Reuses `sast` overlay machinery; satisfies Req 4 without making Youden a merge gate.
- Trade-offs: Pair gate runtime grows by three small overlay scans. Acceptable.
- Follow-up: Do not retune memcpy rules to force these pairs to PASS.
### Decision: Maintainer harvest script, vendored seed, no CI fetch
- **Context**: Grow the catalog later without cloning mozilla-central.
- **Selected Approach**: `benchmarks/pairs/vfc/harvest.py` plus `recipes.toml`. CI never runs harvest. Seed files are committed excerpts with provenance headers.
- **Rationale**: Req 5 and Req 7 (offline testability).
- **Trade-offs**: Harvest needs network when a maintainer grows the set; tests mock extraction locally.
- Context: Grow the catalog later without cloning mozilla-central.
- Selected Approach: `benchmarks/pairs/vfc/harvest.py` plus `recipes.toml`. CI never runs harvest. Seed files are committed excerpts with provenance headers.
- Rationale: Req 5 and Req 7 (offline testability).
- Trade-offs: Harvest needs network when a maintainer grows the set; tests mock extraction locally.
### Decision: Public hg counts as a version-control commit
- **Context**: Firefox canonical history is hg.mozilla.org; gecko-dev git SHAs are not the hg SHAs; GitHub 422s on hg identifiers.
- **Selected Approach**: Record the hg revision and Phabricator/hg URL. Dataset pointers still list gecko-dev and mozilla-firefox/firefox.
- **Rationale**: Brief forbids MFSA-without-commit, not hg. The MAR fix is a public VCS snapshot.
- **Follow-up**: If a git SHA is later mapped, add it as an extra catalog field without changing the excerpt.
- Context: Firefox canonical history is hg.mozilla.org; gecko-dev git SHAs are not the hg SHAs; GitHub 422s on hg identifiers.
- Selected Approach: Record the hg revision and Phabricator/hg URL. Dataset pointers still list gecko-dev and mozilla-firefox/firefox.
- Rationale: Brief forbids MFSA-without-commit, not hg. The MAR fix is a public VCS snapshot.
- Follow-up: If a git SHA is later mapped, add it as an extra catalog field without changing the excerpt.
## Synthesis
- **Generalization**: `vfc` is another honesty slice, not a new product. Overlay-versus-inventory is a report field, not a new engine.
- **Build vs adopt**: Adopt pair catalog schema, overlay eval, provenance comment style. Build only harvest extract + dual scoreboard payload.
- **Simplification**: No Joern, no megarepo, no 90% gate, no rule changes to chase Youden on three memory/UAF pairs.
- Generalization: `vfc` is another honesty slice, not a new product. Overlay-versus-inventory is a report field, not a new engine.
- Build vs adopt: Adopt pair catalog schema, overlay eval, provenance comment style. Build only harvest extract + dual scoreboard payload.
- Simplification: No Joern, no megarepo, no 90% gate, no rule changes to chase Youden on three memory/UAF pairs.
## Risks & Mitigations
- Overlay/inventory LEAK on Heartbleed and MAR memcpy-after-fix — document as known; tests must not assert PASS.
- Chromium UAF is a MISS — still required so the corpus is not only memcpy.
- Overlay/inventory LEAK on Heartbleed and MAR memcpy-after-fix; document as known; tests must not assert PASS.
- Chromium UAF is a MISS; still required so the corpus includes more than memcpy.
- `pair_gate` currently scans the full catalog; three extra pairs are small. If the catalog grows, gate should keep evaluating local-only for the fail condition (already true) and may later skip overlay slices for speed.
- License: OpenSSL (OpenSSL License / SSLeay), Firefox MPL-2.0, Chromium BSD-3-Clause — headers must state them.
- License: OpenSSL (OpenSSL License / SSLeay), Firefox MPL-2.0, Chromium BSD-3-Clause; headers must state them.
- gecko-dev is archived; harvest should accept hg.mozilla.org and github.com/chromium/chromium / openssl/openssl as hosts.
## References
- [OpenSSL Heartbleed fix](https://github.com/openssl/openssl/commit/96db9023b881d7cd9f379b0c154650d6c108e9a3) — CVE-2014-0160
- [Firefox MAR heap overflow](https://bugzilla.mozilla.org/show_bug.cgi?id=1653371) — CVE-2020-15667, hg `b79b6cc78248`
- [Chromium FileReader UAF](https://github.com/chromium/chromium/commit/ba9748e78ec7e9c0d594e7edf7b2c07ea2a90449) — CVE-2019-5786
- [OpenSSL Heartbleed fix](https://github.com/openssl/openssl/commit/96db9023b881d7cd9f379b0c154650d6c108e9a3): CVE-2014-0160
- [Firefox MAR heap overflow](https://bugzilla.mozilla.org/show_bug.cgi?id=1653371): CVE-2020-15667, hg `b79b6cc78248`
- [Chromium FileReader UAF](https://github.com/chromium/chromium/commit/ba9748e78ec7e9c0d594e7edf7b2c07ea2a90449): CVE-2019-5786
- `.kiro/specs/real-world-vfc-slice/brief.md`
- `.kiro/steering/roadmap.md`