mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Opus 4.8
39a1873e5b
Ran the humanizer rules over 79 prose files in the active specs (brief, design, requirements, research and narrative docs). Removed em dashes, significance inflation, forced rule-of-three, bold-label list items and filler. Skipped tasks.md checklists and the retired specs (harnessx-*, openrouter-sast-harness, unseen-repo-evaluation). Every number, path, command, hash and identifier was preserved: checked each file against HEAD for drift in numbers, inline code and code fences, zero drift across all 79 files. Co-Authored-By: Claude Opus 4.8 <[email protected]>
107 lines
8.1 KiB
Markdown
107 lines
8.1 KiB
Markdown
# Research & Design Decisions
|
|
|
|
## Summary
|
|
|
|
- Feature: `real-world-vfc-slice`
|
|
- Discovery Scope: Extension of the existing pair catalog and pair eval CLI
|
|
- Key Findings:
|
|
- Pair eval already concatenates `benchmarks/pairs/catalog.toml` with `sast/catalog.toml`. A third file `vfc/catalog.toml` is the smallest seam.
|
|
- CLI `--slice` choices are a closed set (`all`, `local`, `github`, `sast`). `vfc` must be added there and in tests that assert slice names.
|
|
- `sast` already scores overlay with inventory fallback when nothing is promoted or demoted. `vfc` should use that path as the primary scorer and add a second inventory-only figure for the honesty dashboard.
|
|
- `pair_gate` evaluates the full catalog but fails CI only on `local` 100%. Adding vfc pairs does not change the fail condition if names stay out of `LANGUAGE_MANIFESTS`.
|
|
- Seed commits with public VCS snapshots exist: OpenSSL Heartbleed (`tls1_process_heartbeat`), Firefox libmar (`mar_insert_item`, CVE-2020-15667), Chromium FileReader (`ArrayBufferResult`, CVE-2019-5786). All three keep a sink-shaped call after the fix, so overlay/inventory may LEAK; that is the dashboard, not a 100% gate.
|
|
- mozilla-central / gecko-dev must not be cloned. Phabricator and GitHub raw blobs are enough to extract one function. FixFox stays embargoed.
|
|
|
|
## Research Log
|
|
|
|
### Existing pair catalog and scoring
|
|
|
|
- Context: Brownfield extension; must not invent a second eval engine.
|
|
- Sources Consulted: `src/openultrasast/pairs.py`, `cli.py`, `pair_gate.py`, `tests/test_pair_corpus.py`, `benchmarks/pairs/catalog.toml`, `benchmarks/pairs/sast/catalog.toml`, `benchmarks/pairs/README.md`
|
|
- Findings:
|
|
- `PairCase.slice` is a free string in the catalog loader; the closed set lives only in argparse and tests.
|
|
- Overlay pair eval is gated on `case.slice == "sast"`.
|
|
- GitHub pairs already use provenance comments plus catalog fields (`repo`, `commit`, `commit_url`, `cve`, `license`).
|
|
- Pair names must not appear in `LANGUAGE_MANIFESTS`.
|
|
- Implications: Add `vfc` as a concatenated catalog; extend the overlay branch to `sast` **or** `vfc`; keep the gate local-only.
|
|
|
|
### Harvestable public commits (OpenSSL, Firefox, Chromium)
|
|
|
|
- Context: User asked for those three projects and a labeled corpus that can actually be tested.
|
|
- Sources Consulted:
|
|
- OpenSSL `96db9023b881d7cd9f379b0c154650d6c108e9a3` (parent `0d7717fc9c83dafab8153cbd5e2180e6e04cc802`), `ssl/t1_lib.c` `tls1_process_heartbeat`, CVE-2014-0160
|
|
- Firefox hg `b79b6cc78248eea7fda10bfb76aa273c19c9fa65` (parent `0c0f777161a9499dd149853ff62d356f75d16c2a`), `modules/libmar/src/mar_read.c` `mar_insert_item`, CVE-2020-15667 / Bug 1653371; Phabricator raw diff
|
|
- Chromium `ba9748e78ec7e9c0d594e7edf7b2c07ea2a90449` (parent `9f3fccdba567f30ea39b08f57d8d5b49cd832b4a`), `FileReaderLoader::ArrayBufferResult`, CVE-2019-5786
|
|
- Rejected: CVE-2019-11730 (SOP/file URI policy, not a sink pair); FixFox (embargoed to 2030); MFSA text without a commit
|
|
- Findings:
|
|
- Heartbleed fix adds a record-length check; `memcpy(bp, pl, payload)` remains. Inventory `c-unsafe-memory-copy` will fire on both sides (LEAK) until CWE-121 semantics exist.
|
|
- Firefox fix is `int namelen` → `uint32_t namelen`; `memcpy(item->name, name, namelen + 1)` remains. Same LEAK class.
|
|
- Chromium fix copies the ArrayBuffer on partial results (UAF). No memcpy/eval sink; expected MISS on inventory and overlay. Still a labeled pair.
|
|
- GitHub raw and Phabricator `?diff=1` return the function text without cloning.
|
|
- Implications: Seed three reviewed pairs. Tests assert presence and that the slice **runs**, not that pair_correct is 100%. Memory-safety pairs are allowed as known leaks (Req 6.3).
|
|
|
|
### Dataset pointers versus vendored dumps
|
|
|
|
- Context: `datasets.toml` already lists CVEfixes, MoreFixes, SVEN, CWE-Bench-Java. Roadmap asked for Firefox/Chromium/OpenSSL pointers.
|
|
- Findings: Pointers are metadata only. CI must stay offline on vendored excerpts.
|
|
- Implications: Add dataset rows for OpenSSL, gecko-dev / mozilla-firefox/firefox, and Chromium. Do not fetch dumps in tests.
|
|
|
|
## Architecture Pattern Evaluation
|
|
|
|
| Option | Description | Strengths | Risks / Limitations | Notes |
|
|
|--------|-------------|-----------|---------------------|-------|
|
|
| Concatenate a `vfc` catalog like `sast` | Third TOML file, same `[[pair]]` schema | Matches existing loader; no new types | CLI choices and tests must be updated | **Selected** |
|
|
| Stuff vfc rows into `catalog.toml` | One file | Fewer paths | Mixes local/github with large-project VFCs | Rejected; sast already split |
|
|
| New eval package | Separate scorer | Clean isolation | Duplicates pair eval | Rejected |
|
|
| Clone megarepos in CI | Scan real trees | Maximum fidelity | Multi-GB, scrubbed commit messages, not isolated | Explicitly out of scope |
|
|
|
|
## Design Decisions
|
|
|
|
### Decision: Concatenated `vfc` catalog, overlay primary, inventory sidecar
|
|
|
|
- Context: Need a labeled corpus that is offline, isolated, and honest about overlay versus regex.
|
|
- Alternatives Considered:
|
|
1. Inventory-only like `github`: misses the overlay question the user is evolving.
|
|
2. Overlay-only like `sast` without a sidecar: cannot see whether overlay helped or hurt.
|
|
3. Dual scoreboard with overlay primary and inventory sidecar: **selected**.
|
|
- Selected Approach: `slice = "vfc"` uses the overlay pair path (inventory fallback when unadjudicated). `PairEvalResult.scorers["vfc"]` holds both `overlay` and `inventory` metrics. Pair gate still fails only on `local`.
|
|
- Rationale: Reuses `sast` overlay machinery; satisfies Req 4 without making Youden a merge gate.
|
|
- Trade-offs: Pair gate runtime grows by three small overlay scans. Acceptable.
|
|
- Follow-up: Do not retune memcpy rules to force these pairs to PASS.
|
|
|
|
### Decision: Maintainer harvest script, vendored seed, no CI fetch
|
|
|
|
- Context: Grow the catalog later without cloning mozilla-central.
|
|
- Selected Approach: `benchmarks/pairs/vfc/harvest.py` plus `recipes.toml`. CI never runs harvest. Seed files are committed excerpts with provenance headers.
|
|
- Rationale: Req 5 and Req 7 (offline testability).
|
|
- Trade-offs: Harvest needs network when a maintainer grows the set; tests mock extraction locally.
|
|
|
|
### Decision: Public hg counts as a version-control commit
|
|
|
|
- Context: Firefox canonical history is hg.mozilla.org; gecko-dev git SHAs are not the hg SHAs; GitHub 422s on hg identifiers.
|
|
- Selected Approach: Record the hg revision and Phabricator/hg URL. Dataset pointers still list gecko-dev and mozilla-firefox/firefox.
|
|
- Rationale: Brief forbids MFSA-without-commit, not hg. The MAR fix is a public VCS snapshot.
|
|
- Follow-up: If a git SHA is later mapped, add it as an extra catalog field without changing the excerpt.
|
|
|
|
## Synthesis
|
|
|
|
- Generalization: `vfc` is another honesty slice, not a new product. Overlay-versus-inventory is a report field, not a new engine.
|
|
- Build vs adopt: Adopt pair catalog schema, overlay eval, provenance comment style. Build only harvest extract + dual scoreboard payload.
|
|
- Simplification: No Joern, no megarepo, no 90% gate, no rule changes to chase Youden on three memory/UAF pairs.
|
|
|
|
## Risks & Mitigations
|
|
|
|
- Overlay/inventory LEAK on Heartbleed and MAR memcpy-after-fix; document as known; tests must not assert PASS.
|
|
- Chromium UAF is a MISS; still required so the corpus includes more than memcpy.
|
|
- `pair_gate` currently scans the full catalog; three extra pairs are small. If the catalog grows, gate should keep evaluating local-only for the fail condition (already true) and may later skip overlay slices for speed.
|
|
- License: OpenSSL (OpenSSL License / SSLeay), Firefox MPL-2.0, Chromium BSD-3-Clause; headers must state them.
|
|
- gecko-dev is archived; harvest should accept hg.mozilla.org and github.com/chromium/chromium / openssl/openssl as hosts.
|
|
|
|
## References
|
|
|
|
- [OpenSSL Heartbleed fix](https://github.com/openssl/openssl/commit/96db9023b881d7cd9f379b0c154650d6c108e9a3): CVE-2014-0160
|
|
- [Firefox MAR heap overflow](https://bugzilla.mozilla.org/show_bug.cgi?id=1653371): CVE-2020-15667, hg `b79b6cc78248`
|
|
- [Chromium FileReader UAF](https://github.com/chromium/chromium/commit/ba9748e78ec7e9c0d594e7edf7b2c07ea2a90449): CVE-2019-5786
|
|
- `.kiro/specs/real-world-vfc-slice/brief.md`
|
|
- `.kiro/steering/roadmap.md`
|