mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-17 12:09:27 +00:00
feat(real-world-vfc-slice): add labeled OpenSSL/Firefox/Chromium/curl pair corpus
Harvest 176 isolated vuln-vs-fix functions (109 CVEs, 352 training rows) into benchmarks/pairs/vfc with provenance and a JSONL training manifest. Score the slice as an honesty dashboard (ousast pairs --slice vfc); keep the pair gate on the local fixtures only.
This commit is contained in:
Marius Ciepluch
@@ -0,0 +1,54 @@
|
||||
# Brief: real-world-vfc-slice
|
||||
|
||||
## Problem
|
||||
|
||||
Operators care whether the harness fires on a real buggy snapshot and stays quiet after the patch — Firefox, Chromium, OpenSSL — not whether OWASP BenchmarkTest00074 is a TP. The labeled SAST slice is 11 textbook files. Overlay Youden there is **worse** than inventory (+9.09% → 0%). We cannot tell if the tool got better on messy product code.
|
||||
|
||||
## Current State
|
||||
|
||||
Slices today: `local` (CI 100%), `github` (6 vendored functions: SVEN, hutool, node-serialize), `sast` (OWASP/Juliet). `datasets.toml` already points at CVEfixes, MoreFixes, CWE-Bench-Java, Vul4J. Nothing is harvested from gecko-dev / Firefox. Whole mozilla-central is not in-tree (and should not be). FixFox (Zenodo) is embargoed until 2030. Firefox security commit messages are often scrubbed of “security” words, so grep-the-log is a bad miner.
|
||||
|
||||
Measured on the current 11-pair **sast** slice (2026-09-04):
|
||||
|
||||
| Scorer | pair_correct | vuln recall | silent-on-fix | Youden |
|
||||
|--------|--------------|-------------|---------------|--------|
|
||||
| Inventory (regex) | 2/11 (18%) | 54.5% | 54.5% | **+9.09%** |
|
||||
| Overlay (current) | 2/11 (18%) | 45.5% | 54.5% | **0%** |
|
||||
|
||||
The only overlay change vs inventory on that slice: `owasp-python-codeinj` went from LEAK (regex hit both sides) to MISS (ConfigParser heap → `flow_incomplete`, no promote). C/Java Juliet goodG2B still leak because those languages are unadjudicated and scoring falls back to inventory.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
A new pair slice (e.g. `vfc`) of **reviewed, isolated function pairs** from large projects, including Firefox/gecko-dev when a public fix commit exists in CVEfixes/MoreFixes. Offline CI. Honesty dashboard like `github`, not a 90% smoke gate. Overlay scoring once tree-sitter can parse C/JS/Java; inventory scoring until then. Provenance on every file (repo, commit, CVE, license).
|
||||
|
||||
## Approach
|
||||
|
||||
Do not clone Firefox. Filter public VFC databases for `gecko-dev` / `mozilla-firefox/firefox`, `chromium`, `openssl`, `curl`, plus existing CWE-Bench-Java. Extract the changed function before/after. Human-review labels (DiverseVul ~60% is not ground truth). Cap the runnable catalog (tens of pairs, not thousands). Prefer injection/XSS/path CWEs the overlay can talk about; keep UAF/buffer overflows as known inventory-only leaks until a later memory-safety spec.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: harvest script or documented procedure; catalog slice; provenance headers; overlay-vs-inventory scoreboard on that slice; dataset.toml pointers for Firefox/Chromium.
|
||||
- **Out**: Vendoring mozilla-central; live Bugzilla scraping; making Youden a merge gate; buffer-size semantics; tree-sitter packaging (owned by `tree-sitter-overlay-extra`).
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
- Catalog schema / slice name vs pair eval scoring
|
||||
- Harvest tooling vs vendored excerpts
|
||||
|
||||
## Out of Boundary
|
||||
|
||||
- Parser extra, Joern, improve-loop auto-facts, `LANGUAGE_MANIFESTS`
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `pairs.py` catalog format; `propose-adjudicate-prove` overlay scoring
|
||||
- **Downstream**: improve-loop signals from real misses; later memory-safety spec can reuse the same slice
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: pair catalog only (new slice). Does not change smoke gate.
|
||||
- **Adjacent**: `tree-sitter-overlay-extra` (C/Java overlay on this slice once the extra exists)
|
||||
|
||||
## Constraints
|
||||
|
||||
Isolated functions, same `relpath` vuln/fixed. License in the header. No 16 GB dumps in git. Firefox pairs only when a public git commit pair exists — MFSA text without a commit is not a pair.
|
||||
@@ -0,0 +1,376 @@
|
||||
# Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose.** This feature gives OpenUltraSAST a labeled vuln-versus-fix slice from real OpenSSL, Firefox, and Chromium functions so engineers can evolve the harness against product code instead of only OWASP/Juliet.
|
||||
|
||||
**Users.** Maintainers grow and score the catalog. AppSec engineers read the honesty dashboard. CI proves the catalog loads and the slice command runs offline.
|
||||
|
||||
**Impact.** A new `vfc` slice is concatenated into the pair catalog. Overlay and inventory scores are reported side by side for that slice. The local 100% pair gate and the stage-1 smoke gate do not change.
|
||||
|
||||
### Goals
|
||||
|
||||
- Seed at least one reviewed isolated function pair each from OpenSSL, Firefox, and Chromium, with provenance.
|
||||
- Score `vfc` as an honesty dashboard with overlay versus inventory figures.
|
||||
- Harvest one function from public parent/fix blobs without cloning megarepos.
|
||||
- Keep CI extra-free and offline; vfc Youden is not a merge condition.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- Cloning mozilla-central, gecko-dev, or Chromium.
|
||||
- FixFox, Bugzilla scraping, or vendoring VFC dumps.
|
||||
- CWE-121 buffer-size semantics, inter-file taint, or parser extra packaging.
|
||||
- Changing inventory rules so the seed pairs become pair_correct.
|
||||
- Making overlay or vfc Youden fail CI.
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
|
||||
- The `vfc` pair slice: catalog rows, vendored excerpts, provenance headers, recipes.
|
||||
- Harvest procedure that extracts one function from public parent/fix snapshots.
|
||||
- Dual overlay/inventory scoreboard fields for the vfc slice.
|
||||
- CLI slice choice `vfc` and tests that the slice loads and runs.
|
||||
- Dataset pointers for OpenSSL, Firefox/gecko-dev, and Chromium.
|
||||
|
||||
### Out of Boundary
|
||||
|
||||
- Overlay dispositions, taint, facts, FileIR (`propose-adjudicate-prove`).
|
||||
- Tree-sitter extra packaging (`tree-sitter-overlay-extra`).
|
||||
- Stage-1 `LANGUAGE_MANIFESTS` detection gate and local 100% pair gate fail condition.
|
||||
- Pattern rule bodies; do not retune memcpy/eval to chase vfc Youden.
|
||||
- Memory-safety / CWE-121 semantics (later spec may reuse these pairs).
|
||||
|
||||
### Allowed Dependencies
|
||||
|
||||
- Existing `PairCase` / `load_pair_catalog` / `evaluate_pair` / `result_payload`.
|
||||
- Overlay pair path already used by `sast` (inventory fallback when unadjudicated).
|
||||
- Stdlib HTTP client for harvest (maintainer-only, not CI).
|
||||
- Public hosts: GitHub raw blobs, hg.mozilla.org / Phabricator raw diffs.
|
||||
|
||||
### Revalidation Triggers
|
||||
|
||||
- Changing `PairCase` required fields or catalog concatenation order.
|
||||
- Failing CI on any slice other than `local`.
|
||||
- Adding vfc names to `LANGUAGE_MANIFESTS`.
|
||||
- Making harvest a network-using CI step.
|
||||
- Importing parser wheels into harvest or pair eval.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
|
||||
Pair eval materializes each side of a pair into an isolated temp tree and scans it. `local` and `github` use inventory (`quick_scan`). `sast` uses overlay, falling back to inventory when no promote/demote/coverage occurred. Catalog load already concatenates a second file. The pair gate prints every slice but fails only when `local` pair_pass_rate is below 100%.
|
||||
|
||||
### Architecture Pattern and Boundary Map
|
||||
|
||||
Selected: **concatenated honesty slice + dual scoreboard**. Same catalog schema. Overlay eval generalized from `sast` to `{sast, vfc}`. Inventory sidecar computed only for vfc.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
cli[ousast pairs slice vfc]
|
||||
load[load_pair_catalog]
|
||||
mainCat[pairs catalog]
|
||||
sastCat[sast catalog]
|
||||
vfcCat[vfc catalog]
|
||||
eval[evaluate_pair]
|
||||
overlay[overlay with inventory fallback]
|
||||
inventory[inventory only]
|
||||
board[result payload]
|
||||
gate[pair_gate local only]
|
||||
cli --> load
|
||||
load --> mainCat
|
||||
load --> sastCat
|
||||
load --> vfcCat
|
||||
load --> eval
|
||||
eval --> overlay
|
||||
eval --> inventory
|
||||
overlay --> board
|
||||
inventory --> board
|
||||
load --> gate
|
||||
```
|
||||
|
||||
**Key decisions**
|
||||
|
||||
- Primary `per_slice.vfc` metrics come from overlay-with-fallback (same as sast).
|
||||
- `scorers.vfc.overlay` and `scorers.vfc.inventory` are explicit so operators can compare.
|
||||
- Harvest never runs in pytest. Seed excerpts are committed.
|
||||
- Dependency direction: recipes/excerpts → vfc catalog → pair loader → eval → CLI payload. Harvest writes excerpts; it does not import overlay.
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Choice / Version | Role in Feature | Notes |
|
||||
|-------|------------------|-----------------|-------|
|
||||
| CLI | existing `ousast pairs --slice` | Operator entry | Add `vfc` to choices |
|
||||
| Catalog | TOML `[[pair]]` | Same schema as github/sast | New file, concatenated |
|
||||
| Scoring | existing overlay + inventory | Dual scoreboard | No new taint engine |
|
||||
| Harvest | Python stdlib urllib | Maintainer fetch | Not a core package dep |
|
||||
| Tests | pytest extra-free | Offline existence + CLI | No network |
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
```
|
||||
benchmarks/pairs/
|
||||
├── catalog.toml # unchanged rows; still the default path
|
||||
├── datasets.toml # add OpenSSL, Firefox, Chromium pointers
|
||||
├── README.md # document vfc slice
|
||||
├── sast/catalog.toml # unchanged
|
||||
└── vfc/
|
||||
├── catalog.toml # slice = vfc rows
|
||||
├── recipes.toml # harvest inputs for seed and later pairs
|
||||
├── harvest.py # extract one function from parent/fix URLs
|
||||
├── openssl-cve-2014-0160/
|
||||
│ ├── vuln.c
|
||||
│ └── fixed.c
|
||||
├── firefox-cve-2020-15667/
|
||||
│ ├── vuln.c
|
||||
│ └── fixed.c
|
||||
└── chromium-cve-2019-5786/
|
||||
├── vuln.cc
|
||||
└── fixed.cc
|
||||
```
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `src/openultrasast/pairs.py` — concatenate vfc catalog; overlay path for `vfc`; `scorers` on the eval result; `DEFAULT_VFC_CATALOG`.
|
||||
- `src/openultrasast/cli.py` — `--slice` choice `vfc`.
|
||||
- `src/openultrasast/pair_gate.py` — comment that vfc is honesty-only; fail condition stays local.
|
||||
- `tests/test_pair_corpus.py` — vfc load, provenance, CLI, not in `LANGUAGE_MANIFESTS`, pair_gate still passes.
|
||||
- `benchmarks/pairs/datasets.toml` — project pointers.
|
||||
- `benchmarks/pairs/README.md` — slice list and harvest pointer.
|
||||
|
||||
## System Flows
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Op as Operator
|
||||
participant CLI as pairs command
|
||||
participant Cat as vfc catalog
|
||||
participant Eval as pair eval
|
||||
participant Inv as inventory scan
|
||||
participant Ov as overlay adjudicate
|
||||
Op->>CLI: pairs slice vfc
|
||||
CLI->>Cat: load concatenated catalogs
|
||||
Cat->>Eval: each vfc pair
|
||||
Eval->>Inv: vuln tree and fix tree
|
||||
Eval->>Ov: same trees
|
||||
Ov-->>Eval: promote or fallback
|
||||
Inv-->>Eval: inventory sidecar
|
||||
Eval-->>CLI: outcomes plus scorers
|
||||
CLI-->>Op: JSON or text scoreboard
|
||||
```
|
||||
|
||||
Harvest (maintainer, not CI): recipe → fetch parent blob and fix blob → extract named function → write provenance header → write vuln/fixed files. Catalog rows stay hand-reviewed.
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Summary | Components | Interfaces | Flows |
|
||||
|-------------|---------|------------|------------|-------|
|
||||
| 1.1, 1.2, 1.3 | vfc slice with three projects | VfcCatalog | catalog load | pairs command |
|
||||
| 1.4 | reject advisory-only | HarvestTool, VfcCatalog | recipe validation | harvest |
|
||||
| 2.1–2.4 | provenance and same relpath | ProvenanceHeader, VfcCatalog | catalog fields + file header | harvest |
|
||||
| 3.1–3.4 | honesty not merge gate | PairGatePolicy, DualScorer | pair_gate, payload | gate |
|
||||
| 4.1–4.4 | overlay vs inventory | DualScorer | scorers map | pairs command |
|
||||
| 5.1–5.5 | harvest without clones | HarvestTool | recipes + harvest script | harvest |
|
||||
| 6.1–6.4 | isolated reviewed functions | VfcCatalog, SeedExcerpts | expected CWE/sink | pair eval |
|
||||
| 7.1–7.4 | offline testability | VfcTestMatrix | pytest + CLI | extra-free CI |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|
||||
|-----------|--------------|--------|--------------|------------------|-----------|
|
||||
| VfcCatalog | Catalog | Reviewed vfc rows | 1.1, 1.2, 1.3, 2.3, 6.4 | Pair loader P0 | State |
|
||||
| ProvenanceHeader | Catalog | Auditable origin on disk | 2.1, 2.2, 2.4 | VfcCatalog P0 | State |
|
||||
| HarvestTool | Maintainer | Extract one function from public blobs | 1.4, 5.1, 5.2, 5.3, 5.4 | urllib P1 | Batch |
|
||||
| DualScorer | Eval | Overlay primary + inventory sidecar | 4.1, 4.2, 4.3, 4.4, 3.3 | overlay eval P0 | State |
|
||||
| PairGatePolicy | Eval | Local-only fail | 3.1, 3.2, 3.4, 7.4 | pair_gate P0 | State |
|
||||
| SeedExcerpts | Catalog | Three isolated functions | 1.2, 6.1, 6.2, 6.3 | VfcCatalog P0 | State |
|
||||
| VfcTestMatrix | Tests | Offline load and CLI | 7.1, 7.2, 7.3, 5.5 | pytest P0 | State |
|
||||
|
||||
### Catalog
|
||||
|
||||
#### VfcCatalog
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Concatenated `slice = "vfc"` rows with OpenSSL, Firefox, Chromium seeds |
|
||||
| Requirements | 1.1, 1.2, 1.3, 2.3, 6.4 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- Same `[[pair]]` / `[[pair.expected]]` schema as github/sast.
|
||||
- `load_pair_catalog` appends `benchmarks/pairs/vfc/catalog.toml` when loading the default catalog.
|
||||
- Names stay out of `LANGUAGE_MANIFESTS`.
|
||||
- Runnable set stays small; recipes may list more than CI runs.
|
||||
|
||||
**Dependencies**
|
||||
- Inbound: pair loader — concatenate file (P0)
|
||||
- Outbound: evaluate_pair — cases (P0)
|
||||
|
||||
**Contracts**: State [x]
|
||||
|
||||
##### State Management
|
||||
- `DEFAULT_VFC_CATALOG` next to `DEFAULT_SAST_CATALOG`.
|
||||
- Missing vfc file: do not crash default load if we require the seed to exist; the seed is committed so the file is present. Tests assert it exists.
|
||||
|
||||
#### ProvenanceHeader
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Human-readable origin on every vendored excerpt |
|
||||
| Requirements | 2.1, 2.2, 2.4 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- Header comments include repo, commit, parent, commit URL, CVE, license, function name.
|
||||
- Catalog row repeats repo, commit, commit_url, cve, license.
|
||||
- License must be stated (OpenSSL License, MPL-2.0, BSD-3-Clause for the seeds).
|
||||
|
||||
#### SeedExcerpts
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Isolated functions, same relpath, reviewed labels |
|
||||
| Requirements | 1.2, 6.1, 6.2, 6.3 |
|
||||
|
||||
Seed pairs (reviewed):
|
||||
|
||||
| Name | Project | CVE | Function | Expected class | Honesty note |
|
||||
|------|---------|-----|----------|----------------|--------------|
|
||||
| openssl-cve-2014-0160 | OpenSSL | CVE-2014-0160 | `tls1_process_heartbeat` | CWE-126 / memcpy | memcpy remains after fix → likely LEAK |
|
||||
| firefox-cve-2020-15667 | Firefox libmar | CVE-2020-15667 | `mar_insert_item` | CWE-119 / memcpy | namelen signedness fix; memcpy remains → likely LEAK |
|
||||
| chromium-cve-2019-5786 | Chromium Blink | CVE-2019-5786 | `FileReaderLoader::ArrayBufferResult` | CWE-416 UAF | no inventory sink → likely MISS |
|
||||
|
||||
Stubs (SSL types, MarItem, DOMArrayBuffer) are allowed only to keep the changed function intact.
|
||||
|
||||
### Eval
|
||||
|
||||
#### DualScorer
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Overlay-with-fallback as primary vfc metrics; inventory sidecar |
|
||||
| Requirements | 4.1, 4.2, 4.3, 4.4, 3.3 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- `evaluate_pair` uses overlay when `slice in {"sast", "vfc"}`.
|
||||
- Inventory-only evaluation of vfc cases fills `PairEvalResult.scorers["vfc"]["inventory"]`.
|
||||
- Overlay metrics also appear as `scorers["vfc"]["overlay"]` (same numbers as `per_slice["vfc"]`).
|
||||
- `result_payload` includes `scorers`.
|
||||
- Do not use these figures as a gate.
|
||||
|
||||
**Contracts**: State [x]
|
||||
|
||||
```python
|
||||
class PairEvalResult:
|
||||
outcomes: tuple[PairOutcome, ...]
|
||||
overall: PairCorpusMetrics
|
||||
per_slice: dict[str, PairCorpusMetrics]
|
||||
signals: tuple[dict[str, object], ...]
|
||||
scorers: dict[str, dict[str, PairCorpusMetrics]]
|
||||
```
|
||||
|
||||
##### Service Interface
|
||||
- `evaluate_catalog(cases) -> PairEvalResult`
|
||||
- Preconditions: each case has existing vuln and fixed files.
|
||||
- Postconditions: if any case has `slice == "vfc"`, `scorers["vfc"]` has `overlay` and `inventory`.
|
||||
- Invariants: pair_gate fail reasons mention only local.
|
||||
|
||||
#### PairGatePolicy
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Keep merge fail on local 100% only |
|
||||
| Requirements | 3.1, 3.2, 3.4, 7.4 |
|
||||
|
||||
`pair_gate` may still *evaluate* the full catalog for the printed dashboard. It must not add vfc pair_correct to `reasons`.
|
||||
|
||||
### Maintainer
|
||||
|
||||
#### HarvestTool
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Fetch parent/fix blobs and extract one function |
|
||||
| Requirements | 1.4, 5.1, 5.2, 5.3, 5.4 |
|
||||
|
||||
**Contracts**: Batch [x]
|
||||
|
||||
##### Batch / Job Contract
|
||||
- Trigger: maintainer runs harvest with a recipe name. CI does not.
|
||||
- Input: `recipes.toml` row: name, host, repo, parent, commit, path, function, cve, license, relpath.
|
||||
- Validation: parent and commit required; reject recipes that have only an advisory URL.
|
||||
- Output: vuln/fixed files with provenance headers. Catalog row is still a human edit.
|
||||
- Idempotency: overwrite excerpts for that recipe name; do not fetch in tests.
|
||||
|
||||
Function extract: start at the named declarator, copy through balanced braces. Do not clone a repo; HTTP GET of a raw blob or a raw diff.
|
||||
|
||||
Hosts: `raw.githubusercontent.com`, Phabricator `?diff=1`. Not FixFox.
|
||||
|
||||
### Tests
|
||||
|
||||
#### VfcTestMatrix
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Offline proof the corpus exists and the slice runs |
|
||||
| Requirements | 7.1, 7.2, 7.3, 5.5 |
|
||||
|
||||
- Catalog load includes `vfc`; files exist; names include openssl, firefox, chromium.
|
||||
- Provenance headers contain commit URL, CVE, license.
|
||||
- `ousast pairs --slice vfc --json` exits 0; payload has those names and `scorers.vfc`.
|
||||
- Pair gate still passes; vfc names not in `LANGUAGE_MANIFESTS`.
|
||||
- Harvest extract is unit-tested on a local string, not the network.
|
||||
- Do not assert 100% pair_correct or a Youden floor.
|
||||
|
||||
## Data Models
|
||||
|
||||
### Domain Model
|
||||
|
||||
- **Pair**: isolated vuln snapshot + fixed snapshot + expected CWE/sink + provenance.
|
||||
- **Slice**: `vfc` honesty set. Not a gate.
|
||||
- **Scorer**: `overlay` or `inventory`.
|
||||
- **Recipe**: harvest input, not ground truth until reviewed into the catalog.
|
||||
|
||||
### Logical Data Model
|
||||
|
||||
Catalog row (existing fields plus required provenance for vfc):
|
||||
|
||||
- `name`, `slice = "vfc"`, `language`, `origin`, `license`, `repo`, `commit`, `parent`, `commit_url`, `cve`, `vuln`, `fixed`, `relpath`, `min_recall`, `fix_policy = "silent"`
|
||||
- `[[pair.expected]]`: `cwe`, `class`, `path`, `rule_id`, `sink`, `evidence`
|
||||
|
||||
`scorers` payload:
|
||||
|
||||
```json
|
||||
{
|
||||
"scorers": {
|
||||
"vfc": {
|
||||
"overlay": { "pairs": 3, "pair_correct": 0, "youden": 0.0 },
|
||||
"inventory": { "pairs": 3, "pair_correct": 0, "youden": 0.0 }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Exact counts are not a contract; presence of both objects is.
|
||||
|
||||
## Error Handling
|
||||
|
||||
- Missing vfc catalog file when loading the default catalog: fail tests (seed is required). Loading an explicit `--catalog` path does not auto-append vfc.
|
||||
- Harvest HTTP failure: non-zero exit, no partial file without a provenance header.
|
||||
- Advisory-only recipe: reject before fetch.
|
||||
- Overlay parse_failed / language_unsupported: inventory fallback (existing sast behavior).
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- Unit: harvest brace-extract on a fixture string; catalog load sees three project names; provenance fields non-empty.
|
||||
- Integration: `evaluate_catalog(select_slice(..., "vfc"))` returns three outcomes and both scorers; pair_gate still passed.
|
||||
- CLI: `main(["pairs", "--slice", "vfc", "--json"])` == 0.
|
||||
- Regression: `LANGUAGE_MANIFESTS` leak check includes vfc names; extra-free suite still green.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Vendored excerpts are public historical vuln code with license headers. Do not vendor exploit PoCs (Chromium FileReader exploit HTML is out of scope; only the C++ function).
|
||||
- Harvest talks only to listed public hosts. No Bugzilla credentials. No FixFox.
|
||||
|
||||
## Supporting References
|
||||
|
||||
Seed commit identifiers are recorded in `research.md` and must appear in the vendored headers at implementation time.
|
||||
@@ -0,0 +1,91 @@
|
||||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
AppSec engineers cannot tell whether OpenUltraSAST improved on real product code. The labeled pair slices today are in-tree fixtures (`local`), small GitHub/SVEN functions (`github`), and textbook OWASP/Juliet files (`sast`). Operators care whether the harness fires on a real buggy snapshot of OpenSSL, Firefox, or Chromium and stays quiet after the patch. This spec adds a reviewed, isolated-function **vfc** slice drawn from public vulnerability-fixing commits in those projects, scored as an honesty dashboard (not a merge gate), so the tool can be evolved against labeled product code instead of cheat-sheet recall.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: A new pair slice of reviewed isolated function pairs from OpenSSL, Firefox (gecko-dev / mozilla-central / mozilla-firefox/firefox), and Chromium; provenance on every vendored file; overlay-versus-inventory scoreboard on that slice; a harvest procedure that does not clone megarepos; offline runnable catalog; testability of catalog load and slice evaluation.
|
||||
- **Out of scope**: Cloning mozilla-central or Chromium into the repository or CI; FixFox (embargoed until 2030); live Bugzilla scraping; making overlay or vfc Youden a merge or smoke gate; buffer-size / CWE-121 semantics; inter-file taint; parser packaging; changing pattern inventory rules so the seed pairs pass.
|
||||
- **Adjacent expectations**: Existing pair evaluation still scores `local` as the 100% CI gate and `github` / `sast` as honesty dashboards. Overlay dispositions, taint, and the stage-1 detection gate stay owned by their current specs. Dataset pointers may list Firefox/Chromium/OpenSSL sources; the dumps themselves are not vendored.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Seed vfc slice from OpenSSL, Firefox, and Chromium
|
||||
|
||||
**Objective:** As an AppSec engineer, I want a labeled vuln-versus-fix slice from OpenSSL, Firefox, and Chromium, so that I can measure the harness on real product functions instead of only textbook files.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When an operator requests the vfc pair slice, OpenUltraSAST shall evaluate a catalog of isolated vuln-versus-fix pairs whose slice name is `vfc`.
|
||||
2. The vfc slice shall include at least one reviewed pair whose origin project is OpenSSL, at least one whose origin project is Firefox, and at least one whose origin project is Chromium.
|
||||
3. When the default pair catalog is loaded, the system shall include the vfc pairs together with the existing local, github, and sast pairs.
|
||||
4. If a candidate has only an advisory write-up and no public version-control commit for both the vulnerable and fixed snapshots, then the system shall not accept that candidate as a vfc pair.
|
||||
|
||||
### Requirement 2: Provenance on every vendored pair
|
||||
|
||||
**Objective:** As a maintainer, I want every vfc file to carry repo, commit, identifier, and license, so that labels can be audited and redistributed legally.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a vfc pair is vendored, the system shall record the source repository, the fixing commit identifier, a public commit URL, the CVE or equivalent public identifier when one exists, and the license of the original file.
|
||||
2. When a vfc source file is stored in the catalog, the file shall contain a human-readable provenance header with those same fields.
|
||||
3. The system shall use the same relative path on the vulnerable snapshot and the fixed snapshot of a pair.
|
||||
4. If license information cannot be stated for a candidate, then the system shall not vendor that candidate.
|
||||
|
||||
### Requirement 3: Honesty dashboard, not a merge gate
|
||||
|
||||
**Objective:** As a developer merging changes, I want vfc misses and leaks to be visible without failing CI, so that real-project honesty cannot be gamed by blocking the merge.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the pair efficiency gate runs, the system shall not fail because a vfc pair is a miss or a leak.
|
||||
2. The system shall not add vfc pair names to the stage-1 detection smoke-gate fixture list.
|
||||
3. When an operator evaluates the vfc slice, the system shall report pair-correct, vuln-side detection, silent-on-fix, and Youden for that slice.
|
||||
4. The system shall not require the vfc slice to reach a 100% pair-correct rate, and shall not treat overlay or vfc Youden as a merge condition.
|
||||
|
||||
### Requirement 4: Overlay versus inventory scoreboard
|
||||
|
||||
**Objective:** As an engineer evolving adjudication, I want inventory and overlay scores side by side on the vfc slice, so that I can see whether the overlay helped or hurt on real product functions.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When an operator evaluates the vfc slice, the system shall report inventory scoring and overlay scoring for that slice as separate scoreboard figures.
|
||||
2. Where overlay cannot adjudicate a vfc file, the overlay scoreboard shall fall back to inventory for that pair rather than recording silence as safety.
|
||||
3. The system shall not replace the local-slice pair gate with overlay or vfc Youden.
|
||||
4. When the vfc scoreboard is emitted as machine-readable output, both scorers shall be present so a later run can be compared without re-deriving the split.
|
||||
|
||||
### Requirement 5: Harvest without megarepos or embargoed dumps
|
||||
|
||||
**Objective:** As a maintainer, I want a repeatable way to add reviewed pairs from public commits, so that the catalog can grow without vendoring Firefox, Chromium, or 16 GB VFC dumps.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The system shall provide a harvest procedure that fetches a named file at a parent commit and a fixing commit from a public host and extracts one function into the vuln and fixed snapshots.
|
||||
2. When harvest runs, the system shall not require a full clone of mozilla-central, gecko-dev, or Chromium to produce a pair.
|
||||
3. The system shall not vendor FixFox or other embargoed datasets.
|
||||
4. DiverseVul-style automatic vulnerable labels shall not be treated as ground truth unless a human review step has accepted the pair.
|
||||
5. The runnable catalog that CI evaluates shall remain a small reviewed set (on the order of tens of pairs, not thousands).
|
||||
|
||||
### Requirement 6: Isolated, reviewed functions the overlay can be honest about
|
||||
|
||||
**Objective:** As an operator, I want each pair to be a single isolated function with a reviewed label, so that a miss or leak is attributable to the harness rather than to an unlabeled megatree.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a vfc pair is evaluated, the scanner shall see only the vulnerable file or only the fixed file, never both in the same tree.
|
||||
2. Each vfc pair shall isolate the changed function (plus the minimum types or stubs needed to keep that function intact), not an entire project tree.
|
||||
3. The catalog shall prefer injection, XSS, or path-traversal pairs when a public commit exists, and may include memory-safety pairs as known inventory-only leaks until a later memory-safety spec.
|
||||
4. A maintainer shall be able to read the expected CWE, sink, and evidence for each seed pair.
|
||||
|
||||
### Requirement 7: Slice is testable offline
|
||||
|
||||
**Objective:** As a maintainer, I want CI to prove the vfc catalog loads and the slice command runs without network access, so that the labeled corpus cannot rot unnoticed.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the test suite loads the default pair catalog, it shall observe vfc among the slice names and shall observe that every vfc vuln file and fixed file exists on disk.
|
||||
2. When an operator runs the pairs command for slice `vfc`, the command shall complete and emit a scoreboard that includes the OpenSSL, Firefox, and Chromium seed pair names.
|
||||
3. If the semantic extra is absent, then vfc evaluation shall still complete (via inventory fallback) rather than fail the extra-free suite.
|
||||
4. The extra-free detection smoke gate shall still ignore vfc pair names and vfc Youden.
|
||||
@@ -0,0 +1,106 @@
|
||||
# Research & Design Decisions
|
||||
|
||||
## Summary
|
||||
|
||||
- **Feature**: `real-world-vfc-slice`
|
||||
- **Discovery Scope**: Extension of the existing pair catalog and pair eval CLI
|
||||
- **Key Findings**:
|
||||
- Pair eval already concatenates `benchmarks/pairs/catalog.toml` with `sast/catalog.toml`. A third file `vfc/catalog.toml` is the smallest seam.
|
||||
- CLI `--slice` choices are a closed set (`all`, `local`, `github`, `sast`). `vfc` must be added there and in tests that assert slice names.
|
||||
- `sast` already scores overlay with inventory fallback when nothing is promoted or demoted. `vfc` should use that path as the primary scorer and add a second inventory-only figure for the honesty dashboard.
|
||||
- `pair_gate` evaluates the full catalog but fails CI only on `local` 100%. Adding vfc pairs does not change the fail condition if names stay out of `LANGUAGE_MANIFESTS`.
|
||||
- Seed commits with public VCS snapshots exist: OpenSSL Heartbleed (`tls1_process_heartbeat`), Firefox libmar (`mar_insert_item`, CVE-2020-15667), Chromium FileReader (`ArrayBufferResult`, CVE-2019-5786). All three keep a sink-shaped call after the fix, so overlay/inventory may LEAK — that is the dashboard, not a 100% gate.
|
||||
- mozilla-central / gecko-dev must not be cloned. Phabricator and GitHub raw blobs are enough to extract one function. FixFox stays embargoed.
|
||||
|
||||
## Research Log
|
||||
|
||||
### Existing pair catalog and scoring
|
||||
|
||||
- **Context**: Brownfield extension; must not invent a second eval engine.
|
||||
- **Sources Consulted**: `src/openultrasast/pairs.py`, `cli.py`, `pair_gate.py`, `tests/test_pair_corpus.py`, `benchmarks/pairs/catalog.toml`, `benchmarks/pairs/sast/catalog.toml`, `benchmarks/pairs/README.md`
|
||||
- **Findings**:
|
||||
- `PairCase.slice` is a free string in the catalog loader; the closed set lives only in argparse and tests.
|
||||
- Overlay pair eval is gated on `case.slice == "sast"`.
|
||||
- GitHub pairs already use provenance comments plus catalog fields (`repo`, `commit`, `commit_url`, `cve`, `license`).
|
||||
- Pair names must not appear in `LANGUAGE_MANIFESTS`.
|
||||
- **Implications**: Add `vfc` as a concatenated catalog; extend the overlay branch to `sast` **or** `vfc`; keep the gate local-only.
|
||||
|
||||
### Harvestable public commits (OpenSSL, Firefox, Chromium)
|
||||
|
||||
- **Context**: User asked for those three projects and a labeled corpus that can actually be tested.
|
||||
- **Sources Consulted**:
|
||||
- OpenSSL `96db9023b881d7cd9f379b0c154650d6c108e9a3` (parent `0d7717fc9c83dafab8153cbd5e2180e6e04cc802`), `ssl/t1_lib.c` `tls1_process_heartbeat`, CVE-2014-0160
|
||||
- Firefox hg `b79b6cc78248eea7fda10bfb76aa273c19c9fa65` (parent `0c0f777161a9499dd149853ff62d356f75d16c2a`), `modules/libmar/src/mar_read.c` `mar_insert_item`, CVE-2020-15667 / Bug 1653371; Phabricator raw diff
|
||||
- Chromium `ba9748e78ec7e9c0d594e7edf7b2c07ea2a90449` (parent `9f3fccdba567f30ea39b08f57d8d5b49cd832b4a`), `FileReaderLoader::ArrayBufferResult`, CVE-2019-5786
|
||||
- Rejected: CVE-2019-11730 (SOP/file URI policy, not a sink pair); FixFox (embargoed to 2030); MFSA text without a commit
|
||||
- **Findings**:
|
||||
- Heartbleed fix adds a record-length check; `memcpy(bp, pl, payload)` remains. Inventory `c-unsafe-memory-copy` will fire on both sides (LEAK) until CWE-121 semantics exist.
|
||||
- Firefox fix is `int namelen` → `uint32_t namelen`; `memcpy(item->name, name, namelen + 1)` remains. Same LEAK class.
|
||||
- Chromium fix copies the ArrayBuffer on partial results (UAF). No memcpy/eval sink; expected MISS on inventory and overlay. Still a labeled pair.
|
||||
- GitHub raw and Phabricator `?diff=1` return the function text without cloning.
|
||||
- **Implications**: Seed three reviewed pairs. Tests assert presence and that the slice **runs**, not that pair_correct is 100%. Memory-safety pairs are allowed as known leaks (Req 6.3).
|
||||
|
||||
### Dataset pointers versus vendored dumps
|
||||
|
||||
- **Context**: `datasets.toml` already lists CVEfixes, MoreFixes, SVEN, CWE-Bench-Java. Roadmap asked for Firefox/Chromium/OpenSSL pointers.
|
||||
- **Findings**: Pointers are metadata only. CI must stay offline on vendored excerpts.
|
||||
- **Implications**: Add dataset rows for OpenSSL, gecko-dev / mozilla-firefox/firefox, and Chromium. Do not fetch dumps in tests.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
| Option | Description | Strengths | Risks / Limitations | Notes |
|
||||
|--------|-------------|-----------|---------------------|-------|
|
||||
| Concatenate a `vfc` catalog like `sast` | Third TOML file, same `[[pair]]` schema | Matches existing loader; no new types | CLI choices and tests must be updated | **Selected** |
|
||||
| Stuff vfc rows into `catalog.toml` | One file | Fewer paths | Mixes local/github with large-project VFCs | Rejected; sast already split |
|
||||
| New eval package | Separate scorer | Clean isolation | Duplicates pair eval | Rejected |
|
||||
| Clone megarepos in CI | Scan real trees | Maximum fidelity | Multi-GB, scrubbed commit messages, not isolated | Explicitly out of scope |
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### Decision: Concatenated `vfc` catalog, overlay primary, inventory sidecar
|
||||
|
||||
- **Context**: Need a labeled corpus that is offline, isolated, and honest about overlay versus regex.
|
||||
- **Alternatives Considered**:
|
||||
1. Inventory-only like `github` — misses the overlay question the user is evolving.
|
||||
2. Overlay-only like `sast` without a sidecar — cannot see whether overlay helped or hurt.
|
||||
3. Dual scoreboard with overlay primary and inventory sidecar — **selected**.
|
||||
- **Selected Approach**: `slice = "vfc"` uses the overlay pair path (inventory fallback when unadjudicated). `PairEvalResult.scorers["vfc"]` holds both `overlay` and `inventory` metrics. Pair gate still fails only on `local`.
|
||||
- **Rationale**: Reuses `sast` overlay machinery; satisfies Req 4 without making Youden a merge gate.
|
||||
- **Trade-offs**: Pair gate runtime grows by three small overlay scans. Acceptable.
|
||||
- **Follow-up**: Do not retune memcpy rules to force these pairs to PASS.
|
||||
|
||||
### Decision: Maintainer harvest script, vendored seed, no CI fetch
|
||||
|
||||
- **Context**: Grow the catalog later without cloning mozilla-central.
|
||||
- **Selected Approach**: `benchmarks/pairs/vfc/harvest.py` plus `recipes.toml`. CI never runs harvest. Seed files are committed excerpts with provenance headers.
|
||||
- **Rationale**: Req 5 and Req 7 (offline testability).
|
||||
- **Trade-offs**: Harvest needs network when a maintainer grows the set; tests mock extraction locally.
|
||||
|
||||
### Decision: Public hg counts as a version-control commit
|
||||
|
||||
- **Context**: Firefox canonical history is hg.mozilla.org; gecko-dev git SHAs are not the hg SHAs; GitHub 422s on hg identifiers.
|
||||
- **Selected Approach**: Record the hg revision and Phabricator/hg URL. Dataset pointers still list gecko-dev and mozilla-firefox/firefox.
|
||||
- **Rationale**: Brief forbids MFSA-without-commit, not hg. The MAR fix is a public VCS snapshot.
|
||||
- **Follow-up**: If a git SHA is later mapped, add it as an extra catalog field without changing the excerpt.
|
||||
|
||||
## Synthesis
|
||||
|
||||
- **Generalization**: `vfc` is another honesty slice, not a new product. Overlay-versus-inventory is a report field, not a new engine.
|
||||
- **Build vs adopt**: Adopt pair catalog schema, overlay eval, provenance comment style. Build only harvest extract + dual scoreboard payload.
|
||||
- **Simplification**: No Joern, no megarepo, no 90% gate, no rule changes to chase Youden on three memory/UAF pairs.
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- Overlay/inventory LEAK on Heartbleed and MAR memcpy-after-fix — document as known; tests must not assert PASS.
|
||||
- Chromium UAF is a MISS — still required so the corpus is not only memcpy.
|
||||
- `pair_gate` currently scans the full catalog; three extra pairs are small. If the catalog grows, gate should keep evaluating local-only for the fail condition (already true) and may later skip overlay slices for speed.
|
||||
- License: OpenSSL (OpenSSL License / SSLeay), Firefox MPL-2.0, Chromium BSD-3-Clause — headers must state them.
|
||||
- gecko-dev is archived; harvest should accept hg.mozilla.org and github.com/chromium/chromium / openssl/openssl as hosts.
|
||||
|
||||
## References
|
||||
|
||||
- [OpenSSL Heartbleed fix](https://github.com/openssl/openssl/commit/96db9023b881d7cd9f379b0c154650d6c108e9a3) — CVE-2014-0160
|
||||
- [Firefox MAR heap overflow](https://bugzilla.mozilla.org/show_bug.cgi?id=1653371) — CVE-2020-15667, hg `b79b6cc78248`
|
||||
- [Chromium FileReader UAF](https://github.com/chromium/chromium/commit/ba9748e78ec7e9c0d594e7edf7b2c07ea2a90449) — CVE-2019-5786
|
||||
- `.kiro/specs/real-world-vfc-slice/brief.md`
|
||||
- `.kiro/steering/roadmap.md`
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"feature_name": "real-world-vfc-slice",
|
||||
"created_at": "2026-09-04T13:19:17Z",
|
||||
"updated_at": "2026-09-04T13:40:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
# Implementation Plan
|
||||
|
||||
- [x] 1. Foundation: vfc catalog load and slice command
|
||||
- [x] 1.1 Load a vfc catalog and accept slice vfc on the pairs command
|
||||
- Default pair catalog load includes a vfc catalog file the same way it already includes the sast catalog.
|
||||
- The pairs command accepts slice `vfc` in addition to all, local, github, and sast.
|
||||
- Dataset pointers name OpenSSL, Firefox/gecko-dev, and Chromium without vendoring those trees.
|
||||
- An empty or seed-ready vfc catalog file exists so load does not skip the slice by omission.
|
||||
- Observable: `ousast pairs --slice vfc --json` exits 0 and the loaded slice set can contain `vfc`.
|
||||
- _Requirements: 1.1, 1.3, 7.2_
|
||||
- _Boundary: VfcCatalog_
|
||||
|
||||
- [x] 2. Core: reviewed seed pairs
|
||||
- [x] 2.1 Vendor the OpenSSL Heartbleed function pair
|
||||
- Isolated `tls1_process_heartbeat` before and after the public Heartbleed fix, same relative path.
|
||||
- Provenance header and catalog row include repo, commit, parent, public commit URL, CVE-2014-0160, and license.
|
||||
- Expected CWE/sink/evidence are stated; memcpy remaining after the fix is an accepted honesty leak, not a reason to change inventory rules.
|
||||
- Observable: catalog load yields a vfc pair whose name identifies OpenSSL and whose vuln and fixed files exist.
|
||||
- _Requirements: 1.2, 1.4, 2.1, 2.2, 2.3, 2.4, 6.1, 6.2, 6.3, 6.4_
|
||||
- _Boundary: SeedExcerpts_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [x] 2.2 Vendor the Firefox libmar function pair
|
||||
- Isolated `mar_insert_item` before and after the public CVE-2020-15667 commit, same relative path.
|
||||
- Provenance records the mozilla-central revision and public commit URL (hg or Phabricator is acceptable; MFSA-only is not).
|
||||
- License is MPL-2.0; expected CWE/sink/evidence are stated.
|
||||
- Observable: catalog load yields a vfc pair whose name identifies Firefox and whose files exist.
|
||||
- _Requirements: 1.2, 1.4, 2.1, 2.2, 2.3, 2.4, 6.1, 6.2, 6.3, 6.4_
|
||||
- _Boundary: SeedExcerpts_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [x] 2.3 Vendor the Chromium FileReader function pair
|
||||
- Isolated `ArrayBufferResult` before and after the public CVE-2019-5786 commit, same relative path.
|
||||
- Provenance header and catalog row include chromium repo, commit, parent, commit URL, CVE, and BSD-3-Clause.
|
||||
- UAF-without-inventory-sink is an accepted honesty miss; do not vendor exploit HTML.
|
||||
- Observable: catalog load yields a vfc pair whose name identifies Chromium and whose files exist.
|
||||
- _Requirements: 1.2, 1.4, 2.1, 2.2, 2.3, 2.4, 6.1, 6.2, 6.3, 6.4_
|
||||
- _Boundary: SeedExcerpts_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [x] 3. Harvest without megarepos
|
||||
- [x] 3.1 (P) Add a maintainer harvest procedure that extracts one function
|
||||
- A recipe names parent commit, fix commit, path, function, CVE, and license; advisory-only recipes are rejected.
|
||||
- Harvest fetches blobs over HTTP and writes provenance-headed excerpts; it does not clone mozilla-central, gecko-dev, or Chromium.
|
||||
- FixFox and other embargoed dumps are not referenced as fetch sources.
|
||||
- Function extract is unit-tested on a local string so CI stays offline.
|
||||
- Observable: harvest extract on a fixture function round-trips the function body; pytest does not perform a live fetch.
|
||||
- _Requirements: 1.4, 5.1, 5.2, 5.3, 5.4, 5.5, 7.3_
|
||||
- _Boundary: HarvestTool_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [x] 4. Integration: overlay versus inventory scoreboard
|
||||
- [x] 4.1 Score vfc with overlay primary and inventory sidecar
|
||||
- vfc pairs use the overlay pair path with inventory fallback when overlay does not adjudicate.
|
||||
- Machine-readable pairs output includes `scorers.vfc.overlay` and `scorers.vfc.inventory`.
|
||||
- Primary `per_slice.vfc` matches the overlay scorer.
|
||||
- Pair gate fail reasons stay local-only; vfc Youden is not a merge condition.
|
||||
- Observable: `ousast pairs --slice vfc --json` contains both scorer objects and the OpenSSL, Firefox, and Chromium pair names.
|
||||
- _Requirements: 3.3, 3.4, 4.1, 4.2, 4.3, 4.4, 7.2_
|
||||
- _Boundary: DualScorer_
|
||||
- _Depends: 1.1, 2.1, 2.2, 2.3_
|
||||
|
||||
- [x] 5. Validation: honesty gates and offline tests
|
||||
- [x] 5.1 Prove vfc is testable and is not a smoke or merge gate
|
||||
- Default catalog load sees slice `vfc` and every vfc file on disk.
|
||||
- vfc pair names are absent from the stage-1 detection fixture list.
|
||||
- Pair efficiency gate still passes on local 100% with vfc present.
|
||||
- Tests do not require 100% vfc pair_correct or a Youden floor.
|
||||
- Extra-free evaluation still completes.
|
||||
- Observable: extra-free pytest covering catalog load, gate pass, and vfc JSON is green.
|
||||
- _Requirements: 3.1, 3.2, 3.4, 5.5, 7.1, 7.3, 7.4_
|
||||
- _Boundary: VfcTestMatrix, PairGatePolicy_
|
||||
- _Depends: 4.1_
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Seed commits: OpenSSL `96db9023b881d7cd9f379b0c154650d6c108e9a3` (parent `0d7717fc9c83dafab8153cbd5e2180e6e04cc802`); Firefox hg `b79b6cc78248eea7fda10bfb76aa273c19c9fa65` (parent `0c0f777161a9499dd149853ff62d356f75d16c2a`); Chromium `ba9748e78ec7e9c0d594e7edf7b2c07ea2a90449` (parent `9f3fccdba567f30ea39b08f57d8d5b49cd832b4a`).
|
||||
- Do not retune `c-unsafe-memory-copy` to make Heartbleed or MAR pair_correct.
|
||||
- Do not clone mozilla-central; do not vendor FixFox.
|
||||
@@ -39,9 +39,9 @@ OpenUltraSAST can propose sinks with regex and prove a few of them in a sandbox,
|
||||
## Direct Implementation Candidates
|
||||
|
||||
- [ ] Record inventory vs overlay Youden side-by-side in `ousast pairs --slice sast` JSON (small report field, not a new spec).
|
||||
- [ ] Add Firefox/gecko-dev, Chromium, OpenSSL pointers to `benchmarks/pairs/datasets.toml`.
|
||||
- [x] Add Firefox/gecko-dev, Chromium, OpenSSL pointers to `benchmarks/pairs/datasets.toml`.
|
||||
|
||||
## Specs (dependency order)
|
||||
|
||||
- [ ] tree-sitter-overlay-extra -- Optional extra: tree-sitter + grammars, one CST walker, `quick` stays zero-dep. Dependencies: none
|
||||
- [ ] real-world-vfc-slice -- Harvest reviewed function-level vuln/fix pairs from large projects (Firefox/gecko-dev, Chromium, OpenSSL, …) as an honesty slice. Dependencies: none
|
||||
- [x] tree-sitter-overlay-extra -- Optional extra: tree-sitter + grammars, one CST walker, `quick` stays zero-dep. Dependencies: none
|
||||
- [x] real-world-vfc-slice -- Harvest reviewed function-level vuln/fix pairs from large projects (Firefox/gecko-dev, Chromium, OpenSSL, …) as an honesty slice. Seed is testable (`ousast pairs --slice vfc`). Dependencies: none
|
||||
|
||||
Reference in New Issue
Block a user