marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(pair-corpus-honesty): spec, review record, roadmap; close real-world-vfc-slice as superseded

Browse Source 
Task 7.2 and the spec itself (brief, requirements incl. Req 9/10
amendment, research, design, tasks with eight review rounds recorded).

- real-world-vfc-slice: spec.json closed, superseded_by
  pair-corpus-honesty; brief carries the closure note.
- roadmap: Direction section, per-slice baselines (sast 4/11, vibe-py
  13/35, vfc-js 5/17, vfc 0/176, agent-vfc 0/29 title tier), spec order
  pair-corpus-honesty -> corpus-seeded-mechanisms -> reachability-flow-model
  -> overlay-ir-completeness -> guard-dominance-regime.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-05 10:45:43 +02:00
co-authored by Claude Fable 5.1
parent 25f1512eb9
commit 3eba6e10a0
9 changed files with 997 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/pair-corpus-honesty/brief.md
+56
View File
@@ -0,0 +1,56 @@
# Brief: pair-corpus-honesty
Supersedes `real-world-vfc-slice` for the corpus and scorer plane. That spec is closed: all tasks done, 17 tests green, three seed projects plus curl landed. Its residuals are corpus quality and scorer fidelity, not seed coverage, so they get a new spec instead of reopening.
## Problem
The pair scoreboard cannot tell us whether the overlay engine improved, because the scoreboard itself is lossy:
- Overlay scoring counts only `promote`. Coverage records with a real flow (the Java sqli vuln is already found that way) count as nothing. Every future sink alias lands as coverage, so the scorer will hide the work.
- 129 of 176 vfc labels say `sink = "unknown"` and only 4 carry a rule id. Matching falls through to CWE-string equality, and CWE-119 is a catch-all. Any memcpy promotion anywhere in the file counts as a hit.
- 25 curl pairs (50 files) are `parse_failed`. The harvester anchors on the first mention of the function name, which is usually the doc comment above it, so excerpts start mid-comment.
- Two sast pairs cannot be labeled as vendored: the Juliet strcpy vuln and fixed bodies are byte-identical (buffer declarations lost), and the OWASP Java hash pair needs `benchmark.properties`.
- There is no vibe-coded or web corpus at all. Node, TypeScript, and modern Python web frameworks are where operators are writing code now, and where AI-authored code concentrates.
- Overlay-versus-inventory is only reported for vfc, and there is no per-mechanism or per-provenance view, so a change that helps C guards and hurts Python injection is invisible.
Measured 2026-09-04 (extra on): sast 5/11 recall, 4/11 pair-correct, Youden +27%; vfc 6/176 recall, 0 pair-correct, Youden −14%; overlay column identical to inventory on vfc.
## Desired Outcome
A scoreboard that is honest per slice, per mechanism, and per provenance, on corpora that include real Node/TypeScript/Python web code and agent-authored code, so the two engine specs that follow (`overlay-ir-completeness`, `guard-dominance-regime`) can be measured and the improve loop can be branched by profile without overfitting one slice.
## Approach
Fix the scorer and labels first (no engine change). Anchor harvest at the declarator and add line-range and hunk extraction so Python and JS/TS functions can be harvested. Add a closed mechanism vocabulary and a closed provenance set to every pair. Add three slices: `vibe-py` (Real-Vuln-Benchmark, 40 LLM-generated plus 26 human Python web repos with false-positive traps), `vfc-js` (SecBench.js fix commits and sink locations), `agent-vfc` (agent-trailer security-fix commits, human reviewed). Add a deterministic provenance fingerprint at scan time. Report per-profile metrics and make the improve loop accept only when no profile regresses.
## Scope
- **In**: scorer fidelity; label schema; harvest anchor and modes; corpus hygiene for sast; three new slices; dataset pointers; provenance fingerprint; per-profile and per-mechanism dashboard; per-profile improve-loop gate; offline tests.
- **Out**: any change to taint, facts, CST walker, or dispositions (engine specs); tolerating tree-sitter ERROR nodes (engine); BaxBench generation and exploit-labelled slices (needs deep mode; later); profile-specific fact tables; making any dashboard a merge gate; vendoring SecBench.js test files (no license).
## Boundary Candidates
- Scorer and label schema (`pairs.py`, catalog loaders) vs engine records (`OverlayRecord` gains one additive field)
- Harvest tooling vs vendored excerpts
- Provenance fingerprint (preprocess/manifest) vs improve-loop gate
## Out of Boundary
- Overlay dispositions, taint, CST walker (`propose-adjudicate-prove`, `tree-sitter-overlay-extra`, and the two follow-up engine specs)
- Rule policy internals and CWE severity (`harnessx-self-improving-rulesets`)
- Stage plan, sandbox, hunter (`three-stage-scan`)
## Upstream / Downstream
- **Upstream**: pair catalog schema; overlay eval path; `build_pair_signals`; `run_improvement` accept gate; preprocess tags.
- **Downstream**: `overlay-ir-completeness` and `guard-dominance-regime` measure against these slices and vocabularies; later `baxgen` slice reuses provenance and mechanism fields.
## Existing Spec Touchpoints
- **Supersedes**: `real-world-vfc-slice` (corpus/scorer plane). Its seed pairs, harvest script, and dual scoreboard are kept and extended.
- **Extends**: roadmap direct candidate "record inventory vs overlay Youden side-by-side in sast JSON".
- **Adjacent**: `propose-adjudicate-prove` (one additive field on `OverlayRecord`), `harnessx-self-improving-rulesets` (accept gate grows a per-profile clause, ledger format unchanged).
## Constraints
Core `dependencies = []`. Harvest never runs in CI. Every vendored file carries license and provenance. No LLM in classification. Youden on any non-local slice is never a merge gate. Held-out halves are declared in the catalog, not chosen at run time.
.kiro/specs/pair-corpus-honesty/design.md
+406
View File
@@ -0,0 +1,406 @@
# Design Document
## Overview
**Purpose.** This feature makes the pair scoreboard honest and gives it three web and agent-authored slices, so the follow-up engine specs can be measured per mechanism and per provenance instead of by one Youden on a lossy scorer.
**Users.** Maintainers grow and score corpora. Engineers evolving the overlay read per-mechanism and per-profile numbers. The improve loop reads holdout metrics per profile. Operators see a provenance value in their manifest.
**Impact.** `pairs.py` counts coverage and matches by function. Catalogs carry `mechanism`, `provenance`, `split`, and `known_limit`. Harvest anchors at the declarator and gains line-range and hunk modes. Three catalogs are added. `preprocess` computes provenance. `run_improvement` grows a per-profile clause. `real-world-vfc-slice` is closed and superseded on the corpus plane.
### Goals
- Scorer counts every adjudicated detection and reports parser and adjudication loss.
- Every pair is attributable: function, mechanism, provenance, split, license.
- `vibe-py`, `vfc-js`, `agent-vfc` load offline and score.
- Per-profile and per-mechanism metrics; per-profile improve acceptance.
- No gate changes; Youden on non-local slices is never a merge condition.
### Non-Goals
- Any change to taint, facts, CST walker, dispositions, or ERROR-node tolerance.
- BaxBench generation slices (later, after deep mode).
- Profile-specific fact tables, thresholds, or ledger files.
- LLM classification.
## Boundary Commitments
### This Spec Owns
- `pairs.py` scoring semantics, payload fields, slice list, catalog validation.
- `src/openultrasast/semantic/functions.py` (added in the round-3 debug plan): declarator-aware named function ranges shared by the overlay's range filling, the scorer, and the catalog generator; stdlib plus `semantic/ir.py` only, never imports `overlay` or `pairs`. The scorer boundary owns naming; `ir.py` and `cst.py` stay untouched.
- Catalog label schema and `benchmarks/pairs/mechanisms.toml`.
- Harvest tooling (shared library plus per-slice recipes).
- `benchmarks/pairs/{sast,vfc,vibe-py,vfc-js,agent-vfc}` contents and READMEs, `datasets.toml`.
- `provenance.py` and its manifest field.
- The per-profile clause in the improve accept gate.
### Out of Boundary
- `semantic/*` except one additive optional field on `OverlayRecord`, the range-filling expression in `overlay._flows_for`, and the new `semantic/functions.py` helper listed above.
- `ruleset/store.py` ledger format, CWE policy, project score.
- Stage plan, sandbox, hunter, MCP.
- `LANGUAGE_MANIFESTS` and the stage-1 gate.
### Allowed Dependencies
- Existing `PairCase`, `evaluate_pair`, `adjudicate`, `parse_file` (for function line ranges), `build_pair_signals`, `run_improvement`, `preprocess_repository`, `write_manifest`.
- Stdlib only for new core modules. `git` binary via subprocess for trailers, optional.
- Public hosts for harvest (maintainer only): raw.githubusercontent.com, Real-Vuln-Benchmark manifest, upstream npm package repos.
### Revalidation Triggers
- Changing `OverlayRecord` fields beyond the one optional `function` field.
- Adding a fifth provenance profile.
- Making any slice other than `local` fail CI.
- Vendoring a file without a license field.
- Changing the mechanism vocabulary without bumping its version.
## Architecture
### Existing Architecture Analysis
`load_pair_catalog` concatenates `catalog.toml`, `sast/catalog.toml`, `vfc/catalog.toml`. `evaluate_pair` routes `sast` and `vfc` through `_evaluate_overlay_pair`, which counts only `promote` and falls back to inventory when nothing adjudicated. `scorers` is populated only for `vfc`. `build_pair_signals` emits miss/fp rows with pair name. `run_improvement` accepts on four clauses. `preprocess.detect_tags` is deterministic per file. `vfc/harvest.py` extracts by name with a comment-blind search.
### Architecture Pattern & Boundary Map
Selected: **label-driven scorer plus shared harvest plus deterministic profile**. One scorer, one harvest library, one provenance function; slices are data.
```mermaid
flowchart TD
subgraph data["Catalogs (data)"]
MECH[mechanisms.toml]
SAST[sast]
VFC[vfc]
VPY[vibe-py]
VJS[vfc-js]
AGV[agent-vfc]
end
subgraph harvest["Harvest (maintainer)"]
HL[harvest library: name / line_range / hunk]
REC[recipes.toml per slice]
end
subgraph score["Scorer"]
LOAD[load_pair_catalog + validate]
EVAL[evaluate_pair: promote + coverage, function match]
PAY[payload: per_slice, scorers, per_profile, per_mechanism, loss]
end
subgraph scan["Scan"]
PROV[provenance fingerprint]
MAN[manifest.provenance]
end
subgraph improve["Improve loop"]
SIG[pair signals + profile]
GATE[accept gate + per-profile holdout clause]
end
REC --> HL --> VPY
HL --> VJS
HL --> AGV
HL --> VFC
MECH --> LOAD
SAST --> LOAD
VFC --> LOAD
VPY --> LOAD
VJS --> LOAD
AGV --> LOAD
LOAD --> EVAL --> PAY
PAY --> SIG --> GATE
PROV --> MAN
MAN -. names profile .-> PAY
```
**Key decisions**
- Coverage with a source counts like promote on both sides.
- Function-scoped matching when the label names a function; CWE-only rows are `weak_label`.
- Provenance is closed and deterministic; `synthetic` comes only from the catalog.
- Profiles branch the accept decision, not the ledger.
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| CLI | existing `ousast pairs --slice` | new slice names, `--profile` filter | argparse choices |
| Catalog | TOML `[[pair]]`, `[[pair.expected]]` | new fields | loader validates |
| Vocabulary | `benchmarks/pairs/mechanisms.toml` | closed set, versioned | data, not code |
| Harvest | stdlib urllib, `benchmarks/pairs/harvest.py` | name, line_range, hunk modes | maintainer only |
| Provenance | `src/openultrasast/provenance.py` | fingerprint | stdlib, optional `git` |
| Improve | `improve/evolve.py` | per-profile clause | existing gate kept |
| Tests | pytest, extra-free default | offline | overlay asserts skip without extra |
## File Structure Plan
```
benchmarks/pairs/
├── mechanisms.toml # closed vocabulary v1 with one-line definitions
├── harvest.py # shared library: fetch, extract_by_name (comment-blind), extract_line_range, extract_hunk, python_block, provenance_header
├── datasets.toml # + Real-Vuln-Benchmark, SecBench.js, AIDev, BaxBench, SecurityEval, CodeSecEval, CWEval, CyberSecEval
├── sast/ # juliet strcpy fixture restored; owasp-java-hash known_limit
├── vfc/
│ ├── harvest.py # thin shim -> ../harvest.py (keeps existing tests)
│ ├── recipes.toml # + function already present; curl re-harvested
│ └── generate_catalog.py # emits function, mechanism, provenance, split
├── vibe-py/
│ ├── README.md # educational/CTF caveat, license, matching rule
│ ├── recipes.toml # repo, commit, file, line range, function, cwes, authorship, is_vulnerable
│ ├── catalog.toml
│ └── <pair>/{vuln,fixed}.py
├── vfc-js/
│ ├── README.md
│ ├── recipes.toml # upstream repo, parent, fixCommit, file, sink line, class
│ ├── catalog.toml
│ └── <pair>/{vuln,fixed}.js|ts
└── agent-vfc/
├── README.md # review checklist
├── recipes.toml # repo, parent, commit, file, trailer, mechanism, reviewer
├── catalog.toml
└── <pair>/{vuln,fixed}.<ext>
src/openultrasast/
├── pairs.py # coverage counting, function match, weak_label, per_profile/per_mechanism, loss counters, validation
├── provenance.py # fingerprint(root) -> Provenance(value, signals)
├── preprocess.py # calls provenance once per tree
├── reports.py # manifest.provenance
├── semantic/overlay.py # OverlayRecord.function: str | None = None (additive)
├── improve/evolve.py # per-profile holdout clause
└── cli.py # slice choices, --profile
tests/
├── test_pair_corpus.py # extended
├── test_pair_harvest.py # renamed from test_vfc_harvest.py, + modes
├── test_provenance.py
└── test_improve.py # per-profile rejection
```
### Modified Files
- `src/openultrasast/pairs.py` — see above; `OVERLAY_SLICES` grows; `PairCase` gains `provenance`, `mechanism` (on expected), `split`, `known_limit`.
- `src/openultrasast/semantic/overlay.py` — fill `function` from the `FunctionIR` whose range contains the record line; default `None`.
- `src/openultrasast/improve/evolve.py` — `_Eval` computes `per_profile` on holdout; new reject reason `profile_regression:<profile>`.
- `src/openultrasast/preprocess.py`, `reports.py`, `cli.py` — wiring only.
- `benchmarks/pairs/README.md`, `.kiro/steering/roadmap.md` — slice list and spec order.
## System Flows
### Overlay pair scoring
```mermaid
flowchart TD
A[scan vuln tree, scan fixed tree] --> B{any promote/coverage/demote?}
B -->|no| C[inventory fallback, loss counters set]
B -->|yes| D[detections = promote + coverage with source]
D --> E{expected row}
E -->|function| F[match inside function lines and sink/rule/cwe]
E -->|sink or rule_id| G[match by sink or rule_id]
E -->|cwe only| H[weak_label, no match]
F --> I[vuln matched?]
G --> I
I --> J[fixed side: detections = leaks under silent policy]
J --> K[outcome + counters: parse_failed, unadjudicated, weak_label, known_limit]
```
Known-limit pairs run the same flow but are excluded from `achievable` metrics and listed under `known_limit`.
### Improve acceptance
```mermaid
flowchart LR
P[candidate ledger] --> Q[existing four clauses]
Q -->|pass| R[holdout pairs per profile]
R -->|any profile regresses > tolerance| X[reject profile_regression]
R -->|none| Y[accept, persist]
Q -->|fail| X2[reject as today]
```
## Requirements Traceability
| Requirement | Summary | Components | Flows |
|-------------|---------|------------|-------|
| 1.1–1.6 | count coverage, function match, weak_label, side-by-side, loss, local only | Scorer | overlay pair scoring |
| 2.1–2.6 | mechanism, provenance, known_limit, validation, split, function | LabelSchema, Vocabulary | load |
| 3.1–3.6 | comment-blind anchor, modes, python, curl re-harvest, offline | HarvestLibrary | harvest |
| 4.1–4.3 | Juliet fixture, Java hash known_limit, achievable counts | sast catalog | load, score |
| 5.1–5.6 | three slices, licenses, dataset pointers | SliceCatalogs | harvest, load |
| 6.1–6.4 | deterministic provenance in manifest | ProvenanceFingerprint | scan |
| 7.1–7.5 | per-profile/per-mechanism payload, holdout clause, cap, min size | Scorer, ProfileGate | improve acceptance |
| 8.1–8.4 | offline tests, gates unchanged | tests | CI |
## Components and Interfaces
| Component | Domain | Intent | Req Coverage | Dependencies | Contracts |
|-----------|--------|--------|--------------|--------------|-----------|
| Scorer | Eval | honest pair outcomes and payload | 1, 7.1 | adjudicate, parse_file P0 | Service, State |
| LabelSchema | Catalog | validated pair and expected fields | 2, 4 | tomllib P0 | State |
| Vocabulary | Catalog | closed mechanism set | 2.1 | none | State |
| HarvestLibrary | Maintainer | extract functions by name, range, hunk | 3, 5 | urllib P1 | Batch |
| SliceCatalogs | Catalog | vibe-py, vfc-js, agent-vfc | 5 | HarvestLibrary P0 | State |
| ProvenanceFingerprint | Scan | deterministic profile | 6 | preprocess P0, git P2 | Service |
| ProfileGate | Improve | per-profile holdout clause | 7.2–7.4 | Scorer P0, evolve P0 | Batch |
### Eval
#### Scorer
| Field | Detail |
|-------|--------|
| Intent | Count what the overlay found; report loss; slice by profile and mechanism |
| Requirements | 1.1–1.6, 7.1 |
```python
@dataclass(frozen=True)
class PairOutcome: # existing fields plus
provenance: str
mechanisms: tuple[str, ...]
known_limit: str | None
split: str
weak_labels: int
parse_failed_vuln: int
parse_failed_fixed: int
unadjudicated_vuln: int
unadjudicated_fixed: int
detection_kinds: tuple[str, ...] # promote | coverage | inventory
@dataclass(frozen=True)
class PairEvalResult: # existing fields plus
per_profile: dict[str, PairCorpusMetrics]
per_mechanism: dict[str, PairCorpusMetrics]
achievable: dict[str, PairCorpusMetrics] # per slice, known_limit excluded
known_limit: tuple[str, ...]
```
- Preconditions: catalog validated.
- Postconditions: `scorers[slice]` has `overlay` and `inventory` for every overlay slice, and `hunter` when a model or scripted client is available; counters are non-negative.
- Invariants: `local` is the only slice `pair_gate` may cite in `reasons`.
Matching rule, in order: (1) if `expected.function`, record must lie inside that function's lines (from `OverlayRecord.function` or from `parse_file` ranges on the vulnerable file); (2) `rule_id in proposal_id` or `sink in sinks`; (3) CWE equality only when (1) or (2) also holds; rows with none of function, sink, rule_id are `weak_label` and never match.
### Catalog
#### LabelSchema and Vocabulary
Pair fields added: `provenance` (required for new slices, default `synthetic` for sast, `human` for vfc and github, `human` for local), `split` (`train` default, `holdout` declared), `known_limit` (string reason or absent). Expected row fields added: `mechanism` (required), `function` (required and must be a named function, never `<anon>`, for every slice in `OVERLAY_SLICES` except `sast`; the loader enforces it). Sink matching is exact against the record's fact sink ids, or through the facts' `rule_ids` alias table; never substring. Rows whose recipe carries `reviewer = "pending"` load at `review_tier = "title"` (see the Amendment below; the earlier candidates file is retired).
`mechanisms.toml` v1:
```
version = "1"
[[mechanism]] id = "source_reaches_sink"
[[mechanism]] id = "container_taint"
[[mechanism]] id = "config_sink_weak_literal"
[[mechanism]] id = "unchecked_length_copy"
[[mechanism]] id = "unchecked_alloc_size"
[[mechanism]] id = "null_deref_unguarded"
[[mechanism]] id = "signed_size_operand"
[[mechanism]] id = "fixed_buffer_capacity"
[[mechanism]] id = "identity_from_request_body"
[[mechanism]] id = "missing_auth_guard"
[[mechanism]] id = "secret_in_client"
[[mechanism]] id = "permissive_default"
[[mechanism]] id = "path_join_user_input"
[[mechanism]] id = "prototype_pollution"
[[mechanism]] id = "redos"
[[mechanism]] id = "uaf"
[[mechanism]] id = "type_confusion"
[[mechanism]] id = "validation_strength"
[[mechanism]] id = "cross_artifact"
[[mechanism]] id = "other"
```
Each row carries a one-line `definition`. Loader rejects unknown ids.
### Maintainer
#### HarvestLibrary
Batch contract:
- Trigger: `python benchmarks/pairs/harvest.py <slice> [recipe]`. CI never runs it.
- Modes: `name` (comment-blind declarator search: strip `/* */`, `//`, `#`, and string literals to a mask before searching; the match line must not be inside a mask and must be followed by `(` and a `{` before `;`), `line_range` (exact lines from both blobs), `hunk` (function enclosing the first changed hunk of `git diff parent..commit -- file`, obtained from the two blobs with a stdlib diff).
- Python: block by indentation from the `def` line to the first line with lower indentation.
- Output: provenance header plus body; recipe rejected without parent, commit, license.
- Idempotent per recipe name.
### Scan
#### ProvenanceFingerprint
```python
@dataclass(frozen=True)
class Provenance:
value: str # human | agent | mixed | synthetic
signals: tuple[str, ...]
def fingerprint(root: Path, *, git: bool = True, sample_commits: int = 200) -> Provenance: ...
```
Signals: paths `.claude/`, `CLAUDE.md`, `AGENTS.md`, `.cursor/`, `.cursorrules`, `.github/copilot-instructions.md`, `.windsurf/`; text markers `Generated with Claude Code`, `<meta name="author" content="Lovable"`, `Co-Authored-By: Claude`, `Co-authored-by: Codex`, `Co-authored-by: Cursor`, `Co-authored-by: Copilot`, `Devin`, `google-labs-jules`; git trailers over the last `sample_commits`. `agent` when the trailer share is at least 0.85, else `mixed` when any signal, else `human`. Never `synthetic` from a scan.
### Improve
#### ProfileGate
`run_improvement` gains `profile_tolerance: float = 0.0` and `min_holdout_pairs: int = 5`. After the four existing clauses pass, evaluate holdout pairs per profile before and after; reject with `profile_regression:<profile>` when pair-correct or Youden drops by more than the tolerance for any profile with at least the minimum pairs; profiles under the minimum are reported, not gated. Profiles are the closed provenance set; at most four.
## Data Models
Catalog row (additions in bold):
- `name`, `slice`, `language`, `origin`, `license`, `repo`, `commit`, `parent`, `commit_url`, `cve`, `vuln`, `fixed`, `relpath`, `min_recall`, `fix_policy`, **`provenance`**, **`split`**, **`known_limit`**
- `[[pair.expected]]`: `cwe`, `class`, `path`, `rule_id`, `sink`, `evidence`, **`function`**, **`mechanism`**, `line`
Payload additions:
```json
{
"per_profile": {"agent": {...}, "human": {...}, "synthetic": {...}},
"per_mechanism": {"identity_from_request_body": {...}},
"achievable": {"sast": {...}},
"known_limit": ["owasp-java-hash"],
"loss": {"vfc": {"parse_failed_files": 0, "unadjudicated_proposals": 120, "weak_labels": 0}}
}
```
Manifest addition: `"provenance": {"value": "agent", "signals": ["AGENTS.md", "trailer:Claude 0.91"]}`.
## Error Handling
| Condition | Response |
|-----------|----------|
| Unknown mechanism or provenance in catalog | load fails loud with pair name |
| `sink = "unknown"` without function or rule_id | load fails loud; existing vfc rows are migrated by task 2.4 |
| Harvest name not found or only in comments | `RecipeError`, no partial file |
| Harvest HTTP failure | non-zero exit, no file |
| Git unavailable at scan | provenance from file signals only; signal list says `git:unavailable` |
| Profile under minimum holdout | reported in improve journal, not gated |
| Semantic extra absent | overlay slices fall back to inventory as today; loss counters show it |
## Testing Strategy
- Unit: coverage-with-source counts on both sides; function-scoped match rejects a promotion outside the function; CWE-only row is `weak_label`; catalog rejects unknown mechanism; harvest name mode skips a comment mention and anchors on the declarator; line-range and hunk modes round-trip local strings; Python block extraction; provenance returns `human` with no signals, `mixed` with `AGENTS.md`, `agent` with a synthetic git log of trailers.
- Integration: `evaluate_catalog` on each new slice returns per-profile and per-mechanism metrics; `run_improvement` rejects a change that regresses the `agent` holdout while improving `synthetic`; Juliet strcpy pair is no longer byte-identical; owasp-java-hash appears under `known_limit`.
- CLI: `pairs --slice vibe-py --json`, `vfc-js`, `agent-vfc` exit zero; `--profile agent` filters.
- Regression: stage-1 gate, map gate, and local pair gate outputs unchanged; extra-free suite green; new slice names absent from `LANGUAGE_MANIFESTS`.
- Do not assert any Youden floor on non-local slices.
## Security Considerations
Vendored excerpts are public code with license headers. Harvest talks only to listed public hosts. Agent-authored pairs include no secrets; the review checklist requires redaction of any token found in the excerpt (redaction module already exists). No exploit payloads from SecBench.js are vendored.
## Migration Strategy
1. Land scorer changes with existing catalogs and record the new numbers in the roadmap (expect vfc recall to drop under function matching).
2. Migrate existing vfc rows: add `function` from recipes, add `mechanism`, set `provenance = "human"`, declare a holdout half.
3. Re-harvest curl; restore Juliet; mark Java hash.
4. Add slices one at a time; each lands with its README and dataset pointers.
5. Wire provenance and the profile clause last; default tolerance zero.
`real-world-vfc-slice` stays in the tree as closed history; its `spec.json` gains `superseded_by`.
## Amendment 2026-09-05: review tiers and pointer slices
Discovered while growing the corpus: the license rule (Req 5.4) and the binary review rule (Req 5.3) capped the vibe and web slices at tens of pairs. Two additive changes lift the cap without weakening honesty.
**Review tiers.** `PairCase.review_tier` in `{seeded, advisory, title, reviewed}` with defaults per slice: sast and vfc `advisory` (public CVE or benchmark label), vibe-py `seeded`, vfc-js `advisory`, agent-vfc `title` until `reviewer` is set, local `reviewed`. `PairEvalResult.per_tier` is reported next to `per_profile`. `evolve.evaluate_profiles` filters holdout pairs to `seeded | reviewed` before computing per-profile metrics. `catalog_gen` writes `title` rows into the catalog (the candidates file is retired) and stamps `review_tier` from `reviewer`.
**Pointer slices.** Catalog rows may carry `vendored = false` and no `vuln`/`fixed` paths; the recipe fields (repo, parent, commit, path, mode, line, fix_path, fix_line) travel in the catalog row. `pairs.evaluate_pair` resolves such rows through `harvest.materialize_pointer(case, cache_dir)` which fetches into `OPENULTRASAST_PAIR_CACHE` (default `~/.cache/openultrasast/pairs/<slice>/<name>/`) and returns the two paths; when `OPENULTRASAST_PAIRS_NETWORK` is not `1` the pair is skipped and `degradations` gains `{"stage": "pairs", "reason": "pointer_pair_skipped", "pair": name}`. `load_pair_catalog` loads pointer rows but tests and gates call `select_vendored(cases)`. `ousast pairs --pointers` enables network for one run; CI never sets it. `.gitignore` gains the cache path. Provenance of pointer pairs follows the recipe (`agent` for the Real-Vuln LLM repositories, whose authorship column is upstream-reviewed).
Boundary: unchanged. Pointer harvesting reuses `harvest.py` modes; no new engine dependency; the redaction pass applies to cached files as well.
.kiro/specs/pair-corpus-honesty/requirements.md
+136
View File
@@ -0,0 +1,136 @@
# Requirements Document
## Introduction
Engineers evolving OpenUltraSAST's overlay cannot trust the pair scoreboard: coverage detections are dropped, vfc labels mostly say `sink = "unknown"`, a quarter of the vfc corpus fails to parse because excerpts start mid-comment, two sast pairs are unlabelable as vendored, and there is no Node/TypeScript/Python web corpus and no agent-authored corpus. This spec makes the scoreboard honest per slice, per mechanism, and per provenance, adds three web and agent-authored slices, and lets the improve loop accept a change only when no provenance profile regresses. It supersedes `real-world-vfc-slice` for the corpus and scorer plane.
## Boundary Context
- **In scope**: pair scorer fidelity; catalog label schema (function, mechanism, provenance, known limits, held-out split); harvest anchoring and extraction modes; sast corpus hygiene; `vibe-py`, `vfc-js`, `agent-vfc` slices; dataset pointers; deterministic provenance fingerprint at scan time; per-profile and per-mechanism dashboard; per-profile acceptance in the improve loop; offline tests.
- **Out of scope**: taint, facts, CST walker, dispositions, or tolerance of parser ERROR nodes (engine specs); BaxBench generation slices; profile-specific fact tables or thresholds; any new merge gate; vendoring files without a stated license; LLM-based classification.
- **Adjacent expectations**: `local` stays the 100% pair gate; the stage-1 smoke gate stays inventory-only; `OverlayRecord` may gain one optional additive field; the improve loop's existing accept gate (recall floor, FP ceiling, score, no matched regression) stays and grows one clause; ledger file formats do not change.
## Requirements
### Requirement 1: Scorer counts every adjudicated detection
**Objective:** As an engineer evolving adjudication, I want the pair scorer to count what the overlay actually found, so that sink aliases and coverage flows move the number instead of vanishing.
#### Acceptance Criteria
1. When an overlay slice pair is scored, the system shall count a `coverage` record that carries a source-to-sink flow as a detection on the vulnerable side and as a leak on the fixed side, symmetrically with `promote`.
2. When an expected row names a function, the system shall match a promotion or coverage record only if the record lies inside that function's line range.
3. When an expected row names neither a rule id nor a sink nor a function, the system shall not count a CWE-only match as a detection and shall flag the row as `weak_label` in the payload.
4. When any overlay slice is evaluated, the system shall report inventory and overlay metrics side by side for that slice, not only for `vfc`.
5. When a pair is evaluated, the system shall report how many files on each side were `parse_failed` and how many proposals were `unadjudicated`, so parser loss is visible on the dashboard.
6. The system shall keep `local` as the only slice whose result can fail CI.
7. Where a hunter model is configured, the system shall report a third scorer, `hunter`, that runs the tool hunter on each isolated pair function and scores it with the same pair rules, so the LLM plane is measured on the same corpus; absence of a model shall skip the scorer with a recorded degradation.
### Requirement 2: Labels carry function, mechanism, and known limits
**Objective:** As a maintainer, I want every pair to say what should fire, where, and by which mechanism, so that recall can be read per mechanism and unachievable pairs do not pollute the achievable number.
#### Acceptance Criteria
1. Each expected row shall carry a `mechanism` value from a closed, versioned vocabulary stored as data next to the catalogs.
2. Each pair shall carry a `provenance` value from the closed set `human`, `agent`, `mixed`, `synthetic`.
3. When a pair is only achievable with information outside the vendored files, the catalog shall declare `known_limit` with a reason, and the system shall evaluate the pair but exclude it from the achievable metrics and report it separately.
4. When a catalog is loaded, the system shall reject an expected row whose `sink` is `unknown` and whose `function` and `rule_id` are both absent, and shall reject an unknown `mechanism` or `provenance` value.
5. Each pair shall carry a `split` value of `train` or `holdout`, declared in the catalog and never chosen at run time.
6. Where a recipe names a function, the generated expected row shall carry that function name.
### Requirement 3: Harvest anchors at the declarator
**Objective:** As a maintainer growing the corpus, I want harvest to extract the function body and nothing before it, in C, JavaScript, TypeScript, and Python, so that excerpts parse and pairs are attributable.
#### Acceptance Criteria
1. When harvest searches for a function by name, the system shall ignore occurrences inside comments and string literals and shall anchor on the declarator line.
2. Where a recipe gives a line range instead of a name, the system shall extract exactly that range from the parent and fix blobs.
3. Where a recipe gives a fix commit and a file, the system shall extract the function enclosing the first changed hunk on both sides.
4. When the language is Python, the system shall extract by indentation, not by braces.
5. When harvest is re-run for the curl recipes, the system shall produce excerpts that start at the declarator; the mid-comment excerpts shall no longer exist in the catalog.
6. The system shall not fetch during pytest, and extraction shall be unit-tested on local strings.
### Requirement 4: sast corpus hygiene
**Objective:** As an engineer, I want the textbook slice to contain only pairs that are labelable from the vendored files, so that a miss there is an engine miss.
#### Acceptance Criteria
1. The Juliet CWE-121 strcpy pair shall vendor the buffer declarations that differ between `bad` and `goodG2B`, so the two bodies are not byte-identical.
2. The OWASP Java hash pair shall be declared `known_limit` with reason `cross_artifact`, or shall vendor the property file that makes it labelable; either satisfies this criterion.
3. When the sast slice is evaluated, the payload shall report achievable pairs and known-limit pairs as separate counts.
### Requirement 5: Web and agent-authored slices
**Objective:** As an AppSec engineer, I want labeled Node, TypeScript, and Python web pairs, including agent-authored code, so that the harness is measured on the code operators are writing now.
#### Acceptance Criteria
1. The system shall add a `vibe-py` slice built from Real-Vuln-Benchmark manifests: pinned commit, file, line range, function, primary and acceptable CWEs, and authorship, with false-positive trap entries used as fixed-side twins.
2. The system shall add a `vfc-js` slice built from SecBench.js sink locations and fix commits, harvesting from the upstream package repositories, and shall not vendor SecBench.js test files.
3. The system shall add an `agent-vfc` slice of security-fix commits carrying an AI agent co-author trailer or generation marker, in JavaScript, TypeScript, or Python, each human reviewed before entering the catalog.
4. Each vendored pair shall carry repo, commit, parent, commit URL, license, provenance, mechanism, and split; a pair without a stated license shall not be vendored.
5. Each new slice shall start with a reviewed set on the order of tens of pairs and shall not require network in CI.
6. When the new slices are added, `datasets.toml` shall point at Real-Vuln-Benchmark, SecBench.js, AIDev, BaxBench, SecurityEval, CodeSecEval, CWEval, and CyberSecEval, and shall note that CyberSecEval labels are defined by its own rules and are not recall ground truth.
### Requirement 6: Provenance fingerprint at scan time
**Objective:** As an operator, I want a scan to say whether the tree looks agent-authored, so that profiles can be applied and reported without a model.
#### Acceptance Criteria
1. When a scan preprocesses a tree, the system shall compute a provenance value from deterministic signals only: agent configuration files and directories, generation markers in files, and commit trailers when a git history is readable.
2. The system shall record the provenance value and the signals that produced it in the manifest.
3. The system shall not call a language model to classify provenance.
4. If no signal is found, then the system shall record `human` with an empty signal list rather than guessing.
### Requirement 7: Profile dashboards and per-profile acceptance
**Objective:** As a maintainer running the improve loop, I want metrics per provenance profile and per mechanism, and an accept gate that refuses a change that helps one profile by hurting another, so that tuning can branch without overfitting.
#### Acceptance Criteria
1. When pairs are evaluated, the system shall report metrics per provenance profile and per mechanism in the machine-readable payload.
2. When the improve loop evaluates a candidate change, the system shall compute pair metrics per profile on the `holdout` split and shall reject the change if any profile's pair-correct or Youden regresses beyond a configured tolerance.
3. The system shall keep the existing accept gate clauses unchanged and shall not treat any non-local slice's Youden as a merge condition.
4. The system shall support at most four provenance profiles and shall report when a profile has fewer holdout pairs than a configured minimum.
5. Where a scan carries a provenance value, reports shall name the profile so an operator can relate dashboard numbers to their tree.
### Requirement 8: Offline testability and unchanged gates
**Objective:** As a maintainer, I want CI to prove the corpus loads and the scorer is honest without network, so that the corpus cannot rot and the gates cannot drift.
#### Acceptance Criteria
1. When the test suite loads the default catalog, it shall observe `vibe-py`, `vfc-js`, and `agent-vfc` among the slices and every referenced file on disk.
2. When the pairs command runs for each new slice with JSON output, it shall exit zero and include per-profile and per-mechanism metrics.
3. The stage-1 detection gate, the split-sink map gate, and the local pair gate shall be unchanged by this spec.
4. The extra-free suite shall stay green; overlay-dependent assertions shall skip when the semantic extra is absent.
### Requirement 9: Review tiers instead of a binary review gate
**Objective:** As a maintainer, I want every pair to carry how its label was established, so that the corpus can grow past what one person can review while only trusted tiers steer the improve loop.
#### Acceptance Criteria
1. Each pair shall carry a `review_tier` from the closed set `seeded` (ground truth reviewed by the upstream benchmark), `advisory` (a CVE or GHSA with a public fix commit), `title` (mechanism inferred from a commit message, unreviewed), `reviewed` (a named reviewer in this repository).
2. When pairs are evaluated, the system shall report metrics per review tier in the machine-readable payload.
3. The improve loop's per-profile holdout clause shall use only `seeded` and `reviewed` pairs; `advisory` and `title` pairs shall be scored and reported but shall not gate.
4. When a recipe carries `reviewer = "pending"`, its tier shall be `title` and it shall load into the catalog, replacing the former candidates-only gating; the `title` tier is the review queue.
5. The catalog loader shall reject a pair whose tier is `reviewed` but names no reviewer.
### Requirement 10: Pointer slices score public code without vendoring it
**Objective:** As a maintainer, I want to score pairs whose upstream repository states no license, so that the corpus is not capped by redistribution rights while the repository never redistributes that code.
#### Acceptance Criteria
1. A slice catalog may mark a pair `vendored = false`; such a pair carries repo, parent, commit, path, and extraction recipe but no excerpt files in the repository.
2. When a non-vendored pair is evaluated and the network is allowed, the system shall harvest both sides into a local cache outside the repository tree and score them exactly like a vendored pair.
3. When the network is not allowed, non-vendored pairs shall be skipped with a recorded degradation and shall not fail any test or gate.
4. The default `pytest` run and every CI gate shall evaluate vendored pairs only; a separate nightly command shall evaluate pointer slices.
5. Harvested cache files shall never be committed; the cache directory shall be ignored by git.
6. The `vibe-py` slice shall gain the LLM-generated Real-Vuln repositories as non-vendored `agent` pairs, and the `agent-vfc` slice shall gain unlicensed agent-authored fix commits as non-vendored pairs, each with `review_tier` set.
.kiro/specs/pair-corpus-honesty/research.md
+115
View File
@@ -0,0 +1,115 @@
# Research & Design Decisions
## Summary
- **Feature**: `pair-corpus-honesty`
- **Discovery Scope**: Extension of the pair catalog, scorer, and harvest; supersedes `real-world-vfc-slice`
- **Key Findings**:
- The overlay scorer ignores `coverage`. The Java sqli vuln file already produces a coverage record on `prepareCall` from a servlet source; the scorer reports MISS.
- The overlay column equals the inventory column on all 176 vfc pairs because C facts have no memory sink, so every proposal is `flow_incomplete` and the scorer falls back to inventory. The dashboard cannot show engine progress until the scorer reports parser and adjudication loss explicitly.
- Harvest anchors on the first textual occurrence of the function name. For curl that is the doc comment mention (`Curl_follow()`), so 25 pairs start mid-comment and fail to parse.
- 129 of 176 vfc expected rows have `sink = "unknown"`; 4 have a rule id. Matching degrades to CWE equality.
- Real-Vuln-Benchmark (Apache-2.0) ships 66 pinned Python web repos, 40 of them LLM-generated, with file, line range, function, primary and acceptable CWEs, severity, authorship, and 120 false-positive traps. It is the only released corpus with an authorship column.
- SecBench.js ships 600 npm vulnerabilities with `fixCommit`, `fixedVersion`, and `sinkLocation`, but no license file. Pointer plus harvest from upstream repos only.
- GitHub commit search for a Claude co-author trailer plus "fix IDOR" returns over twenty thousand commits; a sampled June 2026 FastAPI repository with `AGENTS.md` had a 12-line fix moving identity from the request body to the auth context. Agent-authored vuln-to-fix pairs are harvestable.
- VibeApps/VibeVulns (arXiv 2606.23130) is not released, but its identification method is: `.claude/` directory, Lovable meta tag, Claude trailers, and an 85% AI-authored threshold.
## Research Log
### Scorer blind spots
- **Context**: Why the sast overlay number did not move after the tree-sitter extra.
- **Sources Consulted**: `src/openultrasast/pairs.py` (`_evaluate_overlay_pair`, `_overlay_matches_expected`, `_overlay_leaks`), `semantic/overlay.py`, direct runs of `adjudicate` on the sast files.
- **Findings**: only `promote` is counted; `coverage` is excluded on both sides. `_overlay_matches_expected` accepts a CWE-only match. `ExpectedFinding` already has `function` and `line`, and `benchmark._matches_expected` honors them, but the overlay path does not. `OverlayRecord` has no function field; `FunctionIR` has line ranges.
- **Implications**: count coverage symmetrically; add optional `function` to `OverlayRecord`; require function or sink or rule id for a match when present; flag CWE-only rows.
### Harvest anchor defect
- **Context**: 50 vfc files `parse_failed`.
- **Sources Consulted**: `benchmarks/pairs/vfc/harvest.py` `extract_function`; tree-sitter ERROR nodes on `curl-cve-2022-27774/vuln.c` and `curl-cve-2023-38545-do_socks5/vuln.c`.
- **Findings**: the name search accepts a match followed by `(` even inside a comment, and the brace search then finds the real body. A second, smaller class is `#if`-split `else if` chains, which tree-sitter cannot attach; that is an engine tolerance question, not harvest.
- **Implications**: skip comment and string regions before searching; anchor on a line that starts a declarator; add line-range and hunk modes so Python and JS/TS are covered; report parse loss on the dashboard so the engine spec can measure its tolerance work.
### Candidate corpora
- **Context**: User asked for vibe-coded and Node/TypeScript/Python web corpora.
- **Sources Consulted**: Real-Vuln-Benchmark repo and paper; SecBench.js repo and ICSE 2023 paper; AIDev dataset card and paper; BaxBench site, repo, and paper; SecurityEval, CodeSecEval, CWEval; CyberSecEval benchmarks README; VibeApps paper; TS-VulBench page; SecureVibeBench repo; GitHub commit search via `gh api search/commits`.
- **Findings**:
| Corpus | Languages | Labels | Fixed twin | License | Verdict |
|---|---|---|---|---|---|
| Real-Vuln-Benchmark | Python web | file, line range, function, CWEs, authorship, FP traps | traps | Apache-2.0 | `vibe-py` |
| SecBench.js | Node | sinkLocation, fixCommit, class | yes | none stated | `vfc-js` via upstream |
| AIDev + GitHub search | multi | provenance only | via fix commit | per repo | `agent-vfc` source |
| BaxBench | Express, Flask, FastAPI, Django, Nest, more | exploit outcome per generated sample | by generation | MIT | later, needs deep mode |
| SecurityEval, CodeSecEval, CWEval | Python, some C/JS | CWE per prompt, some tests | partial | check per repo | pointers |
| CyberSecEval ICD | 8 languages | defined by its own rules | no | MIT | pointer, circular |
| VibeApps/VibeVulns | TS, JS, React | not released | no | n/a | method only |
| TS-VulBench | TS synthetic | paywalled | no | unknown | skip |
| SecureVibeBench | C/C++ | agent tasks | n/a | MIT | skip for web |
- **Implications**: three slices now; BaxBench after deep mode is stable; CyberSecEval never as recall ground truth.
### Classification and branching
- **Context**: User expects tuning to split into branches per corpus class.
- **Sources Consulted**: `improve/evolve.py` accept gate; `ruleset/store.py` ledger; `calibration.py` scope; `preprocess.detect_tags`; `build_pair_signals`.
- **Findings**: the accept gate already ANDs four clauses and reverts byte-for-byte. Pair signals carry pair name but no profile. Ledger is one file per target. Preprocess tags are deterministic and per file.
- **Implications**: classify on two axes (provenance for gating, mechanism for reporting). Branch the acceptance decision, not the ledger format: reject a change if any profile's holdout regresses. Profile-specific facts and thresholds are deferred to the engine specs where those data live.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Reopen `real-world-vfc-slice` | Add tasks to the closed spec | One place | Mixes closed seed work with scorer redesign; roadmap says do not reopen | Rejected |
| New superseding spec, corpus plane only | Scorer, labels, harvest, slices, provenance, profile gate | Measurable before engine work; engine specs stay pure | Two engine specs still needed | **Selected** |
| One big spec including engine | Corpus plus walker plus guards | Fewer specs | Unreviewable size; engine changes would be measured by a lossy scorer | Rejected |
| Profile as ledger fork | One `rule_policy.<profile>.json` per profile | Full branching | Changes ledger format owned by another spec; overfits per slice | Deferred |
## Design Decisions
### Decision: Count coverage-with-flow as detection and as leak
- **Context**: Sink aliases will always land as coverage.
- **Alternatives Considered**: promote coverage to `promote` in the engine; count coverage only on the vulnerable side.
- **Selected Approach**: scorer treats `coverage` with a non-empty source the same as `promote` on both sides.
- **Rationale**: `propose-adjudicate-prove` already states coverage enters the same state machine. Asymmetry would inflate recall.
- **Trade-offs**: some pairs move from MISS to LEAK; that is the honest reading.
### Decision: Function-scoped matching with weak-label flagging
- **Context**: CWE-only matching over-counts.
- **Selected Approach**: expected rows carry `function`; match requires the record inside that function's lines; rows with no function, sink, or rule id are `weak_label` and never match by CWE alone.
- **Trade-offs**: vfc recall may drop on first run; the drop is real.
### Decision: Provenance is deterministic and closed
- **Context**: No LLM in classification.
- **Selected Approach**: signals from `.claude/`, `CLAUDE.md`, `AGENTS.md`, `.cursor/`, `.github/copilot-instructions.md`, Lovable meta tag, "Generated with Claude Code", and co-author trailers for Claude, Codex, Copilot, Cursor, Devin, Jules when a git log is readable. Threshold for `agent` on a git tree: at least 85% of commits or lines in the last N commits carry a signal, following the VibeApps method; otherwise `mixed` when any signal exists; `human` when none; `synthetic` only from catalog declaration.
- **Trade-offs**: repos that scrub trailers read as human. Acceptable; the label is a profile, not a verdict.
### Decision: Branch the accept decision, not the data
- **Context**: Overfitting risk with per-slice tuning.
- **Selected Approach**: `run_improvement` evaluates holdout pairs per profile; any profile regression beyond tolerance rejects. Profiles capped at four. Minimum holdout size reported.
- **Follow-up**: profile-keyed facts land with the engine specs, gated by this clause.
## Risks & Mitigations
- Recall numbers drop after function-scoped matching and weak-label flags — expected; record before and after in the roadmap.
- SecBench.js has no license — never vendor its files; harvest from upstream package repos and record each package license.
- Agent-authored commits may be false labels (a "fix" that was not a vuln) — human review required; keep `agent-vfc` small.
- Real-Vuln-Benchmark apps are educational and CTF style — say so in the slice README; do not treat `vibe-py` as production.
- Held-out halves are small on day one — report minimum-size warnings; do not gate on a profile under the minimum.
## References
- [Real-Vuln-Benchmark](https://github.com/kolega-ai/Real-Vuln-Benchmark) — Apache-2.0, manifest with pinned SHAs, ground-truth schema
- [RealVuln paper](https://arxiv.org/abs/2604.13764) — 26 human plus 40 LLM-generated Python repos, FP traps
- [SecBench.js](https://github.com/cristianstaicu/SecBench.js) — 600 npm vulns, `fixCommit`, `sinkLocation`
- [SecBench.js paper](https://software-lab.org/publications/icse2023_SecBenchJS.pdf)
- [AIDev dataset](https://huggingface.co/datasets/hao-li/AIDev) — agent-authored PRs, CC-BY-4.0 index
- [BaxBench](https://github.com/logic-star-ai/baxbench) — MIT, scenarios and exploits, generation by CLI
- [SecurityEval](https://github.com/s2e-lab/SecurityEval), [CodeSecEval](https://arxiv.org/pdf/2407.02395), [CWEval](https://github.com/Co1lin/CWEval)
- [CyberSecEval](https://github.com/meta-llama/PurpleLlama/tree/main/CybersecurityBenchmarks) — labels defined by ICD rules
- [VibeApps paper](https://arxiv.org/html/2606.23130v1) — identification method
- `.kiro/specs/real-world-vfc-slice/` — superseded corpus spec
- Memory note `overlay-gaps-diagnosis-2026-09` — measured state and engine gaps
.kiro/specs/pair-corpus-honesty/spec.json
+28
View File
@@ -0,0 +1,28 @@
{
"feature_name": "pair-corpus-honesty",
"created_at": "2026-09-04T16:12:17Z",
"updated_at": "2026-09-05T09:00:00Z",
"language": "en",
"phase": "implementation",
"supersedes": [
"real-world-vfc-slice"
],
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": true,
"amendments": [
"2026-09-05 review tiers (Req 9) and pointer slices (Req 10), approved by the maintainer in session"
]
}
.kiro/specs/pair-corpus-honesty/tasks.md
+216
View File
@@ -0,0 +1,216 @@
# Implementation Plan
- [x] 1. Foundation: label schema and vocabulary
- [x] 1.1 Add the closed mechanism vocabulary as data
- A versioned TOML next to the catalogs lists every mechanism id with a one-line definition.
- Catalog load rejects an expected row whose mechanism is not in the vocabulary.
- Observable: loading a catalog with an unknown mechanism fails with the pair name; the vocabulary file lists the v1 set from the design.
- _Requirements: 2.1, 2.4_
- _Boundary: Vocabulary, LabelSchema_
- [x] 1.2 Extend pair and expected fields with provenance, split, known_limit, function
- Pair rows accept `provenance` from the closed set, `split` of train or holdout, and an optional `known_limit` reason.
- Expected rows accept `function`; rows with `sink = "unknown"` and no function and no rule id are rejected.
- Defaults: sast is synthetic, local/github/vfc are human, split is train.
- Observable: a unit test loads a row with each new field and rejects an unknown provenance value and a sink-unknown row without function or rule id.
- _Requirements: 2.2, 2.3, 2.4, 2.5, 2.6_
- _Boundary: LabelSchema_
- _Depends: 1.1_
- [x] 2. Core: honest scorer
- [x] 2.1 Count coverage with a source as detection and as leak
- Overlay pair scoring treats a coverage record with a non-empty source the same as a promote on both the vulnerable and the fixed side.
- Outcome records which detection kinds contributed.
- Observable: the owasp-java-sqli vulnerable side counts as detected via coverage; a coverage record on a fixed tree under silent policy counts as a leak.
- _Requirements: 1.1_
- _Boundary: Scorer_
- [x] 2.2 Match by function and flag weak labels
- `OverlayRecord` gains an optional `function` filled from the enclosing function's line range; default stays None.
- When an expected row names a function, a record matches only inside that function; CWE equality alone never matches; rows without function, sink, or rule id are counted as weak labels.
- Observable: a promotion outside the named function does not match; a CWE-only row is reported under weak_labels and produces no match.
- _Requirements: 1.2, 1.3_
- _Boundary: Scorer_
- _Depends: 1.2_
- [x] 2.3 Report side-by-side scorers and loss counters for every overlay slice
- Every overlay slice gets `overlay` and `inventory` scorers, not only vfc.
- Payload reports parse_failed files and unadjudicated proposals per side and per slice, plus known_limit and achievable metrics.
- Observable: `pairs --slice sast --json` contains both scorers and a loss block; known-limit pairs are listed and excluded from achievable metrics.
- _Requirements: 1.4, 1.5, 2.3, 4.3_
- _Boundary: Scorer_
- _Depends: 2.1, 2.2_
- [x] 2.4 Migrate existing vfc rows and record the new baseline
- `generate_catalog.py` emits function, mechanism, provenance, and split from recipes; every existing recipe gains a mechanism and half the pairs are declared holdout.
- Roadmap records sast and vfc numbers before and after the scorer change.
- Observable: no vfc expected row is a weak label; the catalog loads under the new validation; the roadmap has a before/after table.
- _Requirements: 2.6, 1.6_
- _Boundary: LabelSchema_
- _Depends: 2.3_
- [x] 2.5 Add the hunter scorer for overlay slices
- With a hunter model configured, each pair function is run through the tool hunter and scored with the same detection and leak rules; results appear as `scorers[slice].hunter`; without a model the scorer is skipped with a degradation.
- Default tests use a scripted client; no network.
- Observable: `pairs --slice sast --json` with a scripted client shows a `hunter` scorer; without a model the payload records `hunter_model_unavailable`.
- _Requirements: 1.7_
- _Boundary: Scorer_
- _Depends: 2.3_
- [x] 3. Core: harvest library (parallel with 2)
- [x] 3.1 (P) Anchor name extraction at the declarator
- Comments and string literals are masked before the name search; the anchor line must not be inside a mask and must be followed by a parameter list and a body before a semicolon.
- Existing vfc harvest tests still pass through a thin shim.
- Observable: a fixture whose doc comment mentions `Curl_follow()` above the definition extracts from the definition line; existing prototype-skip test still passes.
- _Requirements: 3.1, 3.6_
- _Boundary: HarvestLibrary_
- [x] 3.2 (P) Add line-range, hunk, and Python block modes
- A recipe may give a line range, or a fix commit whose first changed hunk selects the enclosing function on both sides; Python functions are cut by indentation.
- Observable: each mode round-trips a local string fixture; no network in tests.
- _Requirements: 3.2, 3.3, 3.4, 3.6_
- _Boundary: HarvestLibrary_
- _Depends: 3.1_
- [x] 3.3 Re-harvest the curl pairs
- All curl recipes are re-run with the anchored extractor; excerpts start at the declarator.
- Pairs that still fail to parse because of preprocessor-split control flow are kept and show up in the loss counters, not removed.
- Observable: no vfc excerpt has a mid-comment first code line; the loss block for vfc reports the remaining parse failures.
- _Requirements: 3.5_
- _Boundary: SliceCatalogs_
- _Depends: 3.1, 2.3_
- [x] 4. Core: sast hygiene (parallel with 3)
- [x] 4.1 (P) Restore the Juliet strcpy buffer declarations and mark the Java hash pair
- The Juliet CWE-121 vuln and fixed excerpts include the differing buffer declarations from the full case file.
- The OWASP Java hash pair is declared known_limit with reason cross_artifact, or vendors the property file.
- Observable: the two Juliet bodies differ; owasp-java-hash appears under known_limit; sast achievable count is 10 or 11 accordingly.
- _Requirements: 4.1, 4.2, 4.3_
- _Boundary: SliceCatalogs_
- [x] 5. Integration: new slices
- [x] 5.1 Add the vibe-py slice from Real-Vuln-Benchmark _(escalated: the "at least one agent provenance" observable cannot be met without vendoring unlicensed LLM-generated repos; see Implementation Notes)_
- Recipes carry pinned commit, file, line range, function, primary and acceptable CWEs, authorship, and vulnerable or trap flag; harvest uses line-range mode; traps become fixed-side twins; provenance from authorship.
- README states the educational and CTF caveat and the license.
- Observable: the slice loads offline with tens of pairs, at least one agent and one human provenance, and scores with per-profile metrics.
- _Requirements: 5.1, 5.4, 5.5_
- _Boundary: SliceCatalogs_
- _Depends: 3.2, 1.2_
- [x] 5.2 (P) Add the vfc-js slice from SecBench.js fix commits
- Recipes point at upstream package repositories with parent and fix commit, file, and sink line; harvest uses hunk mode; command injection, path traversal, and code injection first; prototype pollution and ReDoS rows carry their mechanism and may be known_limit.
- No SecBench.js test files are vendored; each pair records the upstream package license.
- Observable: the slice loads offline with tens of pairs and JavaScript or TypeScript files; no file from the SecBench.js repository is present.
- _Requirements: 5.2, 5.4, 5.5_
- _Boundary: SliceCatalogs_
- _Depends: 3.2, 1.2_
- [x] 5.3 (P) Add the agent-vfc slice from agent-trailer security fixes _(re-scoped by Req 9: rows load at tier `title` and never gate; task closes when 8.1 lands and the slice scores)_
- Recipes record repo, parent, commit, file, the agent trailer or marker, mechanism, and reviewer; harvest uses hunk mode; provenance is agent.
- README carries the review checklist: real vulnerability, single mechanism, no secrets, license stated.
- Observable: the slice loads offline with tens of reviewed pairs in JavaScript, TypeScript, or Python and scores under the agent profile.
- _Requirements: 5.3, 5.4, 5.5_
- _Boundary: SliceCatalogs_
- _Depends: 3.2, 1.2_
- [x] 5.4 Point datasets.toml at the researched corpora and document slices
- Rows for Real-Vuln-Benchmark, SecBench.js, AIDev, BaxBench, SecurityEval, CodeSecEval, CWEval, CyberSecEval with license and caveats; CyberSecEval is marked rule-defined, not recall ground truth.
- Pair README lists the new slices and the profile and mechanism views.
- Observable: dataset test sees the new names; README mentions each slice.
- _Requirements: 5.6_
- _Depends: 5.1, 5.2, 5.3_
- [x] 6. Integration: provenance and profile gate
- [x] 6.1 (P) Compute a deterministic provenance fingerprint at scan time
- Signals from agent configuration paths, generation markers, and git trailers over a bounded commit sample; agent at 85 percent trailer share, mixed on any signal, human otherwise; never synthetic.
- Manifest records the value and the signal list; no model call.
- Observable: an empty tree is human with no signals; a tree with AGENTS.md is mixed; a synthetic git log of Claude trailers is agent; git absence is recorded as a signal.
- _Requirements: 6.1, 6.2, 6.3, 6.4, 7.5_
- _Boundary: ProvenanceFingerprint_
- [x] 6.2 Add per-profile and per-mechanism metrics and a profile filter
- Payload gains per_profile and per_mechanism; the pairs command accepts a profile filter.
- Observable: `pairs --slice all --json` contains both maps; `--profile agent` evaluates only agent pairs.
- _Requirements: 7.1_
- _Boundary: Scorer_
- _Depends: 2.3, 5.4_
- [x] 6.3 Reject improve-loop changes that regress any profile on holdout
- After the existing four clauses pass, holdout pairs are evaluated per profile; a drop in pair-correct or Youden beyond tolerance for a profile with at least the minimum pairs rejects with a profile reason; smaller profiles are reported only; at most four profiles.
- Observable: a test change that improves synthetic and regresses agent holdout is rejected with `profile_regression:agent`; a profile under the minimum appears in the journal and does not gate.
- _Requirements: 7.2, 7.3, 7.4_
- _Boundary: ProfileGate_
- _Depends: 6.2_
- [x] 7. Validation
- [x] 7.1 Prove offline load, CLI, and unchanged gates
- Default catalog load sees the three new slices and every file on disk; each slice command exits zero with per-profile metrics; new names are absent from the stage-1 fixture list; detection gate, map gate, and local pair gate outputs are unchanged; extra-free suite is green with overlay assertions skipping.
- Observable: extra-free pytest green; gate entrypoints produce the same verdicts as before this spec.
- _Requirements: 8.1, 8.2, 8.3, 8.4, 1.6_
- _Depends: 5.4, 6.3_
- [x] 7.2 Close real-world-vfc-slice as superseded
- Its spec.json records superseded_by; its brief carries a closure note; the roadmap lists this spec and the two engine specs in dependency order.
- Observable: spec status for real-world-vfc-slice shows closed and superseded; roadmap has the new order.
- _Requirements: 8.3_
- _Depends: 7.1_
- [ ] 8. Amendment: review tiers and pointer slices
- [x] 8.1 Add review tiers to the label schema and the payload
- `review_tier` on every pair with per-slice defaults; `per_tier` metrics in the payload; `title` rows load and replace the candidates file; `reviewed` requires a reviewer name.
- Observable: agent-vfc rows load with tier `title`; `pairs --json` reports `per_tier`; a `reviewed` row without a reviewer is rejected by the loader.
- _Requirements: 9.1, 9.2, 9.4, 9.5_
- _Boundary: LabelSchema, Scorer_
- [x] 8.2 Gate the improve loop on seeded and reviewed tiers only
- The per-profile holdout clause filters pair cases to `seeded | reviewed` before scoring; `advisory` and `title` pairs never gate.
- Observable: a regression confined to `title` pairs does not reject a round; the same regression on `reviewed` pairs does.
- _Requirements: 9.3_
- _Boundary: ProfileGate_
- _Depends: 8.1_
- [ ] 8.3 Score non-vendored pointer pairs through a local cache
- Catalog rows with `vendored = false` carry the recipe; evaluation harvests both sides into the cache when `OPENULTRASAST_PAIRS_NETWORK=1` (or `pairs --pointers`), otherwise skips with a degradation; the cache is git-ignored; tests and gates select vendored pairs only.
- Observable: a pointer row evaluates to the same outcome shape as a vendored row when network is on, is skipped with `pointer_pair_skipped` when off, and no cache file appears under the repository.
- _Requirements: 10.1, 10.2, 10.3, 10.4, 10.5_
- _Boundary: Scorer, HarvestLibrary_
- _Depends: 8.1_
- [ ] 8.4 Add the LLM-generated Real-Vuln repositories and unlicensed agent fixes as pointer pairs
- vibe-py gains the 40 LLM-generated repositories as `agent`, `seeded`, `vendored = false` rows built by `build_recipes.py --pointers`; agent-vfc gains the license-less candidates as `title`, `vendored = false` rows.
- Observable: `pairs --slice vibe-py --pointers --json` scores both provenances; the default suite and gates are unchanged.
- _Requirements: 10.6, 5.1_
- _Boundary: SliceCatalogs_
- _Depends: 8.3_
## Implementation Notes
- Review round 1 (kiro-review, 2026-09-04) REJECTED groups 1, 2, 3, 5, 6; group 4 APPROVED. Remediated in round 2: mypy/ruff clean; `tool_hunter.py` reverted (HarnessX-preferring hunter client moved to `reachability-flow-model` as a task); weak labels never match on the inventory-fallback path; exact/alias sink matching restored; loader requires a named function on vfc-style slices; `catalog_gen.derive_function` ignores provenance headers and never emits `<anon>`; vfc-js rebuilt without duplicate/misattributed fixes; vendored Google API key redacted and a code-safe redaction pass added to `harvest.write_excerpt`; improve CLI fails loud on a missing pair catalog; profile tolerance compares pass rates; known-limit pairs emit no improve signals. RED evidence for the remediation tests was captured before the fixes (5 + 1 failing tests).
- Review round 2 (2026-09-05) APPROVED groups 1, 3, 4, 6; REJECTED groups 2 and 5. Remediated RED-first (4 failing tests captured, then green): `function_at` skips `<anon>` as well as `<module>`; the scorer matches a labeled function by line-range containment from the vulnerable side's parse (`_in_named_function`), name equality only when no range exists; the loader rejects reserved words as function labels; `catalog_gen` derives labels from the declarator on the range's start line (chained and property arrows covered), drops parse-failed excerpts instead of regex-labeling them, and `--prune` removes excerpt directories that are neither cataloged nor candidates (vfc-js: 16 pairs, 17 dirs incl. one candidate-less prune); the agent-vfc test asserts the empty-until-reviewed invariant explicitly.
- Review round 3 (2026-09-05) REJECTED groups 2 and 5 again; per protocol a fresh kiro-debug agent produced the fix plan (ROOT_CAUSE: scorer ranges used raw parser names while labels used declarator names; labels followed formatting hunks; no sink on vfc-js rows). Remediated RED-first (8 failing tests captured, then green): one shared helper `src/openultrasast/semantic/functions.py` (`declarator_name`, `named_function_ranges`, `function_at`, `spans_named`) used by `overlay._flows_for`, `pairs._overlay_scan`, `pairs._with_functions`, and `catalog_gen`; bare-parameter arrows and callback callees are never names; `_in_named_function` is containment-only and an unresolvable label is counted as `unresolved_function_labels` loss with a `!unresolved` miss marker; hunter matching uses containment too; harvest records `upstream_start` in every excerpt header (`extract_pair_with_lines`); `catalog_gen.derive_function` maps `sink_location` through `upstream_start` (±10 lines, CWE-family sink tokens, regex `.exec(` excluded), falls back to the unique function owning a sink token, then to the mapped line itself for wrapper sinks, and only then to the first non-whitespace hunk; vfc-js re-harvested with headers, regenerated with `--prune` (17 pairs, 17 dirs), `sink` emitted on 13/17 rows.
- Measured after review round 4 remediation (2026-09-05): sast 4/11 (achievable 4/10), Youden +0.273, unresolved 0; vibe-py 13/35, Youden +0.371 (inventory 7), unresolved 0, parse-failed 0; vfc-js 5/17, Youden +0.235, unresolved 0; vfc 0/176, Youden -0.131, parse-failed files 36, unresolved labels 23 after round 5: 18 pairs parse-failed on both sides (36 files, walker gap for overlay-ir-completeness); 3 pairs whose `.in` files `preprocess_repository` never lists as targets (`openssl-cve-2022-1292` and `openssl-cve-2022-2068-hash_dir` Perl `c_rehash.in`, `openssl-cve-2026-31790-rsasve_generate` C template `rsa_kem.c.in`; the Perl `sub` rule is dead on the scorer path until `.in` files are targeted, accepted as visible loss); 2 pairs whose multi-line declarators the walker leaves `<anon>` and the one-line regex cannot rescue (`chromium-cve-2022-2162`, `openssl-cve-2025-69421`: walker gaps). No unresolved label remains inside the Scorer boundary.
- Review round 4 (2026-09-05) REJECTED groups 2 and 5 on two defects, both fixed RED-first (4 failing tests captured): the shared helper now demotes undeclared parser names to `<anon>` only for JavaScript/TypeScript (bare-parameter arrows), so C/Java/Python keep their parser names when the return type sits on the previous line; `declarator_name` knows Perl `sub` and never names a statement (`return foo(`, `print(`, `synchronized (`); `catalog_gen` imports the package at module top and exits loud without it, `_ranges` no longer swallows exceptions, and `--prune` refuses to run when no recipe derived a label; the Python block extractor decides indentation on masked lines so triple-quoted strings never end a block (`vfapi-main-put-user` re-harvested, now parses).
- Review round 5 (2026-09-05) APPROVED group 5; REJECTED group 2 on one defect, fixed RED-first (2 failing tests captured, then green): `declarator_name` returned C++ qualified names (`FileReaderLoader::ArrayBufferResult`) that no unqualified catalog label matches, so both C++ vfc pairs were unresolvable and had been misattributed to the walker; the helper now returns the unqualified tail for the method form, `chromium-cve-2019-5786` resolves (vfc unresolved 24 -> 23), sast/vibe-py/vfc-js numbers unchanged, and the measured note above carries the corrected per-cause breakdown.
- Review round 6 (2026-09-05) APPROVED group 2 (verify-completion: `pytest -q` 468 passed / 4 skipped, extra-free 455 / 17 skipped, ruff/mypy clean, three gates PASS, reviewer reproduced RED by deleting the fix line and re-measured all four slices). Feature-level verdict stays REJECTED for completeness only: groups 7 and 8 open, two escalations undecided. Reviewer suggestions carried to the roadmap, not blocking: a multi-line-aware declarator rescue in `functions.py` would name the two `<anon>` C/C++ ranges without touching `cst.py`; the inventory-fallback miss strings lack the `!unresolved` suffix the overlay path attaches (counter is correct).
- Task 8.1 (2026-09-05, RED first: 6 failing tests captured — `REVIEW_TIERS`/`select_tier` missing, `reviewed` without reviewer accepted, no `per_tier` payload, agent-vfc absent from the default catalog, CLI payload lacked `per_tier`; then green): `PairCase.review_tier`/`reviewer`, `PairOutcome.review_tier`, `PairEvalResult.per_tier`, `select_tier`, `GATING_TIERS`; loader derives the tier from the row, else from `reviewer` (pending -> title, a name -> reviewed), else the slice default, and rejects unknown tiers and `reviewed` without a reviewer; `catalog_gen` stamps `review_tier` (and `reviewer`) on every generated row, `title` rows enter `catalog.toml`, `catalog-candidates.toml` is retired (deleted on regenerate); local rows carry `reviewer = "Marius Ciepluch"` (fixture author per git history). Task 7.1 evidence: default catalog loads 277 pairs across local/github/sast/vfc/vibe-py/vfc-js/agent-vfc with every file on disk and no name on the stage-1 `LANGUAGE_MANIFESTS`; `pairs --slice {vibe-py,vfc-js,agent-vfc} --json` exit 0 with `per_tier`, `per_profile`, `per_mechanism`; detection/map/local-pair gate outputs byte-identical to commit 0116f9b (pre-spec); extra-free suite 459 passed / 17 skipped; full suite 472 passed / 4 skipped; each of the six task commits is green in isolation (worktree bisect: 413 -> 430 -> 430 -> 430 -> 459 -> 465 -> 468 passed, ruff clean). Measured agent-vfc at tier `title`: 0/29, Youden +0.000 ({'typescript': 18, 'python': 6, 'javascript': 5}; no row in any language yields a finding on either side: the excerpts carry no rule-known sink (CORS defaults, missing guards, prototype pollution: title-guessed mechanisms, several wrong on inspection, e.g. `dnf-plugin-anyrepo-config-26d19a` has no path join); inventory rules do run on `.ts` files, but the overlay plane reports `language_unsupported` for TypeScript (18 rows) and counts them neither as parse-failed nor as a separate loss — mechanisms {'source_reaches_sink': 7, 'permissive_default': 6, 'path_join_user_input': 3, 'prototype_pollution': 7, 'missing_auth_guard': 1, 'identity_from_request_body': 5}), unresolved labels 1. This is the honest title-tier baseline the engine specs measure against, not a scoring defect. Task 5.3 closes with 8.1 as re-scoped: the slice loads and scores; human review promotes rows to `reviewed`.
- Review round 7 (2026-09-05) APPROVED tasks 7.1 and 7.2; REJECTED 8.1 on documentation only (REVIEW.md still described the candidates file; READMEs claimed the tier gate filter before 8.2 existed; TS zeros misattributed). Remediated: REVIEW.md/READMEs rewritten to tier semantics, note corrected, `catalog_gen._review_tier` precedence aligned with the loader (explicit tier wins), `training/manifest.jsonl` rows carry `review_tier`, design §LabelSchema candidates sentence superseded, roadmap agent-vfc line updated. Deferred to the engine specs: a `language_unsupported` loss counter (reviewer suggestion).
- Task 8.2 (2026-09-05, RED first: `test_profile_gate_ignores_title_and_advisory_tiers` failed with `reason='profile_regression:agent'` on five title-tier pairs; then green): `evolve.evaluate_profiles` filters holdout pairs to `GATING_TIERS` via `select_tier` before scoring, so `advisory`/`title` pairs never gate and a regression on `reviewed` pairs still rejects the round.
- Review round 8 (2026-09-05) APPROVED 8.1 and 8.2 (verify-completion: `pytest -q` 473 passed / 4 skipped, extra-free 460 / 17, ruff/mypy clean, three gates PASS; reviewer reproduced RED by removing the tier filter and checked the default holdout gate now covers `human` seeded pairs only, with the 16 title-tier agent rows excluded). Task 5.3 closed with 8.1 as re-scoped. Applied suggestion: the agent-vfc tier test accepts `reviewed` rows so promoting a row does not break it. Follow-up for corpus-seeded-mechanisms: report holdout profiles excluded by tier (`profiles_ungated`) so an operator sees why `agent`/`synthetic` do not gate.
- ESCALATION (task 5.1 observable): vibe-py has `human` provenance only because the 40 LLM-generated Real-Vuln repos have no license (Req 5.4 wins). Spec owner decision needed: accept human-only vibe-py, or obtain a license statement upstream.
- ESCALATION (Req 5.3): agent-vfc rows are unreviewed. They are NOT in the loaded catalog; `benchmarks/pairs/agent-vfc/catalog-candidates.toml` and `REVIEW.md` list them for a human reviewer. Task 5.3 stays open until reviewed.
- 2026-09-04 execution evidence (manual mode, main context): full suite `uv run pytest` 440 passed / 4 skipped; `python -m openultrasast.gate` PASS; `map_gate` PASS; `pair_gate` PASS (local 3/3); `ousast --help` boots. Validation commands come from `.github/workflows/ci.yml`.
- Scorer (2.1–2.3): coverage-with-source counts both ways; function-scoped match; CWE-only rows are `weak_label`; `scorers[slice]` has overlay+inventory for every overlay slice, `hunter` with a client; payload gains `per_profile`, `per_mechanism`, `achievable`, `known_limit`, `loss`, `degradations`. `OverlayRecord.function` is the one additive overlay field; `_flows_for` parses once and keeps function ranges.
- 2.4 baseline after the scorer change is in the roadmap (sast 4/11 → achievable 4/10; vfc 0/176 under function matching). Regex findings get an enclosing function from a parse so function-labeled rows can match the inventory column.
- 2.5 hunter scorer: `pairs --hunter [--hunter-model]`; client resolution prefers HarnessX (`resolve_hunter_client(prefer_harnessx=...)`, best-effort adapter, offline-untested) then any OpenAI-compatible endpoint via `OPENROUTER_API_KEY` + `OPENROUTER_BASE_URL` (DeepSeek documented in `.env.example`). Live run not executed in this session; scripted-client test covers the scorer.
- 3.x harvest: shared `benchmarks/pairs/harvest.py` (comment-blind `name`, `line_range`, `hunk` with next-function and whole-small-file fallbacks, `enclosing` with `fix_path`/`fix_line`), `vfc/harvest.py` is a shim; `catalog_gen.py --slice` regenerates catalogs and derives `function` for hunk-harvested pairs from the excerpt diff. 3.3: curl re-harvested; parse failures fell from 50 files to 36 (remaining are `#if`-split control flow and similar grammar limits, visible in `loss.parse_failed_files`; engine tolerance belongs to `overlay-ir-completeness`).
- 5.1 vibe-py: 35 pairs from the 15 licensed human-authored Real-Vuln repos. The 40 LLM-generated repos (`kolega-ai-dev/vc-*`) have no license file and were NOT vendored (Req 5.4); `build_recipes.py --allow-unlicensed` exists for local inspection only. So the slice currently has `human` provenance only; agent provenance comes from `agent-vfc`.
- 5.2 vfc-js: 15 pairs from upstream fix commits after dedupe by fix commit and module/repo attribution check (SecBench.js pointer only; 10 unlicensed upstreams, 3 duplicate fixes, 8 misattributed modules skipped; 2 excerpts without a named function not cataloged). `sink_location` is kept as metadata, not as the expected sink.
- 5.3 agent-vfc: 29 candidate pairs in `catalog-candidates.toml` (not loaded), all `reviewer = "pending"`; `REVIEW.md` is the review sheet. Mechanism/CWE were assigned from commit titles by `build_recipes.py` and must be confirmed by a human. 631 candidates were searched; 202 skipped for missing license; the loaded `catalog.toml` has 0 rows until review.
- 6.1 provenance: `src/openultrasast/provenance.py`, manifest `provenance` block, stage `provenance` in `_run_scan`.
- 6.3 profile gate: `run_round(pair_cases=..., profile_tolerance, min_holdout_pairs)`; CLI `improve --pair-catalog/--no-pair-gate/--profile-tolerance/--min-holdout-pairs`; default on with the full catalog holdout split (slower rounds).
- Measured after round-2 remediation (2026-09-04): sast 4/11 (achievable 4/10), Youden +0.273; vibe-py 13/35 pair-correct, Youden +0.371 (overlay) vs inventory 7/35, 0 weak labels; vfc-js 0/15, Youden −0.200 (2 parse-failed files, 0 weak labels; the earlier 5/32 was inflated by first-function labels and duplicate rows); agent-vfc 0 loaded rows, 29 candidates awaiting review.
- Do not touch `taint.py`, `facts.py`, `cst.py`, or `ir.py`. Parser ERROR-node tolerance belongs to `overlay-ir-completeness`; this spec only makes the loss visible.
- Expect vfc recall to drop when function-scoped matching lands. Record it; do not soften the matcher.
- BaxBench generation is deferred until deep mode can run its exploit harness; provenance and mechanism fields are designed so that slice slots in later.
.kiro/specs/real-world-vfc-slice/brief.md
+4
View File
@@ -52,3 +52,7 @@ Do not clone Firefox. Filter public VFC databases for `gecko-dev` / `mozilla-fir
## Constraints
Isolated functions, same `relpath` vuln/fixed. License in the header. No 16 GB dumps in git. Firefox pairs only when a public git commit pair exists — MFSA text without a commit is not a pair.
## Closure (2026-09-04)
Closed. All tasks done; `tests/test_pair_corpus.py` and `tests/test_vfc_harvest.py` pass (17). Seeds from OpenSSL, Firefox, Chromium, and curl landed (176 pairs). Residuals found in review are corpus quality and scorer fidelity, not seed coverage: 25 curl pairs start mid-comment because harvest anchors on a comment mention; 129 expected rows have `sink = "unknown"`; the overlay column equals inventory because C facts have no memory sink. These are owned by `pair-corpus-honesty`, which supersedes this spec on the corpus and scorer plane. Do not reopen.
.kiro/specs/real-world-vfc-slice/spec.json
+5 -2
View File
@@ -1,7 +1,7 @@
{
"feature_name": "real-world-vfc-slice",
"created_at": "2026-09-04T13:19:17Z",
"updated_at": "2026-09-04T13:40:00Z",
"updated_at": "2026-09-04T16:12:17Z",
"language": "en",
"phase": "implementation",
"approvals": {
@@ -18,5 +18,8 @@
"approved": true
}
},
"ready_for_implementation": true
"ready_for_implementation": true,
"closed": true,
"closed_at": "2026-09-04T16:12:17Z",
"superseded_by": "pair-corpus-honesty"
}
.kiro/steering/roadmap.md
+31 -2
View File
@@ -4,6 +4,10 @@
OpenUltraSAST can propose sinks with regex and prove a few of them in a sandbox, but adjudication is still Python-`ast` plus a tree-sitter CLI stub. Textbook SAST pairs (OWASP/Juliet) did not improve after the overlay: Youden went from **+9.09% inventory to 0% overlay** on the same 11-pair slice. Real operator trees and browser/C/C++ history are not in the catalog. The next work is to put a real tree-sitter extra under the existing overlay, then grow an offline function-level vuln-vs-fix slice from public VFC datasets (including Firefox/gecko-dev where commits exist) without vendoring whole megarepos.
## Direction (2026-09-04)
Detection logic is ~16% of the source; harness plumbing, governance, and ranking are the rest. Recall on real code comes from the model with tools plus proof; precision and silence come from static checks. Shift: grow the corpus past the license and review walls (pointer slices, review tiers), turn every reviewed pair into a searchable mechanism (Clearwing's variant loop), then build the path model (call graph, summaries, fix points), put the hunter on paths, let model answers become gated facts, and stop adding harness plumbing until the detector exists. Recall first under a false-positive ceiling, per profile. Youden on non-local slices stays off the merge gate.
## Approach Decision
- **Chosen**: Optional tree-sitter extra first, then a harvested VFC slice scored like `github` (honesty dashboard, not smoke gate). Isolated functions only. Firefox is one project filter among Chromium, OpenSSL, curl — not a special-case full mozilla-central clone.
@@ -19,6 +23,26 @@ OpenUltraSAST can propose sinks with regex and prove a few of them in a sandbox,
- **In**: tree-sitter + grammars as `openultrasast[semantic]`; CST walker into existing FileIR/taint; harvested function-level pairs from CVEfixes/MoreFixes/CWE-Bench-Java/SVEN with provenance; overlay scoring on that slice; keep `quick` zero-dep.
- **Out**: Cloning mozilla-central/Chromium into the repo; making overlay Youden a merge gate; buffer-size/CWE-121 semantics; inter-file taint; auto-facts from improve; LLM as parser.
## Measured 2026-09-04 (extra on, before pair-corpus-honesty)
| Slice | Vuln recall | Silent on fix | Pair-correct | Youden |
|-------|-------------|---------------|--------------|--------|
| sast (11) | 45.5% | 81.8% | 4/11 | +27.3% |
| vfc (176) | 3.4% | 82.4% | 0/176 | −14.2% |
Overlay column equals inventory on vfc (no C memory sink fact). 25 curl pairs parse_failed (harvest anchored on a comment mention).
## Measured 2026-09-04 (after pair-corpus-honesty scorer: coverage counted, function-scoped matching)
| Slice | Vuln recall | Silent on fix | Pair-correct | Youden | Achievable | Loss |
|-------|-------------|---------------|--------------|--------|------------|------|
| sast (11) | 54.5% | 72.7% | 4/11 | +27.3% | 4/10 (java-hash known_limit) | 4 unadjudicated |
| vfc (176) | 0.0% | 81.8% | 0/176 | −18.2% | 0/176 | 50 parse_failed files, 128 unadjudicated |
New slices (pair-corpus-honesty, 2026-09-05, after the debug round: shared declarator naming, containment matching, sink-location labels): vibe-py (35, human, Real-Vuln licensed repos) 13/35 pair-correct, Youden +37.1% overlay vs 7/35 inventory; vfc-js (17) 5/17, Youden +23.5%; agent-vfc (29, agent, tier `title`) 0/29, Youden 0.0: no excerpt carries a rule-known sink and the overlay has no TypeScript grammar (18 rows); the unreviewed-tier baseline, never a gate. vfc parse-failed files 50 → 36 after the harvest fix.
sast: java-sqli moved MISS→LEAK (coverage counted on both sides; the fixed twin leaks via parameter-as-source). vfc: recall fell to zero because every label now names a function and no C detection lands inside it; that is the honest baseline the engine specs measure against.
## Constraints
- Core `dependencies = []`. Parser wheels live only in an extra.
@@ -38,10 +62,15 @@ OpenUltraSAST can propose sinks with regex and prove a few of them in a sandbox,
## Direct Implementation Candidates
- [ ] Record inventory vs overlay Youden side-by-side in `ousast pairs --slice sast` JSON (small report field, not a new spec).
- [ ] Record inventory vs overlay Youden side-by-side in `ousast pairs --slice sast` JSON -- folded into pair-corpus-honesty task 2.3.
- [x] Add Firefox/gecko-dev, Chromium, OpenSSL pointers to `benchmarks/pairs/datasets.toml`.
## Specs (dependency order)
- [x] tree-sitter-overlay-extra -- Optional extra: tree-sitter + grammars, one CST walker, `quick` stays zero-dep. Dependencies: none
- [x] real-world-vfc-slice -- Harvest reviewed function-level vuln/fix pairs from large projects (Firefox/gecko-dev, Chromium, OpenSSL, …) as an honesty slice. Seed is testable (`ousast pairs --slice vfc`). Dependencies: none
- [x] real-world-vfc-slice -- Harvest reviewed function-level vuln/fix pairs from large projects (Firefox/gecko-dev, Chromium, OpenSSL, …) as an honesty slice. Seed is testable (`ousast pairs --slice vfc`). Dependencies: none. **Closed 2026-09-04, superseded by pair-corpus-honesty on the corpus/scorer plane.**
- [ ] pair-corpus-honesty -- Scorer counts coverage and matches by function; labels carry mechanism/provenance/split/known_limit; harvest anchors at the declarator with line-range and hunk modes; sast hygiene; new slices `vibe-py` (Real-Vuln-Benchmark), `vfc-js` (SecBench.js fix commits), `agent-vfc` (agent-trailer fixes); deterministic provenance fingerprint; per-profile improve acceptance. Dependencies: none (supersedes real-world-vfc-slice)
- [ ] corpus-seeded-mechanisms -- Every seeded or reviewed pair becomes a mechanism record (source shape, sink shape, guard the fix added); structural variant search over the scanned tree proposes matches as `variant` findings that the overlay and sandbox verify; leave-one-out recall on the pair corpus is the corpus's own detection rate; the improve loop gains a recall-raising `mechanisms` lever under the per-profile holdout gate. Principle: recall first under a false-positive ceiling, per profile. Dependencies: pair-corpus-honesty (tiers, pointer slices)
- [ ] reachability-flow-model -- Name-resolved call graph over `FunctionIR`; per-function taint summaries composed along the graph into path records; graph reachability drives rank, map, worth-fixing; hunter gets `path_context` and hunter findings with a locatable sink become sandbox candidates; closed-kind fact oracle with candidate cache; `facts` improve lever under the per-profile gate; fix points ranked by reachable risk removed. Dependencies: pair-corpus-honesty
- [ ] overlay-ir-completeness -- Shared CST walker completeness (C `init_declarator`, lvalue paths for field stores, per-grammar literal kinds, local ERROR-node tolerance, multi-line C/C++ declarators the walker names `<anon>` and `functions.py`'s one-line rescue cannot name: `chromium-cve-2022-2162`, `openssl-cve-2025-69421`; a Scorer-side multi-line-aware rescue is the cheaper alternative) plus Python parity with stdlib ast (f-string constness, augmented assignment, kwargs); injection-regime facts: container/transformer taint, `weak_literals` consulted, parameter sources off for injection. Measured on sast and vibe-py. Dependencies: pair-corpus-honesty
- [ ] guard-dominance-regime -- Guard and Use IR records; guard dominance by nesting or early exit; memory sinks (memcpy/malloc/deref) with operand roles, nullable callees, out-param transformers, fixed-array capacity; same regime applied to web absence bugs (route auth guard, identity from auth context). Consumes path records. Measured on vfc, vfc-js, agent-vfc. Dependencies: reachability-flow-model, overlay-ir-completeness