docs(pair-corpus-honesty): close the spec; escalations resolved, roadmap carries the measured state

- tasks.md: all tasks checked; the two escalations are closed (5.1 by the
  Req 10 pointer rows, 5.3 by the Req 9 review tiers); round 11 recorded
  with the carry-forwards for corpus-seeded-mechanisms and
  overlay-ir-completeness.
- roadmap: measured state after tiers and pointer slices (vibe-py 115,
  agent-vfc 59, gates byte-identical to pre-spec); pair-corpus-honesty
  marked completed.
- spec.json: completed 2026-09-05 after eleven review rounds.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-05 10:45:43 +02:00
co-authored by Claude Fable 5.1
parent 3c36f13e03
commit 27f9ec6a92
3 changed files with 14 additions and 8 deletions
+3 -1
View File
@@ -41,6 +41,8 @@ Overlay column equals inventory on vfc (no C memory sink fact). 25 curl pairs pa
New slices (pair-corpus-honesty, 2026-09-05, after the debug round: shared declarator naming, containment matching, sink-location labels): vibe-py (35, human, Real-Vuln licensed repos) 13/35 pair-correct, Youden +37.1% overlay vs 7/35 inventory; vfc-js (17) 5/17, Youden +23.5%; agent-vfc (29, agent, tier `title`) 0/29, Youden 0.0: no excerpt carries a rule-known sink and the overlay has no TypeScript grammar (18 rows); the unreviewed-tier baseline, never a gate. vfc parse-failed files 50 → 36 after the harvest fix.
Measured 2026-09-05 after review tiers (Req 9) and pointer slices (Req 10): vibe-py 115 = 35 vendored human (13/35, Youden +37.1%) + 80 `agent` pointer pairs over the 40 LLM-generated Real-Vuln repositories (0/80, Youden 0.0; the LLM-repo baseline, scored by the nightly `pairs --slice vibe-py --pointers` only, never in tests or gates); agent-vfc 59 = 29 vendored + 30 pointer, tier `title`, 1/59, Youden 0.0, 2 unresolved labels; sast, vfc-js, vfc unchanged; detection/map/local-pair gate outputs byte-identical to the pre-spec commit. Only `seeded`/`reviewed` pairs gate the improve loop. Carry-forwards: `language_unsupported` loss counter and multi-line C/C++ declarator rescue (overlay-ir-completeness), `profiles_ungated` reporting and an offline test for the agent-vfc pointer builder (corpus-seeded-mechanisms).
sast: java-sqli moved MISS→LEAK (coverage counted on both sides; the fixed twin leaks via parameter-as-source). vfc: recall fell to zero because every label now names a function and no C detection lands inside it; that is the honest baseline the engine specs measure against.
## Constraints
@@ -69,7 +71,7 @@ sast: java-sqli moved MISS→LEAK (coverage counted on both sides; the fixed twi
- [x] tree-sitter-overlay-extra -- Optional extra: tree-sitter + grammars, one CST walker, `quick` stays zero-dep. Dependencies: none
- [x] real-world-vfc-slice -- Harvest reviewed function-level vuln/fix pairs from large projects (Firefox/gecko-dev, Chromium, OpenSSL, …) as an honesty slice. Seed is testable (`ousast pairs --slice vfc`). Dependencies: none. **Closed 2026-09-04, superseded by pair-corpus-honesty on the corpus/scorer plane.**
- [ ] pair-corpus-honesty -- Scorer counts coverage and matches by function; labels carry mechanism/provenance/split/known_limit; harvest anchors at the declarator with line-range and hunk modes; sast hygiene; new slices `vibe-py` (Real-Vuln-Benchmark), `vfc-js` (SecBench.js fix commits), `agent-vfc` (agent-trailer fixes); deterministic provenance fingerprint; per-profile improve acceptance. Dependencies: none (supersedes real-world-vfc-slice)
- [x] pair-corpus-honesty -- Scorer counts coverage and matches by function; labels carry mechanism/provenance/split/known_limit; harvest anchors at the declarator with line-range and hunk modes; sast hygiene; new slices `vibe-py` (Real-Vuln-Benchmark), `vfc-js` (SecBench.js fix commits), `agent-vfc` (agent-trailer fixes); deterministic provenance fingerprint; per-profile improve acceptance; review tiers and pointer slices (amendment). Dependencies: none (supersedes real-world-vfc-slice). **Completed 2026-09-05 (11 review rounds).**
- [ ] corpus-seeded-mechanisms -- Every seeded or reviewed pair becomes a mechanism record (source shape, sink shape, guard the fix added); structural variant search over the scanned tree proposes matches as `variant` findings that the overlay and sandbox verify; leave-one-out recall on the pair corpus is the corpus's own detection rate; the improve loop gains a recall-raising `mechanisms` lever under the per-profile holdout gate. Principle: recall first under a false-positive ceiling, per profile. Dependencies: pair-corpus-honesty (tiers, pointer slices)
- [ ] reachability-flow-model -- Name-resolved call graph over `FunctionIR`; per-function taint summaries composed along the graph into path records; graph reachability drives rank, map, worth-fixing; hunter gets `path_context` and hunter findings with a locatable sink become sandbox candidates; closed-kind fact oracle with candidate cache; `facts` improve lever under the per-profile gate; fix points ranked by reachable risk removed. Dependencies: pair-corpus-honesty
- [ ] overlay-ir-completeness -- Shared CST walker completeness (C `init_declarator`, lvalue paths for field stores, per-grammar literal kinds, local ERROR-node tolerance, multi-line C/C++ declarators the walker names `<anon>` and `functions.py`'s one-line rescue cannot name: `chromium-cve-2022-2162`, `openssl-cve-2025-69421`; a Scorer-side multi-line-aware rescue is the cheaper alternative) plus Python parity with stdlib ast (f-string constness, augmented assignment, kwargs); injection-regime facts: container/transformer taint, `weak_literals` consulted, parameter sources off for injection. Measured on sast and vibe-py. Dependencies: pair-corpus-honesty