From 6fe86829936c7a403c851ff3f1d649f7d1d040fd Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Tue, 8 Sep 2026 11:47:10 +0200 Subject: [PATCH] docs(model-grounded-detection): 16 tasks in 6 groups, with a coupling map so Opus does not crash-discover Design approved. All 49 criteria trace to a task. The tasks carry an agent-guidance preamble aimed squarely at the escalations this session hit: the eighteen-defect lesson (fix the class, not the instance), the gate-baseline as the safety net (the whole learning/ package is off the gate path, so its deletion cannot move a gate -- a moved gate means you touched a retained path), the zero-dep capability-detection pattern to copy verbatim, and -- most important for the subtraction -- a full coupling map naming every touchpoint of the removed code (cli, pairs hunter path, config, reports, semantic seed/loo, the mechanism loop across improve/ and semantic/). The subtraction is not discovery by import error. Group 1 is pure gate-protected subtraction, sequenced so vocabularies are ported (1.1) and importers detached (1.2-1.3) before anything is deleted (1.4), then tests reduced in the same spirit (1.5) and the tree audited for orphans (1.6). Group 2 is a Joern go/no-go: task 2.3 reads go or stop, and groups 3-6 do not begin until it is green -- building past a red 2.3 is the mistake the whole spec exists to prevent. Parallelism is deliberately absent; the subtraction is sequential and the spike gates the rest. Not approved for implementation. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG --- .../specs/model-grounded-detection/spec.json | 10 +- .kiro/specs/model-grounded-detection/tasks.md | 252 ++++++++++++++++++ 2 files changed, 257 insertions(+), 5 deletions(-) create mode 100644 .kiro/specs/model-grounded-detection/tasks.md diff --git a/.kiro/specs/model-grounded-detection/spec.json b/.kiro/specs/model-grounded-detection/spec.json index b7c654b..b106720 100644 --- a/.kiro/specs/model-grounded-detection/spec.json +++ b/.kiro/specs/model-grounded-detection/spec.json @@ -1,9 +1,9 @@ { "feature_name": "model-grounded-detection", "created_at": "2026-09-08T00:00:00Z", - "updated_at": "2026-09-08T03:00:00Z", + "updated_at": "2026-09-08T04:00:00Z", "language": "en", - "phase": "design", + "phase": "tasks", "approvals": { "requirements": { "generated": true, @@ -11,10 +11,10 @@ }, "design": { "generated": true, - "approved": false + "approved": true }, "tasks": { - "generated": false, + "generated": true, "approved": false } }, @@ -24,5 +24,5 @@ "constrained-detector" ], "depends_on": [], - "approval_note": "2026-09-08 corrective brief drafted at the maintainer's direction ('can you produce a corrective spec that supersedes the project and sets a course'). Written after the learning-harness evolve rounds produced zero decisions distinguishable from noise, and after researching Clearwing (/home/mc/Source/clearwing, fork norandom/deepwing), Google Big Sleep, and the DARPA AIxCC 2025 finals. Not approved; requirements not generated. The central recommendation is to adopt Clearwing as the execution substrate rather than rebuild, and to contribute the web/logic detection path (taxonomy + IR candidate enumeration + obligation checker + a differential web oracle) that Clearwing's crash-oracle model does not reach. ; 2026-09-08 renamed execution-grounded-detection -> model-grounded-detection after the maintainer's correction: execution is not possible for all SAST targets and the bugs we own do not crash, so the arbiter is a program model built deterministically from code (IR candidate enumeration, binding-chain dataflow, obligation/guard-dominance) that checks LLM claims, producing a static evidence ladder (suspicion -> model_corroborated -> model_entailed); execution is a deferred tier for memory-safety-in-C only, adopted from Clearwing when a class needs it. ; 2026-09-08 requirements generated at the maintainer's direction ('write the requirements, and include to throw away broken parts. include the reduction of tests to what makes sense. I think there is an outgrowth problem here'): subtraction is Req 1-3 (remove the noise architecture, reduce 832 tests to surviving behavior, audit 99 modules for orphans) and precedes adoption of the Joern CPG arbiter (Req 4-8), the corpus as calibration set (Req 9), and the deferred Clearwing execution tier (Req 10). ; 2026-09-08 requirements approved by the maintainer in session ('approve the requirements and run the design phase'); design generated: subtraction is phase 1 (gate-protected, ~5000 lines removed before any Joern), a Joern spike on the injection family is phase 2 and can end the design before the integration cost is paid." + "approval_note": "2026-09-08 corrective brief drafted at the maintainer's direction ('can you produce a corrective spec that supersedes the project and sets a course'). Written after the learning-harness evolve rounds produced zero decisions distinguishable from noise, and after researching Clearwing (/home/mc/Source/clearwing, fork norandom/deepwing), Google Big Sleep, and the DARPA AIxCC 2025 finals. Not approved; requirements not generated. The central recommendation is to adopt Clearwing as the execution substrate rather than rebuild, and to contribute the web/logic detection path (taxonomy + IR candidate enumeration + obligation checker + a differential web oracle) that Clearwing's crash-oracle model does not reach. ; 2026-09-08 renamed execution-grounded-detection -> model-grounded-detection after the maintainer's correction: execution is not possible for all SAST targets and the bugs we own do not crash, so the arbiter is a program model built deterministically from code (IR candidate enumeration, binding-chain dataflow, obligation/guard-dominance) that checks LLM claims, producing a static evidence ladder (suspicion -> model_corroborated -> model_entailed); execution is a deferred tier for memory-safety-in-C only, adopted from Clearwing when a class needs it. ; 2026-09-08 requirements generated at the maintainer's direction ('write the requirements, and include to throw away broken parts. include the reduction of tests to what makes sense. I think there is an outgrowth problem here'): subtraction is Req 1-3 (remove the noise architecture, reduce 832 tests to surviving behavior, audit 99 modules for orphans) and precedes adoption of the Joern CPG arbiter (Req 4-8), the corpus as calibration set (Req 9), and the deferred Clearwing execution tier (Req 10). ; 2026-09-08 requirements approved by the maintainer in session ('approve the requirements and run the design phase'); design generated: subtraction is phase 1 (gate-protected, ~5000 lines removed before any Joern), a Joern spike on the injection family is phase 2 and can end the design before the integration cost is paid. ; 2026-09-08 design approved and tasks generated ('approve the design and generate the tasks. add infos so that Opus can follow the tasks without running into complexity escalations'): tasks carry an agent-guidance preamble with the eighteen-defect lesson, the gate-baseline safety net, the zero-dep capability-detection pattern, and a full coupling map of every learning/mechanism touchpoint so the subtraction is not crash-driven discovery; group 1 is pure gate-protected subtraction, group 2 is a Joern go/no-go that gates groups 3-6." } diff --git a/.kiro/specs/model-grounded-detection/tasks.md b/.kiro/specs/model-grounded-detection/tasks.md new file mode 100644 index 0000000..c40b699 --- /dev/null +++ b/.kiro/specs/model-grounded-detection/tasks.md @@ -0,0 +1,252 @@ +# Implementation Plan: model-grounded-detection + +## Guidance for the implementing agent + +Read this before starting. It exists so you never have to reverse-engineer a decision this spec already made. + +- **The failure mode to avoid.** The prior work logged eighteen defects, three of them the same shape: a fix + applied to one instance while the *class* of the bug stayed. When you fix a retry, a key, an error path — + ask "where else is this shape?" and fix all of them in the one change. That single discipline prevents most + of the escalations that happened here. +- **The gate baseline is sacred and is your safety net.** `benchmarks/measurements/2026-09-06-gate-baseline.txt` + holds the byte-exact output of `python -m openultrasast.{gate,map_gate,pair_gate}`. Run all three before and + after any change and diff against it. If a gate moves, you changed something that was never supposed to + change — stop and find it. The whole `learning/` package is **absent from the gate path** (the gates import + only `pairs`, `benchmark`, `findings`, `mapping`, `preprocess`, `rank`, `complexity.map`), so its deletion + *cannot* move a gate; a moved gate means you touched a retained path by mistake. +- **RED first, always.** Write the failing test, watch it fail for the right reason, then implement. One + commit per task with selective `git add`. Push when a group's independent review passes. +- **Zero-dependency core.** `pyproject.toml` `dependencies = []` never changes. Joern, the LLM endpoint and the + execution tier are capability-detected, never imported at module load. Copy the exact pattern in + `semantic/extra.py::has_semantic_extra` and `sandbox/probe.py::resolve_sandbox_probe` — probe, don't import. +- **Redaction on every prompt and persisted artifact** (`redaction.redact_secrets`). Non-negotiable. +- **Two phases can end the design cheaply.** Phase 1 (subtraction) is reversible via git and gate-protected. + Phase 2 (the Joern spike) is a go/no-go: if Joern cannot produce a deterministic verdict on the injection + family at acceptable cost, the design stops there, exactly as the candidate ceiling could have stopped the + prior spec. Do not build groups 3–6 until phase 2 is green. + +### The coupling map (for the subtraction, group 1) + +These are every touchpoint of the code being removed. Handle all of them in group 1; do not discover them by +running into import errors. + +- **`learning/` package** — imported by: `cli.py` (the `learning` subcommand group + `_run_family_detectors` + + `_learning*` helpers + `from .learning.families import FamilyTaxonomy` + `from .learning.proposer import + Proposer`), `pairs.py` (the hunter-scoring path — `_evaluate_hunter_pair`, `_family_metrics`, `_hunter_outcome`, + `build_pair_signals`, function-local imports of `learning.{families,scoring,classify,rounds}` at lines + ~519, 1151, 1166, 1184), `semantic/seed.py` and `semantic/loo.py` (the teacher-rule import from + `learning.split`), and `learning/endpoint.py` (retained — re-homed, not deleted). +- **Mechanism/evolve loop** — `improve/evolve.py`, `improve/validator.py`, `semantic/mechanisms.py`, + `semantic/variants.py`, `semantic/variant_search.py`, `semantic/prove_budget.py`, `semantic/loo.py`, + `semantic/seed.py`, and `semantic/__init__.py` (re-exports). `cli.py` has the `--mechanism-candidates` + surface and `_mechanisms_export`. +- **`config.py`** — `LearningConfig`, `_load_learning`, and the `learning` field on `ResolvedConfig`. +- **`reports.py`** — the `learning=` keyword on the report writers and `_append_learning`; the SARIF + `family`/`detector`/`verifier` properties (keep `family`, drop `detector`/`verifier`). +- **`pairs.py` is gate-load-bearing** via `pair_gate` (the overlay/inventory path, + `_evaluate_overlay_pair`/`_evaluate_inventory_pair`). The *hunter-scoring* path is the learning coupling and + is **not** on the gate path. Remove the hunter path and its learning imports; keep the overlay/inventory + path untouched; confirm `pair_gate` byte-identical. + +### What is retained and re-homed (do not delete) + +`learning/families.py` → `model/taxonomy.py`; `learning/candidates.py` → `model/candidates.py`; +`learning/endpoint.py` → `model/endpoint.py`. `semantic/obligations/` stays (its facts feed the dominance +queries in group 4). The gates, `redaction.py`, `pairs.py` overlay/inventory path — unchanged. + +--- + +## Group 1 — Subtract and audit (gate-protected, no Joern yet) + +- [ ] 1.1 Port the security vocabularies out before deleting their modules + - Create `src/openultrasast/model/specs.py` and move into it, as data, the knowledge the deleted modules + carry: `_GUARD_PATTERNS` and the guard-kind vocabulary from `semantic/variants.py`; the sink token tables + (from `benchmarks/pairs/catalog_gen.py::_SINK_TOKENS` and `semantic/facts.py` SinkFacts) as the seed + `TaintSpec` source/sink/sanitizer lists per family; a reference to the obligation facts + (`semantic/obligations/facts.py`) for the `DominanceSpec` operations/dischargers. This is data movement, + not logic — the CPG queries that consume it come in groups 2 and 4. + - Observable: `model/specs.py` imports nothing from `learning/`, `improve/`, `semantic/variants`, + `semantic/mechanisms`; a test asserts every family in the taxonomy has a `TaintSpec` or a `DominanceSpec` + seed, and that the guard vocabulary is preserved verbatim. + - _Requirements: 1.2, 7.1, 7.2, 7.3_ + +- [ ] 1.2 Re-home the three retained modules + - Move `learning/families.py` → `model/taxonomy.py`, `learning/candidates.py` → `model/candidates.py`, + `learning/endpoint.py` → `model/endpoint.py`. Update their imports and every importer. `candidates.py` + keeps its behaviour exactly (it is the suspicion-band feeder, Req 8.4) — its tests move with it and stay + green. + - Observable: the moved modules import no other `learning/` module; `test_candidate_enumeration.py` passes + unchanged against the new path; the taxonomy loads and every family resolves. + - _Requirements: 4.4, 7.1_ + - _Depends: 1.1_ + +- [ ] 1.3 Detach the retained paths from the removed code + - `pairs.py`: remove the hunter-scoring path (`_evaluate_hunter_pair`, `_hunter_outcome`, `_family_metrics`, + `PairEvalResult.per_family`, `build_pair_signals` split arg, the `k_runs`/`hunter` plumbing) and its + `learning.*` imports; keep `_evaluate_overlay_pair`/`_evaluate_inventory_pair` and the `context_files` + machinery. `cli.py`: remove the `learning` subcommand group, `_run_family_detectors`, the `_learning*` + helpers, the `--mechanism-candidates` surface and `_mechanisms_export`; keep `scan`, `benchmark`, + `improve` (rule-status only), `pairs` (overlay), `mcp`/`index` for now (audited in 1.6). `config.py`: + remove `LearningConfig`/`_load_learning`. `reports.py`: drop the `learning=` kwarg, `_append_learning`, + and the SARIF `detector`/`verifier` properties (keep `family`). `semantic/{seed,loo}.py`: remove the + teacher-rule import from `learning.split`. + - Observable: `python -m compileall src` clean; the three gates byte-identical to the baseline; the full + suite still collects (failures expected only in tests that pin removed behaviour, addressed in 1.5). + - _Requirements: 1.3, 1.4, 11.2_ + - _Depends: 1.2_ + +- [ ] 1.4 Remove the noise architecture and the mechanism/evolve loop + - Delete `learning/{rounds,acceptance,proposer,journal,canaries,verifiers,difficulty,scoring,classify, + detectors,split,publish,slices,predicates,judgments}.py` (everything under `learning/` except the three + modules re-homed in 1.2, which are already gone from the directory), then remove the now-empty `learning/` + package. Delete `improve/evolve.py`, `improve/validator.py` (mechanism parts — keep rule-status validation + if `improve` is retained by the 1.6 audit), `semantic/{mechanisms,variants,variant_search,prove_budget, + loo,seed}.py`, and their re-exports in `semantic/__init__.py`. + - Guardrail: run this **after** 1.1–1.3, so every vocabulary is ported and every importer detached. If a + delete breaks an import, the fix is in the importer (a 1.3 miss), never a re-addition of the module. + - Observable: an import-graph check (a test that walks `ast` over every `src/**/*.py` and asserts no import + names a removed module) passes; `compileall` clean; three gates byte-identical; the extra-free matrix and + the full matrix both collect. + - _Requirements: 1.1, 1.2, 1.2b, 1.3, 1.4_ + - _Depends: 1.3_ + +- [ ] 1.5 Reduce the test suite to surviving behaviour + - For every deleted module, delete the test files that existed only to pin it (`test_learning_*`, + `test_mechanism_*`, `test_obligation_*` only where it tested the lever not the checker, `test_pair_*` + hunter-scoring cases, etc.) in this same change. For every surviving test, confirm it maps to a surviving + requirement of this spec or a still-shipping feature; delete any that map to nothing, and delete outright + any test that asserts a shape "does nothing", greps `--help`, or passes with its capability absent. + - Observable: the suite is smaller than 832 tests; a one-line-per-cluster note records the before/after + count and the requirement each removed cluster served; the pinned floor (gate-baseline, redaction, + per-slice honesty, `test_candidate_enumeration`) is green. + - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5_ + - _Depends: 1.4_ + +- [ ] 1.6 The module audit manifest + - Add `model/audit.py` that classifies every `src/**/*.py` as `load_bearing` (reachable from a gate, the + scan CLI, or a surviving requirement), `standalone_capability` (its own CLI/users — e.g. `mcp.py`, + `skills.py`), or `orphaned` (no importer, no entry, no requirement). Remove the orphans. Give + `fusion.py`, `mcp.py`, `skills.py`, `harness_ext.py`, `hunter_harness.py` an explicit keep-or-remove line + each with its reason. Commit `benchmarks/measurements/2026-09-08-module-audit.json`. + - Observable: the manifest classifies every module; a test asserts no module classified `orphaned` remains + in the tree; the five named subsystems each have a recorded decision. + - _Requirements: 1.5, 3.1, 3.2, 3.3, 3.4_ + - _Depends: 1.5_ + +## Group 2 — The Joern spike on one family (go/no-go) + +- [ ] 2.1 The CpgBackend seam, out of process + - `src/openultrasast/cpg/capability.py::has_cpg()` — Joern on PATH (probe `joern-parse --help`), the + `has_semantic_extra` pattern exactly. `cpg/backend.py`: `CpgResult` (a built `cpg.bin` path + a `run(query, + params)->JSON` callable); `JoernBackend` (subprocess: `joern-parse ` then `joern --script + cpg/queries/.sc`, parsing stdout JSON, every step timeout-bounded, fail-closed to `None`); + `NullBackend` (`available()` False, `build()` None); `resolve_cpg_backend()`. **Never import a JVM + binding; the subprocess is the only Joern touch-point in the codebase.** + - Observable: with Joern absent, `resolve_cpg_backend()` is the `NullBackend` and `build()` returns None; a + scripted fake backend (canned JSON) exercises the parsing without Joern; the seam has no import of any + non-stdlib module at load time. + - _Requirements: 4.1, 4.2, 4.3, 4.5, 11.3_ + - _Depends: 1.6_ + +- [ ] 2.2 The injection taint query and its verdict + - `cpg/queries/taint.sc` — a CPGQL script parameterised by a `TaintSpec` (sources, sinks, sanitizers) that + emits, per candidate, the JSON `{sink, source, path, sanitized: bool}`. `model/taint.py::verdict(cpg, + spec, candidate)` → `ENTAILED` when a source→sink path exists with `sanitized=false`, `CORROBORATED` when + a path exists but a sanitizer may apply, else `None`. Injection `TaintSpec` from `model/specs.py`. + - Observable: on a fixture with `os.system(request.args['x'])` the verdict is `ENTAILED`; on the + parameterised-query fixed twin it is not; the query script is deterministic. + - _Requirements: 5.1, 5.2, 6.1_ + - _Depends: 2.1, 1.1_ + +- [ ] 2.3 Determinism, and the go/no-go reading + - `test_two_runs_over_one_cpg_give_an_identical_verdict_sequence`. Run the injection taint verdict over the + injection slice, regenerate the entailment ceiling for injection, and write the reading into the + Implementation Notes: does Joern produce a deterministic verdict at acceptable per-target cost and time? + If not, stop the design here and record why. + - Observable: the determinism test passes repeatedly; the injection entailment number is committed against + the 2.2%/multi-hop baseline; the notes state go or no-go. + - _Requirements: 6.4, 6.5_ + - _Depends: 2.2_ + +## Group 3 — The ladder and the judge + +- [ ] 3.1 The evidence ladder as the verdict type + - `model/ladder.py`: `Rung` enum (suspicion/model_corroborated/model_entailed/execution_confirmed), + `Verdict` (rung, family, witness, contradiction). Findings carry the rung in markdown, SARIF and JSON. + - Observable: a finding at each rung round-trips through every output format with its rung intact; nothing + is reported above `suspicion` without a `Verdict` establishing it. + - _Requirements: 5.1, 5.3, 5.4_ + - _Depends: 2.3_ + +- [ ] 3.2 The judge: one bounded question, checked against the model + - `model/judge.py::judge(candidate, cpg, spec, *, client, model)` — if the model entails, report + `model_entailed` with no LLM call; else ask the LLM one typed question (source expression / guard / family) + with no tools; then check the answer against the CPG — contradiction drops the claim (records why), + corroboration advances to `model_corroborated`, neither leaves `suspicion`. Reuse `model/endpoint.py`. + - Observable: an entailed candidate makes zero LLM calls; a claim the CPG contradicts is dropped with the + contradiction recorded; a claim the CPG supports advances; there is no rounds/acceptance/floor/evolve code + left to import. + - _Requirements: 8.1, 8.2, 8.3, 8.4, 8.5, 11.4_ + - _Depends: 3.1_ + +## Group 4 — Absence and the rest of the taxonomy + +- [ ] 4.1 Guard dominance for the absence families + - `cpg/queries/dominance.sc` + `model/dominance.py::verdict(cpg, spec, candidate)` — `ENTAILED` when an + obligated operation is dominated by no discharger while its siblings are, seeded from the obligation + facts. This is the arbiter for the classes that never crash (access_control). + - Observable: on the ThreatByte/VAmPI absence fixtures the vulnerable handler is `ENTAILED` (guard absent) + and the fixed twin is not; a handler with no obligated operation yields no verdict. + - _Requirements: 6.2, 7.3_ + - _Depends: 3.2_ + +- [ ] 4.2 The remaining TaintSpecs and the ceiling regeneration + - Fill `TaintSpec`s for path, deserialization, untrusted_destination, output_encoding, prototype, and the + config constant-abstraction arm (`model/taint.py`). Regenerate the full entailment ceiling per family and + per slice and commit it; name where each family falls back to `suspicion`. + - Observable: the entailment ceiling artifact carries every web/logic family with its entailed/corroborated + split; the number is higher than the flat-IR 2.2% and the fallbacks are named. + - _Requirements: 6.3, 6.5, 7.2_ + - _Depends: 4.1_ + +## Group 5 — Corpus calibration and reporting + +- [ ] 5.1 The corpus as the model's calibration set + - `model/calibrate.py` — over the vulnerable-parent/fix pairs, run the model on both sides and record + whether it distinguishes them; a pair it cannot split is a named model gap (with the reason), not a miss. + Provide the CVE-history harvest seam (recipe → parent/fix, the fix diff read as oracle). + - Observable: the calibration report lists model gaps by family with reasons; a pair the model splits is + recorded as covered; per-slice with the real-world slice deciding. + - _Requirements: 9.1, 9.2, 9.4, 11.1_ + - _Depends: 4.2_ + +- [ ] 5.2 Reporting: per-slice, overfitting gap, rung + - `model/report.py` — per-slice numbers, the overfitting gap beside every headline, the evidence rung on + every finding; no cross-slice aggregate without its per-slice rows. Re-home the surviving per-slice + honesty discipline (it was in the deleted `scoring`/`publish`). + - Observable: the report carries a row per slice and refuses a bare aggregate; the overfitting gap is + published; every finding shows its rung. + - _Requirements: 9.3, 11.1_ + - _Depends: 5.1_ + +## Group 6 — The execution tier stub (deferred) + +- [ ] 6.1 The execution_confirmed rung and the Clearwing adoption seam + - Wire `Rung.execution_confirmed` as an optional escalation reached only where the model cannot decide + (memory/C first), behind a capability probe for the Clearwing fork. Do **not** reimplement any sandbox + machinery; the seam calls the adopted tool. Not required for any web/logic class at v1. + - Observable: a memory-family finding the model leaves at `suspicion` is the only kind eligible for the + execution tier; with Clearwing absent the rung is simply never reached and a reason is recorded; no + web/logic path depends on it. + - _Requirements: 10.1, 10.2, 10.3_ + - _Depends: 5.2_ + +## Task-plan review notes + +- Group 1 is the largest and the only one with no new capability — it is pure subtraction, ~5,000 lines + removed, gate-protected at every step. It must land and pass an independent review before group 2 begins. +- Group 2 is a go/no-go. If task 2.3 reads no-go on Joern, groups 3–6 do not start; the model layer either + finds another CPG engine or the design returns to the maintainer. Building past a red 2.3 is the exact + mistake this whole spec was written to stop. +- `(P)` parallelism is deliberately absent: the subtraction is sequential by dependency, and the Joern spike + gates everything after it. Do not fan out.