marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-19 04:59:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: plain-English pass over the active specs

Browse Source 
Ran the humanizer rules over 79 prose files in the active specs (brief,
design, requirements, research and narrative docs). Removed em dashes,
significance inflation, forced rule-of-three, bold-label list items and
filler. Skipped tasks.md checklists and the retired specs (harnessx-*,
openrouter-sast-harness, unseen-repo-evaluation).

Every number, path, command, hash and identifier was preserved: checked
each file against HEAD for drift in numbers, inline code and code fences,
zero drift across all 79 files.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-12 18:16:41 +02:00
co-authored by Claude Opus 4.8
parent b08f372022
commit 39a1873e5b
79 changed files with 1103 additions and 1103 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/model-grounded-detection/brief.md
+61 -61
View File
@@ -7,7 +7,7 @@
> deterministically from code**; execution is a later, conditional tier. The correction is integrated below.
**Status:** course-setting. Supersedes `learning-harness` and `constrained-detector`. Not a requirements
document — a decision about what to build, what to delete, and what to adopt, written after the
document, a decision about what to build, what to delete, and what to adopt, written after the
learning-harness line produced numbers its own measurements say to discard.
## 1. The premise we got wrong
@@ -17,19 +17,19 @@ averaged (K runs), floored (per-family noise budget), gated (sign test), and slo
Every mechanism in `learning-harness` follows from that assumption. This session measured all of it and it
does not hold:
- Five evolve rounds, $2.17, **zero decisions distinguishable from chance** — because the acceptance rule
- Five evolve rounds, $2.17, **zero decisions distinguishable from chance**, because the acceptance rule
compares two ensemble votes over a detector that never runs the same investigation twice.
- 120 runs of one detector produced **120 distinct trajectories**. The variance is the loop's freedom to
search, not the model's uncertainty about an answer.
- The noise floor a regression budget is derived from measures *how often an unconstrained search happens
to reach the sink* — so the worse the search wanders, the more collateral damage a round is licensed to do.
to reach the sink*, so the worse the search wanders, the more collateral damage a round is licensed to do.
The assumption is also empirically false as a claim about LLMs, and the evidence is now overwhelming:
- **Google Big Sleep** (Nov 2024): the first LLM-discovered zero-day in widely-used real-world software — a
stack buffer underflow in SQLite — found by *variant analysis*, a sandboxed test case, and an
- Google Big Sleep (Nov 2024): the first LLM-discovered zero-day in widely-used real-world software, a
stack buffer underflow in SQLite, found by *variant analysis*, a sandboxed test case, and an
auto-generated root-cause report. Not pattern-matching; reproduction.
- **DARPA AIxCC finals** (DEF CON 33, Aug 2025): seven autonomous systems processed 54M lines of code,
- DARPA AIxCC finals (DEF CON 33, Aug 2025): seven autonomous systems processed 54M lines of code,
patched 43 of 54 synthetic vulnerabilities and **uncovered 18 previously-unknown real 0-days**. Buttercup
(Trail of Bits, 2nd place) found **28 vulnerabilities across 20 CWEs at 90% accuracy, $181 per point,
using non-reasoning LLMs only**, ~$152 per task, 45 minutes each.
@@ -47,11 +47,11 @@ Lazarus-AI, MIT, an open reproduction of Anthropic's Glasswing) share one shape:
> **The LLM is a judgement layer on top of a deterministic substrate that arbitrates its claims. Truth comes
> from that substrate, not from a score against a labeled corpus.**
In the memory-safety systems above the substrate is *execution* — a crash is the arbiter. But execution is
In the memory-safety systems above the substrate is *execution*. A crash is the arbiter. But execution is
only the arbiter you reach for when the target builds in isolation and the bug crashes, which is the C world,
not ours. The general form of the substrate is a **program model built from the code**: a deterministic
abstraction rich enough to *check* a claim — does the source actually reach the sink through resolved
dataflow; is there a guard on the path or isn't there; is the entry point attacker-reachable. Execution is one
abstraction rich enough to *check* a claim (does the source actually reach the sink through resolved
dataflow; is there a guard on the path or isn't there; is the entry point attacker-reachable). Execution is one
way to arbitrate; a model is the way that works for a SAST project and for bugs that never crash.
Clearwing's evidence ladder is the whole thesis in one type:
@@ -81,24 +81,24 @@ real. **We built K-runs and noise floors because we had no ground truth. They ge
This is not a total loss, and the course is not "throw everything away." Two results here are real,
measured, and *absent from Clearwing*:
1. **IR-driven candidate enumeration with a published ceiling.** We measured that the pattern ruleset caps
1. IR-driven candidate enumeration with a published ceiling. We measured that the pattern ruleset caps
recall at 12.1% while call-sites-and-bindings from the semantic IR reach **96.6%** across the web slices,
a median of ten sites per function. Clearwing's candidate feeder is C-first (taint tables for
`memcpy`-shaped bugs); its `web_framework` specialist is a *single unconstrained prompt* with no
equivalent candidate enumerator for logic/web classes.
2. **Absence-bug detection.** Our `obligations` work — an access-control bug is the *absence* of a guard
around an operation its siblings protect — has no analogue in Clearwing, whose ladder is built around a
2. Absence-bug detection. Our `obligations` work (an access-control bug is the *absence* of a guard
around an operation its siblings protect) has no analogue in Clearwing, whose ladder is built around a
*crash*. An IDOR or a missing authorization check never crashes a sandbox. Clearwing's execution
ground-truth model does not reach these classes at all, and they are exactly the "dangerous but
low-hanging" web bugs this project was chartered to catch.
So the gap Clearwing leaves open is **the web/logic taxonomy without a crash oracle** — which is where our
So the gap Clearwing leaves open is **the web/logic taxonomy without a crash oracle**, which is where our
IR candidate enumerator, our family taxonomy, and our obligation checker are genuinely ahead.
## 4b. The model layer, and the evidence ladder it produces
The arbiter is a program model constructed deterministically from the code — the thing an LLM claim is
The arbiter is a program model constructed deterministically from the code, the thing an LLM claim is
*checked against*, so that a finding's truth does not rest on the LLM that proposed it. We already hold the
first rungs of it; they were buried under a noise layer instead of being made the arbiter.
@@ -113,16 +113,16 @@ The evidence ladder is the model layer made legible. Each rung is a stronger, st
Two properties are the whole point:
- **`model_corroborated` and above are deterministic.** Run the checker twice over the same model and it
returns the same verdict. That removes the run-to-run noise the same way a crash would — no K-runs, no
floor, no sign test — *without needing a runnable target*. The noise this session measured was the LLM
- `model_corroborated` and above are deterministic. Run the checker twice over the same model and it
returns the same verdict. That removes the run-to-run noise the same way a crash would (no K-runs, no
floor, no sign test), *without needing a runnable target*. The noise this session measured was the LLM
free-searching; a model check does not search.
- **The LLM's job shrinks to the residual the model cannot decide** — is this string genuinely
attacker-controlled, is this guard semantically sufficient — and even that answer is checked against the
- The LLM's job shrinks to the residual the model cannot decide (is this string genuinely
attacker-controlled, is this guard semantically sufficient), and even that answer is checked against the
model rather than averaged over runs. The LLM stops being the oracle and becomes a proposer the model
audits.
**Measured, 2026-09-08 — the entailment ceiling** (`benchmarks/measurements/2026-09-08-model-entailment-ceiling.json`,
**Measured, 2026-09-08, the entailment ceiling** (`benchmarks/measurements/2026-09-08-model-entailment-ceiling.json`,
offline, no LLM):
| rung | web/logic taxonomy | for comparison |
@@ -134,17 +134,17 @@ offline, no LLM):
The gap between 96.6% (a site exists) and 2.2% (the model *decides* the vulnerability) is the gap between
enumerating and arbitrating, and it is the whole finding. The model cannot be a *pure* arbiter at v1. But it
is not a wall — the measurement diagnoses depth, not impossibility: one-hop source resolution (real taint sits
is not a wall. The measurement diagnoses depth, not impossibility: one-hop source resolution (real taint sits
1–3 binds upstream: `q = "…" + id; execute(q)`), a closed literal sink table (misses project wrappers and the
path/xss/prototype sinks), and the function-naming gap that eats 20/50 injection pairs as `no_function`.
Deepening the taint to reuse the candidate enumerator's depth-4 chain already moved injection 24% → 48%.
So the architecture the numbers support is a **layered arbiter**, not a pure one:
- where the model **entails** — report `model_entailed`, deterministic, the LLM not needed;
- where the model **corroborates a taint path** (~35% and rising) — the LLM judges, but its claim is *checked*
- where the model **entails**, report `model_entailed`, deterministic, the LLM not needed;
- where the model **corroborates a taint path** (~35% and rising), the LLM judges, but its claim is *checked*
against the resolved path, so run-to-run noise is removed on this band;
- where the model can only **enumerate** (the rest, up to 96.6%) — honest `suspicion`, the LLM proposes with
- where the model can only **enumerate** (the rest, up to 96.6%), honest `suspicion`, the LLM proposes with
no deterministic backstop, and this is exactly the band where the deferred execution / differential tier
earns its place.
@@ -152,32 +152,32 @@ Pure `model_entailed`-as-arbiter is the north star. The v1 is the layered arbite
named by the measurement: fix function naming, build per-family sink/sanitizer tables, and lift the flow
model onto the multi-hop binding chain we already have.
This is the classical SAST arbiter — dataflow / abstract interpretation, the CodeQL and Infer lineage —
This is the classical SAST arbiter (dataflow / abstract interpretation, the CodeQL and Infer lineage),
with the LLM supplying the semantic judgement the model is weakest at (aliasing intent, whether a value is
really untrusted, whether a guard really covers the case). It does not need soundness. A partial model that
can confirm *some* claims and honestly returns "could not confirm — stays suspicion" for the rest is already
can confirm *some* claims and honestly returns "could not confirm, stays suspicion" for the rest is already
the independent arbiter we never had.
## 4c. IL or AST, build or adopt — the substrate decision
## 4c. IL or AST, build or adopt, the substrate decision
**IL, and adopt one.** Both halves are forced by the entailment measurement.
*Why an IL, not the AST we have.* Our `semantic/ir.py` is not an IL — it is a lossy AST projection: per-function
*Why an IL, not the AST we have.* Our `semantic/ir.py` is a lossy AST projection rather than an IL: per-function
flat lists of calls and binds, no control-flow graph, no basic blocks, no data-dependence edges. That is
*why* entailment sits at 2.2%. Real abstract interpretation is a fixpoint over a **CFG**: guard *dominance*
(is the check on every path to the operation) is a dominator-tree question, and path-sensitive taint (a
sanitizer on the taint path specifically) needs the branch structure. A flat bind list cannot express either;
it can only over-approximate taint flow-insensitively by name, which is the one-hop ceiling we measured. The
model layer wants the classical IL that merges AST + CFG + PDG + call graph into one graph — a **Code Property
Graph (CPG)** — over which taint is a reachability query (source → sink minus sanitizer) and absence is a
model layer wants the classical IL that merges AST + CFG + PDG + call graph into one graph (a **Code Property
Graph (CPG)**), over which taint is a reachability query (source → sink minus sanitizer) and absence is a
dominance query.
*Why adopt, not build.* Building a multi-language CFG/PDG/taint engine on top of tree-sitter is precisely
reinventing CodeQL, Semgrep and Joern — a multi-year effort, and the maximal case of the failure mode this
reinventing CodeQL, Semgrep and Joern, a multi-year effort, and the maximal case of the failure mode this
session logged eighteen times. **Joern** is the adoptable answer: Apache-2.0, a CPG engine for exactly our
corpus (C/C++/Java/JavaScript/Python/Kotlin/binary), a built-in taint-propagation engine, embeddable, and
there is already a published CPG-plus-LLM bridge (Codebadger, an MCP server over Joern) and an arXiv line on
CPGs as the substrate for LLM program analysis. Joern *is* "a model built up from code" as a graph — the exact
CPGs as the substrate for LLM program analysis. Joern *is* "a model built up from code" as a graph, the exact
thing the maintainer described. The alternatives: CodeQL is more powerful but least adoptable (restrictive
license, query-only, GitHub-owned); Semgrep's taint engine is lighter and we already have integration
precedent through Clearwing's sidecar, but interprocedural taint is behind the paid Pro engine.
@@ -192,14 +192,14 @@ source ──▶ Joern: compile to CPG (one IL: AST + CFG + PDG + call graph, la
```
The CPG is language-agnostic once its frontend exists, so the taint and dominance logic is written *once*, not
per language — which is the whole reason our per-language tree-sitter approach kept hitting naming and grammar
per language, which is the whole reason our per-language tree-sitter approach kept hitting naming and grammar
gaps.
*What stays ours, and is the contribution.* Joern is a workbench, not an opinionated web/logic detector with
an LLM judge. What we own and port onto it: the closed **family taxonomy**; the **source / sink / sanitizer /
guard models** for the web/logic CWEs — our obligation facts and closed guard vocabulary become CPG queries
and taint specifications; and the **LLM-adjudication-checked-against-the-CPG** orchestration. That security
modelling — expressed as queries over an adopted CPG — is where our work is genuinely ahead of both Joern
guard models** for the web/logic CWEs (our obligation facts and closed guard vocabulary become CPG queries
and taint specifications); and the **LLM-adjudication-checked-against-the-CPG** orchestration. That security
modelling, expressed as queries over an adopted CPG, is where our work is ahead of both Joern
(no opinion on these classes) and Clearwing (crash-oracle, no arbiter for bugs that do not crash).
*The one real cost.* Joern is JVM/Scala; embedding it in a Python tool means its server mode or the MCP bridge,
@@ -212,38 +212,38 @@ noise architecture; keep execution as a later, conditional tier.**
Concretely, superseding both prior specs:
1. **Delete, do not fix.** The K-run machinery, the per-family noise floor, the regression budget, the sign
1. Delete, do not fix. The K-run machinery, the per-family noise floor, the regression budget, the sign
test, the evolve loop, the acceptance rule, the mechanism-lever optimizer. All of it existed to average an
LLM oracle we could not otherwise trust. A deterministic model arbiter removes the need for every piece.
`learning-harness` Req 3.5, 8.2, 9.5 and the evolve tasks are retired, not amended.
2. **Adopt a CPG engine (Joern) as the arbiter substrate; port our security models onto it.** We do not lift
2. Adopt a CPG engine (Joern) as the arbiter substrate; port our security models onto it. We do not lift
our flat IR to a CFG ourselves (§4c). The taxonomy, the source/sink/sanitizer/guard models, and the
obligation/absence logic move from prompt context to **CPG taint specifications and dominance queries** —
obligation/absence logic move from prompt context to **CPG taint specifications and dominance queries**,
deterministic `model_corroborated` and `model_entailed` verdicts over an engine that already has the CFG
and PDG our IR lacks. Our tree-sitter IR retains one job: cheap candidate enumeration for the `suspicion`
band where the CPG has no verdict.
3. **The LLM proposes; the model disposes.** One bounded, typed question per candidate — the constrained
detector we already started — but the answer is not scored against a corpus, it is *checked against the
model*. A claim the model corroborates advances; a claim it contradicts is dropped with the contradiction
3. The LLM proposes; the model disposes. One bounded, typed question per candidate (the constrained
detector we already started), with the answer *checked against the
model* rather than scored against a corpus. A claim the model corroborates advances; a claim it contradicts is dropped with the contradiction
recorded; a claim it can neither confirm nor deny stays `suspicion` and is reported as such. No K-runs, no
evolve loop.
4. **Corpus is the model's calibration set, not the LLM's training signal.** Keep the vulnerable-parent /
fix-commit pairs — but their role changes. The pair is ground truth for *the model's precision*: the fix
4. Corpus is the model's calibration set, not the LLM's training signal. Keep the vulnerable-parent /
fix-commit pairs, but their role changes. The pair is ground truth for *the model's precision*: the fix
is what the model should be able to see (a guard added, a sanitizer inserted, a bound checked), and a pair
the model cannot distinguish is a gap in the model, named and measured, not a detector miss to average.
Harvest more pairs at repository scale from CVE history, as Clearwing does; truth is the fix diff read by
the model, not a label we hand-assign.
5. **Execution is a deferred, conditional tier.** For the classes a static model genuinely cannot arbitrate —
memory-safety-in-C, where reachability and exploitability turn on runtime layout — a sandbox crash oracle
5. Execution is a deferred, conditional tier. For the classes a static model genuinely cannot arbitrate
(memory-safety-in-C, where reachability and exploitability turn on runtime layout), a sandbox crash oracle
is the right arbiter, and Clearwing already is that system. Adopt it *there*, later, only when a class
needs it. It is the `execution_confirmed` rung, not the foundation.
6. **Reporting keeps the honesty disciplines that stand:** per-slice numbers with the real-world slice
deciding, the overfitting gap published, and the evidence rung on every finding — now meaning a model
6. Reporting keeps the honesty disciplines that stand: per-slice numbers with the real-world slice
deciding, the overfitting gap published, and the evidence rung on every finding, now meaning a model
verdict, not a sandbox result.
## 6. Build vs. adopt, revised
@@ -252,14 +252,14 @@ The earlier draft of this brief over-rotated to "adopt Clearwing wholesale," bec
foregrounds execution. The maintainer's correction re-centres it, and the honest split is cleaner than either
extreme:
- **The model layer is adopt-plus-contribute, not build.** The arbiter *engine* is Joern's CPG (Apache-2.0,
our languages, taint + dominance built in); the *contribution* is our security modelling on top of it — the
- The model layer is adopt-plus-contribute, not build. The arbiter *engine* is Joern's CPG (Apache-2.0,
our languages, taint + dominance built in); the *contribution* is our security modelling on top of it, the
taxonomy and the source/sink/sanitizer/guard specs for the web/logic classes, plus the LLM adjudication
checked against the CPG. The entailment measurement (2.2% on our flat IR) is what rules out building the
engine ourselves: that number is the distance between our AST projection and a real CPG, and closing it by
hand is reimplementing Joern.
- **The execution tier is Clearwing's to adopt**, later, for the memory-safety classes where it is strong and
we are weak — via the fork the maintainer already maintains. We do not reimplement its sandbox lifecycle,
- The execution tier is Clearwing's to adopt, later, for the memory-safety classes where it is strong and
we are weak, via the fork the maintainer already maintains. We do not reimplement its sandbox lifecycle,
container pooling, sanitizer images or PoC stability; those are exactly the parts this session proved we get
wrong.
@@ -268,24 +268,24 @@ arbiter we cannot, when a class demands it.
## Open questions for the requirements phase
- **How far up the ladder can a partial model reach without soundness?** `model_corroborated` needs only a
- How far up the ladder can a partial model reach without soundness? `model_corroborated` needs only a
consistency check; `model_entailed` needs a real interprocedural dataflow/guard-dominance result. What
fraction of our corpus can the model *entail* today, and where does it honestly fall back to `suspicion`?
This is the first measurement to run — the model analogue of the 96.6% candidate ceiling — and it can end
This is the first measurement to run (the model analogue of the 96.6% candidate ceiling), and it can end
or reshape the design before any LLM cost, exactly as the ceiling did.
- **What is the model's arbiter for an absence bug?** A missing guard has no dataflow path to confirm. The
- What is the model's arbiter for an absence bug? A missing guard has no dataflow path to confirm. The
obligation/sibling-dominance model is the candidate arbiter; does it distinguish the vulnerable-parent from
the fixed twin across the corpus on its own, deterministically?
- **Where does the LLM's residual judgement resist checking?** Some claims — "this value is attacker-
controlled by convention", "this guard is semantically insufficient" — the model cannot audit. How large is
- Where does the LLM's residual judgement resist checking? Some claims ("this value is attacker-
controlled by convention", "this guard is semantically insufficient") the model cannot audit. How large is
that residual, and is `suspicion` an acceptable home for it or does it need the execution tier sooner?
- **Which classes cross into execution, and when.** Memory-safety-in-C is the clear case for the deferred
- Which classes cross into execution, and when. Memory-safety-in-C is the clear case for the deferred
tier. Is anything in the web/logic taxonomy genuinely un-arbitrable by a model, forcing the differential
request oracle earlier than "later, if needed"?
- **What of the existing corpus survives** as the model's calibration set, given the twin/straddle/tier
- What of the existing corpus survives as the model's calibration set, given the twin/straddle/tier
defects the learning-harness reviews surfaced?
Sources: [Big Sleep / SQLite zero-day](https://thehackernews.com/2024/11/googles-ai-tool-big-sleep-finds-zero.html) ·
[AIxCC finals — Buttercup, 28 vulns / 20 CWEs / non-reasoning LLMs](https://blog.trailofbits.com/2025/08/09/trail-of-bits-buttercup-wins-2nd-place-in-aixcc-challenge/) ·
[AIxCC results — 18 real 0-days, 43/54 patched](https://semiengineering.com/aixcc-2025-what-it-means-for-device-security/) ·
[AIxCC finals, Buttercup, 28 vulns / 20 CWEs / non-reasoning LLMs](https://blog.trailofbits.com/2025/08/09/trail-of-bits-buttercup-wins-2nd-place-in-aixcc-challenge/) ·
[AIxCC results, 18 real 0-days, 43/54 patched](https://semiengineering.com/aixcc-2025-what-it-means-for-device-security/) ·
[Clearwing](https://github.com/Lazarus-AI/clearwing) · local: `/home/mc/Source/clearwing`, fork `norandom/deepwing`.
.kiro/specs/model-grounded-detection/design.md
+20 -20
View File
@@ -4,14 +4,14 @@
The detector stops being a noisy oracle and becomes a proposer that a deterministic program model audits. The
model is a **Code Property Graph** (Joern, Apache-2.0): one graph merging AST, CFG, PDG and call graph, over
which taint reachability and guard dominance produce verdicts on the evidence ladder — `suspicion`,
which taint reachability and guard dominance produce verdicts on the evidence ladder, `suspicion`,
`model_corroborated`, `model_entailed`, `execution_confirmed`. The LLM is asked one bounded question per
candidate; its answer is checked against the CPG, not averaged over runs.
Two disciplines shape every decision below. **Subtraction first:** the noise architecture and the outgrown
test surface are removed and audited before the CPG arbiter is adopted, because the session that produced this
spec logged eighteen defects building on an overgrown surface. **Adopt, don't build:** the CFG/PDG our flat IR
lacks comes from Joern, not from a hand-built engine — the 2.2% entailment ceiling is the measured distance
lacks comes from Joern, not from a hand-built engine. The 2.2% entailment ceiling is the measured distance
between our AST projection and a real CPG, and closing it by hand is reimplementing Joern.
### Goals
@@ -19,12 +19,12 @@ between our AST projection and a real CPG, and closing it by hand is reimplement
- Remove the noise architecture and reduce the test suite with it, gate-protected and reversible.
- A deterministic model verdict on the band the model covers; honest `suspicion` on the rest.
- The CPG engine, the LLM endpoint and any execution tier are optional; the core stays zero-dependency.
- The security modelling — taxonomy, source/sink/sanitizer/guard specs — is ours, expressed as CPG queries.
- The security modelling (taxonomy, source/sink/sanitizer/guard specs) is ours, expressed as CPG queries.
### Non-Goals
- Building a control-flow or dataflow engine (adopted, Req 4). Prompt optimisation, evolve loops, acceptance
rules (deleted, Req 8.5). The execution tier itself (deferred to Clearwing, Req 10). Soundness — the
rules (deleted, Req 8.5). The execution tier itself (deferred to Clearwing, Req 10). Soundness, the
north-star `model_entailed`-as-pure-arbiter is not v1.
## Boundary Commitments
@@ -32,18 +32,18 @@ between our AST projection and a real CPG, and closing it by hand is reimplement
### This Spec Owns
- The subtraction: which modules are removed, the dangling-import check, the module-audit manifest.
- `cpg/` — the Joern seam (`CpgBackend`, the subprocess/query implementation, the null backend).
- `model/` — `Verdict`/`Rung`, taint reachability, guard dominance, constant abstraction; the security specs
- `cpg/`, the Joern seam (`CpgBackend`, the subprocess/query implementation, the null backend).
- `model/`, `Verdict`/`Rung`, taint reachability, guard dominance, constant abstraction; the security specs
ported from the deleted modules' vocabularies.
- The proposer/judge orchestration where the LLM's claim is checked against the model.
- The corpus-as-calibration harness and its model-gap report.
### Out of Boundary
- The family **taxonomy** (`learning/families.py`) — retained unchanged, re-homed under `model/`.
- The corpus loader, per-slice split and gates — retained; the gates stay byte-identical.
- Redaction, the LLM endpoint client (`learning/endpoint.py`) — retained.
- Clearwing's sandbox lifecycle — never reimplemented; the execution tier adopts it.
- The family **taxonomy** (`learning/families.py`), retained unchanged, re-homed under `model/`.
- The corpus loader, per-slice split and gates, retained; the gates stay byte-identical.
- Redaction, the LLM endpoint client (`learning/endpoint.py`), retained.
- Clearwing's sandbox lifecycle, never reimplemented; the execution tier adopts it.
### Allowed Dependencies
@@ -63,7 +63,7 @@ between our AST projection and a real CPG, and closing it by hand is reimplement
### Existing Architecture Analysis
The `learning/` package (4,091 lines, 16 modules) implements the noise architecture and is **absent from the
gate path** — the gates import only from `pairs`, `benchmark`, `findings`, `mapping`, `preprocess`, `rank`,
gate path**. The gates import only from `pairs`, `benchmark`, `findings`, `mapping`, `preprocess`, `rank`,
`complexity.map`. So the subtraction is clean: removing `rounds`, `acceptance`, `proposer`, `journal`, the
noise machinery in `scoring`, and the mechanism loop cannot move a gate. What survives and re-homes: the
taxonomy (`families.py`), the candidate enumerator (`candidates.py`, the `suspicion`-band feeder), the LLM
@@ -140,7 +140,7 @@ src/openultrasast/model/
- `learning/candidates.py` → `model/candidates.py` (the suspicion-band enumerator, Req 8.4).
- `learning/endpoint.py` → `model/endpoint.py` (the judge client).
- `semantic/obligations/` retained; its operation/discharge facts feed `model/dominance.py`.
- `pairs.py`, the gates, redaction — unchanged.
- `pairs.py`, the gates, redaction, unchanged.
### The audit manifest (Req 3)
@@ -288,21 +288,21 @@ is dominated by no discharger and its siblings are, else `CORROBORATED`/`None`.
## Migration and Phases
1. **Subtract and audit (Req 1–3).** Remove the noise architecture; port the guard/sink vocabularies into
1. Subtract and audit (Req 1–3). Remove the noise architecture; port the guard/sink vocabularies into
`model/specs.py` before deleting their modules; run the import-graph check and the gate-baseline; reduce
the tests in the same change; commit the audit manifest. Gate-protected and reversible; no CPG yet.
2. **Joern spike on one family (Req 4, 6.1).** The `CpgBackend` seam and the injection `TaintSpec`, proven
2. Joern spike on one family (Req 4, 6.1). The `CpgBackend` seam and the injection `TaintSpec`, proven
end to end on the injection slice: build a CPG, run the taint query, produce entailed/corroborated
verdicts, assert determinism. This phase can stop the design if Joern cannot deliver a deterministic
verdict at acceptable cost — the analogue of the candidate-ceiling gate.
3. **The ladder and the judge (Req 5, 8).** `Verdict`/`Rung`, and `judge.py` checking the LLM's residual
verdict at acceptable cost, the analogue of the candidate-ceiling gate.
3. The ladder and the judge (Req 5, 8). `Verdict`/`Rung`, and `judge.py` checking the LLM's residual
answer against the verdict. No K-runs, nothing to average.
4. **Absence and the rest of the taxonomy (Req 6.2, 7).** Dominance queries from the obligation facts; the
4. Absence and the rest of the taxonomy (Req 6.2, 7). Dominance queries from the obligation facts; the
remaining `TaintSpec`s. Regenerate the entailment ceiling (Req 6.5) and watch it rise as depth improves.
5. **Corpus calibration and reporting (Req 9, 11).** The vulnerable-parent/fix differential, the model-gap
5. Corpus calibration and reporting (Req 9, 11). The vulnerable-parent/fix differential, the model-gap
report, per-slice + overfitting gap.
6. **Execution tier stub (Req 10).** The `execution_confirmed` rung and the Clearwing adoption seam, built
only when a class needs it — memory/C first, and not for web/logic v1.
6. Execution tier stub (Req 10). The `execution_confirmed` rung and the Clearwing adoption seam, built
only when a class needs it, memory/C first, and not for web/logic v1.
Phase 1 removes ~5,000 lines before a line of Joern is written. Phase 2 can end the design before the
integration cost is paid, exactly as the candidate ceiling could have in the prior spec.
.kiro/specs/model-grounded-detection/requirements.md
+21 -21
View File
@@ -6,12 +6,12 @@
This feature supersedes `learning-harness` and `constrained-detector`. It replaces the assumption that the LLM
is a noisy oracle to be averaged, floored and gated with an architecture in which a **deterministic program
model arbitrates the LLM's claims**. The arbiter is a Code Property Graph (CPG) — an intermediate language
merging AST, control-flow graph, program-dependence graph and call graph — over which taint reachability and
model arbitrates the LLM's claims**. The arbiter is a Code Property Graph (CPG), an intermediate language
merging AST, control-flow graph, program-dependence graph and call graph, over which taint reachability and
guard dominance produce verdicts the LLM's assertion is no longer needed for. The measurements that force this
design are committed: the candidate ceiling (96.6% of labeled functions have an enumerable site), and the
entailment ceiling (a deterministic model *decides* only 2.2% today on our flat AST-projection IR, ~24-35%
with deeper taint — the distance to a real CPG). Context: `brief.md`,
with deeper taint, the distance to a real CPG). Context: `brief.md`,
`benchmarks/measurements/2026-09-06-trajectory-divergence.json`, `2026-09-06-candidate-ceiling-all-slices.json`,
`2026-09-08-model-entailment-ceiling.json`.
@@ -22,15 +22,15 @@ overgrown, superseded surface is how this session logged eighteen defects.
## Boundary Context
- **In scope**: removing the noise architecture and the superseded arbiter; reducing the test suite to
- In scope: removing the noise architecture and the superseded arbiter; reducing the test suite to
surviving behavior; auditing the whole tree for orphaned modules; adopting a CPG engine (Joern) as the
model-layer substrate behind an optional extra; the evidence ladder as the verdict type; taint + dominance
verdicts; porting our taxonomy and source/sink/sanitizer/guard models to the CPG; the LLM as a proposer
checked against the model; the corpus as the model's calibration set; per-slice reporting.
- **Out of scope**: building a dataflow/CFG engine by hand (the CPG is adopted, §Req 4); the execution tier
- Out of scope: building a dataflow/CFG engine by hand (the CPG is adopted, §Req 4); the execution tier
itself (deferred to Clearwing, `execution_confirmed` rung only, Req 10); prompt optimisation, reflective
evolution, any optimiser (deleted, not deferred); fine-tuning weights.
- **North star, not v1**: pure `model_entailed`-as-arbiter. The entailment ceiling says the model decides a
- North star, not v1: pure `model_entailed`-as-arbiter. The entailment ceiling says the model decides a
minority today; v1 is the *layered* arbiter (entail where it can, corroborate the taint band, honest
suspicion for the rest). This is stated so the requirements are not read as promising a soundness the
measurement denies.
@@ -45,12 +45,12 @@ so that nothing is maintained, tested, or built upon that this feature's measure
#### Acceptance Criteria
1. The system shall remove the round loop, the per-family noise floor, the regression budget, the acceptance
rule, the sign test, the evolve journal and the meta-agent proposer — concretely `learning/rounds.py`,
rule, the sign test, the evolve journal and the meta-agent proposer, concretely `learning/rounds.py`,
`learning/acceptance.py`, `learning/proposer.py`, `learning/journal.py`, and the K-run / floor / sign-test
machinery in `learning/scoring.py`.
2. The system shall remove the mechanism-optimisation loop — the mechanism store and lever in
2. The system shall remove the mechanism-optimisation loop (the mechanism store and lever in
`improve/evolve.py`, and the shape-search machinery in `semantic/mechanisms.py` and the optimisation
surface of `semantic/variants.py` — while preserving the *security models* those modules carried (the
surface of `semantic/variants.py`), while preserving the *security models* those modules carried (the
guard vocabulary and source/sink classification) for porting under Req 7.
2b. The system shall remove the per-family evolvable detector configuration and its canary/verifier surface
(`learning/detectors.py` config-evolution parts, `learning/canaries.py`, `learning/verifiers.py`,
@@ -70,7 +70,7 @@ ratchet that pins deleted mechanisms and the suite is once again a description o
1. When a mechanism is removed under Req 1, the tests that existed only to pin its behavior shall be removed
with it, in the same change, not left skipped or xfailed.
2. Every surviving test shall map to a surviving requirement — of this spec or of a still-shipping feature —
2. Every surviving test shall map to a surviving requirement (of this spec or of a still-shipping feature)
and a test that maps to nothing shall be removed.
3. The reduced suite shall be smaller than the current 832 tests, and the change shall report the count before
and after with the requirement each removed test cluster served.
@@ -78,7 +78,7 @@ ratchet that pins deleted mechanisms and the suite is once again a description o
determinism test (Req 6.4), the per-slice honesty tests, and the two ceiling measurements' regeneration
tests.
5. A test that asserts a shape "does nothing" or greps `--help`, or that passes with its capability absent,
shall be removed rather than kept — the session logged several; none survive.
shall be removed rather than kept. The session logged several; none survive.
### Requirement 3: Audit the whole tree for outgrowth
@@ -89,7 +89,7 @@ found and removed rather than carried indefinitely.
1. The system shall classify every source module as one of: load-bearing for the model-grounded architecture;
a retained standalone capability with its own users; or orphaned.
2. If a module is orphaned — no importer, no CLI entry, no gate dependency, no surviving requirement — then the
2. If a module is orphaned (no importer, no CLI entry, no gate dependency, no surviving requirement), then the
system shall remove it and record the classification.
3. The system shall name, for the audit, the older-phase subsystems whose status is not obvious
(`fusion.py`, `mcp.py`, `skills.py`, `harness_ext.py`, `hunter_harness.py`) and give each an explicit
@@ -122,7 +122,7 @@ model result is never confused with an LLM suspicion.
#### Acceptance Criteria
1. The system shall assign every finding one rung: `suspicion` (a site was enumerated / the LLM flagged it),
`model_corroborated` (the LLM's claim is consistent with the CPG — the source it names reaches the sink
`model_corroborated` (the LLM's claim is consistent with the CPG, the source it names reaches the sink
through resolved dataflow, or a guard its siblings carry is absent), `model_entailed` (the CPG alone
establishes it, independent of trusting the LLM), or `execution_confirmed` (a sandbox reproduced it).
2. The system shall establish `model_corroborated` and `model_entailed` from the CPG deterministically, with
@@ -145,21 +145,21 @@ project measured is removed on the band the model covers.
4. When the system runs the model layer twice over the same CPG, the sequence of verdicts shall be identical,
and a test shall assert it.
5. The system shall report, per family and per slice, the fraction of the corpus it can `model_entail` and
`model_corroborate` — the regeneration of the committed entailment ceiling — and shall name where it falls
`model_corroborate` (the regeneration of the committed entailment ceiling), and shall name where it falls
back to `suspicion`.
### Requirement 7: Port the taxonomy and security models onto the CPG
**Objective:** As a maintainer, I want our security knowledge — the classes we target and how their bugs look
— expressed as queries over the adopted CPG, so that the contribution is the modelling, not the engine.
**Objective:** As a maintainer, I want our security knowledge (the classes we target and how their bugs look)
expressed as queries over the adopted CPG, so that the contribution is the modelling, not the engine.
#### Acceptance Criteria
1. The system shall retain the closed family taxonomy as the routing key, unchanged in meaning.
2. The system shall express the source, sink and sanitizer models for the web/logic CWEs as CPG taint
specifications, seeded from the models the deleted modules carried.
3. The system shall express the obligation / absence model — an operation on a protected resource, and the
guards that discharge it — as CPG dominance queries, as the arbiter for the classes that never crash.
3. The system shall express the obligation / absence model (an operation on a protected resource, and the
guards that discharge it) as CPG dominance queries, as the arbiter for the classes that never crash.
4. The system shall keep the verifier boundary: the models and gold labels stay outside anything an LLM or a
future optimiser may write.
@@ -177,7 +177,7 @@ with its claim checked against the model, so that there is no oracle to average.
`model_corroborated`.
4. When the model can neither confirm nor contradict, the system shall report the claim at `suspicion`, and
this band is where a future execution tier earns its place.
5. The system shall contain no K-run averaging, no noise floor, no acceptance rule, no evolve loop — the model
5. The system shall contain no K-run averaging, no noise floor, no acceptance rule, no evolve loop. The model
check replaces all of them.
### Requirement 9: The corpus is the model's calibration set
@@ -188,7 +188,7 @@ vote, so that a gap is a gap in the model, named and fixable.
#### Acceptance Criteria
1. The system shall use the vulnerable-parent / fix-commit pair as ground truth for whether the model can
distinguish the two — the fix is what the model should be able to see.
distinguish the two. The fix is what the model should be able to see.
2. If the model cannot distinguish a pair's two sides, then the system shall record it as a named model gap
with the reason, not as a detector miss to be averaged away.
3. The system shall keep the per-slice split with the real-world slices deciding, and shall not present the
@@ -204,7 +204,7 @@ arbitrate, without building or requiring it at v1.
#### Acceptance Criteria
1. The system shall treat `execution_confirmed` as an optional rung reached only for classes where the model
layer cannot decide — memory-safety in C first.
layer cannot decide, memory-safety in C first.
2. When execution confirmation is used, the system shall adopt it from Clearwing (the existing fork), not
reimplement the sandbox lifecycle, container pooling, sanitizer images or PoC stability.
3. The system shall not require the execution tier for any web/logic class at v1.