mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
docs(model-grounded-detection): record what 16/16 does and does not mean
Asked whether the prior spec is finished. Audited rather than counted, and the answer is: all 16 tasks are complete as written, and not all requirements are satisfied in the product. Four are built but unreachable, so the requirement is not met where it matters: model/report.py implements the per-slice and overfitting-gap discipline (Req 11.1) and has no production caller; resolve_execution_backend returns NullExecutionBackend on BOTH branches so the execution_confirmed rung (Req 10.1) can never be reached; harvest_recipe (Req 9.4) is a seam with nothing plugged into it; and the pipeline that measures 90.0% pair-correct is called only by a measurement harness, never by ousast scan. That is a spec defect rather than an execution one. No task ever said "make this reachable", and Req 11.2 -- the gates stay byte-identical -- is satisfied PRECISELY BECAUSE nothing is wired. I read that as reassurance for a whole session when it was evidence. The spec is not reopened; it remains the complete record of building and measuring the arbiter. contributor-scan takes all four: its Req 1 closes 8.x and 11.1 in the product, Req 7.1 forces the memory decision 10.1 defers, and Req 9 covers the harvest 9.4 left as a seam. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
4e4681f162
commit
64a27c4e30
@@ -555,3 +555,37 @@ same error class as reading an absent engine as "no vulnerability".
|
||||
|
||||
With no adopted tool present — the normal case — the rung is simply never reached and `clearwing_unavailable`
|
||||
is recorded.
|
||||
|
||||
## Completion status — audited 2026-09-09
|
||||
|
||||
**All 16 tasks are complete as written. Not all requirements are satisfied in the product.** The distinction
|
||||
matters and was not visible from the checkboxes: several tasks delivered a library, verified it, measured it,
|
||||
and stopped — because no task said to connect it. Recorded here so a later reader is not misled by 16/16.
|
||||
|
||||
**Satisfied and reachable:**
|
||||
|
||||
- Req 1–3 (subtraction, test reduction, module audit) — done and durable; the audit manifest is committed and
|
||||
a test holds the tree to it.
|
||||
- Req 4–7 (the CPG seam, the ladder, deterministic taint/dominance/constant-abstraction verdicts, the ported
|
||||
security models) — built, tested, and measured at 41.3% entailed against the flat IR's 2.2%.
|
||||
- Req 5.3 (rung in every output format) — markdown, SARIF and JSON all carry it.
|
||||
- Req 8 (the LLM proposes, the model disposes) — works, and measures 90.0% pair-correct / 6.7% leak against
|
||||
the hunter's 76.7% / 12.3%. But only through a measurement harness (see below).
|
||||
|
||||
**Built but not reachable — the requirement is not met in the product:**
|
||||
|
||||
| Requirement | State |
|
||||
|---|---|
|
||||
| 11.1 per-slice reporting, overfitting gap, rung on every finding | `model/report.py` exists and is tested; **no production caller**. The discipline is implemented and applied to nothing a user sees. |
|
||||
| 10.1 `execution_confirmed` reachable for the classes the model cannot decide | `resolve_execution_backend` returns `NullExecutionBackend` on **both** branches. The rung can never be reached. Honest as the stub task 6.1 specified, but not operational. |
|
||||
| 9.4 CVE-history harvest at repository scale | `harvest_recipe` exists and is tested; no caller. A seam with nothing plugged into it. |
|
||||
| 8.1–8.5 in production | The pipeline is real and measured, but `ousast scan` calls none of it. |
|
||||
|
||||
**Why this happened, and it is a spec defect rather than an execution one.** There was never a task saying
|
||||
*make this reachable*, and Req 11.2 — the gates stay byte-identical — is satisfied **precisely because
|
||||
nothing is wired**. That was read as reassurance for a whole session when it was evidence.
|
||||
|
||||
**Where it is being closed:** `contributor-scan` takes all four. Req 1 there covers scan integration
|
||||
(closing 8.x and 11.1 in the product), Req 7.1 forces the memory decision that 10.1 defers, and Req 9 covers
|
||||
the corpus harvest 9.4 left as a seam. This spec is **not reopened**; it stays complete as the record of
|
||||
building and measuring the arbiter.
|
||||
|
||||
Reference in New Issue
Block a user