mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
docs(contributor-scan): brief -- the model layer is complete and unshipped
Asked what the spec state was and whether anything was forgotten. It was, and it is the largest thing outstanding: model-grounded-detection is 16/16 tasks complete and its arbiter measures 86.7% pair-correct at 6.7% leak, but the model layer has ZERO production callers. `ousast scan` still runs preprocess -> map -> overlay -> obligations -> regress -> report and touches none of it. Every number quoted today came from a measurement harness in a scratch directory. That is a scope gap in the predecessor spec rather than a slip in executing it: there was never a task saying "make this reachable", and its Req 11.2 (gates stay byte-identical) is satisfied PRECISELY BECAUSE nothing is wired. The spec is complete and the feature is unshipped, both at once. A new spec rather than an amendment, because the predecessor is coherent and closed -- it built an arbiter and measured it -- while this is a different question: making it something a developer runs on a repository they are about to contribute to. What it covers, in the order it bites: scan integration; whole-repo scanning (entry-point region selection, ranking across thousands of candidates, a budget policy -- the enumerator caps at 8 per region, meaningless at repo scale -- and a performance envelope never measured); running against a known-vulnerable checkout, since the 176 real-world CVE pairs are single-function excerpts that exercise none of it; the deferred memory decision, with 173 of those pairs idle behind a stub tier; and retiring evidence_level. It also records what was built outside any spec: the pipeline itself, the ops work (one image, compose, shell wrappers, pyinfra -- with the Docker image never actually built), and the corpus additions. And it carries the known limits forward rather than losing them: the flat permissive_values list that cannot tell debug=True from httponly=True, interprocedural resolution as the most repeated miss cause, and the five bugs that shared one shape -- a taint-specific assumption applied to every family, each caught by a measurement rather than a check. Not approved. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
adbb824eac
commit
41266ce472
@@ -0,0 +1,68 @@
|
||||
# Brief: contributor-scan — make the model layer a tool someone runs
|
||||
|
||||
**Status:** brief only, unapproved. Successor to `model-grounded-detection`, which is complete (16/16 tasks)
|
||||
and is not reopened by this.
|
||||
|
||||
## The gap this exists to close
|
||||
|
||||
`model-grounded-detection` built and measured an arbiter. It works: 41.3% of the corpus reaches
|
||||
`model_entailed` against the flat IR's 2.2%, and the wired pipeline scores 86.7% pair-correct at 6.7% leak
|
||||
against the LLM hunter's 76.7% / 12.3%, at a fifth of the model calls and ~100× cheaper.
|
||||
|
||||
**And none of it is reachable by a user.** `ousast scan` still runs
|
||||
`preprocess → map → overlay → obligations → regress → report` and touches the model layer nowhere;
|
||||
`model/pipeline`, `judge`, `config_value`, `report`, `calibrate` and `execution` have zero production callers
|
||||
outside `model/`. Every number quoted above came from a measurement harness in a scratch directory.
|
||||
|
||||
That is a scope gap in the predecessor spec, not an oversight in its execution: it had no task saying *make
|
||||
this reachable*, and its Req 11.2 (the gates stay byte-identical) is satisfied precisely **because** nothing
|
||||
is wired. The spec is complete and the feature is unshipped, both at once.
|
||||
|
||||
## What the goal actually is
|
||||
|
||||
> "a real world tool that helps developers when they want to contribute"
|
||||
|
||||
That reframes the work. A contributor does not have a labelled function and a known weakness class — they
|
||||
have a repository they are about to touch and a question about what is risky in it. Everything measured so
|
||||
far is function-level pairs: a labelled function, a declared family, a known target. None of it exercises the
|
||||
questions a whole repository asks.
|
||||
|
||||
## What is missing, in the order it bites
|
||||
|
||||
1. **Scan integration.** Wire `model/pipeline` into `ousast scan` so a finding with a rung reaches a report.
|
||||
Without this nothing else matters.
|
||||
2. **Whole-repo scanning.** Region selection from entry points instead of a labelled function; ranking across
|
||||
thousands of candidates rather than one; a budget policy (the enumerator caps at 8 candidates per region,
|
||||
which is meaningless at repo scale); and the performance envelope — a CPG build on a 100k-line tree, not a
|
||||
40-line excerpt. **This has never been run.**
|
||||
3. **A known-vulnerable repository as the target.** The 176 real-world CVE pairs already in the corpus are
|
||||
single-function excerpts and do not exercise any of the above. Running against a real checkout — curl,
|
||||
libpng, a WordPress plugin — is the only way to find what breaks at scale.
|
||||
4. **The memory decision.** 173 of those 176 pairs are `memory` family, routed to a deferred execution tier
|
||||
that is a stub. Either adopt Clearwing for them or scope C memory-safety out explicitly. Right now it is
|
||||
neither, and a sixth of the corpus sits idle.
|
||||
5. **Retire `evidence_level`.** It duplicates `rung` in intent; noted as a follow-up in group 3 and never
|
||||
tracked.
|
||||
|
||||
## Already built and unrecorded
|
||||
|
||||
These were delivered during `model-grounded-detection` without being in any spec. They belong here so the
|
||||
record is not scattered:
|
||||
|
||||
- **The pipeline itself** (`model/pipeline.py`) — Req 8 existed, task 3.2 built `judge()`, and nothing ever
|
||||
said to connect an enumerator to it. Built, tested, measured, unspecified.
|
||||
- **Ops** — a single image carrying tool and engine, a compose file with read-only target and tmpfs scratch,
|
||||
bash/zsh/PowerShell wrappers that make the container invisible, and a pyinfra deploy for host installs.
|
||||
The Docker path is written and syntax-validated but **the image has never been built**.
|
||||
- **Corpus** — the `owasp` slice (40 validated xss pairs; `output_encoding` went from 1 pair to 41) and
|
||||
authored sink/sanitizer facts for Java and JavaScript.
|
||||
|
||||
## Known limits to carry forward
|
||||
|
||||
- `permissive_values` is a flat literal list with no keyword association, so it holds both `True` and `False`.
|
||||
`debug=True` is dangerous where `httponly=True` is safe and the model cannot tell them apart.
|
||||
- Interprocedural resolution is the most repeated miss cause across every experiment: OWASP's helper classes,
|
||||
`angular-http-server`'s `resolveUrl`. Joern supports it; our queries may not be configured for it.
|
||||
- Five bugs in `model-grounded-detection` shared one shape — a taint-specific assumption applied to every
|
||||
family, each found by a measurement rather than a check. Any new family-specific path should be audited for
|
||||
siblings before it is measured, not after.
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"feature_name": "contributor-scan",
|
||||
"created_at": "2026-09-08T20:00:00Z",
|
||||
"updated_at": "2026-09-08T20:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "brief",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"design": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false,
|
||||
"depends_on": [
|
||||
"model-grounded-detection"
|
||||
],
|
||||
"approval_note": "2026-09-08 brief drafted after the maintainer asked what the spec state was and whether anything was forgotten. It was: model-grounded-detection is 16/16 complete and its arbiter is measured at 86.7% pair-correct / 6.7% leak, but the model layer has ZERO production callers -- ousast scan does not use it, so the feature is complete and unshipped at once. That is a scope gap in the predecessor (it had no task saying 'make this reachable', and its byte-identical-gates requirement is satisfied precisely because nothing is wired). This spec covers scan integration, whole-repo scanning against known vulnerable repositories, the deferred memory-family decision, and records the pipeline, ops and corpus work delivered outside any spec. Not approved; requirements not generated."
|
||||
}
|
||||
Reference in New Issue
Block a user