mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 20:49:28 +00:00
docs: plain-English pass over the active specs
Ran the humanizer rules over 79 prose files in the active specs (brief, design, requirements, research and narrative docs). Removed em dashes, significance inflation, forced rule-of-three, bold-label list items and filler. Skipped tasks.md checklists and the retired specs (harnessx-*, openrouter-sast-harness, unseen-repo-evaluation). Every number, path, command, hash and identifier was preserved: checked each file against HEAD for drift in numbers, inline code and code fences, zero drift across all 79 files. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
b08f372022
commit
39a1873e5b
@@ -1,4 +1,4 @@
|
||||
# Brief: contributor-scan — make the model layer a tool someone runs
|
||||
# Brief: contributor-scan, make the model layer a tool someone runs
|
||||
|
||||
**Status:** brief only, unapproved. Successor to `model-grounded-detection`, which is complete (16/16 tasks)
|
||||
and is not reopened by this.
|
||||
@@ -22,7 +22,7 @@ is wired. The spec is complete and the feature is unshipped, both at once.
|
||||
|
||||
> "a real world tool that helps developers when they want to contribute"
|
||||
|
||||
That reframes the work. A contributor does not have a labelled function and a known weakness class — they
|
||||
That reframes the work. A contributor does not have a labelled function and a known weakness class; they
|
||||
have a repository they are about to touch and a question about what is risky in it. Everything measured so
|
||||
far is function-level pairs: a labelled function, a declared family, a known target. None of it exercises the
|
||||
questions a whole repository asks.
|
||||
@@ -33,11 +33,11 @@ questions a whole repository asks.
|
||||
Without this nothing else matters.
|
||||
2. **Whole-repo scanning.** Region selection from entry points instead of a labelled function; ranking across
|
||||
thousands of candidates rather than one; a budget policy (the enumerator caps at 8 candidates per region,
|
||||
which is meaningless at repo scale); and the performance envelope — a CPG build on a 100k-line tree, not a
|
||||
which is meaningless at repo scale); and the performance envelope: a CPG build on a 100k-line tree, not a
|
||||
40-line excerpt. **This has never been run.**
|
||||
3. **A known-vulnerable repository as the target.** The 176 real-world CVE pairs already in the corpus are
|
||||
single-function excerpts and do not exercise any of the above. Running against a real checkout — curl,
|
||||
libpng, a WordPress plugin — is the only way to find what breaks at scale.
|
||||
single-function excerpts and do not exercise any of the above. Running against a real checkout (curl,
|
||||
libpng, a WordPress plugin) is the only way to find what breaks at scale.
|
||||
4. **The memory decision.** 173 of those 176 pairs are `memory` family, routed to a deferred execution tier
|
||||
that is a stub. Either adopt Clearwing for them or scope C memory-safety out explicitly. Right now it is
|
||||
neither, and a sixth of the corpus sits idle.
|
||||
@@ -49,12 +49,12 @@ questions a whole repository asks.
|
||||
These were delivered during `model-grounded-detection` without being in any spec. They belong here so the
|
||||
record is not scattered:
|
||||
|
||||
- **The pipeline itself** (`model/pipeline.py`) — Req 8 existed, task 3.2 built `judge()`, and nothing ever
|
||||
- The pipeline itself (`model/pipeline.py`): Req 8 existed, task 3.2 built `judge()`, and nothing ever
|
||||
said to connect an enumerator to it. Built, tested, measured, unspecified.
|
||||
- **Ops** — a single image carrying tool and engine, a compose file with read-only target and tmpfs scratch,
|
||||
- Ops: a single image carrying tool and engine, a compose file with read-only target and tmpfs scratch,
|
||||
bash/zsh/PowerShell wrappers that make the container invisible, and a pyinfra deploy for host installs.
|
||||
The Docker path is written and syntax-validated but **the image has never been built**.
|
||||
- **Corpus** — the `owasp` slice (40 validated xss pairs; `output_encoding` went from 1 pair to 41) and
|
||||
- Corpus: the `owasp` slice (40 validated xss pairs; `output_encoding` went from 1 pair to 41) and
|
||||
authored sink/sanitizer facts for Java and JavaScript.
|
||||
|
||||
## Known limits to carry forward
|
||||
@@ -63,7 +63,7 @@ record is not scattered:
|
||||
`debug=True` is dangerous where `httponly=True` is safe and the model cannot tell them apart.
|
||||
- Interprocedural resolution is the most repeated miss cause across every experiment: OWASP's helper classes,
|
||||
`angular-http-server`'s `resolveUrl`. Joern supports it; our queries may not be configured for it.
|
||||
- Five bugs in `model-grounded-detection` shared one shape — a taint-specific assumption applied to every
|
||||
- Five bugs in `model-grounded-detection` shared one shape: a taint-specific assumption applied to every
|
||||
family, each found by a measurement rather than a check. Any new family-specific path should be audited for
|
||||
siblings before it is measured, not after.
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Task 2.19 — measured input retention and census completeness
|
||||
# Task 2.19, measured input retention and census completeness
|
||||
|
||||
The maintainer accepted the proposed contributor-scan follow-up after the pre-push
|
||||
NO-GO report. The Kiro root-cause-first workflow identified a concrete shared-boundary
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Design Document: contributor-scan
|
||||
|
||||
## Integration amendment — 2026-09-12 (draft, unapproved)
|
||||
## Integration amendment, 2026-09-12 (draft, unapproved)
|
||||
|
||||
The maintainer has clarified the product as an actionable pre-push safety net across supported
|
||||
ecosystems. The proposed successor `../pre-push-safety-net/design.md` consumes this driver/backend
|
||||
@@ -9,7 +9,7 @@ of both targets and graphs. Existing approvals below apply to the prior design,
|
||||
|
||||
Proposed shared seams needing requirements/design review before implementation:
|
||||
|
||||
- Accept an absolute deadline across graph build, overlays, retries and queries, not only region/call caps.
|
||||
- Accept an absolute deadline across graph build, overlays, retries and queries as well as region/call caps.
|
||||
- Accept an optional validated reusable graph lease while preserving ordinary fresh-build scans;
|
||||
backend remains authoritative for artifact identity, census, compatibility and cleanup.
|
||||
- Return exact selected/deferred question IDs and context boundaries from ranker decisions;
|
||||
@@ -45,13 +45,13 @@ for someone who does not have Joern.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- New arbiters or families — the three forms exist and are measured.
|
||||
- New arbiters or families; the three forms exist and are measured.
|
||||
- Any optimiser, prompt search, or K-run averaging. The predecessor deleted that machinery; this does not
|
||||
reintroduce it by another name.
|
||||
- Building a sandbox. Req 10 of the predecessor stands: adopt, never build.
|
||||
- Changing the gates, the taxonomy, or the fact-table schema.
|
||||
|
||||
## The fact tables are an ontology, and it is general-purpose — 2026-09-21
|
||||
## The fact tables are an ontology, and it is general-purpose, 2026-09-21
|
||||
|
||||
The `[[source]]`, `[[sink]]`, `[[sanitizer]]`, `[[operation]]`, `[[discharger]]` and `[[layout]]` rows
|
||||
are not a pattern list. They are an ontology of program elements and their security relations: what
|
||||
@@ -87,18 +87,18 @@ repository is in front of the maintainer, which is what task 7.1 exists to make
|
||||
|
||||
### This Spec Owns
|
||||
|
||||
- `model/regions.py` — entry points and file fallbacks to regions, with the family routing for each.
|
||||
- `model/scan.py` — the repository-level driver: rank, budget, dispatch to `model/pipeline`, collect.
|
||||
- `model/regions.py`: entry points and file fallbacks to regions, with the family routing for each.
|
||||
- `model/scan.py`: the repository-level driver: rank, budget, dispatch to `model/pipeline`, collect.
|
||||
- The `Stage.MAP` wiring in `cli.py` and the scan's model-layer payload and report section.
|
||||
- `benchmarks/repos/` — pinned known-vulnerable checkouts and their baselines.
|
||||
- `benchmarks/repos/`: pinned known-vulnerable checkouts and their baselines.
|
||||
- The decisions in Req 7 and their record.
|
||||
|
||||
### Out of Boundary
|
||||
|
||||
- `model/pipeline`, `taint`, `dominance`, `config_value`, `ladder`, `specs`, `candidates` — built, measured,
|
||||
- `model/pipeline`, `taint`, `dominance`, `config_value`, `ladder`, `specs`, `candidates`: built, measured,
|
||||
used unchanged. If this feature needs to change one, that is a signal the change belongs in its own commit
|
||||
with its own measurement.
|
||||
- `mapping.analyze_entry_points`, the overlay, the obligations checker, the gates — untouched.
|
||||
- `mapping.analyze_entry_points`, the overlay, the obligations checker, the gates: untouched.
|
||||
|
||||
### Allowed Dependencies
|
||||
|
||||
@@ -117,8 +117,8 @@ repository is in front of the maintainer, which is what task 7.1 exists to make
|
||||
### Existing Architecture Analysis
|
||||
|
||||
`ousast scan` runs `policy_load → ruleset_load → preprocess → entry_point_mapping → rank → calibrate →
|
||||
STATIC → MAP → REGRESS → report`. Inside MAP it already builds a complexity map, runs the overlay, and — the
|
||||
closest analogue to what we are adding — runs `_check_obligations(root, targets, entries, facts, settings,
|
||||
STATIC → MAP → REGRESS → report`. Inside MAP it already builds a complexity map, runs the overlay, and (the
|
||||
closest analogue to what we are adding) runs `_check_obligations(root, targets, entries, facts, settings,
|
||||
runtime, hunter_model)`, which takes entry points, produces findings, and merges them by re-ranking.
|
||||
|
||||
The model layer slots in exactly there, with the same shape and the same failure discipline. `entry_points`
|
||||
@@ -167,10 +167,10 @@ benchmarks/repos/
|
||||
|
||||
### Changed
|
||||
|
||||
- `cli.py` — a `model` stage inside MAP, mirroring `_check_obligations`; its payload into the manifest and a
|
||||
- `cli.py`: a `model` stage inside MAP, mirroring `_check_obligations`; its payload into the manifest and a
|
||||
report section.
|
||||
- `model/report.py` — a per-rung summary block for a scan (it currently reports per-slice for corpus runs).
|
||||
- `config.py` — a `ModelConfig`-style block: enabled, budget, max regions. Zero-dep, defaults conservative.
|
||||
- `model/report.py`: a per-rung summary block for a scan (it currently reports per-slice for corpus runs).
|
||||
- `config.py`: a `ModelConfig`-style block: enabled, budget, max regions. Zero-dep, defaults conservative.
|
||||
|
||||
### Unchanged and load-bearing
|
||||
|
||||
@@ -194,8 +194,8 @@ class ScanRegion:
|
||||
def regions_for(entries, targets, *, taxonomy, language_of) -> tuple[ScanRegion, ...]
|
||||
```
|
||||
|
||||
Ranking uses what `EntryPointRecord` already carries — an externally-reachable entry point at a trust
|
||||
boundary outranks an internal one — rather than inventing a new signal.
|
||||
Ranking uses what `EntryPointRecord` already carries (an externally-reachable entry point at a trust
|
||||
boundary outranks an internal one) rather than inventing a new signal.
|
||||
|
||||
### model/scan.py
|
||||
|
||||
@@ -264,10 +264,10 @@ each; everything past exhaustion counted into `regions_unjudged`.
|
||||
1. **Wire it (Req 1, 2).** Regions, the MAP stage, findings with rungs reaching a report. Gate-protected.
|
||||
Nothing else matters until a user can see a rung.
|
||||
2. **Scale it (Req 3, 4).** Budget and ranking, then the first run against a pinned known-vulnerable
|
||||
checkout — the measurement that has never been taken and can reshape the rest.
|
||||
checkout, the measurement that has never been taken and can reshape the rest.
|
||||
3. **Ship it (Req 5, 6).** Contributor output; build and verify the image that is currently only defined.
|
||||
4. **Close the deferred (Req 7).** The memory decision; retire `evidence_level`.
|
||||
5. **Grow the corpus (Req 9).** PHP/WordPress first — the families the model arbitrates best. C CVEs only
|
||||
5. **Grow the corpus (Req 9).** PHP/WordPress first: the families the model arbitrates best. C CVEs only
|
||||
after 7.1.
|
||||
6. **Regression baselines (Req 10).**
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Release remediation research — 2026-09-14
|
||||
# Release remediation research, 2026-09-14
|
||||
|
||||
Status: investigation under the maintainer's instruction to continue towards release.
|
||||
This record does not approve rollout, replace the existing design, or mark a production
|
||||
|
||||
@@ -12,7 +12,7 @@ hunter's 76.7% / 12.3%, at a fifth of the model calls. All 16 of its tasks are c
|
||||
from a measurement harness, not from the tool. This feature closes that gap and answers the question the
|
||||
maintainer actually posed: *a real world tool that helps developers when they want to contribute.*
|
||||
|
||||
That framing changes the work. A contributor has no labelled function and no declared weakness class — they
|
||||
That framing changes the work. A contributor has no labelled function and no declared weakness class; they
|
||||
have a repository they are about to touch. Everything measured so far is function-level pairs, which
|
||||
exercise none of the questions a whole repository asks: which regions to look at, how to rank thousands of
|
||||
candidates, what to spend, and whether a CPG build on a 100k-line tree finishes at all.
|
||||
@@ -22,13 +22,13 @@ Context: `brief.md`, `.kiro/specs/model-grounded-detection/`, and the committed
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: wiring `model/pipeline` into `ousast scan`; region selection from entry points; ranking and
|
||||
- In scope: wiring `model/pipeline` into `ousast scan`; region selection from entry points; ranking and
|
||||
budget at repository scale; the performance envelope; running against known-vulnerable checkouts;
|
||||
contributor-facing output; the packaging already built (image, compose, shell wrappers, pyinfra); the
|
||||
deferred memory-family decision; retiring `evidence_level`.
|
||||
- **Out of scope**: new arbiters or families (the three forms exist); prompt optimisation of any kind;
|
||||
building a sandbox (Req 10 of the predecessor stands — adopt, never build); changing the gates.
|
||||
- **Carried forward, not re-litigated**: the evidence ladder, the taxonomy, the fact tables, and the finding
|
||||
- Out of scope: new arbiters or families (the three forms exist); prompt optimisation of any kind;
|
||||
building a sandbox (Req 10 of the predecessor stands: adopt, never build); changing the gates.
|
||||
- Carried forward, not re-litigated: the evidence ladder, the taxonomy, the fact tables, and the finding
|
||||
that the model supplies precision while the LLM supplies recall.
|
||||
|
||||
## Requirements
|
||||
@@ -45,7 +45,7 @@ built is something I can run rather than something a harness demonstrates.
|
||||
2. The system shall report model findings alongside the existing pattern and overlay findings, without
|
||||
removing either.
|
||||
3. When no CPG engine is available, the scan shall behave exactly as it does today and record
|
||||
`cpg_unavailable` — the model layer is additive, never a precondition.
|
||||
`cpg_unavailable`; the model layer is additive, never a precondition.
|
||||
4. The detection, map and local pair gates shall remain byte-identical to the committed baseline.
|
||||
5. The system shall expose the model layer's per-rung counts in the scan's own output, so a user can see how
|
||||
much of the result the model arbitrated rather than inferring it.
|
||||
@@ -109,7 +109,7 @@ in the corpus exercises it.
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Every finding shall carry its rung, its witness, and the file and line it concerns.
|
||||
2. The system shall state plainly what a `suspicion` means — that the model could not confirm it — so a
|
||||
2. The system shall state plainly what a `suspicion` means (that the model could not confirm it) so a
|
||||
reader does not treat it as established.
|
||||
3. The system shall write findings to the writable output location, never into the repository under analysis.
|
||||
4. The system shall summarise a scan in a form readable without opening a JSON file.
|
||||
@@ -121,7 +121,7 @@ in the corpus exercises it.
|
||||
throughout, and no criterion here forbade it.)*
|
||||
6. Where a family's abstraction cannot decide a weakness class, the system shall state that limit in the
|
||||
report, and no scan shall present an absence of findings as evidence of absence. *(Added 2026-09-09.
|
||||
`c/memory` scored zero true positives across 89k lines of C for a structural reason — it models string
|
||||
`c/memory` scored zero true positives across 89k lines of C for a structural reason: it models string
|
||||
functions and libpng overflows through arithmetic. A silent scan reading as a clean bill of health is
|
||||
worse than noise because it is quiet, and it is the one failure mode the rung ladder does not guard.)*
|
||||
|
||||
@@ -148,7 +148,7 @@ silently caps what the corpus can tell us.
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The system shall either reach the `memory` family through an adopted execution tier, or record explicitly
|
||||
that C memory-safety is out of scope — 173 of the 176 real-world CVE pairs currently sit idle behind a
|
||||
that C memory-safety is out of scope; 173 of the 176 real-world CVE pairs currently sit idle behind a
|
||||
stub.
|
||||
2. The system shall retire `evidence_level` in favour of `rung`, or record why both are kept.
|
||||
3. Each decision shall be recorded with its reason where a later reader will find it.
|
||||
@@ -167,7 +167,7 @@ this one, because each exists due to a specific past error.
|
||||
4. The core install shall remain zero-dependency, with the CPG engine, the LLM endpoint and any execution
|
||||
tier optional and degrading to a recorded reason.
|
||||
5. Any new family-specific code path shall be checked for sibling paths that need the same change, before it
|
||||
is measured — five bugs in the predecessor shared exactly that shape.
|
||||
is measured; five bugs in the predecessor shared exactly that shape.
|
||||
|
||||
### Requirement 9: Grow the corpus where the engine can use it
|
||||
|
||||
@@ -177,14 +177,14 @@ what developers actually contribute to rather than against 30 pairs of one slice
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The system shall add a **PHP/WordPress** slice covering plugin and core weaknesses. WordPress plugin CVEs
|
||||
are overwhelmingly injection, XSS and authorization bypass — the families the model layer already
|
||||
arbitrates well — and `php2cpg` works once a PHP interpreter is present, which the shipped image now
|
||||
are overwhelmingly injection, XSS and authorization bypass (the families the model layer already
|
||||
arbitrates well) and `php2cpg` works once a PHP interpreter is present, which the shipped image now
|
||||
provides.
|
||||
2. The system shall add **real-world C library CVEs** beyond the existing curl and OpenSSL rows — libpng and
|
||||
2. The system shall add **real-world C library CVEs** beyond the existing curl and OpenSSL rows: libpng and
|
||||
the image libraries, OpenSSH, libpam, and vulnerable kernel modules were named. These are gated on Req 7.1:
|
||||
until the memory-family decision is closed they would join 173 pairs already sitting idle, so the decision
|
||||
comes first and the harvest follows it.
|
||||
3. The system shall add further **vibe-code** pairs — agent-written and human-written web application bugs —
|
||||
3. The system shall add further **vibe-code** pairs (agent-written and human-written web application bugs)
|
||||
because that slice is where the tool's stated audience actually works, and it is currently 35 pairs.
|
||||
4. Before any slice is vendored at scale, the system shall verify on a sample that the model can **read** its
|
||||
pairs, and shall vendor only the categories that a measurement shows it can arbitrate. The OWASP harvest
|
||||
@@ -194,7 +194,7 @@ what developers actually contribute to rather than against 30 pairs of one slice
|
||||
contiguous key-shaped literal.
|
||||
6. Each new slice shall report as its own row and shall never be blended into a headline, and a synthetic
|
||||
slice shall say in its catalog that it measures breadth rather than real-world performance.
|
||||
7. The system shall record, per slice, how many pairs the model can arbitrate — so corpus growth is measured
|
||||
7. The system shall record, per slice, how many pairs the model can arbitrate, so corpus growth is measured
|
||||
by what it teaches rather than by how many rows it adds.
|
||||
|
||||
### Requirement 10: Regression tuning over a large codebase
|
||||
@@ -206,7 +206,7 @@ tables and queries can be checked for regressions that pair scoring cannot see.
|
||||
|
||||
1. The system shall support a repeatable scan over a pinned checkout of a large repository, recorded as a
|
||||
committed baseline.
|
||||
2. When the fact tables or queries change, the system shall report the delta against that baseline —
|
||||
2. When the fact tables or queries change, the system shall report the delta against that baseline:
|
||||
findings gained, findings lost, and rung movements.
|
||||
3. The system shall treat an unexplained loss of a previously-entailed finding as a regression to be
|
||||
investigated, not a number to be accepted.
|
||||
@@ -214,7 +214,7 @@ tables and queries can be checked for regressions that pair scoring cannot see.
|
||||
|
||||
---
|
||||
|
||||
## Amendment record — group 2 (approved 2026-09-09)
|
||||
## Amendment record, group 2 (approved 2026-09-09)
|
||||
|
||||
Group 2 was specified as the phase that can reshape the rest, and it did. Three amendments were drafted from
|
||||
what it measured and **approved by the maintainer on 2026-09-09**; they are folded into the numbered
|
||||
@@ -227,7 +227,7 @@ requirements above and marked there with the evidence that produced them.
|
||||
| C — a checkout that can demonstrate detection | Req 4.6 | libpng satisfies 4.1 and cannot demonstrate detection |
|
||||
|
||||
**A is only half implemented, deliberately.** Task 2.11 made pattern matches proportionate. The same argument
|
||||
applies to the model layer's own suspicion band — 38 of vampi's 60 findings — which is an enumerated
|
||||
applies to the model layer's own suspicion band (38 of vampi's 60 findings), which is an enumerated
|
||||
candidate the graph could not decide and an LLM affirmed: a pattern match with an opinion attached, not a
|
||||
reasoned claim. Task 2.17 carries that half.
|
||||
|
||||
@@ -238,5 +238,5 @@ reasoned claim. Task 2.17 carries that half.
|
||||
(2.6), where the leak fell 42.9% → 0.0% with pair-correct unchanged.
|
||||
* **Req 3.3's honesty counters were the load-bearing part.** `regions_unjudged` is what made a budget that
|
||||
stopped free work visible at all.
|
||||
* **Req 11.2's byte-identical gates held through every one of nine defect fixes** — which says the same thing
|
||||
* **Req 11.2's byte-identical gates held through every one of nine defect fixes**, which says the same thing
|
||||
from the other side: the corpus could not have caught any of them.
|
||||
|
||||
Reference in New Issue
Block a user