mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 20:49:28 +00:00
docs: plain-English pass over the active specs
Ran the humanizer rules over 79 prose files in the active specs (brief, design, requirements, research and narrative docs). Removed em dashes, significance inflation, forced rule-of-three, bold-label list items and filler. Skipped tasks.md checklists and the retired specs (harnessx-*, openrouter-sast-harness, unseen-repo-evaluation). Every number, path, command, hash and identifier was preserved: checked each file against HEAD for drift in numbers, inline code and code fences, zero drift across all 79 files. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
b08f372022
commit
39a1873e5b
@@ -1,4 +1,4 @@
|
||||
# Brief: finding-feedback-loop — what happens after a finding is judged wrong
|
||||
# Brief: finding-feedback-loop, what happens after a finding is judged wrong
|
||||
|
||||
**Status:** brief only, unapproved. Successor scope lifted out of `contributor-scan`, which is mid-implementation
|
||||
and does not cover this.
|
||||
@@ -7,13 +7,13 @@ and does not cover this.
|
||||
|
||||
Two task groups were drafted into `contributor-scan/tasks.md` on 2026-09-09 without passing through
|
||||
requirements or design. That was a workflow error: they are not refinements of scan integration, they are a
|
||||
different feature — a loop that learns from findings a human rejected — and writing them as tasks skipped the
|
||||
different feature (a loop that learns from findings a human rejected) and writing them as tasks skipped the
|
||||
two phases that decide whether the feature is shaped right. They are lifted here to be specified properly.
|
||||
|
||||
## The gap this exists to close
|
||||
|
||||
Nine engine defects were found in one session against two repositories. **Three of them were the same bug in
|
||||
three different matchers**: a spec token matched as a substring rather than as a word — `resolve` inside
|
||||
three different matchers**: a spec token matched as a substring rather than as a word: `resolve` inside
|
||||
`resolveUrl` in the sanitizer matcher, `user` inside `users` in the discharger matcher, `gets` inside `fgets`
|
||||
in the sink matcher. Each was found by a measurement, fixed locally, and left no trace. The third survived
|
||||
until a 89k-line C library made it visible as 26 identical false positives.
|
||||
@@ -60,17 +60,17 @@ sound. The same machinery writing sink tables is the thing that produced the lea
|
||||
|
||||
## What the shape looks like
|
||||
|
||||
* **Per-repository, declarative, committed by the developer.** A dismissal file in the *target* repository —
|
||||
site, rung, reason, author — read by the tool and never written by it, the same rule the declared policy
|
||||
* **Per-repository, declarative, committed by the developer.** A dismissal file in the *target* repository (
|
||||
site, rung, reason, author), read by the tool and never written by it, the same rule the declared policy
|
||||
file already follows. Reviewed like code, diffable, and a dismissal without a reason is how a suppression
|
||||
list becomes a graveyard nobody can audit.
|
||||
* **Dismissals are labels, not suppressions.** With the known vulnerabilities a `benchmarks/repos/` recipe
|
||||
already declares, a dismissal file gives a per-repository evaluation set with *both* classes in it — which
|
||||
already declares, a dismissal file gives a per-repository evaluation set with *both* classes in it, which
|
||||
is what an optimiser needs and what a suppression list normally throws away.
|
||||
* **A ledger for the maintainer.** Counter-examples grouped by what they share (same token, same matcher
|
||||
clause, same witness shape), so a recurring cause is visible as a cause rather than as five separate bugs.
|
||||
* **Split-respecting adoption.** Train split only, holdout never read, the three gates byte-identical, and
|
||||
precision *and* recall reported on both sides — an adoption that quietly trades recall is worse than none.
|
||||
precision *and* recall reported on both sides; an adoption that quietly trades recall is worse than none.
|
||||
|
||||
## The acceptance test for the model's role
|
||||
|
||||
@@ -80,7 +80,7 @@ recover a cause we already know should not be trusted to propose one we do not.
|
||||
## Known limits to carry forward
|
||||
|
||||
* The pair corpus could not see any of the nine defects; whole-repository measurement is what found them, and
|
||||
this loop inherits that. Its evaluation set has to include repositories, not only pairs.
|
||||
this loop inherits that. Its evaluation set has to include repositories as well as pairs.
|
||||
* Only two repositories exist today (one Python, one C), and the C one has never produced a true positive.
|
||||
An optimiser tuned on that evidence base would be tuned on very little.
|
||||
* `contributor-scan` Req 8 already warns about the +16% overfitting gap between the slice a change was
|
||||
|
||||
Reference in New Issue
Block a user