mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 04:29:27 +00:00
docs: plain-English pass over the active specs
Ran the humanizer rules over 79 prose files in the active specs (brief, design, requirements, research and narrative docs). Removed em dashes, significance inflation, forced rule-of-three, bold-label list items and filler. Skipped tasks.md checklists and the retired specs (harnessx-*, openrouter-sast-harness, unseen-repo-evaluation). Every number, path, command, hash and identifier was preserved: checked each file against HEAD for drift in numbers, inline code and code fences, zero drift across all 79 files. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
b08f372022
commit
39a1873e5b
@@ -1,4 +1,4 @@
|
||||
# Brief: finding-feedback-loop — what happens after a finding is judged wrong
|
||||
# Brief: finding-feedback-loop, what happens after a finding is judged wrong
|
||||
|
||||
**Status:** brief only, unapproved. Successor scope lifted out of `contributor-scan`, which is mid-implementation
|
||||
and does not cover this.
|
||||
@@ -7,13 +7,13 @@ and does not cover this.
|
||||
|
||||
Two task groups were drafted into `contributor-scan/tasks.md` on 2026-09-09 without passing through
|
||||
requirements or design. That was a workflow error: they are not refinements of scan integration, they are a
|
||||
different feature — a loop that learns from findings a human rejected — and writing them as tasks skipped the
|
||||
different feature (a loop that learns from findings a human rejected) and writing them as tasks skipped the
|
||||
two phases that decide whether the feature is shaped right. They are lifted here to be specified properly.
|
||||
|
||||
## The gap this exists to close
|
||||
|
||||
Nine engine defects were found in one session against two repositories. **Three of them were the same bug in
|
||||
three different matchers**: a spec token matched as a substring rather than as a word — `resolve` inside
|
||||
three different matchers**: a spec token matched as a substring rather than as a word: `resolve` inside
|
||||
`resolveUrl` in the sanitizer matcher, `user` inside `users` in the discharger matcher, `gets` inside `fgets`
|
||||
in the sink matcher. Each was found by a measurement, fixed locally, and left no trace. The third survived
|
||||
until a 89k-line C library made it visible as 26 identical false positives.
|
||||
@@ -60,17 +60,17 @@ sound. The same machinery writing sink tables is the thing that produced the lea
|
||||
|
||||
## What the shape looks like
|
||||
|
||||
* **Per-repository, declarative, committed by the developer.** A dismissal file in the *target* repository —
|
||||
site, rung, reason, author — read by the tool and never written by it, the same rule the declared policy
|
||||
* **Per-repository, declarative, committed by the developer.** A dismissal file in the *target* repository (
|
||||
site, rung, reason, author), read by the tool and never written by it, the same rule the declared policy
|
||||
file already follows. Reviewed like code, diffable, and a dismissal without a reason is how a suppression
|
||||
list becomes a graveyard nobody can audit.
|
||||
* **Dismissals are labels, not suppressions.** With the known vulnerabilities a `benchmarks/repos/` recipe
|
||||
already declares, a dismissal file gives a per-repository evaluation set with *both* classes in it — which
|
||||
already declares, a dismissal file gives a per-repository evaluation set with *both* classes in it, which
|
||||
is what an optimiser needs and what a suppression list normally throws away.
|
||||
* **A ledger for the maintainer.** Counter-examples grouped by what they share (same token, same matcher
|
||||
clause, same witness shape), so a recurring cause is visible as a cause rather than as five separate bugs.
|
||||
* **Split-respecting adoption.** Train split only, holdout never read, the three gates byte-identical, and
|
||||
precision *and* recall reported on both sides — an adoption that quietly trades recall is worse than none.
|
||||
precision *and* recall reported on both sides; an adoption that quietly trades recall is worse than none.
|
||||
|
||||
## The acceptance test for the model's role
|
||||
|
||||
@@ -80,7 +80,7 @@ recover a cause we already know should not be trusted to propose one we do not.
|
||||
## Known limits to carry forward
|
||||
|
||||
* The pair corpus could not see any of the nine defects; whole-repository measurement is what found them, and
|
||||
this loop inherits that. Its evaluation set has to include repositories, not only pairs.
|
||||
this loop inherits that. Its evaluation set has to include repositories as well as pairs.
|
||||
* Only two repositories exist today (one Python, one C), and the C one has never produced a true positive.
|
||||
An optimiser tuned on that evidence base would be tuned on very little.
|
||||
* `contributor-scan` Req 8 already warns about the +16% overfitting gap between the slice a change was
|
||||
|
||||
@@ -10,7 +10,7 @@ anything.
|
||||
|
||||
The design turns on one asymmetry: **two of those three bugs needed no judgement at all.** Once the first was
|
||||
understood, the other two were the same fix carried across. So the automation that pays is *propagation and
|
||||
memory*, and the model is confined to the one step that genuinely needs inference — naming what a group has in
|
||||
memory*, and the model is confined to the one step that genuinely needs inference: naming what a group has in
|
||||
common.
|
||||
|
||||
### Goals
|
||||
@@ -22,7 +22,7 @@ common.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- **Automated fact-table edits.** Not deferred — prohibited. See Boundary Commitments.
|
||||
- **Automated fact-table edits.** Prohibited rather than deferred. See Boundary Commitments.
|
||||
- Suppressing findings silently. A dismissal makes a finding *labelled*, never invisible.
|
||||
- A general-purpose optimiser. The only surfaces evolved are two prompt constants.
|
||||
- Replacing the pair corpus. It is the holdout that keeps per-repository tuning honest.
|
||||
@@ -38,10 +38,10 @@ common.
|
||||
|
||||
### Out of Boundary
|
||||
|
||||
- `semantic/facts`, `semantic/obligations`, `ruleset/`, the declared policy file — **no automated writer, by
|
||||
- `semantic/facts`, `semantic/obligations`, `ruleset/`, the declared policy file: **no automated writer, by
|
||||
any path, at any time.** This is the boundary the whole design exists to respect.
|
||||
- The three gates. They are the invariant an adoption is measured against, not a thing this spec changes.
|
||||
- `model/taint.py`, `dominance.py`, `config_value.py` — arbiters are fixed here; a proposal that would change
|
||||
- `model/taint.py`, `dominance.py`, `config_value.py`: arbiters are fixed here; a proposal that would change
|
||||
one is a proposal for a human, not an edit.
|
||||
|
||||
### Allowed Dependencies
|
||||
@@ -64,7 +64,7 @@ Three pieces already exist and are reused rather than rebuilt:
|
||||
- **`benchmarks/repos/*.toml`** declares known vulnerabilities per checkout, with `in_scope` stating whether a
|
||||
family can decide one at all. That is the positive half of an evaluation set, already committed.
|
||||
- **`pairs.py`** already carries a train/holdout split and the machinery that respects it. The leak of
|
||||
2026-09-05 was in the *lever*, not the splitter — `semantic/loo.evaluate_loo` was honest throughout.
|
||||
2026-09-05 was in the *lever*, not the splitter; `semantic/loo.evaluate_loo` was honest throughout.
|
||||
- **`redaction.py`** is applied to every prompt and persisted trajectory, and applies here unchanged.
|
||||
|
||||
### Boundary Map
|
||||
@@ -87,21 +87,21 @@ benchmarks/repos/*.toml ── read ────▶ evaluation set
|
||||
|
||||
**A dismissal is a label, not a suppression.** A suppressed finding disappears and its reason rots; a
|
||||
labelled one is reported as dismissed with its reason, and a dismissal that stops matching is reported too.
|
||||
This is what turns a suppression list into the negative half of an evaluation set — the thing that makes the
|
||||
This is what turns a suppression list into the negative half of an evaluation set, the thing that makes the
|
||||
rest of the feature possible.
|
||||
|
||||
**The tool never writes the dismissal file.** Same rule as the declared policy file, for the same reason: a
|
||||
file the tool edits is a file no reviewer trusts. It is committed by a developer and reviewed like code.
|
||||
|
||||
**Grouping is deterministic; only the hypothesis is not.** Entries are grouped by shared spec token, matcher
|
||||
clause, family and witness shape — all mechanical. The model is asked one question about a group it did not
|
||||
clause, family and witness shape; all mechanical. The model is asked one question about a group it did not
|
||||
choose, and its answer goes to a queue. This keeps the inference confined to the step that needs it.
|
||||
|
||||
**Evolution targets the question, never the facts.** A prompt is how we ask; a fact table is what we claim.
|
||||
`CANDIDATE_QUESTION` and `residual_question` are already the LLM-facing surface and are the only mutable
|
||||
targets. Enforced by a test that fails if any writer touches the fact modules, not by convention.
|
||||
|
||||
**The holdout is never read.** Not "should not" — the evaluation-set API has no path to it during
|
||||
**The holdout is never read.** The evaluation-set API has no path to it during
|
||||
optimisation, and a test asserts the absence. The 2026-09-05 leak was possible because reading it was merely
|
||||
discouraged.
|
||||
|
||||
@@ -123,9 +123,9 @@ benchmarks/ledger/
|
||||
|
||||
### Changed
|
||||
|
||||
- `cli.py` — a `feedback` subcommand: `ledger`, `groups`, `propose`, `evolve`. Maintainer surface; `scan`
|
||||
- `cli.py`: a `feedback` subcommand: `ledger`, `groups`, `propose`, `evolve`. Maintainer surface; `scan`
|
||||
gains only the dismissal read.
|
||||
- `reports.py` — a dismissed finding is rendered as dismissed with its reason.
|
||||
- `reports.py`: a dismissed finding is rendered as dismissed with its reason.
|
||||
|
||||
### Unchanged and load-bearing
|
||||
|
||||
|
||||
@@ -5,22 +5,22 @@
|
||||
## Introduction
|
||||
|
||||
Nine engine defects were found in one session against two repositories. **Three were the same bug in three
|
||||
matchers** — a spec token matched as a substring rather than a word: `resolve` inside `resolveUrl` in the
|
||||
matchers**: a spec token matched as a substring rather than a word: `resolve` inside `resolveUrl` in the
|
||||
sanitizer matcher, `user` inside `users` in the discharger matcher, `gets` inside `fgets` in the sink matcher.
|
||||
Each was found by a measurement, fixed locally, and left no trace; the third survived until an 89k-line C
|
||||
library made it visible as 26 identical false positives.
|
||||
|
||||
Nothing in the system remembers that a finding was wrong. There is no artifact saying *this site, at this
|
||||
rung, is not a defect, and here is why* — so a class recurs, and each recurrence costs another measurement.
|
||||
rung, is not a defect, and here is why*, so a class recurs, and each recurrence costs another measurement.
|
||||
|
||||
Two audiences want that artifact for different reasons. A maintainer wants the *propagation* automated: two of
|
||||
the three matcher bugs needed no judgement at all, only the fix carried across. A developer wants to dismiss a
|
||||
finding in their own repository, have it stay dismissed, and have the reason reviewable beside the code — a
|
||||
finding in their own repository, have it stay dismissed, and have the reason reviewable beside the code; a
|
||||
tool that cannot be told it is wrong is one they stop running.
|
||||
|
||||
This feature is bounded by a rule the project has held from the start and a leak that shows why. An optimiser
|
||||
may evolve **the question** — the judge and residual prompts, which are how we ask. It may never author **the
|
||||
facts** — tables, rulesets, the declared policy file, which are what we claim. On 2026-09-05 the improve lever
|
||||
may evolve **the question**: the judge and residual prompts, which are how we ask. It may never author **the
|
||||
facts**: tables, rulesets, the declared policy file, which are what we claim. On 2026-09-05 the improve lever
|
||||
admitted candidate shapes because they "recover a currently missed holdout pair"; five of eleven were taught by
|
||||
the holdout pairs they then recovered, and a holdout Youden of **−0.059 was reported as +0.059**.
|
||||
|
||||
@@ -38,11 +38,11 @@ recorded and my colleagues can see why.
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The system shall read dismissals from a declarative file in the **target** repository, and shall never
|
||||
write to it — the same rule the declared policy file already follows.
|
||||
write to it, the same rule the declared policy file already follows.
|
||||
2. A dismissal shall carry the site it concerns, the rung it was dismissed at, a reason, and an author.
|
||||
3. The system shall reject a dismissal that carries no reason, because a suppression list without reasons is
|
||||
a graveyard nobody can audit.
|
||||
4. The system shall report a dismissed finding as dismissed, with its reason, rather than omitting it — so a
|
||||
4. The system shall report a dismissed finding as dismissed, with its reason, rather than omitting it, so a
|
||||
stale dismissal is visible rather than silent.
|
||||
5. When a dismissal no longer matches any finding, the system shall say so, because a dismissal that has
|
||||
stopped applying is either a fixed bug or a moved target and both are worth knowing.
|
||||
@@ -54,20 +54,20 @@ cause is visible as a cause rather than as five separate bugs.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a finding is contradicted — by a dismissal, by the judge with coverage, or by a declared contract —
|
||||
1. When a finding is contradicted (by a dismissal, by the judge with coverage, or by a declared contract)
|
||||
the system shall record the site, the rung, the witness, the spec tokens that produced it, and the commit.
|
||||
2. Each ledger entry shall be runnable as a regression case: this site, at this commit, shall not be reported
|
||||
at that rung.
|
||||
3. The ledger's regression cases shall run offline in CI, so that re-introducing a fixed cause fails a test
|
||||
rather than a repository scan four weeks later.
|
||||
4. The system shall group ledger entries by what they share — spec token, matcher clause, family, witness
|
||||
shape — and report the groups by size.
|
||||
4. The system shall group ledger entries by what they share (spec token, matcher clause, family, witness
|
||||
shape) and report the groups by size.
|
||||
5. The ledger shall be committed evidence, not a local artifact.
|
||||
|
||||
### Requirement 3: A model may propose a cause, and only propose it
|
||||
|
||||
**Objective:** As a maintainer, I want the model to do the part that needs judgement — naming what a group of
|
||||
counter-examples has in common — without it deciding anything.
|
||||
**Objective:** As a maintainer, I want the model to do the part that needs judgement (naming what a group of
|
||||
counter-examples has in common) without it deciding anything.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
@@ -132,7 +132,7 @@ quietly loses recall is worse than none.
|
||||
|
||||
1. The evaluation set shall include whole repositories, not pairs alone: the pair corpus could not see any of
|
||||
the nine defects that motivate this feature, and the gates stayed byte-identical through every fix.
|
||||
2. The system shall state how many repositories its evidence base contains, because two — one of which has
|
||||
never produced a true positive — is very little to tune against.
|
||||
2. The system shall state how many repositories its evidence base contains, because two (one of which has
|
||||
never produced a true positive) is very little to tune against.
|
||||
3. Where an adoption is measured on one repository only, the system shall label it as such rather than
|
||||
reporting it as a general improvement.
|
||||
|
||||
Reference in New Issue
Block a user