mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-17 12:09:27 +00:00
Marius CiepluchandClaude Opus 5
72932ed879
Following the Kiro workflow rather than continuing to write tasks. **A workflow error, corrected.** Two task groups -- a counter-example ledger, and per-repository evolutionary prompting so developers can commit dismissals and evolve the harness -- were appended to contributor-scan's tasks.md on 2026-09-09 without passing requirements or design. They are not refinements of scan integration; they are a different feature, and writing them as tasks skipped the two phases that decide whether a feature is shaped right. They are lifted to .kiro/specs/finding-feedback-loop/, at brief, unapproved. That brief states the boundary the design turns on: an optimiser may evolve the QUESTION (the judge and residual prompts, which are how we ask) and never the FACTS (tables, rulesets, the declared policy file, which are what we claim). The closed-loop leak measured on 2026-09-05 is why -- five of eleven candidate shapes were taught by the holdout pairs they then recovered, reporting a -0.059 holdout Youden as +0.059. It also states the acceptance test for the model's role: revert the three matcher fixes and see whether the grouping rediscovers the known causes. **Three requirements amendments, drafted and unapproved**, at the end of contributor-scan/requirements.md: - Req 5 should state what a report may repeat. libpng emitted 198 findings and nothing in the spec forbade it. Task 2.11 fixed pattern noise; the model layer's own suspicion band is still 38 of vampi's 60, and it is the same epistemic class -- an enumerated candidate the graph could not decide and an LLM affirmed. - A family that cannot decide a class must say so. c/memory scored zero true positives on 89k lines of C for a structural reason, and a silent scan reads as a clean bill of health. Worse than noise because it is quiet, and the one failure mode the rung ladder does not already guard. - Req 4.1 is satisfied by a checkout that cannot demonstrate detection at all -- libpng's CVE is out of family and no version of it can be otherwise. It should require one that can. And what group 2 confirmed rather than changed: Req 8's overfitting warning stopped a rule being extended until one repository was clean and forced the access-control work onto pairs nobody tuned against; Req 3.3's honesty counters are what made a budget stopping free work visible at all; and Req 11.2's byte-identical gates held through all nine defect fixes, which says from the other side that the corpus could not have caught any of them. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
89 lines
5.2 KiB
Markdown
89 lines
5.2 KiB
Markdown
# Brief: finding-feedback-loop — what happens after a finding is judged wrong
|
||
|
||
**Status:** brief only, unapproved. Successor scope lifted out of `contributor-scan`, which is mid-implementation
|
||
and does not cover this.
|
||
|
||
## Why this is its own spec
|
||
|
||
Two task groups were drafted into `contributor-scan/tasks.md` on 2026-09-09 without passing through
|
||
requirements or design. That was a workflow error: they are not refinements of scan integration, they are a
|
||
different feature — a loop that learns from findings a human rejected — and writing them as tasks skipped the
|
||
two phases that decide whether the feature is shaped right. They are lifted here to be specified properly.
|
||
|
||
## The gap this exists to close
|
||
|
||
Nine engine defects were found in one session against two repositories. **Three of them were the same bug in
|
||
three different matchers**: a spec token matched as a substring rather than as a word — `resolve` inside
|
||
`resolveUrl` in the sanitizer matcher, `user` inside `users` in the discharger matcher, `gets` inside `fgets`
|
||
in the sink matcher. Each was found by a measurement, fixed locally, and left no trace. The third survived
|
||
until a 89k-line C library made it visible as 26 identical false positives.
|
||
|
||
Nothing in the system remembers that a finding was wrong. There is no artifact recording *this site, at this
|
||
rung, is not a defect, and here is why*, so the same class recurs and each recurrence costs another
|
||
measurement to find.
|
||
|
||
Two audiences want that artifact for different reasons:
|
||
|
||
* **A maintainer** wants the propagation automated. Two of the three matcher bugs needed no judgement at all,
|
||
only the fix carried across.
|
||
* **A developer** wants to dismiss a finding in their own repository, have it stay dismissed, and have the
|
||
reason reviewable beside the code. Today they have no way to say so, and a tool that cannot be told it is
|
||
wrong is one they stop running.
|
||
|
||
## What the maintainer asked for
|
||
|
||
> "is there a way for an evolution (gepa, simba) style prompting with a declarative and persistent approach
|
||
> when evaluating findings per target repo? so that devs can commit state and evolve the harness / dismiss
|
||
> findings"
|
||
|
||
And, as the standing test for all of it:
|
||
|
||
> "it must find the bugs, otherwise it's SAST noise. most devs know codeql / semgrep have dataflow
|
||
> capabilities but never use it"
|
||
|
||
## The boundary that shapes the design
|
||
|
||
This project has refused LLM-authored patterns and policy from the start, and a closed-loop leak measured on
|
||
2026-09-05 is why that is not negotiable: the improve lever admitted candidate shapes because they "recover a
|
||
currently missed holdout pair", five of eleven were taught by the holdout pairs they then recovered, and a
|
||
holdout Youden of **−0.059 was reported as +0.059**.
|
||
|
||
So a reflective optimiser has exactly one legitimate target here:
|
||
|
||
| may be evolved | may not be evolved |
|
||
| --- | --- |
|
||
| the judge and residual **questions** — how we ask | fact tables, rulesets, the declared policy file — what we claim |
|
||
|
||
A prompt is how a question is put; a fact table is an assertion about the world. GEPA/SIMBA-style evolution
|
||
over `CANDIDATE_QUESTION` and `residual_question`, scored against a repository's own labelled dismissals, is
|
||
sound. The same machinery writing sink tables is the thing that produced the leak.
|
||
|
||
## What the shape looks like
|
||
|
||
* **Per-repository, declarative, committed by the developer.** A dismissal file in the *target* repository —
|
||
site, rung, reason, author — read by the tool and never written by it, the same rule the declared policy
|
||
file already follows. Reviewed like code, diffable, and a dismissal without a reason is how a suppression
|
||
list becomes a graveyard nobody can audit.
|
||
* **Dismissals are labels, not suppressions.** With the known vulnerabilities a `benchmarks/repos/` recipe
|
||
already declares, a dismissal file gives a per-repository evaluation set with *both* classes in it — which
|
||
is what an optimiser needs and what a suppression list normally throws away.
|
||
* **A ledger for the maintainer.** Counter-examples grouped by what they share (same token, same matcher
|
||
clause, same witness shape), so a recurring cause is visible as a cause rather than as five separate bugs.
|
||
* **Split-respecting adoption.** Train split only, holdout never read, the three gates byte-identical, and
|
||
precision *and* recall reported on both sides — an adoption that quietly trades recall is worse than none.
|
||
|
||
## The acceptance test for the model's role
|
||
|
||
Revert the three matcher fixes and run the grouping: does it rediscover the known causes? A model that cannot
|
||
recover a cause we already know should not be trusted to propose one we do not.
|
||
|
||
## Known limits to carry forward
|
||
|
||
* The pair corpus could not see any of the nine defects; whole-repository measurement is what found them, and
|
||
this loop inherits that. Its evaluation set has to include repositories, not only pairs.
|
||
* Only two repositories exist today (one Python, one C), and the C one has never produced a true positive.
|
||
An optimiser tuned on that evidence base would be tuned on very little.
|
||
* `contributor-scan` Req 8 already warns about the +16% overfitting gap between the slice a change was
|
||
developed against and one nobody tuned against. Per-repository evolution makes that risk structural rather
|
||
than incidental.
|