mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-19 13:10:03 +00:00
docs(specs): approve the group-2 amendments, generate finding-feedback-loop requirements
**Amendments approved and folded into the numbered requirements** rather than left as a proposal appended to the file: - Req 5.5 — a report proportionate to the evidence behind a finding. A rule whose only evidence is a text match is summarised past a stated number of sites; a finding a proposer reasoned about keeps its detail whatever its rung; nothing leaves findings.json, which is the record a baseline is compared against. - Req 5.6 — a family that cannot decide a class states the limit, and no scan presents an absence of findings as evidence of absence. - Req 4.6 — at least one pinned checkout carries a vulnerability a family can decide in principle, so a scan producing nothing is distinguishable from an engine that cannot see. Each carries the evidence that produced it, and the amendment record keeps what group 2 confirmed rather than changed: Req 8's overfitting warning earned its place twice, Req 3.3's honesty counters made an invisible failure visible, and Req 11.2's gates held through all nine defect fixes. Req 5.5 is half implemented. Task 2.11 made pattern matches proportionate; task 2.17 now carries the model layer's own suspicion band, which is 38 of vampi's 60 findings and is an enumerated candidate the graph could NOT decide with a judge's opinion attached -- a pattern match with an opinion, not a reasoned claim, and weaker evidence than the obligations checker's structural output. **finding-feedback-loop requirements generated**, seven of them: a dismissal file the developer owns and the tool never writes, rejecting one without a reason because a suppression list without reasons is a graveyard nobody can audit; a counter-example ledger whose entries are runnable regression cases, grouped by shared cause so a recurring bug is visible as one; a model that proposes a cause into a review queue and decides nothing, validated by reverting the three known matcher fixes and measuring what it recovers; dismissals plus a recipe's known vulnerabilities as one split evaluation set with the holdout unread and asserted; evolution scoped to the judge and residual questions with the fact tables off limits to any automated means; an adoption that reports what it traded and reverts itself when net-negative; and the loop inheriting the corpus blind spot, so its evidence base is repositories and it states how few it has. Phase: requirements, generated, not approved. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
72932ed879
commit
bccc89be52
@@ -95,6 +95,10 @@ in the corpus exercises it.
|
||||
4. The measurement shall state whether the known vulnerability was found, and at which rung.
|
||||
5. The system shall not require the whole repository to fit in one CPG if that proves infeasible; the
|
||||
fallback shall be recorded and measured rather than assumed.
|
||||
6. At least one pinned checkout shall carry a known vulnerability that a family can decide in principle, so
|
||||
that a scan producing nothing is distinguishable from an engine that cannot see. *(Added 2026-09-09.
|
||||
Criterion 4.1 is satisfied by libpng, whose CVE is out of family and which no version can fix, because
|
||||
the library never calls a sink any family models: the letter was met and the intent was not.)*
|
||||
|
||||
### Requirement 5: Output a contributor can act on
|
||||
|
||||
@@ -107,6 +111,17 @@ in the corpus exercises it.
|
||||
reader does not treat it as established.
|
||||
3. The system shall write findings to the writable output location, never into the repository under analysis.
|
||||
4. The system shall summarise a scan in a form readable without opening a JSON file.
|
||||
5. The report shall be proportionate to the evidence behind a finding: a rule whose only evidence is a text
|
||||
match shall be summarised once it exceeds a stated number of sites, and a finding a proposer reasoned
|
||||
about shall keep its detail whatever its rung. Nothing shall be dropped from `findings.json`, which
|
||||
remains the record a baseline is compared against. *(Added 2026-09-09. libpng emitted 198 findings, 163 of
|
||||
them a pattern asserting `memcpy` and `strcpy` are PRESENT in a library that uses them correctly
|
||||
throughout, and no criterion here forbade it.)*
|
||||
6. Where a family's abstraction cannot decide a weakness class, the system shall state that limit in the
|
||||
report, and no scan shall present an absence of findings as evidence of absence. *(Added 2026-09-09.
|
||||
`c/memory` scored zero true positives across 89k lines of C for a structural reason — it models string
|
||||
functions and libpng overflows through arithmetic. A silent scan reading as a clean bill of health is
|
||||
worse than noise because it is quiet, and it is the one failure mode the rung ladder does not guard.)*
|
||||
|
||||
### Requirement 6: Packaging is part of the product
|
||||
|
||||
@@ -197,55 +212,22 @@ tables and queries can be checked for regressions that pair scoring cannot see.
|
||||
|
||||
---
|
||||
|
||||
## Proposed amendments from group 2 (drafted 2026-09-09, NOT approved)
|
||||
## Amendment record — group 2 (approved 2026-09-09)
|
||||
|
||||
Group 2 was specified as the phase that can reshape the rest, and it did. These are the changes it argues for
|
||||
at the requirements level rather than the task level. They are recorded here unapproved; the tasks already
|
||||
merged into group 2 implement what was within the approved requirements, and these are the parts that are not.
|
||||
Group 2 was specified as the phase that can reshape the rest, and it did. Three amendments were drafted from
|
||||
what it measured and **approved by the maintainer on 2026-09-09**; they are folded into the numbered
|
||||
requirements above and marked there with the evidence that produced them.
|
||||
|
||||
### Amendment A — Req 5 should state what a report may repeat
|
||||
| amendment | folded into | evidence |
|
||||
| --- | --- | --- |
|
||||
| A — what a report may repeat | Req 5.5 | libpng: 198 findings, 163 a pattern match; task 2.11 fixed that half |
|
||||
| B — a family that cannot decide must say so | Req 5.6 | `c/memory`: 0 true positives on 89k lines of C |
|
||||
| C — a checkout that can demonstrate detection | Req 4.6 | libpng satisfies 4.1 and cannot demonstrate detection |
|
||||
|
||||
**Why.** libpng emitted 198 findings, 163 of them a static pattern asserting that `memcpy` and `strcpy` are
|
||||
*present* in a library that uses them correctly throughout. Req 5 says the output must be one a contributor
|
||||
can act on; it does not say anything about volume, so nothing in the spec was violated by a report no
|
||||
contributor would read.
|
||||
|
||||
**Proposed criterion.** *The report shall be proportionate to the evidence behind a finding: a rule whose only
|
||||
evidence is a text match shall be summarised once it exceeds a stated number of sites, and a finding a
|
||||
proposer reasoned about shall keep its detail whatever its rung. Nothing shall be dropped from
|
||||
`findings.json`, which remains the record a baseline is compared against.*
|
||||
|
||||
Implemented in task 2.11 for pattern matches. **Not implemented, and the reason this is an amendment rather
|
||||
than a closed task:** the same argument applies to the model layer's own suspicion band, which is 38 of vampi's
|
||||
60 findings — an enumerated candidate that the graph could not decide and an LLM affirmed. That is the same
|
||||
epistemic class as a pattern match with an opinion attached, and treating it as a reasoned claim is what
|
||||
leaves 51 sections on a 520-line application.
|
||||
|
||||
### Amendment B — a family that cannot decide a class shall say so
|
||||
|
||||
**Why.** `c/memory` produced zero true positives across an 89k-line C library and three distinct classes of
|
||||
false one. The reason is structural: it models string functions, and libpng overflows through arithmetic. A
|
||||
silent scan currently reads as a clean bill of health, and the stated goal is that an LLM will help a
|
||||
developer dive in — so it will fluently explain a silence it has no basis for.
|
||||
|
||||
**Proposed criterion.** *Where a family's abstraction cannot decide a weakness class, the system shall state
|
||||
the limit in the report rather than be silent about it, and no scan shall present the absence of findings as
|
||||
evidence of absence.*
|
||||
|
||||
This is worse than noise because it is quiet, and it is the one failure mode the rung ladder does not already
|
||||
guard against.
|
||||
|
||||
### Amendment C — Req 4 should require a checkout that can produce a true positive
|
||||
|
||||
**Why.** Req 4.1 asks for "at least one known-vulnerable repository checkout at a vulnerable commit", and
|
||||
libpng satisfies it while being incapable of demonstrating detection: its CVE is out of family, and no
|
||||
version of it can be otherwise, because the library never calls a sink any family models. The requirement is
|
||||
met and the intent is not.
|
||||
|
||||
**Proposed criterion.** *At least one pinned checkout shall carry a known vulnerability that a family can
|
||||
decide in principle, so that a scan producing nothing is distinguishable from an engine that cannot see.*
|
||||
|
||||
vampi satisfies this for Python. Nothing satisfies it for C, and 2.15 exists to find out whether anything can.
|
||||
**A is only half implemented, deliberately.** Task 2.11 made pattern matches proportionate. The same argument
|
||||
applies to the model layer's own suspicion band — 38 of vampi's 60 findings — which is an enumerated
|
||||
candidate the graph could not decide and an LLM affirmed: a pattern match with an opinion attached, not a
|
||||
reasoned claim. Task 2.17 carries that half.
|
||||
|
||||
### What group 2 confirmed rather than changed
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "contributor-scan",
|
||||
"created_at": "2026-09-08T20:00:00Z",
|
||||
"updated_at": "2026-09-09T12:00:00Z",
|
||||
"updated_at": "2026-09-09T12:30:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
@@ -22,5 +22,5 @@
|
||||
"depends_on": [
|
||||
"model-grounded-detection"
|
||||
],
|
||||
"approval_note": "2026-09-08 brief drafted after the maintainer asked what the spec state was and whether anything was forgotten. It was: model-grounded-detection is 16/16 complete and its arbiter is measured at 86.7% pair-correct / 6.7% leak, but the model layer has ZERO production callers -- ousast scan does not use it, so the feature is complete and unshipped at once. That is a scope gap in the predecessor (it had no task saying 'make this reachable', and its byte-identical-gates requirement is satisfied precisely because nothing is wired). This spec covers scan integration, whole-repo scanning against known vulnerable repositories, the deferred memory-family decision, and records the pipeline, ops and corpus work delivered outside any spec. Not approved; requirements not generated. ; 2026-09-08 requirements generated at the maintainer's direction ('we follow the spec and build integration afterwards'). ; 2026-09-09 requirements approved and design generated ('let us continue with these'): the model layer joins as one stage inside MAP beside the obligations checker, one CPG per scan, regions from entry points, a total budget rather than a per-region cap. Phase 2 (the first run on a pinned real repository) can invalidate the rest and comes before shipping. ; 2026-09-09 tasks approved by the maintainer ('approve the tasks and start group 1'); implementing group 1 (regions, driver, MAP stage). ; 2026-09-09 group 2 complete except 2.8/2.9/2.13-2.16, and it reshaped the phase as intended. Nine engine defects found by measurement and none by review, gates byte-identical through every fix -- which says the pair corpus could not have caught any of them. Three requirements amendments are DRAFTED AND UNAPPROVED at the end of requirements.md: (A) Req 5 should state what a report may repeat, because libpng emitted 198 findings with nothing in the spec forbidding it, and the model layer's own suspicion band is still 38 of vampi's 60; (B) a family that cannot decide a class must say so, because c/memory scored zero true positives on 89k lines of C for a structural reason and a silent scan reads as a clean bill of health; (C) Req 4.1 is satisfied by a checkout that cannot demonstrate detection at all, so it should require one that can. Separately, two task groups appended to tasks.md without passing requirements or design have been LIFTED OUT to the new finding-feedback-loop spec, where they belong."
|
||||
"approval_note": "2026-09-08 brief drafted after the maintainer asked what the spec state was and whether anything was forgotten. It was: model-grounded-detection is 16/16 complete and its arbiter is measured at 86.7% pair-correct / 6.7% leak, but the model layer has ZERO production callers -- ousast scan does not use it, so the feature is complete and unshipped at once. That is a scope gap in the predecessor (it had no task saying 'make this reachable', and its byte-identical-gates requirement is satisfied precisely because nothing is wired). This spec covers scan integration, whole-repo scanning against known vulnerable repositories, the deferred memory-family decision, and records the pipeline, ops and corpus work delivered outside any spec. Not approved; requirements not generated. ; 2026-09-08 requirements generated at the maintainer's direction ('we follow the spec and build integration afterwards'). ; 2026-09-09 requirements approved and design generated ('let us continue with these'): the model layer joins as one stage inside MAP beside the obligations checker, one CPG per scan, regions from entry points, a total budget rather than a per-region cap. Phase 2 (the first run on a pinned real repository) can invalidate the rest and comes before shipping. ; 2026-09-09 tasks approved by the maintainer ('approve the tasks and start group 1'); implementing group 1 (regions, driver, MAP stage). ; 2026-09-09 group 2 complete except 2.8/2.9/2.13-2.16, and it reshaped the phase as intended. Nine engine defects found by measurement and none by review, gates byte-identical through every fix -- which says the pair corpus could not have caught any of them. Three requirements amendments are DRAFTED AND UNAPPROVED at the end of requirements.md: (A) Req 5 should state what a report may repeat, because libpng emitted 198 findings with nothing in the spec forbidding it, and the model layer's own suspicion band is still 38 of vampi's 60; (B) a family that cannot decide a class must say so, because c/memory scored zero true positives on 89k lines of C for a structural reason and a silent scan reads as a clean bill of health; (C) Req 4.1 is satisfied by a checkout that cannot demonstrate detection at all, so it should require one that can. Separately, two task groups appended to tasks.md without passing requirements or design have been LIFTED OUT to the new finding-feedback-loop spec, where they belong. ; 2026-09-09 the three group-2 amendments APPROVED by the maintainer and folded into the numbered requirements: Req 5.5 (a report proportionate to the evidence behind a finding), Req 5.6 (a family that cannot decide a class states the limit, and no scan presents absence of findings as evidence of absence), Req 4.6 (at least one checkout carrying a vulnerability a family can decide in principle). Req 5.5 is half implemented by task 2.11 -- pattern matches -- and task 2.17 was added for the model layer's own suspicion band, which is 38 of vampi's 60 findings and is an enumerated candidate the graph could not decide with a judge's opinion attached rather than a reasoned claim."
|
||||
}
|
||||
|
||||
@@ -307,6 +307,21 @@
|
||||
- _Requirements: 3.1, 8.2_
|
||||
- _Depends: 2.15_
|
||||
|
||||
- [ ] 2.17 The model's own suspicion band is a pattern match with an opinion attached
|
||||
- **Req 5.5's other half.** Task 2.11 made pattern matches proportionate and left this untouched, so vampi
|
||||
still emits 61 sections for a 520-line application -- 38 of them model suspicions, roughly one per
|
||||
fourteen lines of code.
|
||||
- The epistemic case for treating them as reasoned claims does not survive inspection. A model suspicion is
|
||||
what the enumerator proposed at a site the graph could NOT decide, affirmed by a judge with no coverage
|
||||
there. That is a pattern match with an opinion attached, and it is weaker evidence than the obligations
|
||||
checker's output, which comes from a structural analysis rather than a guess.
|
||||
- So group them by family the way 2.11 groups pattern rules, and keep obligation findings and everything
|
||||
at `model_corroborated` or above in full. Expect vampi to fall from 61 sections to roughly 23.
|
||||
- Measure both repositories, and measure what it costs: if a real finding only ever appeared at model
|
||||
suspicion, this hides it, and that is the number that decides whether the rule is right.
|
||||
- _Requirements: 5.1, 5.2, 5.5_
|
||||
- _Depends: 2.11_
|
||||
|
||||
## Group 3 — Ship it
|
||||
|
||||
- [ ] 3.1 Contributor output
|
||||
|
||||
Reference in New Issue
Block a user