marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-18 04:29:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(learning-harness): round 5, and what it says about the experiment itself

Browse Source 
Round 5 re-ran round 1's content through the lever it should have used --
counterexamples, rendered to the detector as "Worked examples" rather than
"Known non-issues, do not report these" -- and was rejected because one pair
of eight moved, at p = 1.000.

So the experiment I designed to give the demo lever a fair test could not
give it one. Correcting my authoring error changed the reason for the verdict
and not the verdict, because the rule cannot tell a lever that helps from a
lever that does nothing.

Five rounds, $2.17, zero decisions distinguishable from chance.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-07 08:09:50 +02:00
co-authored by Claude Opus 5
parent 78924316be
commit 66a51c674c
2 changed files with 42 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/learning-harness/tasks.md
+4 -3
View File
@@ -247,7 +247,7 @@
- **28 rows resolved a span on the vulnerable side and none on the fixed side** (12% of the corpus), so every fixed-side finding read as outside the labeled function and the pair scored as silence whatever the detector said — the same flattering direction as the range defect, one-seventh the size. `fix_function` reached only the harvester; it now reaches the catalog and `PairCase`, `_spans` uses it for the fixed side, and the catalog generator derives it from the fixed excerpt itself for the 25 vibe-py trap rows that never declared one. A row that still resolves no fixed-side span reports `unresolved_label:fixed` instead of scoring silent. Measured after: **0 rows**.
- Suggestions applied: one spelling per corpus fact everywhere (`PairCase.unscorable` verbatim, no invented prefix); leave-one-out records non-teachers withheld for tier or an unscorable reason, not only holdout refusals; the hygiene test asserts the declared twins still hash equal, so a declaration that stopped being true would not be invisible; `is_post_cutoff` refuses a cutoff coarser than a day, which was the one over-claim path that existed, and month precision is pinned; `UNSCORABLE_REASONS` covers the whole vocabulary rather than `identical_twin` alone; the zero-run score carries its date; `_pairs_named_by` filters by substring before the regex.
- Carry-forward, measured 2026-09-06 after group 4 (`benchmarks/measurements/2026-09-06-trajectory-divergence.json`): **the K-run numbers this spec publishes are ensemble votes, not repeated measurements.** Across 120 runs of the injection family at temperature 0 with thinking disabled there was not one repeated tool-call trajectory, and outcome disagreement tracks trajectory divergence (30.9% at low similarity, 4.2% at mid). Consequences for anyone reading these numbers: Req 3.5's K is an ensemble over distinct search procedures; Req 8.2's noise floor measures how often an unconstrained search reaches the sink rather than the detector's uncertainty, so Req 9.5's regression budget grows with the search's own wandering; and the instability is entirely one-sided — every pair produced a single finding set on the fixed side, so specificity is solid and recall carries the caveat. `.kiro/specs/detector-optimizer/brief.md` proposes the repair, and it is a program change rather than a prompt change.
- Task 5.3 (2026-09-07, four rounds on `injection`, $1.76, `benchmarks/measurements/2026-09-07-evolve-rounds-injection.json`). The task was amended to target injection because access_control has one train teacher.
- Task 5.3 (2026-09-07, five rounds on `injection`, $2.17, `benchmarks/measurements/2026-09-07-evolve-rounds-injection.json`). The task was amended to target injection because access_control has one train teacher.
| round | lever | outcome | train | p | holdout | p | distinguishable from noise |
|---|---|---|---|---|---|---|---|
@@ -255,14 +255,15 @@
| 2 | checklist | rejected `train_regressed` | 0+/1− | 1.000 | 1+/1− | 1.000 | no |
| 3 | prompt | **accepted `improved` → v1** | 2+/0− | 0.500 | 2+/1− | 1.000 | no |
| 4 | meta-agent | rejected `no_proposal` | — | — | — | — | never ran |
| 5 | counterexample | rejected `train_regressed` | 0+/1− | 1.000 | 1+/3− | 0.625 | no |
**Nothing the loop decided carried evidence, in either direction.** Round 2 rejected a change because **one pair of eight** moved, at p = 1.000 — the maximum the sign test can return. Round 3 accepted one because two of eight improved and none regressed, at p = 0.500, and that acceptance is now versioned as v1 and will be built on. On a family whose measured floor says **11 of 20 pairs disagree with themselves between runs of the unchanged detector**, both readings are what changing nothing looks like.
**Nothing the loop decided carried evidence, in either direction — five rounds, zero reliable decisions.** Round 2 rejected a change because **one pair of eight** moved, at p = 1.000 — the maximum the sign test can return. Round 3 accepted one because two of eight improved and none regressed, at p = 0.500, and that acceptance is now versioned as v1 and will be built on. On a family whose measured floor says **11 of 20 pairs disagree with themselves between runs of the unchanged detector**, both readings are what changing nothing looks like.
I overstated this once and correct it here: after rounds 1 and 2 I wrote that the rule was "close to structurally incapable of accepting anything". Round 3 accepted. The accurate statement is symmetric and worse — the rule accepts on noise as readily as it rejects on it.
- **Escalation to the maintainer, not applied here.** Req 9.5's threshold rule (reject if either side regresses, accept if one improves and neither regresses, tie rejects) should be replaced by an acceptance that survives the measured variance. `Archive` already holds the per-pair Pareto frontier and `acceptance.reliability` already computes the test; neither decides anything. The honest caveat is that at a minibatch of 8 and a holdout of 10 a reliability-gated rule would accept almost nothing either: **the corpus is the binding constraint, not the rule**, and no acceptance rule fixes a train split of twelve.
- **The v1 configuration rests on p = 0.500** and should probably be reverted before any further round builds on it. Left in place, and flagged, because reverting an accepted round is the maintainer's call.
- Round 1's rejection was correct but for my error, not the loop's insight: the `memory` lever writes `hard_negatives.jsonl`, which the detector renders as *"Known non-issues, do not report these"*, and I fed it the four vulnerability shapes the train split teaches. Round 5 re-runs the same content through `counterexamples` (*"Worked examples"*), which is the lever I meant.
- Round 1's rejection was correct but for my error, not the loop's insight: the `memory` lever writes `hard_negatives.jsonl`, which the detector renders as *"Known non-issues, do not report these"*, and I fed it the four vulnerability shapes the train split teaches. Round 5 re-ran the same content through `counterexamples` (*"Worked examples"*), which is the lever I meant — and was rejected because **one pair of eight** moved, at p = 1.000. **The experiment I designed to give the demo lever a fair test could not give it one.** The rule cannot distinguish a lever that helps from a lever that does nothing, so correcting my authoring error changed the reason for the verdict and not the verdict.
- What did work: the cross-family sweep caught a genuine negative flip in `access_control` in round 2 — a real regression found by the before/after comparison that replaced the absolute-failure count, which under the old rule would have rejected rounds for families that were already failing.
- **The finding that outranks the rounds (maintainer, 2026-09-07: "the error is not the model, the error is the elsewhere").** Eighteen defects were found in this session's measurement and round work. Every one was in this repository; none was the model's. The list is kept because it should change how the next bad number is triaged: the prose parser that discarded findings the model had already reached; the CLI that never read the cost meter it published; the baseline that lost everything on an exception, and again on a timeout; two keys that collapsed a slice into another, in two different modules; the pattern ruleset at a 12.1% ceiling; a harvester that truncated an excerpt at both ends; a family shape that could not see a permissive default inside a call argument; a lever wired to the file that says *do not report these*; a provider seam hard-coded to a vendor with no key; a proposer reason the journal threw away; a retry that did not cover a body which stopped mid-read; an exception path wrapped around half a round; log-probabilities read from the wrong object; `.mjs` absent from the language table; and an access-level check that matched prose inside a decorator. Three of them were the same shape — fixing an instance and missing the class. **The default assumption for an unexplained number is that we broke it.** `deepseek-v4-pro` scoring 46.7% against flash's 83.3% is unexplained and is therefore, by this prior, ours until shown otherwise.