mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
fix(learning-harness): the round-3 review findings, remediated RED-first
Eight of nine group-3 tasks were rejected. Each defect was hiding a number, and the numbers are why the loop was not learning: - The fixed side was judged against the *vulnerable* side's function ranges. 191 of 227 pairs with a resolvable labeled span (84%) have different spans on the two sides, so a leak was recorded as silence, always in the direction that flatters Youden. - Round zero measured every family with the generalist. `family:unknown` earns no credit against any real family, so every family but `unknown` baselined at a structural zero and every noise floor with it. - The cross-family sweep counted absolute failures instead of negative flips. On a corpus where most pairs fail at round zero that rejects every round for damage it did not do. Unscorable rows voted too. - K runs was declared and never threaded; `pairs --k-runs` was parsed and discarded, and its only test grepped `--help`. - The meta-agent proposer was a stub, so `learning round` could never propose anything. It now composes MetaAgent.evolve against a scratch copy of the family directory, and `--proposals FILE` runs a recorded sequence offline. - `refused_holdout` was a vocabulary word no code path could produce, and the covering test accepted either outcome. - Cost was metered once, after everything had been paid for; a stage that raised left no record at all. - The corrected lever number was published beside the earlier one rather than in place of it, and the test named for that property never checked. - Refusals were `continue` statements: `Refusal.degradation()` existed and nothing called it. A train-split identical twin was reported as `split:train`, which is untrue. Test repairs that each changed a number: two split tests measured an `x = 1` fixture that teaches nothing whatever the rule says; two CLI scoreboard tests asserted against all-zero output produced by a stub that answers twice and puts its finding outside the labeled function. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
c2d9c4ad01
commit
1843ef9a02
@@ -91,9 +91,13 @@ def test_family_findings_never_become_sandbox_candidates(tmp_path: Path, monkeyp
|
||||
assert main(["scan", str(repo), "--mode", "deep", "--config", str(repo / "openultrasast.toml")]) == 0
|
||||
run_dir = _run_dir(repo)
|
||||
verdicts = json.loads((run_dir / "verdicts.json").read_text())["verdicts"]
|
||||
findings = {item["finding_id"] for item in json.loads((run_dir / "findings.json").read_text())["findings"]}
|
||||
del findings
|
||||
assert all(not str(item.get("path", "")).startswith("family:") for item in verdicts)
|
||||
payload = json.loads((run_dir / "findings.json").read_text())["findings"]
|
||||
family_findings = {item["finding_id"] for item in payload if any(str(tag).startswith("family:") for tag in item["tags"])}
|
||||
assert family_findings, "the fixture must actually produce family findings, or this proves nothing"
|
||||
# A family detector's claim is a suspicion until a verifier speaks; handing it to the sandbox would let an
|
||||
# unverified model claim drive execution (Req 7.1).
|
||||
candidates = {str(item.get("finding_id") or item.get("id") or "") for item in verdicts}
|
||||
assert candidates.isdisjoint(family_findings)
|
||||
|
||||
|
||||
def test_the_sarif_report_carries_the_family_properties(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
|
||||
Reference in New Issue
Block a user