mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
fix(learning-harness): the round-4 review findings
Three defects, two of them the same shape as ones already rejected once: - The meta-agent proposer still could not propose. `MetaAgent(inner_model=)` takes a ModelConfig; it was handed a model name, a `# type: ignore` hid that from mypy, and a bare `except Exception` turned the resulting AttributeError into `meta_agent_failed` -- which reads exactly like the model declining. The composition is now a seam a test can pin, and a construction error is re-raised rather than reported as a quiet round. - `ousast pairs --hunter` crashed in its documented form: the K floor landed with `--k-runs` still defaulting to 1. The covering test always passed `--k-runs 4`, which is why nothing noticed. - 28 rows (12% of the corpus) resolved a span on the vulnerable side and none on the fixed side, so every fixed-side finding read as outside the labeled function and the pair scored as silence whatever the detector said. `fix_function` now reaches the catalog and the scorer, and the generator derives it from the fixed excerpt for the 25 trap rows that never declared one. Measured after: 0 rows. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
75ae9ee47b
commit
2dad25d312
@@ -189,7 +189,9 @@ def main(argv: list[str] | None = None) -> int:
|
||||
)
|
||||
pairs.add_argument("--loo-out", type=Path, default=None, help="where to write loo.json (default: reports/loo.json)")
|
||||
pairs.add_argument("--json", action="store_true", help="print the pair scoreboard as JSON")
|
||||
pairs.add_argument("--k-runs", type=int, default=1, help="runs per pair on the hunter path; the family scorer needs at least three")
|
||||
pairs.add_argument(
|
||||
"--k-runs", type=int, default=None, help="runs per pair on the hunter path; never fewer than three, and three by default"
|
||||
)
|
||||
|
||||
subparsers.add_parser("mcp", help="run the narrow MCP server over stdio for OpenCode integration")
|
||||
mechanisms = subparsers.add_parser("mechanisms", help="maintainer: mechanism memory seeded from trusted pairs")
|
||||
@@ -982,7 +984,10 @@ def _pairs(
|
||||
from .learning.families import load_families
|
||||
|
||||
scan = make_hunter_scan(client, model, taxonomy=load_families())
|
||||
result = evaluate_catalog(cases, hunter=scan, pointers=True if pointers else None, k_runs=k_runs)
|
||||
try:
|
||||
result = evaluate_catalog(cases, hunter=scan, pointers=True if pointers else None, k_runs=k_runs)
|
||||
except ValueError as exc:
|
||||
raise SystemExit(f"--k-runs: {exc}") from exc
|
||||
if json_out:
|
||||
print(json.dumps(result_payload(result), indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
Reference in New Issue
Block a user