mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
Marius CiepluchandClaude Opus 5.5
7f6bbacb6e
learn/compile.toml: the pre-registered spec -- balanced Brier score of the raw score on C_val (cells by family x label, repositories capped at 5, unsure = 0.5, ties within 0.005 to fewer prompt tokens), seed, the compile split (25%, boot 60/40, compile-split-only memory), bootstrap (150 candidates, confidence >= 0.7, 4 sets of 3), search (6 proposals + baseline, 20 errors shown, 60 validation candidates), k = 1, the retrieval settings, and the $3 / 1,200-call ceiling. learn/compile.py: bootstrap few-shot keeps only right, confident rationales that cite a line and name no repository, path or CVE/GHSA id; seeded demo sets (positive, negative, any; families mixed) with other-framework alternates; one proposal call; 7 instructions scored with set 1, the best with sets 2-4. The artifact is canonical JSON stored as programs/<program_id>.json, program_id = sha256 of the artifact without its id; it records the spec sha, memory/compile-groups/folds digests and the response-cache digest, with spend priced from cached usage, so a rerun from the cache is byte-identical at $0 (tested on FileStore and MinioStore through the fake client; real MinIO gated as today). A ceiling ends the compile `unfinished` with no artifact. packageable() keeps only permissively licensed demonstrations (MIT, BSD, Apache-2.0, ISC, zlib), swapping in a permissive alternate or dropping, under a new program id. learn/calibrate.py: Platt scaling on logit(s), pure Python Newton, per family / pooled with family intercepts / insufficient data by the floors; cross-fitted (fold j's map is fitted on the other folds only); the check (5-bin reliability, slope and intercept with repository-bootstrap CIs, ECE <= 0.05), ADVISORY only when it fails. learn/evaluate.py: predictions under each fold's boundary, families below the floor never asked; operating points read from the calibrated, cross-fitted predictions of the other folds (BLOCK: Wilson lower bound >= 0.95, never on a majority unsure; ADVISORY: recall >= 0.90); repository-cluster bootstrap intervals; strata with "insufficient data"; >5% parse failures is an instrument failure; explanations keep only signals the record holds; version mismatch names the fallback reason; the memory-size learning curve. `ousast learn compile|evaluate|curve`: paid runs need --budget-usd (exit 3 at the ceiling), --replay-only runs from the cache. Nothing was spent: every test and the dry run used scripted clients. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo