marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(learn): compile, calibrate, evaluate (decision-engine 6.5)

Browse Source 
learn/compile.toml: the pre-registered spec -- balanced Brier score of the
raw score on C_val (cells by family x label, repositories capped at 5,
unsure = 0.5, ties within 0.005 to fewer prompt tokens), seed, the
compile split (25%, boot 60/40, compile-split-only memory), bootstrap
(150 candidates, confidence >= 0.7, 4 sets of 3), search (6 proposals +
baseline, 20 errors shown, 60 validation candidates), k = 1, the
retrieval settings, and the $3 / 1,200-call ceiling.

learn/compile.py: bootstrap few-shot keeps only right, confident
rationales that cite a line and name no repository, path or CVE/GHSA id;
seeded demo sets (positive, negative, any; families mixed) with
other-framework alternates; one proposal call; 7 instructions scored with
set 1, the best with sets 2-4. The artifact is canonical JSON stored as
programs/<program_id>.json, program_id = sha256 of the artifact without
its id; it records the spec sha, memory/compile-groups/folds digests and
the response-cache digest, with spend priced from cached usage, so a
rerun from the cache is byte-identical at $0 (tested on FileStore and
MinioStore through the fake client; real MinIO gated as today). A ceiling
ends the compile `unfinished` with no artifact. packageable() keeps only
permissively licensed demonstrations (MIT, BSD, Apache-2.0, ISC, zlib),
swapping in a permissive alternate or dropping, under a new program id.

learn/calibrate.py: Platt scaling on logit(s), pure Python Newton, per
family / pooled with family intercepts / insufficient data by the floors;
cross-fitted (fold j's map is fitted on the other folds only); the check
(5-bin reliability, slope and intercept with repository-bootstrap CIs,
ECE <= 0.05), ADVISORY only when it fails.

learn/evaluate.py: predictions under each fold's boundary, families below
the floor never asked; operating points read from the calibrated,
cross-fitted predictions of the other folds (BLOCK: Wilson lower bound
>= 0.95, never on a majority unsure; ADVISORY: recall >= 0.90);
repository-cluster bootstrap intervals; strata with "insufficient data";
>5% parse failures is an instrument failure; explanations keep only
signals the record holds; version mismatch names the fallback reason;
the memory-size learning curve.

`ousast learn compile|evaluate|curve`: paid runs need --budget-usd (exit
3 at the ceiling), --replay-only runs from the cache. Nothing was spent:
every test and the dry run used scripted clients.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-01 10:36:34 +02:00
co-authored by Claude Opus 5.5
parent ea61dc1bb3
commit 7f6bbacb6e
10 changed files with 1694 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/measurements/2026-09-08-module-audit.json
+57 -5
View File
@@ -1,7 +1,7 @@
{
"modules": 150,
"modules": 153,
"counts": {
"load_bearing": 143,
"load_bearing": 146,
"standalone_capability": 7
},
"orphaned": [],
@@ -114,6 +114,7 @@
"cli",
"harness",
"hunter_tools",
"learn.cli",
"model.endpoint",
"pairs",
"plane.memory",
@@ -352,8 +353,11 @@
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
"importers": [
"cli",
"learn.calibrate",
"learn.cli",
"learn.compile",
"learn.embeddings",
"learn.evaluate",
"learn.examples",
"learn.excerpt",
"learn.features",
@@ -366,6 +370,14 @@
"plane.tasks.roles"
]
},
{
"module": "learn.calibrate",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
"importers": [
"learn.evaluate"
]
},
{
"module": "learn.cli",
"classification": "load_bearing",
@@ -374,6 +386,15 @@
"cli"
]
},
{
"module": "learn.compile",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
"importers": [
"learn.cli",
"learn.evaluate"
]
},
{
"module": "learn.embeddings",
"classification": "load_bearing",
@@ -383,12 +404,23 @@
"learn.retrieve"
]
},
{
"module": "learn.evaluate",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
"importers": [
"learn.cli",
"learn.compile"
]
},
{
"module": "learn.examples",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
"importers": [
"learn.cli",
"learn.compile",
"learn.evaluate",
"learn.folds",
"learn.program",
"learn.retrieve"
@@ -415,8 +447,11 @@
{
"module": "learn.folds",
"classification": "load_bearing",
"reason": "imported by learn.program, learn.retrieve",
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
"importers": [
"learn.cli",
"learn.compile",
"learn.evaluate",
"learn.program",
"learn.retrieve"
]
@@ -427,7 +462,10 @@
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
"importers": [
"cli",
"learn.calibrate",
"learn.cli",
"learn.compile",
"learn.evaluate",
"learn.examples",
"learn.excerpt",
"learn.retrieve"
@@ -437,13 +475,18 @@
"module": "learn.program",
"classification": "load_bearing",
"reason": "learned-decision-engine Req 3.1/3.3: the AI classifier program (signature, Retrieve -> Classify(k), response cache); `ousast learn compile` (task 6.5) and the decide task (task 8) are its callers",
"importers": []
"importers": [
"learn.cli",
"learn.compile",
"learn.evaluate"
]
},
{
"module": "learn.retrieve",
"classification": "load_bearing",
"reason": "imported by learn.program",
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
"importers": [
"learn.evaluate",
"learn.program"
]
},
@@ -462,6 +505,8 @@
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
"importers": [
"learn.compile",
"learn.evaluate",
"learn.features",
"learn.program",
"learn.retrieve",
@@ -511,6 +556,7 @@
"cli",
"cpg.backend",
"cpg.session",
"learn.cli",
"learn.features",
"learn.labels",
"model.candidates",
@@ -610,6 +656,7 @@
"reason": "model-grounded-detection Req 8: the judge client, wired by model/judge.py in group 3",
"importers": [
"cli",
"learn.cli",
"plane.budget",
"plane.generate",
"push.assistance",
@@ -795,6 +842,7 @@
"cli",
"improve.memory",
"learn.cli",
"learn.compile",
"learn.embeddings",
"learn.examples",
"learn.program",
@@ -818,6 +866,7 @@
"reason": "ai-service-plane Req 2.3/2.4: metered chat client with usd and calls ceilings; task integration in tasks 3, 6",
"importers": [
"learn.cli",
"learn.compile",
"learn.embeddings",
"learn.program",
"plane.generate",
@@ -876,6 +925,7 @@
"importers": [
"improve.memory",
"learn.cli",
"learn.compile",
"learn.embeddings",
"learn.examples",
"learn.program",
@@ -1088,6 +1138,7 @@
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
"importers": [
"learn.cli",
"learn.compile",
"learn.program",
"model.endpoint",
"tool_hunter"
@@ -1099,6 +1150,7 @@
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
"importers": [
"learn.cli",
"learn.compile",
"learn.program",
"model.endpoint",
"tool_hunter"