mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
Marius CiepluchandClaude Fable 5.1
48510fd5c2
Reject B. 160 paired units (injection 119, access_control 41; 56 repositories), both arms, two replicates each, no early stop. Canary agreement A 0.875 -> B 0.906 (CI [-0.03, +0.10], spans; B meets 0.90). Within-pair AUC A 0.815 -> B 0.741 (CI [-0.13, -0.02], against B). AUC -0.02 (against), ADVISORY recall -0.06 and precision +0.01 (span), usd per candidate A $0.0021 -> B $0.0051 (against). Priced usage A $0.68 / B $1.63; balance $16.72 -> $16.04 at the run's end (lags). Counts only; task 7.4 marked done. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
24 lines
1.5 KiB
JSON
24 lines
1.5 KiB
JSON
{
|
|
"experiment": "exp-002-retrieval-ensemble",
|
|
"registered": "2026-10-02",
|
|
"store": "s3://sast-memory",
|
|
"manifest": "plane/experiments/exp-002-retrieval-ensemble.yaml",
|
|
"manifest_digest": "f5004fb8618b1f7cc7dcf298d42b4ac1b6c0c1aa72a97299c77f937bd6175148",
|
|
"pin": "9213e5ffd978b387439e5568d32f4269dd81c021",
|
|
"units_status": "frozen",
|
|
"units_file": "plane/experiments/exp-002-retrieval-ensemble.units.jsonl",
|
|
"units_digest": "0de66bfa3ff844825e06f4f4227a14210d1b77f1cc08c0441c6d23e23fa5d91f",
|
|
"units": {
|
|
"total": 160,
|
|
"paired": 108,
|
|
"groups": 56,
|
|
"by_family": {
|
|
"injection": {"units": 119, "paired": 76, "pairs": 38, "groups": 34, "labels": {"1": 63, "0": 56}},
|
|
"access_control": {"units": 41, "paired": 32, "pairs": 16, "groups": 24, "labels": {"1": 25, "0": 16}}
|
|
},
|
|
"note": "frozen on the 937-example static memory (slice 2 evaluated 110 + 37 candidates on the 904-example memory of the same day; the recovered negatives of ef6945e moved the seeded compile split and the outer folds). Freezing needed two repairs first: example rows carried the plane-only features as null in x and the renderer refused them (1f525ff), and the RustFS store held none of the 788 embedding vectors the file store had (copied over with put_blob, no embedding call)."
|
|
},
|
|
"first_registration": {"registered": "2026-10-02", "pin": "fbc3220bbe1c8a4f8afa0a724fd41a51b390dd22", "units_status": "pending"},
|
|
"status": "run and analysed 2026-10-02: reject (result.json; benchmarks/measurements/2026-10-02-exp-002/record.json)"
|
|
}
|