mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Opus 5.5
67699e8737
A third verify pass on the 10 disputed candidates, 2-of-3 agreement (Req 6.4), run on ax by extending the same Run: 16/20 declared sites agreed (reference 16), $0.0205 per candidate with recorded triage (reference $0.022). Caveats recorded: a design change on the same set; 17 vs 19 sites flagged in any pass; three non-declared pgadmin candidates promoted. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
66 lines
2.7 KiB
JSON
66 lines
2.7 KiB
JSON
{
|
|
"measurement": "ai-service-plane increment re-measured after the design revisit (Requirement 6.4): passes a and b as recorded in plane-increment-1.json, plus a tie-break pass c on the 10 candidates a and b disputed; agreement 2-of-3. Executed on google/ax by rerunning the same Run (the 60 done tasks were reused, 30 new tasks ran).",
|
|
"date": "2026-09-30",
|
|
"run": {
|
|
"name": "validation-46",
|
|
"tasks": 90,
|
|
"status": "done",
|
|
"runner_image": "localhost:5001/ousast-runner@sha256:fb9a743027518976659fd11505e58cc3fc84fb42e07627196e13b76ac01d0617"
|
|
},
|
|
"previous": "benchmarks/independent/plane-increment-1.json",
|
|
"gate": {
|
|
"declared_sites_agreed": {
|
|
"plane": 16,
|
|
"reference": 16,
|
|
"of": 20,
|
|
"required": ">= 16",
|
|
"met": true
|
|
},
|
|
"cost_per_candidate_with_recorded_triage_usd": {
|
|
"plane_over_46": 0.0205,
|
|
"plane_over_43_after_triage": 0.02195,
|
|
"reference_over_46": 0.022,
|
|
"required": "< 0.022",
|
|
"met": true,
|
|
"note": "the reference divides by the 46 candidates before triage; on that basis the plane is 7% cheaper, on the 43 after triage it is 0.2% under"
|
|
},
|
|
"result": "MET"
|
|
},
|
|
"detail": {
|
|
"tiebreak": {
|
|
"asked": 10,
|
|
"flipped_to_agreed": 5,
|
|
"declared_sites_recovered": [
|
|
"froxlor listing",
|
|
"froxlor listingCount"
|
|
],
|
|
"declared_site_still_disputed": "froxlor validateIpAddresses (b only)",
|
|
"non_declared_flipped_to_agreed": [
|
|
"pgadmin enable_disable",
|
|
"pgadmin get_job_args_value",
|
|
"pgadmin run_job"
|
|
]
|
|
},
|
|
"agreed_candidates": {
|
|
"two_pass": 23,
|
|
"two_of_three": 28
|
|
},
|
|
"declared_sites_flagged_in_any_pass": {
|
|
"plane": 17,
|
|
"reference": 19,
|
|
"of": 20,
|
|
"missed_by_every_plane_pass": [
|
|
"budibase read",
|
|
"fuxa getDaqValue"
|
|
]
|
|
},
|
|
"usd": {
|
|
"plane": 0.8567,
|
|
"recorded_triage": 0.0872,
|
|
"with_triage": 0.9439,
|
|
"reference": 1.0123
|
|
}
|
|
},
|
|
"reading": "The plane on ax matches the script pipeline's pre-registered recall (16/20 agreed) at 7% lower cost, so Requirement 6.2 is met after the 6.4 revisit. Two caveats travel with the result: (1) the tie-break is a design change chosen after seeing the first measurement on the same validation set, so this compares plane with script like for like and qualifies nothing on an independent population; (2) the plane's passes flag fewer declared sites than the reference's (17 vs 19 in any pass), and 2-of-3 also promoted three non-declared pgadmin candidates, whose truth is unassessed (an earlier pgadmin spot check found 3 of 5 findings truly privileged). Precision is not part of this gate and remains the open problem."
|
|
}
|