marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

bench(plane): first increment measured on ax -- gate not met (14/20 agreed vs 16)

Browse Source 
validation-46 ran done on google/ax: 60 tasks, 357 DeepSeek calls, $0.83 with
recorded triage ($0.0194 per candidate, reference $0.022: cost met). Declared
sites agreed 14/20 against the reference's 16/20: recall criterion not met.
Both runs find 19/20 sites in at least one pass; the two-pass agreement
flicker moved three sites. Design revisit before task 9.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-29 22:19:38 +02:00
co-authored by Claude Opus 5.5
parent c97e216d45
commit c5e6ed3221
2 changed files with 38 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/ai-service-plane/tasks.md
+1 -1
View File
@@ -122,7 +122,7 @@ mypy are clean, and the commit is gated on pytest's own exit code. Script-level
- Workspaces generated for the 15 validation cases' vulnerable pins; the runner image rebuilt and loaded.
- _Requirements: 1.1, 1.2, 6.1_
- [ ] 8. Measurement of the first increment
- [x] 8. Measurement of the first increment
- Run `validation-46` through the reconciler on ax (needs DeepSeek credit); record cost per candidate, tool turns
per hunt, declared sites agreed, against `verifier-batched-check-2026-09-29.json`.
- Run `benchmarks/dev/token_report.py` on the increment's development session.