marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(pre-push-safety-net): preserve uncertainty in benchmark scoring

Browse Source
This commit is contained in:
Marius Ciepluch
2026-09-12 19:04:20 +02:00
parent 8acada0d98
commit 051f3b0058
13 changed files with 1197 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/pre-push-safety-net/design.md
+6
View File
@@ -216,6 +216,12 @@ question/context/depth, facts/sanitizers, query code and relevant analysis confi
also includes ranking/admission versions and any model/prompt identity. Comparison results include
both base and head identities. Persist manifests with source byte counts and graph census.
These identities deliberately separate graph construction from ranking experiments. A ranking-only
change can reuse a compatible graph and unchanged query evidence while invalidating affected ranked
results. A query/fact change invalidates its answers without automatically rebuilding a graph whose
input/frontend/overlay semantics are unchanged. This supports iteration on the core as well as
repeated push checks; it does not authorize reusing stale evidence after a source change.
Only validated, complete artifacts can satisfy complete-result lookups. Partial artifacts may be
retained for diagnostics but never promoted by cache presence. Publish atomically from private
temporary paths under a per-key lock; another push may wait only within its deadline. Apply a
.kiro/specs/pre-push-safety-net/tasks.md
+2 -1
View File
@@ -38,12 +38,13 @@ started with group 1 (tasks 1.1–1.4); each task remains pending until independ
- _Boundary: Evaluation harness_
- _Requirements: 6.1, 6.3, 8.3, 8.4_
- _Verified 2026-09-12:_ independent review APPROVED; 902 tests passed, 9 skipped, then 34 focused tests passed after a diagnostic-only correction. Fresh offline validation opened 506,502 bytes and retained all 11 cases/9 snapshots: 7 cases input-ready, 4 with named missing prerequisites. See `benchmarks/push/inputs.json` and `input-validation.json`. Ghost checkout/reviewed labels and independent per-capability PHP/Python populations remain missing; every capability stays experimental. No detector run or admission claim.
- [ ] 1.3 Make the baseline and push scorer fail honestly
- [x] 1.3 Make the baseline and push scorer fail honestly
- Retain unmatched targets and unanswered queries in the declared population; require a queried witness before labeling a target transitively detected.
- Emit actual vulnerable/fixed outcomes beside rank, selected mode, engine/facts/scope and elapsed stage costs. Freeze a versioned workload/profile before measuring.
- Done when a deliberately missing target and failed query cannot improve reported recall, while a verified transitive positive is counted with its witness.
- _Boundary: Evaluation harness, repository benchmark integration_
- _Requirements: 6.2, 6.3, 6.4, 6.5, 7.3_
- _Verified 2026-09-12:_ independent review APPROVED; 925 tests passed, 9 skipped. Parent reproduced the frozen profile and actual VAmPI static rerun (15,942 pinned bytes, all 3 targets retained, SQLI unmatched, no transitive/detection claim, exit 1). Empty-run scoring retains 11 unresolved cases and explicitly reports unmeasured/0 recorded cases. Historical baselines remain unchanged with a caveat; new scorer consumes associated execution/review evidence and does not implement replay or admission.
- [ ] 1.4 Establish typed push and engine integration contracts
- Add importable push components and validated configuration for comparison base, shared deadline/cancellation allowance, local cache limit, advisory/blocking and separate coverage policy.
- Define ownership of update/comparison, change context, selected/deferred scope, graph identity/lease and separate finding/coverage/push statuses as described in design.