marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

learn: dry-run estimate from measured cached-response cost; docs: exp-004 result; tidy instrument check

Browse Source 
The estimator priced all input at the full rate (\$5.58 estimated vs \$1.24
spent); it now uses the family mean billed cost of cached responses when usage
is recorded. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-03 14:23:36 +02:00
co-authored by Claude Opus 5.5
parent b50f7afccb
commit 6d9850578e
4 changed files with 152 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/decision-engine.md
+43 -5
View File
@@ -150,7 +150,7 @@ Four experiments are registered:
0.875 against 0.906 and an interval spanning zero
(`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`).
- exp-003 compared two BLOCK rules on the same scores. It was **inconclusive**. Neither rule flags anything.
- exp-004 tests one pooled BLOCK threshold with a per-family floor. It is registered and has not run.
- exp-004 tested one pooled BLOCK threshold with a per-family floor. It was **inconclusive**.
### exp-003: precision-bound blocking (2026-10-03)
@@ -216,8 +216,44 @@ Calibration holds only for untrusted destination.
There arm A still finds no BLOCK point that reaches the bound.
Record: `benchmarks/experiments/exp-003-precision-bound-blocking/result-complete-cache.json`.
### exp-004: pooled block gate (2026-10-03)
The experiment tested one pooled BLOCK threshold across families.
Each family needed at least 10 clean flags to block.
Each family also had its own quiet threshold.
The run evaluated 501 units across 220 repository groups.
Of these, 390 units were paired.
The cache fill cost $1.238223 for 2,073 calls.
The replay cost $0.0 and made 0 client calls.
The verdict was **inconclusive**.
No fold found a `t_block` threshold.
The instrument check confirmed real scores.
The pooled top 30 contained 27 positives.
The first negative appeared at rank 12.
The bound needs 73 clean flags without an error.
A within-pair win means the vulnerable side scored above its fixed counterpart.
| Family | Within-pair wins | Pairs |
| --- | --- | --- |
| access control | 22 | 28 |
| deserialization | 18 | 25 |
| injection | 30 | 53 |
| output encoding | 18 | 28 |
| path | 25 | 29 |
| untrusted destination | 18 | 32 |
Precision at the top of the ranking now limits blocking.
Data volume alone cannot resolve this.
The next levers are graph evidence in the prompt and the combiner.
Records: `benchmarks/experiments/exp-004-pooled-block-gate/record.json` and `benchmarks/experiments/exp-004-pooled-block-gate/result.json`.
### exp-004: pooled block gate with a family floor (registered, not run)
This registration describes the setup before the run reported above.
The maintainer amended the BLOCK rule on 2026-10-03 (requirements, Req 6.4).
Precision is now shown across all families together, not per family.
Each family's score is first mapped by its own Platt fit, on training folds.
@@ -230,8 +266,8 @@ No held-out label moves a threshold.
- **Primary metric:** the pooled held-out Wilson lower bound of flagged precision, paired units, before the floor.
- **Decision:** reject below point precision 0.95; adopt if the bound reaches 0.95 and a family passes the floor.
- **Units:** frozen at run time, from the memory after the negatives harvest.
- **Cost:** the run replays the cache. The new memory needs a paid cache fill first.
- **Units:** frozen from the memory after the negatives harvest.
- **Cost:** the run replayed the cache after a paid cache fill.
Manifest: `plane/experiments/exp-004-pooled-block-gate.yaml`.
@@ -267,8 +303,10 @@ which writes nothing. Exit codes are 0 for completion or dry-run, 1 for a provid
failure, 2 for invalid setup and 3 for budget exhaustion. Resume a partial fill with a
new invocation and budget; that budget covers only the new invocation, including smoke.
Dry-run prices each missing prompt using the program's input-token estimate and the
family's mean cached completion-token usage, falling back to **300 output tokens**.
Dry-run uses each family's mean billed cached response cost when recorded usage exists.
Model prices account for cache-hit input, cache-miss input, and output tokens.
Without cached usage, it uses the program's input-token estimate and **300 output tokens**.
Each family's `estimate_method` reports `measured` or `estimated`.
It reports potential retries separately as `conditional_retries`: an unseen answer
may need one parse retry, whose necessity cannot be known without calling the model.
Known invalid cached answers include their retry requests in the inventory.