marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

learn: experiment fill -- fill a rule experiment's response cache with a hard ceiling, dry-run estimate

Browse Source 
Implemented with Codex; review fix: the bounded client keeps the configured
thinking setting so a fill answers as the run would.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-03 12:35:27 +02:00
co-authored by Claude Opus 5.5
parent 212a391872
commit 8d14b0c13f
6 changed files with 545 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/decision-engine.md
+26
View File
@@ -252,6 +252,32 @@ The reading is on [Where we stand](where-we-stand.md#2-our-approach-to-detection
| `ousast learn curve --program ID` | the metric over memory size on a fixed candidate subset |
| `ousast learn audit-leaks` | how well each signal alone separates the sides of a pair (no model) |
| `ousast learn experiment register\|units\|run\|analyse MANIFEST` | A/B experiments from a pre-registered manifest (`plane/experiments/<id>.yaml`): record its digest, freeze the units (no model), run the arms paired (arm A may replay from the cache at $0), analyse (repository bootstrap, exact McNemar, two looks, the adoption verdict) |
| `ousast learn experiment fill MANIFEST --budget-usd X` | Fill missing responses for a registered `kind: rule` experiment's frozen units before its zero-cost run; `--dry-run` inspects and estimates without calls or writes |
`experiment fill` accepts `--memory URL`, `--profile static|plane` (default `static`),
`--manifest-model PATH` (default `plane/models/deepseek-flash.yaml`) and `--out PATH`.
Freeze, commit and register the units first. Fill uses the incumbent programs and the
same retrieval, sampling and cache keys as replay. The first missing request is the
smoke call; a successful smoke response is kept. Any provider failure stops the fill.
The JSON report on stdout gives each family's units, distinct requests, initially
cached and missing responses, newly filled responses and still-missing responses,
plus total client calls and metered USD. `--out` also saves it, except during dry-run,
which writes nothing. Exit codes are 0 for completion or dry-run, 1 for a provider/cache
failure, 2 for invalid setup and 3 for budget exhaustion. Resume a partial fill with a
new invocation and budget; that budget covers only the new invocation, including smoke.
Dry-run prices each missing prompt using the program's input-token estimate and the
family's mean cached completion-token usage, falling back to **300 output tokens**.
It reports potential retries separately as `conditional_retries`: an unseen answer
may need one parse retry, whose necessity cannot be known without calling the model.
Known invalid cached answers include their retry requests in the inventory.
Fill reserves at least **$0.01** and the largest observed call cost before each call.
For the production client it also reserves a conservative input-byte token bound plus
the program's enforced output-token limit; hidden provider retries are disabled.
Consequently a small budget may stop before making any call, and a fill may finish
with unused budget. Calls without token usage stop as failures, never as free calls.
The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](ops/ax/README.md)).
The harvests that produced the plane signals ran as plane Runs generated by