mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-17 12:09:27 +00:00
docs: audit against v2.0.0; deployment guide (local ax and a separate Kubernetes cluster)
Every README and docs page checked against ousast --help, the code and the committed records. New docs/deployment.md (in the nav): what runs where today, this host's setup (up.sh, doctor, smoke Run, footprint), step-by-step guidance for a separate Kubernetes cluster with implemented vs planned stated per item (no cluster deployment has been made; kube context and doctor's registry check are hard-coded in doctor.py, egress.py, router.py), and what the plane/ manifests need changed. Decision-engine pages state slice 2 (six families, pooled and within-pair AUC, BLOCK offered nowhere, canary below 0.9 for three families, positives-only inflation) and that the engine is not adopted; exp-002 registered, not run, no result quoted. Fixed: model/judge.py reference (now model/pipeline.py), missing-key behaviour, .env reference (OPENROUTER_EMBEDDING_MODEL, store variables), docs extra, plane memory-normalise and ousast index listed, ax README's cluster table made precise. RELEASE_NOTES gets an Unreleased entry. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
a90e570f48
commit
5aac7eac36
+52
-5
@@ -61,9 +61,55 @@ sides of a pair by construction: `2026-10-01-decision-engine-leak-audit/`); fram
|
||||
| Canary agreement on re-run | 0.83 (30 candidates) |
|
||||
| Spend | $0.91 metered at list price for DeepSeek, $0.0040 for embeddings; $0.0032 per evaluated candidate |
|
||||
|
||||
The other evaluable families are not evaluated yet. Leave-one-source-out, leave-one-framework-out
|
||||
and learning curves, experiments, integration into scan and pre-push, adoption, and the one-time
|
||||
v3 check are still ahead.
|
||||
## Second slice: six families (2026-10-02)
|
||||
|
||||
Record: `benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json` (counts only).
|
||||
After the advisory-fix harvest the memory holds 904 static examples (724 before). One program was
|
||||
compiled per evaluable family (injection recompiled, since the memory changed) and evaluated on
|
||||
that family's own candidates, out of repository as above: compile split never evaluated, outer
|
||||
grouped 5-fold, calibration cross-fitted, operating points nested, 95% repository-cluster
|
||||
bootstrap intervals (2,000 resamples); input profile v1, priors off, k = 5.
|
||||
|
||||
| Family | Candidates (groups; positive) | Pooled AUC of `s` (95% CI) | Within-pair AUC (paired groups) | ADVISORY recall / precision | Canary agreement |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| access control | 37 (22; 23) | 0.96 (0.90 - 1.0) | 0.93 (14) | 0.91 / 0.91 | 0.83 |
|
||||
| deserialization | 46 (30; 30) | 0.90 (0.82 - 0.97) | 0.81 (16) | 0.83 / 0.76 | 0.93 |
|
||||
| injection | 110 (29; 58) | 0.77 (0.67 - 0.88) | 0.72 (22) | 0.88 / 0.57 | 0.77 |
|
||||
| output encoding | 44 (27; 29) | 0.91 (0.82 - 0.97) | 0.82 (15) | 0.86 / 0.81 | 0.90 |
|
||||
| path | 35 (21; 21) | 0.89 (0.79 - 0.97) | 0.83 (14) | 0.86 / 0.69 | 0.90 |
|
||||
| untrusted destination | 62 (37; 38) | 0.85 (0.76 - 0.94) | 0.72 (18) | 0.84 / 0.73 | 0.87 |
|
||||
|
||||
- **BLOCK is offered for no family.** Calibration holds only for output encoding (ECE 0.045), and
|
||||
there no fold reaches the required precision (`precision_unreachable`: no fold reaches a Wilson
|
||||
lower bound of 0.95). For the other five families calibration does not hold (ECE 0.071 to 0.119).
|
||||
- **The pooled AUC is the more favourable number.** Every family's pool includes candidates that
|
||||
carry only a positive label, with no fixed-side counterpart (7 to 19 per family; there are no
|
||||
single-label negatives), so the pooled AUC is inflated by positives that never had to be told
|
||||
apart from their own fix. The within-pair AUC over the paired groups, 0.72 to 0.93, is the
|
||||
stricter reading (`paired_groups_check` in the record).
|
||||
- **Re-run agreement (canary, 30 candidates per family) is below 0.9 for three families:**
|
||||
injection 0.77, access control 0.83, untrusted destination 0.87.
|
||||
- **One memory-size point for injection** (paired: the same program, candidates and folds, with a
|
||||
retrieval memory of 724 vs 904 examples): AUC 0.76 vs 0.77, within the intervals. No injection
|
||||
example was added, so the new examples reach an injection prompt only as contrast.
|
||||
- **Spend:** $2.88 metered at list price for DeepSeek under an $8 ceiling, $0.0009 for
|
||||
embeddings; $0.0027 to $0.0032 per evaluated candidate. The access-control evaluation replayed
|
||||
from the response cache at $0 with identical predictions.
|
||||
- **An instrument fix is recorded:** compiled demonstrations ignored the near-duplicate rule
|
||||
(cosine >= 0.98) until the pre-call boundary assertion stopped the untrusted-destination
|
||||
evaluation; demonstrations are now swapped or dropped like retrieved examples, and the stopped
|
||||
part was replayed after the fix.
|
||||
|
||||
**Not adopted.** Still ahead: leave-one-source-out, leave-one-framework-out, the remaining
|
||||
learning-curve points (memory size 0/25/50%, number of sources), integration into scan and
|
||||
pre-push, adoption, and the one-time v3 check.
|
||||
|
||||
## Experiments
|
||||
|
||||
Changes to prompts, models, pass counts or features are compared by controlled experiments, so
|
||||
nothing is adopted on one run's number. One is registered and has not run: exp-002 (variance
|
||||
reduction of the score). No result exists; this page carries one only once a record under
|
||||
`benchmarks/measurements/` does.
|
||||
|
||||
## Commands
|
||||
|
||||
@@ -81,5 +127,6 @@ v3 check are still ahead.
|
||||
| `ousast learn audit-leaks` | how well each signal alone separates the sides of a pair (no model) |
|
||||
|
||||
The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](ops/ax/README.md)).
|
||||
The harvest that produced the plane signals runs as plane Runs generated by
|
||||
`ousast plane harvest` (`benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json`).
|
||||
The harvests that produced the plane signals ran as plane Runs generated by
|
||||
`ousast plane harvest` (`benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json`
|
||||
and, for the advisory-fix pairs, `2026-10-02-decision-engine-harvest-advisory/record.json`).
|
||||
|
||||
Reference in New Issue
Block a user