docs: audit against v2.0.0; deployment guide (local ax and a separate Kubernetes cluster)

Every README and docs page checked against ousast --help, the code and the committed records. New docs/deployment.md (in the nav): what runs where today, this host's setup (up.sh, doctor, smoke Run, footprint), step-by-step guidance for a separate Kubernetes cluster with implemented vs planned stated per item (no cluster deployment has been made; kube context and doctor's registry check are hard-coded in doctor.py, egress.py, router.py), and what the plane/ manifests need changed. Decision-engine pages state slice 2 (six families, pooled and within-pair AUC, BLOCK offered nowhere, canary below 0.9 for three families, positives-only inflation) and that the engine is not adopted; exp-002 registered, not run, no result quoted. Fixed: model/judge.py reference (now model/pipeline.py), missing-key behaviour, .env reference (OPENROUTER_EMBEDDING_MODEL, store variables), docs extra, plane memory-normalise and ousast index listed, ax README's cluster table made precise. RELEASE_NOTES gets an Unreleased entry.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 16:03:46 +02:00
co-authored by Claude Fable 5.1
parent a90e570f48
commit 5aac7eac36
16 changed files with 448 additions and 53 deletions
+52 -5
View File
@@ -61,9 +61,55 @@ sides of a pair by construction: `2026-10-01-decision-engine-leak-audit/`); fram
| Canary agreement on re-run | 0.83 (30 candidates) |
| Spend | $0.91 metered at list price for DeepSeek, $0.0040 for embeddings; $0.0032 per evaluated candidate |
The other evaluable families are not evaluated yet. Leave-one-source-out, leave-one-framework-out
and learning curves, experiments, integration into scan and pre-push, adoption, and the one-time
v3 check are still ahead.
## Second slice: six families (2026-10-02)
Record: `benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json` (counts only).
After the advisory-fix harvest the memory holds 904 static examples (724 before). One program was
compiled per evaluable family (injection recompiled, since the memory changed) and evaluated on
that family's own candidates, out of repository as above: compile split never evaluated, outer
grouped 5-fold, calibration cross-fitted, operating points nested, 95% repository-cluster
bootstrap intervals (2,000 resamples); input profile v1, priors off, k = 5.
| Family | Candidates (groups; positive) | Pooled AUC of `s` (95% CI) | Within-pair AUC (paired groups) | ADVISORY recall / precision | Canary agreement |
| --- | --- | --- | --- | --- | --- |
| access control | 37 (22; 23) | 0.96 (0.90 - 1.0) | 0.93 (14) | 0.91 / 0.91 | 0.83 |
| deserialization | 46 (30; 30) | 0.90 (0.82 - 0.97) | 0.81 (16) | 0.83 / 0.76 | 0.93 |
| injection | 110 (29; 58) | 0.77 (0.67 - 0.88) | 0.72 (22) | 0.88 / 0.57 | 0.77 |
| output encoding | 44 (27; 29) | 0.91 (0.82 - 0.97) | 0.82 (15) | 0.86 / 0.81 | 0.90 |
| path | 35 (21; 21) | 0.89 (0.79 - 0.97) | 0.83 (14) | 0.86 / 0.69 | 0.90 |
| untrusted destination | 62 (37; 38) | 0.85 (0.76 - 0.94) | 0.72 (18) | 0.84 / 0.73 | 0.87 |
- **BLOCK is offered for no family.** Calibration holds only for output encoding (ECE 0.045), and
there no fold reaches the required precision (`precision_unreachable`: no fold reaches a Wilson
lower bound of 0.95). For the other five families calibration does not hold (ECE 0.071 to 0.119).
- **The pooled AUC is the more favourable number.** Every family's pool includes candidates that
carry only a positive label, with no fixed-side counterpart (7 to 19 per family; there are no
single-label negatives), so the pooled AUC is inflated by positives that never had to be told
apart from their own fix. The within-pair AUC over the paired groups, 0.72 to 0.93, is the
stricter reading (`paired_groups_check` in the record).
- **Re-run agreement (canary, 30 candidates per family) is below 0.9 for three families:**
injection 0.77, access control 0.83, untrusted destination 0.87.
- **One memory-size point for injection** (paired: the same program, candidates and folds, with a
retrieval memory of 724 vs 904 examples): AUC 0.76 vs 0.77, within the intervals. No injection
example was added, so the new examples reach an injection prompt only as contrast.
- **Spend:** $2.88 metered at list price for DeepSeek under an $8 ceiling, $0.0009 for
embeddings; $0.0027 to $0.0032 per evaluated candidate. The access-control evaluation replayed
from the response cache at $0 with identical predictions.
- **An instrument fix is recorded:** compiled demonstrations ignored the near-duplicate rule
(cosine >= 0.98) until the pre-call boundary assertion stopped the untrusted-destination
evaluation; demonstrations are now swapped or dropped like retrieved examples, and the stopped
part was replayed after the fix.
**Not adopted.** Still ahead: leave-one-source-out, leave-one-framework-out, the remaining
learning-curve points (memory size 0/25/50%, number of sources), integration into scan and
pre-push, adoption, and the one-time v3 check.
## Experiments
Changes to prompts, models, pass counts or features are compared by controlled experiments, so
nothing is adopted on one run's number. One is registered and has not run: exp-002 (variance
reduction of the score). No result exists; this page carries one only once a record under
`benchmarks/measurements/` does.
## Commands
@@ -81,5 +127,6 @@ v3 check are still ahead.
| `ousast learn audit-leaks` | how well each signal alone separates the sides of a pair (no model) |
The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](ops/ax/README.md)).
The harvest that produced the plane signals runs as plane Runs generated by
`ousast plane harvest` (`benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json`).
The harvests that produced the plane signals ran as plane Runs generated by
`ousast plane harvest` (`benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json`
and, for the advisory-fix pairs, `2026-10-02-decision-engine-harvest-advisory/record.json`).