Maintainer decision 2026-10-03: BLOCK precision is shown pooled across
families. Req 6.4 (amended; 7.4 annotated): one t_block on a per-family
Platt-normalised score (calibrate.fit_map, training folds, inner
cross-fit), Wilson 95% LB >= 0.95 over all families' training flags; a
family blocks only with >= F = 10 held-out flags and zero errors, else
advisory; t_quiet per family keeps >= 90% of training paired positives.
evaluate: normalised, pooled_threshold, quiet_threshold,
pooled_block_gate, family_floor, apply_floor. experiments: rule
pooled_block_gate (normalise: platt, floor, recall), decide_pooled on the
pre-floor pooled held-out LB, floor table, quiet_recall and after-floor
totals in analyse_rule; exp-003's analysis is unchanged.
plane/experiments/exp-004-pooled-block-gate.yaml: procedure and decision
rule registered now; units frozen at run time from the post-harvest
memory (units_status pending until then).
Tests (tests/test_learn_rules.py): pooled threshold where no family alone
reaches the bound, no held-out label moves a threshold or the map, a
family with one held-out error cannot block, t_quiet and no blind
silencing, the pooled verdict, the exp-004 manifest.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Kind proofs: records 01 and 02. Task 2.6 (receiver removal) is paused while the
safety net and the decision engine come first. deployment.md conflicts resolved
in the plain-language style; corrects the RuntimeClass claim (Substrate runs
runsc inside its worker pod).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Live on kind with the runner image that carries 2.2
(ousast-runner@sha256:d2a55293...; templates and validation-46 Tasks repinned,
128 lines; kind-images.json updated): the smoke Run on store delivery finished
in 26 s and 24 s; the tar was PUT to runs/ax-e2e-smoke/facts/output.tar
(10240 B) and collected 2 s later; read back from the store its members
(facts.json 451 B, summary.json 238 B, units.jsonl 291 B) equal the extracted
files; facts.json, summary.json and done.json (149 B) re-put, state.json
(350 B) mirrored; a rerun from an empty results root seeded state.json from the
store and finished in 2 s without resubmitting. The actor's egress policy read
back live: one tlsPassthrough rule on 443 naming the store host, nothing else.
`ousast plane status ax-e2e-smoke` printed the table each time. Record:
benchmarks/measurements/2026-10-02-k8s-02-store-delivery/record.json with the
smoke outputs, store read-back and policy beside it. Task 2.6 (receiver
removal) is paused by the maintainer's reprioritisation.
Gates: pytest 2031 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0. (The first run, at 0.97 GiB free disk, failed 11 tests and errored 5
on the FileStore's 1 GiB free-space refusal; rerun after the disk was freed.)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Rerun records, serve-static and unrolled/secure (same planted commits) with
this build installed from its wheel: 6 replays and 10 hook pushes, all exit 0
in 0.69-1.21 s against the 30 s deadline, input read proven per run. The
planted JS eval in a new file surfaces in the quick tier on the host and, in
one Docker engine run (190.9 s, deadline 300), as two advisory engine findings
classified new/file_added_in_head. The planted Python %-formatted SQL is still
missed (no quick rule, no host engine): recorded as a coverage gap, no rule
edited. Go is named as not covered. The run caught a wrong install hint in
the hook (the package is not on PyPI); corrected here.
Record: benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
`ousast pre-push install|uninstall [REPO] [--force] [--docker]` ships in the
wheel with the hook as package data; it honours core.hooksPath, refuses an
existing hook unless --force (which chains it as pre-push.before-ousast), and
uninstall restores the chained hook. ops/ousast-docker is a hook-callable
script: the repository read-only and the artifact directory at their own
paths, stdin passed through, no network, --memory 3g, the GHCR 2.0.1 image by
default, advisory exit unless blocking. Help text for every pre-push flag; a
first-run guide, settings, skip-reason glossary, CI recipe and full --help in
docs/scanning.md; README tiers 1 and 2 pull the published image. The image
was not rebuilt: 1.1 GB free, below the 4 GB threshold. Requirements 7.4, 9.6.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
--engine inline|background|off (hook: OUSAST_PUSH_ENGINE). background returns
after the quick tier and starts one detached explicit replay of the same base
and head (push/background.py), one at a time per artifact directory; its
artifact lands under <artifact dir>/engine/<repo key>/<head>.json and the next
run prints "Engine result for <head> from the previous push" once. A deadline
miss says so in one line naming the deadline and the background option. No
engine speed work. Requirement 9.5.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ChangeContext records the paths Git reports as added. An entailed,
undischarged operation in an added file, with complete head evidence and no
identical operation in any base answer, is classified new
(file_added_in_head) instead of base_incomplete; admission accepts that reason
as change evidence. Unpaired add/delete pushes and every other unknown
comparison keep their vetoes. An engine finding whose novelty is established
and whose only vetoes are capability vetoes is shown as an advisory engine
finding, not an alert. Requirement 9.4.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Before the engine and inside the same deadline, the shipped quick rules run
over the head side of the changed files (push/quick.py) and keep matches on
changed lines or in changed functions. They print as advisory pattern matches,
labeled apart from admitted alerts, are kept in the artifact's quick_tier with
the files and bytes read, and never admit or block. A new branch with no
remote base runs the quick tier over its last commit and says so. The planted
eval in serve-static's new debug.js now surfaces; the planted Python
%-formatted SQL in records does not (a quick-rule coverage gap, no rule
edited). Requirement 9.3.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
One terminal line per distinct coverage reason (push/explain.py): hex-encoded
paths decoded, per-question sha256 tokens dropped and counted, unknown reasons
still printed; the JSON artifact keeps the raw values. A changed language with
no engine frontend and no quick rules is recorded and printed as "language not
covered: <lang> (...)", in pre-push and in `ousast scan`. The scan report
renders the eight degradations cli.py recorded but never showed, and any
reason without a sentence. The hook forwards OUSAST_COMPARISON_BASE.
Requirement 9.2.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The installed hook exits 0 with one line when ousast (or OUSAST_COMMAND) is
not on PATH, still chaining a prior hook. In Git hook mode without an explicit
--mode blocking, argument/usage errors (an invalid OUSAST_PUSH_DEADLINE, a cache
directory inside the repository, a refused experimental flag) and unexpected
exceptions print one "check did not run" line and exit 0. Blocking mode keeps
its usage errors. Requirement 9.1.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
_execute builds the task's `delivery {put, inputs}` from Delivery, passes it to
router.starter, grants the store's host in the egress policy, awaits
Delivery.delivered (the tar's exact key in the store), then collect()s:
download, extract with the member checks, re-put the declared artifacts, write
done.json; the outcome is the collected summary.json. Three findings from the
tests: a rerun of an unfinished task completed at once on the previous
attempt's tar, so Delivery.reset(task) deletes output.tar and done.json before
every attempt; a worker thread that raised left the run blocked on its queue,
so the worker now turns any exception into a failed task; a consumer may
declare a producer's summary.json without the producer listing it (the loop
Run does), so declared(run) publishes outputs[] plus every artifact a consumer
names. The receiver thread survives behind OUSAST_DELIVERY=receiver for this
group only (no presigned URLs, the http rule, OUSAST_ARTIFACT_URL in the env);
the default path renders no receiver address into any Task. The fake ax PUTs
to the start body's URL and GETs its inputs from theirs; the reconciler tests
run through the S3-shaped fake store and assert the objects, done.json, the
seeded rerun and the receiver mode. reconciler.py: 549 lines (bound 550).
Gates: pytest 2031 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Append task group 17 "Operational readiness (2026-10-02)" with one task per
maintainer-approved item (never block, name every skip, quick tier, new-file
classification, engine honesty and background mode, install/Docker/docs, proof),
add Requirement 9 (9.1-9.6, marked added 2026-10-02, maintainer-approved), and
check 13.4 and the parent of 14 as the audit found them done.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
policy_for(task, workspaces, model, store_host, receiver_host=None) emits one
tlsPassthrough rule on 443 over the sorted union of the memory store's host,
the bound Workspaces' https Git hosts and the bound Model's egress hosts. There
is no http rule: an http Git repo is refused by name (it could not be reached),
a store given as an address is refused naming S3_ENDPOINT (the gateway allows
hostnames only). receiver_host adds the old receiver's http rule for the path
group 2 keeps until 2.6; the reconciler passes it by keyword (passing the
receiver's address positionally would have raised outside the worker's try and
hung the run on the queue -- 2.4 hardens the worker). Tests: the store-only
policy, the Model-bound policy, the two refusals, the receiver exception, and
every committed template under plane/tasks/ rendering exactly the store plus,
for verify and roles, the provider host.
Gates: pytest 2027 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
StartGate.offer validates an optional `delivery` beside the credentials: https
URLs to a hostname (never an address, no userinfo; AX_RUNNER_INSECURE_DELIVERY=1
admits http in unit tests only), names ^[A-Z][A-Z0-9_]*$, a 400 that names the
field and echoes no URL. The runner PUTs its output tar to delivery.put and
GETs each declared input from delivery.inputs[NAME] (deliver() gains a method;
log lines and DeliveryError carry the URL without its query, so no presigned
signature reaches a log, a marker or a summary). OUSAST_ARTIFACT_URL/DIAL stay
honoured until task 2.6 so both paths run in the same build; the completion
marker and the retry-only boot are unchanged, and a re-start brings a fresh
URL for the retry. Router.start_task carries `delivery` in the body and
redacts its URLs like credential values. Tests: the gate's acceptance and
refusals, an end-to-end PUT/GET against the S3-shaped fake with no URL in any
log or file, the failed-PUT-then-retry-only boot, deliver() with PUT.
Gates: pytest 2024 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
src/openultrasast/plane/delivery.py: Delivery(store, run, base) mints a presigned
PUT for runs/<run>/<task>/output.tar and a GET per declared input
(runs/<run>/<producer>/<artifact>), expiry OUSAST_TASK_TIMEOUT + 600 s;
delivered() asks the store for the exact key; collect() downloads, extracts
under the run directory with the receiver's member checks (no symlink, hard
link or escape; nothing written on a rejection), re-puts the declared outputs
as single objects and writes done.json; put_input() for inputs no task
produces; save_state()/load_state() mirror state.json. The reconciler opens the
store first (no fallback; the CLI reports a store that cannot open and submits
nothing), mirrors state.json after every change and seeds a rerun with no local
state from the store (the 2.1 bullet's state mirroring is what gives the new
module its importer). The reconciler crossed 500 lines here (517) rather than at
2.4 as 1.6 projected: the bound moves to 550 with the reason in the assertion,
as 1.6 decided. tests/plane_fake_store.py is the S3-shaped fake (objects in a
dict, presigned URLs on a local server); tests/test_plane_delivery.py covers the
body, expiry, polling, rejection, declared-only re-put, state seeding and that
no URL reaches a log. Module-audit manifest regenerated (148/5, no orphan).
Gates: pytest 2020 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
benchmarks/measurements/2026-10-02-k8s-01-profile/record.json with the live
evidence of group 1: `ousast plane doctor --profile kind` passes every line
(exit 0, 1.4 s; the store check reaches the bucket: verify 0.62 s, probe object
24 B); `--profile k3s` fails on the unreachable context and on the images file
the release has not published yet (two expected lines; the namespaces are not
checked behind an unreachable context); the literal scan has 0 offenders (23
before 1.3) and fails naming workspaces.py:37 when a literal is reinserted;
up.sh reran idempotently in 38 s and the smoke Run finished in 24 s on the
rebuilt image (facts.json 451 B, summary.json 238 B); the placement probe says
`any`; the reconciler bound numbers of 1.6. Doctor, literal, smoke and up.sh
outputs beside it.
Gates: pytest full suite on this tree's src/ and tests/ content: 2010 passed,
42 skipped, exit 0 (the 1.5 run; the 1.8 run reported two module-audit
failures caused by the uncommitted 2.1 module present in the working tree);
ruff check 0; ruff format 0; mypy 0; mkdocs build --strict 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
docs/deployment.md B.1: Substrate's ateom-gvisor worker runs runsc from the
release tarball named in manifests/ate-install/sandboxconfig-gvisor.yaml
(gs://gvisor/releases/nightly/2026-09-02/<arch>/gvisor.tar.zstd, sha256-pinned
per architecture), fetched by the atelet on the node; the node must reach it or
a copied SandboxConfig must name a mirror with the same sha256. The check is the
atelet's fetch line in the worker pod's logs, run by hand on the day k3s exists;
labelled planned until then (design open question 3 stays open).
Gates: docs only; mkdocs build --strict passed with this paragraph present in
the 1.4 gate; no file a test or linter reads changed since the 1.5 gate.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
reconciler.py: 477 lines at 6bd44ca, 489 after 1.2 (the profile parameter of
run() and its docstring). Task 2.4 adds the Delivery wiring (about +25) while
the receiver path is kept for group 2, and 2.6 removes the receiver wiring
(about -20): projected 514 during group 2, 494 after 2.6. The estimate crosses
500 while both paths coexist, so 2.4 moves tests/test_plane_reconciler.py's
bound to 550 with the reason in the assertion message; 2.6 records the final
count. Numbers in benchmarks/measurements/2026-10-02-k8s-01-profile/record.json
(record 01, drafted here, completed by 1.8).
Gates: no file a test or linter reads changed since the 1.5 gate (pytest 2010
passed, 42 skipped, exit 0; ruff 0; format 0; mypy 0).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ops/ax/probe-placement.sh scales ousast-pool to one worker, applies probe-pool
(1 x 2 CPU / 3Gi, the already-built ateom-gvisor image, no ko build), runs one
model-free repo-facts Task requesting 2560Mi (limit 3Gi) and one requesting
256Mi, records `kubectl ate get workers -o json` after each resume, and removes
everything it created. Live on kind: both actors landed on the 1536Mi
ousast-pool worker (status.allocated.actors=1 there, 0 on probe-pool) ->
placement: any. design.md section 4 records the outcome and the fallback group
5 must take (an engine atespace with engine-pool as its only pool if Substrate
scopes pools by atespace; else a plain Kubernetes Job on k3s). Evidence under
benchmarks/measurements/2026-10-02-k8s-01-profile/probe/.
Gates (tree identical to 1.4 in src/ and tests/; ops/ gains the probe script,
which the literal scan covers): pytest 2010 passed, 42 skipped, exit 0; ruff
check 0; ruff format 0; mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ops/ax/up.sh reads kube_context, registry, pools.default and the images path
with `python -m openultrasast.plane.profile --print`, writes the runner digest to
ops/k8s/profiles/kind-images.json instead of ~/.cache/.../runner-image, renders
ops/ax/workerpool.yaml.tmpl from the profile's pool (POOL_NAME/REPLICAS/CPU/
MEMORY, WORKER_IMAGE), and keeps an existing pool and an ax CLI newer than its
checkout instead of rebuilding them (ko and go install would refill several GB
of Go cache on this disk). smoke-run.sh takes the runner from `--print
images.runner` and runs the checkout's code with OUSAST_PYTHON. repin_templates
reads the JSON images file (runner key; the one-line pin still reads). `--print
images` returns the file path (the first live rerun fell over this). Templates
and the generated validation-46 Tasks re-pinned to the rebuilt image (128
lines, one digest).
Live: up.sh reran idempotently on kind in 38 s (every step kept except the
runner build); smoke-run.sh done in 24 s on the new image, facts.json 451 B,
summary.json 238 B delivered (record: benchmarks/measurements/2026-10-02-k8s-01-profile/).
Gates: pytest 2010 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0; mkdocs build --strict 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
tests/test_plane_literals.py scans src/, plane/, ops/ (minus ops/ax/up.sh and
ops/k8s/profiles/) and docs/ for the kind context, the kind registry, the kind
network's gateway, a loopback address with a port and port-forward; router.py
keeps the last two, a line tagged ax-tunnel or loopback is allowed, and a
committed template's digest-pinned image is accepted only while it names the
kind profile's registry (checked by a third test). A self-check reinserts each
literal into a fixture and sees it caught. The runner's health URL and the Joern
session's query URL are tagged loopback; docs/deployment.md and ops/ax/README.md
stop naming this host and describe the kube context and the registry as the
profile's (what 1.2 made configurable).
Gates: pytest 2009 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0; mkdocs build --strict 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
doctor(profile) checks the profile's kube context, the two namespaces behind a
reachable context only, the images file's runner reference in the profile's
registry (a local registry over its v2 API, a remote one via docker manifest
inspect) and opens the memory store; its first line reports every configured
address (Req 1.2). Egress(executable, context), open_router(context, url=...),
receiver_cluster_ip(context) and Receiver(..., context) take the profile's
context and build none; generate.increment/render and harvest take the profile's
atespace. `ousast plane doctor|run|workspaces|harvest --profile`. The test fakes
(kubectl-ate, kubectl, ax) record the context they received and the tests assert
it is the profile's. grep "kind-" src/openultrasast/plane: no match.
Gates: pytest 2006 passed, 42 skipped, exit 0; ruff check 0; ruff format 0; mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ops/k8s/profiles/kind.toml and k3s.toml hold the values that differ per cluster
(exec, kube_context, registry, images, router_url, ax_server, memory, atespace,
snapshots_bucket, image_pull_secret, pools); kind-images.json carries the digest
pin ops/ax/up.sh wrote. load_profile() reads OUSAST_PLANE_PROFILE (name or path,
default kind), applies OUSAST_<FIELD> and OUSAST_POOL_<NAME>_<FIELD>, and refuses
an unknown key, a credential-shaped key or value, an exec outside local|remote
and an images file without digest pins. `python -m openultrasast.plane.profile
--print FIELD` serves shell scripts. The module-audit manifest is regenerated
for the new module (counts 147/5, no orphan).
Gates: pytest full suite 2002 passed, 1 failed (the module-audit manifest, fixed
by regenerating it; tests/test_module_audit.py re-run exit 0), ruff check 0,
ruff format 0, mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Legacy inventory 2, the six STATUS NOTE items (the archive README row landed with the R2 commit).
ai-service-plane: 11 of 11 tasks checked, spec.json phase set to implementation-complete as 5b96199
did for four other specs. contributor-scan: dated note in requirements.md -- paused since 2026-09-23,
27 of 51 tasks, v1.2.0-alpha.1 NO-GO, remaining tasks wait on pre-push M2/M3. finding-feedback-loop:
dated note -- dormant, design unapproved since 2026-09-09, depends on the paused spec. The steering
overview's spec table carries the same three states, its leftover 2026-09-09 module-count line is
removed and the collected-test count is the measured 2,023. docs/memory-and-detection.md names the
archive path of model_sinks, the one stale script name left in docs after 067adb9.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer: the whole system moves to a Kubernetes cluster soon. Seven host-bound
assumptions become cluster resources or configuration; the engine becomes a
plane task on a larger pool; a delta Run command serves the pre-push use; every
step is proven on kind first.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer decision 2026-10-02 (no symlink). AGENTS.md, README, the two spec
docs, the kiro-spec-batch skill and the provenance ignore list are repointed;
the OpenCode plugin package.json stays under .opencode.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reject B. 160 paired units (injection 119, access_control 41; 56 repositories),
both arms, two replicates each, no early stop. Canary agreement A 0.875 -> B
0.906 (CI [-0.03, +0.10], spans; B meets 0.90). Within-pair AUC A 0.815 -> B
0.741 (CI [-0.13, -0.02], against B). AUC -0.02 (against), ADVISORY recall
-0.06 and precision +0.01 (span), usd per candidate A $0.0021 -> B $0.0051
(against). Priced usage A $0.68 / B $1.63; balance $16.72 -> $16.04 at the
run's end (lags). Counts only; task 7.4 marked done.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Registration rows at pin fbc3220 in s3://sast-memory; in-repo records under
benchmarks/experiments/<id>/registration.json. exp-002's units could not be
frozen: every example row in the store carries null plane-only instruments
and learn.examples.load_examples raises on them (unchanged code since
a90e570) -- a store-side change to diagnose before any paid run.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/experiments.py: YAML manifests under plane/experiments/<id>.yaml,
digest-checked at register, run and analyse time; `experiment`,
`arm_outcome` and `experiment_result` rows; paired arms in a seeded
repository order with a per-unit coin; arm A replay-only at $0; canary
replicates under a fresh cache salt; repository-cluster paired bootstrap,
exact McNemar, two O'Brien-Fleming looks, adopt/reject/equivalent/
inconclusive. `ousast learn experiment register|units|run|analyse`.
Program: `ProgramSpec.sampling = retrieval_ensemble` (k samples at
temperature 0 over k disjoint retrieved example sets; default unchanged);
`Caller.salt`; `retrieve(exclude=)`. verify: OUSAST_VERIFY_CALLERS=off.
Manifests exp-001-known-callers (plane arms, units pending) and
exp-002-retrieval-ensemble (slice-2 incumbents, ceiling $4) -- not run.
18 scripted tests; module audit regenerated (151 modules).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Decision-engine task 6.7 (partial). After the advisory-fix harvest (memory 724 -> 904 static examples),
one compile per evaluable family and an outer grouped 5-fold evaluation of that family's own candidates,
input profile v1, priors off, k = 5, cross-fitted Platt, nested operating points (repository-cluster
bootstrap 95%):
family n AUC(s) ADVISORY R / P 'vulnerable' P / R ECE canary
injection 110 0.77 [0.67, 0.88] 0.88 / 0.57 0.84 / 0.36 0.093 0.77
untrusted_dest. 62 0.85 [0.76, 0.94] 0.84 / 0.73 0.96 / 0.66 0.071 0.87
path 35 0.89 [0.79, 0.97] 0.86 / 0.69 0.89 / 0.76 0.119 0.90
output_encoding 44 0.91 [0.82, 0.97] 0.86 / 0.81 0.94 / 0.55 0.045 0.90
deserialization 46 0.90 [0.82, 0.97] 0.83 / 0.76 0.93 / 0.83 0.095 0.93
access_control 37 0.96 [0.90, 1.00] 0.91 / 0.91 1.00 / 0.61 0.076 0.83
BLOCK offered for none: calibration fails everywhere except output_encoding, where no fold reaches a
Wilson lower bound of 0.95. Within paired groups (a function and its own fix) AUC is lower, 0.72-0.93:
every single-label group in the new families holds positives only. Injection learning-curve point,
paired (same program, candidates, folds): memory 724 -> 904, AUC 0.76 -> 0.77 (no injection example
was added). $2.88 metered DeepSeek (ceiling $8; balance 17.73 -> 16.73), $0.0009 embeddings.
Instrument fix: compiled demonstrations now follow the near-duplicate rule (cos >= 0.98) that retrieval
and the pre-call assertion apply; the assertion had stopped the untrusted_destination evaluation.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The plane's memory store no longer names one vendor. `MinioStore`/`SdkClient`/
`minio_settings`/`MINIO_KEYS` become `S3Store`/`S3Client`/`s3_settings`/`S3_KEYS`;
the URL scheme is `s3://<bucket>[/<prefix>]` (`s3://` alone uses `S3_BUCKET`); the
settings are `S3_ENDPOINT` (a full URL), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`,
optional `AWS_SESSION_TOKEN` and `S3_REGION`. The client is boto3 (extra `s3`,
imported lazily): SigV4, path-style addressing, SDK checksum trailers off, paginated
listing, versions and delete markers listed and deleted by version (`purge`, used by
the contract tests' teardown so the bucket keeps nothing of a run).
Behaviour is unchanged: the store never configures its bucket and verifies
versioning, the `runs/` expiry rule, S3 Select and tag reads at startup; no local
fallback; every row carries every queryable field with "" placeholders. The admin
hints read `--endpoint-url "$S3_ENDPOINT"`. The fake client still reproduces RustFS's
two Select failures (unbound leading-row field, null-typed column).
Docs: rustfs.md stays the tested server's setup page (policy JSON unchanged, env
names updated, any S3-compatible store with versioning, lifecycle, tags and Select
works); memory.md, ops/ax/README.md, README.md, steering and the two current spec
designs use the S3 naming. Retired specs and committed measurements are untouched.
Real server (RustFS, OUSAST_MEMORY_TEST_S3=1): test_plane_memory + test_learn_compile
75 passed, 1 skipped. Store copy of the live local store to a temporary prefix: every
count matched (776 row objects, 2982 rows, 22 facts, 788 excerpts, 2601 responses,
2 programs, 1793 index entries), 4190 versions purged afterwards.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The 97 advisory-fix pairs (194 sides) harvested for the decision engine: verify a/b + agree on the 34
untrusted_destination sides and model roles on every side as plane Runs on ax ($0.38 metered, task
budgets $1.49, ceiling $5; DeepSeek balance 17.88 -> 17.73), engine features on the host (frozen
7b17eac, one container, --memory 3g: 54 ran, 58 asked no question, 82 no language model), feature
records for all 194 sides, static memory rebuilt with the merged harvest units: 724 -> 904 examples.
No unit was skipped for the 20 KB inline limit.
Instrument fix: SideReader read only the first pair-kind source, so the first rebuild left all 194
advisory sides `unread` (0 examples). It now reads every pair-kind source; test added.
Leak audit: the static profile flags only engine signals and facts.function_lines (withheld by v1);
the plane profile newly flags verify.flag_b and ms.operation (plane-only, not in any v1 prompt).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A4: roadmap-2026-09-04.md moves to .kiro/archive/steering/ so it no longer
loads as steering (roadmap.md and safety-net.md replaced it); roadmap.md's
pointer follows it. overview.md's spec table is recounted from tasks.md and
spec.json (adds ai-service-plane, harnessx-removal, learned-decision-engine,
pre-push-safety-net; drops the stale "active one"), and the module line now
cites the regenerated audit: 150 modules, 145 load-bearing, 5 standalone,
0 orphaned; 1,964 tests collected.
A5: a dated status note at the top of the requirements (or brief) of 16
specs: retired, superseded, dormant or complete, and for
model-grounded-detection the removal of judge/calibrate/execution (closing
the Req 10 execution-tier option). spec.json phase corrected to
implementation-complete where every task is checked (model-grounded-
detection, propose-adjudicate-prove, tree-sitter-overlay-extra,
corpus-seeded-mechanisms, pair-corpus-honesty); openrouter-sast-harness gets
a spec.json with status retired. Nothing deleted, no spec moved.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
6.2: the 724 static examples' 616 distinct excerpts embedded with openai/text-embedding-3-small via
OpenRouter, 198,819 tokens, $0.0040 metered (ceiling $1).
6.6, family injection only, memory = every family (at most 2 contrast examples per prompt), input
profile v1: smoke fixed k = 5 (repeat prefix-cache hit share 0.96); one compile (program c1237a49,
proposed instruction 5, balanced Brier 0.128 on 13 C_val candidates vs 0.199 baseline); outer grouped
5-fold, 110 out-of-repository candidates in 29 groups: ADVISORY recall 0.90 [0.82, 0.97], precision
0.54 [0.44, 0.65], AUC 0.74; BLOCK not offered (ECE 0.064 > 0.05; no fold reached a Wilson lower bound
of 0.95); canary re-ask agreement 0.83. $0.91 metered at list price; DeepSeek balance $18.20 -> $17.90.
Instrument fixes found on the way, each with a test:
- memory build: pair sides join the record stored under their harvest unit pin (--units), and only the
built profile's records are read (static and plane shared a key; 85 -> 724 examples);
- retrieval: stage 1 per side (family / contrast) and at most 2 contrast examples, never padded;
- program: the JSON answer contract is part of the signature, appended to proposed instructions
(DeepSeek JSON mode refused a prompt without "json");
- compile: candidate_families for a one-family compile; compile.toml k = 5 from the smoke rule.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer decision "Neutralise both for v1".
1. Phantom candidates. A pair side's candidates are now only the functions that
side declares (plane/harvest.py `declares`, the excerpt builder's
function_bounds); a labelled function absent from a side is listed under
`absent_on_side` and never asked, and a side declaring none of its verify
family's functions is no verify unit. The recorded harvest units held 58
phantom verify candidates (53 fixed, 5 vulnerable sides). build_features.py
skips label rows whose side does not declare the function and drops their
earlier rows (MemoryStore.drop_rows); rebuilt the pair records from the
recorded signals, no model call: 126 label rows absent_on_side (117 fixed,
9 vulnerable) -- access_control 13, config_secrets 4, deserialization 20,
injection 28, memory 4, output_encoding 16, path 15, untrusted_destination
26 -- so 252 feature rows dropped (1712 -> 1460); no other row changed.
2. Engine withheld. Input profiles in learn/schema.py (`v1` withholds the
engine instrument, `full` withholds nothing); ProgramSpec.inputs defaults to
v1 and compile.toml records it. Neither the engine's state nor its values
reach the candidate, examples, demonstrations or the instruction proposer's
error summary, and the Gower retrieval distance leaves the engine block out.
Stored records are unchanged. design.md Risks records the decision and that
the engine returns only through a pre-registered A/B on assumed-benign and
benign-control negatives.
3. `ousast learn audit-leaks` (learn/leaks.py): per feature and per instrument
state, separation = (pairs the signal alone puts on the right side - on the
wrong side) / pairs, over all pairs and per family; flagged at >= 0.20 over
>= 10 pairs. Counts under
benchmarks/measurements/2026-10-01-decision-engine-leak-audit/ (before and
after the rebuild). After: facts.function_lines (0.80 untrusted_destination,
0.75 config_secrets, 0.26 injection; higher on fixed), verify.turns_mean
0.67 config_secrets, eng.* 0.35-0.40 output_encoding, verify.flag_a/votes/
site_offset and agree.final 0.20-0.30 on 10-12 pair families. Before the
rebuild the phantom fixed sides drove state.source/state.roles/ms.flagged
to 1.0 and state.engine to 0.95 on deserialization.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The pair-corpus engine job finished: 682 of 686 sides (199 ran, 475 read their input
and asked no question, 8 no engine model; the 4 sides of the two family-unknown
pairs were not run). Feature records rebuilt (1712); counts-only record updated.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
