Author SHA1 Message Date
Marius CiepluchandClaude Opus 5.5 e13ce35a75 engine: admit TypeScript (jssrc2cpg, JavaScript taint specs via one alias), .mts/.cts, mixed JS/TS builds one CPG
Implemented with Codex. Not merged: wall-clock deadline tests failed at host
load 47; to be re-checked against main at normal load.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 10:34:50 +02:00
Marius CiepluchandClaude Opus 5.5 20777d8863 chore(audit): module audit gains learn.slice
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 17:04:33 +02:00
Marius Ciepluch 77e47ac7d3 Merge branch 'graph-slice' 2026-10-03 16:58:37 +02:00
Marius CiepluchandClaude Opus 5.5 094f6d226c learn: exp-005 graph slice -- inconclusive (within-pair AUC 0.762 -> 0.746, CI [-0.063, 0.032]); unsure rate 0.29 -> 0.24; fill \$1.59
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 16:58:37 +02:00
Marius CiepluchandClaude Opus 5.5 0acbc7ec41 learn: exp-005 graph slice -- slice in the prompt (opt-in), program-arm fill, manifest and units
Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 15:06:08 +02:00
Marius CiepluchandClaude Opus 5.5 1b2c7993a7 learn: in-function def-use slice from the tree-sitter layer + leak audit (\$0)
slice.py renders source -> step -> sink lines with fixed wording for flow,
no-flow and not-analysable; the prompt option is default-off (byte-identical
default prompt). semantic IR: JS/C declarations count as assignments, augmented
assignment keeps its target (gate, map_gate, pair_gate outputs unchanged).
Audit over exp-004's 501 units: 59 flow, 161 no-flow, 281 not analysable;
has-flow within-pair accuracy 0.52, pooled AUC 0.55 -- no leak. Fixes a
numbering bug that rejected every excerpt with a blank line. With Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 14:56:52 +02:00
Marius CiepluchandClaude Opus 5.5 463fe1503a learn: build_features reads v2 _a/_b scans and TypeScript repository-scan witnesses; --dry-run
Population v2 scans are named <case>--<side>_a/_b.json; the script looked only
for <case>--<side>.json and recorded those examples as engine not-run. The two
repeats combine symmetrically for both sides. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 14:41:55 +02:00
Marius CiepluchandClaude Opus 5.5 2836b78819 merge(learn): exp-004 -- inconclusive, pooled top-30 precision 0.90; experiment fill command; registration fix
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 14:29:22 +02:00
Marius CiepluchandClaude Opus 5.5 6d9850578e learn: dry-run estimate from measured cached-response cost; docs: exp-004 result; tidy instrument check
The estimator priced all input at the full rate (\$5.58 estimated vs \$1.24
spent); it now uses the family mean billed cost of cached responses when usage
is recorded. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 14:23:36 +02:00
Marius CiepluchandClaude Opus 5.5 b50f7afccb learn: exp-004 run -- inconclusive, no pooled t_block; fill \$1.24 of a \$2 ceiling, replay \$0; instrument check recorded
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 14:18:17 +02:00
Marius CiepluchandClaude Opus 5.5 90a2642019 learn: registered() returns the frozen registration, not the first row
A pending-then-frozen registration left two rows and rows[0] could be the stale
pending one, so fill and run refused a properly frozen experiment. Implemented
with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 12:42:39 +02:00
Marius CiepluchandClaude Opus 5.5 8aeca972c0 learn: freeze exp-004 units (501 units, 390 paired, 220 groups) after the negatives harvest; audit gains learn.fill
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 12:36:02 +02:00
Marius CiepluchandClaude Opus 5.5 8d14b0c13f learn: experiment fill -- fill a rule experiment's response cache with a hard ceiling, dry-run estimate
Implemented with Codex; review fix: the bounded client keeps the configured
thinking setting so a fill answers as the run would.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 12:35:27 +02:00
Marius CiepluchandClaude Opus 5.5 212a391872 merge(learn): cache fill and exp-004 registration -- pooled block gate with family floor, Req 6.4
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 09:53:10 +02:00
Marius CiepluchandClaude Opus 5.5 da91ef0dd9 exp-003: supplementary all-units replay after the cache fill; exp-004 registered
Supplementary analysis, not a new decision: exp-003's verdict stands.
The 142 units excluded for an uncached response were filled with the same
six incumbent programs, parameters and request keys, on exp-003's
937-example memory (the harvest's 196 rows and its 2026-10-03 embeddings
left out; a zero-cost replay first reproduced exp-003 exactly: 211
decided, 1056 replayed, $0.582876). Fill: MeteredClient, ceiling $0.99
under the approved $1.00, 697 client calls, $0.331759, no provider error.

Replay on all 353 units (244 paired, 146 groups): 0 client calls, $0.0,
1768 responses replayed; no arm flags any unit, the registered rule reads
inconclusive. Calibration holds only for untrusted destination (ECE
0.0454) and arm A still finds no BLOCK point there.
benchmarks/experiments/exp-003-precision-bound-blocking/result-complete-cache.json

exp-004 registered in s3://sast-memory (manifest 87fceb65..., pin
2ad9c1d, units pending): benchmarks/experiments/exp-004-pooled-block-gate/
registration.json. docs/decision-engine.md: both, briefly.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 09:47:20 +02:00
Marius CiepluchandClaude Opus 5.5 2ad9c1db7a learn(exp-004): Req 6.4 pooled BLOCK gate with a family floor; manifest before units
Maintainer decision 2026-10-03: BLOCK precision is shown pooled across
families. Req 6.4 (amended; 7.4 annotated): one t_block on a per-family
Platt-normalised score (calibrate.fit_map, training folds, inner
cross-fit), Wilson 95% LB >= 0.95 over all families' training flags; a
family blocks only with >= F = 10 held-out flags and zero errors, else
advisory; t_quiet per family keeps >= 90% of training paired positives.

evaluate: normalised, pooled_threshold, quiet_threshold,
pooled_block_gate, family_floor, apply_floor. experiments: rule
pooled_block_gate (normalise: platt, floor, recall), decide_pooled on the
pre-floor pooled held-out LB, floor table, quiet_recall and after-floor
totals in analyse_rule; exp-003's analysis is unchanged.

plane/experiments/exp-004-pooled-block-gate.yaml: procedure and decision
rule registered now; units frozen at run time from the post-harvest
memory (units_status pending until then).

Tests (tests/test_learn_rules.py): pooled threshold where no family alone
reaches the bound, no held-out label moves a threshold or the map, a
family with one held-out error cannot block, t_quiet and no blind
silencing, the pooled verdict, the exp-004 manifest.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 09:40:58 +02:00
Marius CiepluchandClaude Opus 5.5 5b2bd9bb91 merge(learn): negatives harvest -- 98 real fix pairs with surviving functions (79 post-cutoff), paired groups 151 -> 249
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:29:32 +02:00
Marius CiepluchandClaude Opus 5.5 50c93edfb8 learn: 98 advisory-fix pairs whose function survives the fix (advisory-fixes-2)
Real (vulnerable function, fixed function) pointer pairs for the six compiled
families, mined at zero model cost from reviewed GHSA advisories (pip, npm,
maven, composer), newest first: 79 of 98 published after 2026-06-01
(post_cutoff = true). Each pair: one linked one-parent fix commit, <= 4 source
files / 120 lines, the function holding the changed code lines declared once on
both sides with a changed body; every function diff read and kept only when the
named function holds the fix (145 reviewed, 100 kept); 2 dropped by the
reserved-population test; permissive licences only, no repository named
elsewhere under benchmarks/.

New label source advisory-fixes-2; the advisory-fix tests run over both
batches. Memory 937 -> 1133 static examples (98 paired groups added);
positives-only groups stay 99 (vibe-py/dev-php/v2 data gaps), their share of
positive groups in the six families 0.40 -> 0.29. Embeddings: 7 calls,
$0.0014. No chat-model call. Record:
benchmarks/measurements/2026-10-03-decision-engine-negatives-harvest.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:23:42 +02:00
Marius CiepluchandClaude Opus 5.5 6ecf0a304e merge(learn): exp-003 precision-bound blocking -- inconclusive; the Wilson bound needs 73 clean flags, injection has 38 paired positives
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:21:27 +02:00
Marius CiepluchandClaude Opus 5.5 b31c85678f exp-003: result -- inconclusive, no BLOCK rule flags a held-out unit
Zero-cost replay of the six slice-2 programs: meter 0 client calls, $0.0;
211 of 353 units scored, 142 excluded for an uncached response (path lost
all 39). Arm A (calibrated) fails calibration (ECE 0.077 access control,
0.1067 injection); arm B (Wilson LB >= 0.95, nested) chooses no threshold on
any fold, as the registration's power note predicted (73 clean flags needed,
38 paired injection positives); secondary arm C flags nothing either. The
instrument check shows real scores; the zero is the rule's.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:15:22 +02:00
Marius CiepluchandClaude Opus 5.5 7bb97ad0bc exp-003: registration, before the run
Registered in s3://sast-memory at dd7ebc1: manifest 5917f9a1, units 338de998
(353 units, 146 groups). Primary metric, decision rule and the threshold
selection are fixed in the manifest; the power note says arm B cannot reach
a Wilson LB of 0.95 on this data (73 clean flags needed, 38 paired
positives at most).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:03:22 +02:00
Marius CiepluchandClaude Opus 5.5 dd7ebc119b learn(exp-003): decision-rule experiments and the precision-bound BLOCK arm
kind: rule experiments score once from the response cache behind a
zero-ceiling meter (a miss is excluded and counted, never asked) and apply
each arm's BLOCK rule to the same held-out scores. Arm B chooses a per-family
raw-score threshold on the other outer folds only (lowest t with Wilson 95%
LB >= 0.95); a test fails if a fold's own labels move its threshold.
Manifest and frozen units (353 units, 146 groups, 244 paired) for exp-003.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:02:51 +02:00
Marius CiepluchandClaude Opus 5.5 b24eeb15f3 merge(plane-on-kubernetes): groups 1-2 -- cluster profile, no host literals, delivery through the S3 store
Kind proofs: records 01 and 02. Task 2.6 (receiver removal) is paused while the
safety net and the decision engine come first. deployment.md conflicts resolved
in the plain-language style; corrects the RuntimeClass claim (Substrate runs
runsc inside its worker pod).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 07:57:13 +02:00
Marius CiepluchandClaude Opus 5.5 c8b418c34c plane(2.5): kind proof and record 02 -- artifacts through the store, one TLS rule
Live on kind with the runner image that carries 2.2
(ousast-runner@sha256:d2a55293...; templates and validation-46 Tasks repinned,
128 lines; kind-images.json updated): the smoke Run on store delivery finished
in 26 s and 24 s; the tar was PUT to runs/ax-e2e-smoke/facts/output.tar
(10240 B) and collected 2 s later; read back from the store its members
(facts.json 451 B, summary.json 238 B, units.jsonl 291 B) equal the extracted
files; facts.json, summary.json and done.json (149 B) re-put, state.json
(350 B) mirrored; a rerun from an empty results root seeded state.json from the
store and finished in 2 s without resubmitting. The actor's egress policy read
back live: one tlsPassthrough rule on 443 naming the store host, nothing else.
`ousast plane status ax-e2e-smoke` printed the table each time. Record:
benchmarks/measurements/2026-10-02-k8s-02-store-delivery/record.json with the
smoke outputs, store read-back and policy beside it. Task 2.6 (receiver
removal) is paused by the maintainer's reprioritisation.

Gates: pytest 2031 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0. (The first run, at 0.97 GiB free disk, failed 11 tests and errored 5
on the FileStore's 1 GiB free-space refusal; rerun after the disk was freed.)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 07:45:19 +02:00
Marius CiepluchandClaude Opus 5.5 432fd2947e merge(pre-push): operational readiness increment 1 -- never block, never hide, quick tier, new files, background engine, install
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 07:29:17 +02:00
Marius CiepluchandClaude Opus 5.5 2ec8071c26 pre-push(17.7): prove it on the audit's three repositories
Rerun records, serve-static and unrolled/secure (same planted commits) with
this build installed from its wheel: 6 replays and 10 hook pushes, all exit 0
in 0.69-1.21 s against the 30 s deadline, input read proven per run. The
planted JS eval in a new file surfaces in the quick tier on the host and, in
one Docker engine run (190.9 s, deadline 300), as two advisory engine findings
classified new/file_added_in_head. The planted Python %-formatted SQL is still
missed (no quick rule, no host engine): recorded as a coverage gap, no rule
edited. Go is named as not covered. The run caught a wrong install hint in
the hook (the package is not on PyPI); corrected here.
Record: benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 23:02:32 +02:00
Marius CiepluchandClaude Opus 5.5 591818d5fc pre-push(17.6): install, Docker wrapper and first-run docs
`ousast pre-push install|uninstall [REPO] [--force] [--docker]` ships in the
wheel with the hook as package data; it honours core.hooksPath, refuses an
existing hook unless --force (which chains it as pre-push.before-ousast), and
uninstall restores the chained hook. ops/ousast-docker is a hook-callable
script: the repository read-only and the artifact directory at their own
paths, stdin passed through, no network, --memory 3g, the GHCR 2.0.1 image by
default, advisory exit unless blocking. Help text for every pre-push flag; a
first-run guide, settings, skip-reason glossary, CI recipe and full --help in
docs/scanning.md; README tiers 1 and 2 pull the published image. The image
was not rebuilt: 1.1 GB free, below the 4 GB threshold. Requirements 7.4, 9.6.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 22:51:22 +02:00
Marius CiepluchandClaude Opus 5.5 66357ffea1 pre-push(17.5): say when the engine cannot finish, and offer it in the background
--engine inline|background|off (hook: OUSAST_PUSH_ENGINE). background returns
after the quick tier and starts one detached explicit replay of the same base
and head (push/background.py), one at a time per artifact directory; its
artifact lands under <artifact dir>/engine/<repo key>/<head>.json and the next
run prints "Engine result for <head> from the previous push" once. A deadline
miss says so in one line naming the deadline and the background option. No
engine speed work. Requirement 9.5.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 22:31:14 +02:00
Marius CiepluchandClaude Opus 5.5 511de4b75f pre-push(17.4): a file new in head is new, not comparison unknown
ChangeContext records the paths Git reports as added. An entailed,
undischarged operation in an added file, with complete head evidence and no
identical operation in any base answer, is classified new
(file_added_in_head) instead of base_incomplete; admission accepts that reason
as change evidence. Unpaired add/delete pushes and every other unknown
comparison keep their vetoes. An engine finding whose novelty is established
and whose only vetoes are capability vetoes is shown as an advisory engine
finding, not an alert. Requirement 9.4.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 22:24:24 +02:00
Marius CiepluchandClaude Opus 5.5 15999a09e1 pre-push(17.3): quick rules as the fast first tier
Before the engine and inside the same deadline, the shipped quick rules run
over the head side of the changed files (push/quick.py) and keep matches on
changed lines or in changed functions. They print as advisory pattern matches,
labeled apart from admitted alerts, are kept in the artifact's quick_tier with
the files and bytes read, and never admit or block. A new branch with no
remote base runs the quick tier over its last commit and says so. The planted
eval in serve-static's new debug.js now surfaces; the planted Python
%-formatted SQL in records does not (a quick-rule coverage gap, no rule
edited). Requirement 9.3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 22:18:14 +02:00
Marius CiepluchandClaude Opus 5.5 f555dee640 pre-push(17.2): name every skipped check in plain words
One terminal line per distinct coverage reason (push/explain.py): hex-encoded
paths decoded, per-question sha256 tokens dropped and counted, unknown reasons
still printed; the JSON artifact keeps the raw values. A changed language with
no engine frontend and no quick rules is recorded and printed as "language not
covered: <lang> (...)", in pre-push and in `ousast scan`. The scan report
renders the eight degradations cli.py recorded but never showed, and any
reason without a sentence. The hook forwards OUSAST_COMPARISON_BASE.
Requirement 9.2.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 22:10:33 +02:00
Marius CiepluchandClaude Opus 5.5 bf07cd4d1d pre-push(17.1): never block a push the user did not choose to block
The installed hook exits 0 with one line when ousast (or OUSAST_COMMAND) is
not on PATH, still chaining a prior hook. In Git hook mode without an explicit
--mode blocking, argument/usage errors (an invalid OUSAST_PUSH_DEADLINE, a cache
directory inside the repository, a refused experimental flag) and unexpected
exceptions print one "check did not run" line and exit 0. Blocking mode keeps
its usage errors. Requirement 9.1.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:57:42 +02:00
Marius CiepluchandClaude Fable 5.1 605b7af5ba plane(2.4): reconciler on Delivery -- the start body carries the URLs, the store completes the task
_execute builds the task's `delivery {put, inputs}` from Delivery, passes it to
router.starter, grants the store's host in the egress policy, awaits
Delivery.delivered (the tar's exact key in the store), then collect()s:
download, extract with the member checks, re-put the declared artifacts, write
done.json; the outcome is the collected summary.json. Three findings from the
tests: a rerun of an unfinished task completed at once on the previous
attempt's tar, so Delivery.reset(task) deletes output.tar and done.json before
every attempt; a worker thread that raised left the run blocked on its queue,
so the worker now turns any exception into a failed task; a consumer may
declare a producer's summary.json without the producer listing it (the loop
Run does), so declared(run) publishes outputs[] plus every artifact a consumer
names. The receiver thread survives behind OUSAST_DELIVERY=receiver for this
group only (no presigned URLs, the http rule, OUSAST_ARTIFACT_URL in the env);
the default path renders no receiver address into any Task. The fake ax PUTs
to the start body's URL and GETs its inputs from theirs; the reconciler tests
run through the S3-shaped fake store and assert the objects, done.json, the
seeded rerun and the receiver mode. reconciler.py: 549 lines (bound 550).

Gates: pytest 2031 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:49:58 +02:00
Marius CiepluchandClaude Opus 5.5 a656fda2f4 pre-push(17): operational-readiness task group and Requirement 9
Append task group 17 "Operational readiness (2026-10-02)" with one task per
maintainer-approved item (never block, name every skip, quick tier, new-file
classification, engine honesty and background mode, install/Docker/docs, proof),
add Requirement 9 (9.1-9.6, marked added 2026-10-02, maintainer-approved), and
check 13.4 and the parent of 14 as the audit found them done.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:46:25 +02:00
Marius CiepluchandClaude Fable 5.1 c49eae4f30 plane(2.3): egress -- one TLS rule over store host, Git hosts and Model hosts; no http rule
policy_for(task, workspaces, model, store_host, receiver_host=None) emits one
tlsPassthrough rule on 443 over the sorted union of the memory store's host,
the bound Workspaces' https Git hosts and the bound Model's egress hosts. There
is no http rule: an http Git repo is refused by name (it could not be reached),
a store given as an address is refused naming S3_ENDPOINT (the gateway allows
hostnames only). receiver_host adds the old receiver's http rule for the path
group 2 keeps until 2.6; the reconciler passes it by keyword (passing the
receiver's address positionally would have raised outside the worker's try and
hung the run on the queue -- 2.4 hardens the worker). Tests: the store-only
policy, the Model-bound policy, the two refusals, the receiver exception, and
every committed template under plane/tasks/ rendering exactly the store plus,
for verify and roles, the provider host.

Gates: pytest 2027 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:29:32 +02:00
Marius CiepluchandClaude Opus 5.5 49e11e0252 docs(readme): one paragraph for the published site and local preview
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:17:44 +02:00
Marius CiepluchandClaude Opus 5.5 f0fcb6686f docs: publish the MkDocs site to GitHub Pages
A Docs workflow builds the site strictly with the docs extra and deploys it to
https://norandom.github.io/OpenUltraSAST/ on pushes to main that touch the docs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:16:49 +02:00
Marius CiepluchandClaude Fable 5.1 9506a18b56 plane(2.2): runner -- delivery {put, inputs} in the start body, PUT to the store
StartGate.offer validates an optional `delivery` beside the credentials: https
URLs to a hostname (never an address, no userinfo; AX_RUNNER_INSECURE_DELIVERY=1
admits http in unit tests only), names ^[A-Z][A-Z0-9_]*$, a 400 that names the
field and echoes no URL. The runner PUTs its output tar to delivery.put and
GETs each declared input from delivery.inputs[NAME] (deliver() gains a method;
log lines and DeliveryError carry the URL without its query, so no presigned
signature reaches a log, a marker or a summary). OUSAST_ARTIFACT_URL/DIAL stay
honoured until task 2.6 so both paths run in the same build; the completion
marker and the retry-only boot are unchanged, and a re-start brings a fresh
URL for the retry. Router.start_task carries `delivery` in the body and
redacts its URLs like credential values. Tests: the gate's acceptance and
refusals, an end-to-end PUT/GET against the S3-shaped fake with no URL in any
log or file, the failed-PUT-then-retry-only boot, deliver() with PUT.

Gates: pytest 2024 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:12:19 +02:00
Marius CiepluchandClaude Fable 5.1 35d8032c0a plane(2.1): Delivery -- presigned store URLs, collection, state mirrored to the store
src/openultrasast/plane/delivery.py: Delivery(store, run, base) mints a presigned
PUT for runs/<run>/<task>/output.tar and a GET per declared input
(runs/<run>/<producer>/<artifact>), expiry OUSAST_TASK_TIMEOUT + 600 s;
delivered() asks the store for the exact key; collect() downloads, extracts
under the run directory with the receiver's member checks (no symlink, hard
link or escape; nothing written on a rejection), re-puts the declared outputs
as single objects and writes done.json; put_input() for inputs no task
produces; save_state()/load_state() mirror state.json. The reconciler opens the
store first (no fallback; the CLI reports a store that cannot open and submits
nothing), mirrors state.json after every change and seeds a rerun with no local
state from the store (the 2.1 bullet's state mirroring is what gives the new
module its importer). The reconciler crossed 500 lines here (517) rather than at
2.4 as 1.6 projected: the bound moves to 550 with the reason in the assertion,
as 1.6 decided. tests/plane_fake_store.py is the S3-shaped fake (objects in a
dict, presigned URLs on a local server); tests/test_plane_delivery.py covers the
body, expiry, polling, rejection, declared-only re-put, state seeding and that
no URL reaches a log. Module-audit manifest regenerated (148/5, no orphan).

Gates: pytest 2020 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:05:48 +02:00
Marius Ciepluch 9833571e94 Merge branch 'worktree-agent-af2e4a098d650d0a2' 2026-10-02 21:03:41 +02:00
Marius Ciepluch c217d1d1d3 Merge branch 'worktree-agent-aed844222b4750ed7' 2026-10-02 21:03:41 +02:00
Marius CiepluchandClaude Opus 5.5 352f8c07aa docs: plain-language pass -- detection techniques, architecture, evaluation, decision engine
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:03:24 +02:00
Marius CiepluchandClaude Opus 5.5 552671ca6a docs: plain-language pass -- README, index, where we stand, scanning, examples, threat model
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:01:10 +02:00
Marius CiepluchandClaude Opus 5.5 7e23379e96 docs: plain-language pass -- deployment, plane, memory, RustFS, token economics
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:00:37 +02:00
Marius CiepluchandClaude Fable 5.1 79296bb4a4 plane(1.8): kind proof and record 01 -- profile, literals, doctor, probe, bound
benchmarks/measurements/2026-10-02-k8s-01-profile/record.json with the live
evidence of group 1: `ousast plane doctor --profile kind` passes every line
(exit 0, 1.4 s; the store check reaches the bucket: verify 0.62 s, probe object
24 B); `--profile k3s` fails on the unreachable context and on the images file
the release has not published yet (two expected lines; the namespaces are not
checked behind an unreachable context); the literal scan has 0 offenders (23
before 1.3) and fails naming workspaces.py:37 when a literal is reinserted;
up.sh reran idempotently in 38 s and the smoke Run finished in 24 s on the
rebuilt image (facts.json 451 B, summary.json 238 B); the placement probe says
`any`; the reconciler bound numbers of 1.6. Doctor, literal, smoke and up.sh
outputs beside it.

Gates: pytest full suite on this tree's src/ and tests/ content: 2010 passed,
42 skipped, exit 0 (the 1.5 run; the 1.8 run reported two module-audit
failures caused by the uncommitted 2.1 module present in the working tree);
ruff check 0; ruff format 0; mypy 0; mkdocs build --strict 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:55:57 +02:00
Marius CiepluchandClaude Fable 5.1 2cca03c1c0 plane(1.7): gvisor tarball reachability documented as planned (manual, k3s)
docs/deployment.md B.1: Substrate's ateom-gvisor worker runs runsc from the
release tarball named in manifests/ate-install/sandboxconfig-gvisor.yaml
(gs://gvisor/releases/nightly/2026-09-02/<arch>/gvisor.tar.zstd, sha256-pinned
per architecture), fetched by the atelet on the node; the node must reach it or
a copied SandboxConfig must name a mirror with the same sha256. The check is the
atelet's fetch line in the worker pod's logs, run by hand on the day k3s exists;
labelled planned until then (design open question 3 stays open).

Gates: docs only; mkdocs build --strict passed with this paragraph present in
the 1.4 gate; no file a test or linter reads changed since the 1.5 gate.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:50:14 +02:00
Marius CiepluchandClaude Fable 5.1 2afa20e2ea plane(1.6): reconciler bound recorded -- 489 lines, bound moves to 550 in 2.4
reconciler.py: 477 lines at 6bd44ca, 489 after 1.2 (the profile parameter of
run() and its docstring). Task 2.4 adds the Delivery wiring (about +25) while
the receiver path is kept for group 2, and 2.6 removes the receiver wiring
(about -20): projected 514 during group 2, 494 after 2.6. The estimate crosses
500 while both paths coexist, so 2.4 moves tests/test_plane_reconciler.py's
bound to 550 with the reason in the assertion message; 2.6 records the final
count. Numbers in benchmarks/measurements/2026-10-02-k8s-01-profile/record.json
(record 01, drafted here, completed by 1.8).

Gates: no file a test or linter reads changed since the 1.5 gate (pytest 2010
passed, 42 skipped, exit 0; ruff 0; format 0; mypy 0).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:49:45 +02:00
Marius CiepluchandClaude Fable 5.1 86ec237483 plane(1.5): placement probe -- Substrate places by any free worker, not by fit
ops/ax/probe-placement.sh scales ousast-pool to one worker, applies probe-pool
(1 x 2 CPU / 3Gi, the already-built ateom-gvisor image, no ko build), runs one
model-free repo-facts Task requesting 2560Mi (limit 3Gi) and one requesting
256Mi, records `kubectl ate get workers -o json` after each resume, and removes
everything it created. Live on kind: both actors landed on the 1536Mi
ousast-pool worker (status.allocated.actors=1 there, 0 on probe-pool) ->
placement: any. design.md section 4 records the outcome and the fallback group
5 must take (an engine atespace with engine-pool as its only pool if Substrate
scopes pools by atespace; else a plain Kubernetes Job on k3s). Evidence under
benchmarks/measurements/2026-10-02-k8s-01-profile/probe/.

Gates (tree identical to 1.4 in src/ and tests/; ops/ gains the probe script,
which the literal scan covers): pytest 2010 passed, 42 skipped, exit 0; ruff
check 0; ruff format 0; mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:49:03 +02:00
Marius CiepluchandClaude Fable 5.1 8099b29deb plane(1.4): up.sh by profile -- context, registry, pool and images file from kind.toml
ops/ax/up.sh reads kube_context, registry, pools.default and the images path
with `python -m openultrasast.plane.profile --print`, writes the runner digest to
ops/k8s/profiles/kind-images.json instead of ~/.cache/.../runner-image, renders
ops/ax/workerpool.yaml.tmpl from the profile's pool (POOL_NAME/REPLICAS/CPU/
MEMORY, WORKER_IMAGE), and keeps an existing pool and an ax CLI newer than its
checkout instead of rebuilding them (ko and go install would refill several GB
of Go cache on this disk). smoke-run.sh takes the runner from `--print
images.runner` and runs the checkout's code with OUSAST_PYTHON. repin_templates
reads the JSON images file (runner key; the one-line pin still reads). `--print
images` returns the file path (the first live rerun fell over this). Templates
and the generated validation-46 Tasks re-pinned to the rebuilt image (128
lines, one digest).

Live: up.sh reran idempotently on kind in 38 s (every step kept except the
runner build); smoke-run.sh done in 24 s on the new image, facts.json 451 B,
summary.json 238 B delivered (record: benchmarks/measurements/2026-10-02-k8s-01-profile/).

Gates: pytest 2010 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0; mkdocs build --strict 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:43:40 +02:00
Marius CiepluchandClaude Fable 5.1 36f5927999 plane(1.3): host-literal scan test over src, plane, ops and docs
tests/test_plane_literals.py scans src/, plane/, ops/ (minus ops/ax/up.sh and
ops/k8s/profiles/) and docs/ for the kind context, the kind registry, the kind
network's gateway, a loopback address with a port and port-forward; router.py
keeps the last two, a line tagged ax-tunnel or loopback is allowed, and a
committed template's digest-pinned image is accepted only while it names the
kind profile's registry (checked by a third test). A self-check reinserts each
literal into a fixture and sees it caught. The runner's health URL and the Joern
session's query URL are tagged loopback; docs/deployment.md and ops/ax/README.md
stop naming this host and describe the kube context and the registry as the
profile's (what 1.2 made configurable).

Gates: pytest 2009 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0; mkdocs build --strict 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:32:51 +02:00
Marius CiepluchandClaude Fable 5.1 77a03bad61 plane(1.2): literals out of doctor, egress, router, generate -- the profile is handed in
doctor(profile) checks the profile's kube context, the two namespaces behind a
reachable context only, the images file's runner reference in the profile's
registry (a local registry over its v2 API, a remote one via docker manifest
inspect) and opens the memory store; its first line reports every configured
address (Req 1.2). Egress(executable, context), open_router(context, url=...),
receiver_cluster_ip(context) and Receiver(..., context) take the profile's
context and build none; generate.increment/render and harvest take the profile's
atespace. `ousast plane doctor|run|workspaces|harvest --profile`. The test fakes
(kubectl-ate, kubectl, ax) record the context they received and the tests assert
it is the profile's. grep "kind-" src/openultrasast/plane: no match.

Gates: pytest 2006 passed, 42 skipped, exit 0; ruff check 0; ruff format 0; mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:26:46 +02:00
Marius CiepluchandClaude Opus 5.5 50f4e69321 release: v2.0.1 -- Apache-2.0 licence, GHCR images, CI green, front-door README
v2.0.0 was tagged but its release job stopped at the format check, so no release
or package was published; v2.0.1 is the first published 2.x release.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:18:39 +02:00
Marius CiepluchandClaude Fable 5.1 8eac1b2c69 plane(1.1): PlaneProfile -- one TOML per cluster, env-overridable, no secrets
ops/k8s/profiles/kind.toml and k3s.toml hold the values that differ per cluster
(exec, kube_context, registry, images, router_url, ax_server, memory, atespace,
snapshots_bucket, image_pull_secret, pools); kind-images.json carries the digest
pin ops/ax/up.sh wrote. load_profile() reads OUSAST_PLANE_PROFILE (name or path,
default kind), applies OUSAST_<FIELD> and OUSAST_POOL_<NAME>_<FIELD>, and refuses
an unknown key, a credential-shaped key or value, an exec outside local|remote
and an images file without digest pins. `python -m openultrasast.plane.profile
--print FIELD` serves shell scripts. The module-audit manifest is regenerated
for the new module (counts 147/5, no orphan).

Gates: pytest full suite 2002 passed, 1 failed (the module-audit manifest, fixed
by regenerating it; tests/test_module_audit.py re-run exit 0), ruff check 0,
ruff format 0, mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:16:23 +02:00
Marius Ciepluch b1acc865e8 Merge branch 'worktree-agent-a94cfc2aafc87f382' 2026-10-02 20:09:34 +02:00
Marius CiepluchandClaude Opus 5.5 d1aabf0f66 ci: green again -- format and lint the whole tree; push the scan and runner images to GHCR on tags
CI and the release job check `ruff format --check .` and `ruff check .` over the
whole repository, while the local gate checked only src and tests, so eight
benchmark scripts drifted and the v2.0.0 release job stopped before building.
The fixes are formatting, two noqa comments and one renamed loop variable; no
script computes anything differently. The local gate now checks the whole tree.

The release workflow gains an images job: Dockerfile -> ghcr.io/<owner>/openultrasast,
plane/Dockerfile.runner -> ghcr.io/<owner>/openultrasast-runner, tagged with the
version, and the digests written to the job summary.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:09:28 +02:00
Marius CiepluchandClaude Fable 5.1 bdabb978f8 docs(readme): front door -- current commands, status, and links to every docs page
Rewrite README.md as the front door of the docs site: a short "What it is", the
Status (NO-GO against M4, figures quoted from docs/where-we-stand.md), a tiered
install (quick scan without Docker; standard/deep with the openultrasast:dev
image; the plane on ax for maintainers; k3s planned), then one section per
docs page, each linked by relative path so it works on GitHub and on the site.

Stale claims removed: `ousast index` (deleted), `OPENROUTER_EMBEDDING_MODEL`
(no longer read), the duplicated scan-mode, rule and gate tables (now on
docs/scanning.md and docs/evaluation.md), the HarnessX-era wording. Adds a
Licence section stating that no licence file exists yet.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:06:21 +02:00
Marius CiepluchandClaude Fable 5.1 c978de9ff8 merge(cleanup): second legacy inventory -- vector index, equality test, unread config keys, status notes
Also drops the removed ousast index line from the scan skill.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:02:53 +02:00
Marius CiepluchandClaude Fable 5.1 61ff5f3c33 docs(kiro): status notes -- ai-service-plane complete, contributor-scan paused, finding-feedback-loop design unapproved; stale lines
Legacy inventory 2, the six STATUS NOTE items (the archive README row landed with the R2 commit).
ai-service-plane: 11 of 11 tasks checked, spec.json phase set to implementation-complete as 5b96199
did for four other specs. contributor-scan: dated note in requirements.md -- paused since 2026-09-23,
27 of 51 tasks, v1.2.0-alpha.1 NO-GO, remaining tasks wait on pre-push M2/M3. finding-feedback-loop:
dated note -- dormant, design unapproved since 2026-09-09, depends on the paused spec. The steering
overview's spec table carries the same three states, its leftover 2026-09-09 module-count line is
removed and the collected-test count is the measured 2,023. docs/memory-and-detection.md names the
archive path of model_sinks, the one stale script name left in docs after 067adb9.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:57:04 +02:00
Marius CiepluchandClaude Fable 5.1 515f94e3c2 docs(where-we-stand): a scan without a provider key records learning_endpoint_unavailable
cli.py appends the degradation when resolve_chat_endpoint returns nothing; the
gap is that report.md has no line for it, not that nothing is recorded.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgMh3zqD1QMwEo
2026-10-02 19:55:22 +02:00
Marius CiepluchandClaude Fable 5.1 6bd44caa50 spec(plane-on-kubernetes): tasks approved; groups 1-2 first
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:54:25 +02:00
Marius Ciepluch 4c1e24ceae Merge branch 'worktree-agent-a630761c8cf5ded05' 2026-10-02 19:54:24 +02:00
Marius Ciepluch ed769e2334 Merge branch 'worktree-agent-a3e8fa755c00b3e20' 2026-10-02 19:53:52 +02:00
Marius CiepluchandClaude Fable 5.1 03c9f745a4 merge(skills): apply the 2026-10-02 review -- current CLI and evidence, merges, removals, steering names
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:53:52 +02:00
Marius CiepluchandClaude Fable 5.1 a097240702 spec(plane-on-kubernetes): tasks for review
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:53:27 +02:00
Marius CiepluchandClaude Fable 5.1 58e788cecb spec(plane-on-kubernetes): design for review
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:48:48 +02:00
Marius CiepluchandClaude Fable 5.1 d0ac8b5f3d docs: detection techniques -- representations, taint, obligations, ranking, the model's role
New page docs/detection-techniques.md: the three representations (regex over the
file, tree-sitter IR, Joern CPG) and what each can see; Joern as a section of its
own (frontends, one graph per language partition, census as the instrument check,
batching against the measured cost curve, deadlines, degradations, where the engine
sits in the plane, cost and misses with records); taint from fact tables to query to
verdict with each refinement traced to its record or commit; obligations, dominance,
constant abstraction, census; regions, ranking ties and budgets; the residual
question and MAX_JUDGED_CANDIDATES; REGRESS verdicts and the unassigned
execution_confirmed rung; an honest table of what each technique misses.

Nav entry under "How it works" after Scanning; one link from the index; one
cross-link line each in scanning.md and architecture.md.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:48:32 +02:00
Marius CiepluchandClaude Fable 5.1 16d9ef7110 chore(skills): apply the 2026-10-02 review -- current CLI and evidence, merges, removals, steering names
openultrasast-scan: evidence ladder as verification.py defines it, with the levels nothing produces today; --mode prerequisites, --fail-on, recorded degradations, run artifacts, ousast mcp; retired [evidence] threshold language removed. openultrasast-triage: reasons verified against calibration.py; deterministic fusion; adjudications as labels, plane verdicts as features, rule changes through improve or a registered experiment; fix-audit merged as a fix-and-re-scan section. kiro: spec-quick and spec-batch removed; validate-gap and validate-design merged into spec-design (--gap, --review; the review gate stays); steering-custom merged into steering; core steering names overview.md, roadmap.md, safety-net.md; instrument rule in debug and verify-completion; repo gates named in validate-impl. AGENTS.md, README skills paragraph and tests/test_agent_skills.py (.agents/skills, two product skills, drift tests) updated.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:48:01 +02:00
Marius Ciepluch 9f58525f0e Merge branch 'worktree-agent-ad1b3451052ec3033' 2026-10-02 19:46:46 +02:00
Marius CiepluchandClaude Fable 5.1 94d2bfcef4 spec(plane-on-kubernetes): remove the last in-cluster reconciler wording
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:46:46 +02:00
Marius CiepluchandClaude Fable 5.1 91a35a4406 spec(plane-on-kubernetes): Req 2/3 -- the reconciler stays the CLI with a remote mode; artifacts land in the store
Maintainer: trigger the checks in kube from the CLI and feed the results back;
execution is remote instead of local, not a separate service.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:46:22 +02:00
Marius CiepluchandClaude Fable 5.1 6db5a35f07 chore(config): retire unread keys [sandbox] network, [sandbox] workspace_readonly, [complexity] top_k
Legacy inventory 2, item R3. The three keys loaded into ResolvedConfig and nothing read them: no
`.network`, `.workspace_readonly` or `.top_k` access exists in src outside config.py. The sandbox
runner always passes `--network none` and `--read-only` and rejects host networking by argv
inspection, so the two sandbox keys were knobs that turned nothing; no stage read top_k, and
`[complexity] max_hunter_hotspots` is the one complexity key the scan reads. They join RETIRED_KEYS
with the same 2026-10-02 mechanism as the other unread keys (RetiredConfigError naming the reason).
The fields and their loaders are removed; memory_mb, timeout_seconds, pids_limit (read by
regress/recipes.py and sandbox/runner.py) and max_hunter_hotspots stay. test_config.py covers the three
retirements and the override test now exercises max_hunter_hotspots. RELEASE_NOTES Unreleased carries
the line.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:45:37 +02:00
Marius CiepluchandClaude Fable 5.1 297f6a8af2 docs(deployment): production topology -- k3s and GHCR target, kind as a first-class profile, the reconciler stays the CLI
Reworks the production topology section after the maintainer's three
decisions of 2026-10-02: the target cluster is k3s (containerd, no Docker
socket, runsc + RuntimeClass for gVisor workers, Traefik) with digest-pinned
images published to GHCR by GitHub Actions; kind stays a first-class local
execution mode (two profiles of one code base, every change proven on kind
first); and the reconciler stays the CLI, with a planned remote mode in which
the same `ousast plane run` submits to ax-server in k3s, tasks deliver their
artifacts to the S3 memory store through presigned URLs, and the CLI polls the
store for the same status and attribution table. The in-cluster reconciler
and the in-cluster receiver become a later option. The diagram is redrawn
without an inbound path; the pre-push section says the hook or CI calls the
CLI in remote mode and either waits with a deadline or reads a later `plane
status`. Every item that is not built stays labelled planned.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:45:27 +02:00
Marius CiepluchandClaude Fable 5.1 ac9c30da0a docs(deployment): production topology on Kubernetes and the pre-push hook at scale
Answers the maintainer's questions on deployment.md: ax is a Kubernetes
application (ax-server Deployment + Redis in ax-system) that schedules every
Task as a gVisor actor on Agent Substrate (ate-system, WorkerPool CRD); there
is one runner image and concurrency equals warm workers, one actor per worker.
Names the three host-bound components (reconciler, receiver, Joern engine) with
their planned in-cluster replacements, the production shape of the pre-push
hook (local deterministic check in the deadline, delta Run on the plane
asynchronously), the estimated per-push cost from the two measured
per-candidate rates, what is not built (no `ousast plane scan`, no diff
candidates task, population-only Workspace generation, no in-cluster
reconciler), and a sizing table from plane/tasks/*.yaml. plane.md gets a
two-line pointer to the section.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:42:36 +02:00
Marius CiepluchandClaude Fable 5.1 adb70b6b98 chore(tests): drop the HarnessX equality test that guards an archived one-off
Legacy inventory 2, item R2. tests/test_harnessx_removal_equality.py loaded
benchmarks/archive/harnessx_removal_equality.py by path and checked that its comparison sees a
difference. The spec is implementation-complete and the proof is the committed pair of records
(2026-09-30-harnessx-removal-{baseline,equality}); the archive README says these scripts are
reference only and a record is re-run at the commit it names, not with today's package. A unit test
that keeps the archived copy importable contradicted that and would break the next time cli or
learn.cli move. The archive README row now says the test was dropped. test_removed_plane_references.py
(the Req 2.4 name search) is a different test and stays.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:39:27 +02:00
Marius CiepluchandClaude Fable 5.1 17ac7ad69d spec(plane-on-kubernetes): kind stays a first-class local execution mode
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:38:03 +02:00
Marius CiepluchandClaude Fable 5.1 85107bfbcc spec(plane-on-kubernetes): requirements approved; target is k3s with images on GitHub (GHCR)
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:37:48 +02:00
Marius CiepluchandClaude Fable 5.1 6187214be3 spec(plane-on-kubernetes): brief and requirements for review
Maintainer: the whole system moves to a Kubernetes cluster soon. Seven host-bound
assumptions become cluster resources or configuration; the engine becomes a
plane task on a larger pool; a delta Run command serves the pre-push use; every
step is proven on kind first.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:36:59 +02:00
Marius CiepluchandClaude Fable 5.1 e5c338070d merge(docs): where we stand -- capability, approach, ax integration, token economics, open items
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:36:23 +02:00
Marius CiepluchandClaude Fable 5.1 e1db7e1920 chore(index): remove the unused vector index, the ousast index subcommand and the [embeddings] section
Legacy inventory 2, item R1. The vector half of index.py (VectorIndex, VectorRecord, RetrievalHit,
RetrievalPackage, the EmbeddingClient protocol, index_reuse_key, build/write/read/query_vector_index,
build_retrieval_package and their helpers) had no importer in src; `ousast index` wrote index/chunks.json
that nothing read, and was the only reader of `[embeddings]`. index.py keeps CodeChunk and
chunk_text_namespace, which skills.py imports; chunk_file/build_code_chunks go with their only caller.

`[embeddings] model` and `[embeddings] store` join RETIRED_KEYS (RetiredConfigError naming the reason),
the same way the [variants] keys were retired; OPENROUTER_EMBEDDING_MODEL is no longer read, so the
README and .env.example drop it (`ousast learn memory embed --model` names the model). test_index.py
now tests the chunker only; test_config.py covers the two retired keys and no longer asserts the
removed field. The module audit record is regenerated (counts unchanged: 151 modules, 0 orphaned;
the index module's importer list shrinks to skills). RELEASE_NOTES Unreleased carries the line.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:32:46 +02:00
Marius CiepluchandClaude Fable 5.1 3344b1fc45 chore(skills): repoint the remaining references to .agents/skills
The previous commit moved the directory but a failed git add left these seven
reference edits unstaged.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:30:48 +02:00
Marius CiepluchandClaude Fable 5.1 5010b15ffc chore(skills): .agents/skills -- the plural directory OpenCode searches by default
Verified against opencode.ai/docs/skills: project-local search paths are
.opencode/skills, .claude/skills and .agents/skills, not configurable.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:24:33 +02:00
Marius CiepluchandClaude Fable 5.1 6b12546531 docs: where we stand -- capability, approach, ax integration, token economics, open items
One page, docs/where-we-stand.md, composed with the COBESY composition method (governing idea,
SCQA opener, MECE sections): the M4 gates and the state against them, the detection approach step
by step, ax on the server, token economics before and after, and the open items. Every figure is
quoted from a committed record whose path stands next to it.

decision-engine.md, evaluation.md and token-ergonomics.md link to the page where they repeated its
content (detail tables kept); decision-engine.md's Experiments section now records exp-002's
result instead of saying it has not run. Linked from the nav, the index and the README.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:18:54 +02:00
Marius CiepluchandClaude Fable 5.1 69b5e0efcc chore(skills): move the project skills from .opencode/skills to .agent/skills
Maintainer decision 2026-10-02 (no symlink). AGENTS.md, README, the two spec
docs, the kiro-spec-batch skill and the provenance ignore list are repointed;
the OpenCode plugin package.json stays under .opencode.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:17:19 +02:00
Marius CiepluchandClaude Fable 5.1 48510fd5c2 bench(learn): exp-002 result -- retrieval ensemble vs temperature sampling
Reject B. 160 paired units (injection 119, access_control 41; 56 repositories),
both arms, two replicates each, no early stop. Canary agreement A 0.875 -> B
0.906 (CI [-0.03, +0.10], spans; B meets 0.90). Within-pair AUC A 0.815 -> B
0.741 (CI [-0.13, -0.02], against B). AUC -0.02 (against), ADVISORY recall
-0.06 and precision +0.01 (span), usd per candidate A $0.0021 -> B $0.0051
(against). Priced usage A $0.68 / B $1.63; balance $16.72 -> $16.04 at the
run's end (lags). Counts only; task 7.4 marked done.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:05:34 +02:00
Marius CiepluchandClaude Fable 5.1 780caaa558 fix(learn): the experiment_result row keeps one JSON type per column of its object
An experiment's register, run, result and arm_outcome rows share one object,
and RustFS's S3 Select infers one schema per object: exp-002's result row
carried `units` as an object beside the run row's integer (and `looks` as
objects beside the register row's fractions), and after `analyse --record` no
row of the experiment could be read (JSONParsingError). The result row now
writes `unit_counts` and `looks_taken`; a test asserts one JSON shape per
column across the four row kinds. The live object was repaired by dropping
the one row (drop_rows) and recording the analysis again.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 18:57:56 +02:00
Marius CiepluchandClaude Fable 5.1 8d45483d10 bench(learn): exp-002 registration recorded with the frozen units
The experiment row in s3://sast-memory now carries units_digest 0de66bfa... at
pin 9213e5f (manifest digest unchanged, f5004fb8...). Counts per family and the
two repairs the freeze needed are in the record; no model.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 17:28:55 +02:00
Marius CiepluchandClaude Fable 5.1 9213e5ffd9 bench(learn): exp-002 units frozen and registered
160 units (injection 119 in 34 repositories, 76 paired as 38 candidate pairs;
access_control 41 in 24 repositories, 32 paired as 16 pairs), 56 repositories
in all, under the incumbent folds' seed on the 937-example memory (slice 2
evaluated 110 + 37 on the 904-example memory; the memory grew by the recovered
negatives on 2026-10-02). No model. sha256 0de66bfa3ff844825e06f4f4227a14210d1b77f1cc08c0441c6d23e23fa5d91f.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 17:27:26 +02:00
Marius CiepluchandClaude Fable 5.1 1f525ff540 fix(learn): example rows drop another profile's null features when written and read
Static-profile feature records carry the nine plane-only features (agree.*,
ms.*, verify.*) as null in x; after cb85542 the 937 live example rows loaded,
but the renderer's validate_x refused every one of them ("features outside the
static allow-list"), so no program could prepare a prompt (found while probing
exp-002 model-free). profile_x drops a null feature outside the row's profile
and keeps a valued one, so a real leak still fails. Test added.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 17:24:49 +02:00
Marius CiepluchandClaude Fable 5.1 cb85542578 fix(learn): example rows skip null instrument entries when written and read
Static-profile feature records carry the plane-only instruments as null; every
one of the 937 live example rows was unreadable (TypeError), which blocked
ousast learn evaluate and freezing exp-002 units.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:54:24 +02:00
Marius CiepluchandClaude Fable 5.1 ef1f4daa99 merge(learn): recovered negatives; excerpt matcher covers more declaration forms
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:48:09 +02:00
Marius CiepluchandClaude Fable 5.1 4c8c3ce1c4 merge(learn): A/B experiment runner; exp-001 and exp-002 registered (decision-engine task 7)
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:37:57 +02:00
Marius CiepluchandClaude Fable 5.1 ef6945ec53 bench(learn): labels, features and memory rebuilt with the recovered negatives
Labels regenerated (identical sha: the builder's fixed-side rule already
accepted these pairs), feature records built for the 9 corpus pairs the
matcher now resolves (36 rows; the store's ingest index had remembered them
from a build before the declares() gate and skipped them as a repeat after a
later build dropped them -- build_features now checks presence after an index
hit and restores the rows), static memory rebuilt on RustFS: 904 -> 937
examples (+16 fixed-side negatives, +17 positives: 7 advisory Ruby pairs, 9
corpus pairs, 1 population positive), 9 wrong excerpts corrected (Ruby blocks,
a TS default-value brace), no embedding call (33 new + 9 changed excerpts
without a vector, ~$0.0003 to embed). Positives-only groups 98 -> 99: all
data gaps (94 vibe-py same-snapshot fixed sides, 3 dev-php, 2 population
fixes that did not change the function). Leak audit: pairs 376 -> 385; static
no new flag; plane two new injection flags (agree.final, verify.votes at
0.2292 over 48 pairs), v1 profile unchanged, reported.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:31:30 +02:00
Marius CiepluchandClaude Fable 5.1 9d74839128 fix(learn): excerpt matcher covers more declaration forms
The brace matcher now finds a declaration head in more forms -- name(...) with
any return type, qualifiers, generics or a parameter list spread over many
lines (a paren-aware scan replaces the 8-line lookahead; a prototype's ';' is
not a body, a ')' closing an outer parenthesis marks a call), JS object-literal
methods and property/chained assignments of function expressions and arrows,
$ as an identifier character, Perl 'sub name {', C# expression bodies -- and a
file given as 'other'/'unknown' is sniffed for Perl and Ruby so their def/sub
forms are matched by the right pattern instead of a {} block inside them. The
memory build hands the matcher the label's language where the record's closed
vocabulary language ('other' for Go, Ruby, C#) resolves nothing. One test per
form. Fixed-side negatives still require the fixed side to declare the
function; the 94 vibe-py positives-only groups stay counted.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:12:08 +02:00
Marius CiepluchandClaude Fable 5.1 3b91a52788 bench(learn): exp-001 and exp-002 registered in the memory store, units pending (decision-engine task 7.2)
Registration rows at pin fbc3220 in s3://sast-memory; in-repo records under
benchmarks/experiments/<id>/registration.json. exp-002's units could not be
frozen: every example row in the store carries null plane-only instruments
and learn.examples.load_examples raises on them (unchanged code since
a90e570) -- a store-side change to diagnose before any paid run.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:08:36 +02:00
Marius CiepluchandClaude Fable 5.1 fbc3220bbe feat(learn): A/B experiment runner -- registered manifests, paired arms, bootstrap and McNemar (decision-engine task 7)
learn/experiments.py: YAML manifests under plane/experiments/<id>.yaml,
digest-checked at register, run and analyse time; `experiment`,
`arm_outcome` and `experiment_result` rows; paired arms in a seeded
repository order with a per-unit coin; arm A replay-only at $0; canary
replicates under a fresh cache salt; repository-cluster paired bootstrap,
exact McNemar, two O'Brien-Fleming looks, adopt/reject/equivalent/
inconclusive. `ousast learn experiment register|units|run|analyse`.
Program: `ProgramSpec.sampling = retrieval_ensemble` (k samples at
temperature 0 over k disjoint retrieved example sets; default unchanged);
`Caller.salt`; `retrieve(exclude=)`. verify: OUSAST_VERIFY_CALLERS=off.
Manifests exp-001-known-callers (plane arms, units pending) and
exp-002-retrieval-ensemble (slice-2 incumbents, ceiling $4) -- not run.
18 scripted tests; module audit regenerated (151 modules).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:04:33 +02:00
Marius CiepluchandClaude Fable 5.1 5aac7eac36 docs: audit against v2.0.0; deployment guide (local ax and a separate Kubernetes cluster)
Every README and docs page checked against ousast --help, the code and the committed records. New docs/deployment.md (in the nav): what runs where today, this host's setup (up.sh, doctor, smoke Run, footprint), step-by-step guidance for a separate Kubernetes cluster with implemented vs planned stated per item (no cluster deployment has been made; kube context and doctor's registry check are hard-coded in doctor.py, egress.py, router.py), and what the plane/ manifests need changed. Decision-engine pages state slice 2 (six families, pooled and within-pair AUC, BLOCK offered nowhere, canary below 0.9 for three families, positives-only inflation) and that the engine is not adopted; exp-002 registered, not run, no result quoted. Fixed: model/judge.py reference (now model/pipeline.py), missing-key behaviour, .env reference (OPENROUTER_EMBEDDING_MODEL, store variables), docs extra, plane memory-normalise and ousast index listed, ax README's cluster table made precise. RELEASE_NOTES gets an Unreleased entry.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:03:46 +02:00
Marius CiepluchandClaude Fable 5.1 090265b247 bench(learn): where the positives-only groups come from
Counts-only record of every repository group of the static memory that holds
positives and no negatives (98), traced label -> feature record -> excerpt:
94 are vibe-py pairs whose fixed side is another function of the same
snapshot (data gap, counted), 3 dev-php recipes without a fixed pin, 1
population-v2 fix that did not change the function. Separately, 16 pairs are
absent from the memory on both sides for tooling reasons: 7 advisory Ruby
pairs matched under the feature vocabulary's 'other' language, 9 corpus pairs
whose declaration form the excerpt matcher misses (JS object-literal methods,
property-assignment arrows, a $ identifier, a C parameter list past the
8-line lookahead, Perl sub).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:00:37 +02:00
Marius CiepluchandClaude Fable 5.1 a90e570f48 fix(learn): v1 withholds verify.flag_b and ms.operation (plane-profile leak audit)
Maintainer decision 2026-10-02. Not pushed with the release; main is otherwise
at v2.0.0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 15:25:21 +02:00
253 changed files with 33017 additions and 3035 deletions
@@ -55,6 +55,14 @@ Extract:
- The command that produced the failure
- Whether the failure is deterministic or intermittent
### 1b. Verify the Instrument Before You Believe the Measurement
Every wrong diagnosis recorded in this project had the same shape: the input was unreadable and the tool reported a plausible zero (`file_exists()` false, the parser wrote nothing, the frontend exited 0), and "no errors, 0 findings" read exactly like a clean result. Before trusting any measurement in the failure report:
- **Prove the tool read its input.** Make it open one file and print the size. A zero from an unread tree and a zero from clean code are indistinguishable afterwards.
- **Distrust a number that is too good.** A Joern script "finishing" in 0.5 s when a JVM takes 15 s to start is a failed launch. A harness must fail loudly with the exit code.
- **Prefer the shipped environment** (the Docker image has a native `php`); local wrappers that proxy stdio are not valid instruments for `php2cpg`.
- **When a result contradicts a working baseline, suspect the instrument first.**
A plausible zero is a failed instrument until proven otherwise; classify it as `CONFIG_GAP` or `RUNTIME_MISMATCH`, not as `LOGIC_ERROR` in the code under test.
### 2. Inspect Local Runtime and Repository State
Inspect the repository for local evidence:
- `package.json`, `pyproject.toml`, `go.mod`, `Makefile`, `README*`
@@ -134,6 +142,7 @@ If the issue is fixable by repo changes inside the current task plan, do not esc
| “Let’s try a few fixes” | Multi-fix guessing hides root cause. |
| “The spec is probably wrong, I’ll adapt it” | Spec conflicts must be surfaced explicitly. |
| “The docs search is optional” | For runtime/dependency issues, docs and version issues often contain the shortest path to root cause. |
| “It ran clean: no errors, zero findings” | A tool that never read its input reports the same. Prove the read (one file, its size) before believing the zero. |
## Output Format
@@ -20,7 +20,7 @@ description: Entry point for new work. Determines the best action path or work d
Gather **only metadata** to determine the action path. Do NOT read full file contents yet.
- **Specs inventory**: Scan `.kiro/specs/*/spec.json` for `name`, `phase` fields and `approvals` status. Note feature names and their current status.
- **Steering existence**: Check which files exist in `.kiro/steering/` (product.md, tech.md, structure.md, roadmap.md). Do NOT read their contents yet.
- **Steering existence**: Check which files exist in `.kiro/steering/` (core: overview.md, roadmap.md, safety-net.md; custom files may exist too). Do NOT read their contents yet.
- **Roadmap check**: If `.kiro/steering/roadmap.md` exists, read it. This contains project-level context (approach, scope, constraints, spec list) from a previous discovery session. Use it to restore project context.
- **Top-level structure**: List the project root directory to note key directories and files. Do NOT recurse into subdirectories.
@@ -60,7 +60,7 @@ For Path A/B, recommend the next action and stop.
**Only for Path C, D, and E.** Now load the context needed for discovery.
**In main context** (essential for dialogue with user):
- **Steering documents**: Read product.md and tech.md (if they exist) for project goals, constraints, and tech stack
- **Steering documents**: Read overview.md and safety-net.md (if they exist) for project goals, constraints, and the current state of the stack
- **Relevant specs**: If the request is adjacent to an existing spec, read that spec's requirements.md to understand boundaries and avoid overlap
**Delegate to sub-agent** (keeps exploration out of main context):
@@ -198,7 +198,7 @@ Use this roadmap structure:
- [ ] feature-c -- [one-line description]. Dependencies: feature-a, feature-b
```
Then write `.kiro/specs/<feature>/brief.md` for **every** feature listed under `## Specs (dependency order)` using the Path C brief format. This enables parallel spec creation via `/kiro-spec-batch`.
Then write `.kiro/specs/<feature>/brief.md` for **every** feature listed under `## Specs (dependency order)` using the Path C brief format. Each spec is then created one at a time with `/kiro-spec-init`, in dependency order.
**For Path E (mixed decomposition)**:
@@ -219,7 +219,7 @@ Use the same roadmap structure as Path D, plus these additional sections:
```
Path E rules:
- Keep `## Specs (dependency order)` reserved for **new specs only** so `/kiro-spec-batch` can still parse it unchanged
- Keep `## Specs (dependency order)` reserved for **new specs only** so the list stays a plain, parseable creation order
- Record existing-spec extensions under `## Existing Spec Updates`
- Record true no-spec work under `## Direct Implementation Candidates`
- Write `brief.md` only for the **new specs** listed under `## Specs (dependency order)`
@@ -235,15 +235,12 @@ Suggest the next command and stop. Do NOT automatically run downstream spec gene
- Path A: `/kiro-spec-requirements {feature}` to update the existing spec
- Path B: Recommend direct implementation without creating a spec
- Path C: Default to `/kiro-spec-init <feature-name>`
- Optional fast path: `/kiro-spec-quick <feature-name>` when the user explicitly wants to continue immediately
- Path D: Default to `/kiro-spec-batch` (creates all specs in parallel based on roadmap.md dependency order)
- Optional cautious path: `/kiro-spec-init <first-feature-name>` when the user wants to validate the first slice before batching the rest
- Path C: `/kiro-spec-init <feature-name>`, then the step-by-step phases (requirements, design, tasks), each human-reviewed
- Path D: `/kiro-spec-init <first-feature-name>` for the first spec in roadmap.md dependency order; validate that slice before the next one
- Path E: Choose the next command based on the new-spec portion of the decomposition
- If there is exactly one new spec: `/kiro-spec-init <new-feature-name>`
- If there are multiple new specs: `/kiro-spec-batch`
- `/kiro-spec-init <new-feature-name>` for the first (or only) new spec, the rest in dependency order afterwards
- Also note which existing specs should be revisited with `/kiro-spec-requirements <feature>`
- Re-entry: `/kiro-spec-init <next-feature-name>` or `/kiro-spec-batch` if multiple specs remain
- Re-entry: `/kiro-spec-init <next-feature-name>` for the next spec that remains
If the decomposition contains only existing-spec updates plus direct implementation candidates, do NOT use Path E. Prefer Path A when one existing spec is the clear home, or recommend the existing-spec update plus direct implementation work without creating roadmap entries.
@@ -26,7 +26,7 @@ You operate in two modes:
If steering/spec context is already available from conversation, skip redundant file reads.
Otherwise, load all necessary context:
- `.kiro/specs/{feature}/spec.json`, `requirements.md`, `design.md`, `tasks.md`
- Core steering context: `product.md`, `tech.md`, `structure.md`
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to the selected task's boundary, runtime prerequisites, integrations, domain rules, security/performance constraints, or team conventions that affect implementation or validation
- Relevant local agent skills or playbooks only when they clearly match the task's host environment or use case; read the specific artifact(s) you need, not entire directories
@@ -1,13 +1,18 @@
---
name: kiro-spec-design
description: Create comprehensive technical design for a specification
description: Create comprehensive technical design for a specification; optionally run the implementation gap analysis first (--gap) or an interactive design review of an existing design (--review)
metadata:
shared-rules: "design-principles.md, design-discovery-full.md, design-discovery-light.md, design-synthesis.md, design-review-gate.md"
shared-rules: "design-principles.md, design-discovery-full.md, design-discovery-light.md, design-synthesis.md, design-review-gate.md, gap-analysis.md, design-review.md"
---
# Technical Design Generator
**Arguments**: `$1` is the feature. `$2` may be `-y` (auto-approve requirements), `--gap` (run the
optional gap analysis of Step 1b before designing) or `--review` (run only the optional interactive
design review of Step 7 on an existing `design.md`). The design review gate of Step 5 always runs
when a design is generated; the optional steps never replace it.
<background_information>
- **Success Criteria**:
- All requirements mapped to technical components with clear interfaces
@@ -24,8 +29,8 @@ metadata:
**Read all necessary context**:
- `.kiro/specs/$1/spec.json`, `requirements.md`, `design.md` (if exists)
- `.kiro/specs/$1/research.md` (if exists, contains gap analysis from `/kiro-validate-gap`)
- Core steering context: `product.md`, `tech.md`, `structure.md`
- `.kiro/specs/$1/research.md` (if exists, contains the gap analysis from Step 1b or an earlier run)
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (this repository's core steering; custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to requirement coverage, architecture boundaries, integrations, runtime prerequisites, security/performance constraints, or team conventions that affect implementation readiness
- `.kiro/settings/templates/specs/design.md` for document structure
- Read `rules/design-principles.md` from this skill's directory for design principles
@@ -35,6 +40,16 @@ metadata:
- If `-y` flag provided ($2 == "-y"): Auto-approve requirements in spec.json
- Otherwise: Verify approval status (stop if unapproved, see Safety & Fallback)
### Step 1b (optional, `--gap`): Implementation Gap Analysis
Recommended for brownfield features; skip for greenfield. Analyze the gap between the approved requirements and the existing codebase to inform the design, following `rules/gap-analysis.md` from this skill's directory.
1. Read `rules/gap-analysis.md` for the analysis framework.
2. Research in parallel where possible (sub-agents if available, otherwise sequentially): existing implementations, architecture patterns, integration points and extension possibilities; external dependency compatibility and version constraints when needed; the requirements, core steering, task-relevant extra steering and relevant local agent skills or playbooks.
3. Execute the gap analysis: Grep and Read the codebase, use WebSearch/WebFetch for external dependencies if needed, evaluate multiple implementation approaches (extend / new / hybrid) with trade-offs, flag areas needing further research. Provide information and options, not final implementation choices.
4. Write the analysis to `.kiro/specs/$1/research.md` (append after a `---` rule if the file exists; never overwrite earlier research) and read it back to verify the write. Use the language specified in spec.json.
5. Unapproved requirements are a warning here, not a stop: gap analysis can inform requirement revisions. Then continue with Step 2, or stop and report if the user asked for the gap analysis alone.
### Step 2: Discovery & Analysis
**Critical: This phase ensures design is based on complete, accurate information.**
@@ -137,6 +152,15 @@ After all findings return, synthesize in main context before proceeding.
- Set `approvals.requirements.approved: true`
- Update `updated_at` timestamp
### Step 7 (optional, `--review`): Interactive Design Review
A human-paced quality review of an existing `.kiro/specs/$1/design.md`, in addition to the automatic gate of Step 5. Follow `rules/design-review.md` from this skill's directory.
1. Gather context: spec.json, requirements.md, design.md, core steering, review-relevant extra steering and local playbooks; survey existing codebase patterns, naming conventions and component structure for reference (parallel sub-agents if available).
2. Review: Analysis -> Critical Issues -> Strengths -> GO/NO-GO. Limit to the 3 most important concerns; recognize 1-2 strengths; engage interactively (clarifying questions, alternatives); leverage conversation history for requirements context and user intent. Quality assurance, not perfection seeking; accept acceptable risk; every suggestion must be implementable.
3. Decide: a clear GO/NO-GO with rationale. GO: `/kiro-spec-tasks $1` (or `-y`). NO-GO: address the critical issues, re-run `/kiro-spec-design $1`, then `/kiro-spec-design $1 --review` again.
4. If design.md does not exist, stop: "Run `/kiro-spec-design $1` first to generate design document". If the design phase is not marked generated in spec.json, warn but proceed.
## Critical Constraints
- **Type Safety**:
- Enforce strong typing aligned with the project's technology stack.
@@ -199,7 +223,7 @@ Provide brief summary in the language specified in spec.json:
**If Design Approved**:
- Review generated design at `.kiro/specs/$1/design.md`
- **Optional**: Run `/kiro-validate-design $1` for interactive quality review
- **Optional**: Run `/kiro-spec-design $1 --review` for the interactive quality review (Step 7)
- Then `/kiro-spec-tasks $1 -y` to generate implementation tasks
**If Modifications Needed**:
@@ -24,7 +24,7 @@ metadata:
- Read `.kiro/specs/$1/spec.json` for language and metadata
- Read `.kiro/specs/$1/brief.md` if it exists (discovery context: problem, approach, scope decisions, boundary candidates)
- Read `.kiro/specs/$1/requirements.md` for project description
- Core steering context: `product.md`, `tech.md`, `structure.md`
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to feature scope, user personas, business/domain rules, compliance/security constraints, operational constraints, or existing product boundaries
- Relevant local agent skills or playbooks only when they clearly match the feature's host environment or use case and contain domain terminology or workflow rules that shape user-observable requirements
@@ -40,7 +40,7 @@ The following research areas are independent. Decide the optimal decomposition b
**In main context** (essential for requirements generation):
- Spec files: spec.json, brief.md, requirements.md (project description)
- EARS format rules, requirements review gate, requirements template
- Core steering: product.md, tech.md (directly inform scope and constraints)
- Core steering: overview.md, safety-net.md (directly inform scope and constraints)
**Delegate to sub-agent** (keeps exploration out of main context):
- **Codebase hints** (brownfield projects): Spawn a sub-agent to explore existing implementations that inform requirement scope. Ask it to summarize: (1) what already exists, (2) relevant interfaces/APIs, (3) patterns that new requirements should align with. Return a summary under 150 lines.
@@ -133,7 +133,7 @@ Provide output in the language specified in spec.json with:
**If Requirements Approved**:
- Review generated requirements at `.kiro/specs/$1/requirements.md`
- **Optional Gap Analysis** (for existing codebases):
- Run `/kiro-validate-gap $1` to analyze implementation gap with current code
- Run `/kiro-spec-design $1 --gap` to analyze the implementation gap with the current code before designing
- Identifies existing components, integration points, and implementation strategy
- Recommended for brownfield projects; skip for greenfield
- Then `/kiro-spec-design $1 -y` to proceed to design phase
@@ -25,7 +25,7 @@ metadata:
**Read all necessary context**:
- `.kiro/specs/$1/spec.json`, `requirements.md`, `design.md`
- `.kiro/specs/$1/tasks.md` (if exists, for merge mode)
- Core steering context: `product.md`, `tech.md`, `structure.md`
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to requirements coverage, design boundaries, runtime prerequisites, or team conventions that affect task executability
**Validate approvals**:
@@ -1,6 +1,6 @@
---
name: kiro-steering
description: Manage .kiro/steering/ as persistent project knowledge
description: Manage .kiro/steering/ as persistent project knowledge, including custom steering documents for specialized contexts
metadata:
shared-rules: "steering-principles.md"
---
@@ -11,9 +11,12 @@ metadata:
<background_information>
**Role**: Maintain `.kiro/steering/` as persistent project memory.
**Core steering in this repository**: `overview.md` (what the project is and what is actually true of it, dated current state), `roadmap.md` (milestones and status), `safety-net.md` (product direction). Custom files may exist beside them (for example a domain or standards file); all `.kiro/steering/*.md` are loaded as project memory. The generic bootstrap templates under `.kiro/settings/templates/steering/` are named `product.md`, `tech.md`, `structure.md`; they are starting points for a fresh project, not this repository's file names.
**Mission**:
- Bootstrap: Generate core steering from codebase (first-time)
- Sync: Keep steering and codebase aligned (maintenance)
- Custom: Create specialized steering documents beyond the core files (optional, see below)
- Preserve: User customizations are sacred, updates are additive
**Success Criteria**:
@@ -27,8 +30,9 @@ metadata:
Check `.kiro/steering/` status:
**Bootstrap Mode**: Empty OR missing core files (product.md, tech.md, structure.md)
**Sync Mode**: All core files exist
**Bootstrap Mode**: Empty OR missing core files (here `overview.md`, `roadmap.md`, `safety-net.md`; in a fresh project the template names `product.md`, `tech.md`, `structure.md`)
**Sync Mode**: All core files exist
**Custom Mode**: The user asks for a specialized steering document (see Custom Steering below)
---
@@ -75,6 +79,20 @@ After all parallel research completes, synthesize patterns for steering files.
---
## Custom Steering (optional)
Create a specialized steering document for one domain beyond the core files.
1. **Ask the user** for the domain/topic (e.g. "API standards", "testing approach") and the specific patterns to document
2. **Check for a template** in `.kiro/settings/templates/steering-custom/`: `api-standards.md`, `testing.md`, `security.md`, `database.md`, `error-handling.md`, `authentication.md`, `deployment.md`. Load the matching one as a starting point; without a template, generate from domain knowledge
3. **Analyze the codebase** (JIT) for the domain's patterns with Glob/Grep/Read (template and principles in parallel with the domain analysis when multi-agent is available)
4. **Generate** the document: follow the template structure if any, apply `rules/steering-principles.md`, patterns not exhaustive lists, one domain per file, 100-200 lines (a 2-3 minute read), never secrets
5. **Create** `.kiro/steering/{name}.md`; ensure it does not duplicate core steering content
Custom files are loaded as project memory like the core files and carry equal weight.
---
## Granularity Principle
From `rules/steering-principles.md` (in this skill's directory):
@@ -106,9 +124,9 @@ Chat summary only (files updated directly).
✅ Steering Created
## Generated:
- product.md: [Brief description]
- tech.md: [Key stack]
- structure.md: [Organization]
- overview.md: [What it is, current state]
- roadmap.md: [Milestones, status]
- safety-net.md: [Product direction]
Review and approve as Source of Truth.
```
@@ -118,8 +136,8 @@ Review and approve as Source of Truth.
✅ Steering Updated
## Changes:
- tech.md: React 18 → 19
- structure.md: Added API pattern
- overview.md: Current state dated, retired component removed
- roadmap.md: Milestone status updated
## Code Drift:
- Components not following import conventions
@@ -128,6 +146,20 @@ Review and approve as Source of Truth.
- Consider api-standards.md
```
### Custom:
```
✅ Custom Steering Created
## Created:
- .kiro/steering/api-standards.md
## Based On:
- Template: api-standards.md
- Analyzed: src/api/ directory patterns
Review and customize as needed.
```
## Examples
### Bootstrap
@@ -136,7 +168,12 @@ Review and approve as Source of Truth.
### Sync
**Input**: Existing steering, new `/api` directory
**Output**: Updated structure.md, flagged non-compliant files, suggested api-standards.md
**Output**: Updated overview.md, flagged non-compliant files, suggested api-standards.md
### Custom
**Input**: "Create API standards steering"
**Action**: Load template, analyze src/api/, extract patterns
**Output**: api-standards.md with project-specific REST conventions
## Safety & Fallback
@@ -84,7 +84,11 @@ Never include:
## File-Specific Focus
- **product.md**: Purpose, value, business context (not exhaustive features)
- **tech.md**: Key frameworks, standards, conventions (not all dependencies)
- **structure.md**: Organization patterns, naming rules (not directory trees)
- **Custom files**: Specialized patterns (API, testing, security, etc.)
This repository's core steering files:
- **overview.md**: What the project is and what is actually true of it; a dated current-state section wins over older history
- **roadmap.md**: Milestones, their measured status, and the accepted order of work (not a task list)
- **safety-net.md**: Product direction and acceptance criteria (not feature catalogs)
- **Custom files**: Specialized patterns (API, testing, security, etc.); they may exist beside the core files and carry equal weight
(The generic bootstrap templates `product.md`, `tech.md`, `structure.md` under `.kiro/settings/templates/steering/` cover purpose, key frameworks and organization patterns for a fresh project.)
@@ -68,10 +68,16 @@ For each detected feature:
- Read `.kiro/specs/<feature>/requirements.md` for requirements
- Read `.kiro/specs/<feature>/design.md` for design structure
- Read `.kiro/specs/<feature>/tasks.md` for task list and Implementation Notes
- Core steering context: `product.md`, `tech.md`, `structure.md`
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to the validated boundaries, runtime prerequisites, integrations, domain rules, security/performance constraints, or team conventions that affect the GO/NO-GO call
**Discover canonical validation commands**:
**This repository's gates** (`.github/workflows/ci.yml`; run them as written, judge each by its own exit code):
- `TEST_COMMANDS`: `pytest -q -p no:cacheprovider < /dev/null; echo "exit=$?"` -- judge ONLY by the printed exit code, never by a summary line; `ruff check src tests`; `ruff format --check src tests`; `mypy src/openultrasast`
- Detection gates: `python -m openultrasast.gate` (>=90% recall, <10% FP per language; hard, score-independent), `python -m openultrasast.map_gate`, and the pair gate `python -m openultrasast.pair_gate` (local vuln vs fix). A ruleset change that fails a gate is NO-GO.
- Docs: `mkdocs build --strict` when `docs/` or `mkdocs.yml` changed
- `SMOKE_COMMANDS`: `ousast --help`, and for a scan change `ousast scan <fixture> --mode quick` whose `manifest.json` names the stages that ran and the files preprocess kept. A plausible zero (0 findings, 0 degradations) from a tool that never read its input is a failed instrument, not a pass: prove the tool read its input first.
**Discover canonical validation commands** (for other repositories, or anything the list above does not cover):
- Inspect repository-local sources of truth in this order: project scripts/manifests (`package.json`, `pyproject.toml`, `go.mod`, `Cargo.toml`, app manifests), task runners (`Makefile`, `justfile`), CI/workflow files, existing e2e/integration configs, then `README*`
- Derive a feature-level validation set for this repo: `TEST_COMMANDS`, `BUILD_COMMANDS`, and `SMOKE_COMMANDS`
- Prefer commands already used by repo automation over ad hoc shell pipelines
@@ -61,7 +61,8 @@ Use the language specified in `spec.json`.
4. Check exit code, failure count, skipped scope, and missing coverage.
5. Reject claims that are broader than the evidence.
6. If mandatory validation cannot be completed, return `MANUAL_VERIFY_REQUIRED`.
7. Only then allow the claim.
7. Verify the instrument before you believe the measurement: prove the tool read its input (it opened one file and printed the size; the stage list names what ran). A plausible zero -- "0 findings, 0 errors, exit 0" from a tree the tool never read, or a Joern script that "finished" before a JVM could start -- is a failed instrument, not evidence. For pytest, judge only by the printed exit code (`pytest -q -p no:cacheprovider < /dev/null; echo "exit=$?"`), never by a summary line.
8. Only then allow the claim.
## Claim-Specific Rules
@@ -116,6 +117,7 @@ Return `NOT_VERIFIED` when:
| “Build should be fine because lint passed” | Lint does not prove build success. |
| “Tests passed and build succeeded, so it must run” | Type erasure, module loading, native ABI, and boot-time config issues can still fail at runtime. |
| “The feature is done because all tasks are checked off” | `FEATURE_GO` also requires coverage, integration, and design alignment. |
| “The scan found nothing, so the code is clean” | An unread input and clean code both report zero. Prove the tool read its input first. |
## Output Format
+123
View File
@@ -0,0 +1,123 @@
---
name: openultrasast-scan
description: OpenUltraSAST scan workflows - `ousast scan` in quick, standard or deep mode, the evidence ladder, `--fail-on` exit codes, recorded degradations, the run directory and `ousast mcp`. Use when running or planning a scan, reading a run directory, or wiring OpenUltraSAST into an agent.
---
# OpenUltraSAST Scan
Use this skill to run `ousast scan PATH --mode quick|standard|deep` from an agent and to read what
it wrote.
## Workflow
1. Confirm the target is a local checkout (a path, not a URL).
2. Run through the project entry point: `uv run ousast ...` inside the repository, or `ousast` from an
environment where it is installed. Never activate a virtualenv by hand.
3. Start with `--mode quick` unless the task needs the Joern model layer (`standard`) or sandboxed
regression snippets (`deep`). Check the prerequisites below first; a mode whose engine is missing
still exits 0 and records a degradation.
4. Read the printed `scan_id=` and `run_dir=` lines; the run directory is `.openultrasast/runs/<scan-id>/`.
5. Report the finding count, how many the verifier accepted, and **every** recorded degradation
(`manifest.json`, key `degradations`). No findings plus a degradation is not a clean scan: the scan
answered a smaller question than it was asked.
6. Never call a finding verified from its title or severity. `verification.json` holds the status.
## Commands
```bash
uv run ousast scan <repo> --mode quick
uv run ousast scan <repo> --mode standard --fail-on verified
uv run ousast scan <repo> --mode deep --fail-on worth-fixing
uv run ousast scan <repo> --mode quick --config openultrasast.toml
uv run ousast mcp # the MCP server over stdio, for agent hosts
```
## Modes and prerequisites
| Mode | Adds | Needs | Without it |
| --- | --- | --- | --- |
| `quick` | language-scoped pattern rules, entry-point hints, ranking, verification, score; deterministic | nothing | - |
| `standard` | the MAP stage: complexity map, semantic overlay, authorization obligations, the Joern model layer (one code property graph per repository; taint, guard-dominance and configuration questions), deterministic fusion | Joern (shipped in the Docker image, with `php-cli` for PHP); a provider key for the residual LLM question | `cpg_unavailable`; without a key the LLM question is not asked and only the graph's entailed findings are reported |
| `deep` | REGRESS: promoted candidates run as snippets in a Docker sandbox (no network, read-only source, non-root) and get a `triggerable` / `not_triggerable` / ... verdict in `verdicts.json` | a working `docker` | `sandbox_unavailable` |
Model stages read `DEEPSEEK_API_KEY` (every LLM call); with only `OPENROUTER_API_KEY` set, chat
calls fall back to OpenRouter. An exported shell variable wins over `.env` silently. `[models]
hunter` in the config enables the tool hunter in `standard`; it is off by default.
## Exit code: `--fail-on`
| Policy | Exit 1 when |
| --- | --- |
| `never` (default) | never |
| `findings` | any finding was reported |
| `verified` | any verification result is `verified` (accepted at `static_corroboration` or above) |
| `worth-fixing` | a deep-mode sandbox verdict is marked `worth_fixing` |
A degradation never changes the exit code. Read `manifest.json` for it.
## Recorded degradations
Each entry in `manifest.json` `degradations` is `{"stage": ..., "reason": ...}`. The ones to expect:
- `cpg_unavailable` (stage `model`): no Joern on the host; the model layer ran nothing.
- `sandbox_unavailable` (stage `regress`): no working `docker`; deep mode fell back to standard.
- `learning_endpoint_unavailable` (stage `model`): no provider key resolved. The graph's entailed
findings are still reported; the residual question is skipped. This is recorded in `manifest.json`
but has no line in `report.md`'s "What could not be analysed" section.
- `hunter_model_unavailable` (stage `map`): `[models] hunter` is unset, so the tool hunter did not run.
Recorded on every standard scan without that setting; it is expected, not an error.
- coverage facts from the graph, rendered in `report.md` under "What could not be analysed":
`cpg_empty`, `cpg_build_failed`, `cpg_sharded`, `files_unparsed`, `query_failed`,
`query_budget_reserved`, `query_too_expensive`, `budget_exhausted`, `regions_truncated`,
`region_failed`; plus `max_findings_exceeded` and `facts_unavailable` (overlay, obligations).
Prove the instrument read its input before believing a zero: `manifest.json` names the stages that
ran and the files the preprocess step kept (`preprocess/file_targets.json`).
## Evidence ladder
`src/openultrasast/verification.py` defines the levels in this order. A level moves one rung at a
time and never backward.
| Level | Meaning | Produced today by |
| --- | --- | --- |
| `suspicion` | a rule, model or weak signal raised a hypothesis | the Joern model layer (its rung, `model_entailed` or `suspicion`, travels in the finding's tags), the tool hunter, authorization-obligation checks |
| `static_corroboration` | deterministic code evidence at the line | quick rules, the semantic overlay |
| `crash_reproduced` | an artifact-backed reproducer | nothing: there is no crash reproducer. Deep-mode sandbox verdicts (`verdicts.json`, `worth_fixing`) do not raise the level |
| `root_cause_explained` | independent verifier reasoning | nothing: the LLM judge was retired in v2.0.0 |
| `exploit_demonstrated` | a demonstrated exploit | nothing |
| `patch_validated` | relevant checks re-run after a patch | nothing: there is no patch stage |
The verifier's status per finding (`verification.json`): below `static_corroboration` is
`needs_evidence`; `static_corroboration` with reachability `unknown` is `rejected` (the tie-breaker
asks for call-graph, route, CLI, parser or dynamic evidence); `static_corroboration` with
function-level reachability is `accepted`. `verified` means accepted at `static_corroboration` or
above, and that is what `--fail-on verified` counts. There is no configurable report threshold: the
`[evidence]` config section was retired in v2.0.0 and is ignored.
## Run artifacts
Under `.openultrasast/runs/<scan-id>/`:
- `report.md`, `report.sarif` (SARIF 2.1.0 for code hosts and IDEs), `manifest.json` (stages, degradations, model payload)
- `findings.json`, `verification.json`, `score.json`, `shadow_findings.json` (rules shipped as `shadow`)
- `harness.json`: the resolved harness configuration the run used
- standard: `fusion.json`, `complexity_map.json`, `overlay.json`; deep: `verdicts.json`
- `calibration/false_positive_learnings.json`, `rank/ranking.json`, `mapping/`, `preprocess/`, `trace/events.jsonl`
## Changed code only
`ousast pre-push --base <rev> --head <rev> --artifact <file> <repo>` replays a revision pair and
reports only new or worsened defects (experimental; `--mode advisory` is the default and never
blocks). The capability registry is empty, so every candidate is diagnostic.
## MCP
`ousast mcp` serves a narrow MCP server over stdio (newline-delimited JSON-RPC) with ten tools:
`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`,
`explain`, `propose_patch` (degrades visibly) and `export_report`. No tool runs a shell, Docker or a
free-form command.
```jsonc
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
```
@@ -0,0 +1,75 @@
---
name: openultrasast-triage
description: OpenUltraSAST finding triage - false-positive adjudication with the fixed reason taxonomy, the evidence ladder, what a verdict may feed (labels, never rule edits), and fixing then re-scanning. Use when reviewing findings, reducing false positives, deciding whether evidence is enough, or auditing a fix for a finding.
---
# OpenUltraSAST Triage
Use this skill when triaging OpenUltraSAST findings or auditing a fix for one.
## Triage Loop
1. Identify the finding: id, path, rule, CWE and policy severity, confidence, evidence level and
verification status (`findings.json`, `verification.json` in the run directory).
2. Separate deterministic evidence (rule match, graph entailment, attached reachability) from model
assumptions.
3. Check attacker control, reachability, sanitizer behaviour, duplicates, impact, and contradictory
evidence.
4. If rejected or unresolved, record the false-positive reason and its scope.
5. Prefer scoped demotion over deleting a vulnerability class globally: a rejection in `auth/`
demotes `auth/...`, never the class (`calibration.py`).
6. Record the adjudication. Do not edit a rule because of one finding (see below).
## False-Positive Reasons
The taxonomy is `FalsePositiveReason` in `src/openultrasast/calibration.py`; use these values exactly:
- `unreachable_path`
- `missing_attacker_control`
- `sanitizer_disproved`
- `static_rule_mismatch`
- `incorrect_model_assumption`
- `duplicate`
- `insufficient_impact`
- `unsupported`
- `contradicted`
- `unverified`
## Decision Standard
If evidence is uncertain, mark the finding `needs_evidence` and name the next evidence instead of
escalating severity. Fusion (`fusion.json`, standard mode) is deterministic: it adjudicates between
the quick-rule and MAP panels by fixed triggers and is not a second opinion from a model. Independent
model agreement exists only in the plane's verify passes a and b, with pass c on disputes, and the
`agree` step; those verdicts arrive as features of the decision engine, never as a verification
status.
## What an Adjudication Feeds
- Recorded adjudications (a per-finding verdict with a reason) become decision-engine labels through
the fail-closed source list `src/openultrasast/learn/sources.toml` (`[[source]] id =
"adjudications"`). Aggregate counts label nothing; only per-finding verdicts do.
- Plane verdicts (`agreed`, `rejected`, `disputed`) are features, never labels. "Rejected by the
plane" is not a negative.
- Rule changes go through the ledger: `ousast improve` (the validator and the detection gate, holdout
pairs per round) or a registered experiment (`ousast learn experiment register|units|run|analyse`).
Never a hand edit of `rules.toml` per miss or per false positive.
## Fix and Re-scan
1. Intake: restate the finding id, evidence level, affected path, and the acceptance criterion.
2. Plan: the smallest defensive change that addresses the root cause.
3. Implement: modify only the files the finding requires.
4. Adversarial review: look for bypasses, regressions, false assumptions, and unrelated edits. Use a
differential review for changes in auth, crypto, config, parsers, and sandbox behaviour.
5. Reconcile: fix each review finding or document why it does not apply.
6. Verify: a re-scan of the checkout in the same mode, or a replay of the change with
`ousast pre-push --base <before> --head <after> --artifact <file> <repo>`, shows the finding gone
and no new findings. That re-scan is the evidence. Nothing in the tool records `patch_validated`;
do not claim it.
Guardrails:
- Do not rewrite modules when a local fix is enough.
- Do not change public APIs unless the vulnerability requires it.
- Do not call a finding fixed from model reasoning alone; the re-scan or replay decides.
+1 -2
View File
@@ -6,9 +6,8 @@
DEEPSEEK_API_KEY=
# DEEPSEEK_BASE_URL=https://api.deepseek.com
# Embeddings only: OpenAI text-embedding-3-small through OpenRouter.
# Embeddings only (`ousast learn memory embed`, default openai/text-embedding-3-small through OpenRouter).
OPENROUTER_API_KEY=
OPENROUTER_EMBEDDING_MODEL=openai/text-embedding-3-small
# Plane memory store. Default is a local FileStore under ~/ousast-results/plane/memory.
# For an S3-compatible server (RustFS is the tested one): OUSAST_MEMORY=s3://<bucket>[/<prefix>], or
+43
View File
@@ -0,0 +1,43 @@
name: Docs
# Builds the MkDocs site and publishes it to GitHub Pages on every push to main that touches it.
on:
push:
branches: [main]
paths: ["docs/**", "mkdocs.yml", "README.md", ".github/workflows/docs.yml"]
workflow_dispatch:
permissions:
contents: read
pages: write
id-token: write
concurrency:
group: pages
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: astral-sh/setup-uv@v7
with:
enable-cache: true
- uses: actions/setup-python@v6
with:
python-version-file: .python-version
- run: uv sync --extra docs
- run: uv run mkdocs build --strict
- uses: actions/upload-pages-artifact@v3
with:
path: site
deploy:
needs: build
runs-on: ubuntu-latest
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- id: deployment
uses: actions/deploy-pages@v4
+42
View File
@@ -34,3 +34,45 @@ jobs:
generate_release_notes: true
prerelease: ${{ contains(github.ref_name, '-') }}
make_latest: ${{ !contains(github.ref_name, '-') }}
images:
# The scan image (Joern + php-cli, `docker run ... scan`) and the plane's task runner, pushed to GHCR under the
# tag's version. Pin them by the digests this job prints, not by tag: Agent Substrate rejects tags.
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
strategy:
fail-fast: false
matrix:
include:
- name: openultrasast
dockerfile: Dockerfile
- name: openultrasast-runner
dockerfile: plane/Dockerfile.runner
steps:
- uses: actions/checkout@v5
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository_owner }}/${{ matrix.name }}
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}},enable=${{ !contains(github.ref_name, '-') }}
type=raw,value=latest,enable=${{ !contains(github.ref_name, '-') }}
- id: build
uses: docker/build-push-action@v6
with:
context: .
file: ${{ matrix.dockerfile }}
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
- run: |
echo "ghcr.io/${{ github.repository_owner }}/${{ matrix.name }}@${{ steps.build.outputs.digest }}" >> "$GITHUB_STEP_SUMMARY"
+2 -2
View File
@@ -1,9 +1,9 @@
{
"feature_name": "ai-service-plane",
"created_at": "2026-09-29T15:40:00Z",
"updated_at": "2026-09-29T17:55:00Z",
"updated_at": "2026-10-02T00:00:00Z",
"language": "en",
"phase": "implementation",
"phase": "implementation-complete",
"approvals": {
"requirements": {
"generated": true,
@@ -1,5 +1,7 @@
# Requirements Document: contributor-scan
> **Status 2026-10-02:** paused since 2026-09-23 (its last change). 27 of 51 tasks done: the MAP integration was delivered and `v1.2.0-alpha.1` rolled out NO-GO (`census-repair-verification.md`); the remaining tasks wait on `pre-push-safety-net` M2/M3 and nothing here moved after the ax plane landed. Not active.
## Introduction
`model-grounded-detection` built an arbiter and measured it: 41.3% of the corpus reaches `model_entailed`
@@ -1,5 +1,7 @@
# Requirements Document: finding-feedback-loop
> **Status 2026-10-02:** dormant. Requirements approved and design generated on 2026-09-09; the design was never approved and no tasks exist. It depends on `contributor-scan`, which is paused; reopen it with that spec.
## Introduction
Nine engine defects were found in one session against two repositories. **Three were the same bug in three
@@ -119,6 +119,19 @@ rule hits.
2. Without a trained model available the tools fall back to today's behaviour and say so in the report.
3. Hand-written rules remain as signal producers; changing their enabled/shadow status no longer changes what is
reported unless the engine's decision changes.
4. *(Amended 2026-10-03, maintainer decision; refines the BLOCK rule of 7.3 and 7.4. Tested by exp-004.)* BLOCK
precision is demonstrated pooled across families, not per family:
- **Normalisation**, declared before any run: each family's raw score is put on one scale by a per-family Platt
map (`learn.calibrate.fit_map` with its floors) fitted on training folds only. A family the training folds
cannot map does not block on that fold.
- **t_block**: one threshold on the normalised score for all families, chosen on training folds as the lowest
one whose Wilson 95% lower bound on precision, over the training flags of all families together, is >= 0.95.
It is reported by the Wilson 95% lower bound over all held-out flags of all families, before the floor.
- **Family floor**: a family may block only if its own held-out flags have zero errors and number at least F
(F = 10, reasoned in the exp-004 registration); otherwise that family stays ADVISORY.
- **t_quiet**: per family, the highest threshold keeping >= 90% of the paired positives at or above it (chosen
on training folds, its held-out recall reported); findings below it are silent.
- No held-out label moves a threshold or a normalisation map; candidates whose verdict is `unsure` never block.
### Requirement 7: Built for unseen repositories -- a repo safety net
@@ -136,7 +149,8 @@ rule hits.
ADVISORY, trading precision for recall; both reported with out-of-repository intervals. The decision unit is a
candidate in changed code (`pre-push` delta), with whole-repository scans as a secondary mode.
4. Calibration is checked on held-out repositories; where it does not hold, only ADVISORY is offered for that
family.
family. *(Amended 2026-10-03: once exp-004 adopts the pooled gate of 6.4, BLOCK precision is shown by its held-out
bound and family floor, and calibration is reported, not required.)*
5. Any adaptation to a user's own repository (facts reuse, their dismissals) is opt-in, separate from the
generalisation model, and never contributes to its reported numbers.
+35 -6
View File
@@ -116,12 +116,41 @@ a model or embedding call.
- _Requirements: 5.1, 5.2, 8.3_
- [ ] 7. Experiments (exp-001 and program variants need budget go-ahead)
- `learn/experiments.py`, `register`, `experiment-outcome`, Run annotation checks, `OUSAST_VERIFY_CALLERS`;
exp-001 (known-callers line on/off, ~$7.7 expected, ceiling $15) registered, run and recorded -- an
inconclusive result is pre-registered as likely.
- Program-variant experiments (paired on candidates, repository cluster, arm A replayed from cache at $0): the
machinery and its test; the first variant (`k = 5` vs `k = 1`, or the confidence source) **(budget go-ahead:
~$0.9-1.35 per arm-B pass of 300 candidates, ceiling $2 each)**.
- [x] 7.1 Machinery (no model in tests): `learn/experiments.py` -- YAML manifests under `plane/experiments/<id>.yaml`
(hypothesis, arms as program-spec overrides or plane `task_env`, unit, pairing, repository cluster, metrics with
`[label=]` selectors and targets, MDE, two looks with O'Brien-Fleming boundaries, budget ceiling, families, folds,
frozen units file), `register` (`experiment` row with both digests; refuses an uncommitted, modified or re-declared
manifest), `units` (freeze the families' evaluation candidates, paired by candidate across pins; no model), `run`
(paired arms in a seeded repository order, a seeded coin per unit for which arm goes first, `arm_outcome` rows,
resumable, arm A replay-only at $0, canary replicates under a fresh cache salt, the first look during the run),
`analyse` (paired repository bootstrap, exact McNemar, the looks, adopt / reject / equivalent / inconclusive,
`experiment_result` row + `benchmarks/experiments/<id>/result.json`), `ousast learn experiment
register|units|run|analyse`; `OUSAST_VERIFY_CALLERS=off` in `plane/tasks/verify.py`; the retrieval-ensemble
sampling variant (`ProgramSpec.sampling = retrieval_ensemble`: `k` samples at temperature 0 over `k` disjoint
retrieved example sets; default unchanged). Tests: `test_learn_experiments.py` (18, scripted clients, FileStore).
- [x] 7.2 Registered, not run: `exp-001-known-callers` (plane arms, `flag_rate[label=1]`, MDE 0.15, ceiling $15;
units `pending` -- model-free candidate generation on the 41 development repositories is not built, and plane Run
annotation checks / the `experiment-outcome` task are not written) and `exp-002-retrieval-ensemble` (arm A today's
k = 5 temperature sampling, arm B the retrieval ensemble; primary canary agreement (target 0.90) and within-pair
AUC, secondary usd per candidate, AUC, ADVISORY recall/precision; families injection and access_control on the
slice-2 candidates; ceiling $4: A $1, B $3). Both registered in `s3://sast-memory` at fbc3220
(`benchmarks/experiments/<id>/registration.json`). exp-002's units are `pending` too: on 2026-10-02 every example
row in the store carried null plane-only instruments (`agree`, `model_sinks`, `verify`) and `load_examples` raised
on them, so `ousast learn experiment units` could not freeze the candidates.
- [ ] 7.3 exp-001 run and recorded (~$7.7 expected, ceiling $15) -- an inconclusive result is pre-registered as
likely; needs the frozen units file first.
- [x] 7.4 exp-002 run and recorded 2026-10-02 (budget go-ahead ~$2 expected, ceiling $4): **reject B**. 160 paired
units (injection 119, access_control 41; 56 repositories; 108 units in 54 candidate pairs), both arms, two
replicates each, no early stop. Primary: canary agreement A 0.875 -> B 0.906 (diff +0.03, CI [-0.03, +0.10],
spans; B meets the 0.90 target), within-pair AUC A 0.815 -> B 0.741 (diff -0.07, CI [-0.13, -0.02], **against B**).
Secondary: AUC -0.02 (CI [-0.04, -0.003], against), ADVISORY recall -0.06 (spans; McNemar 7/2, p = 0.18),
precision +0.01 (spans), usd per candidate A $0.0021 -> B $0.0051 (CI [+0.0027, +0.0032], against: the five
disjoint example sets share only the prompt prefix in DeepSeek's cache). Priced usage A $0.68 (1206 paid calls,
395 replayed), B $1.63 (1350 paid, 250 replayed). Result: `benchmarks/experiments/exp-002-retrieval-ensemble/result.json`,
`benchmarks/measurements/2026-10-02-exp-002/record.json`. Three repairs on the way (each its own commit): example
rows' null plane-only features (1f525ff), the 788 embedding vectors absent from RustFS (copied from the file
store), the result row's column-type collision under S3 Select (780caaa). Further program variants (`k = 5` vs
`k = 1`, the confidence source) as new manifests.
- _Requirements: 3.6, 4.1-4.4_
- [ ] 8. Integration
+33
View File
@@ -0,0 +1,33 @@
# Brief: the plane on a Kubernetes cluster
Maintainer, 2026-10-02: "the plan is to move the whole to a kube system soon. what is in the way?" and earlier
"later i will deploy this in a separate kube env."
## Where we are
The plane already runs on google/ax over Agent Substrate, on a single-node kind cluster on the maintainer's
laptop (`ops/ax/`). Its manifests are ax's own kinds, every task is one digest-pinned runner image, the memory
store is external (S3-compatible RustFS), and provider credentials travel only in the start request. What still
assumes the laptop is small and listed in `docs/deployment.md` and `ops/ax/README.md`:
1. the reconciler is a host CLI process (`ousast plane run`);
2. the artifact receiver is a thread of that process, reached through a Service + EndpointSlice to the kind
gateway; the router is reached through a kubectl port-forward;
3. the kube context `kind-ousast` and the registry `localhost:5001` are hard-coded in `doctor.py`, `egress.py`,
`router.py`;
4. images come from the kind registry;
5. ax's deploy manifest points `AX_SNAPSHOTS_BUCKET` at the ax authors' bucket;
6. the Joern engine runs as a host Docker container (2.8 GB image, 3 GB memory), not as an ax task;
7. the worker pool is two 1 CPU / 1.5 GiB workers with no autoscaling.
## Direction
Make the plane deployable to any Kubernetes cluster that runs Agent Substrate and ax, prove every change on the
kind cluster first (it already behaves like the target), and leave nothing that only works on this host. The
production use (a pre-push safety net fed by CI) needs, in addition, a delta Run command and Workspaces from a
URL and commit; those are in scope as the last increment because they define what the cluster is for.
## Not in scope
Provisioning the maintainer's cluster, Substrate's and ax's own deployment procedures (followed, not changed),
and any detection change.
+411
View File
@@ -0,0 +1,411 @@
# Design Document
## Overview
The plane keeps its shape: `ousast plane run` is the reconciler, ax is the only executor, every task is one
digest-pinned image under the runner contract, memory is the S3 store. What changes is where the reconciler's
peers live and how they are named. Three facts from the code and the upstream checkouts decide the design:
- **Every host assumption is an address or a process in four places.** `kind-` + `KIND_CLUSTER_NAME` is built in
`doctor.py:39`, `egress.py:115`/`:148` and `router.py:145`; `localhost:5001` in `doctor.py:45-47` and
`up.sh:10`; the receiver is a thread of the CLI (`reconciler.py:350-352`) reached by an EndpointSlice to the kind
gateway (`receiver-service.yaml.tmpl`, `up.sh:55-57`); the router and ax-server are reached by `kubectl
port-forward` (`router.py:146-148`; ax's own CLI does the same, `ax/internal/tunnel/tunnel.go:180-238`).
- **The store is already the one thing every profile shares.** `S3Store` is verified at every open
(`memory.py:608-640`), its `runs/` prefix expires after 30 days (`RUNS_RULE_ID`, `memory.py:589-591`), and no
task reaches it today. A presigned URL is a credential, and the plane already has a channel for credentials that
never touches a manifest or the golden boot: the start request (`router.py`, `start_task`, `runner.py:219-275`).
- **ax-server authenticates nobody; Kubernetes does.** `deploy/ax-server.yaml` runs `--addr=:8080` with no auth
flag; the `ax` CLI tunnels to `svc/ax-server` through kubectl; `kubectl-ate` port-forwards to `ate-api-server`
and sends a Kubernetes ServiceAccount token (`cmd/kubectl-ate/internal/cmd/root.go:59-60`). So the only
authenticated door into either control plane is the Kubernetes API with a kubeconfig.
Hence (maintainer, 2026-10-02): the reconciler stays the CLI and gains an execution mode. `local` is the kind
profile as today; `remote` is the same `ousast plane run` (later `ousast plane scan`) from a laptop or a CI job
against ax in k3s, with task artifacts delivered to the S3 store through presigned URLs minted by the CLI and
carried in the start request, and the CLI polling the store for completion. No in-cluster reconciler is needed
for the move; the receiver thread goes in both profiles once the store path passes its kind proof.
### Goals
- Two profiles of one code base, `kind` and `k3s`; every change proven on kind first (Req 2.4, 7.1).
- No module names a host (`kind-ousast`, `localhost:5001`, `172.19.0.1`); a test enforces it (Req 1.3).
- One delivery path (the store) in both profiles; one credential channel (the start request).
- The Joern engine as a plane task on its own pool (Req 4); a delta Run for a push (Req 6).
### Non-Goals
Provisioning k3s; changing Substrate or ax (their manifests are applied with our values, not edited in place);
any detection, scoring or population change; an unattended in-cluster reconciler (deferred, section 2.4).
## Architecture
```mermaid
flowchart LR
subgraph laptop ["kind profile: laptop"]
cli1["ousast plane run --exec local"]
env1[".env: provider + S3 keys"]
end
subgraph kind ["kind cluster (same host)"]
ax1["ax-server (ax-system)"]
sub1["Substrate: atenet-router, atenet-egress, ate-api"]
pool1["ousast-pool: 2 gVisor workers"]
epool1["engine-pool: 1 worker, 4 GiB"]
reg1["kind registry: images by digest"]
end
s3[("S3 store (RustFS, external): runs/, repos/, facts/")]
cli1 -- "ax CLI tunnel; kubectl-ate; router port-forward" --> ax1
cli1 --> sub1
sub1 --> pool1
sub1 --> epool1
pool1 -- "presigned PUT/GET via egress (TLS 443)" --> s3
epool1 --> s3
cli1 -- "mint URLs, poll, ingest" --> s3
reg1 -.-> pool1
```
```mermaid
flowchart LR
subgraph ci ["k3s profile: laptop or CI job"]
cli2["ousast plane run|scan --exec remote"]
kc["kubeconfig: SA ousast-ci token"]
sec["CI secrets: provider + S3 keys"]
end
subgraph k3s ["k3s cluster"]
api["Kubernetes API (RBAC: portforward in ax-system, ate-system)"]
ax2["ax-server"]
sub2["Substrate router / egress / ate-api"]
pool2["ousast-pool + HPA"]
epool2["engine-pool + HPA"]
end
ghcr["GHCR ghcr.io/norandom: runner + engine images by digest"]
s3b[("S3 store (RustFS)")]
cli2 -- "port-forward tunnels only" --> api
api --> ax2
api --> sub2
sub2 --> pool2
sub2 --> epool2
pool2 -- "presigned PUT/GET" --> s3b
epool2 --> s3b
cli2 --> s3b
ghcr -. "imagePullSecret if private" .-> pool2
```
The two diagrams differ in the kubeconfig principal, the registry, the pool sizes and the HPA. Nothing else.
New code: `plane/profile.py` (the profile), `plane/delivery.py` (presign, poll, extract), `plane/k8s.py`
(manifest rendering per profile), `plane/tasks/engine.py`, `plane/scan.py` (Req 6). `reconciler.py` loses the
receiver wiring and gains a `Delivery` dependency; it stays under the 500 lines `test_plane_reconciler.py:803`
enforces because every new concern is a sibling module, as `router.py` and `egress.py` already are.
## Components and Interfaces
### 1. Configuration surface: `PlaneProfile` (Req 1)
**Decision.** One frozen dataclass `PlaneProfile` in `plane/profile.py`, loaded by `load_profile()` from a TOML
file named by `OUSAST_PLANE_PROFILE` (a name resolved under `ops/k8s/profiles/<name>.toml`, or a path; default
`kind`), with every field overridable by an environment variable of the same upper-cased name
(`OUSAST_KUBE_CONTEXT`, `OUSAST_REGISTRY`, ...). Profiles hold no secret: keys stay in the environment or
`.env`, as today.
| Field | `kind.toml` | `k3s.toml` | Replaces |
| --- | --- | --- | --- |
| `exec` | `local` | `remote` | the implicit host mode |
| `kube_context` | `kind-ousast` | `ousast-k3s` (the maintainer's context name) | `"kind-" + KIND_CLUSTER_NAME` in three modules |
| `registry` | `localhost:5001` | `ghcr.io/norandom` | `doctor.py:45`, `up.sh:10` |
| `images` | `kind-images.json` | `images.json` (from the release, section 8) | the `image:` literals in `plane/tasks/*.yaml` |
| `router_url` | `""` (port-forward) | `""` (port-forward) | `OUSAST_ROUTER_URL` stays as the test override |
| `ax_server` | `""` (ax CLI tunnel) | `""` | `$AX_SERVER` stays as ax's own override |
| `memory` | `s3://sast-memory` | `s3://sast-memory` | `OUSAST_MEMORY` default `file://` is removed for the plane (no fallback) |
| `atespace` | `default` | `default` | `generate.ATESPACE` |
| `snapshots_bucket` | `s3://ax-snapshots` | `s3://ax-snapshots` | ax's `gs://dberkov-gke-dev3/ate-env/` (rendered into our ax-server patch) |
| `pools.default`, `pools.engine` | `{replicas: 2, cpu: 1, memory: 1536Mi}`, `{1, 2, 4Gi}` | documented production values | `workerpool.yaml.tmpl` |
| `image_pull_secret` | `""` | `ghcr-pull` | none today |
**Why a file plus env, not env only.** Eleven values change together per cluster; env-only has no reviewable
record of what "k3s" is (the `ops/ax/README.md` table at lines 193-204 already struggles to list them), and a
CI job would carry eleven exports. The env override keeps the existing tests (`OUSAST_ROUTER_URL`,
`OUSAST_ARTIFACT_HOST`) and one-off experiments working. **Rejected:** a `[plane]` section in the scan config
TOML (the scan config is a user's tool configuration; the plane is a maintainer surface and must not leak into
`ousast scan`); a Kubernetes ConfigMap (there is no in-cluster reconciler to read it).
`doctor()` takes the profile: it checks `kube_context` reachability, the two namespaces, that the `images` file's
runner reference resolves (`docker manifest inspect` for GHCR, the `/v2/` catalog for a local registry), that
`memory` opens (`open_store` verifies the bucket), and prints each configured address (Req 1.2). The literal scan
(section 9) covers `src/`, `ops/`, `plane/` and `docs/` except the profile files themselves and
`ops/ax/up.sh`, which may name kind because it creates it.
### 2. Execution mode: the CLI with `--exec local|remote` (Req 2)
**Decision.** `ousast plane run RUN.yaml [--exec local|remote]` (default from the profile). Both modes run the
same `reconciler.run`: render, `ax apply`, egress policy, resume, start, await, delete. What differs is
addressing and principal, and none of it is new client code:
| Call | Through | Principal in `remote` |
| --- | --- | --- |
| `ax apply/resume/get/delete` (`reconciler.Ax`) | ax's own tunnel: `kubectl port-forward -n ax-system svc/ax-server` (`tunnel.go:203-238`), context from the profile | kubeconfig for SA `ousast-ci` |
| `kubectl ate create/get/update egress-policy` (`egress.Egress`) | kubectl-ate's own port-forward to `ate-api-server`, bearer = the SA token (`root.go:59-60`) | same SA; Substrate's authorization of that token for `egress-policy` is an open question (below) |
| `POST /ousast/v1/start` (`router.open_router`) | `kubectl port-forward svc/atenet-router` in `ate-system`, as today | same SA |
| artifacts, inputs, state | the S3 store (section 3) | the store's agent key from the CI secrets |
**Auth: kubeconfig with a ServiceAccount token, not an ingress.** ax-server has no authentication
(`deploy/ax-server.yaml:71-79`: `--addr=:8080`, nothing else) and Substrate's router routes on a header. Exposing
either through Traefik would publish an unauthenticated control plane behind whatever middleware we bolt on;
tunnelling through the Kubernetes API keeps one authenticated, audited door and uses the clients as they ship.
**Rejected:** Traefik IngressRoute for ax-server and the router (unauthenticated upstreams, a second auth layer to
design, and ax's CLI would still tunnel); a bespoke ax HTTP client (Req boundary: no replacement of the `ax` CLI
beyond what addressing needs, and the tunnel already handles addressing).
**RBAC (`ops/k8s/base/rbac-ci.yaml`).** ServiceAccount `ousast-ci` in `ax-system`; Role in `ax-system` and in
`ate-system`: `pods` get/list, `services` get, `pods/portforward` create; nothing cluster-wide, no verbs on
Substrate's or ax's CRDs (both are reached through their servers, not through the Kubernetes API). The token is a
bound TokenRequest (`kubectl create token ousast-ci --duration=...`) stored as the CI secret `KUBECONFIG`. Secrets
for the provider and the store stay where the reconciler runs: `.env` on a laptop, repository secrets in CI,
Kubernetes Secrets only on the day an in-cluster reconciler exists (2.4). The start-request path is unchanged
(Req 2.3).
**Run state of record moves to the store.** `state.json` is written locally under `$OUSAST_RESULTS/plane/<run>/`
as today and mirrored to `runs/<run>/state.json` on the store after each change; a rerun from another machine
seeds its local copy from the store. This is what lets CI job N+1 resume CI job N's Run (Runs are long,
resumable and artifact-driven: `reconciler.py:272-312`).
**Kind keeps `local` (Req 2.4).** Same command, same manifests, local registry, laptop as reconciler; the only
differences are the profile's values. The kind proof of each increment is run in `local`, and increment 3 runs
`remote` against kind from a container that holds only the kubeconfig and the secrets (section 7).
#### 2.4 Deferred: an in-cluster reconciler
Not needed for the move. It would add: survival of a laptop sleeping or a CI job timing out mid-Run, scheduled
Runs, and Secrets injected by Kubernetes. It would cost: a workload (Job per Run is the natural shape, keeping
the CLI path identical), a PVC or store-only state, a submit path, and RBAC for ax's and Substrate's Services from
inside the cluster. Revisit when unattended Runs are wanted; the store-backed state above is its prerequisite.
### 3. Delivery: presigned URLs in the start request, both profiles (Req 3)
**Decision.** The runner PUTs `output.tar` to a presigned S3 URL and GETs each declared input from a presigned
URL; the CLI mints them and polls the store. The URLs travel in the start body, never in env:
```json
{"run": "...", "task": "...", "credentials": {"DEEPSEEK_API_KEY": "..."},
"delivery": {"put": "<presigned PUT runs/<run>/<task>/output.tar>",
"inputs": {"CANDIDATES": "<presigned GET runs/<run>/<producer>/<artifact>>"}}}
```
- **Why the start body.** A presigned URL is a bearer capability, so the credentials-only-in-the-start-request
rule applies; env would bake it into the ActorTemplate and hand it to the golden boot; and the body has no
32 KB ceiling (Substrate rejects env values over 32768 characters, `reconciler.py:141`). `StartGate.offer`
(`runner.py:250-275`) validates `delivery` the way it validates `credentials`: `https` URLs only, hostnames,
no echo. Expiry = `OUSAST_TASK_TIMEOUT` (7200 s) + 600 s; a re-start after a re-resume re-mints.
- **Why presigned rather than a scoped delivery credential.** A per-task S3 key needs a per-task policy on RustFS,
gives the sandbox `ListBucket` and reads beyond its inputs, and would travel in the same body. Presigned URLs
name exactly the objects a task may write and read, which is what the receiver enforces today
(`Receiver.input_file`, `egress.py:230-247`).
- **Why not an in-cluster receiver Service.** It is a server to run, holds state in a process, and is reachable
only from a pod; that is the gap remote mode has. **Rejected.**
- **Objects.** `runs/<run>/<task>/output.tar` (the runner's PUT); after the CLI extracts it, the declared
`outputs[]` are re-put as `runs/<run>/<task>/<artifact>` so a consumer's presigned GET names one artifact, and
`runs/<run>/<task>/done.json` carries the summary's status. The 30-day `runs/` expiry already applies; memory
rows are ingested from the local extraction by `remember` as today (f: annotations and `memory_key` seeding
unchanged, `generate.py:_chain_inputs`, `memory.seed`).
- **Completion.** `_await`'s `delivered()` becomes `Delivery.delivered(task)`: a HEAD on `output.tar`, polled at
`OUSAST_POLL_SECONDS` (3 s). The runner's completion marker logic (`runner.py`, `_run_once`) is unchanged.
- **Inputs no task produces** (`candidates.json`, `functions.json`, `case.json`) may also be store objects
declared as inputs, which lifts the ~20 KB inline-files limit (`ops/ax/README.md:117`); Workspace `files`
stay for the smoke Task and small cases.
**One path for both profiles (d).** The store is required and verified in both; keeping the receiver for kind
would keep `receiver_address`, `OUSAST_ARTIFACT_DIAL`, the `Host`-dial trick and the EndpointSlice alive for one
profile and leave the kind proof proving a path k3s never runs. The receiver thread, `receiver-service.yaml.tmpl`,
`receiver_cluster_ip` and the `up.sh` step go after increment 2's proof (the `harnessx-removal` sequencing rule:
nothing deleted before its replacement passes). Tests that used `OUSAST_ARTIFACT_HOST` move to a local HTTP PUT
target that accepts presigned-shaped URLs.
**Egress per task (c).** `policy_for` drops the `http` rule and emits one `tls_passthrough` rule on 443 for the
sorted union of the store endpoint's hostname (`urlsplit(S3_ENDPOINT).hostname`, IPs rejected as today), the bound
Workspaces' Git hosts (`git_hosts`) and the bound Model's `openultrasast.io/egress-hosts`. A model-free task's
policy names the store and its Git hosts only. The gateway already passes TLS to the Model host by hostname, so
the store host is the same mechanism (`egress.py:7-16`). Open question: the store endpoint must be a DNS name
the gateway resolves; `files.because-security.com` is; an in-cluster RustFS would need a resolvable Service name.
### 4. The engine as a plane task (Req 4)
**Decision.** `plane/tasks/engine.py` runs in the Joern image under the runner contract, on a second WorkerPool
`engine-pool`. The root `Dockerfile` gains the two lines `Dockerfile.runner:25-29` has (the `ax-task-runner`
shim and `/workspace`), so one *contract* in two images: `ousast-runner` (slim, PyYAML) and `ousast-engine`
(JRE + Joern + php-cli, ~2.8 GB). **Rejected:** Joern inside the single runner image (every task would pull
2.8 GB and every golden snapshot carry it; the engine runs for a minority of cases); Docker-in-pod (no socket on
containerd; `Dockerfile:1-7` already argues against it).
- **Task.** Template `plane/tasks/engine.yaml`: `command: ["engine"]`, env `OUSAST_WORKSPACE_DIR`,
`OUSAST_FIXED_DIR`, `OUSAST_INPUT_CASE`, pins and `OUSAST_FAMILIES`, bound to the same two Workspaces the
`alerts` task binds (`generate.py:342-353`). The module lifts `benchmarks/push/finding_dump.py`'s scan into
`openultrasast.cpg.dump` (the image ships `src/`, not `benchmarks/`) and reuses `engine_alerts.engine_rows`,
`read_record` and the `alerts` helpers (`scan`, `coverage`, `write_rows`, `write_summary`). It emits the same
`alerts.jsonl` rows (`rule_id: engine:<family>`, `source: "engine"`) and `summary.json` (coverage with the
engine's languages, files and bytes read, questions, seconds, degradations per pin) as `alerts-engine`, with
`host: false`. The instrument rule stays: no `read N files` line or `questions == 0` fails the task loudly.
- **Sizing from the record** (`benchmarks/measurements/2026-09-30-php-engine-alerts/record.json`): container
peak 2.129 GiB under a 3 GiB limit, 789 s for both pins of froxlor, deadline 1800 s per pin. Task resources:
requests 3 Gi / 1 CPU, limits 4 Gi / 2 CPU; `engine-pool` workers 4 Gi / 2 CPU, replicas 1 on kind (the 7 GB
host holds it only with `ousast-pool` at one worker), production replicas in `k3s.toml`. Task timeout: two pins
times 1800 s plus grace fits `OUSAST_TASK_TIMEOUT` 7200.
- **Placement is the open question of this increment.** ax's templates set no worker selector
(`workerpool.yaml.tmpl:1-3`), so Substrate picks any free worker of the sandbox class. The kind proof must show
a 3 Gi actor lands only on `engine-pool`. If Substrate does not place by resource fit, the fallback is a
separate atespace with its own pool if Substrate scopes pools that way; failing that, the engine stays
`alerts-engine` on a Docker host and the k3s engine becomes a plain Kubernetes Job outside ax, recorded as such.
**Probe outcome (task 1.5, 2026-10-02, `benchmarks/measurements/2026-10-02-k8s-01-profile/`): `placement: any`.**
With `ousast-pool` at one 1536Mi worker and a `probe-pool` of one 3Gi worker both ACTIVE, a `repo-facts` actor
requesting 2560Mi (limit 3Gi) was placed on the 1536Mi `ousast-pool` worker (`kubectl ate get workers`: ACTORS
1/1000 there, 0/1000 on `probe-pool`), and so was the 256Mi actor. Substrate does not place by resource fit, so
group 5 takes the fallback: the engine pool must be the only pool its actors can land on. Before 5.2, check whether
Substrate scopes pools by atespace (an `engine` atespace with `engine-pool` as its only pool); if it does not, the
k3s engine runs as a plain Kubernetes Job outside ax and `alerts-engine` stays the kind path.
- **`alerts-engine` stays** for development on a host with Docker (kind profile), its help text and
`docs/plane.md` row marked "development only; the plane task is `engine`" (Req 4.3). Generated Runs choose
`engine` when the profile's `images` file has an engine image, else leave `alerts` for `alerts-engine`.
### 5. `ops/k8s/` layout (Req 5)
```
ops/k8s/
profiles/kind.toml, k3s.toml # section 1; k3s values documented, not secrets
base/
workerpool-default.yaml.tmpl # replicas, limits, nodeSelector from the profile (render: plane/k8s.py)
workerpool-engine.yaml.tmpl # engine-pool, same template, engine values
ax-server-snapshots.yaml # a strategic-merge patch setting AX_SNAPSHOTS_BUCKET (applied over ax's deploy)
rbac-ci.yaml # SA ousast-ci; Roles + RoleBindings in ax-system and ate-system (section 2)
secrets.example.env # the variable names only: DEEPSEEK_API_KEY, OPENROUTER_API_KEY, S3_*, AWS_*
k3s/
imagepullsecret.yaml.tmpl # ghcr-pull from a GitHub PAT, referenced by the pools' pod template
hpa-default.yaml, hpa-engine.yaml # External metric ate_workerpool_workers{state=at_capacity}, averageValue 0.7
prometheus-adapter.yaml # from substrate/demos/autoscaled-workerpool, namespace ax-system
render.sh # ousast plane manifests --profile <p> --out DIR; kubectl apply -f DIR
```
`ops/ax/up.sh` keeps creating kind and calls `render.sh kind` for the pools; its receiver step is deleted in
increment 2. **gVisor on k3s is a correction to the requirements text:** Substrate does not use a Kubernetes
`RuntimeClass`. The `ateom-gvisor` worker (`workerImage: ko://.../cmd/ateom-gvisor`) fetches the gvisor release
tarball named by `manifests/ate-install/sandboxconfig-gvisor.yaml` (`gs://gvisor/releases/nightly/...`) and runs
`runsc` itself inside the worker pod (`cmd/ateom-gvisor/actorpath.go:35-45`). The node checklist is therefore:
PodSecurity in `ax-system` admits the worker pod's security context (recorded from kind in increment 6), the
node reaches the tarball URL or a mirror named in a copied SandboxConfig, and `/var/lib/ate` sits on local-path
storage with room for golden actors (24 MB each, `ops/ax/README.md:112`). Autoscaling is k3s-only: the HPA
needs Prometheus and the adapter (the Substrate demo README says kind-only for the demo; we take its manifests),
and the 7 GB host cannot scale anyway. The queueing consequence of one actor per worker is recorded in the
runbook: a Run with `--workers N` beyond the pool's replicas queues in ax as Suspended until a worker frees.
### 6. `ousast plane scan <url> --base <c> --head <c>` (Req 6)
`plane/scan.py`, reusing `generate.py`'s render path with a new case source:
1. **Workspace generation** (`generate.scan_cases(url, base, head, cache)`): a shallow clone into the case cache
(`runner.find_cached_checkout` layout), two pinned Workspaces `<repo_dir>-<head7>` and `<repo_dir>-<base7>`
(`workspaces.py` shape, pins in `openultrasast.io/git-commits`), repository key by `memory.repo_key`.
2. **Candidates from changed functions**: `generate.fix_ranges(repo, base, head, side="new")` gives the head-side
hunks (the `push/` delta primitive `_diff_records` is the same `git diff -U0`), `repo_facts.enclosing` names
the function per hunk line, filtered to `repo_facts.product_files`. Quick rules on the changed files
(`tasks.alerts.scan`) add alerted functions; `roles` runs once per repository when memory has no `roles` rows
for it; `repo-facts` supplies callers. `candidates.json` and `functions.json` are store inputs (section 3).
3. **Run composition**: per changed-function set `facts -> va, vb -> agree -> vc -> final -> features ->
remember`, plus `engine` when a changed file's language is engine-covered and not quick-covered, or PHP
(`ENGINE_ALONGSIDE_QUICK`). Budgets from `generate.verify_budget` (0.022 USD per candidate, headroom 1.5);
`--ceiling` caps the sum. `--all` (whole repository, secondary) takes every product function.
4. **Results** land in memory under `repos/<host>__<owner>__<name>/<head>.jsonl` through `remember` and
`ousast plane remember` (rows already carry `repo` and `pin`, `memory.py:77`); `runs/<run>/` holds the raw
artifacts for 30 days. The report prints the attribution table and the plane's agreement (`final`); the
decision engine's verdict only when a model is adopted (learned-decision-engine Req 6 unchanged).
5. **Latency split (e).** `ousast pre-push` stays the blocking hook: quick rules and the delta engine inside
`OUSAST_PUSH_DEADLINE` (30 s, `ops/README.md:322`). `ousast plane scan --exec remote --no-wait` submits and
prints the Run name in the hook's report; `--wait SECONDS` blocks for CI. `ousast plane status <run>` reports
later. The wrapper `ops/pre-push` gains an opt-in `OUSAST_PLANE_SCAN=1`; it never blocks on the plane.
### 7. Increment order, kind proof and measurement
| # | Increment | Kind proof (`local` unless stated) | Recorded under `benchmarks/measurements/` |
| --- | --- | --- | --- |
| 1 | `PlaneProfile`, literals replaced, `doctor` by profile | `ousast plane doctor` passes with `kind.toml`; literal scan 0 | `<date>-k8s-01-profile/record.json`: doctor lines per profile, scan count |
| 2 | Store delivery + receiver-less egress; receiver removed after | `ops/ax/smoke-run.sh` delivers `facts.json` via the store; policy has one TLS rule | `-02-store-delivery/`: bytes, PUT-to-poll seconds, rendered policies |
| 3 | `--exec remote` from a CI-shaped container (kubeconfig + secrets only) against kind | smoke Run completes; audit log lists only the Role's verbs | `-03-remote-exec/`: kube audit verbs vs Role, run status |
| 4 | Images on GHCR, digests, repin (section 8) | kind pulls `ghcr.io/norandom/ousast-runner@sha256` (pull secret if private); smoke Run | `-04-ghcr-images/`: sizes, pull seconds per image |
| 5 | `engine` task + `engine-pool` | froxlor case as `engine`: rows equal the 2026-09-30 host record (6/6), placement on engine-pool shown | `-05-engine-task/`: rows diff, seconds, peak memory, placement |
| 6 | Pools per profile, HPA manifests, node checklist | kind pools rendered and applied; worker securityContext recorded; HPA renders (k3s-only live) | `-06-pools/`: footprint table, securityContext |
| 7 | `plane scan` on a real pair (a PHP CVE pair from the dev corpus) | Run completes; rows under `repos/<repo>/<head>` | `-07-plane-scan/`: candidates, cost, agreement, rows |
| 8 | Runbook (`docs/deployment.md`), checklist mapped to R1-R7 | n/a (documentation; every "planned" label removed only with a recorded run) | the checklist links the records above |
### 8. GitHub Actions and re-pinning
`.github/workflows/images.yml` on tags `v*` (separate from `release.yml`, which builds the wheel): build
`plane/Dockerfile.runner` as `ghcr.io/norandom/ousast-runner` and the root `Dockerfile` as
`ghcr.io/norandom/ousast-engine` with `docker/build-push-action`, tag by version, and write the two digests to a
release asset `images.json` (`{"runner": "ghcr.io/...@sha256:...", "engine": "..."}`) and to job outputs.
`generate.repin_templates` (`generate.py:484-497`, one image for all templates) becomes `repin(plane, images)`:
`engine.yaml` gets `images["engine"]`, every other template `images["runner"]`; `ousast plane repin images.json`
exposes it and `--runner-image FILE` keeps working for a single-image file. Private packages: the pools' pod
template names `imagePullSecret` from the profile.
### 9. Tests
- `tests/test_plane_literals.py`: scans `src/`, `plane/`, `ops/` (minus `ops/ax/up.sh`, `ops/k8s/profiles/`) and
`docs/` for `kind-ousast`, `localhost:5001`, `172.19.`, `127.0.0.1:` and `port-forward` outside `router.py`
and the ax tunnel note; built at runtime like `test_removed_plane_references.py`.
- `tests/test_plane_profile.py`: defaults, file + env precedence, unknown key rejected, no secret field, both
committed profiles load.
- `tests/test_plane_k8s.py`: render both profiles; kind has no HPA and no pull secret; k3s has both; engine pool
limits >= the engine task's limits; every image reference is digest-pinned.
- `tests/test_plane_rbac.py`: the rendered Role's verbs equal the set of Kubernetes API calls the CLI makes
(fake `kubectl`/`ax`/`kubectl-ate` record their argv; the test maps them to `pods/portforward`, `pods`,
`services`) and nothing more.
- `tests/test_plane_egress.py`: for every committed Task template, `policy_for` yields one TLS rule whose hosts are
exactly the store host, the bound Git hosts and the Model hosts; no `http` rule; an IP endpoint is refused.
- `tests/test_plane_delivery.py`: start body with `delivery` accepted; URLs never logged; expiry; PUT and GET
against a local server; `delivered()` polling; declared-only inputs.
- `tests/test_plane_engine_task.py`: fixture record -> rows and summary equal `engine_alerts.produce`'s shape;
the instrument failures.
- `tests/test_plane_scan.py`: candidates from a two-commit fixture repository; Run composition; budgets.
- **Only live on kind:** placement on `engine-pool`; Substrate accepting the SA token for `egress-policy`;
the gateway passing TLS to the store host; GHCR pulls; the HPA (k3s only). These are the kind proofs above.
## Data Models
Store layout additions: `runs/<run>/state.json`, `runs/<run>/<task>/output.tar`, `runs/<run>/<task>/<artifact>`,
`runs/<run>/<task>/done.json`, `runs/<run>/inputs/<name>`; all under the existing 30-day `runs/` rule. Start body:
`credentials` as today plus optional `delivery {put, inputs{name: url}}`. `PlaneProfile` as in section 1.
`images.json` as in section 8.
## Error Handling
A profile naming an unreadable context, a bucket that fails `verify_bucket`, or an images file without a digest
fails before `ax apply` with the field named. A presigned PUT that fails three times ends the task `failed` with
the HTTP status (`runner.deliver` semantics kept); an expired URL after a re-resume is re-minted by the next
start. A store HEAD error during polling is retried with the resume backoff and reported after ten consecutive
failures, like `ax get` (`reconciler.py`, `_await`). A `delivery` URL that is not `https` to a hostname is a 400
from `StartGate.offer`. The engine task fails loudly on an unread input, as `read_record` does.
## Requirement Traceability
| Req | Components | Tests | Kind proof / record |
| --- | --- | --- | --- |
| R1 nothing assumes this host | `profile.py`, `doctor.py`, `egress.py`, `router.py`, `ops/k8s/profiles/` | literals, profile | increment 1 |
| R2 reconciler in the cluster -> CLI with `--exec remote`; kind first-class | `reconciler.py` (`Delivery`), `Ax` with context, `rbac-ci.yaml`, store-backed state | rbac, profile | increments 3, 1 |
| R3 receiver-less artifacts and egress | `delivery.py`, `runner.py` start body and PUT/GET, `egress.policy_for` | delivery, egress | increment 2 |
| R4 engine as a task | `tasks/engine.py`, `cpg/dump.py`, `Dockerfile` shim, `engine.yaml`, `engine-pool` | engine task, k8s | increment 5 |
| R5 sizing and autoscaling | `k8s.py`, `ops/k8s/base`, `ops/k8s/k3s` | k8s | increment 6 (HPA live only on k3s) |
| R6 delta Run | `scan.py`, `generate.scan_cases`, `ops/pre-push` opt-in | scan | increment 7 |
| R7 proven on kind, documented | every increment's record; `docs/deployment.md` | n/a | increment 8 |
## Open Questions
1. **Substrate authorization for the CI token.** `kubectl-ate` sends a ServiceAccount token; which audience and
which Substrate-side grant let `ousast-ci` create `egress-policy` in `default` is not in the checkouts read
here. Proven or refuted in increment 3.
2. **Engine placement.** Whether Substrate places a 3 Gi actor only on workers that fit it (section 4). The
fallback is named.
3. **gVisor tarball reachability on k3s** (`gs://gvisor/...` in the SandboxConfig): mirror or egress needed.
4. **Store endpoint hostname** for any future in-cluster RustFS; today's external endpoint is fine.
5. **Two images under one contract** versus the maintainer's "one runner image for all tasks": the design
reads it as one contract; confirm.
6. **GHCR visibility** (public packages need no pull secret) and whether Substrate's and ax's `ko` images go to
the same registry namespace.
7. **Reconciler size**: remote mode and `Delivery` must stay under the 500-line test; if they do not, the test's
bound moves with a reason, not silently.
@@ -0,0 +1,135 @@
# Requirements Document
## Introduction
The plane moves from this laptop's kind cluster to a Kubernetes cluster of the maintainer's choosing without
changing what runs: the same ax manifests, runner image, memory store and credential path. Every host-bound
piece becomes a cluster resource or a configuration value, each change is proven on kind first, and the
production entry point (a delta Run for a push) is added. Context: `brief.md`, `docs/deployment.md`.
## Boundary Context
- **In scope**: two profiles of one code base, `kind` (local, kept) and `k3s` (production); a remote execution mode of the CLI; artifacts delivered to the store; configurable
cluster context, registry and image pins; our own snapshot and memory buckets; the Joern engine as a plane task
on a larger worker pool; worker pool sizing and autoscaling; a delta Run command and Workspaces from URL +
commit; a deployment runbook with a proof on kind and a checklist for the target cluster.
- **Out of scope**: provisioning or operating the target cluster; modifying Agent Substrate or ax; any change to
detection, scoring, populations or the decision engine; replacing the `ax` CLI with a bespoke client beyond what
remote submission needs.
- **Adjacent expectations**: `ai-service-plane` requirements (one Model + budget per task, attribution,
credentials only in the start request, egress policies, verify-at-startup memory store) keep holding;
`harnessx-removal` Req 6 (memory on the S3 store) keeps holding; nothing is deleted before its replacement
passes its test (the `harnessx-removal` sequencing rule).
## Target Environment (maintainer, 2026-10-02)
- **Cluster: k3s.** Consequences: containerd (no Docker socket; the engine's host-Docker path does not exist there),
gVisor needs `runsc` and a `RuntimeClass` installed on the nodes Substrate's workers run on, k3s ships Traefik
and local-path storage, and single-node k3s is close enough to kind that the kind proof transfers.
- **Images and source: GitHub.** Images are published to GitHub Container Registry (`ghcr.io/norandom/...`),
built by GitHub Actions from tagged commits, digest-pinned in the manifests; private packages need an
`imagePullSecret` on the worker ServiceAccounts and the CI principal. Substrate's and ax's own images are built with
`ko` from their checkouts and pushed to the same registry.
## Requirements
### Requirement 1: Nothing assumes this host
**User Story:** As the maintainer, I want every host-bound assumption replaced by a cluster resource or a
configuration value, so the plane runs the same on kind and on the target cluster.
#### Acceptance Criteria
1. The kube context, the registry (GHCR in production), the ax-server address, the router address, the receiver address and the
snapshot and memory buckets are configuration (environment or one config file), with the kind values as the
documented defaults for development; no module names `kind-ousast`, `localhost:5001` or `172.19.0.1`.
2. `ousast plane doctor` checks the configured cluster, not kind by name, and reports each configured address.
3. A test enumerates the source tree for host-only literals and fails on any.
### Requirement 2: The reconciler stays the CLI, with a remote execution mode
**User Story:** As the maintainer, I want to trigger the checks in the cluster from the CLI and get the results
fed back, so that remote execution is a mode of the same tool, not a separate service.
#### Acceptance Criteria
1. `ousast plane run` (and the later `ousast plane scan`) takes an execution mode, `local` or `remote`, from the
profile or a flag; `remote` submits the Run to ax-server in the target cluster (over the cluster's ingress or a
kubeconfig), `local` is today's kind path. The manifests, the memory-key seeding, the budgets and the
attribution table are identical in both modes.
2. In remote mode the CLI needs no inbound path: it polls the memory store for task completion and artifacts and
prints the same status, attribution table and verdicts; `ousast plane status <run>` reports a Run started
earlier from any machine with the store credentials.
3. Provider and store credentials come from the caller's environment (a laptop `.env`, a CI secret), travel only
in the start request, and never appear in a manifest, an ActorTemplate or a log.
4. The principal the CLI uses against the cluster has the least privilege the ax API needs and nothing against
Agent Substrate directly; a test compares the rendered role with the calls the CLI makes.
5. kind stays a first-class local execution mode, not a stepping stone: `ousast plane run` from a laptop
against the kind cluster keeps working with the development profile (same code, same manifests, local
registry), and every release is proven on it before anything is applied to k3s.
6. An unattended in-cluster reconciler (a Deployment or Job that runs without a caller) is a later option, not
part of this spec.
### Requirement 3: Artifacts land in the store, not on a host
**User Story:** As the maintainer, I want task artifacts delivered to the memory store so that no host has to
be reachable from the cluster.
#### Acceptance Criteria
1. Tasks deliver their artifacts to the S3 memory store (presigned URLs minted by the CLI and carried in the
start request, or a scoped delivery credential; the design chooses one and states why) and the CLI reads
completion from the store. The receiver thread and the kind gateway EndpointSlice remain only if the design
shows the local profile needs them; one delivery path for both profiles is preferred.
2. Egress policies grant each task exactly the hosts it needs (the store's endpoint, git hosts, the bound
Model's host) and nothing else; a test renders the policy for every committed Task.
### Requirement 4: The engine is a plane task
**User Story:** As the maintainer, I want Joern to run where the other tasks run, on a worker sized for it.
#### Acceptance Criteria
1. A second WorkerPool sized for the engine (memory and CPU from a measured run) exists alongside the default
pool; an `engine` task module wraps today's host engine run and emits the same `alerts.jsonl` rows and
coverage as `alerts-engine`.
2. The engine image is digest-pinned in the registry like the runner image; its size and pull time are recorded.
3. `alerts-engine` on the host remains available for development and is marked as such.
### Requirement 5: Sizing and autoscaling
**User Story:** As the maintainer, I want the pool to follow the load of pushes without manual resizing.
#### Acceptance Criteria
1. Worker pool replicas, limits and the autoscaling rule (Substrate's assigned-worker signal) are manifests under
`ops/k8s/` with the kind values as a small profile and a documented production profile.
2. The runbook records the measured footprint per task type and the queueing consequence of one actor per
worker.
### Requirement 6: A delta Run for a push
**User Story:** As a user with a repository the project has never seen, I want a push checked by the plane.
#### Acceptance Criteria
1. `ousast plane scan <repo-url> --base <commit> --head <commit>` generates Workspaces from the URL and commits,
derives candidates from the changed functions (quick rules, inferred roles, repository facts), and submits a
Run of facts, verify a/b, tie-break and agree with per-task budgets; a whole-repository mode is secondary.
2. Results are written to the memory store under the repository and head commit and reported with the
attribution table; the decision engine's verdict is reported only when a model is adopted, else the plane's
agreement (ai-service-plane Req 6 and learned-decision-engine Req 6 unchanged).
3. The latency split is explicit: a blocking pre-push carries quick rules and the delta engine within its
deadline; plane verdicts arrive asynchronously.
### Requirement 7: Proven on kind, then documented for the target
**User Story:** As the maintainer, I want each step shown working on kind before I touch the real cluster.
#### Acceptance Criteria
1. Every requirement above is exercised end to end on the kind cluster (remote mode included, with the kind cluster as the remote target), with
a recorded run under `benchmarks/measurements/`.
2. `docs/deployment.md` becomes the runbook: prerequisites for the target cluster (Substrate, ax, registry,
buckets, Secrets, node requirements for gVisor workers), the profile switch, and a checklist whose every item
maps to a requirement here; planned items are labelled planned until their run is recorded.
+22
View File
@@ -0,0 +1,22 @@
{
"feature_name": "plane-on-kubernetes",
"created_at": "2026-10-02T20:00:00Z",
"updated_at": "2026-10-02T22:00:00Z",
"language": "en",
"phase": "implementation",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": true
}
+280
View File
@@ -0,0 +1,280 @@
# Implementation Plan
One task group per increment of `design.md` section 7, in that order; a later group never starts before the
earlier group's kind proof is recorded. Every task ships with its tests; a task is done when its tests pass in
the host suite, ruff and mypy are clean, and nothing in `src/`, `plane/`, `ops/` (minus `ops/ax/up.sh` and
`ops/k8s/profiles/`) or `docs/` names `kind-ousast`, `localhost:5001`, `172.19.` or a port-forward address
(task 1.3 makes that a test). Nothing is deleted before its replacement passes its proof. Each group ends with
its kind proof and the record it writes under `benchmarks/measurements/<date>-k8s-NN-<slug>/record.json`
(commit, images, the instrument checks, the numbers). Steps that call a model are marked **paid: needs a budget
go-ahead** with a ceiling; steps that need a hand on a cluster that does not exist yet are marked **manual**.
- [ ] 1. Profile and literals (design section 1; open-question checks 2, 3, 7)
- [x] 1.1 `PlaneProfile`
- `src/openultrasast/plane/profile.py`: frozen dataclass with the fields of design section 1; `load_profile()`
reads `ops/k8s/profiles/<OUSAST_PLANE_PROFILE>.toml` (a name or a path; default `kind`), then applies
`OUSAST_<FIELD>` overrides; rejects an unknown key, a secret-looking key (`*_KEY`, `*_SECRET`), an `exec`
outside `local|remote`, an `images` file without digest pins. `ops/k8s/profiles/kind.toml` and `k3s.toml`
committed with the design's values; `ops/k8s/profiles/kind-images.json` from `~/.cache/ousast/ax-src/runner-image`.
- `tests/test_plane_profile.py`: defaults; file then env precedence; each rejection names the field; both
committed profiles load; no field holds a value matching a credential pattern.
- _Requirements: 1.1_
- [x] 1.2 Literals out of `doctor.py`, `egress.py`, `router.py`, `generate.py`
- `doctor(profile)`: context from `profile.kube_context`; the registry check resolves the `images` file's runner
reference (`docker manifest inspect` for a remote registry, `/v2/_catalog` for a local one); `open_store`
verifies `profile.memory`; prints every configured address (Req 1.2). `Egress(context=...)`,
`open_router(context=...)` and `receiver_cluster_ip(context)` take the profile's context; `generate.ATESPACE`
becomes `profile.atespace`. `cli.py`: `ousast plane doctor|run|workspaces|harvest` accept `--profile`.
- `tests/test_plane_reconciler.py`, `test_plane_router.py`, `test_plane_egress.py`: the fakes receive the
profile's context, never a built one.
- Evidence: `grep -rn "kind-" src/openultrasast/plane` matches only comments citing the kind profile.
- _Requirements: 1.1, 1.2_
- [x] 1.3 Host-literal scan test
- `tests/test_plane_literals.py`: pattern built at runtime (as `tests/test_removed_plane_references.py` does)
over `src/`, `plane/`, `ops/` minus `ops/ax/up.sh` and `ops/k8s/profiles/`, and `docs/`, for `kind-ousast`,
`localhost:5001`, `172.19.`, `127.0.0.1:` and `port-forward` outside `router.py` and lines tagged
`ax-tunnel`; fails listing `path:line`.
- Evidence: the test passes on this commit and fails when a literal is reinserted (shown once in the record).
- _Requirements: 1.3_
- [x] 1.4 `up.sh` by profile
- `ops/ax/up.sh`: `KO_DOCKER_REPO` and the context from `kind.toml` (read with `python -m
openultrasast.plane.profile --print registry|kube_context`); writes `kind-images.json` instead of
`runner-image`; `generate.repin_templates` accepts that file (runner key only until task 4.3).
- Evidence: `ops/ax/up.sh` reruns idempotently on kind; `ops/ax/smoke-run.sh` passes.
- _Requirements: 1.1, 2.4_
- [x] 1.5 Placement probe (open question 2, decided before the engine work) -- result: `placement: any`
- `ops/ax/probe-placement.sh`: applies a second WorkerPool `probe-pool` (1 replica, 3 Gi) from the design's
engine template, then one model-free `repo-facts` Task with `resources.requests.memory: 2560Mi`, and records
with `kubectl ate get workers` which worker took the actor; then the same Task with 256Mi. Removed after.
- Evidence: `record.json` says whether Substrate places by resource fit (`placement: fit|any`). `any` selects
the design's fallback for group 5 and is written into `design.md` section 4 before group 5 starts.
- _Requirements: 4.1_
- [x] 1.6 Reconciler bound (open question 7) -- 489 lines after 1.2; projected 514 during group 2: the bound moves to 550 in 2.4
- Record `wc -l src/openultrasast/plane/reconciler.py` (477 at 17ac7ad) and the lines task 2.4 may add
(receiver wiring removed: `-12`, `Delivery` calls: `+8` estimated). If the estimate crosses 500, the bound
in `tests/test_plane_reconciler.py:803` moves to 550 in task 2.4 with the reason in the assertion message.
- Evidence: the numbers in `record.json`.
- _Requirements: 2.1_
- [x] 1.7 gvisor tarball reachability (open question 3) **manual** -- documented as planned in `docs/deployment.md` B.1; the live check waits for k3s
- `docs/deployment.md` (B.1, planned): the k3s node must fetch
`gs://gvisor/releases/nightly/2026-09-02/<arch>/gvisor.tar.zstd` (`substrate/manifests/ate-install/sandboxconfig-gvisor.yaml`)
or a mirrored copy named in a copied SandboxConfig; the check is `kubectl -n ate-system logs` of the worker
pod's atelet fetch line, run by hand on the day k3s exists. Labelled planned until then.
- _Requirements: 7.2_
- [x] 1.8 Kind proof and record 01
- `ousast plane doctor --profile kind` passes every line; `ousast plane doctor --profile k3s` fails only on the
unreachable context (the expected line); `test_plane_literals` 0 offenders; the probe of 1.5; the bound of 1.6.
- `benchmarks/measurements/<date>-k8s-01-profile/record.json`.
- _Requirements: 1.1, 1.2, 1.3, 7.1_
- [ ] 2. Store delivery, receiver-less egress (design section 3)
- [x] 2.1 Presigned URLs and the store run directory
- `src/openultrasast/plane/delivery.py`: `Delivery(store, run)` mints presigned PUT for
`runs/<run>/<task>/output.tar` and GET for `runs/<run>/<producer>/<artifact>` (boto3 `generate_presigned_url`
on `S3Client.sdk`, expiry `OUSAST_TASK_TIMEOUT + 600`); `delivered(task)` HEADs the tar; `collect(task)`
downloads, extracts under the run dir with the member checks `Receiver.accept` has, re-puts declared
`outputs[]` as single objects, writes `done.json`; `put_input(name, path)` for inputs no task produces;
`state.json` mirrored to `runs/<run>/state.json` and seeded from it on a rerun.
- `tests/test_plane_delivery.py`: against a local S3-shaped fake (PUT/GET/HEAD by key); URLs never appear in
logs; expiry; member rejection; declared-only re-put; state seeding.
- _Requirements: 3.1, 2.1_
- [x] 2.2 Runner: `delivery` in the start body
- `src/openultrasast/plane/runner.py`: `StartGate.offer` validates an optional `delivery {put, inputs}` (https,
hostname, no echo); `_deliver_output` PUTs the tar to `delivery.put` (same retry and `DeliveryError`
semantics); `fetch_inputs` GETs from `delivery.inputs[name]`; `OUSAST_ARTIFACT_URL` and `OUSAST_ARTIFACT_DIAL`
stay honoured until task 2.6 so both paths run in the same build.
- `tests/test_plane_runner.py`: accepted body with `delivery`; 400 on an `http` or IP URL; PUT to a local
server; inputs fetched from presigned-shaped URLs; the marker and retry-only boot unchanged.
- _Requirements: 3.1_
- [x] 2.3 Egress without the receiver
- `src/openultrasast/plane/egress.py`: `policy_for(task, workspaces, model, store_host)` emits one
`tlsPassthrough` rule on 443 over the sorted union of store host, Git hosts, Model hosts; no `http` rule; an
IP store host raises naming `S3_ENDPOINT`.
- `tests/test_plane_egress.py`: for every committed template under `plane/tasks/` the rendered policy is the
expected set; the IP refusal.
- _Requirements: 3.2_
- [x] 2.4 Reconciler on `Delivery`
- `src/openultrasast/plane/reconciler.py`: `run()` opens the store first (no fallback), builds `Delivery`,
passes `delivery` to `router.starter`/`start_task` (body extended in `router.py`), `_await` on
`Delivery.delivered`, outcome from the collected `summary.json`; the receiver thread is kept behind
`OUSAST_DELIVERY=receiver` for this group only. Line bound per task 1.6.
- `tests/test_plane_reconciler.py`: the ax fake run completes through the delivery fake; a rerun from an empty
results root resumes from the store's `state.json`.
- _Requirements: 2.1, 3.1_
- [x] 2.5 Kind proof and record 02
- `ops/ax/smoke-run.sh` with `OUSAST_DELIVERY=store`: `facts.json` and `summary.json` arrive through the store;
the actor's policy read back has one TLS rule naming the store host; `ousast plane status ax-e2e-smoke`
prints the table. Instrument: the tar's byte size from the store equals the extracted files' total.
- `benchmarks/measurements/<date>-k8s-02-store-delivery/record.json`: bytes, PUT-to-poll seconds, rules.
- _Requirements: 3.1, 3.2, 7.1_
- [ ] 2.6 Remove the receiver (last; after 2.5 is recorded)
- Delete `Receiver`, `_Handler`, `receiver_address`, `receiver_cluster_ip`, `RECEIVER_HOST`, `RECEIVER_PORT`
from `egress.py`; `OUSAST_ARTIFACT_URL/DIAL/HOST/PORT` from `runner.py` and `reconciler.py`; the
`OUSAST_DELIVERY` switch; `ops/ax/receiver-service.yaml.tmpl` and the `up.sh` receiver step; the EndpointSlice
from kind (`kubectl delete`). Docs: `docs/plane.md` sequence diagram, `docs/deployment.md`, `ops/ax/README.md`.
- Tests using `OUSAST_ARTIFACT_HOST` move to the delivery fake; `test_plane_literals` covers `172.19.`.
- Evidence: `ops/ax/smoke-run.sh` passes again on the build without the receiver; recorded in 02's record as
`receiver_removed: <commit>`.
- _Requirements: 3.1, 1.3_
- [ ] 3. `--exec remote` from a CI-shaped principal (design section 2; open-question check 1)
- [ ] 3.1 Execution mode and RBAC manifests
- `cli.py`: `ousast plane run --exec local|remote` (default `profile.exec`); `reconciler.Ax` passes the
profile's context to `ax` (its context flag, read from `ax --help` and pinned in a test); `remote` refuses a
`file://` store. `ops/k8s/base/rbac-ci.yaml`: SA `ousast-ci` in `ax-system`; Roles and RoleBindings in
`ax-system` and `ate-system` with `pods` get/list, `services` get, `pods/portforward` create, nothing else.
- `tests/test_plane_rbac.py`: fake `kubectl`, `ax`, `kubectl-ate` record argv over a full fake Run; the
verbs implied (`port-forward` -> `pods/portforward` create + `pods` get + `services` get) equal the rendered
Role's rules exactly.
- _Requirements: 2.1, 2.2_
- [ ] 3.2 Substrate's authorization of the SA token (open question 1)
- `ops/ax/probe-ci-token.sh`: `kubectl create token ousast-ci --duration=2h` into a kubeconfig; `kubectl ate
--token-file - get workers` and `create egress-policy` on a throwaway actor with that kubeconfig. If refused,
read `substrate/docs/api-guide.md` for the grant (atespace role or token audience) and add the manifest to
`ops/k8s/base/rbac-ci.yaml`; if no grant exists for a non-admin token, record it and the design's remote mode
uses the maintainer's admin kubeconfig for the proof, with the gap listed in the runbook.
- Evidence: `record.json` names the grant used (`grant: <kind/name>` or `admin-kubeconfig`).
- _Requirements: 2.1_
- [ ] 3.3 CI-shaped container
- `ops/k8s/ci-shell.Dockerfile`: `python:3.12-slim` + this package + `kubectl`, `kubectl-ate`, `ax` (built from
the checkouts); no `.env`; runs with `-e KUBECONFIG=/run/kc -e DEEPSEEK_API_KEY -e S3_*` only.
`.github/workflows/plane-remote.yml` (dispatch only, no schedule): the same image running `ousast plane run
--exec remote --profile k3s` with repository secrets; disabled until k3s exists (labelled in the file).
- _Requirements: 2.2, 2.3_
- [ ] 3.4 Kind proof and record 03 **paid: needs a budget go-ahead, ceiling 0.50 USD**
- From the container of 3.3 against kind (`kind.toml` with `exec = "remote"`): the smoke Run (model-free), then
one `verify` pass over one case of `plane/runs/validation-46.yaml` (budget `{usd: 0.5, calls: 60}`) so the
credential path is exercised remotely. Kube audit log (kind apiserver `--audit-policy` of the SA's verbs)
lists only the Role's verbs.
- `benchmarks/measurements/<date>-k8s-03-remote-exec/record.json`: audit verbs vs Role, grant of 3.2, run
status, usd spent.
- _Requirements: 2.1, 2.2, 2.3, 2.4, 7.1_
- [ ] 4. Images on GHCR (design section 8)
- [ ] 4.1 `images.yml`
- `.github/workflows/images.yml` on tags `v*`: `docker/build-push-action` for `plane/Dockerfile.runner` ->
`ghcr.io/norandom/ousast-runner:<tag>` and the root `Dockerfile` -> `ghcr.io/norandom/ousast-engine:<tag>`;
digests as job outputs and as the release asset `images.json` (`{"runner": "...@sha256:...", "engine": ...}`);
`GITHUB_TOKEN` with `packages: write`.
- Evidence: a dry run on a `v*-rc` tag publishes both and the asset (the engine build is ~2.8 GB: record the
build minutes).
- _Requirements: 1.1, 4.2_
- [ ] 4.2 Engine image under the runner contract
- Root `Dockerfile`: the `ax-task-runner` shim and `/workspace` as `plane/Dockerfile.runner:25-29`; the `s3`
extra is not needed in either image (the runner PUTs by URL).
- `tests/test_plane_images.py`: both Dockerfiles declare the shim and entrypoint (text check).
- _Requirements: 4.2_
- [ ] 4.3 `repin` for two images
- `generate.repin(plane, images)`: `engine.yaml` (added in 5.1; until then absent is fine) takes
`images["engine"]`, every other template `images["runner"]`; `ousast plane repin images.json`;
`--runner-image FILE` keeps the single-image file working. `ops/k8s/profiles/k3s.toml` names `images.json`
downloaded from the release.
- `tests/test_plane_increment.py`: repin with a two-key file; a template with two image lines is refused.
- _Requirements: 1.1, 4.2_
- [ ] 4.4 Kind proof and record 04
- Pools' pod template takes `imagePullSecret` from the profile (`ops/k8s/base/workerpool-*.tmpl`, task 6.1
brings the full template; here the field only); kind pulls `ghcr.io/norandom/ousast-runner@sha256:...` with
`ghcr-pull` if the package is private; the smoke Run passes on the GHCR image.
- `benchmarks/measurements/<date>-k8s-04-ghcr-images/record.json`: image sizes, pull seconds, visibility.
- _Requirements: 1.1, 4.2, 7.1_
- [ ] 5. The engine as a plane task (design section 4)
- [ ] 5.1 `cpg/dump.py` and `tasks/engine.py`
- Lift `benchmarks/push/finding_dump.py`'s scan into `src/openultrasast/cpg/dump.py` (the script becomes a
thin caller; `ops/README.md:78` command unchanged). `src/openultrasast/plane/tasks/engine.py`: env
`OUSAST_WORKSPACE_DIR`, `OUSAST_FIXED_DIR`, `OUSAST_INPUT_CASE`, pins, `OUSAST_FAMILIES`, `OUSAST_DEADLINE`;
per pin the quick scan (`tasks.alerts.scan`) and the engine in-process; rows via `engine_alerts.engine_rows`,
the instrument checks via `read_record`'s rules; `summary.json` as `engine_alerts.produce` writes it with
`host: false`. `plane/tasks/engine.yaml`: `command: ["engine"]`, requests 3Gi/1, limits 4Gi/2, image from
`images["engine"]`. `generate.TEMPLATES` gains `engine`; `--loop` emits `engine` instead of `alerts` for a
case needing the engine when the images file has an engine key.
- `tests/test_plane_engine_task.py`: a fixture record -> rows and summary equal to `produce`'s shape; the
unread-input and zero-question failures; the generator's choice.
- _Requirements: 4.1_
- [ ] 5.2 `engine-pool`
- `ops/k8s/base/workerpool-engine.yaml.tmpl` rendered by `plane/k8s.py` (task 6.1 generalises; here the engine
values): kind replicas 1, 4Gi/2; if 1.5 recorded `any`, apply the design's fallback and write it here.
- Evidence: `kubectl -n ax-system get workerpool engine-pool` ready on kind with `ousast-pool` at 1 replica.
- _Requirements: 4.1, 5.1_
- [ ] 5.3 `alerts-engine` marked development-only
- `cli.py` help, `docs/plane.md` task table, `ops/README.md`: "development only (needs Docker on the host);
the plane task is `engine`". No code removed.
- _Requirements: 4.3_
- [ ] 5.4 Kind proof and record 05
- The froxlor case of `validation-46` as an `engine` task on kind (model-free, deadline 1800 s per pin):
rows equal `benchmarks/measurements/2026-09-30-php-engine-alerts/` (6/6 per pin), placement on `engine-pool`
shown by `kubectl ate get workers`, peak memory from `kubectl top`.
- `benchmarks/measurements/<date>-k8s-05-engine-task/record.json`: rows diff, seconds per pin, peak memory,
placement, image size and pull seconds.
- _Requirements: 4.1, 4.2, 7.1_
- [ ] 6. Pools, autoscaling, node checklist (design section 5)
- [ ] 6.1 `plane/k8s.py` and the manifests
- `src/openultrasast/plane/k8s.py`: `render(profile, out)` writes `workerpool-default.yaml`,
`workerpool-engine.yaml`, `ax-server-snapshots.yaml` (patch with `profile.snapshots_bucket`), `rbac-ci.yaml`;
for `k3s` also `imagepullsecret.yaml`, `hpa-default.yaml`, `hpa-engine.yaml`, `prometheus-adapter.yaml`
(copied from `substrate/demos/autoscaled-workerpool`, namespace `ax-system`, metric
`ate_workerpool_workers{state=at_capacity}`, `averageValue 0.7`). `ousast plane manifests --profile P --out
DIR`; `ops/k8s/render.sh`. `ops/ax/workerpool.yaml.tmpl` deleted once `up.sh` uses the renderer.
- `tests/test_plane_k8s.py`: both profiles render; kind has no HPA and no pull secret; k3s has both; engine
pool limits >= `engine.yaml` limits; every image digest-pinned; no host literal in the output.
- _Requirements: 5.1_
- [ ] 6.2 Node checklist and footprint
- `docs/deployment.md`: the gVisor correction (no RuntimeClass; `ateom-gvisor` runs `runsc` from the
SandboxConfig tarball), PodSecurity for the worker pod's securityContext (recorded from kind), local-path for
`/var/lib/ate`, the measured footprint per task type (`verify`, `repo-facts`, `engine` from 05) and the
queueing consequence of one actor per worker with `--workers`.
- _Requirements: 5.2, 7.2_
- [ ] 6.3 Kind proof and record 06 (HPA: render only; live is k3s-only, **manual** later)
- `render.sh kind` applied on kind replaces the hand template with identical pools (`kubectl diff` empty but
for labels); worker pod `securityContext` captured; `render.sh k3s` validates with `kubectl apply
--dry-run=client` against kind's API (CRDs present).
- `benchmarks/measurements/<date>-k8s-06-pools/record.json`: footprint table, securityContext, dry-run result.
- _Requirements: 5.1, 5.2, 7.1_
- [ ] 7. `ousast plane scan` (design section 6)
- [ ] 7.1 Workspaces and candidates from URL + commits
- `generate.scan_cases(url, base, head, cache, *, all=False)`: shallow clone into the case cache, two pinned
Workspaces, head-side hunks via `fix_ranges(side="new")`, functions via `repo_facts.enclosing` over
`product_files`, quick alerts via `tasks.alerts.scan` on the changed files, `candidates.json` and
`functions.json` as store inputs (`Delivery.put_input`); `--all` takes every product function.
- `tests/test_plane_scan.py`: a two-commit fixture repository yields exactly the changed functions; a rename
and a deleted function are handled; `--all` count.
- _Requirements: 6.1_
- [ ] 7.2 Run composition and the command
- `src/openultrasast/plane/scan.py` + `cli.py`: `ousast plane scan URL --base C --head C [--all] [--ceiling USD]
[--exec] [--wait S | --no-wait]`: per set `facts -> va, vb -> agree -> vc -> final -> features -> remember`,
`engine` when a changed file's language needs it, `roles` once per repository when memory has no `roles`
rows for `repo_key(url)`; budgets from `verify_budget`; Run annotated `openultrasast.io/population: scan`,
`split: <repo_dir>`; after the Run, `memory.ingest`, then the attribution table and the plane's agreement
(`final`), the decision engine's verdict only when adopted. `--no-wait` prints the Run name and exits 0.
- `tests/test_plane_scan.py`: the rendered Run for the fixture; budgets sum under the ceiling; `roles`
included only when the store lacks rows; the report's verdict line.
- _Requirements: 6.1, 6.2_
- [ ] 7.3 Hook opt-in
- `ops/pre-push`: with `OUSAST_PLANE_SCAN=1`, after the blocking check, `ousast plane scan <remote url>
--base <old> --head <new> --exec remote --no-wait` and the Run name in the report; the hook's exit code never
depends on it. `ops/README.md` and `docs/deployment.md`: the latency split.
- `tests/test_push_hook.py`: the wrapper calls the scan only with the variable set and ignores its failure.
- _Requirements: 6.3_
- [ ] 7.4 Kind proof and record 07 **paid: needs a budget go-ahead, ceiling 3.00 USD**
- A PHP pair from the development corpus (pmpro's CVE commit and its parent, never a v2 or v3 case) scanned on
kind: candidates, the Run through `final`, `engine` for the PHP files, rows under
`repos/<host>__<owner>__<name>/<head>.jsonl` on the store, the report; then `ousast plane status`.
- `benchmarks/measurements/<date>-k8s-07-plane-scan/record.json`: candidates, usd, calls, agreement, rows,
seconds from submit to verdict, whether the known CVE site is among the agreed.
- _Requirements: 6.1, 6.2, 6.3, 7.1_
- [ ] 8. Runbook (design section 7, Req 7)
- [ ] 8.1 `docs/deployment.md` as the runbook
- Sections: prerequisites for k3s (Substrate, ax with the snapshots patch, GHCR and the pull secret, buckets,
the CI SA and its grant from 3.2, the node checklist of 6.2), the profile switch, `--exec remote` from a laptop
and from CI, and the checklist whose every item names its requirement and its record 01-07; items without a
recorded run (HPA live, 1.7's fetch) labelled planned. `ops/ax/README.md` and `ops/README.md` point to it;
the "moving to a separate cluster" tables are replaced by the checklist.
- _Requirements: 7.2_
- [ ] 8.2 Spec closure
- `design.md` open questions updated with the outcomes of 1.5, 1.6, 3.2 (1.7 stays open until k3s); the
`kind-tooling` memory and `.kiro/steering/overview.md` current-state line updated; `/kiro-spec-status`.
- Evidence: every record 01-07 linked from the runbook's checklist.
- _Requirements: 7.1, 7.2_
@@ -67,7 +67,7 @@ real capability; task 8.3 still owns actual eligibility. Task 4.2 brings progres
## Task 4.3 verification
The agent service rejected the task 4.3 dispatch with `agent thread limit reached`.
The [kiro-impl skill](../../../.opencode/skills/kiro-impl/SKILL.md) explicitly provides:
The [kiro-impl skill](../../../.agents/skills/kiro-impl/SKILL.md) explicitly provides:
“If multi-agent is not available, fall back to manual mode execution for all tasks.”
Implementation, review and completion checks therefore ran in the main context.
This is a manual review record, not an independent-review claim.
@@ -7,7 +7,7 @@ Real capability eligibility and rollout readiness remain unproven.
## Execution and review mode
Fresh implementer dispatch for 5.1 returned `agent thread limit reached`.
The [Kiro implementation skill](../../../.opencode/skills/kiro-impl/SKILL.md) explicitly
The [Kiro implementation skill](../../../.agents/skills/kiro-impl/SKILL.md) explicitly
states: “If multi-agent is not available, fall back to manual mode execution for all tasks.”
Its manual review rule permits review in the main context when a fresh reviewer is unavailable.
This run therefore uses manual implementation and adversarial review, not independent review.
@@ -147,6 +147,43 @@ with Express-style routes and middleware, including vulnerable, fixed and benign
frontend/framework adaptations and new abstract capabilities, and shall report C/C++ envelope or
unsupported-property results separately from demonstrated vulnerability detection.
## Requirement 9: Operate as an advisory safety net on any repository
Added 2026-10-02, maintainer-approved, from the operational-readiness audit. The maintainer's
definition: a user installs the hook on ANY repository and gets honest, advisory-only results;
nothing ever blocks a push unless blocking was explicitly chosen.
9.1 (added 2026-10-02, maintainer-approved) When the installed hook cannot run the check, because the
`ousast` executable is missing or an argument or setting is invalid, the hook shall allow the push and
print one line saying the check did not run and how to fix it; only an explicit `--mode blocking`
opt-in may turn an analysis result into a rejected push.
9.2 (added 2026-10-02, maintainer-approved) When a check is skipped or incomplete, the terminal output
shall name each distinct reason in plain words, one line per reason, with repository paths decoded and
no opaque hash tokens; a language with no engine frontend and no quick rules shall be named with its
file count; a first push of a new branch shall say there was no base to compare against and what was
checked instead. The JSON artifact keeps the raw reason values.
9.3 (added 2026-10-02, maintainer-approved) Before the engine, within the same deadline, the Safety Net
shall run the shipped quick pattern rules over the head side of the changed lines and show their
findings as advisory pattern matches, labeled separately from admitted alerts. This amends 3.2 for this
tier only: quick-rule findings are shown, never admitted, and never block.
9.4 (added 2026-10-02, maintainer-approved) When a finding's operation lies in a file that is new in the
head revision, the comparison shall classify it as new rather than unknown; an engine finding whose
novelty is established but whose capability is not yet qualified shall be shown as an advisory engine
finding, not an admitted alert. Genuinely unknown comparisons keep their vetoes.
9.5 (added 2026-10-02, maintainer-approved) When the engine cannot finish within the deadline, one line
shall say so; on explicit request (`--engine background`) the Safety Net shall start the engine
detached after the push, record its result in the artifact directory, and show it on the next run as
the engine result for that head from the previous push.
9.6 (added 2026-10-02, maintainer-approved) The hook installer and remover shall ship in the wheel
(`ousast pre-push install|uninstall`), refuse to overwrite an existing hook unless forced, and a
hook-callable container wrapper shall pass the push input and artifact path through; the documentation
shall give a first-run guide, a glossary of skip reasons, a CI recipe and help text for every flag.
## Accepted experimental acceptance profile
These values are the initial implementation/evaluation profile accepted for task generation, not
+51 -2
View File
@@ -475,7 +475,7 @@ Preserve all completed task evidence below.
- _Depends: 13.1_
- _Requirements: 6.2, 6.3, 8.1, 8.2_
- [ ] 14. Ask every declared family whether it establishes anything (precondition for M4)
- [x] 14. Ask every declared family whether it establishes anything (precondition for M4)
- Seven families are declared and two were proven end to end. A declared family that has never been asked whether it finds anything is advertised coverage, and M4 cannot qualify a capability that establishes nothing.
- [x] 14.1 Census the seven declared families on NodeGoat and VAmPI
- Run production discovery and scan over each whole checkout and report, per family, the regions that offered it, the questions asked, the questions completed, and what was found with the witness and rung. No comparison, no novelty, no admission: detection only.
@@ -575,7 +575,7 @@ Preserve all completed task evidence below.
- _Boundary: Evaluation harness, full-run integration_
- _Depends: 13.2, contributor-scan 7.3, contributor-scan 7.8_
- _Requirements: 6.2, 6.3, 8.1, 8.2_
- [ ] 13.4 Let a question inside one partition complete, whatever else the repository holds
- [x] 13.4 Let a question inside one partition complete, whatever else the repository holds
- **Diagnosed 2026-09-21, and PHP was never the variable.** A repository holding TWO languages can never complete a question, so it can never admit a finding. `partitions.py` raises `cross_partition_semantics_unresolved` whenever more than one language partition exists; that reason is a member of `GRAPH_INTEGRITY_GAPS`; and a graph-integrity gap demotes every completed outcome in the scan.
- The evidence is one column wide: NodeGoat is JavaScript alone and completes 299 of 299, VAmPI is Python alone and completes 35 of 35, PMPro is PHP with 36 regions of bundled JavaScript and completes 0 of 127 and 0 of 520 while establishing 19 and 43 findings. NodeGoat's own `vendor_semantics_unresolved` is a DECLARED exclusion rather than an integrity gap, which is why it does not demote.
- **This is larger than the PHP runtime limit and it lands on the declared v0.1 target.** A WordPress plugin is PHP plus its admin JavaScript, so every one of them is multi-language, so on the target this tool was built for it can detect defects and never admit one. Neither subject that currently passes could reveal it.
@@ -587,6 +587,7 @@ Preserve all completed task evidence below.
- _Three causes, not one, and the third was hidden by the record rather than by the code._ The same census gap is raised in TWO passes, arbitration and evidence, and carrying the partition in one but not the other left an unscoped copy that kept demoting everything, so the fix looked applied and was not. It was found by making the census print each outcome's reason instead of only its status, after which three separate causes stopped looking identical.
- _Admission is scoped too._ `_blocking_degradations` already exempted declared exclusions; it now also ignores a gap another language partition raised, so a JavaScript build artefact cannot block a PHP finding. A gap naming no partition still reaches everything, and so does every gap when the claim's own language is unknown, because unknown scope is not evidence of independence. Evidence: `benchmarks/measurements/2026-09-21-partition-gap-blast-radius.json`.
- _Boundary: contributor-scan query/context evidence, Ranker Scope Contract_
- _Checked 2026-10-02 per the operational-readiness audit:_ implemented and verified on PMPro (120 of 127, above). 13.3 stays open on its own blocker; this box was left unchecked only because it waits on it.
- _Depends: 13.3_
- _Requirements: 2.1, 2.2, 6.3, 8.1_
@@ -619,6 +620,54 @@ Preserve all completed task evidence below.
- _Requirements: 8.1, 8.2_
- [x] 17. Operational readiness (2026-10-02)
- **Named by the 2026-10-02 operational-readiness audit**, maintainer-approved as the first increment. The maintainer's definition: a user installs the hook on ANY repository and gets honest, advisory-only results; nothing ever blocks a push. The audit found the plumbing sound (snapshots, resolution, deadline accounting, advisory exit 0, the JSON artifact) and five things in the way: the hook can block a push, quick rules are not part of pre-push, the engine cannot finish a 26-file repository inside 30 s, the terminal hides every skip reason, and there is no supported install path without the source checkout. Before numbers for every item are the audit's live runs on records (Python), serve-static (JavaScript) and unrolled/secure (Go), each with one planted commit.
- This group is plumbing and honesty. It enables no capability, edits no rule for a planted sample, and leaves graph reuse (the engine speed work decided in 11.6) as a later task.
- _Done 2026-10-02:_ all seven subtasks done. The engine still does not fit the 30 s hook deadline (graph reuse, the 11.6 direction, stays a later task), and plane verdicts after the push were not in this increment.
- _Requirements: 9.1–9.6_
- [x] 17.1 Never block a push the user did not choose to block
- The installed hook exits 0 with one line when `ousast` is missing (the check did not run, and how to install). In hook (stdin) mode, an argument or usage error (an invalid `OUSAST_PUSH_DEADLINE`, a cache directory inside the repository) is advisory: one line and exit 0. `--mode blocking` stays the explicit opt-in that can reject a push, and keeps its usage errors. The prior-hook input-retention failure keeps exit 1, because it is the chained hook's verdict that cannot be obtained, not ours.
- Files: `ops/pre-push`, `src/openultrasast/cli.py`. Tests: `tests/test_push_hook.py` (a push with `ousast` absent from PATH succeeds and prints the line; a bad deadline in hook mode exits 0; blocking mode keeps exit 2).
- Evidence: the audit's `hook.sh` push 1 (rejected before) and `fail.sh` (rc=2 before) rerun with exit 0.
- _Done 2026-10-02:_ the hook prints one line and exits 0 (chaining a prior hook if present) when `ousast` (or `OUSAST_COMMAND`) is missing; in hook mode without `--mode blocking`, argparse/usage errors and unexpected exceptions print one line and exit 0. Tests: `test_missing_ousast_never_blocks_the_push`, `test_hook_mode_usage_errors_are_advisory_unless_blocking` (blocking keeps exit 2), and `test_cli_refuses_the_flag_outside_explicit_replay` updated (the experimental flag is still refused in hook mode, now advisory). Gates: 1987 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.1, 5.3, 5.4_
- [x] 17.2 Name every skipped check in plain words
- One terminal line per distinct coverage reason: hex-encoded paths decoded, sha256-suffixed reasons grouped with a count, JSON unchanged. "language not covered: <lang> (<n> files)" for a language with no engine frontend and no quick rules, in both `pre-push` and `scan`. A first push of a new branch says there is no base and what was checked instead. `reports.py` gains lines for the eight scan degradations `cli.py` records and the report did not render (`learning_endpoint_unavailable`, `cpg_unavailable`, `facts_unavailable`, `policy_invalid`, `model_withheld_execution_budget`, `max_findings_exceeded`, `hunter_model_unavailable`, `sandbox_unavailable`).
- Files: `src/openultrasast/push/report.py`, `src/openultrasast/push/explain.py` (new), `src/openultrasast/reports.py`, `src/openultrasast/cli.py`. Tests: `tests/test_push_report.py`, `tests/test_reports.py`.
- Evidence: the three repositories' terminal output names engine-missing, Go not covered and the new branch.
- _Done 2026-10-02:_ new `push/explain.py` turns every raw coverage reason into one plain `Skipped:` line per distinct kind (hex paths decoded, sha256 tokens dropped and counted, unknown reasons still printed); the existing one-line `Notice:` keeps its next action and gains an engine-missing variant. `language_not_covered:<lang>:<changed>:<total>` is recorded for a changed language with no engine frontend and no quick rules; `ousast scan` prints `notice: language not covered: go (N files)` and its report says so. `reports.py` renders the eight previously dropped degradations and prints any reason it has no sentence for. The hook passes `OUSAST_COMPARISON_BASE`. The new-branch line names the base it lacks; what was checked instead lands with the quick tier (17.3). Tests: `tests/test_push_explain.py` (6), two in `tests/test_reports.py`; module-audit manifest regenerated for the new module. Gates: 1995 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.2, 5.2, 8.2_
- [x] 17.3 Run the quick rules as the fast first tier
- Inside the deadline and before the engine, the shipped quick rules run over the head side of the changed files and keep the findings on changed lines or in changed functions. They print as advisory pattern matches, labeled apart from admitted alerts, recorded in the artifact's `quick_tier`, and never block.
- Files: `src/openultrasast/push/quick.py` (new), `src/openultrasast/push/runner.py`, `src/openultrasast/push/report.py`. Tests: `tests/test_push_quick.py` (new).
- Evidence: serve-static's planted `eval` in the new `debug.js` is shown; whether records' planted `%`-formatted SQL is shown is recorded as is, with no rule edits for it.
- _Done 2026-10-02:_ new `push/quick.py` runs the shipped quick rules (no shadow rules, no model) over the changed head files inside `_prepare` under the shared deadline, before the engine, keeping matches on a changed line or inside a changed function (entry-point ranges); it records `files`/`bytes_read` as proof of input. The terminal shows them under "Quick rules: N pattern match(es) on changed code ... Advisory: not verified by the engine."; the artifact keeps `quick_tier`; admission and push disposition are untouched. A new branch runs the quick tier over its last commit's changes and says so. `_analyze` gained `engine="off"`. On the audit's serve-static planted commit the `eval` in the new `debug.js` is now shown (js-eval, js-xss-response at debug.js:2, ~1.1 s on the host); the planted `%`-formatted SQL in records is NOT matched by any quick rule (coverage gap, no rule edited). Tests: `tests/test_push_quick.py` (3). Gates: 1998 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.3_
- [x] 17.4 Classify a file new in head as new, not unknown
- `ChangeContext` records the paths Git reports as added. An operation in an added file, when no add/delete pairing is unresolved and its head evidence is complete, is `new` with reason `file_added_in_head`, and admission accepts that reason as change evidence. An engine finding with established novelty whose only vetoes are capability vetoes renders as an advisory engine finding, not an alert. Every other unknown comparison keeps its veto.
- Files: `src/openultrasast/model/contracts.py`, `src/openultrasast/push/snapshot.py`, `src/openultrasast/push/policy.py`, `src/openultrasast/push/report.py`. Tests: `tests/test_push_policy.py`, `tests/test_push_snapshot.py`, `tests/test_push_report.py`.
- Evidence: unit tests on the comparison and admission; a live engine run if the host allows one (free memory >= 3 GB).
- _Done 2026-10-02:_ `ChangeContext.added_paths` records Git status-A paths (rename targets excluded). In `_compare_evidence`, after every existing context/head/deadline check, an operation in an added file is `new`/`file_added_in_head` when the claim is entailed, undischarged and has change evidence, and no base answer holds the same operation (else `operation_moved_within_change`); an unpaired add/delete still blocks as `change_context_unresolved`. Admission accepts `(new, file_added_in_head)`; with the empty registry such a candidate's only veto is `capability_unavailable`, and the terminal now shows it under "Engine: N finding(s) this push introduced. Advisory ... not an alert." Any comparison, evidence or location veto keeps a candidate out of that line. Tests: `tests/test_push_new_file.py` (9). Live engine evidence depends on host memory (17.7). Gates: 2007 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.4, 3.4_
- [x] 17.5 Say when the engine cannot finish, and offer it in the background
- One line when the engine did not finish (deadline, engine not installed). `--engine background` starts the engine detached after the quick tier, writes its result under the artifact directory and, on the next run, prints "engine result for <head> from the previous push". One background engine at a time; `--engine off` skips it with a stated line. No speed work.
- Files: `src/openultrasast/push/background.py` (new), `src/openultrasast/push/runner.py`, `src/openultrasast/cli.py`, `ops/pre-push`. Tests: `tests/test_push_background.py` (new).
- Evidence: a hook push with `OUSAST_PUSH_ENGINE=background` followed by a second push showing the previous result.
- _Done 2026-10-02:_ `--engine inline|background|off` (hook: `OUSAST_PUSH_ENGINE`) and `--background-deadline` (default 900 s). `off` and `background` run the quick tier and record `engine_off`/`engine_background` with a plain skip line; the deadline line names the deadline and the background option. New `push/background.py` starts one detached explicit replay (`start_new_session`, stdin closed, output to a log) per artifact directory and repository (lock file with a live-pid check: a second start reports busy), writes its artifact under `<artifact dir>/engine/<repo key>/<head>.json`, and the next run prints "Engine result for <head> from the previous push: ..." once, then marks it shown. Not usable through the Docker wrapper (the container exits with the hook); documented in 17.6. Tests: `tests/test_push_background.py` (4, one launching a real detached run). Gates: 2011 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.5, 4.1, 4.2_
- [x] 17.6 Install, Docker and documentation for a first-time user
- `ousast pre-push install|uninstall [--force] [--docker] [REPO]` ships the hook as package data, honours `core.hooksPath`, refuses an existing hook unless `--force` (which chains it as `pre-push.before-ousast`), and works from a pip install. `ops/ousast-docker` is a hook-callable script (not a shell function) that passes stdin and the artifact path through. The `openultrasast:dev` image is rebuilt only if `df -h /` shows at least 4 GB free; otherwise the rebuild command is documented and the record says it was not rebuilt. `docs/scanning.md` gains a first-run guide, a glossary of skip reasons, a GitHub Actions recipe and full `--help` text; README tiers 1 and 2 updated, pulling `ghcr.io/norandom/openultrasast:2.0.1` where the wrapper allows.
- Files: `src/openultrasast/push/hook/pre-push`, `src/openultrasast/push/install.py` (new), `src/openultrasast/cli.py`, `ops/ousast-docker`, `docs/scanning.md`, `README.md`, `ops/README.md`. Tests: `tests/test_push_install.py` (new), `mkdocs build --strict`.
- Evidence: install from the package into a scratch clone, push through it, uninstall restores the prior hook.
- _Done 2026-10-02:_ `ousast pre-push install|uninstall [REPO] [--force] [--docker]` (`push/install.py`) installs the hook shipped as package data (`openultrasast/push/hook/pre-push`, byte-identical to `ops/pre-push`, verified in the built wheel), honours `core.hooksPath` without editing it, refuses an existing hook unless `--force` (ours is replaced; a foreign hook moves to `pre-push.before-ousast` and is chained), and uninstall removes only a hook it wrote and restores the chained one. `ops/ousast-docker` (also package data; `--docker` installs it beside the hook, which then uses it) is a script that mounts the repository read-only and the artifact directory at their own absolute paths, passes stdin with `-i`, runs as the user with no network and `--memory 3g`, defaults to `ghcr.io/norandom/openultrasast:2.0.1`, never pulls inside a hook, and exits 0 unless `OUSAST_PUSH_MODE=blocking`. The hook passes `--engine`/`--comparison-base` only when set, so an older ousast or image still runs. Every pre-push flag has help text; `docs/scanning.md` has the first-run guide, settings table, new-branch and background notes, the artifact keys, a skip-reason glossary (a test keeps it in sync with `push/explain.py`), a GitHub Actions recipe and the full `--help` text; README tiers 1 and 2 and `ops/README.md` updated (pull and tag the GHCR image instead of building). **Image not rebuilt:** `df -h /` showed 1.1 GB free (< 4 GB); rebuild with `docker build -t openultrasast:dev .`. Tests: `tests/test_push_install.py` (11) and the glossary test. Gates: 2022 passed, 42 skipped; ruff, format, mypy clean; `mkdocs build --strict` exit 0.
- _Requirements: 9.6, 7.4_
- [x] 17.7 Prove it on the audit's three repositories
- Rerun records, serve-static and unrolled/secure (same planted commits) with the new build in quick mode and with the hook installed, plus one Docker engine run on serve-static if free memory >= 3 GB and free disk >= 1 GB. Prove each run read its input (files, bytes).
- Files: `benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json`. Evidence: before (the audit) and after: exit codes, wall time against the deadline, findings shown, every skip line printed. A planted sample still missed is a coverage gap, recorded, never a rule edit.
- _Done 2026-10-02:_ `benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json` (before = the audit, after = this build installed from its wheel into a scratch venv). 10 hook pushes and 6 replays, all exit 0, 0.69-1.21 s against 30 s on the host; the quick tier read exactly the changed file and every head snapshot read the full tree (records 43,577 B, serve-static 61,525 B, secure 103,023 B). The planted JS `eval` in the new `debug.js` surfaces: quick tier on the host, and as two advisory engine findings classified `new`/`file_added_in_head` in one Docker engine run (190.9 s against a 300 s deadline; MemAvailable 3.28 GB, 1.16 GB disk). **The planted Python `%`-formatted SQL is still missed**: no quick rule matches it and the host has no engine. Recorded as a coverage gap, no rule edited. Go is named as not covered. The proof caught a wrong install hint (`pip install openultrasast`, not on PyPI), corrected in the hook in this commit.
- _Requirements: 9.1–9.6, 6.3_
## Implementation Notes
- Task5.2 measured PMPro timing out after both snapshots were materialized but before change context/graph analysis. Group6 must account for compatible snapshot/discovery reuse as well as graph reuse; identical-tip hits cannot substitute for the later representative changed-code latency gate. No new detector, scope heuristic or incremental CPG mutation is implied.
+5 -6
View File
@@ -200,10 +200,10 @@ Counted 2026-10-02 from each spec's `tasks.md` checkboxes and `spec.json`.
|---|---|---|
| `learned-decision-engine` | implementation | 11 of 17 tasks done |
| `pre-push-safety-net` | implementation | 67 of 79 tasks done |
| `contributor-scan` | implementation | 27 of 51 tasks done |
| `ai-service-plane` | implementation | 11 of 11 tasks done; the ax plane is the agentic path |
| `contributor-scan` | paused | 27 of 51 tasks done; paused since 2026-09-23 (`v1.2.0-alpha.1` NO-GO), waits on pre-push M2/M3 |
| `ai-service-plane` | complete | 11 of 11; the ax plane is the agentic path |
| `harnessx-removal` | complete | 8 of 8; HarnessX removed 2026-09-30 |
| `finding-feedback-loop` | design | requirements approved, design awaiting approval |
| `finding-feedback-loop` | dormant | requirements approved, design unapproved since 2026-09-09; depends on `contributor-scan` |
| `model-grounded-detection` | complete | 16 of 16; built the arbiter this all rests on |
| `authorization-obligations` | dormant | tasks approved, 0 of 24 started; superseded in practice by the dominance arbiter |
| `reachability-flow-model` | dormant | 23 tasks generated, nothing approved; largely delivered by adopting Joern |
@@ -214,6 +214,5 @@ Counted 2026-10-02 from each spec's `tasks.md` checkboxes and `spec.json`.
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax (`ai-service-plane`),
> and evolutionary proposals come from the plane memory. The HarnessX specs in the table above are history.
As of 2026-09-09: 97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests.
150 source modules, 145 load-bearing and 5 standalone capabilities, 0 orphaned
(`benchmarks/measurements/2026-09-08-module-audit.json`, regenerated 2026-10-02); 1,964 tests collected.
151 source modules, 146 load-bearing and 5 standalone capabilities, 0 orphaned
(`benchmarks/measurements/2026-09-08-module-audit.json`, regenerated 2026-10-02); 2,023 tests collected.
-165
View File
@@ -1,165 +0,0 @@
---
name: kiro-spec-batch
description: Create complete specs (requirements, design, tasks) for all features in roadmap.md using parallel sub-agent dispatch by dependency wave.
---
# Spec Batch
<background_information>
- **Success Criteria**:
- All features have complete spec files (spec.json, requirements.md, design.md, tasks.md)
- Dependency ordering respected (upstream specs complete before downstream)
- Independent features processed in parallel via sub-agent dispatch
- Cross-spec consistency verified (data models, interfaces, naming)
- Mixed roadmap context understood without breaking `## Specs (dependency order)` parsing
- Controller context stays lightweight (sub-agents do the heavy work)
</background_information>
<instructions>
## Step 1: Read Roadmap and Validate
1. Read `.kiro/steering/roadmap.md`
2. Parse the `## Specs (dependency order)` section to extract:
- Feature names
- One-line descriptions
- Dependencies for each feature
- Completion status (`[x]` = done, `[ ]` = pending)
3. If present, also read for context:
- `## Existing Spec Updates`
- `## Direct Implementation Candidates`
Do not include these in dependency-wave execution; they are awareness-only inputs for sequencing and consistency review.
4. For each pending feature in `## Specs (dependency order)`, verify `.kiro/specs/<feature>/brief.md` exists
5. If any brief.md is missing, stop and report: "Missing brief.md for: [list]. Run `/kiro-discovery` to generate briefs first."
## Step 2: Build Dependency Waves
Group pending features into waves based on dependencies:
- **Wave 1**: Features with no dependencies (or all dependencies already completed `[x]`)
- **Wave 2**: Features whose dependencies are all in Wave 1 or already completed
- **Wave N**: Features whose dependencies are all in earlier waves or already completed
Display the execution plan:
```
Spec Batch Plan:
Wave 1 (parallel): app-foundation
Wave 2 (parallel): block-editor, page-management
Wave 3 (parallel): sidebar-navigation, database-views
Wave 4 (parallel): cli-integration
Total: 6 specs across 4 waves
```
If roadmap contains `## Existing Spec Updates` or `## Direct Implementation Candidates`, mention them separately as non-batch items so the user can see the whole decomposition.
## Step 3: Execute Waves
For each wave, dispatch all features in the wave as **parallel sub-agents**.
**For each feature in the wave**, spawn a sub-agent with this task:
```
Create a complete specification for feature "{feature-name}".
1. Read the brief at .kiro/specs/{feature-name}/brief.md for feature context
2. Read the roadmap at .kiro/steering/roadmap.md for project context
3. Execute the full spec pipeline. For each phase, read the corresponding skill's SKILL.md for complete instructions (templates, rules, review gates):
a. Initialize: Read .opencode/skills/kiro-spec-init/SKILL.md, then create spec.json and requirements.md
b. Generate requirements: Read .opencode/skills/kiro-spec-requirements/SKILL.md, then follow its steps
c. Generate design: Read .opencode/skills/kiro-spec-design/SKILL.md, then follow its steps
d. Generate tasks: Read .opencode/skills/kiro-spec-tasks/SKILL.md, then follow its steps
4. Set all approvals to true in spec.json (auto-approve mode, equivalent of -y flag)
5. Report completion with file list and task count
```
If multi-agent is not available, execute features in the wave sequentially.
**After all sub-agents in the wave complete**:
1. Verify each feature has: spec.json, requirements.md, design.md, tasks.md
2. If any feature failed, report the error and continue with features that succeeded
3. Display wave completion: "Wave N complete: [features]. Files verified."
4. Proceed to next wave
## Step 4: Cross-Spec Review
After all waves complete, spawn a **single sub-agent** for cross-spec consistency review. This is the highest-value quality gate -- it catches issues that per-spec review gates cannot.
**Sub-agent task**:
Read ALL generated specs and check for consistency across the entire project:
- `.kiro/specs/*/design.md` (primary: contains interfaces, data models, architecture)
- `.kiro/specs/*/requirements.md` (for scope and acceptance criteria)
- `.kiro/specs/*/tasks.md` (for boundary annotations only -- read _Boundary:_ lines, skip task descriptions)
- `.kiro/steering/roadmap.md`
Reading priority: Focus on design.md files (they contain interfaces, data models, architecture). For requirements.md, focus on section headings and acceptance criteria. For tasks.md, focus on _Boundary:_ annotations.
Check:
1. **Data model consistency**: Same entities defined consistently across specs (field names, types, relationships)
2. **Interface alignment**: Where spec A outputs what spec B consumes, do contracts match exactly?
3. **No duplicate functionality**: Any capability specified in more than one spec?
4. **Dependency completeness**: Every design.md references correct upstream specs? Implicit dependencies not in roadmap?
5. **Naming conventions**: Component names, file paths, API routes, table names consistent across specs?
6. **Shared infrastructure**: Shared concerns (auth, error handling, logging) handled in one spec and correctly referenced?
7. **Task boundary alignment**: Task _Boundary:_ annotations partition codebase cleanly? No files claimed by multiple specs?
8. **Roadmap boundary continuity**: If roadmap includes `Existing Spec Updates` or `Direct Implementation Candidates`, do the generated new specs avoid absorbing that work by accident?
9. **Architecture boundary integrity**: Do the specs preserve clean responsibility seams, avoid shared ownership, keep dependency direction coherent, and include enough revalidation triggers to catch downstream impact?
10. **Change-friendly decomposition**: Has any spec absorbed multiple independent seams that should probably be split instead of kept together?
Output: CONSISTENT areas + ISSUES with (which specs, what's inconsistent, suggested fix).
**After the review sub-agent returns**:
- **Critical/important issues found**: Dispatch fix sub-agents for each affected spec to apply the suggested fixes. If the issue is really a decomposition problem (for example boundary overlap or one spec carrying multiple independent seams), stop and return to roadmap/discovery instead of papering over it locally. Re-run cross-spec review after fixes (max 3 remediation rounds).
- **Minor issues only**: Report them for user awareness, proceed to Step 5.
- **No issues**: Proceed to Step 5.
## Step 5: Finalize
1. Scan `.kiro/specs/*/tasks.md` to verify all specs exist
2. For each completed spec, read spec.json to confirm phase and approvals
3. Update roadmap.md: mark completed specs as `[x]`
4. If roadmap.md includes `Existing Spec Updates` or `Direct Implementation Candidates`, leave them untouched and mention them as remaining follow-up items unless already explicitly completed elsewhere
Display final summary:
```
Spec Batch Complete:
✓ app-foundation: X requirements, Y design components, Z tasks
✓ block-editor: ...
✓ page-management: ...
...
Total: N specs created, M tasks generated
Cross-spec review: PASSED / N issues found (M fixed)
Existing spec updates pending: <count or none>
Direct implementation candidates pending: <count or none>
Next: Review generated specs, then start implementation with /kiro-impl <feature>
```
</instructions>
## Critical Constraints
- **Controller stays lightweight**: Only read roadmap.md and brief.md existence checks in main context. All spec generation happens in sub-agents.
- **Wave ordering is strict**: Never start a wave until all features in previous waves are complete.
- **Parallel within waves**: All features in the same wave should be dispatched in parallel if multi-agent is available.
- **No partial waves**: If a feature in a wave fails, still complete the other features in that wave before reporting.
- **Skip completed specs**: Features with `[x]` in roadmap.md or existing tasks.md are skipped.
- **`## Specs (dependency order)` remains authoritative for batch execution**: Other roadmap sections are context, not wave inputs.
## Safety & Fallback
**Sub-agent failure**:
- Log the error, skip the failed feature
- Continue with remaining features in the wave
- Report failed features in the summary
- Suggest: "Run `/kiro-spec-quick <feature> --auto` manually for failed features."
**Circular dependencies**:
- If dependency graph has cycles, report the cycle and stop
- Suggest: "Fix dependency ordering in roadmap.md"
**Roadmap not found**:
- Stop and report: "No roadmap.md found. Run `/kiro-discovery` first."
**All specs already complete**:
- Report: "All specs in roadmap.md are already complete. Nothing to do."
-256
View File
@@ -1,256 +0,0 @@
---
name: kiro-spec-quick
description: Quick spec generation with interactive or automatic mode
---
# Quick Spec Generator
<instructions>
## CRITICAL: Automatic Mode Execution Rules
**If `--auto` flag is present in `$ARGUMENTS`, you are in AUTOMATIC MODE.**
In Automatic Mode:
- Execute ALL 4 phases in a continuous loop without stopping
- Display progress after each phase (e.g., "Phase 1/4 complete: spec initialized")
- IGNORE any "Next Step" messages from Phase 2-4 (they are for standalone usage)
- After Phase 4, run the final sanity review before exiting
- Stop ONLY after the sanity review completes or if error occurs
---
## Core Task
Execute 4 spec phases sequentially. In automatic mode, execute all phases without stopping. In interactive mode, prompt user for approval between phases.
Before claiming quick generation is complete, run one lightweight sanity review over the generated requirements, design, and tasks. If the host supports fresh sub-agents, use one. Otherwise run the sanity review inline.
## Execution Steps
### Step 1: Parse Arguments and Initialize
Parse `$ARGUMENTS`:
- If contains `--auto`: **Automatic Mode** (execute all 4 phases)
- Otherwise: **Interactive Mode** (prompt at each phase)
- Extract description (remove `--auto` flag if present)
Example:
```
"User profile with avatar upload --auto" → mode=automatic, description="User profile with avatar upload"
"User profile feature" → mode=interactive, description="User profile feature"
```
Display mode banner and proceed to Step 2.
### Step 2: Execute Phase Loop
Execute these 4 phases in order:
---
#### Phase 1: Initialize Spec (Direct Implementation)
**Core Logic**:
1. **Check for Brief**:
- If `.kiro/specs/{feature-name}/brief.md` exists (created by `/kiro-discovery`), read it for discovery context (problem, approach, scope, constraints)
- Use brief content as the project description instead of `$ARGUMENTS`
2. **Generate Feature Name**:
- Convert description to kebab-case
- Example: "User profile with avatar upload" → "user-profile-avatar-upload"
- Keep name concise (2-4 words ideally)
3. **Check Uniqueness**:
- Use Glob to check `.kiro/specs/*/`
- If directory exists with only `brief.md` (no `spec.json`), use that directory (discovery created it)
- Otherwise if feature name exists, append `-2`, `-3`, etc.
4. **Create Directory**:
- Use Bash: `mkdir -p .kiro/specs/{feature-name}` (skip if already exists from discovery)
5. **Initialize Files from Templates**:
a. Read templates:
```
- .kiro/settings/templates/specs/init.json
- .kiro/settings/templates/specs/requirements-init.md
```
b. Replace placeholders:
```
{{FEATURE_NAME}} → feature-name
{{TIMESTAMP}} → current ISO 8601 timestamp (use `date -u +"%Y-%m-%dT%H:%M:%SZ"`)
{{PROJECT_DESCRIPTION}} → description
en → language code (detect from user's input language, default to `en`)
```
c. Write files using Write tool:
```
- .kiro/specs/{feature-name}/spec.json
- .kiro/specs/{feature-name}/requirements.md
```
6. **Output Progress**: "Phase 1/4 complete: Spec initialized at .kiro/specs/{feature-name}/"
**Automatic Mode**: IMMEDIATELY continue to Phase 2.
**Interactive Mode**: Prompt "Continue to requirements generation? (yes/no)"
- If "no": Stop, show current state
- If "yes": Continue to Phase 2
---
#### Phase 2: Generate Requirements
Invoke `/kiro-spec-requirements {feature-name}`.
Wait for completion. IGNORE any "Next Step" message (it is for standalone usage).
**Output Progress**: "Phase 2/4 complete: Requirements generated"
**Automatic Mode**: IMMEDIATELY continue to Phase 3.
**Interactive Mode**: Prompt "Continue to design generation? (yes/no)"
- If "no": Stop, show current state
- If "yes": Continue to Phase 3
---
#### Phase 3: Generate Design
Invoke `/kiro-spec-design {feature-name} -y`. The `-y` flag auto-approves requirements.
Wait for completion. IGNORE any "Next Step" message.
**Output Progress**: "Phase 3/4 complete: Design generated"
**Automatic Mode**: IMMEDIATELY continue to Phase 4.
**Interactive Mode**: Prompt "Continue to tasks generation? (yes/no)"
- If "no": Stop, show current state
- If "yes": Continue to Phase 4
---
#### Phase 4: Generate Tasks
Invoke `/kiro-spec-tasks {feature-name} -y`. The `-y` flag auto-approves requirements, design, and tasks.
Wait for completion.
**Output Progress**: "Phase 4/4 complete: Tasks generated"
#### Final Sanity Review
After Phase 4, run a lightweight sanity review before claiming completion.
- Review `requirements.md`, `design.md`, and `tasks.md` directly from disk. If `brief.md` exists, use it only as supporting context.
- Prefer a fresh review sub-agent when the host supports it. Pass only file paths and the review objective; the reviewer should read the generated files itself.
- Review focus:
- Do requirements, design, and tasks tell a coherent story?
- Are there obvious contradictions, missing prerequisites, or missing task coverage for required design work?
- Are `_Depends:_`, `_Boundary:_`, and `(P)` markers plausible for implementation?
- If the review finds only task-plan-local issues, repair or update the generated `tasks.md` once, then re-run the sanity review.
- If the review finds a real requirements/design gap or contradiction, stop and report follow-up instead of claiming the quick spec is implementation-ready.
**All 4 phases plus sanity review complete.**
Output final completion summary (see Output Description section) and exit.
---
## Important Constraints
### Error Handling
- Any phase failure stops the workflow
- Display error and current state
- Suggest manual recovery command
</instructions>
## Output Description
### Mode Banners
**Interactive Mode**:
```
Quick Spec Generation (Interactive Mode)
You will be prompted at each phase.
Note: Skips gap analysis and design validation.
```
**Automatic Mode**:
```
Quick Spec Generation (Automatic Mode)
All phases execute automatically without prompts.
Note: Skips optional validations (gap analysis, design review) and user approval prompts. Internal review gates still run.
Final sanity review still runs.
```
### Intermediate Output
After each phase, show brief progress:
```
Spec initialized at .kiro/specs/{feature}/
Requirements generated → Continuing to design...
Design generated → Continuing to tasks...
```
### Final Completion Summary
Provide output in the language specified in `spec.json`:
```
Quick Spec Generation Complete!
## Generated Files:
- .kiro/specs/{feature}/spec.json
- .kiro/specs/{feature}/requirements.md ({X} requirements)
- .kiro/specs/{feature}/design.md ({Y} components, {Z} endpoints)
- .kiro/specs/{feature}/tasks.md ({N} tasks)
Quick generation skipped:
- `/kiro-validate-gap` - Gap analysis (integration check)
- `/kiro-validate-design` - Design review (architecture validation)
Sanity review: PASSED | FOLLOW-UP REQUIRED
## Next Steps:
1. Review generated specs (especially design.md)
2. Optional validation:
- `/kiro-validate-gap {feature}` - Check integration with existing codebase
- `/kiro-validate-design {feature}` - Verify architecture quality
3. Start implementation: `/kiro-impl {feature}`
```
## Safety & Fallback
### Error Scenarios
**Template Missing**:
- Check `.kiro/settings/templates/specs/` exists
- Report specific missing file
- Exit with error
**Directory Creation Failed**:
- Check permissions
- Report error with path
- Exit with error
**Phase Execution Failed** (Phase 2-4):
- Stop workflow
- Show current state and completed phases
- Suggest: "Continue manually from `/kiro-spec-{next-phase} {feature}`"
**Sanity Review Failed**:
- Stop workflow
- Report the exact contradiction, missing prerequisite, or task-plan issue
- Suggest targeted follow-up with `/kiro-spec-design {feature}`, `/kiro-spec-tasks {feature}`, or manual edits depending on the finding
**User Cancellation** (Interactive Mode):
- Stop gracefully
- Show completed phases
- Suggest manual continuation
@@ -1,145 +0,0 @@
---
name: kiro-steering-custom
description: Create custom steering documents for specialized project contexts
metadata:
shared-rules: "steering-principles.md"
---
# Kiro Custom Steering Creation
<background_information>
**Role**: Create specialized steering documents beyond core files (product, tech, structure).
**Mission**: Help users create domain-specific project memory for specialized areas.
**Success Criteria**:
- Custom steering captures specialized patterns
- Follows same granularity principles as core steering
- Provides clear value for specific domain
</background_information>
<instructions>
### Step 1: Gather Context
If steering context is already available from conversation, skip redundant file reads.
Otherwise:
- Check `.kiro/settings/templates/steering-custom/` for available templates
- Read `rules/steering-principles.md` from this skill's directory for steering principles
## Workflow
1. **Ask user** for custom steering needs:
- Domain/topic (e.g., "API standards", "testing approach")
- Specific requirements or patterns to document
2. **Check if template exists**:
- Load from `.kiro/settings/templates/steering-custom/{name}.md` if available
- Use as starting point, customize based on project
3. **Analyze codebase** (JIT) for relevant patterns:
#### Parallel Research
The following research areas are independent and can be executed in parallel:
1. **Template & principles**: Load matching template and steering-principles.md
2. **Domain patterns**: Analyze codebase for domain-specific patterns using Glob/Grep/Read
If multi-agent is enabled, spawn sub-agents for each area above. Otherwise execute sequentially.
After all parallel research completes, synthesize findings for steering document.
4. **Generate custom steering**:
- Follow template structure if available
- Apply principles from `rules/steering-principles.md` from this skill's directory
- Focus on patterns, not exhaustive lists
- Keep to 100-200 lines (2-3 minute read)
5. **Create file** in `.kiro/steering/{name}.md`
## Available Templates
Templates available in `.kiro/settings/templates/steering-custom/`:
1. **api-standards.md** - REST/GraphQL conventions, error handling
2. **testing.md** - Test organization, mocking, coverage
3. **security.md** - Auth patterns, input validation, secrets
4. **database.md** - Schema design, migrations, query patterns
5. **error-handling.md** - Error types, logging, retry strategies
6. **authentication.md** - Auth flows, permissions, session management
7. **deployment.md** - CI/CD, environments, rollback procedures
Load template when needed, customize for project.
## Steering Principles
From `rules/steering-principles.md` (in this skill's directory):
- **Patterns over lists**: Document patterns, not every file/component
- **Single domain**: One topic per file
- **Concrete examples**: Show patterns with code
- **Maintainable size**: 100-200 lines typical
- **Security first**: Never include secrets or sensitive data
</instructions>
## Tool guidance
- **Read**: Load template, analyze existing code
- **Glob**: Find related files for pattern analysis
- **Grep**: Search for specific patterns
- **Bash** with `ls`: Understand relevant structure
**JIT Strategy**: Load template only when creating that type of steering.
## Output description
Chat summary with file location (file created directly).
```
✅ Custom Steering Created
## Created:
- .kiro/steering/api-standards.md
## Based On:
- Template: api-standards.md
- Analyzed: src/api/ directory patterns
- Extracted: REST conventions, error format
## Content:
- Endpoint naming patterns
- Request/response format
- Error handling conventions
- Authentication approach
Review and customize as needed.
```
## Examples
### Success: API Standards
**Input**: "Create API standards steering"
**Action**: Load template, analyze src/api/, extract patterns
**Output**: api-standards.md with project-specific REST conventions
### Success: Testing Strategy
**Input**: "Document our testing approach"
**Action**: Load template, analyze test files, extract patterns
**Output**: testing.md with test organization and mocking strategies
## Safety & Fallback
- **No template**: Generate from scratch based on domain knowledge
- **Security**: Never include secrets (load principles)
- **Validation**: Ensure doesn't duplicate core steering content
## Notes
- Templates are starting points, customize for project
- Follow same granularity principles as core steering
- All steering files loaded as project memory
- Custom files equally important as core files
- Avoid documenting agent-specific tooling directories (e.g. `.cursor/`, `.gemini/`, `.claude/`)
- Light references to `.kiro/specs/` and `.kiro/steering/` are acceptable; avoid other `.kiro/` directories
@@ -1,90 +0,0 @@
# Steering Principles
Steering files are **project memory**, not exhaustive specifications.
---
## Content Granularity
### Golden Rule
> "If new code follows existing patterns, steering shouldn't need updating."
### ✅ Document
- Organizational patterns (feature-first, layered)
- Naming conventions (PascalCase rules)
- Import strategies (absolute vs relative)
- Architectural decisions (state management)
- Technology standards (key frameworks)
### ❌ Avoid
- Complete file listings
- Every component description
- All dependencies
- Implementation details
- Agent-specific tooling directories (e.g. `.cursor/`, `.gemini/`, `.claude/`)
- Detailed documentation of `.kiro/` metadata directories (settings, automation)
### Example Comparison
**Bad** (Specification-like):
```markdown
- /components/Button.tsx - Primary button with variants
- /components/Input.tsx - Text input with validation
- /components/Modal.tsx - Modal dialog
... (50+ files)
```
**Good** (Project Memory):
```markdown
## UI Components (`/components/ui/`)
Reusable, design-system aligned primitives
- Named by function (Button, Input, Modal)
- Export component + TypeScript interface
- No business logic
```
---
## Security
Never include:
- API keys, passwords, credentials
- Database URLs, internal IPs
- Secrets or sensitive data
---
## Quality Standards
- **Single domain**: One topic per file
- **Concrete examples**: Show patterns with code
- **Explain rationale**: Why decisions were made
- **Maintainable size**: 100-200 lines typical
---
## Preservation (when updating)
- Preserve user sections and custom examples
- Additive by default (add, don't replace)
- Add `updated_at` timestamp
- Note why changes were made
---
## Notes
- Templates are starting points, customize as needed
- Follow same granularity principles as core steering
- All steering files loaded as project memory
- Light references to `.kiro/specs/` and `.kiro/steering/` are acceptable; avoid other `.kiro/` directories
- Custom files equally important as core files
---
## File-Specific Focus
- **product.md**: Purpose, value, business context (not exhaustive features)
- **tech.md**: Key frameworks, standards, conventions (not all dependencies)
- **structure.md**: Organization patterns, naming rules (not directory trees)
- **Custom files**: Specialized patterns (API, testing, security, etc.)
@@ -1,103 +0,0 @@
---
name: kiro-validate-design
description: Interactive technical design quality review and validation
metadata:
shared-rules: "design-review.md"
---
# Technical Design Validation
<background_information>
- **Mission**: Conduct interactive quality review of technical design to ensure readiness for implementation
- **Success Criteria**:
- Critical issues identified (maximum 3 most important concerns)
- Balanced assessment with strengths recognized
- Clear GO/NO-GO decision with rationale
- Actionable feedback for improvements if needed
</background_information>
<instructions>
## Core Task
Interactive design quality review for feature **$1** based on approved requirements and design document.
## Execution Steps
1. **Gather Context**:
- Read `.kiro/specs/$1/spec.json` for language and metadata
- Read `.kiro/specs/$1/requirements.md` for requirements
- Read `.kiro/specs/$1/design.md` for design document
- Core steering context: `product.md`, `tech.md`, `structure.md`
- Additional steering files only when directly relevant to architecture boundaries, integrations, runtime prerequisites, domain rules, security/performance constraints, or team conventions that affect implementation readiness
- Relevant local agent skills or playbooks only when they clearly match the feature's host environment or use case and provide review-relevant context
#### Parallel Research
The following research areas are independent and can be executed in parallel:
1. **Context & rules loading**: Spec documents, core steering, task-relevant extra steering, relevant local agent skills/playbooks, and `rules/design-review.md` from this skill's directory for review criteria
2. **Codebase pattern survey**: Gather existing architecture patterns, naming conventions, and component structure from the codebase to use as reference during review
If multi-agent is enabled, spawn sub-agents for each area above. Otherwise execute sequentially.
After all parallel research completes, synthesize findings for review.
2. **Execute Design Review**:
- Reference conversation history when available: leverage prior requirements discussion and user's stated design intent
- Follow design-review.md process: Analysis → Critical Issues → Strengths → GO/NO-GO
- Limit to 3 most important concerns
- Engage interactively with user — ask clarifying questions, propose alternatives
- Use language specified in spec.json for output
3. **Decision and Next Steps**:
- Clear GO/NO-GO decision with rationale
- Provide specific actionable next steps (see Next Phase below)
## Important Constraints
- **Quality assurance, not perfection seeking**: Accept acceptable risk
- **Critical focus only**: Maximum 3 issues, only those significantly impacting success
- **Conversation-aware**: Leverage discussion history for requirements context and user intent when available
- **Interactive approach**: Engage in dialogue, ask clarifying questions, propose alternatives
- **Balanced assessment**: Recognize both strengths and weaknesses
- **Actionable feedback**: All suggestions must be implementable
- **Context Discipline**: Start with core steering and expand only with review-relevant steering or use-case-aligned local agent skills/playbooks
</instructions>
## Tool Guidance
- **Read first**: Load spec, core steering, relevant local playbooks/agent skills, and rules before review
- **Grep if needed**: Search codebase for pattern validation or integration checks
- **Interactive**: Engage with user throughout the review process
## Output Description
Provide output in the language specified in spec.json with:
1. **Review Summary**: Brief overview (2-3 sentences) of design quality and readiness
2. **Critical Issues**: Maximum 3, following design-review.md format
3. **Design Strengths**: 1-2 positive aspects
4. **Final Assessment**: GO/NO-GO decision with rationale and next steps
**Format Requirements**:
- Use Markdown headings for clarity
- Follow design-review.md output format
- Keep summary concise
## Safety & Fallback
### Error Scenarios
- **Missing Design**: If design.md doesn't exist, stop with message: "Run `/kiro-spec-design $1` first to generate design document"
- **Design Not Generated**: If design phase not marked as generated in spec.json, warn but proceed with review
- **Empty Steering Directory**: Warn user that project context is missing and may affect review quality
- **Language Undefined**: Default to English (`en`) if spec.json doesn't specify language
### Next Phase: Task Generation
**If Design Passes Validation (GO Decision)**:
- Review feedback and apply changes if needed
- Run `/kiro-spec-tasks $1` to generate implementation tasks
- Or `/kiro-spec-tasks $1 -y` to auto-approve and proceed directly
**If Design Needs Revision (NO-GO Decision)**:
- Address critical issues identified
- Re-run `/kiro-spec-design $1` with improvements
- Re-validate with `/kiro-validate-design $1`
**Note**: Design validation is recommended but optional. Quality review helps catch issues early.
-109
View File
@@ -1,109 +0,0 @@
---
name: kiro-validate-gap
description: Analyze implementation gap between requirements and existing codebase
metadata:
shared-rules: "gap-analysis.md"
---
# Implementation Gap Validation
<background_information>
- **Mission**: Analyze the gap between requirements and existing codebase to inform implementation strategy
- **Success Criteria**:
- Comprehensive understanding of existing codebase patterns and components
- Clear identification of missing capabilities and integration challenges
- Multiple viable implementation approaches evaluated
- Technical research needs identified for design phase
</background_information>
<instructions>
## Core Task
Analyze implementation gap for feature **$1** based on approved requirements and existing codebase.
## Execution Steps
1. **Load Context**:
- Read `.kiro/specs/$1/spec.json` for language and metadata
- Read `.kiro/specs/$1/requirements.md` for requirements
- Core steering context: `product.md`, `tech.md`, `structure.md`
- Additional steering files only when directly relevant to the feature's domain rules, integrations, runtime prerequisites, compliance/security constraints, or existing product boundaries
- Relevant local agent skills or playbooks only when they clearly match the feature's host environment or use case and provide analysis-relevant context
2. **Read Analysis Guidelines**:
- Read `rules/gap-analysis.md` from this skill's directory for comprehensive analysis framework
#### Parallel Research
The following research areas are independent and can be executed in parallel:
1. **Codebase analysis**: Existing implementations, architecture patterns, integration points, extension possibilities
2. **External dependency research**: Dependency compatibility, version constraints, known integration challenges (when needed)
3. **Context loading**: Requirements, core steering, task-relevant extra steering, relevant local agent skills/playbooks, and gap-analysis rules
If multi-agent is enabled, spawn sub-agents for each area above. Otherwise execute sequentially.
After all parallel research completes, synthesize findings for gap analysis.
3. **Execute Gap Analysis**:
- Follow gap-analysis.md framework for thorough investigation
- Analyze existing codebase using Grep and Read tools
- Use WebSearch/WebFetch for external dependency research if needed
- Evaluate multiple implementation approaches (extend/new/hybrid)
- Use language specified in spec.json for output
4. **Generate Analysis Document**:
- Create comprehensive gap analysis following the output guidelines in gap-analysis.md
- Present multiple viable options with trade-offs
- Flag areas requiring further research
5. **Write Gap Analysis to Disk**:
**Write the gap analysis to disk so it survives session boundaries and can be referenced during design phase.**
- Save the gap analysis to `.kiro/specs/$1/research.md`
- If the file already exists, append the new analysis (separated by a horizontal rule `---`) rather than overwriting previous research
- Verify the file was written by reading it back
## Important Constraints
- **Information over Decisions**: Provide analysis and options, not final implementation choices
- **Multiple Options**: Present viable alternatives when applicable
- **Thorough Investigation**: Use tools to deeply understand existing codebase
- **Explicit Gaps**: Clearly flag areas needing research or investigation
- **Context Discipline**: Start with core steering and expand only with analysis-relevant steering or use-case-aligned local agent skills/playbooks
</instructions>
## Tool Guidance
- **Read first**: Load spec, core steering, relevant local playbooks/agent skills, and rules before analysis
- **Grep extensively**: Search codebase for patterns, conventions, and integration points
- **WebSearch/WebFetch**: Research external dependencies and best practices when needed
- **Write last**: Generate analysis only after complete investigation
## Output Description
Provide output in the language specified in spec.json with:
1. **Analysis Summary**: Brief overview (3-5 bullets) of scope, challenges, and recommendations
2. **Document Status**: Confirm analysis approach used
3. **Next Steps**: Guide user on proceeding to design phase
**Format Requirements**:
- Use Markdown headings for clarity
- Keep summary concise (under 300 words)
- Detailed analysis follows gap-analysis.md output guidelines
## Safety & Fallback
### Error Scenarios
- **Missing Requirements**: If requirements.md doesn't exist, stop with message: "Run `/kiro-spec-requirements $1` first to generate requirements"
- **Requirements Not Approved**: If requirements not approved, warn user but proceed (gap analysis can inform requirement revisions)
- **Empty Steering Directory**: Warn user that project context is missing and may affect analysis quality
- **Complex Integration Unclear**: Flag for comprehensive research in design phase rather than blocking
- **Language Undefined**: Default to English (`en`) if spec.json doesn't specify language
### Next Phase: Design Generation
**If Gap Analysis Complete**:
- Review gap analysis insights
- Run `/kiro-spec-design $1` to create technical design document
- Or `/kiro-spec-design $1 -y` to auto-approve requirements and proceed directly
**Note**: Gap analysis is optional but recommended for brownfield projects to inform design decisions.
@@ -1,26 +0,0 @@
---
name: openultrasast-fix-audit
description: OpenUltraSAST patch, fix audit, differential review, adversarial review, and patch validation workflow. Use when proposing or auditing fixes for OpenUltraSAST findings from opencode.
---
# OpenUltraSAST Fix Audit
Use this skill to steer fixes for OpenUltraSAST findings.
## Fix Lifecycle
1. Intake: restate the finding ID, evidence level, affected path, and acceptance criteria.
2. Plan: choose the smallest defensive change that addresses the root cause.
3. Implement: modify only files required by the finding.
4. Adversarial review: look for bypasses, regressions, false assumptions, and unrelated edits.
5. Reconcile: either fix the review finding or document why it is not applicable.
6. Verify: rerun the reproducer or affected checks.
7. Ready: only then consider advancing to `patch_validated`.
## Guardrails
- Do not rewrite modules when a local fix is enough.
- Do not change public APIs unless the vulnerability requires it.
- Do not mark a patch validated from model reasoning alone.
- Use differential review for risky changes in auth, crypto, config, parsers, and sandbox behavior.
- Run Semgrep or CodeQL delta checks when those analyzers supplied the original evidence.
@@ -1,35 +0,0 @@
---
name: openultrasast-scan
description: OpenUltraSAST scan, index, quick, standard, deep, SARIF, and report workflows. Use when running or planning `ousast scan`, repository indexing, or evidence-gated SAST analysis from opencode.
---
# OpenUltraSAST Scan
Use this skill to steer OpenUltraSAST scan workflows from opencode.
## Workflow
1. Confirm the target is a local repository path.
2. Run commands through uv, never by activating a virtualenv manually.
3. Start with quick mode unless the task explicitly requires standard or deep analysis.
4. Treat quick findings as `static_corroboration`, not verified vulnerabilities.
5. Preserve run artifacts under `.openultrasast/runs/<scan-id>/`.
6. Report `scan_id`, `run_dir`, finding count, and the strongest available evidence level.
## Commands
```bash
uv run ousast scan <repo> --mode quick
uv run ousast index <repo>
uv run pytest
```
## Evidence Rules
- `suspicion` means a model or weak signal raised a hypothesis.
- `static_corroboration` means deterministic code evidence exists.
- `crash_reproduced` requires an artifact-backed reproducer.
- `root_cause_explained` requires independent verifier reasoning.
- `patch_validated` requires rerunning relevant checks after the patch.
Do not describe a finding as verified unless its evidence level meets the configured report threshold.
@@ -1,34 +0,0 @@
---
name: openultrasast-triage
description: OpenUltraSAST false-positive elimination, ranking calibration, evidence ladder, verifier triage, and finding adjudication. Use when reviewing findings, reducing false positives, or deciding whether evidence is enough.
---
# OpenUltraSAST Triage
Use this skill when triaging OpenUltraSAST findings from opencode.
## Triage Loop
1. Identify the finding, path, rule, severity, confidence, and evidence level.
2. Separate deterministic evidence from model assumptions.
3. Check attacker control, reachability, sanitizer behavior, duplicates, impact, and contradictory evidence.
4. If rejected or unresolved, record the false-positive reason and scope.
5. Prefer scoped demotion over deleting a vulnerability class globally.
6. Feed the outcome into ranking calibration and prompt constraints.
## False-Positive Reasons
- `unreachable_path`
- `missing_attacker_control`
- `sanitizer_disproved`
- `static_rule_mismatch`
- `incorrect_model_assumption`
- `duplicate`
- `insufficient_impact`
- `unsupported`
- `contradicted`
- `unverified`
## Decision Standard
If evidence is uncertain, mark the finding as requiring next evidence instead of escalating severity. Difficult or high-impact disagreements should trigger fusion/deeper reasoning when available.
+15 -15
View File
@@ -28,17 +28,14 @@ Project memory keeps persistent guidance (steering, specs notes, component docs)
- Think in English, generate responses in English. All Markdown content written to project files (e.g., requirements.md, design.md, tasks.md, research.md, validation reports) MUST be written in the target language configured for this specification (see spec.json.language).
## Minimal Workflow
- Phase 0 (optional): `/kiro-steering`, `/kiro-steering-custom`
- Phase 0 (optional): `/kiro-steering` (bootstrap, sync, or a custom steering document)
- Discovery: `/kiro-discovery "idea"` — determines action path, writes brief.md + roadmap.md for multi-spec projects
- Phase 1 (Specification):
- Single spec: `/kiro-spec-quick {feature} [--auto]` or step by step:
- `/kiro-spec-init "description"`
- `/kiro-spec-requirements {feature}`
- `/kiro-validate-gap {feature}` (optional: for existing codebase)
- `/kiro-spec-design {feature} [-y]`
- `/kiro-validate-design {feature}` (optional: design review)
- `/kiro-spec-tasks {feature} [-y]`
- Multi-spec: `/kiro-spec-batch` — creates all specs from roadmap.md in parallel by dependency wave
- Phase 1 (Specification), one spec at a time, each phase human-reviewed:
- `/kiro-spec-init "description"`
- `/kiro-spec-requirements {feature}`
- `/kiro-spec-design {feature} [-y | --gap | --review]` (`--gap`: optional gap analysis against the existing code first; `--review`: optional interactive review of an existing design; the design-review gate always runs)
- `/kiro-spec-tasks {feature} [-y]`
- Multi-spec projects: the specs of roadmap.md in dependency order, with the same steps each
- Phase 2 (Implementation): `/kiro-impl {feature} [tasks]`
- Without task numbers: autonomous mode (subagent per task + independent review + final validation)
- With task numbers: manual mode (selected tasks in main context, still reviewer-gated before completion)
@@ -46,14 +43,17 @@ Project memory keeps persistent guidance (steering, specs notes, component docs)
- Progress check: `/kiro-spec-status {feature}` (use anytime)
## Skills Structure
Skills are located in `.opencode/skills/kiro-*/SKILL.md`
Skills are located in `.agents/skills/*/SKILL.md` (the plural directory OpenCode searches by default; moved from `.opencode/skills` on 2026-10-02)
- Each skill is a directory with a `SKILL.md` file
- Use `/skills` to inspect currently available skills
- Invoke a skill directly with `/kiro-<skill-name>`
- **If there is even a 1% chance a skill applies to the current task, invoke it.** Do not skip skills because the task seems simple.
- Workflow skills: `kiro-steering`, `kiro-discovery`, `kiro-spec-init`, `kiro-spec-requirements`, `kiro-spec-design`, `kiro-spec-tasks`, `kiro-impl`, `kiro-validate-impl`, `kiro-spec-status`
- `kiro-review` — task-local adversarial review protocol used by reviewer subagents
- `kiro-debug` — root-cause-first debug protocol used by debugger subagents
- `kiro-verify-completion` — fresh-evidence gate before success or completion claims
- `kiro-debug` — root-cause-first debug protocol used by debugger subagents; carries the instrument rule below
- `kiro-verify-completion` — fresh-evidence gate before success or completion claims; carries the instrument rule below
- Product skills for users of the tool (the only ones user documentation names): `openultrasast-scan`, `openultrasast-triage`
- Reviewed 2026-10-02: `kiro-spec-quick`, `kiro-spec-batch` removed (no fast-track or parallel spec creation); `kiro-validate-gap` and `kiro-validate-design` merged into `kiro-spec-design` (`--gap`, `--review`); `kiro-steering-custom` merged into `kiro-steering`; `openultrasast-fix-audit` merged into `openultrasast-triage`
## Development Rules
- 3-phase approval workflow: Requirements → Design → Tasks → Implementation
@@ -81,5 +81,5 @@ So, before reporting any measurement:
## Steering Configuration
- Load entire `.kiro/steering/` as project memory
- Default files: `product.md`, `tech.md`, `structure.md`
- Custom files are supported (managed via `/kiro-steering-custom`)
- Core files in this repository: `overview.md` (current state wins over dated history), `roadmap.md`, `safety-net.md`
- Custom files may exist beside them and carry equal weight (created via `/kiro-steering`, templates under `.kiro/settings/templates/steering-custom/`)
+2
View File
@@ -10,6 +10,8 @@
# contributor a working tool from `docker compose run` with nothing installed on their machine.
FROM eclipse-temurin:21-jre-noble
LABEL org.opencontainers.image.licenses="Apache-2.0" \
org.opencontainers.image.source="https://github.com/norandom/OpenUltraSAST"
ARG JOERN_VERSION=v4.0.625
ARG ARCHIVE=joern-cli-linux-x86_64.zip
+202
View File
@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
+298 -183
View File
@@ -1,239 +1,326 @@
# OpenUltraSAST
**v2.0.0 (Python package `2.0.0`), state as of 2026-10-02.** An experimental static
security analyser. The pre-push safety net is **NO-GO** for rollout: no hook capability is
enabled and no independent population has passed the qualification gates (see
[docs/evaluation.md](docs/evaluation.md)). See the [release notes](RELEASE_NOTES.md).
OpenUltraSAST is an experimental static security analyser. It is meant to be a safety net on
repositories it has never seen. It scans a checkout, or the commits a `git push` would publish. It
keeps what it can prove apart from what it only suspects.
OpenUltraSAST separates what it can prove from what it merely suspects, and says which is which.
A scan combines language-scoped pattern rules, a semantic overlay, a Joern-based taint engine
and optional model calls; every finding carries the evidence that earned its place, and a scan
that could not look at something records that as a degradation instead of reporting a clean
result.
A scan combines up to four parts:
`ousast` is the interface. Model-driven work over whole repositories runs on a separate
agentic plane (google/ax), and a learned decision engine that weighs all the signals is in
development; neither is needed for a scan.
- language-scoped pattern rules;
- a tree-sitter overlay (a syntax tree of each file, built by the tree-sitter parser);
- a Joern code property graph (a graph of the program's syntax, control flow and data flow);
- optionally, a model. It is asked only the question the graph cannot settle.
## Install and quickstart
Every finding carries the evidence that earned its place. When a scan could not look at
something, it records a degradation. It does not report a clean result.
`ousast` is the interface. Model work over whole repositories runs on a separate agentic plane
(google/ax). A learned decision engine is in development. A scan needs neither of them.
This file is the front door. Each section links to its page of the documentation site.
The site is published at <https://norandom.github.io/OpenUltraSAST/>. Its sources are in `docs/` and
`mkdocs.yml`; to preview it locally, run `uv sync --extra docs && uv run mkdocs serve`.
## Status
**v2.0.0, state as of 2026-10-02.** The pre-push safety net is **NO-GO** against every M4 gate.
- No hook capability is enabled.
- Two independent populations (case sets frozen before any scan) have been spent so far. Engine
recall was 1 of 11 and then 0 of 17.
- The learned decision engine ranks but may not block. BLOCK is offered for no family.
The project has stopped trying to reach the gates by hand. Every detection change now has to win
a controlled measurement on held-out repositories. One page states all of this, with the record
behind each figure: [docs/where-we-stand.md](docs/where-we-stand.md). The release history is in
[RELEASE_NOTES.md](RELEASE_NOTES.md).
## Install and first scan
Pick the tier you need:
1. Tier 1: a deterministic scan of a checkout, or a CI gate.
2. Tier 2: the Joern engine (`standard`, `deep`).
3. Tier 3, for maintainers: model work over whole repositories on the plane.
4. Tier 4 does not exist yet.
### Tier 1: quick scan, no Docker
The core install depends only on PyYAML. From a checkout, `uv sync` (or `pip install .`) installs
the `ousast` entry point. `uv sync --group dev` adds the test and lint tools.
```bash
uv sync --group dev
# Deterministic scan, no model calls
uv sync
uv run ousast scan /path/to/repo --mode quick
# CI: exit 1 on any evidence-verified finding
uv run ousast scan . --mode quick --fail-on verified
uv run ousast scan . --mode quick --fail-on verified # CI: exit 1 on any evidence-verified finding
```
The core install depends only on PyYAML. Optional extras: `semantic` (tree-sitter grammars
for the overlay) and `s3` (the plane's memory store on an S3-compatible bucket, boto3).
This runs the language-scoped pattern rules, entry-point reachability hints, ranking,
verification and scoring. It is deterministic and reproducible. It needs no network and no keys.
For `standard` and `deep` scans, use the Docker image, which ships Joern (with `php-cli` for
the PHP frontend) next to the tool. Source the shell wrapper once and `ousast` runs in the
container with your directory mounted read-only:
The same install gives you the advisory pre-push hook for any repository. It runs the quick rules
on the lines a push changes, names every check it could not run, and never blocks a push:
```bash
cd /path/to/your/repository && ousast pre-push install # remove: ousast pre-push uninstall
```
What it cannot do:
- There is no engine. Nothing follows a value across statements or files.
- The PHP quick rules are a low-precision, in-sample development set. Per-rule precision lower
bounds run from 0.03 to 1.0 on the real pins
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`).
The optional `semantic` extra (`uv sync --extra semantic`) installs the tree-sitter grammars for
the overlay.
### Tier 2: standard and deep scans (Docker)
`standard` and `deep` need Joern. The Docker image ships three things in one image: the tool,
Joern v4.0.625 and `php-cli` (the PHP frontend needs it). Each release publishes it to GHCR. It is
big: a JRE plus about 2 GB of Joern. Pull it instead of building, and tag it with the name the
shell wrapper uses:
```bash
docker pull ghcr.io/norandom/openultrasast:2.0.1
docker tag ghcr.io/norandom/openultrasast:2.0.1 openultrasast:dev # or build: docker compose build
```
The shell wrapper runs `ousast` in the container. It builds the image only if `openultrasast:dev`
is missing. Your directory is mounted read-only and the container network is off. For the pre-push
hook, use `ousast pre-push install --docker`: Git cannot call a shell function, so it installs the
script `ousast-docker` beside the hook.
```bash
source /path/to/OpenUltraSAST/ops/shell/ousast.sh # PowerShell: ops/shell/ousast.ps1
ousast scan . --mode standard
ousast scan . --mode standard # the graph decides; no model is asked
ousast-with-judge scan . --mode standard # network on, the model is asked (DEEPSEEK_API_KEY)
```
Details, including a pinned host install of Joern, are in [ops/README.md](ops/README.md).
`docker compose build` and `TARGET=/path/to/repo docker compose run --rm ousast scan /target` do
the same without the wrapper.
Every run writes auditable artifacts under `<target>/.openultrasast/runs/<scan-id>/`:
`manifest.json`, `findings.json`, `verification.json`, `score.json`, `report.md`,
`report.sarif` and `trace/events.jsonl`, plus the stage artifacts of the mode.
`deep` adds REGRESS. REGRESS runs each reproduction snippet with `docker run` from wherever the
scan runs. The engine container has no Docker socket. So run `deep` in one of two ways:
- from a host install with Joern on the PATH (`pyinfra @local ops/joern.py`, pinned and checksum-
verified) and a working `docker`;
- or accept the recorded `sandbox_unavailable` degradation.
Details, including the host install, are in [ops/README.md](ops/README.md).
Keys and store settings live in `.env` in the working directory. The file is gitignored;
`.env.example` lists the names.
- `DEEPSEEK_API_KEY` serves every LLM call.
- `OPENROUTER_API_KEY` is used for embeddings only (`ousast learn memory embed`).
- `.env` never overrides a variable already exported in your shell. If a key change seems to have
no effect, `unset` the stale export.
- The plane's memory store defaults to a local file store under `~/ousast-results/plane/memory`.
- An S3-compatible bucket is optional: `uv sync --extra s3`, then `OUSAST_MEMORY=s3://<bucket>`
with `S3_ENDPOINT` and the AWS credentials in `.env`. Nothing in a scan needs it.
### Tier 3: the plane on ax (maintainers)
Tier 1 and 2 need none of this. Today's profile is a single-node kind cluster on the maintainer's
host.
- `ops/ax/up.sh` needs `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` on the PATH. It refuses to
start when one is missing.
- It builds Agent Substrate and ax from their checkouts under `~/.cache/ousast/ax-src/`.
- Measured idle footprint with two workers: the kind node takes 1.7 GiB of memory (27% of the
7.7 GiB host) and 4.9 GB of disk for images. The build uses several GB more for Go caches.
- The worker pool is sized for that 7 GB host.
- The verify and roles tasks need `DEEPSEEK_API_KEY`.
- Memory lives in the local file store or the S3 store. RustFS is the tested server.
Bring-up, doctor and the smoke Run: [ops/ax/README.md](ops/ax/README.md).
### Tier 4: Kubernetes (k3s), planned
Nothing has been deployed to any cluster other than that kind cluster. The target is two profiles
of one code base: `kind` locally and `k3s` in production. The same `ousast plane run` would
submit a Run in a remote execution mode. It needs:
- a kubeconfig context or the cluster's ingress (and an `imagePullSecret` on the worker
ServiceAccount for a private package);
- runner and engine images published to GHCR and pinned by digest;
- the S3 store as the artifact path, with no receiver on a laptop.
The plan, and the code changes it needs with file paths, are in
[docs/deployment.md](docs/deployment.md#production-topology-ax-on-a-kubernetes-cluster).
## Scan modes
| Mode | What runs | Needs |
| --- | --- | --- |
| `quick` | Language-scoped pattern rules, entry-point reachability hints, ranking, verification, scoring. Deterministic and reproducible. | nothing |
| `standard` | `quick` plus the MAP stage: complexity map, semantic overlay, authorization obligations, and the model layer, which builds one Joern code property graph per repository and decides taint, guard-dominance and configuration questions on it. An LLM is asked only the residual question the graph cannot settle (a `suspicion`); `[models] hunter` adds the tool hunter. | Joern for the model layer (else a `cpg_unavailable` degradation); a provider key for the LLM parts (else they are skipped and recorded) |
| `deep` | The MAP stage of `standard` plus REGRESS: promoted candidates are loaded by a small snippet inside a Docker sandbox (no network, read-only source, non-root, memory/pid/time limits) and get a `triggerable` / `not_triggerable` / ... verdict. | a working `docker` (else a `sandbox_unavailable` degradation) |
There are three modes. Each adds to the one before.
`--fail-on never|findings|verified|worth-fixing` sets the exit code: `findings` fails on any
reported finding, `verified` on any evidence-verified one, `worth-fixing` on a deep-mode
verdict that is worth fixing. `--config openultrasast.toml` loads settings.
- `quick` runs the pattern rules, hints, ranking, verification and score. It needs nothing.
- `standard` adds the MAP stage: complexity map, semantic overlay, authorization obligations and
the Joern model layer. The model layer builds one code property graph per repository. It decides
taint, guard-dominance and configuration questions on that graph. A model is asked only the
residual question. Without a key, that question is skipped and only the graph's entailed
findings are reported.
- `deep` adds REGRESS. Promoted candidates are loaded by a small snippet in a Docker sandbox. Each
gets a `triggerable` or `not_triggerable` verdict.
## Detection coverage (quick rules)
A missing engine is a recorded degradation (`cpg_unavailable`, `sandbox_unavailable`), never a
clean result.
Rules run only against files of their language. The bundled ruleset lives in
`src/openultrasast/ruleset/<language>/rules.toml`:
`--fail-on never|findings|verified|worth-fixing` sets the exit code. `--config openultrasast.toml`
loads settings. Every run writes these files under `<target>/.openultrasast/runs/<scan-id>/`:
`manifest.json`, `findings.json`, `verification.json`, `score.json`, `report.md`, `report.sarif`
and `trace/events.jsonl`.
| Language | Classes (CWE) |
| --- | --- |
| C / C++ | stack buffer overflow (121), format string (134), command injection (78) |
| Python | command injection (78), code injection (95), SSTI (94), deserialization (502), SQLi (89), path traversal (22), SSRF (918), weak hash (327), insecure default (489) |
| JavaScript / TS | command injection (78), code injection (95), XSS (79), SQLi (89), path traversal (22), SSRF (918), weak hash (327), deserialization (502) |
| Java + Groovy templates | command injection (78), SQLi (89), weak hash (327), deserialization (502), unescaped template XSS (79) |
| PHP | SQLi (89), command injection (78), file inclusion (98), path (22), open redirect (601), SSRF (918); code eval (95), unserialize (502) and echo XSS (79) ship as `shadow` rules |
`ousast pre-push` (experimental) analyses the commits a push would publish against their base.
First, the quick rules run on the changed lines. Then the engine compares head with base, and only
new or worsened defects become candidates. It is advisory by default, and it prints one line per
check it could not run. Its capability registry is empty, so it emits no normal alert today. An
engine finding the push introduced is shown as advisory. Install, settings, skip reasons and a CI
recipe: [docs/scanning.md](docs/scanning.md#first-run-install-the-hook-on-any-repository).
The detection gate (`uv run python -m openultrasast.gate`, also `tests/test_detection_benchmarks.py`)
enforces the project goal of **at least 90% recall and under 10% false positives** per language
on the bundled corpora in `benchmarks/manifests/`. Its output on this tree:
Flows and the per-language rule table: [docs/scanning.md](docs/scanning.md).
| Language | Recall | False-positive rate |
| --- | --- | --- |
| C / C++ | 93.3% (14/15) | 0.0% |
| Python | 93.3% (14/15) | 0.0% |
| JavaScript | 92.3% (12/13) | 0.0% |
| Java | 100% (4/4) | 0.0% |
| **Overall** | **93.6% (44/47)** | **0.0%** |
## How detection works
These corpora are cheat-sheet fixtures modelled on deliberately vulnerable applications. They
show that a rule change did not break a known detection; they are not a recall estimate on
real code. Real-code results are in [docs/evaluation.md](docs/evaluation.md).
The tool uses three representations of code, from cheap to expensive:
**PHP is not in the gate, and its numbers are in-sample only.** The PHP rules were written and
tightened while looking at the development corpus they were measured on
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`): 13 of 14 expected
findings on the `php-vulnerable` fixture with no false positive, 2 false positives on
`php-benign`, and per-rule precision lower bounds from 0.03 to 1.0 on the real pins. Treat
them as development numbers, not as a holdout result.
1. Text: regex quick rules, one statement at a time.
2. Tree: the tree-sitter CST (concrete syntax tree) and an IR (intermediate representation) per
function. The overlay and the obligations use it.
3. Graph: the Joern CPG. It is the only one that follows a value across statements, calls and
files.
**Framework knowledge is optional.** Rule and taint-fact entries that know a framework or
library (WordPress, Flask, Django, Express, Spring, ...) carry a `framework` or `library` tag
naming a row of `src/openultrasast/ruleset/frameworks.toml`. The loaders take `priors=`
(`"all"`, the default and today's behaviour; `"off"`, language-level knowledge only; or a set
of ids), so their contribution can be measured and switched off.
Taint runs as fact tables, queries and verdicts on the graph. Absence bugs are operations that
should have been guarded. They are found as obligations, dominance and configuration census. A
model answers only the residual question.
## Pre-push checks (experimental)
- [docs/detection-techniques.md](docs/detection-techniques.md) explains each technique, with the
code it comes from and what it misses.
- [docs/architecture.md](docs/architecture.md) gives the stage order and the evidence ladder (a
state machine, not a label a model may assign). It also covers the central CWE policy, the
project score and the self-improving loops (`ousast improve`, with `--memory` for proposals from
plane rows).
`ousast pre-push` analyses the commits a `git push` would publish, or an explicit
`--base`/`--head` pair, and compares the head against its base so that only new or worsened
defects are candidates. `--mode advisory` (the default) never blocks; `--mode blocking` and
`--incomplete-coverage block` opt into enforcement. The artifact must be written outside the
analysed repository.
Quick rules cover C/C++, Python, JavaScript/TypeScript, Java with Groovy templates, and PHP. Each
rule runs only against files of its language.
```bash
uv run ousast pre-push /path/to/repo --base BASE --head HEAD \
--artifact /outside/the/repo/result.json
```
The detection gate (`uv run python -m openultrasast.gate`) enforces at least 90% recall and under
10% false positives per language on the bundled cheat-sheet corpora. On this tree it reports
93.62% recall (44/47) and 0.0% false positives. Those corpora show that a rule change did not
break a known detection. They are not a recall estimate on real code, and PHP is not in the gate.
The default capability registry is empty, so no normal alert is emitted today; every
candidate stays diagnostic in the artifact. Installation into a repository's hooks
(`ops/install-pre-push`), caching and the optional witness model are documented in
[ops/README.md](ops/README.md).
Framework knowledge (WordPress, Flask, Django, Express, Spring) is tagged as an optional prior in
`src/openultrasast/ruleset/frameworks.toml`. So its contribution can be measured and switched off.
## Model providers and keys
## The plane on ax
Keys live in `.env` in the working directory (gitignored, never committed):
Model-driven work over whole repositories runs as a Run manifest. A Run is a DAG of ax Tasks. Each
Task runs in its own gVisor actor (an isolated sandbox) on Agent Substrate. ax is the only agentic
executor; there is no local subprocess path.
```bash
DEEPSEEK_API_KEY=... # every LLM call: model layer, tool hunter, plane tasks
OPENROUTER_API_KEY=... # only OpenAI embeddings (openai/text-embedding-3-small)
```
- Each Task binds at most one Model and its own `usd`/`calls` budget.
- Egress is deny-by-default per Task, by hostname only.
- The provider key travels only in the Task's start request, never in a manifest or an image.
The project uses DeepSeek (`deepseek-flash`) for LLM calls and OpenRouter only for embeddings.
Set `DEEPSEEK_API_KEY`: without it, chat calls fall back to OpenRouter when that key is set.
**`.env` never overrides a variable already exported in your shell.** A stale
`export DEEPSEEK_API_KEY=...` in your profile wins over the `.env` file silently; `unset` it
when a key change in `.env` seems to have no effect.
## Benchmarks and pair evaluation
```bash
# Score the scan against a ground-truth manifest
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
# Vulnerable-vs-fixed pairs: fire on the vulnerable side, stay silent on the fix
uv run ousast pairs
uv run ousast pairs --slice sast # OWASP Benchmark + Juliet
uv run ousast pairs --slice vibe-py --pointers # also fetch non-vendored pointer pairs (network)
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
```
Each benchmark run writes `.openultrasast/benchmarks/<id>/` (`benchmark_result.json`,
`calibration_records.json` with every miss, `external_baseline_deltas.json`). The pair
catalogs live in `benchmarks/pairs/`.
## Self-improvement
`ousast improve` runs a bounded loop over the governed ruleset: benchmark, propose edits to
rule status and score constants only, validate, re-benchmark, and accept a round only if
recall stays at least 90%, false positives under 10%, the project score does not regress and
no matched finding is lost. Accepted rounds land in
`<target>/.openultrasast/calibration/rule_policy.json`, which the next scan of that target loads.
```bash
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
```
With `--memory`, proposals also come from the rows plane runs stored (see below), through the
same validator and gate; rows from holdout pairs, from the gated manifest's own cases and from
any `--qualify-population` are dropped first. Details:
[docs/architecture.md](docs/architecture.md#self-improvement).
## The agentic plane (maintainers)
Model work over whole repositories runs as a Run manifest on [google/ax](ops/ax/README.md)
over Agent Substrate (a kind cluster on the maintainer's host), one isolated actor per task.
Each task binds its own Model and its own `usd`/`calls` budget, egress is deny-by-default per
task, and the provider key travels only in the task's start request.
Measured on the first increment: the plane agreed on 16 of 20 declared sites at $0.0205 per
candidate.
```bash
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
uv run ousast plane run plane/runs/validation-46.yaml # a rerun skips tasks already done
uv run ousast plane status validation-46 # per-task status and token attribution
uv run ousast plane run plane/runs/validation-46.yaml # a rerun skips tasks already done; --workers bounds tasks in flight
uv run ousast plane status validation-46 # per-task status and the token attribution table
uv run ousast plane remember validation-46 # ingest the run's rows into the memory store
```
`ousast plane workspaces`, `alerts-engine` and `harvest` generate Runs and inputs. Bring-up,
the task catalogue, the memory store (`OUSAST_MEMORY`) and the checklist for moving the plane
to another Kubernetes cluster are in [ops/ax/README.md](ops/ax/README.md).
`workspaces`, `harvest`, `alerts-engine` and `memory-normalise` generate Runs, their inputs and
the store's rewrites. The flow and the task catalogue: [docs/plane.md](docs/plane.md). What runs
where today, and what a separate cluster would take: [docs/deployment.md](docs/deployment.md).
> HarnessX, the earlier optional agentic extra, was retired 2026-09-30, and a leftover
> `[harnessx]` section (retired 2026-09-30) is ignored with one warning. `[models] verifier`,
> `[fusion] panel_model` and `[fusion] decider_model` fail with a message naming the plane
> replacement. The LLM judge and the LLM fusion panels were retired with it; fusion is
> deterministic. See [RELEASE_NOTES.md](RELEASE_NOTES.md).
HarnessX, the earlier agentic extra, was retired 2026-09-30. A `[harnessx]` section is ignored
with one warning. `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail and
name the plane replacement. Config keys that nothing ever read fail the same way. Fusion is
deterministic.
## Decision engine (in development, not adopted)
## Memory
A learned decision engine is being built to replace hand-tuned detection edits: each
instrument (quick rules, the Joern engine, model sink classification, verify passes, repository
facts) produces signals, never verdicts, and a compiled AI classifier with local memory turns
them into a calibrated probability per candidate with BLOCK and ADVISORY operating points.
No scan, pre-push or report uses it yet. Status and the first measured slice:
[docs/decision-engine.md](docs/decision-engine.md).
One store holds what plane runs and the decision engine learned. It is keyed by repository and pin.
It holds two kinds of data:
## Agent integrations
- rows: facts, verdicts, unit costs, alerts, coverage and feature records;
- content-addressed blobs: excerpts, embeddings, cached responses and compiled programs.
An agent that can run shell commands needs nothing but the CLI. For
[OpenCode](https://opencode.ai) there are optional project skills in `.opencode/skills/`:
`openultrasast-scan`, `openultrasast-triage` and `openultrasast-fix-audit`.
`OUSAST_MEMORY` selects the backend. It is either a local file store or `s3://<bucket>` on any
S3-compatible server. The S3 server must support versioning, lifecycle rules, object tags and S3
Select. RustFS is the tested server.
`ousast mcp` runs a narrow MCP server over stdio (newline-delimited JSON-RPC) with ten tools:
`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`,
`benchmark`, `explain`, `propose_patch` (degrades visibly) and `export_report`. No tool runs a
shell, Docker or a free-form command.
An admin sets the bucket up once. The store verifies it at every start. If anything is missing, it
refuses to run and names each missing piece. It does not fall back.
```jsonc
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
- Layout and the read and write paths: [docs/memory.md](docs/memory.md).
- The bucket policy and admin steps: [docs/rustfs.md](docs/rustfs.md).
- How stored rows are meant to turn into better decisions:
[docs/memory-and-detection.md](docs/memory-and-detection.md).
## Decision engine and evaluation
A learned decision engine is being built. It is meant to replace hand-tuned detection edits. It
follows three rules:
- Every instrument produces signals, never verdicts.
- Labels come from ground truth only.
- A compiled AI classifier with local memory turns retrieved similar cases and the candidate's
signals into a calibrated probability. It has BLOCK and ADVISORY operating points.
It is **in development, not adopted**. No scan, pre-push check or report uses it.
It was measured on six families, on repositories the compiled program was not shown:
- The pooled AUC (area under the ROC curve) of its score runs from 0.77 (injection) to 0.96
(access control).
- The stricter within-pair AUC is 0.72 to 0.93 across the same families.
- BLOCK is offered for no family
(`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`).
Maintainer surface: `ousast learn labels|memory|compile|evaluate|curve|experiment|audit-leaks`.
Design, slices and the A/B protocol: [docs/decision-engine.md](docs/decision-engine.md).
[docs/evaluation.md](docs/evaluation.md) explains how anything is measured on code the tool was not
tuned on. It covers the pair corpora, the independent populations and the M4 gates. It also covers
the fold discipline that keeps learned decisions out of their own training data. Population v3
(PHP, 15 cases) is frozen as the one-time final check. Nothing in the tree may read it.
```bash
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
uv run ousast pairs --slice sast # fire on the vulnerable side, stay silent on the fix
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
```
> The `kiro-*` skills, `AGENTS.md` and `.kiro/` are the maintainer's development tooling and
> can be ignored by users.
## Token economics
## Further reading
Model tokens are the one running cost. At runtime the plane routes them:
Documentation site with flow diagrams (`docs/`, `mkdocs.yml`): `uv sync --extra docs && uv run mkdocs serve`.
- one Model and one budget per Task;
- a metered client that refuses the next call at the ceiling;
- repository facts computed without a model and reused;
- blobs replayed at $0, with no new model call;
- triage before the hunt;
- a tie-break only on disputed candidates.
- [docs/architecture.md](docs/architecture.md): the scan pipeline, evidence ladder, CWE policy
and project score, and the self-improving loops.
- [docs/evaluation.md](docs/evaluation.md): independent populations, the qualification gates,
pair corpora and the plane increment results.
- [docs/decision-engine.md](docs/decision-engine.md): the learned decision engine.
- [docs/threat-model.md](docs/threat-model.md): trust boundaries, sandbox and hardening.
- [docs/token-ergonomics.md](docs/token-ergonomics.md): model spend at runtime (budgets, reuse,
measured costs) and in coding sessions.
- [docs/examples.md](docs/examples.md): end-to-end walkthroughs.
In development, 85.5% of the maintainers' coding-session tokens were tool input and output rather
than reasoning. That is down from 90%. The remedies in `benchmarks/dev/` target that share.
Mechanisms, the measured costs and the before-and-after table:
[docs/token-ergonomics.md](docs/token-ergonomics.md).
## Development
## Examples, threat model
- End-to-end walkthroughs, from a quick scan to an MCP client: [docs/examples.md](docs/examples.md).
- What the tool trusts and what it does not: [docs/threat-model.md](docs/threat-model.md). This
covers scanned code as untrusted input, secret redaction, per-task egress, budgets and the
`deep` sandbox.
## Contributing and maintainer tooling
```bash
uv run pytest # tests, incl. the 90/10 detection gate
@@ -245,6 +332,34 @@ uv run python -m openultrasast.pair_gate # local vuln-vs-fix pairs
uv run python dagger/ci.py # containerized CI pipeline
```
Maintainer commands not covered above: `ousast repos` (pinned known-vulnerable checkouts),
`ousast model candidates` (what the candidate enumerator can reach) and `ousast learn`
(the decision engine's data; see [docs/decision-engine.md](docs/decision-engine.md)).
An agent that can run shell commands needs nothing but the CLI. For [OpenCode](https://opencode.ai)
there are two project skills in `.agents/skills/`:
- `openultrasast-scan`: modes and their prerequisites, `--fail-on`, the evidence ladder, recorded
degradations and run artifacts.
- `openultrasast-triage`: false-positive reasons, what an adjudication may feed, and fixing and
re-scanning.
`ousast mcp` runs a narrow MCP (Model Context Protocol) server over stdio. It has ten tools:
`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`,
`explain`, `propose_patch` and `export_report`. No tool runs a shell, Docker or a free-form
command.
```jsonc
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
```
The `kiro-*` skills, `AGENTS.md` and the Kiro-style specifications under `.kiro/` are the
maintainer's development tooling. Users can ignore them. Other maintainer commands:
- `ousast repos`: pinned known-vulnerable checkouts.
- `ousast model candidates`: what the candidate enumerator can reach.
## Licence
OpenUltraSAST is licensed under the [Apache License 2.0](LICENSE). Third-party material keeps its
own terms:
- Joern (Apache-2.0) is downloaded at image build time.
- The tree-sitter wheels of the `semantic` extra are MIT.
- The published benchmark cases were licence-checked one by one. v1's OpenCVE case is BUSL-1.1, see [RELEASE_NOTES.md](RELEASE_NOTES.md).
+38
View File
@@ -1,3 +1,41 @@
# v2.0.1 (2026-10-02)
v2.0.0 was tagged but never published: its release job stopped at the format check (see below). v2.0.1 is
the first published 2.x release and carries everything listed under v2.0.0 plus the following.
- **Licence: Apache-2.0.** `LICENSE` added; `pyproject.toml` declares `license = "Apache-2.0"`; both images
carry the `org.opencontainers.image.licenses` label.
- **Container images on GHCR.** Each version tag pushes `ghcr.io/norandom/openultrasast` (the scan image:
Joern and php-cli) and `ghcr.io/norandom/openultrasast-runner` (the plane's task runner), tagged with the
version and, for full releases, `latest`. The digests are in the release job's summary; pin by digest
(Agent Substrate rejects tags).
- **CI green again.** CI and the release job check formatting and lint over the whole repository; eight
benchmark scripts had drifted. They were reformatted (two `noqa` comments, one renamed loop variable; no
script computes anything differently), and the local gate now runs the same whole-tree commands.
- **Front-door README.** Install in four tiers (quick scan without Docker; standard and deep with the scan
image; the plane on kind; Kubernetes on k3s, planned), every claim checked against the CLI, and links to
every documentation page.
## Also in this release (previously listed as Unreleased)
- **Documentation audited against v2.0.0** (no behaviour change). New `docs/deployment.md`: where the plane runs
today, this host's setup, and step-by-step guidance for a separate Kubernetes cluster with what is implemented
and what is planned; no cluster deployment has been made, and the kube context (`kind-$KIND_CLUSTER_NAME`) and
`doctor`'s registry check (`localhost:5001`) are still hard-coded. The decision-engine pages state slice 2
(six families, `benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`: pooled AUC 0.77-0.96,
within-pair 0.72-0.93, BLOCK offered nowhere, canary below 0.9 for three families) and that the engine is not
adopted. The model layer's residual question is documented at `model/pipeline.py` (`model/judge.py` was
deleted); a missing provider key is described as it behaves (no question asked, entailed findings reported),
not as a recorded degradation. The `.env` reference lists the memory-store variables; `ousast plane memory-normalise` is listed.
- **Removed: the unused vector index, `ousast index` and `[embeddings]`** (legacy cleanup, 2026-10-02). `index.py`
keeps only the text chunker the skill router uses; the `VectorIndex` half had no caller, and nothing read the
`chunks.json` the subcommand wrote. `[embeddings] model`/`store` fail to load with the reason, like the other
retired keys; `OPENROUTER_EMBEDDING_MODEL` is no longer read (`ousast learn memory embed --model` names the model).
- **Retired three config keys nothing read** (2026-10-02): `[sandbox] network`, `[sandbox] workspace_readonly` and
`[complexity] top_k` now fail to load naming why. The sandbox always ran with `--network none` and a read-only
workspace regardless of the first two; no stage read `top_k`. `memory_mb`, `timeout_seconds`, `pids_limit` and
`max_hunter_hotspots` are read and stay.
# v2.0.0 (2026-10-02)
A new major release: the agentic plane runs on google/ax, HarnessX is gone, detection decisions move
+1 -1
View File
@@ -16,7 +16,7 @@ maps each old path to the new one. The records themselves are not rewritten.
| `model_sinks.py` | Stage-1 source-only model classification of sinks. Moved into the package as `plane/tasks/roles.py`. | `independent/selection-v2-model-sinks.json`, `measurements/2026-10-01-decision-engine-harvest/record.json`, both `2026-10-01-decision-engine-leak-audit` records | `05d5699` (named in `selection-v2-model-sinks.json`) |
| `sink_candidates.py` | Diagnostic: vocabulary-declared sink candidates per case. Its declaration patterns live on in `plane/tasks/repo_facts.py`. | `independent/results-v2-model-pipeline.json` ("vocabulary sink-first: 2/17") | no commit recorded |
| `hunt.py` | The v2 tool-hunter run over the population (protocol `hunter-protocol-v2.md`, including its `FAMILY_WORDS` table). Replaced by the plane's `verify` / `agree` tasks. | `independent/hunter-protocol-v2.md`, `independent/results-v2-hunter.json` (a label source in `learn/sources.toml`) | `6bfb05f`, the commit that introduced the protocol, which pins the runner |
| `harnessx_removal_equality.py` | One-off equality proof for `harnessx-removal` (record and compare deterministic outputs). The spec is closed. `tests/test_harnessx_removal_equality.py` still checks that the comparison detects differences. | `measurements/2026-09-30-harnessx-removal-baseline/`, `measurements/2026-09-30-harnessx-removal-equality/` | baseline `ca5caf7`, candidate `b9afb66` |
| `harnessx_removal_equality.py` | One-off equality proof for `harnessx-removal` (record and compare deterministic outputs). The spec is closed; its unit test was dropped on 2026-10-02 so the archived copy is reference only like the rest. | `measurements/2026-09-30-harnessx-removal-baseline/`, `measurements/2026-09-30-harnessx-removal-equality/` | baseline `ca5caf7`, candidate `b9afb66` |
The six `independent` scripts still read the population, `evaluate.py` and the results from
`benchmarks/independent/`. Their `HERE` points there, so outputs land where they always did.
+38 -3
View File
@@ -40,8 +40,34 @@ from openultrasast.preprocess import build_file_target, enumerate_source_files
# The model names a family in its own words; the protocol scores three. Fixed before any v2 pin is hunted.
FAMILY_WORDS = {
"untrusted_destination": ("ssrf", "server-side request", "open redirect", "redirect", "url fetch", "request forgery"),
"config_secrets": ("cors", "debug", "secret", "hardcoded", "hard-coded", "credential", "misconfig", "configuration", "tls", "certificate"),
"injection": ("inject", "sql", "sqli", "nosql", "command", "rce", "code execution", "eval", "template", "ssti", "xss", "deserializ", "ldap", "xpath"),
"config_secrets": (
"cors",
"debug",
"secret",
"hardcoded",
"hard-coded",
"credential",
"misconfig",
"configuration",
"tls",
"certificate",
),
"injection": (
"inject",
"sql",
"sqli",
"nosql",
"command",
"rce",
"code execution",
"eval",
"template",
"ssti",
"xss",
"deserializ",
"ldap",
"xpath",
),
}
@@ -182,7 +208,16 @@ def main() -> int:
result["root"] = "/case"
out.write_text(json.dumps(result, indent=2) + "\n")
print(
json.dumps({"case": case["id"], "pin": label, "findings": len(result["findings"]), "hunts": f"{result['completed']}/{result['questions']}", "usd": result["cost_usd"], "total_usd": round(sum(spent), 4)}),
json.dumps(
{
"case": case["id"],
"pin": label,
"findings": len(result["findings"]),
"hunts": f"{result['completed']}/{result['questions']}",
"usd": result["cost_usd"],
"total_usd": round(sum(spent), 4),
}
),
flush=True,
)
return 0
+13 -5
View File
@@ -37,7 +37,9 @@ from openultrasast.config import load_config # noqa: E402
from openultrasast.model.endpoint import DEFAULT_DETECTOR_MODEL # noqa: E402
from openultrasast.preprocess import detect_language, enumerate_source_files # noqa: E402
NOT_PRODUCT = re.compile(r"(^|/)(tests?|__tests__|spec|specs|docs?|examples?|fixtures?|e2e|cypress|storybook)/|\.(test|spec|stories)\.[jt]sx?$|_test\.py$|(^|/)test_[^/]*\.py$|\.min\.js$|(^|/)dist/|(^|/)build/")
NOT_PRODUCT = re.compile(
r"(^|/)(tests?|__tests__|spec|specs|docs?|examples?|fixtures?|e2e|cypress|storybook)/|\.(test|spec|stories)\.[jt]sx?$|_test\.py$|(^|/)test_[^/]*\.py$|\.min\.js$|(^|/)dist/|(^|/)build/"
)
OPERATIONS = {
"injection": (
"executes a database query (SQL, NoSQL, JCR/graph query), an operating-system command or shell, code or "
@@ -69,7 +71,9 @@ def product_files(root: Path, languages: set[str]) -> list[Path]:
def chunks(text: str, size: int) -> list[str]:
lines = text.splitlines()
return ["\n".join(f"{n + 1:5d}| {line}" for n, line in enumerate(lines[i : i + size], start=i)) for i in range(0, max(len(lines), 1), size)]
return [
"\n".join(f"{n + 1:5d}| {line}" for n, line in enumerate(lines[i : i + size], start=i)) for i in range(0, max(len(lines), 1), size)
]
class Meter:
@@ -85,7 +89,9 @@ class Meter:
return sum(float(c.cost_usd()) for c in self.clients)
def classify(root: Path, path: Path, family: str, args: argparse.Namespace, meter: Meter, local: threading.local) -> tuple[list[tuple[str, str, int]], int, int]:
def classify(
root: Path, path: Path, family: str, args: argparse.Namespace, meter: Meter, local: threading.local
) -> tuple[list[tuple[str, str, int]], int, int]:
from openultrasast import tool_hunter
if not hasattr(local, "client"):
@@ -99,7 +105,9 @@ def classify(root: Path, path: Path, family: str, args: argparse.Namespace, mete
asked += 1
prompt = PROMPT.format(operation=OPERATIONS[family], path=relative, code=code)
try:
response = local.client.complete(model=args.model, messages=[{"role": "user", "content": prompt}], tools=[], timeout_seconds=90, json_object=True)
response = local.client.complete(
model=args.model, messages=[{"role": "user", "content": prompt}], tools=[], timeout_seconds=90, json_object=True
)
items = json.loads(response.content or "{}").get("functions", [])
except Exception: # noqa: BLE001 -- counted as unclassified, never as "no sinks"
failed += 1
@@ -138,7 +146,7 @@ def main() -> int:
files = product_files(checkout, {case["language"], *case.get("also", [])})
found, failed, asked = [], 0, 0
with concurrent.futures.ThreadPoolExecutor(max_workers=args.workers) as pool:
for f, fail, ask in pool.map(lambda p: classify(checkout, p, case["family"], args, meter, local), files):
for f, fail, ask in pool.map(lambda p, c=checkout, fam=case["family"]: classify(c, p, fam, args, meter, local), files):
found += f
failed += fail
asked += ask
+2 -1
View File
@@ -66,7 +66,8 @@ def main() -> int:
"agreed_findings": len(vulnerable["findings"]), "disputed_findings": len(vulnerable.get("disputed", [])),
"triaged_out": vulnerable.get("triaged_out"), "candidates": vulnerable.get("questions"),
"fixed_side_alerts": fixed_alerts, "fixed_agreed_total": len(fixed["findings"]) if fixed else None,
"removed_by_fix": fixed.get("removed_by_fix") if fixed else None, "usd": vulnerable.get("usd"), "usage": vulnerable.get("usage"),
"removed_by_fix": fixed.get("removed_by_fix") if fixed else None,
"usd": vulnerable.get("usd"), "usage": vulnerable.get("usage"),
}
) # fmt: skip
pool += [{"case": case["id"], **f} for f in vulnerable["findings"]]
+3 -1
View File
@@ -36,7 +36,9 @@ from openultrasast.preprocess import detect_language, enumerate_source_files #
DECLARATION = {
"python": re.compile(r"^\s*(?:async\s+)?def\s+(\w+)"),
"php": re.compile(r"^\s*(?:(?:public|private|protected|static|final|abstract)\s+)*function\s+&?(\w+)"),
"javascript": re.compile(r"^\s*(?:export\s+)?(?:async\s+)?function\s*\*?\s*(\w+)|^\s*(?:(?:public|private|protected|static|async)\s+)*(\w+)\s*\([^;]*\)\s*(?::[^{=]+)?\{\s*$|^\s*(?:export\s+)?(?:const|let|var)\s+(\w+)\s*=\s*(?:async\s*)?(?:function|\()"),
"javascript": re.compile(
r"^\s*(?:export\s+)?(?:async\s+)?function\s*\*?\s*(\w+)|^\s*(?:(?:public|private|protected|static|async)\s+)*(\w+)\s*\([^;]*\)\s*(?::[^{=]+)?\{\s*$|^\s*(?:export\s+)?(?:const|let|var)\s+(\w+)\s*=\s*(?:async\s*)?(?:function|\()"
),
}
DECLARATION["typescript"] = DECLARATION["javascript"]
KEYWORDS = {"if", "for", "while", "switch", "catch", "return", "function", "else"}
+87 -14
View File
@@ -101,13 +101,22 @@ def triage(root: Path, path: str, candidates: list[list], family: str, model: st
except OSError:
return {}
names = sorted({fn for _, fn, _ in candidates})
prompt = TRIAGE.format(operation=OPERATIONS[family], functions=json.dumps(names), path=path, code=_relevant_code(text, [ln for _, _, ln in candidates], chunk_lines))
prompt = TRIAGE.format(
operation=OPERATIONS[family],
functions=json.dumps(names),
path=path,
code=_relevant_code(text, [ln for _, _, ln in candidates], chunk_lines),
)
response = client.complete(model=model, messages=[{"role": "user", "content": prompt}], tools=[], timeout_seconds=90, json_object=True)
try:
items = json.loads(response.content or "{}").get("functions", [])
except (json.JSONDecodeError, AttributeError):
return {}
return {str(i.get("function")): str(i.get("data_origin", "")).strip().lower() for i in items if isinstance(i, dict) and i.get("function") in names}
return {
str(i.get("function")): str(i.get("data_origin", "")).strip().lower()
for i in items
if isinstance(i, dict) and i.get("function") in names
}
def hunt(root: Path, path: str, group: list[list], family: str, model: str, steps: int) -> tuple[list[dict], dict]:
@@ -120,8 +129,30 @@ def hunt(root: Path, path: str, group: list[list], family: str, model: str, step
f"Operations to judge, all in `{path}`, each of which {OPERATIONS[family]}:\n{listing}\n"
"The file is above. For each, trace where the operation's data comes from and whether it is guarded."
)
spots = [Hotspot(path=path, function_name=fn, score=1.0, band="candidate", signals={}, rationale="model-classified sink", test_hint=None, inventory_finding_ids=()) for _, fn, _ in group]
found = tool_hunter.run_tool_hunter(root, spots, client=recording, model=model, max_steps=steps, system_prompt=SYSTEM, user_prompt=prompt, context_files=[path], tags=("sink-verifier",))
spots = [
Hotspot(
path=path,
function_name=fn,
score=1.0,
band="candidate",
signals={},
rationale="model-classified sink",
test_hint=None,
inventory_finding_ids=(),
)
for _, fn, _ in group
]
found = tool_hunter.run_tool_hunter(
root,
spots,
client=recording,
model=model,
max_steps=steps,
system_prompt=SYSTEM,
user_prompt=prompt,
context_files=[path],
tags=("sink-verifier",),
)
names = {fn for _, fn, _ in group}
rows = []
for f in found:
@@ -131,7 +162,16 @@ def hunt(root: Path, path: str, group: list[list], family: str, model: str, step
# The verdict is about the candidate operation, so the site is the candidate's file and function; the
# model's own location is kept beside it (it often names the file where the input enters instead).
line = f.line if f.path == path else next(c[2] for c in group if c[1] == fn)
rows.append({"candidate": f"{path}::{fn}", "site": f"{path}:{line or 0}:{fn}", "reported_at": f"{f.path}:{f.line or 0}", "family": family, "title": f.title, "witness": f.rationale[:600]})
rows.append(
{
"candidate": f"{path}::{fn}",
"site": f"{path}:{line or 0}:{fn}",
"reported_at": f"{f.path}:{f.line or 0}",
"family": family,
"title": f.title,
"witness": f.rationale[:600],
}
)
return rows, recording.usage()
@@ -143,7 +183,12 @@ def verify_file(root: str, path: str, candidates: list[list], family: str, args_
try:
if not args_["no_triage"]:
record["triage"] = triage(Path(root), path, candidates, family, args_["model"], args_["chunk_lines"])
kept = [c for c in candidates if record["triage"].get(c[1], "unclassified") in VERIFY | {"unclassified"} or record["triage"].get(c[1]) not in {"internal", "constant"}]
kept = [
c
for c in candidates
if record["triage"].get(c[1], "unclassified") in VERIFY | {"unclassified"}
or record["triage"].get(c[1]) not in {"internal", "constant"}
]
record["kept"] = [c[1] for c in kept]
usage: dict[str, int] = {}
for _pass in range(args_["passes"]):
@@ -169,14 +214,16 @@ def agreement(record: dict) -> tuple[list[dict], list[dict]]:
some = set.union(*(set(b) for b in by_pass))
first = by_pass[0]
pick = lambda c: next(b[c] for b in by_pass if c in b) # noqa: E731
return [dict(first[c], passes=len(by_pass)) for c in sorted(every)], [dict(pick(c), passes=sum(c in b for b in by_pass)) for c in sorted(some - every)]
return [dict(first[c], passes=len(by_pass)) for c in sorted(every)], [
dict(pick(c), passes=sum(c in b for b in by_pass)) for c in sorted(some - every)
]
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--population", type=Path, default=HERE / "population-v2.toml")
parser.add_argument("--only", default="")
parser.add_argument("--subset", type=Path, help='JSON {case_id: [[path, function], ...]}: verify only these candidates')
parser.add_argument("--subset", type=Path, help="JSON {case_id: [[path, function], ...]}: verify only these candidates")
parser.add_argument("--model", default=DEFAULT_DETECTOR_MODEL)
parser.add_argument("--max-steps", type=int, default=6)
parser.add_argument("--per-hunt", type=int, default=6)
@@ -206,7 +253,11 @@ def main() -> int:
if out.is_file():
continue
log = out_dir / f"{case['id']}--{label}.jsonl"
done = {r["path"]: r for r in (json.loads(row) for row in log.read_text().splitlines()) if "unverified" not in r} if log.is_file() else {}
done = (
{r["path"]: r for r in (json.loads(row) for row in log.read_text().splitlines()) if "unverified" not in r}
if log.is_file()
else {}
)
spent += sum(float(r.get("usd", 0.0)) for r in done.values())
record = json.loads((candidates_dir / f"{case['id']}.json").read_text())
unique = list({(p, fn): [p, fn, ln] for p, fn, ln in sorted(record["candidates"])}.values())
@@ -265,19 +316,41 @@ def main() -> int:
agreed += a
disputed += d
triaged_out = sum(len(r["candidates"]) - len(r.get("kept", [])) for r in done.values())
usage = {k: sum(int(r.get("usage", {}).get(k, 0)) for r in done.values()) for k in ("prompt_tokens", "prompt_cache_hit_tokens", "completion_tokens", "calls")}
usage = {
k: sum(int(r.get("usage", {}).get(k, 0)) for r in done.values())
for k in ("prompt_tokens", "prompt_cache_hit_tokens", "completion_tokens", "calls")
}
complete = len(done) >= len(by_file) and not account_error
result = {
"root": "/case", "families": [case["family"]], "questions": len(unique), "completed": sum(len(r["candidates"]) for r in done.values()),
"root": "/case", "families": [case["family"]],
"questions": len(unique), "completed": sum(len(r["candidates"]) for r in done.values()),
"completed_regions": sorted(f"{c[0]}:{c[1]}" for r in done.values() for c in r["candidates"]),
"triaged_out": triaged_out, "passes": args.passes, "usage": usage, "usd": round(sum(float(r.get("usd", 0.0)) for r in done.values()), 4),
"seconds": round(time.monotonic() - started, 1), "findings": sorted(agreed, key=lambda f: f["site"]), "disputed": sorted(disputed, key=lambda f: f["site"]),
"triaged_out": triaged_out, "passes": args.passes, "usage": usage,
"usd": round(sum(float(r.get("usd", 0.0)) for r in done.values()), 4),
"seconds": round(time.monotonic() - started, 1),
"findings": sorted(agreed, key=lambda f: f["site"]),
"disputed": sorted(disputed, key=lambda f: f["site"]),
} # fmt: skip
if args.fixed:
result["removed_by_fix"] = removed
if complete:
out.write_text(json.dumps(result, indent=1) + "\n")
print(json.dumps({"case": case["id"], "files": f"{len(done)}/{len(by_file)}", "candidates": len(unique), "triaged_out": triaged_out, "agreed": len(agreed), "disputed": len(disputed), "usd": result["usd"], "total_usd": round(spent, 4), "complete": complete}), flush=True)
print(
json.dumps(
{
"case": case["id"],
"files": f"{len(done)}/{len(by_file)}",
"candidates": len(unique),
"triaged_out": triaged_out,
"agreed": len(agreed),
"disputed": len(disputed),
"usd": result["usd"],
"total_usd": round(spent, 4),
"complete": complete,
}
),
flush=True,
)
if account_error:
print(json.dumps({"stopped": "account", "error": account_error}), flush=True)
return 2
+53 -7
View File
@@ -104,7 +104,16 @@ def verify(root: str, candidate: list, family: str, model: str, max_steps: int,
client = _client()
before = float(client.cost_usd())
path, function, line = candidate
spot = Hotspot(path=path, function_name=function, score=1.0, band="candidate", signals={}, rationale="model-classified sink", test_hint=None, inventory_finding_ids=())
spot = Hotspot(
path=path,
function_name=function,
score=1.0,
band="candidate",
signals={},
rationale="model-classified sink",
test_hint=None,
inventory_finding_ids=(),
)
prompt = (
f"Operation to judge: in `{path}`, function `{function}`, around line {line}, code that {OPERATIONS[family]}. "
"The file is above. Trace where the operation's data comes from and whether it is guarded."
@@ -120,9 +129,19 @@ def verify(root: str, candidate: list, family: str, model: str, max_steps: int,
second = recording.usage()
usage = {k: usage.get(k, 0) + second.get(k, 0) for k in set(usage) | set(second)}
except Exception as error: # noqa: BLE001 -- recorded; an unverified candidate is not a clean one
return {"candidate": candidate, "unverified": f"{type(error).__name__}: {str(error)[:120]}", "usd": float(client.cost_usd()) - before}
return {
"candidate": candidate,
"unverified": f"{type(error).__name__}: {str(error)[:120]}",
"usd": float(client.cost_usd()) - before,
}
findings = [
{"site": f"{f.path}:{f.line or line}:{f.function_name or function}", "family": family, "title": f.title, "witness": f.rationale[:600], "candidate": f"{path}::{function}"}
{
"site": f"{f.path}:{f.line or line}:{f.function_name or function}",
"family": family,
"title": f.title,
"witness": f.rationale[:600],
"candidate": f"{path}::{function}",
}
for f in found
]
return {"candidate": candidate, "findings": findings, "usd": float(client.cost_usd()) - before, "escalated": escalated, "usage": usage}
@@ -134,7 +153,12 @@ def main() -> int:
parser.add_argument("--only", default="")
parser.add_argument("--model", default=DEFAULT_DETECTOR_MODEL)
parser.add_argument("--max-steps", type=int, default=6)
parser.add_argument("--escalate-steps", type=int, default=0, help="rerun a hunt that used every step without a verdict; 0 disables (measured: 88% escalate, +45% cost)")
parser.add_argument(
"--escalate-steps",
type=int,
default=0,
help="rerun a hunt that used every step without a verdict; 0 disables (measured: 88% escalate, +45% cost)",
)
parser.add_argument("--workers", type=int, default=4)
parser.add_argument("--budget-usd", type=float, default=40.0)
parser.add_argument("--fixed", action="store_true", help="recheck detected sites at the fixed pin")
@@ -185,7 +209,9 @@ def main() -> int:
with concurrent.futures.ProcessPoolExecutor(max_workers=args.workers) as pool, log.open("a") as sink:
futures = []
for candidate in pending:
futures.append(pool.submit(verify, str(checkout), list(candidate), case["family"], args.model, args.max_steps, args.escalate_steps))
futures.append(
pool.submit(verify, str(checkout), list(candidate), case["family"], args.model, args.max_steps, args.escalate_steps)
)
for future in concurrent.futures.as_completed(futures):
if future.cancelled(): # cancelled at the ceiling: never run, nothing to record
continue
@@ -219,9 +245,29 @@ def main() -> int:
} # fmt: skip
if complete:
out.write_text(json.dumps(result, indent=1) + "\n")
tokens = {k: sum(int(d.get("usage", {}).get(k, 0)) for d in done) for k in ("prompt_tokens", "prompt_cache_hit_tokens", "completion_tokens")}
tokens = {
k: sum(int(d.get("usage", {}).get(k, 0)) for d in done)
for k in ("prompt_tokens", "prompt_cache_hit_tokens", "completion_tokens")
}
result["usage"] = tokens | {"escalated": sum(1 for d in done if d.get("escalated"))}
print(json.dumps({"case": case["id"], "pin": label, "candidates": len(todo), "verified": len(verified), "findings": len(findings), "unverified": len(unverified), "escalated": result["usage"]["escalated"], "cache_share": round(tokens["prompt_cache_hit_tokens"] / max(tokens["prompt_tokens"], 1), 2), "usd": result["usd"], "total_usd": round(spent, 4), "complete": complete}), flush=True)
print(
json.dumps(
{
"case": case["id"],
"pin": label,
"candidates": len(todo),
"verified": len(verified),
"findings": len(findings),
"unverified": len(unverified),
"escalated": result["usage"]["escalated"],
"cache_share": round(tokens["prompt_cache_hit_tokens"] / max(tokens["prompt_tokens"], 1), 2),
"usd": result["usd"],
"total_usd": round(spent, 4),
"complete": complete,
}
),
flush=True,
)
if account_error:
print(json.dumps({"stopped": "account", "error": account_error, "spent_usd": round(spent, 4)}), flush=True)
return 2
+7 -2
View File
@@ -36,9 +36,9 @@ def main() -> int:
parser.add_argument("files", nargs="+", type=Path)
parser.add_argument("--max-usd", type=float, default=0.05)
args = parser.parse_args()
from openultrasast import tool_hunter
from openultrasast.config import load_config
from openultrasast.model.endpoint import DEFAULT_DETECTOR_MODEL
from openultrasast import tool_hunter
load_config()
client = tool_hunter.resolve_hunter_client()
@@ -51,7 +51,12 @@ def main() -> int:
if float(client.cost_usd()) >= args.max_usd:
answers.append(f"[stopped at the ${args.max_usd} ceiling before: {chunk.splitlines()[0]}]")
break
response = client.complete(model=DEFAULT_DETECTOR_MODEL, messages=[{"role": "user", "content": PROMPT.format(question=args.question, files=chunk)}], tools=[], timeout_seconds=90)
response = client.complete(
model=DEFAULT_DETECTOR_MODEL,
messages=[{"role": "user", "content": PROMPT.format(question=args.question, files=chunk)}],
tools=[],
timeout_seconds=90,
)
answers.append((response.content or "").strip())
print("\n\n".join(a for a in answers if a))
print(f"\n[bulk_read: {len(chunks)} chunk(s), ${float(client.cost_usd()):.4f}]", file=sys.stderr)
+5 -1
View File
@@ -37,7 +37,11 @@ def main() -> int:
f"{path} has {lines} lines (limit {LIMIT} for a whole-file read). Read a range with offset/limit, or ask "
f"the bulk reader for what you need: .venv/bin/python benchmarks/dev/bulk_read.py '<question>' {path}"
)
print(json.dumps({"hookSpecificOutput": {"hookEventName": "PreToolUse", "permissionDecision": "deny", "permissionDecisionReason": reason}}))
print(
json.dumps(
{"hookSpecificOutput": {"hookEventName": "PreToolUse", "permissionDecision": "deny", "permissionDecisionReason": reason}}
)
)
return 0
@@ -0,0 +1,11 @@
{
"experiment": "exp-001-known-callers",
"registered": "2026-10-02",
"store": "s3://sast-memory",
"manifest": "plane/experiments/exp-001-known-callers.yaml",
"manifest_digest": "76b54aa5ae908a4cd7797f3b4654acf03b90e5ad5d068216036e722104313929",
"pin": "fbc3220bbe1c8a4f8afa0a724fd41a51b390dd22",
"units_status": "pending",
"units_note": "model-free candidate generation on the 41 development repositories is not built; the plane's experiment-outcome task and Run annotation checks are not written",
"status": "registered, not run"
}
@@ -0,0 +1,23 @@
{
"experiment": "exp-002-retrieval-ensemble",
"registered": "2026-10-02",
"store": "s3://sast-memory",
"manifest": "plane/experiments/exp-002-retrieval-ensemble.yaml",
"manifest_digest": "f5004fb8618b1f7cc7dcf298d42b4ac1b6c0c1aa72a97299c77f937bd6175148",
"pin": "9213e5ffd978b387439e5568d32f4269dd81c021",
"units_status": "frozen",
"units_file": "plane/experiments/exp-002-retrieval-ensemble.units.jsonl",
"units_digest": "0de66bfa3ff844825e06f4f4227a14210d1b77f1cc08c0441c6d23e23fa5d91f",
"units": {
"total": 160,
"paired": 108,
"groups": 56,
"by_family": {
"injection": {"units": 119, "paired": 76, "pairs": 38, "groups": 34, "labels": {"1": 63, "0": 56}},
"access_control": {"units": 41, "paired": 32, "pairs": 16, "groups": 24, "labels": {"1": 25, "0": 16}}
},
"note": "frozen on the 937-example static memory (slice 2 evaluated 110 + 37 candidates on the 904-example memory of the same day; the recovered negatives of ef6945e moved the seeded compile split and the outer folds). Freezing needed two repairs first: example rows carried the plane-only features as null in x and the renderer refused them (1f525ff), and the RustFS store held none of the 788 embedding vectors the file store had (copied over with put_blob, no embedding call)."
},
"first_registration": {"registered": "2026-10-02", "pin": "fbc3220bbe1c8a4f8afa0a724fd41a51b390dd22", "units_status": "pending"},
"status": "run and analysed 2026-10-02: reject (result.json; benchmarks/measurements/2026-10-02-exp-002/record.json)"
}
@@ -0,0 +1,241 @@
{
"alpha": 0.05,
"arms": {
"A": {
"budget_usd": 1.0,
"label": "today's sampling -- k = 5, sample 1 at temperature 0 and samples 2..5 at 0.7, one retrieved example set",
"repeats": 2,
"replay_only": false
},
"B": {
"budget_usd": 3.0,
"label": "retrieval ensemble -- k = 5 at temperature 0, each sample over a disjoint retrieved example set",
"repeats": 2,
"replay_only": false
}
},
"code_commit": "780caaa558f2cc091fd6b0fd5c618c3b488030c2",
"decision": "reject",
"experiment": "exp-002-retrieval-ensemble",
"hypothesis": "Drawing the k = 5 samples from five disjoint retrieved example sets at temperature 0 (a retrieval ensemble) makes the decision more repeatable than five temperature samples over one example set, without hurting the ranking within a candidate pair, at no higher cost per candidate.\n",
"looks": [
{
"boundary": 2.797,
"crossed": false,
"estimate": {
"a": 0.8611111111111112,
"b": 0.8888888888888888,
"ci": [
-0.08333333333333326,
0.1325301204819277
],
"diff": 0.02777777777777768,
"groups": 28,
"metric": "canary_agreement",
"resamples": 10000,
"se": 0.05449719877526111,
"units": 72
},
"fraction": 0.5,
"groups": 28,
"z": 0.5097101943226363
},
{
"boundary": 1.977,
"crossed": false,
"estimate": {
"a": 0.875,
"b": 0.90625,
"ci": [
-0.03448275862068961,
0.0972972972972973
],
"diff": 0.03125,
"groups": 56,
"metric": "canary_agreement",
"resamples": 10000,
"se": 0.0337315848833629,
"units": 160
},
"fraction": 1.0,
"groups": 56,
"z": 0.9264314175588332
}
],
"manifest_digest": "f5004fb8618b1f7cc7dcf298d42b4ac1b6c0c1aa72a97299c77f937bd6175148",
"mde": 0.1,
"primary": [
{
"name": "canary_agreement",
"target": 0.9
},
{
"name": "within_pair_auc"
}
],
"reason": "the CI of within_pair_auc excludes 0 against B",
"resamples": 10000,
"secondary": [
{
"name": "usd_per_candidate"
},
{
"name": "auc"
},
{
"name": "advisory_recall"
},
{
"name": "advisory_precision"
}
],
"spend_usd": {
"A": 0.682094,
"B": 1.632737
},
"tests": {
"advisory_precision": {
"ci_vs_zero": "spans",
"direction": 1,
"estimate": {
"a": 0.8979591836734694,
"b": 0.9069767441860465,
"ci": [
-0.02020202020202022,
0.05081826012058566
],
"diff": 0.009017560512577072,
"groups": 56,
"metric": "advisory_precision",
"resamples": 10000,
"se": 0.018550028052620342,
"units": 160
},
"mcnemar": null,
"target": null
},
"advisory_recall": {
"ci_vs_zero": "spans",
"direction": 1,
"estimate": {
"a": 0.5,
"b": 0.4431818181818182,
"ci": [
-0.1235955056179775,
0.01010101010101011
],
"diff": -0.05681818181818182,
"groups": 56,
"metric": "advisory_recall",
"resamples": 10000,
"se": 0.033343898686966876,
"units": 160
},
"mcnemar": {
"b": 7,
"c": 2,
"concordant": 79,
"p_value": 0.1796875
},
"target": null
},
"auc": {
"ci_vs_zero": "against",
"direction": 1,
"estimate": {
"a": 0.8012941919191919,
"b": 0.7802241161616161,
"ci": [
-0.04139254385964919,
-0.0034279184917158645
],
"diff": -0.0210700757575758,
"groups": 56,
"metric": "auc",
"resamples": 10000,
"se": 0.009565373841115053,
"units": 160
},
"mcnemar": null,
"target": null
},
"canary_agreement": {
"ci_vs_zero": "spans",
"direction": 1,
"estimate": {
"a": 0.875,
"b": 0.90625,
"ci": [
-0.03448275862068961,
0.0972972972972973
],
"diff": 0.03125,
"groups": 56,
"metric": "canary_agreement",
"resamples": 10000,
"se": 0.0337315848833629,
"units": 160
},
"mcnemar": {
"b": 10,
"c": 15,
"concordant": 135,
"p_value": 0.42435622215270996
},
"target": {
"b_meets": true,
"level": 0.9
}
},
"usd_per_candidate": {
"ci_vs_zero": "against",
"direction": -1,
"estimate": {
"a": 0.002131543750000001,
"b": 0.005102303125,
"ci": [
0.002724568181818181,
0.00317872222222222
],
"diff": 0.002970759374999999,
"groups": 56,
"metric": "usd_per_candidate",
"resamples": 10000,
"se": 0.00011577410791736543,
"units": 160
},
"mcnemar": null,
"target": null
},
"within_pair_auc": {
"ci_vs_zero": "against",
"direction": 1,
"estimate": {
"a": 0.8148148148148148,
"b": 0.7407407407407407,
"ci": [
-0.12857142857142867,
-0.015151515151515138
],
"diff": -0.07407407407407407,
"groups": 56,
"metric": "within_pair_auc",
"resamples": 10000,
"se": 0.029550669251750233,
"units": 160
},
"mcnemar": null,
"target": null
}
},
"tost": null,
"units": {
"by_label": {
"0": 72,
"1": 88
},
"groups": 56,
"outcomes": 640,
"paired": 160
}
}
@@ -0,0 +1,39 @@
{
"experiment": "exp-003-precision-bound-blocking",
"registered": "2026-10-03",
"store": "s3://sast-memory",
"manifest": "plane/experiments/exp-003-precision-bound-blocking.yaml",
"manifest_digest": "5917f9a12035f09eae76978f3b5827b9fe59c8a9a5275f9ab54c3c1b55e8db3b",
"pin": "dd7ebc119b5ad8e77c05d418eed9cbd46e193587",
"units_status": "frozen",
"units_file": "plane/experiments/exp-003-precision-bound-blocking.units.jsonl",
"units_digest": "338de998b1cfa36857cd793dbc4cf7fd0a13ab517a91f6790335881e941b3006",
"units": {
"total": 353,
"paired": 244,
"groups": 146,
"by_family": {
"access_control": {"units": 41, "positives": 25, "paired": 32, "paired_positives": 16, "groups": 24},
"deserialization": {"units": 48, "positives": 31, "paired": 34, "paired_positives": 17, "groups": 31},
"injection": {"units": 119, "positives": 63, "paired": 76, "paired_positives": 38, "groups": 34},
"output_encoding": {"units": 45, "positives": 29, "paired": 32, "paired_positives": 16, "groups": 27},
"path": {"units": 39, "positives": 23, "paired": 32, "paired_positives": 16, "groups": 23},
"untrusted_destination": {"units": 61, "positives": 37, "paired": 38, "paired_positives": 19, "groups": 36}
},
"note": "frozen model-free on the 937-example static memory; the outer folds (k 5, seed 0, compile fraction 0.25) are the incumbent evaluation's; injection and access_control equal exp-002's 160 units"
},
"kind": "rule: one zero-cost scoring pass of the six slice-2 incumbent programs; arms differ only in the BLOCK rule",
"zero_cost": "responses replayed from the response cache only, behind a meter with zero USD and call ceilings around a refusing client; a unit with any uncached response is excluded and counted; the run is void unless the meter reads 0 calls and $0",
"arms": {
"A": "today's rule: Platt map cross-fitted per family; BLOCK only when the calibration check holds (slope CI contains 1, intercept CI contains 0, ECE <= 0.05); BLOCK point nested on the other folds' calibrated p (Wilson LB >= 0.95)",
"B": "selective blocking: per family and outer fold, threshold chosen on the family's paired units in the other four folds as the lowest raw score t with Wilson 95% LB of flagged precision >= 0.95 (unsure verdicts never flag); none -> no BLOCK on that fold",
"C": "secondary, non-deciding: as B with point precision >= 0.95 over at least 10 training flags"
},
"primary_metric": "held-out (outer-fold) Wilson 95% lower bound of flagged precision, per family, arm B, paired population",
"populations": {"primary": "paired (both sides of the candidate present; excludes every positives-only group and any unpaired unit)", "secondary": ["pooled (every unit)"]},
"secondary_metrics": ["flagged_precision", "coverage (flagged true positives / positives)", "flagged", "repository-cluster bootstrap 95% CI of flagged precision"],
"decision_rule": "REJECT B if in any family B flags held-out paired units at a point precision < 0.95; else ADOPT B if in some family B's held-out paired Wilson LB >= 0.95; else INCONCLUSIVE (the incumbent rule stays)",
"m4_reading": "a family reaches the M4 precision bar on held-out repositories when its held-out paired Wilson LB >= 0.95; point precision reported beside it",
"power": "a Wilson LB of 0.95 needs >= 73 flagged units with no false flag; the largest family holds 38 paired positives over all five folds, so arm B cannot offer a BLOCK in any family by construction; expected verdict inconclusive",
"status": "run and analysed 2026-10-03: inconclusive -- no arm flags any held-out unit (result.json); meter 0 client calls, $0.0; 142 of 353 units excluded for an uncached response"
}
@@ -0,0 +1,981 @@
{
"arms": {
"A": {
"label": "today's rule -- Platt map cross-fitted per family, BLOCK only when the calibration check holds, BLOCK point nested on the other folds' calibrated p",
"rule": {
"name": "calibrated_block"
}
},
"B": {
"label": "selective blocking -- per-family raw-score threshold from the other folds' paired units, lowest t with Wilson 95% LB of flagged precision >= 0.95",
"rule": {
"bound": "wilson",
"name": "precision_bound_block",
"precision": 0.95,
"select_on": "paired"
}
},
"C": {
"label": "secondary, non-deciding -- as B with point precision >= 0.95 over at least 10 training flags",
"rule": {
"bound": "point",
"min_flagged": 10,
"name": "precision_bound_block",
"precision": 0.95,
"select_on": "paired"
}
}
},
"code_commit": "7bb97ad0bca62b5f986c5768297d8b8fca50fae4",
"decision": "inconclusive",
"experiment": "exp-003-precision-bound-blocking",
"families": {
"access_control": {
"A": {
"calibration_holds": false,
"ece": 0.077,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 16,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 16,
"true_flagged": 0,
"units": 32
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 24,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 25,
"true_flagged": 0,
"units": 41
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 16,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 16,
"true_flagged": 0,
"units": 32
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 24,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 25,
"true_flagged": 0,
"units": 41
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"C": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 16,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 16,
"true_flagged": 0,
"units": 32
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 24,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 25,
"true_flagged": 0,
"units": 41
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"capacity": {
"evaluated": 41,
"excluded_replay_miss": 0,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 12,
"outer-1": 13,
"outer-2": 13,
"outer-3": 13,
"outer-4": 13
}
}
},
"deserialization": {
"A": {
"calibration_holds": false,
"ece": null,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 3,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 2,
"true_flagged": 0,
"units": 5
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 12,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 11,
"true_flagged": 0,
"units": 14
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 3,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 2,
"true_flagged": 0,
"units": 5
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 12,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 11,
"true_flagged": 0,
"units": 14
},
"thresholds_by_fold": {
"outer-0": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"C": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 3,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 2,
"true_flagged": 0,
"units": 5
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 12,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 11,
"true_flagged": 0,
"units": 14
},
"thresholds_by_fold": {
"outer-0": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"capacity": {
"evaluated": 14,
"excluded_replay_miss": 34,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 0,
"outer-2": 2,
"outer-3": 2,
"outer-4": 2
}
}
},
"injection": {
"A": {
"calibration_holds": false,
"ece": 0.1067,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 26,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 38,
"true_flagged": 0,
"units": 76
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 34,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 63,
"true_flagged": 0,
"units": 119
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 26,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 38,
"true_flagged": 0,
"units": 76
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 34,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 63,
"true_flagged": 0,
"units": 119
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"C": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 26,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 38,
"true_flagged": 0,
"units": 76
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 34,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 63,
"true_flagged": 0,
"units": 119
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"capacity": {
"evaluated": 119,
"excluded_replay_miss": 0,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 29,
"outer-1": 33,
"outer-2": 33,
"outer-3": 26,
"outer-4": 31
}
}
},
"output_encoding": {
"A": {
"calibration_holds": false,
"ece": null,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 8,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 8,
"true_flagged": 0,
"units": 16
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 15,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 16,
"true_flagged": 0,
"units": 24
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 8,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 8,
"true_flagged": 0,
"units": 16
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 15,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 16,
"true_flagged": 0,
"units": 24
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"C": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 8,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 8,
"true_flagged": 0,
"units": 16
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 15,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 16,
"true_flagged": 0,
"units": 24
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"capacity": {
"evaluated": 24,
"excluded_replay_miss": 21,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 7,
"outer-1": 6,
"outer-2": 6,
"outer-3": 7,
"outer-4": 6
}
}
},
"path": {
"A": {
"paired": {
"coverage": null,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 0,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 0,
"true_flagged": 0,
"units": 0
},
"pooled": {
"coverage": null,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 0,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 0,
"true_flagged": 0,
"units": 0
}
},
"B": {
"paired": {
"coverage": null,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 0,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 0,
"true_flagged": 0,
"units": 0
},
"pooled": {
"coverage": null,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 0,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 0,
"true_flagged": 0,
"units": 0
}
},
"C": {
"paired": {
"coverage": null,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 0,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 0,
"true_flagged": 0,
"units": 0
},
"pooled": {
"coverage": null,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 0,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 0,
"true_flagged": 0,
"units": 0
}
},
"capacity": {
"evaluated": 0,
"excluded_replay_miss": 39,
"not_run": 0,
"paired_positives_by_training_folds": {}
}
},
"untrusted_destination": {
"A": {
"calibration_holds": false,
"ece": null,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 6,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 4,
"true_flagged": 0,
"units": 9
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 9,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 7,
"true_flagged": 0,
"units": 13
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null
}
},
"B": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 6,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 4,
"true_flagged": 0,
"units": 9
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 9,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 7,
"true_flagged": 0,
"units": 13
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null
}
},
"C": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 6,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 4,
"true_flagged": 0,
"units": 9
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 9,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 7,
"true_flagged": 0,
"units": 13
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null
}
},
"capacity": {
"evaluated": 13,
"excluded_replay_miss": 48,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 4,
"outer-1": 3,
"outer-2": 3,
"outer-3": 2
}
}
}
},
"hypothesis": "The calibrated-probability BLOCK rule (Platt map + calibration gate, ECE <= 0.05) is the wrong instrument for M4, which asks for precision on what is flagged. Selective blocking -- a per-family raw-score threshold chosen on training folds only as the lowest one whose Wilson 95% lower bound on flagged precision is >= 0.95 -- reaches the M4 precision bar on held-out repositories where the calibrated rule offers no BLOCK.\n",
"instrument_check": {
"families": {
"access_control": {
"distinct_s": 30,
"paired": {
"blockable": 28,
"positives_before_first_negative": 10
},
"pooled": {
"blockable": 37,
"positives_before_first_negative": 19
},
"verdicts": {
"not_vulnerable": 19,
"unsure": 4,
"vulnerable": 18
}
},
"deserialization": {
"distinct_s": 5,
"paired": {
"blockable": 4,
"positives_before_first_negative": 1
},
"pooled": {
"blockable": 13,
"positives_before_first_negative": 10
},
"verdicts": {
"not_vulnerable": 2,
"unsure": 1,
"vulnerable": 11
}
},
"injection": {
"distinct_s": 48,
"paired": {
"blockable": 50,
"positives_before_first_negative": 3
},
"pooled": {
"blockable": 77,
"positives_before_first_negative": 11
},
"verdicts": {
"not_vulnerable": 46,
"unsure": 42,
"vulnerable": 31
}
},
"output_encoding": {
"distinct_s": 14,
"paired": {
"blockable": 11,
"positives_before_first_negative": 0
},
"pooled": {
"blockable": 19,
"positives_before_first_negative": 8
},
"verdicts": {
"not_vulnerable": 9,
"unsure": 5,
"vulnerable": 10
}
},
"untrusted_destination": {
"distinct_s": 9,
"paired": {
"blockable": 4,
"positives_before_first_negative": 1
},
"pooled": {
"blockable": 8,
"positives_before_first_negative": 4
},
"verdicts": {
"not_vulnerable": 3,
"unsure": 5,
"vulnerable": 5
}
}
},
"note": "descriptive, all folds together, ranking by raw score s among non-unsure verdicts; not nested, chooses nothing. It shows the scores are real and the zero is the rule's, not an unread input."
},
"manifest_digest": "5917f9a12035f09eae76978f3b5827b9fe59c8a9a5275f9ab54c3c1b55e8db3b",
"meter": {
"client_calls": 0,
"decided": 211,
"excluded_replay_miss": 142,
"folds_digest": "a1f78e141a572a4adcc9cd57eb8a784c440c177f0b60d5ef5bed9e94c8753c5f",
"metered_usd": 0.0,
"replay_priced_usd_not_spent": 0.582876,
"replayed_responses": 1056
},
"min_flagged_for_bound": 73,
"overall": {
"A": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 59,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 68,
"true_flagged": 0,
"units": 138
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 89,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 122,
"true_flagged": 0,
"units": 211
}
},
"B": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 59,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 68,
"true_flagged": 0,
"units": 138
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 89,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 122,
"true_flagged": 0,
"units": 211
}
},
"C": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 59,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 68,
"true_flagged": 0,
"units": 138
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 89,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 122,
"true_flagged": 0,
"units": 211
}
}
},
"populations": {
"primary": "paired",
"secondary": [
"pooled"
]
},
"primary": [
{
"name": "flagged_precision_lb",
"target": 0.95
}
],
"reason": "B offers no BLOCK on held-out paired units in any family; the incumbent stays",
"resamples": 2000,
"secondary": [
{
"name": "flagged_precision"
},
{
"name": "coverage"
},
{
"name": "flagged"
}
],
"spend_usd": {
"score": 0.0
},
"target_precision": 0.95,
"units": {
"by_label": {
"0": 89,
"1": 122
},
"evaluated": 211,
"excluded_replay_miss": 142,
"frozen": 353,
"groups": 89,
"not_run": 0,
"paired_evaluated": 138,
"unsure": 57
}
}
@@ -0,0 +1,53 @@
"""Instrument check for exp-004: are the scores real, and how clean is the pooled top of the ranking?
Reads the run's arm_outcome rows from the memory store (OUSAST_MEMORY) and the frozen units file; makes no model call.
usage: python benchmarks/experiments/exp-004-pooled-block-gate/instrument_check.py (from the repository root)
"""
from __future__ import annotations
import collections
import json
from pathlib import Path
from openultrasast.config import load_dotenv
from openultrasast.learn import experiments as ex
from openultrasast.plane.memory import open_store
EXPERIMENT = "exp-004-pooled-block-gate"
UNITS = Path(f"plane/experiments/{EXPERIMENT}.units.jsonl")
def main() -> None:
load_dotenv(Path(".env"))
rows = [r.row for r in open_store(None).rows(repo=ex.experiment_repo(EXPERIMENT), kind="arm_outcome")]
with UNITS.open() as handle:
units = {u["unit"]: u for u in (json.loads(line) for line in handle)}
print("arm_outcome rows", len(rows), "units file", len(units))
by_family: dict[str, list[tuple[float | None, int, str | None, str]]] = collections.defaultdict(list)
for row in rows:
unit = units.get(row.get("unit"))
if unit:
by_family[unit["family"]].append((row.get("s"), unit["label"], unit.get("pair"), str(row.get("verdict") or "")))
for family, xs in sorted(by_family.items()):
scored = [x for x in xs if x[0] is not None]
pairs: dict[str, dict[int, float]] = collections.defaultdict(dict)
for score, label, pair, _ in scored:
if pair:
pairs[pair][label] = score
full = [p for p in pairs.values() if 0 in p and 1 in p]
wins = sum(1 for p in full if p[1] > p[0])
ties = sum(1 for p in full if p[1] == p[0])
top = sorted((x for x in scored if x[2]), key=lambda x: -(x[0] or 0.0))
first_negative = next((i for i, x in enumerate(top) if x[1] == 0), len(top))
print(
f"{family}: n={len(xs)} distinct_scores={len({round(x[0] or 0.0, 4) for x in scored})} "
f"unsure={sum(1 for x in xs if x[3] == 'unsure')} pairs={len(full)} within_pair_win={wins} tie={ties} "
f"pos_above_first_neg={first_negative}"
)
pooled = sorted((x for xs in by_family.values() for x in xs if x[0] is not None and x[2]), key=lambda x: -(x[0] or 0.0))
print("pooled raw top-30 labels:", "".join(str(x[1]) for x in pooled[:30]))
if __name__ == "__main__":
main()
@@ -0,0 +1,81 @@
{
"experiment": "exp-004-pooled-block-gate",
"date": "2026-10-03",
"fill": {
"client_calls": 2073,
"usd": 1.238223,
"ceiling_usd": 2.0,
"still_missing": 0,
"dry_run_estimate_usd": 5.582,
"note": "the dry-run estimator over-priced 4.5x (no prompt-cache pricing); measured spend is the meter's"
},
"run": {
"units": 501,
"client_calls": 0,
"usd": 0.0,
"replayed": 2509,
"replay_miss": 0
},
"verdict": "inconclusive",
"reason": "no t_block on any outer fold, pooled or per family; B offers no BLOCK",
"instrument_check": {
"per_family": {
"access_control": {
"n": 66,
"distinct_scores": 41,
"unsure": 5,
"pairs": 28,
"within_pair_wins": 22,
"ties": 1,
"positives_above_first_negative": 12
},
"deserialization": {
"n": 64,
"distinct_scores": 38,
"unsure": 7,
"pairs": 25,
"within_pair_wins": 18,
"ties": 4,
"positives_above_first_negative": 5
},
"injection": {
"n": 149,
"distinct_scores": 52,
"unsure": 54,
"pairs": 53,
"within_pair_wins": 30,
"ties": 17,
"positives_above_first_negative": 6
},
"output_encoding": {
"n": 69,
"distinct_scores": 31,
"unsure": 32,
"pairs": 28,
"within_pair_wins": 18,
"ties": 6,
"positives_above_first_negative": 2
},
"path": {
"n": 66,
"distinct_scores": 43,
"unsure": 15,
"pairs": 29,
"within_pair_wins": 25,
"ties": 3,
"positives_above_first_negative": 13
},
"untrusted_destination": {
"n": 87,
"distinct_scores": 36,
"unsure": 33,
"pairs": 32,
"within_pair_wins": 18,
"ties": 4,
"positives_above_first_negative": 3
}
},
"pooled_raw_top30_labels": "111111111110111110111111110111",
"reading": "scores are real (31-52 distinct values per family); pooled raw top 30 holds 27 positives (0.90), first negative at rank 12; the Wilson bound needs >= 73 flags with no error, so precision at the top, not data volume alone, now binds"
}
}
@@ -0,0 +1,35 @@
{
"experiment": "exp-004-pooled-block-gate",
"registered": "2026-10-03",
"store": "s3://sast-memory",
"manifest": "plane/experiments/exp-004-pooled-block-gate.yaml",
"manifest_digest": "87fceb6580c6de629d6a6089eb5888cdfc4df0f791576184f987c14c192d85f5",
"pin": "2ad9c1db7ae673c2a2e48a0afadcaa795dfb1ced",
"units_status": "pending",
"units_file": "plane/experiments/exp-004-pooled-block-gate.units.jsonl",
"units_digest": "",
"units": "frozen at run time by `ousast learn experiment units` from the memory after the negatives harvest (merged in 5b2bd9b, 1133 static examples), committed, and registered again before the run; until then the run refuses (check_registered: the units file does not exist)",
"requirement": "learned-decision-engine Req 6.4, amended 2026-10-03 by maintainer decision (refines the BLOCK rule of Req 7.3 and 7.4)",
"kind": "rule: one zero-cost scoring pass of the six slice-2 incumbent programs; arms differ only in the BLOCK rule",
"zero_cost": "the run replays the response cache behind a zero-ceiling meter; a unit with any uncached response is excluded and counted. The harvest changed the memory the programs retrieve from, so a paid cache fill, approved separately with its own ceiling, must precede the run",
"normalisation": {
"choice": "per-family Platt scaling: learn.calibrate.fit_map with its existing floors (own slope and intercept per family at >= 20 positive and negative repository groups, a pooled slope with a per-family intercept at >= 10, no map below that and so no BLOCK for that family on that fold), fitted on the training folds' paired units only",
"why": "a rank-to-quantile map gives every family the same flag rate whatever its precision, so one pooled threshold would flag the top slice of a weak family as readily as of a strong one; Platt puts each family on the scale of its own training precision, which a pooled precision bound needs. The map is a normaliser only: precision is shown by the held-out bound, not by calibration (exp-003: ECE <= 0.05 does not hold per family)"
},
"arms": {
"A": "today's rule: Platt map cross-fitted per family; BLOCK only when the calibration check holds; BLOCK point nested on the other folds' calibrated p (Wilson LB >= 0.95)",
"B": "pooled gate: per outer fold j, training units normalised by an inner cross-fit; t_block = the lowest normalised score whose Wilson 95% LB of precision over the training paired flags of all families together is >= 0.95 (unsure never flags); t_quiet per family = the highest normalised score keeping >= 90% of that family's training paired positives; fold j's units BLOCK at p >= t_block, ADVISORY at p >= t_quiet (or without a map or t_quiet), silent below"
},
"family_floor": {
"F": 10,
"rule": "a family may block only if its own held-out paired flags (all five folds) number >= 10 with zero errors; otherwise its flags are reported as ADVISORY. The floor uses held-out labels to decide which families block and moves no threshold",
"why": "zero errors in 10 flags is unlikely for a family much worse than the bar (probability 0.056 at precision 0.75, 0.107 at 0.80); a larger F is out of reach of the smaller families (16-19 paired positives in exp-003's units over all folds); 10 is exp-003 arm C's training minimum"
},
"primary_metric": "held-out (outer-fold) Wilson 95% lower bound of flagged precision over arm B's paired flags of all families together, before the family floor (so dropping a failing family cannot raise it)",
"populations": {"primary": "paired (both sides of the candidate present)", "secondary": ["pooled (every unit)"]},
"secondary_metrics": ["per-family flagged precision and Wilson LB", "the family-floor table", "quiet_recall per family (held-out positives at or above t_quiet / positives) against 0.90", "after-floor totals (what users would get)", "repository-cluster bootstrap 95% CI of flagged precision", "coverage", "flagged"],
"decision_rule": "on arm B's held-out paired flags of all families together, before the floor: REJECT B if their point precision < 0.95; else ADOPT B if their Wilson 95% LB >= 0.95 and at least one family passes the floor (>= 10 held-out paired flags, zero errors) -- those families block, every other family stays advisory; else INCONCLUSIVE (the incumbent rule stays)",
"power": "the pooled bound needs >= 73 flags with no error (110 with one, 142 with two). On exp-003's units (all 353 scored, result-complete-cache.json) few paired positives rank above the first negative per family on the raw score (access control 10, deserialization 7, injection 2, output encoding 0, path 8, untrusted destination 5); normalisation keeps each family's order, so the pooled top set reaches 73 only if the harvest's pairs separate better. An inconclusive verdict is a plausible outcome",
"tests": "tests/test_learn_rules.py: test_one_pooled_threshold_blocks_where_no_family_alone_reaches_the_bound, test_no_held_out_label_moves_a_threshold_or_the_map, test_a_family_with_one_held_out_error_cannot_block, test_quiet_threshold_keeps_ninety_percent_of_positives_and_nothing_is_silenced_blind, test_the_pooled_verdict_follows_the_registered_rule, test_the_exp004_manifest_registers_the_procedure_before_its_units",
"status": "registered 2026-10-03, units pending, not run"
}
@@ -0,0 +1,822 @@
{
"arms": {
"A": {
"label": "today's rule -- Platt map cross-fitted per family, BLOCK only when the calibration check holds, BLOCK point nested on the other folds' calibrated p",
"rule": {
"name": "calibrated_block"
}
},
"B": {
"label": "pooled gate -- one t_block over all families on a per-family Platt score (training folds, inner cross-fit), t_quiet per family at 90% recall, family floor 10 clean held-out flags",
"rule": {
"floor": 10,
"name": "pooled_block_gate",
"normalise": "platt",
"precision": 0.95,
"recall": 0.9,
"select_on": "paired"
}
}
},
"code_commit": "90a2642019df414cb67b5dcea6e4ec235f35650a",
"decision": "inconclusive",
"experiment": "exp-004-pooled-block-gate",
"families": {
"access_control": {
"A": {
"calibration_holds": false,
"ece": 0.0715,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 28,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 28,
"true_flagged": 0,
"units": 56
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 37,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 38,
"true_flagged": 0,
"units": 66
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"floor": {
"errors": 0,
"flagged": 0,
"floor": 10,
"may_block": false
},
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 28,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 28,
"true_flagged": 0,
"units": 56
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 37,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 38,
"true_flagged": 0,
"units": 66
},
"quiet_recall": {
"paired": 0.8929,
"pooled": 0.9211
},
"t_block_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
},
"t_quiet_by_fold": {
"outer-0": 0.44900641988143913,
"outer-1": 0.3723384028044276,
"outer-2": 0.4202772526614184,
"outer-3": 0.4481849709797749,
"outer-4": 0.36346174463707154
}
},
"capacity": {
"evaluated": 66,
"excluded_replay_miss": 0,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 22,
"outer-1": 22,
"outer-2": 22,
"outer-3": 23,
"outer-4": 23
}
}
},
"deserialization": {
"A": {
"calibration_holds": false,
"ece": 0.0725,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 25,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 25,
"true_flagged": 0,
"units": 50
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 39,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 39,
"true_flagged": 0,
"units": 64
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"floor": {
"errors": 0,
"flagged": 0,
"floor": 10,
"may_block": false
},
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 25,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 25,
"true_flagged": 0,
"units": 50
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 39,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 39,
"true_flagged": 0,
"units": 64
},
"quiet_recall": {
"paired": 0.88,
"pooled": 0.9231
},
"t_block_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
},
"t_quiet_by_fold": {
"outer-0": 0.17507194490500488,
"outer-1": 0.22766488322270723,
"outer-2": 0.2163745870743959,
"outer-3": 0.3112201878893598,
"outer-4": 0.363243722721776
}
},
"capacity": {
"evaluated": 64,
"excluded_replay_miss": 0,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 20,
"outer-1": 20,
"outer-2": 20,
"outer-3": 20,
"outer-4": 20
}
}
},
"injection": {
"A": {
"calibration_holds": false,
"ece": 0.1042,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 41,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 53,
"true_flagged": 0,
"units": 106
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 49,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 78,
"true_flagged": 0,
"units": 149
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"floor": {
"errors": 0,
"flagged": 0,
"floor": 10,
"may_block": false
},
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 41,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 53,
"true_flagged": 0,
"units": 106
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 49,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 78,
"true_flagged": 0,
"units": 149
},
"quiet_recall": {
"paired": 0.8679,
"pooled": 0.8974
},
"t_block_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
},
"t_quiet_by_fold": {
"outer-0": 0.287983504097999,
"outer-1": 0.17524866355153343,
"outer-2": 0.12918182665453967,
"outer-3": 0.14521040470628938,
"outer-4": 0.23005714687697673
}
},
"capacity": {
"evaluated": 149,
"excluded_replay_miss": 0,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 38,
"outer-1": 44,
"outer-2": 45,
"outer-3": 43,
"outer-4": 42
}
}
},
"output_encoding": {
"A": {
"calibration_holds": false,
"ece": 0.0736,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 28,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 28,
"true_flagged": 0,
"units": 56
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 39,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 41,
"true_flagged": 0,
"units": 69
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"floor": {
"errors": 0,
"flagged": 0,
"floor": 10,
"may_block": false
},
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 28,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 28,
"true_flagged": 0,
"units": 56
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 39,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 41,
"true_flagged": 0,
"units": 69
},
"quiet_recall": {
"paired": 0.8571,
"pooled": 0.9024
},
"t_block_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
},
"t_quiet_by_fold": {
"outer-0": 0.3723166587726901,
"outer-1": 0.5329072757249385,
"outer-2": 0.3785066578507705,
"outer-3": 0.4389011844848602,
"outer-4": 0.43855966643440264
}
},
"capacity": {
"evaluated": 69,
"excluded_replay_miss": 0,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 23,
"outer-1": 22,
"outer-2": 22,
"outer-3": 22,
"outer-4": 23
}
}
},
"path": {
"A": {
"calibration_holds": false,
"ece": 0.0621,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 29,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 29,
"true_flagged": 0,
"units": 58
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 37,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 37,
"true_flagged": 0,
"units": 66
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"floor": {
"errors": 0,
"flagged": 0,
"floor": 10,
"may_block": false
},
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 29,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 29,
"true_flagged": 0,
"units": 58
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 37,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 37,
"true_flagged": 0,
"units": 66
},
"quiet_recall": {
"paired": 0.8621,
"pooled": 0.8919
},
"t_block_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
},
"t_quiet_by_fold": {
"outer-0": 0.4680507207548809,
"outer-1": 0.3955717127313657,
"outer-2": 0.43694703358760895,
"outer-3": 0.4450583743716199,
"outer-4": 0.46396657833309796
}
},
"capacity": {
"evaluated": 66,
"excluded_replay_miss": 0,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 23,
"outer-1": 23,
"outer-2": 24,
"outer-3": 23,
"outer-4": 23
}
}
},
"untrusted_destination": {
"A": {
"calibration_holds": false,
"ece": 0.1143,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 31,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 32,
"true_flagged": 0,
"units": 64
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 49,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 50,
"true_flagged": 0,
"units": 87
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"floor": {
"errors": 0,
"flagged": 0,
"floor": 10,
"may_block": false
},
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 31,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 32,
"true_flagged": 0,
"units": 64
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 49,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 50,
"true_flagged": 0,
"units": 87
},
"quiet_recall": {
"paired": 1.0,
"pooled": 1.0
},
"t_block_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
},
"t_quiet_by_fold": {
"outer-0": 0.13676746287025585,
"outer-1": 0.11255775703980551,
"outer-2": 0.13763698150768486,
"outer-3": 0.09787938647374639,
"outer-4": 0.13763698150768486
}
},
"capacity": {
"evaluated": 87,
"excluded_replay_miss": 0,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 26,
"outer-1": 26,
"outer-2": 24,
"outer-3": 26,
"outer-4": 26
}
}
}
},
"hypothesis": "One BLOCK threshold pooled across families on a per-family Platt-normalised score, chosen on training folds only as the lowest one whose Wilson 95% lower bound on flagged precision is >= 0.95, reaches that bound on held-out repositories' paired units, where per-family thresholds (exp-003) could not for lack of flags; a per-family floor (>= 10 held-out flags, none wrong) keeps a family that the pooling masks from blocking.\n",
"manifest_digest": "87fceb6580c6de629d6a6089eb5888cdfc4df0f791576184f987c14c192d85f5",
"min_flagged_for_bound": 73,
"overall": {
"A": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 182,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 195,
"true_flagged": 0,
"units": 390
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 220,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 283,
"true_flagged": 0,
"units": 501
}
},
"B": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 182,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 195,
"true_flagged": 0,
"units": 390
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 220,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 283,
"true_flagged": 0,
"units": 501
}
}
},
"overall_after_floor": {
"B": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 182,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 195,
"true_flagged": 0,
"units": 390
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 220,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 283,
"true_flagged": 0,
"units": 501
}
}
},
"populations": {
"primary": "paired",
"secondary": [
"pooled"
]
},
"primary": [
{
"name": "flagged_precision_lb",
"target": 0.95
}
],
"reason": "B offers no BLOCK on held-out paired units; the incumbent stays",
"resamples": 2000,
"secondary": [
{
"name": "flagged_precision"
},
{
"name": "coverage"
},
{
"name": "flagged"
},
{
"name": "quiet_recall"
}
],
"spend_usd": {
"score": 0.0
},
"target_precision": 0.95,
"units": {
"by_label": {
"0": 218,
"1": 283
},
"evaluated": 501,
"excluded_replay_miss": 0,
"frozen": 501,
"groups": 220,
"not_run": 0,
"paired_evaluated": 390,
"unsure": 146
}
}
@@ -0,0 +1,84 @@
{
"arms": [
"B"
],
"client_calls": 2394,
"experiment": "exp-005-graph-slice",
"families": {
"access_control": {
"cached": 0,
"conditional_retries": 330,
"estimate_method": "estimated",
"estimated_output_tokens": 300,
"estimated_usd": 0.9956892000000007,
"filled": 330,
"missing": 330,
"requests": 330,
"still_missing": 0,
"units": 66
},
"deserialization": {
"cached": 0,
"conditional_retries": 310,
"estimate_method": "estimated",
"estimated_output_tokens": 300,
"estimated_usd": 0.8843295999999988,
"filled": 310,
"missing": 310,
"requests": 310,
"still_missing": 0,
"units": 64
},
"injection": {
"cached": 0,
"conditional_retries": 675,
"estimate_method": "estimated",
"estimated_output_tokens": 300,
"estimated_usd": 2.270278999999999,
"filled": 675,
"missing": 675,
"requests": 675,
"still_missing": 0,
"units": 149
},
"output_encoding": {
"cached": 0,
"conditional_retries": 325,
"estimate_method": "estimated",
"estimated_output_tokens": 300,
"estimated_usd": 1.0014180000000004,
"filled": 328,
"missing": 325,
"requests": 328,
"still_missing": 0,
"units": 69
},
"path": {
"cached": 0,
"conditional_retries": 320,
"estimate_method": "estimated",
"estimated_output_tokens": 300,
"estimated_usd": 0.9399192000000005,
"filled": 325,
"missing": 320,
"requests": 325,
"still_missing": 0,
"units": 66
},
"untrusted_destination": {
"cached": 0,
"conditional_retries": 425,
"estimate_method": "estimated",
"estimated_output_tokens": 300,
"estimated_usd": 1.174316,
"filled": 426,
"missing": 425,
"requests": 426,
"still_missing": 0,
"units": 87
}
},
"reason": "",
"status": "done",
"usd": 1.5890382240000016
}
@@ -0,0 +1,165 @@
{
"alpha": 0.05,
"arms": {
"A": {
"budget_usd": 0.0,
"label": "six slice-2 incumbents unchanged, replaying exp-004's response cache",
"repeats": 1,
"replay_only": true
},
"B": {
"budget_usd": 6.0,
"label": "the same six programs with the in-function slice added to their prompts",
"repeats": 1,
"replay_only": false
}
},
"code_commit": "0acbc7ec41dcaeca805c4720bc3f6d02cb940ee7",
"decision": "inconclusive",
"experiment": "exp-005-graph-slice",
"hypothesis": "Adding the in-function def-use slice to the prompt improves within-pair ranking of the six slice-2 incumbent programs on held-out repositories, at no more than 1.5 times the cost per candidate.\n",
"looks": [
{
"boundary": 1.959964,
"crossed": false,
"estimate": {
"a": 0.7615384615384615,
"b": 0.7461538461538462,
"ci": [
-0.0628019323671497,
0.03155339805825241
],
"diff": -0.01538461538461533,
"groups": 220,
"metric": "within_pair_auc",
"resamples": 10000,
"se": 0.024030780235707972,
"units": 501
},
"fraction": 1.0,
"groups": 220,
"z": -0.6402045723740141
}
],
"manifest_digest": "1b1ec814cde12801491376d2086c8c2d6db70f4a86fc308c0adc5a530398e554",
"mde": 0.0,
"primary": [
{
"name": "within_pair_auc"
}
],
"reason": "the CI of within_pair_auc spans 0; the incumbent stays",
"resamples": 10000,
"secondary": [
{
"name": "top_30_precision"
},
{
"name": "unsure_rate"
},
{
"name": "usd_per_candidate"
}
],
"spend_usd": {
"A": 1.444503,
"B": 1.589038
},
"tests": {
"top_30_precision": {
"ci_vs_zero": "spans",
"direction": 1,
"estimate": {
"a": 1.0,
"b": 1.0,
"ci": [
0.0,
0.0
],
"diff": 0.0,
"groups": 220,
"metric": "top_30_precision",
"resamples": 10000,
"se": 0.0006666666666666664,
"units": 501
},
"mcnemar": null,
"target": null
},
"unsure_rate": {
"ci_vs_zero": "favour",
"direction": -1,
"estimate": {
"a": 0.29141716566866266,
"b": 0.23952095808383234,
"ci": [
-0.08583690987124465,
-0.01996007984031939
],
"diff": -0.05189620758483032,
"groups": 220,
"metric": "unsure_rate",
"resamples": 10000,
"se": 0.01685991406098136,
"units": 501
},
"mcnemar": {
"b": 47,
"c": 21,
"concordant": 433,
"p_value": 0.0021862431776861705
},
"target": null
},
"usd_per_candidate": {
"ci_vs_zero": "against",
"direction": -1,
"estimate": {
"a": 0.002883239520958083,
"b": 0.0031717325349301397,
"ci": [
0.0002458205128205199,
0.0003319800399201574
],
"diff": 0.0002884930139720565,
"groups": 220,
"metric": "usd_per_candidate",
"resamples": 10000,
"se": 2.1746161270931252e-05,
"units": 501
},
"mcnemar": null,
"target": null
},
"within_pair_auc": {
"ci_vs_zero": "spans",
"direction": 1,
"estimate": {
"a": 0.7615384615384615,
"b": 0.7461538461538462,
"ci": [
-0.0628019323671497,
0.03155339805825241
],
"diff": -0.01538461538461533,
"groups": 220,
"metric": "within_pair_auc",
"resamples": 10000,
"se": 0.024030780235707972,
"units": 501
},
"mcnemar": null,
"target": null
}
},
"tost": null,
"units": {
"by_label": {
"0": 218,
"1": 283
},
"groups": 220,
"outcomes": 1002,
"paired": 501
}
}
+4 -2
View File
@@ -258,7 +258,7 @@ def score_v2() -> int:
per_case, pool, unstable_pool = [], [], 0
changing, total_questions, total_completed = [0, 0], 0, 0
for case in cases():
path = lambda label: RESULTS / "scans" / f"{case['id']}--{label}.json" # noqa: E731
path = lambda label: RESULTS / "scans" / f"{case['id']}--{label}.json" # noqa: E731, B023 -- called only inside this iteration
scans = {label: json.loads(path(label).read_text()) for label in pins(case) if path(label).is_file()}
if len(scans) < 6:
per_case.append({"id": case["id"], "status": "incomplete", "scans": sorted(scans)})
@@ -279,7 +279,9 @@ def score_v2() -> int:
]
site_files = {site.partition("::")[0] for site in case.get("sites", [])}
covered = set(old) | site_files
answered = any(r.split(":")[0] in covered for label in ("vulnerable_a", "vulnerable_b") for r in scans[label].get("completed_regions", []))
answered = any(
r.split(":")[0] in covered for label in ("vulnerable_a", "vulnerable_b") for r in scans[label].get("completed_regions", [])
)
diff_v, union_v = stability(va, vb)
diff_f, union_f = stability(fa, fb)
changing[0] += diff_v + diff_f
+1 -1
View File
@@ -44,7 +44,7 @@ EXTENSIONS = {
# scopes to API tokens" -- access control and input validation, which is exactly what a benign control must not be.
SECURITY_WORDS = re.compile(
r"secur|cve|ghsa|vuln|inject|xss|csrf|ssrf|sanitiz|escap|exploit|attack|auth|permission|cors|access|restrict|harden"
r"|token|scope|valid|filter|privileg|role|owner|password|secret|redirect|sql|query|login|log in|ldap|session|ssl|tls|verif|cert|crypt|hash|cookie",
r"|token|scope|valid|filter|privileg|role|owner|password|secret|redirect|sql|query|login|log in|ldap|session|ssl|tls|verif|cert|crypt|hash|cookie", # noqa: E501
re.I,
)
MAX_BENIGN_LINES = 200
+143 -19
View File
@@ -7,7 +7,7 @@ from:
- quick rules and deterministic roles, run here on the unit's tree: a pair side materialised from its excerpt, a
population or recipe pin exported from its clone or checkout (one at a time, deleted after, disk-guarded);
- the engine: ``benchmarks/learn/engine_pairs.py`` records for pairs, the recorded protocol scans
(``~/ousast-results/independent-v{1,2}/scans/<case>--<pin role>.json``) for populations;
(``~/ousast-results/independent-v{1,2}/scans/<case>--<pin role>[_a|_b].json``) for populations;
- verify passes a/b and ``agree`` from the harvest Run (``harvest-verify``) and, for population v2, the recorded
``validation-46`` Run; model roles from ``harvest-roles``;
- callers and entry-point distance on a repository (``not_applicable`` on a pair excerpt: no repository exists).
@@ -21,6 +21,11 @@ An instrument that has not run for a unit yet (the engine job still running, a R
memory store (``OUSAST_MEMORY``, default ``~/ousast-results/plane/memory``) as kind ``features``; a rerun replaces
them by id. ``--counts`` writes the counts-only coverage record (no candidate, path or repository name).
``--dry-run`` instead compares existing local store examples with the corrected engine features, printing
per-source transitions from not-run/none to ran (and recorded failures separately). It reads only the spent
v1/v2 manifests, saved scans and local Git objects; it writes neither the store nor ``--counts`` and does not
rebuild other instruments. No label snapshot, harvest units or scratch directory is needed in this mode.
Usage::
python benchmarks/learn/build_features.py --labels labels-<sha>.jsonl --units <harvest plane>/units.json --counts counts.json
@@ -30,9 +35,11 @@ from __future__ import annotations
import argparse
import json
import os
import shutil
import subprocess
import tempfile
import tomllib
from collections import Counter, defaultdict
from collections.abc import Mapping
from pathlib import Path
@@ -61,6 +68,7 @@ from openultrasast.learn.features import (
source_part,
verify_part,
)
from openultrasast.learn.labels import Clone
from openultrasast.learn.roles import RoleSet, from_model_roles, infer_for_checkout
from openultrasast.pairs import DEFAULT_CATALOG, _materialize_side, _targets, load_pair_catalog
from openultrasast.plane import memory
@@ -89,6 +97,109 @@ def read_json(path: Path) -> Any:
return json.loads(path.read_text(encoding="utf-8")) if path.is_file() else None
def resolve_scan(scans: Path, case: str, side: str) -> dict[str, Any] | None:
"""Read the v1 single scan, otherwise combine the v2 repeats for this side."""
single = read_json(scans / f"{case}--{side}.json")
if single is not None:
return dict(single)
repeats = [read_json(scans / f"{case}--{side}_{repeat}.json") for repeat in ("a", "b")]
available = [r for r in repeats if r is not None]
if not available:
return None
# Identical rule for vulnerable and fixed: a finding in either repeat counts;
# exact duplicates count once. engine_part takes the highest rung and minimum
# witness steps across the union. Completion is sum(completed)/sum(questions).
# A successful repeat survives a failed/missing peer, which marks degradation.
ran = [r for r in available if engine_for(r) is None]
findings = {json.dumps(f, sort_keys=True): f for r in ran for f in r.get("findings", [])}
degradations = sorted({d for r in available for d in r.get("degradations", [])})
if len(ran) != len(repeats):
degradations.append("repeat_missing_or_failed")
return {
"state": "ran" if ran else "none" if all(engine_for(r) == NONE for r in available) else "failed",
"questions": sum(int(r.get("questions") or 0) for r in available),
"completed": sum(int(r.get("completed") or 0) for r in available),
"findings": list(findings.values()),
"degradations": degradations,
"image": "+".join(sorted({str(r.get("image") or "recorded") for r in available})),
}
def recorded_engine(language: str, findings: list, result: Mapping | None, repository: bool) -> Part:
# Recorded findings take precedence over the shared engine language coverage.
if language not in ENGINE_LANGUAGES and not findings:
return NONE
state = engine_for(result)
if state is not None:
return state
result = result or {}
questions, completed = int(result.get("questions") or 0), int(result.get("completed") or 0)
return engine_part(
findings, completion=completed / questions if questions else None, degraded=bool(result.get("degradations")),
vocabulary=Vocabulary.load(language), version=str(result.get("image") or "recorded"),
) # fmt: skip
class PinSourceIndex(SourceIndex):
"""Resolve findings with the same matcher as a checkout, without exporting it."""
def __init__(self, clone: Path, pin: str) -> None:
super().__init__(clone)
self.clone, self.pin = Clone(clone), pin
def lines(self, path: str) -> list[str]:
if path not in self._lines:
self._lines[path] = self.clone.git("show", f"{self.pin}:{path}").splitlines()
return self._lines[path] or []
def dry_run(store: memory.FileStore, results: Path, cache: Path) -> dict[str, Any]:
"""Count existing examples whose engine becomes available; never write rows or blobs.
Only spent v1/v2 populations are read. Counts are examples, not the two
static/plane feature records. Other sources are reported with zero changes.
"""
root = Path(__file__).resolve().parents[2]
cases = {}
for population in ("population-v1", "population-v2"):
manifest = root / "benchmarks" / "independent" / f"{population}.toml"
for case in tomllib.loads(manifest.read_text(encoding="utf-8"))["case"]:
for side in ("vulnerable", "fixed"):
cases[(population, memory.repo_key(case["repo"]), case[side], side)] = case["id"]
rows = [r.row for r in store.rows(kind="example")]
if not rows:
raise ValueError("dry run read no examples from the store")
counts: dict[str, Counter[str]] = defaultdict(Counter)
for row in rows:
count = counts[row["source"]]
count["examples"] += 1
old = row["instruments"]["engine"]
reason = "not_run" if old.get("version") == "missing:not-run" else "none" if old["state"] == "none" else None
if reason is None:
continue
key = (row["split"], row["repo"], row["pin"], row["pin_role"])
case_id = cases.get(key)
if case_id is None:
continue
scans = results / row["split"].replace("population", "independent") / "scans"
result = resolve_scan(scans, case_id, row["pin_role"])
path, _, function = row["candidate"].partition("::")
engine = {}
if engine_for(result) is None:
index = PinSourceIndex(cache / "independent" / case_id, row["pin"])
lines = index.lines(path) # fail loudly if the candidate's input cannot be read
count["source_bytes_read"] += len("\n".join(lines).encode("utf-8"))
selected = dict(result or {})
selected["findings"] = [f for f in selected.get("findings", []) if f["site"].partition(":")[0] == path]
_, engine = _signals(index, [], selected, {})
part = recorded_engine(row["language"], engine.get((path, function, row["family"]), []), result, True)
if part.state == "ran":
count[f"{reason}_to_ran"] += 1
elif result is not None and part.state == "failed":
count[f"{reason}_to_recorded_failed"] += 1
return {source: {**{"not_run_to_ran": 0, "none_to_ran": 0}, **dict(count)} for source, count in sorted(counts.items())}
def done(run: Path, entry: str) -> bool:
state = (read_json(run / "state.json") or {}).get("tasks", {})
return state.get(entry, {}).get("status") == "done"
@@ -188,17 +299,7 @@ def static_parts(root: Path, index: SourceIndex, quick: Mapping, engine: Mapping
key = (path, function, family)
parts: dict[str, Part] = {}
parts["quick"] = quick_part(quick.get(key, []), rules, quick_version) if covers(QUICK_LANGUAGES, language) else NONE
if language not in ENGINE_LANGUAGES:
parts["engine"] = NONE
elif engine_state is not None:
parts["engine"] = engine_state
else:
result = engine_result or {}
questions, completed = int(result.get("questions") or 0), int(result.get("completed") or 0)
parts["engine"] = engine_part(
engine.get(key, []), completion=completed / questions if questions else None, degraded=bool(result.get("degradations")),
vocabulary=Vocabulary.load(language), version=str(result.get("image") or "recorded"),
) # fmt: skip
parts["engine"] = recorded_engine(language, engine.get(key, []), engine_result, repository)
lines = index.lines(path)
parts["source"] = source_part(lines, language, function)
parts["roles"] = roles_part(function_of(lines, path, language, function), inferred, language)
@@ -237,21 +338,34 @@ def export(clone: Path, pin: str, target: Path) -> bool:
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--labels", type=Path, required=True)
parser.add_argument("--units", type=Path, required=True, help="units.json of the harvest plane")
parser.add_argument("--labels", type=Path)
parser.add_argument("--units", type=Path, help="units.json of the harvest plane")
parser.add_argument("--catalog", type=Path, default=DEFAULT_CATALOG)
parser.add_argument("--cache", type=Path, default=Path.home() / ".cache" / "openultrasast")
parser.add_argument("--engine", type=Path, default=RESULTS / "plane" / "harvest-engine")
parser.add_argument("--verify-run", type=Path, default=RESULTS / "plane" / "harvest-verify")
parser.add_argument("--roles-run", type=Path, default=RESULTS / "plane" / "harvest-roles")
parser.add_argument("--v46", type=Path, default=RESULTS / "plane" / "validation-46")
parser.add_argument("--scratch", type=Path, required=True)
parser.add_argument("--scratch", type=Path)
parser.add_argument("--min-free-gb", type=float, default=1.2)
parser.add_argument("--counts", type=Path, required=True)
parser.add_argument("--counts", type=Path)
parser.add_argument("--store", help="OUSAST_MEMORY spec (default: the environment's)")
parser.add_argument("--limit", type=int, default=0, help="stop after this many targets (a test)")
parser.add_argument("--kinds", default="pairs,pin", help="which targets: pairs, pin (population and recipe pins)")
parser.add_argument("--results-root", type=Path, default=RESULTS)
parser.add_argument("--dry-run", action="store_true", help="read local stored examples and print engine changes by source; no writes")
args = parser.parse_args()
if args.dry_run:
spec = args.store or os.environ.get("OUSAST_MEMORY") or ""
if spec and not spec.startswith("file://"):
parser.error("--dry-run requires a local file:// store (no network)")
store = memory.open_store(spec)
if not isinstance(store, memory.FileStore):
parser.error("--dry-run requires a local file:// store (no network)")
print(json.dumps({"dry_run": True, "by_source": dry_run(store, args.results_root, args.cache)}, sort_keys=True))
return 0
if any(getattr(args, name) is None for name in ("labels", "units", "scratch", "counts")):
parser.error("--labels, --units, --scratch and --counts are required unless --dry-run is used")
labels = [r for r in jsonl(args.labels) if r["unit"] == "pin" and r["source"] not in CONDITIONAL_SOURCES and not r.get("conditional")]
units = json.loads(args.units.read_text(encoding="utf-8"))
@@ -275,6 +389,7 @@ def main() -> int:
skipped: Counter[str] = Counter()
absent: Counter[str] = Counter() # family -> labelled pair candidates their side does not declare
dropped = 0
reingested = 0 # rows the index remembered from an earlier build but a later one had dropped
args.scratch.mkdir(parents=True, exist_ok=True)
chosen = [(k, r) for k, r in sorted(targets.items()) if k[0] in args.kinds.split(",")]
for number, ((kind, ref, role_or_pin), rows) in enumerate(chosen):
@@ -313,7 +428,7 @@ def main() -> int:
continue
split = {r["split"] for r in rows} | {unit["source"]}
population = "independent-v1" if "population-v1" in split else "independent-v2"
engine_result = read_json(RESULTS / population / "scans" / f"{case_id}--{rows[0]['pin_role']}.json")
engine_result = resolve_scan(args.results_root / population / "scans", case_id, rows[0]["pin_role"])
stems = [(args.verify_run, n) for n in names] + [(args.v46, unit["ref"])]
owner = next(
(n for n in names if f"{n}-roles" in {e for e in (read_json(args.roles_run / "state.json") or {}).get("tasks", {})}),
@@ -357,15 +472,24 @@ def main() -> int:
for inst, info in rec["instruments"].items():
state = "missing" if info.get("version") == "missing:not-run" else info["state"]
coverage[f"{family}:{inst}"][state] += 1
store.ingest_rows("harvest", f"features:{owner}", out_rows)
ingested = store.ingest_rows("harvest", f"features:{owner}", out_rows)
built = {r["id"] for r in out_rows}
stored = store.rows(repo=memory.repo_key(repo), pin=pin, kind="features", where={"run": "harvest", "task": owner})
present = {r.row["id"] for r in stored}
restored = 0
if ingested.skipped and not built <= present:
# the index remembers these very rows from an earlier build, but a later build dropped them as stale
# (the side did not declare the function then); an index hit is not evidence the rows are there
store.put_rows(out_rows)
restored = len(built - present)
reingested += restored
stale = [r.row["id"] for r in stored if r.row["id"] not in built]
dropped += store.drop_rows(memory.repo_key(repo), pin, stale)
print(json.dumps({"target": owner, "rows": len(out_rows), "dropped": len(stale)}), flush=True)
print(json.dumps({"target": owner, "rows": len(out_rows), "dropped": len(stale), "restored": restored}), flush=True)
counts = {
"absent_on_side": dict(sorted(absent.items())),
"dropped_stale_rows": dropped,
"restored_after_drop": reingested,
"records": {f"{p}:{f}": n for (p, f), n in sorted(tally.items())},
"instrument_states_plane": {k: dict(sorted(v.items())) for k, v in sorted(coverage.items())},
"skipped_label_rows": dict(skipped),
+175
View File
@@ -0,0 +1,175 @@
"""Read-only, zero-model slice audit. Run: env -u OPENROUTER_API_KEY python benchmarks/learn/slice_leak_audit.py.
Only sources.toml itself is read, never its referenced corpora. Stored source IDs
are the reporting strata (the store merges advisory pair catalogs into `pairs`).
Ties get half credit. Both signed and best-direction scores are recorded; like
learn.leaks, a reversed label shortcut is still a leak. Missing classes/pairs
produce null metrics, not plausible zeros. Unsupported slices score zero for both
indicators and remain separately counted in coverage.
"""
from __future__ import annotations
import hashlib
import json
import os
from collections import Counter, defaultdict
from collections.abc import Mapping, Sequence
from pathlib import Path
from typing import Any
from openultrasast.config import load_dotenv
from openultrasast.learn.examples import load_examples
from openultrasast.learn.labels import load_sources
from openultrasast.learn.slice import function_slice
from openultrasast.plane.memory import MemoryStore, open_store
UNITS = Path("plane/experiments/exp-004-pooled-block-gate.units.jsonl")
OUTPUT = Path("benchmarks/measurements/2026-10-03-graph-slice-leak-audit/record.json")
THRESHOLD = 0.70
def complete_pairs(rows: Sequence[Mapping[str, Any]]) -> list[tuple[Mapping[str, Any], Mapping[str, Any]]]:
groups: dict[tuple[str, str], list[Mapping[str, Any]]] = defaultdict(list)
for row in rows:
if row.get("pair"):
groups[(row["family"], row["pair"])].append(row)
pairs = []
for members in groups.values():
if len(members) == 2 and {m["label"] for m in members} == {0, 1}:
v, f = sorted(members, key=lambda m: -m["label"])
if v["group"] != f["group"] or v["fold"] != f["fold"]:
raise ValueError("Pair crosses group or fold")
pairs.append((v, f))
elif len(members) != 1:
raise ValueError("Ambiguous pair")
return pairs
def metrics(rows: Sequence[Mapping[str, Any]], signal: str) -> dict[str, Any]:
pairs = complete_pairs(rows)
higher = sum(v[signal] > f[signal] for v, f in pairs)
lower = sum(v[signal] < f[signal] for v, f in pairs)
ties = len(pairs) - higher - lower
pair_score = (higher + 0.5 * ties) / len(pairs) if pairs else None
positive = [r[signal] for r in rows if r["label"] == 1]
negative = [r[signal] for r in rows if r["label"] == 0]
auc = (
sum((v > f) + 0.5 * (v == f) for v in positive for f in negative) / (len(positive) * len(negative))
if positive and negative
else None
)
best_pair = max(pair_score, 1 - pair_score) if pair_score is not None else None
best_auc = max(auc, 1 - auc) if auc is not None else None
return {
"rows": len(rows),
"pairs": len(pairs),
"positive": len(positive),
"negative": len(negative),
"higher_on_vulnerable": higher,
"higher_on_fixed": lower,
"ties": ties,
"within_pair_accuracy": pair_score,
"pooled_auc": auc,
"best_direction_within_pair_accuracy": best_pair,
"best_direction_pooled_auc": best_auc,
"flagged": any(v is not None and v >= THRESHOLD for v in (best_pair, best_auc)),
}
def audit(store: MemoryStore, units: Sequence[Mapping[str, Any]], *, expected_rows: int = 501, expected_pairs: int = 195) -> dict[str, Any]:
if len(units) != expected_rows or len({u["unit"] for u in units}) != expected_rows:
raise ValueError(f"Expected {expected_rows} unique units, read {len(units)}")
if len(complete_pairs(units)) != expected_pairs:
raise ValueError(f"Expected {expected_pairs} complete pairs")
examples = load_examples(store)
index = {e.id: e for e in examples}
if len(index) != len(examples):
raise ValueError("Duplicate example IDs in memory")
sources = {s.id for s in load_sources().sources}
coverage: dict[str, Counter[str]] = defaultdict(Counter)
rows = []
excerpt_bytes = 0
blobs: dict[str, str] = {}
for unit in units:
e = index[unit["unit"]] # missing input is fatal, never counted as no-flow
if (e.family, e.label, e.group) != (unit["family"], unit["label"], unit["group"]):
raise ValueError(f"Unit metadata differs from store: {e.id}")
if e.source not in sources:
raise ValueError(f"Source absent from sources.toml: {e.source}")
if e.excerpt_sha not in blobs:
# MemoryStore.get_blob(verify=True) DELETES corrupt blobs: do not use it.
data = store.get_blob("excerpts", e.excerpt_sha, verify=False)
if not data or hashlib.sha256(data).hexdigest() != e.excerpt_sha:
raise ValueError(f"Missing, empty or corrupt excerpt: {e.id}")
excerpt_bytes += len(data)
blobs[e.excerpt_sha] = data.decode("utf-8")
result = function_slice(blobs[e.excerpt_sha], e.language, e.family)
status = "not-analysable" if result is None else "flow" if result.steps else "no-flow"
coverage[e.family][status] += 1
rows.append(
{
**unit,
"source": e.source,
"status": status,
"has_flow": int(status == "flow"),
"slice_length": len(result.steps) if result else 0,
}
)
pairs = complete_pairs(rows)
asymmetry = Counter(
"both" if v["has_flow"] and f["has_flow"] else "vulnerable_only" if v["has_flow"] else "fixed_only" if f["has_flow"] else "neither"
for v, f in pairs
)
scopes = {"all": rows, **{f"source:{s}": [r for r in rows if r["source"] == s] for s in sorted({r["source"] for r in rows})}}
return {
"cost_usd": 0,
"threshold": THRESHOLD,
"unit_rows_read": len(units),
"example_rows_read": len(examples),
"matched_rows": len(rows),
"excerpts_read": len(blobs),
"excerpt_units": len(rows),
"excerpt_bytes": excerpt_bytes,
"complete_pairs": len(pairs),
"coverage": {f: {s: c[s] for s in ("flow", "no-flow", "not-analysable")} for f, c in sorted(coverage.items())},
"pair_asymmetry": {k: asymmetry[k] for k in ("vulnerable_only", "fixed_only", "both", "neither")},
"signals": {
signal: {scope: metrics(members, signal) for scope, members in scopes.items()} for signal in ("has_flow", "slice_length")
},
"source_strata": "Stored source IDs from sources.toml; advisory pair catalogs are merged into pairs by the example builder.",
"limitations": (
"Flat IR: no dominance/guards, textual branch order, potential parameter inputs, conservative call returns; no alias analysis."
),
"units": rows,
}
def main() -> None:
load_dotenv(Path(".env"))
os.environ.pop("OPENROUTER_API_KEY", None) # dotenv must not restore the removed key
raw = UNITS.read_bytes()
units = [json.loads(line) for line in raw.decode().splitlines() if line.strip()]
report = audit(open_store(), units)
report["units_bytes"] = len(raw)
report["units_sha256"] = hashlib.sha256(raw).hexdigest()
OUTPUT.parent.mkdir(parents=True, exist_ok=True)
OUTPUT.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n")
totals = Counter({s: sum(c[s] for c in report["coverage"].values()) for s in ("flow", "no-flow", "not-analysable")})
lines = [
f"Slice leak audit: $0; units bytes={len(raw)}",
f"Rows read: units={len(units)}, examples={report['example_rows_read']}, matched={report['matched_rows']}",
f"Excerpts read: {report['excerpts_read']}; units={report['excerpt_units']}; bytes={report['excerpt_bytes']}",
f"Coverage: {dict(totals)}",
f"Complete pairs: {report['complete_pairs']}",
f"Pair asymmetry: {report['pair_asymmetry']}",
f"Has flow: {report['signals']['has_flow']['all']}",
f"Slice length: {report['signals']['slice_length']['all']}",
f"Source strata: {', '.join(report['signals']['has_flow'])}; threshold={THRESHOLD}",
f"Report: {OUTPUT}",
]
print("\n".join(lines))
if __name__ == "__main__":
main()
@@ -1,7 +1,7 @@
{
"modules": 150,
"modules": 159,
"counts": {
"load_bearing": 145,
"load_bearing": 154,
"standalone_capability": 5
},
"orphaned": [],
@@ -229,6 +229,8 @@
"model.report",
"pairs",
"plane.tasks.alerts",
"push.quick",
"push.runner",
"rank",
"regress.candidate",
"reports",
@@ -338,7 +340,6 @@
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"cli",
"skills"
]
},
@@ -355,13 +356,16 @@
"learn.evaluate",
"learn.examples",
"learn.excerpt",
"learn.experiments",
"learn.features",
"learn.fill",
"learn.folds",
"learn.labels",
"learn.leaks",
"learn.program",
"learn.retrieve",
"learn.roles",
"learn.slice",
"plane.harvest",
"plane.tasks.features",
"plane.tasks.remember",
@@ -390,7 +394,8 @@
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"learn.cli",
"learn.evaluate"
"learn.evaluate",
"learn.experiments"
]
},
{
@@ -408,7 +413,9 @@
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"learn.cli",
"learn.compile"
"learn.compile",
"learn.experiments",
"learn.fill"
]
},
{
@@ -419,6 +426,8 @@
"learn.cli",
"learn.compile",
"learn.evaluate",
"learn.experiments",
"learn.fill",
"learn.folds",
"learn.program",
"learn.retrieve"
@@ -432,9 +441,19 @@
"learn.embeddings",
"learn.examples",
"learn.labels",
"learn.slice",
"plane.harvest"
]
},
{
"module": "learn.experiments",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"learn.cli",
"learn.fill"
]
},
{
"module": "learn.features",
"classification": "load_bearing",
@@ -444,6 +463,14 @@
"plane.tasks.features"
]
},
{
"module": "learn.fill",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"learn.cli"
]
},
{
"module": "learn.folds",
"classification": "load_bearing",
@@ -452,6 +479,7 @@
"learn.cli",
"learn.compile",
"learn.evaluate",
"learn.experiments",
"learn.program",
"learn.retrieve"
]
@@ -487,7 +515,9 @@
"importers": [
"learn.cli",
"learn.compile",
"learn.evaluate"
"learn.evaluate",
"learn.experiments",
"learn.fill"
]
},
{
@@ -516,6 +546,7 @@
"importers": [
"learn.compile",
"learn.evaluate",
"learn.examples",
"learn.features",
"learn.leaks",
"learn.program",
@@ -525,6 +556,14 @@
"plane.tasks.roles"
]
},
{
"module": "learn.slice",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"learn.program"
]
},
{
"module": "map_gate",
"classification": "load_bearing",
@@ -546,6 +585,7 @@
"pairs",
"plane.tasks.alerts",
"preprocess",
"push.quick",
"push.runner"
]
},
@@ -565,9 +605,12 @@
"cli",
"cpg.backend",
"cpg.session",
"findings",
"learn.cli",
"learn.features",
"learn.fill",
"learn.labels",
"learn.slice",
"model.candidates",
"model.config_value",
"model.dominance",
@@ -588,6 +631,7 @@
"push.cache",
"push.contracts",
"push.policy",
"push.quick",
"push.report",
"push.reuse",
"push.runner",
@@ -635,6 +679,7 @@
"push.cache",
"push.contracts",
"push.policy",
"push.quick",
"push.report",
"push.reuse",
"push.runner",
@@ -658,6 +703,7 @@
"importers": [
"cli",
"learn.cli",
"learn.fill",
"plane.budget",
"plane.generate",
"push.assistance",
@@ -702,7 +748,9 @@
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"model.scan"
"findings",
"model.scan",
"push.runner"
]
},
{
@@ -761,6 +809,7 @@
"classification": "load_bearing",
"reason": "model-grounded-detection Req 7.2-7.4: the security vocabularies the CPG queries are parameterised by",
"importers": [
"learn.slice",
"model.config_value",
"model.dominance",
"model.pipeline",
@@ -831,7 +880,11 @@
"learn.compile",
"learn.embeddings",
"learn.examples",
"learn.experiments",
"learn.fill",
"learn.program",
"plane.delivery",
"plane.doctor",
"plane.egress",
"plane.engine_alerts",
"plane.generate",
@@ -855,12 +908,22 @@
"learn.cli",
"learn.compile",
"learn.embeddings",
"learn.experiments",
"learn.fill",
"learn.program",
"plane.generate",
"plane.tasks.roles",
"plane.tasks.verify"
]
},
{
"module": "plane.delivery",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"plane.reconciler"
]
},
{
"module": "plane.doctor",
"classification": "load_bearing",
@@ -908,6 +971,7 @@
"reason": "ai-service-plane Req 1.1-1.4: ax-shaped Task/Workspace/Model and the Run kind, validated before anything runs",
"importers": [
"learn.cli",
"plane.delivery",
"plane.egress",
"plane.generate",
"plane.harvest",
@@ -920,7 +984,7 @@
{
"module": "plane.memory",
"classification": "load_bearing",
"reason": "harnessx-removal Req 6.1/6.4: the memory store (file, MinIO), ingest and fact reuse; `ousast plane remember`",
"reason": "harnessx-removal Req 6.1/6.4: the memory store (file, S3), ingest and fact reuse; `ousast plane remember`",
"importers": [
"cli",
"improve.memory",
@@ -928,12 +992,28 @@
"learn.compile",
"learn.embeddings",
"learn.examples",
"learn.experiments",
"learn.fill",
"learn.program",
"plane.delivery",
"plane.doctor",
"plane.generate",
"plane.reconciler",
"plane.tasks.loop",
"plane.tasks.remember"
]
},
{
"module": "plane.profile",
"classification": "load_bearing",
"reason": "python -m openultrasast.plane.profile",
"importers": [
"cli",
"plane.doctor",
"plane.generate",
"plane.reconciler"
]
},
{
"module": "plane.reconciler",
"classification": "load_bearing",
@@ -1076,6 +1156,7 @@
"improve.evolve",
"improve.validator",
"plane.tasks.loop",
"push.quick",
"scoring.project_score",
"semantic.overlay"
]
@@ -1102,7 +1183,6 @@
"hunter",
"hunter_tools",
"improve.evolve",
"index",
"learn.features",
"learn.labels",
"learn.roles",
@@ -1116,6 +1196,7 @@
"plane.tasks.alerts",
"plane.tasks.features",
"plane.tasks.repo_facts",
"push.quick",
"push.runner",
"rank",
"regress.candidate",
@@ -1170,6 +1251,7 @@
"importers": [
"cli",
"push.assistance",
"push.background",
"push.eligibility",
"push.policy",
"push.report",
@@ -1187,6 +1269,14 @@
"push.runner"
]
},
{
"module": "push.background",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"push.runner"
]
},
{
"module": "push.cache",
"classification": "load_bearing",
@@ -1218,6 +1308,23 @@
"push.runner"
]
},
{
"module": "push.explain",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"push.report",
"push.runner"
]
},
{
"module": "push.install",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"cli"
]
},
{
"module": "push.policy",
"classification": "load_bearing",
@@ -1229,12 +1336,21 @@
"push.vetoes"
]
},
{
"module": "push.quick",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"push.runner"
]
},
{
"module": "push.report",
"classification": "load_bearing",
"reason": "pre-push-safety-net Req 5.1/5.2/5.5/7.3: compact output and complete artifacts; runner integration pending",
"importers": [
"push.assistance",
"push.background",
"push.runner"
]
},
@@ -1299,7 +1415,8 @@
"improve.evolve",
"map_gate",
"pairs",
"plane.tasks.alerts"
"plane.tasks.alerts",
"push.quick"
]
},
{
@@ -1485,6 +1602,7 @@
"hunter_tools",
"learn.features",
"learn.roles",
"learn.slice",
"mapping",
"model.candidates",
"model.layout",
@@ -1541,6 +1659,7 @@
"hunter_tools",
"learn.features",
"learn.roles",
"learn.slice",
"mapping",
"model.candidates",
"model.layout",
@@ -1574,6 +1693,7 @@
"importers": [
"cli",
"hunter_tools",
"learn.slice",
"model.candidates",
"pairs",
"semantic.cst",
@@ -1707,6 +1827,7 @@
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"learn.slice",
"semantic.overlay"
]
},
@@ -1732,6 +1853,8 @@
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"cli",
"learn.cli",
"learn.fill",
"model.endpoint",
"plane.budget",
"plane.tasks.roles",
@@ -162,7 +162,7 @@ def per_rule(result: dict) -> dict:
rows[fid.split(":", 1)[0]][label] += 1
for fid in pin["at_known_site"]:
rows[fid.split(":", 1)[0]][label + "_at_site"] += 1
for name, fixture in result["fixtures"].items():
for _name, fixture in result["fixtures"].items():
for fid in fixture["true_positives"]:
rows[fid.split(":", 1)[0]]["fixture_tp"] += 1
for fid in fixture["false_positives"]:
@@ -0,0 +1,69 @@
{
"created": "2026-10-02",
"note": "COUNTS ONLY. Where the decision engine's positives-only repository groups come from, before any change. Memory = the static example rows of the RustFS store as of the 2026-10-02 slice-2 rebuild (904 examples); labels = the 2026-10-02 regeneration (6,589 rows, identical in counts to 2026-10-01-decision-engine-labels-advisory-fixes); feature records = the 924 static records of the store; harvest units = the merged units of the corpus and advisory harvests. Every (family, group) of the memory that holds positive and no negative example was traced back: negative label row present? feature record for it? excerpt of it? The reason is the first link that is missing. A second table accounts for the pair sides that have a label but no example at all (those groups are absent from the memory on BOTH sides, so they do not show as positives-only; they are where the fixed-side negatives the matcher owes come from). No candidate, path, function, commit or repository name is recorded here; the per-group detail stays in scratch.",
"memory_before": {
"examples": 904,
"profiles": {"static": 904, "plane": 0},
"by_family": {
"access_control": {"examples_pos": 31, "examples_neg": 18, "groups_pos": 30, "groups_neg": 18, "groups_both": 18, "groups_pos_only": 12},
"config_secrets": {"examples_pos": 18, "examples_neg": 13, "groups_pos": 5, "groups_neg": 2, "groups_both": 2, "groups_pos_only": 3},
"deserialization": {"examples_pos": 39, "examples_neg": 19, "groups_pos": 39, "groups_neg": 19, "groups_both": 19, "groups_pos_only": 20},
"injection": {"examples_pos": 75, "examples_neg": 65, "groups_pos": 38, "groups_neg": 28, "groups_both": 28, "groups_pos_only": 10},
"memory": {"examples_pos": 171, "examples_neg": 171, "groups_pos": 5, "groups_neg": 5, "groups_both": 5, "groups_pos_only": 0},
"output_encoding": {"examples_pos": 76, "examples_neg": 60, "groups_pos": 35, "groups_neg": 21, "groups_both": 21, "groups_pos_only": 14},
"path": {"examples_pos": 30, "examples_neg": 18, "groups_pos": 30, "groups_neg": 18, "groups_both": 18, "groups_pos_only": 12},
"unknown": {"examples_pos": 2, "examples_neg": 2, "groups_pos": 2, "groups_neg": 2, "groups_both": 2, "groups_pos_only": 0},
"untrusted_destination": {"examples_pos": 61, "examples_neg": 35, "groups_pos": 51, "groups_neg": 24, "groups_both": 24, "groups_pos_only": 27}
},
"groups_neg_only": 0
},
"positives_only_groups": {
"total": 98,
"by_family": {"access_control": 12, "config_secrets": 3, "deserialization": 20, "injection": 10, "output_encoding": 14, "path": 12, "untrusted_destination": 27},
"by_family_source_reason": [
{"family": "access_control", "source": "pairs", "slice": "vibe-py", "reason": "fixed side lacks the function: the label builder emitted no fixed-side negative (fixed side is another function of the same snapshot)", "groups": 12},
{"family": "config_secrets", "source": "pairs", "slice": "vibe-py", "reason": "fixed side lacks the function (same as above)", "groups": 3},
{"family": "deserialization", "source": "pairs", "slice": "vibe-py", "reason": "fixed side lacks the function (same as above)", "groups": 20},
{"family": "injection", "source": "pairs", "slice": "vibe-py", "reason": "fixed side lacks the function (same as above)", "groups": 8},
{"family": "injection", "source": "dev-php", "reason": "no fixed pin recorded: development recipes name a vulnerable checkout only", "groups": 2},
{"family": "output_encoding", "source": "pairs", "slice": "vibe-py", "reason": "fixed side lacks the function (same as above)", "groups": 14},
{"family": "path", "source": "pairs", "slice": "vibe-py", "reason": "fixed side lacks the function (same as above)", "groups": 11},
{"family": "path", "source": "dev-php", "reason": "no fixed pin recorded (development recipe)", "groups": 1},
{"family": "untrusted_destination", "source": "pairs", "slice": "vibe-py", "reason": "fixed side lacks the function (same as above)", "groups": 26},
{"family": "untrusted_destination", "source": "population-v2", "reason": "the fix did not change the labelled function: the same code at both pins is not a negative", "groups": 1}
],
"by_reason": {
"fixed side lacks the function (pairs, vibe-py: 108 labels, 80 seeded pointer pairs whose fixed side is a guard helper of the same commit + 28 Real-Vuln-Benchmark traps whose fixed side is a sibling handler)": 94,
"no fixed pin recorded (dev-php recipes)": 3,
"fix did not change the function (population-v2)": 1,
"matcher cannot resolve the function on the fixed pin": 0,
"excerpt missing": 0
},
"check": "all 108 fixed sides of the vibe-py pairs were scanned for a `def <labelled name>` line: 0 hold one (the name occurs only in calls); these are data gaps, counted, never invented",
"advisory_fixes_note": "no advisory-fixes group is positives-only: every one of the 97 pairs gave a fixed-side negative label; 7 of them are absent from the memory on both sides (below)"
},
"pair_sides_with_a_label_but_no_example": {
"total_sides": 32,
"pairs": 16,
"by_catalog_state": {
"advisory-fixes: feature record present, excerpt missing": {"sides": 14, "pairs": 7, "language": "ruby", "by_family": {"access_control": 2, "deserialization": 2, "output_encoding": 1, "path": 1, "untrusted_destination": 1}},
"corpus (vfc, vfc-js): no feature record (declares() false on both sides)": {"sides": 18, "pairs": 9, "by_family": {"injection": 5, "path": 2, "memory": 2}, "by_language": {"javascript": 6, "c_cpp": 1, "perl (catalog: unknown)": 2}}
},
"reasons": {
"ruby under the record's language": "build_features stores the closed feature-vocabulary language (schema.LANGUAGES; `other` for Ruby, Go, C#); the memory build hands that `other` to the excerpt matcher, which then runs the brace matcher: a Ruby `def` has no brace body, so 7 pairs (14 sides) get no excerpt; 4 further Ruby pairs (8 stored examples) matched a `{ |x| ... }` block inside the method instead of the method -- 2-line excerpts, wrong spans. The 11 Go pairs (22 examples) under `other` are brace-matched to the exact body (a correct span) and 9 C# pairs likewise.",
"javascript object-literal method": {"pairs": 3, "form": "`name: function (...) {` and `name: function anonymous(...) {`"},
"javascript property-assignment arrow": {"pairs": 2, "form": "`module.exports.name = (a, b) => {` and `const name = module.exports.name = async a => {` (a chained assignment)"},
"javascript prototype named function expression": {"pairs": 1, "form": "`X.prototype.name = function Name$withDollar (req, res) {`: `$` is not a `\\w` character for the declaration pattern or the brace matcher"},
"c parameter list longer than the brace matcher's lookahead": {"pairs": 1, "form": "`Type name(a,\\n b,\\n#ifndef X\\n c\\n#else\\n d\\n#endif\\n )\\n{`: the `{` is 9 lines down, the matcher looks 8"},
"perl sub": {"pairs": 2, "form": "`sub name {` (no parameter list; catalog language `unknown`)"}
},
"recoverable_by_tooling": {"fixed_side_negatives": 16, "positives": 16, "by_family": {"access_control": 2, "deserialization": 2, "output_encoding": 1, "path": 3, "untrusted_destination": 1, "injection": 5, "memory": 2}},
"note": "each recovered pair adds a paired group (positive and negative), not a negative to a positives-only group; the 98 positives-only groups are data gaps and stay as they are"
},
"label_builder_fixed_side_rule": "a fixed-side negative only where the fixed side declares the labelled function (or a deterministic rename at a distinct fix commit, fixed_side_moved = 0); unchanged",
"instruments": {
"memory_rows": "store.rows(kind=example|features) dumped once to scratch, 904 + 1,848 rows read",
"matcher_sweep": "function_bounds on both sides of every `pairs` pin label: vulnerable ok 431, fixed ok 323, fixed side named-but-undeclared 108 (vibe-py, python), both sides unmatched 9 pairs",
"language_check": "every pair record whose language is `other`: span under the record's language vs the label's: csharp same 18 sides, go different 18 (brace body vs to-next-declaration; stored spans kept), ruby none 14 (unstored) and different 8 (stored, wrong)"
}
}
@@ -0,0 +1,85 @@
{
"created": "2026-10-02",
"note": "COUNTS ONLY. Labels, feature records and the static memory rebuilt after the excerpt matcher learned the declaration forms recorded in 2026-10-02-decision-engine-negatives-gap. No model or embedding call: feature records reuse the recorded engine, verify and roles outputs of the harvest Runs (instruments that never ran stay `missing`); the new and changed excerpts have no vector yet (`ousast learn memory embed`, estimated below). The memory store is the RustFS bucket (versioned: prior objects stay). Candidate, path, function, commit and repository names stay local.",
"labels": {
"command": "ousast learn labels --root . --out <scratch> --snapshot <scratch>",
"rows": 6589,
"groups": 202,
"labels_sha256_prefix": "c17daeac522090d4",
"same_as": "the 2026-10-02 regeneration recorded in 2026-10-02-decision-engine-harvest-advisory (identical sha): the label builder's fixed-side rule already accepted every pair the matcher now resolves (its name-on-a-code-line fallback), so no label row changed",
"pair_fixed_side": {"absent_fixed_side_labels": 108, "fixed_side_moved": 0}
},
"features": {
"command": "benchmarks/learn/build_features.py --labels <the 36 label rows of the 9 corpus pairs> --units <merged units.json> (defaults: harvest-engine, harvest-verify, harvest-roles)",
"targets": 18,
"records": {"static:injection": 10, "static:memory": 4, "static:path": 4, "plane:injection": 10, "plane:memory": 4, "plane:path": 4},
"absent_on_side": {},
"instrument_states_plane_new_records": {
"injection:engine": {"failed": 2, "none": 2, "ran": 6}, "injection:roles": {"failed": 8, "none": 2}, "injection:verify": {"missing": 2, "ran": 8},
"memory:engine": {"failed": 2, "none": 2}, "memory:roles": {"none": 4}, "memory:verify": {"none": 4},
"path:engine": {"failed": 4}, "path:roles": {"failed": 4}, "path:verify": {"none": 4}
},
"instrument_fix": "the store's ingest index remembered these 36 rows from the 2026-10-01 build (before the declares() gate) and skipped them as a repeat, although the 17:25 build had dropped them as stale: the first rebuild stored nothing (store rows 1,848 -> 1,848, memory 919). build_features now checks the rows are present after an index hit and restores them (restored_after_drop = 36). An index hit is not evidence the rows are there.",
"store_rows": {"before": 1848, "after": 1884, "static": 942, "plane": 942},
"advisory_ruby_pairs": "their 14 records were already in the store (language `other`); nothing rebuilt, the excerpt was the gap"
},
"memory": {
"command": "ousast learn memory build --labels <labels> --units <merged units.json> --profile static --root . (exit 2: adjudications 9 of 43 anonymous functions, unchanged since 2026-10-01; rows kept)",
"examples": {"before": 904, "after": 937, "added": 33},
"by_source_after": {"adjudications": 34, "dev-php": 3, "pairs": 772, "population-v1": 85, "population-v2": 43},
"pairs_build_counts": {"before": {"recorded": 754, "examples": 740, "absent_fixed_side": 7, "no_excerpt": 7}, "after": {"recorded": 772, "examples": 772, "absent_fixed_side": 0, "no_excerpt": 0}},
"added_examples": {
"advisory-fixes ruby pairs (record present, excerpt recovered)": {"pairs": 7, "positives": 7, "negatives": 7, "by_family": {"access_control": 2, "deserialization": 2, "output_encoding": 1, "path": 1, "untrusted_destination": 1}},
"corpus pairs (record and excerpt recovered)": {"pairs": 9, "positives": 9, "negatives": 9, "by_family": {"injection": 5, "path": 2, "memory": 2}},
"population-v2 (a `this.name = function (` assignment now matched)": {"positives": 1, "negatives": 0, "family": "injection", "note": "its fix did not change the function, so it has no negative label: one more positives-only group"}
},
"changed_excerpts": {"count": 9, "why": "4 advisory Ruby pairs (8 examples) had a `{ |x| ... }` block inside the method as their excerpt; 1 population-v2 TypeScript method (1 example) had the `{}` default value of a parameter as its body; both corrected by the paren-aware matcher. Their old vectors no longer name the content."},
"unchanged_excerpts": 895,
"vectors": {"distinct_excerpts_after": 821, "distinct_excerpts_before": 788, "without_vector_at_most": 42, "embed_dry_run_all_usd": 0.0043, "note": "no embedding call was made; `ousast learn memory embed` for the 33 new + 9 changed excerpts is a fraction of a cent. Retrieval scores an example without a vector by its signal similarity (retrieve.py), so they take part now."},
"per_family_before_after": {
"access_control": {"before": {"pos": 31, "neg": 18, "groups_pos": 30, "groups_neg": 18, "pos_only": 12}, "after": {"pos": 33, "neg": 20, "groups_pos": 32, "groups_neg": 20, "pos_only": 12}},
"config_secrets": {"before": {"pos": 18, "neg": 13, "groups_pos": 5, "groups_neg": 2, "pos_only": 3}, "after": {"pos": 18, "neg": 13, "groups_pos": 5, "groups_neg": 2, "pos_only": 3}},
"deserialization": {"before": {"pos": 39, "neg": 19, "groups_pos": 39, "groups_neg": 19, "pos_only": 20}, "after": {"pos": 41, "neg": 21, "groups_pos": 41, "groups_neg": 21, "pos_only": 20}},
"injection": {"before": {"pos": 75, "neg": 65, "groups_pos": 38, "groups_neg": 28, "pos_only": 10}, "after": {"pos": 81, "neg": 70, "groups_pos": 44, "groups_neg": 33, "pos_only": 11}},
"memory": {"before": {"pos": 171, "neg": 171, "groups_pos": 5, "groups_neg": 5, "pos_only": 0}, "after": {"pos": 173, "neg": 173, "groups_pos": 5, "groups_neg": 5, "pos_only": 0}},
"output_encoding": {"before": {"pos": 76, "neg": 60, "groups_pos": 35, "groups_neg": 21, "pos_only": 14}, "after": {"pos": 77, "neg": 61, "groups_pos": 36, "groups_neg": 22, "pos_only": 14}},
"path": {"before": {"pos": 30, "neg": 18, "groups_pos": 30, "groups_neg": 18, "pos_only": 12}, "after": {"pos": 33, "neg": 21, "groups_pos": 33, "groups_neg": 21, "pos_only": 12}},
"unknown": {"before": {"pos": 2, "neg": 2, "groups_pos": 2, "groups_neg": 2, "pos_only": 0}, "after": {"pos": 2, "neg": 2, "groups_pos": 2, "groups_neg": 2, "pos_only": 0}},
"untrusted_destination": {"before": {"pos": 61, "neg": 35, "groups_pos": 51, "groups_neg": 24, "pos_only": 27}, "after": {"pos": 62, "neg": 36, "groups_pos": 52, "groups_neg": 25, "pos_only": 27}}
},
"positives_only_groups": {
"before": 98,
"after": 99,
"by_reason_after": {
"fixed side lacks the function (pairs, vibe-py: another function of the same snapshot; 108 labels)": 94,
"no fixed pin recorded (dev-php recipes)": 3,
"fix did not change the function (population-v2)": 2,
"matcher cannot resolve the function on the fixed pin": 0,
"excerpt missing": 0
},
"reading": "every recovered pair adds a paired group (both sides); the positives-only groups are data gaps that the rule `a fixed-side negative only where the fixed side declares the function` keeps counted, never invented. The one new positives-only group is a population positive that was invisible before (no excerpt) and whose fix did not change its function."
}
},
"leak_audit": {
"command": "ousast learn audit-leaks --profile {static,plane} --units <merged units.json> (threshold 0.20, min 10 pairs)",
"pairs": {"before": {"all": 376, "injection": 43, "memory": 171, "path": 19}, "after": {"all": 385, "injection": 48, "memory": 173, "path": 21}, "unchanged": ["access_control 20", "config_secrets 12", "deserialization 21", "output_encoding 61", "unknown 2", "untrusted_destination 27"]},
"rows_read": {"before": 1848, "after": 1884},
"skipped": {"one side only": 172},
"static": {"new_flags": [], "gone_flags": [], "changed": {"facts.function_lines/path": [0.7368, 0.6667], "facts.function_lines/injection": [0.2558, 0.2292]}, "flagged": 13},
"plane": {
"new_flags": [
{"signal": "agree.final", "scope": "injection", "separation": 0.2292, "pairs": 48, "correct": 15, "wrong": 4, "differ": 19},
{"signal": "verify.votes", "scope": "injection", "separation": 0.2292, "pairs": 48, "correct": 15, "wrong": 4, "differ": 19, "direction": "higher on vulnerable"}
],
"gone_flags": [],
"changed": {"facts.function_lines/path": [0.7368, 0.6667], "facts.function_lines/injection": [0.2558, 0.2292]},
"flagged": {"before": 25, "after": 27},
"reading": "the five injection pairs the matcher recovered carry verify verdicts (8 of 10 sides ran) that separate their sides; agree.final and verify.votes were already flagged for untrusted_destination and config_secrets and are the verifier doing its job, not a label leak. The v1 input profile is unchanged (it withholds engine, facts.function_lines, verify.flag_b and ms.operation); agree.final and verify.votes stay in, reported here for the profile's owner."
}
},
"instruments": {
"store_dumps": "store.rows(kind=example|features) read once before and once after (904/1,848 then 937/1,884 rows)",
"matcher_sweep_after": "function_bounds on both sides of every pairs pin label: vulnerable ok 440 (was 431), fixed ok 332 (was 323), the 108 vibe-py fixed sides still undeclared",
"dry_build_check": "a dry build_examples over the stored records before any write: 15 new rows, 9 changed shas, 0 gone -- as expected; the live build then matched it plus the 18 restored corpus records"
}
}
@@ -0,0 +1,377 @@
{
"created": "2026-10-02",
"decision": "reject",
"instruments": {
"distinct_scores_by_arm": {
"A": 72,
"B": 57
},
"outcomes_by_arm_replicate": {
"A:r0": 160,
"A:r1": 160,
"B:r0": 160,
"B:r1": 160
},
"parse_failed_by_arm": {
"A": 0,
"B": 0
}
},
"looks": [
{
"boundary": 2.797,
"crossed": false,
"estimate": {
"a": 0.8611111111111112,
"b": 0.8888888888888888,
"ci": [
-0.08333333333333326,
0.1325301204819277
],
"diff": 0.02777777777777768,
"groups": 28,
"metric": "canary_agreement",
"resamples": 10000,
"se": 0.05449719877526111,
"units": 72
},
"fraction": 0.5,
"groups": 28,
"z": 0.5097101943226363
},
{
"boundary": 1.977,
"crossed": false,
"estimate": {
"a": 0.875,
"b": 0.90625,
"ci": [
-0.03448275862068961,
0.0972972972972973
],
"diff": 0.03125,
"groups": 56,
"metric": "canary_agreement",
"resamples": 10000,
"se": 0.0337315848833629,
"units": 160
},
"fraction": 1.0,
"groups": 56,
"z": 0.9264314175588332
}
],
"note": "COUNTS ONLY. Experiment exp-002 (learned-decision-engine task 7.4): arm A = the slice-2 compiled programs as they are (k = 5, sample 1 at temperature 0, samples 2..5 at 0.7, one retrieved example set); arm B = the same programs with ProgramSpec.sampling = retrieval_ensemble (k = 5 at temperature 0, each sample over a disjoint retrieved example set). Paired on the frozen units (every unit in both arms, two replicates per arm under different response-cache salts; replicate 1 is the canary), seeded repository order and a seeded per-unit coin for which arm goes first, the incumbent folds' seed on the 937-example static memory, 788 embedding vectors. Intervals: repository-cluster paired bootstrap 95% (10,000 resamples); exact McNemar on discordant units for binary metrics; two looks (O'Brien-Fleming 2.797 / 1.977). The overall table is the runner's result.json; the per-family table here uses the same estimator on each family's pairs. Driver: `ousast learn experiment run|analyse` (learn/experiments.py). No candidate, path, function or repository name.",
"overall": {
"advisory_precision": {
"A": 0.8979591836734694,
"B": 0.9069767441860465,
"ci95": [
-0.02020202020202022,
0.05081826012058566
],
"ci_vs_zero": "spans",
"diff_B_minus_A": 0.009017560512577072,
"mcnemar": null,
"target": null
},
"advisory_recall": {
"A": 0.5,
"B": 0.4431818181818182,
"ci95": [
-0.1235955056179775,
0.01010101010101011
],
"ci_vs_zero": "spans",
"diff_B_minus_A": -0.05681818181818182,
"mcnemar": {
"b": 7,
"c": 2,
"concordant": 79,
"p_value": 0.1796875
},
"target": null
},
"auc": {
"A": 0.8012941919191919,
"B": 0.7802241161616161,
"ci95": [
-0.04139254385964919,
-0.0034279184917158645
],
"ci_vs_zero": "against",
"diff_B_minus_A": -0.0210700757575758,
"mcnemar": null,
"target": null
},
"canary_agreement": {
"A": 0.875,
"B": 0.90625,
"ci95": [
-0.03448275862068961,
0.0972972972972973
],
"ci_vs_zero": "spans",
"diff_B_minus_A": 0.03125,
"mcnemar": {
"b": 10,
"c": 15,
"concordant": 135,
"p_value": 0.42435622215270996
},
"target": {
"b_meets": true,
"level": 0.9
}
},
"usd_per_candidate": {
"A": 0.002131543750000001,
"B": 0.005102303125,
"ci95": [
0.002724568181818181,
0.00317872222222222
],
"ci_vs_zero": "against",
"diff_B_minus_A": 0.002970759374999999,
"mcnemar": null,
"target": null
},
"within_pair_auc": {
"A": 0.8148148148148148,
"B": 0.7407407407407407,
"ci95": [
-0.12857142857142867,
-0.015151515151515138
],
"ci_vs_zero": "against",
"diff_B_minus_A": -0.07407407407407407,
"mcnemar": null,
"target": null
}
},
"per_family": {
"access_control": {
"groups": 24,
"metrics": {
"advisory_precision": {
"A": 1.0,
"B": 1.0,
"ci95": [
0.0,
0.0
],
"ci_vs_zero": "spans",
"diff_B_minus_A": 0.0,
"mcnemar": null
},
"advisory_recall": {
"A": 0.72,
"B": 0.68,
"ci95": [
-0.16666666666666674,
0.08333333333333337
],
"ci_vs_zero": "spans",
"diff_B_minus_A": -0.039999999999999925,
"mcnemar": {
"b": 2,
"c": 1,
"concordant": 22,
"p_value": 1.0
}
},
"auc": {
"A": 0.94625,
"B": 0.95125,
"ci95": [
-0.023157894736842155,
0.02941176470588236
],
"ci_vs_zero": "spans",
"diff_B_minus_A": 0.0050000000000000044,
"mcnemar": null
},
"canary_agreement": {
"A": 0.926829268292683,
"B": 0.8780487804878049,
"ci95": [
-0.18421052631578938,
0.08108108108108103
],
"ci_vs_zero": "spans",
"diff_B_minus_A": -0.04878048780487809,
"mcnemar": {
"b": 5,
"c": 3,
"concordant": 33,
"p_value": 0.7265625
}
},
"usd_per_candidate": {
"A": 0.0019149512195121951,
"B": 0.0043439878048780484,
"ci95": [
0.0021030595238095253,
0.00275970731707317
],
"ci_vs_zero": "against",
"diff_B_minus_A": 0.0024290365853658533,
"mcnemar": null
},
"within_pair_auc": {
"A": 0.9375,
"B": 0.875,
"ci95": [
-0.125,
0.0
],
"ci_vs_zero": "spans",
"diff_B_minus_A": -0.0625,
"mcnemar": null
}
},
"paired_units": 41
},
"injection": {
"groups": 34,
"metrics": {
"advisory_precision": {
"A": 0.8387096774193549,
"B": 0.8461538461538461,
"ci95": [
-0.043269230769230727,
0.06763285024154597
],
"ci_vs_zero": "spans",
"diff_B_minus_A": 0.007444168734491274,
"mcnemar": null
},
"advisory_recall": {
"A": 0.4126984126984127,
"B": 0.3492063492063492,
"ci95": [
-0.140625,
0.0
],
"ci_vs_zero": "spans",
"diff_B_minus_A": -0.06349206349206349,
"mcnemar": {
"b": 5,
"c": 1,
"concordant": 57,
"p_value": 0.21875
}
},
"auc": {
"A": 0.7475907029478458,
"B": 0.7140022675736961,
"ci95": [
-0.061038961038961004,
-0.011363636363636465
],
"ci_vs_zero": "against",
"diff_B_minus_A": -0.03358843537414968,
"mcnemar": null
},
"canary_agreement": {
"A": 0.8571428571428571,
"B": 0.9159663865546218,
"ci95": [
-0.016129032258064502,
0.1322314049586777
],
"ci_vs_zero": "spans",
"diff_B_minus_A": 0.05882352941176472,
"mcnemar": {
"b": 5,
"c": 12,
"concordant": 102,
"p_value": 0.143463134765625
}
},
"usd_per_candidate": {
"A": 0.002206168067226891,
"B": 0.005363571428571428,
"ci95": [
0.0028887247706422007,
0.0033776267605633826
],
"ci_vs_zero": "against",
"diff_B_minus_A": 0.003157403361344537,
"mcnemar": null
},
"within_pair_auc": {
"A": 0.7631578947368421,
"B": 0.6842105263157895,
"ci95": [
-0.15384615384615385,
0.0
],
"ci_vs_zero": "spans",
"diff_B_minus_A": -0.07894736842105265,
"mcnemar": null
}
},
"paired_units": 119
}
},
"reason": "the CI of within_pair_auc excludes 0 against B",
"registration": "benchmarks/experiments/exp-002-retrieval-ensemble/registration.json",
"result": "benchmarks/experiments/exp-002-retrieval-ensemble/result.json",
"run": {
"decided": {
"A": 320,
"B": 320
},
"folds_digest": "a1f78e141a572a4adcc9cd57eb8a784c440c177f0b60d5ef5bed9e94c8753c5f",
"reason": "",
"skipped": {
"A": 0,
"B": 0
},
"status": "done",
"stopped_at_look": null,
"units": 160
},
"spend": {
"caller": {
"A": {
"cache_digest": "39ba875897ccbc9dee39b07029d26e1fee84392d230776b63bb6f8aa2189d7b9",
"client_calls": 1206,
"replayed": 395,
"usd": 0.682094
},
"B": {
"cache_digest": "5f6187e9445f180fe234b2f48a984eb8fbecf0000552a4acc0a412780c6e628a",
"client_calls": 1350,
"replayed": 250,
"usd": 1.632737
}
},
"deepseek_balance_usd": {
"after": 16.04,
"before": 16.72,
"metered_delta": 0.68
},
"note": "priced usage prices every response a decision used, a replayed one the same as a paid one (Caller.usd); client_calls are the paid calls and replayed the response-cache hits (arm A: 33 units whose slice-2 sample-0 request was still cached, plus identical prompts across identical-twin candidates and retries; arm B: twins and retries only). The provider balance is read at the run's end and lags; the metered spend is bounded by the arm meters (A $1, B $3).",
"priced_usage_usd_by_arm": {
"A": 0.6821,
"B": 1.6327
},
"priced_usage_usd_by_arm_replicate": {
"A:r0": 0.4299,
"A:r1": 0.2522,
"B:r0": 1.3797,
"B:r1": 0.253
}
},
"units": {
"by_label": {
"0": 72,
"1": 88
},
"groups": 56,
"outcomes": 640,
"paired": 160
}
}
@@ -0,0 +1,5 @@
[ok] profile k3s: exec=remote kube_context=ousast-k3s registry=ghcr.io/norandom images=/home/mc/Source/OpenUltraSAST/.claude/worktrees/agent-a7d4724e0c0c407a8/ops/k8s/profiles/images.json router_url=(tunnel through the kube context) ax_server=(tunnel through the kube context) memory=s3://sast-memory atespace=default snapshots_bucket=s3://ax-snapshots image_pull_secret=ghcr-pull
[FAIL] cluster ousast-k3s: context ousast-k3s unreachable (error: context "ousast-k3s" does not exist); docs/deployment.md lists the prerequisites
[FAIL] runner image in ghcr.io/norandom: images file /home/mc/Source/OpenUltraSAST/.claude/worktrees/agent-a7d4724e0c0c407a8/ops/k8s/profiles/images.json: No such file or directory; docs/deployment.md lists the prerequisites
[ok] memory store s3://sast-memory: s3://sast-memory opens (an S3 bucket is verified)
exit=1 seconds=1.1
@@ -0,0 +1,7 @@
[ok] profile kind: exec=local kube_context=kind-ousast registry=localhost:5001 images=/home/mc/Source/OpenUltraSAST/.claude/worktrees/agent-a7d4724e0c0c407a8/ops/k8s/profiles/kind-images.json router_url=(tunnel through the kube context) ax_server=(tunnel through the kube context) memory=s3://sast-memory atespace=default snapshots_bucket=s3://ax-snapshots image_pull_secret=(none)
[ok] cluster kind-ousast: reachable
[ok] agent substrate (ate-system): 9 pods ready
[ok] ax controller (ax-system): 4 pods ready
[ok] runner image in localhost:5001: ousast-runner@sha256:95edc7ed5621... present in localhost:5001
[ok] memory store s3://sast-memory: s3://sast-memory opens (an S3 bucket is verified)
exit=0 seconds=1.4
@@ -0,0 +1,19 @@
F.. [100%]
=================================== FAILURES ===================================
________ test_no_host_literal_outside_the_profile_and_the_kind_bring_up ________
def test_no_host_literal_outside_the_profile_and_the_kind_bring_up() -> None:
kind = load_profile("kind", environ={})
found = offenders(_files(), kind.registry)
> assert found == [], "host-only literals (Req 1.3):\n" + "\n".join(found)
E AssertionError: host-only literals (Req 1.3):
E src/openultrasast/plane/workspaces.py:37: [kind context] CONTEXT = "kind-ousast" # reinserted for the record 01 demonstration
E assert ['src/openult...emonstration'] == []
E
E Left contains one more item: 'src/openultrasast/plane/workspaces.py:37: [kind context] CONTEXT = "kind-ousast" # reinserted for the record 01 demonstration'
E Use -v to get more diff
tests/test_plane_literals.py:81: AssertionError
=========================== short test summary info ============================
FAILED tests/test_plane_literals.py::test_no_host_literal_outside_the_profile_and_the_kind_bring_up
1 failed, 2 passed in 0.26s
@@ -0,0 +1,2 @@
... [100%]
3 passed in 0.23s
@@ -0,0 +1,16 @@
apiVersion: ax.io/v1alpha1
kind: Workspace
metadata: {name: ousast-probe-big-ws, atespace: default}
spec:
files:
- {path: "hello.py", content: "def hello():\n return greet()\n"}
---
apiVersion: ax.io/v1alpha1
kind: Task
metadata: {name: ousast-probe-big, atespace: default}
spec:
image: "localhost:5001/ousast-runner@sha256:412db0a1504af8f1b23a395af8d1dcbd152c1bb6d16594f2b733067163a31884"
command: ["repo-facts"]
env: [{name: OUSAST_OUTPUT_DIR, value: "/workspace/.ousast-out/probe"}]
resources: {requests: {cpu: "250m", memory: "2560Mi"}, limits: {cpu: "1", memory: "3Gi"}}
workspaces: [{name: ousast-probe-big-ws, path: "/workspace"}]
@@ -0,0 +1,16 @@
apiVersion: ax.io/v1alpha1
kind: Workspace
metadata: {name: ousast-probe-small-ws, atespace: default}
spec:
files:
- {path: "hello.py", content: "def hello():\n return greet()\n"}
---
apiVersion: ax.io/v1alpha1
kind: Task
metadata: {name: ousast-probe-small, atespace: default}
spec:
image: "localhost:5001/ousast-runner@sha256:412db0a1504af8f1b23a395af8d1dcbd152c1bb6d16594f2b733067163a31884"
command: ["repo-facts"]
env: [{name: OUSAST_OUTPUT_DIR, value: "/workspace/.ousast-out/probe"}]
resources: {requests: {cpu: "250m", memory: "256Mi"}, limits: {cpu: "1", memory: "512Mi"}}
workspaces: [{name: ousast-probe-small-ws, path: "/workspace"}]

Some files were not shown because too many files have changed in this diff Show More