Author SHA1 Message Date
Marius CiepluchandClaude Opus 4.8 485aacca02 wip(archive): fix-mechanisms arm — parked WIP preserved before worktree prune
Uncommitted work from the fix-mechanisms worktree (mechanisms.py + cst/extra/cli
edits), committed so the worktree can be removed without loss. Branch is 0 commits
on top of a 172-behind base; re-integrate via rebase if resumed.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-12 15:47:24 +02:00
Marius CiepluchandClaude Opus 5.5 ee422b63e0 spec(learned-decision-engine): fix-mechanisms plan approved -- discharge facts first, >=2 proposing repos, per language
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 17:25:35 +02:00
Marius CiepluchandClaude Opus 5.5 02b2cbf99c engine: retry a pin once when its AX sandbox dies, then hand it to the VM lane
Two engine actors died at gVisor level during a 12-pin run (AX phase Failed, no
result). Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 17:14:20 +02:00
Marius CiepluchandClaude Opus 5.5 eaf436d3db merge(engine): one resident Joern session per pin; ActiveProcessorCount=2 on the cluster
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 16:50:21 +02:00
Marius CiepluchandClaude Opus 5.5 187e3c14c5 engine: one resident Joern session per pin (frontend + one query JVM instead of four), ActiveProcessorCount=2 on the cluster
Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 16:41:22 +02:00
Marius CiepluchandClaude Opus 5.5 b715ff7d0e engine: stage presigned objects as .dat; task entry reports the failing step and HTTP status
Through Cloudflare, signed requests for keys ending .tar/.gz/.zip/.bin reached
RustFS altered and failed with 403 (cf-cache-status BYPASS); the same signed
request straight to RustFS on the LAN returned 200. A Cloudflare cache-bypass
rule on files.because-security.com fixes it (all extensions 200, DYNAMIC); the
engine executors keep .dat as a safeguard, the plane keeps output.tar.
Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 16:18:01 +02:00
Marius CiepluchandClaude Opus 5.5 035710493c merge(engine): AX-task executor for the kube-ax cluster
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 15:30:26 +02:00
Marius CiepluchandClaude Opus 5.5 abca40b072 engine: AX-task executor for kube-ax (engine-task image FROM ax-task-runner v0.3.1, presigned HTTPS I/O, ousast-engine- names, 60 s egress-ready retry, 1.7 GB cluster heap, VM OOM fallback)
Implemented with Codex. Live smoke pending the operator egress watcher.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 15:22:44 +02:00
Marius CiepluchandClaude Opus 5.5 afff080874 merge(engine): Kubernetes engine workers (Deployment from the image, S3 queue); leak-audit records carry digests, not repository names
The Joern leak-audit record named spent population v1/v2 repositories, which
the reserved-repository guard forbids outside benchmarks/independent/ (no v3
repository was involved: 0 matches). Records now carry counts and SHA-256
digests. With Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 14:39:36 +02:00
Marius CiepluchandClaude Opus 5.5 e174c06ee7 engine: worker Deployment from the image (S3 queue with leases, dispatcher --executor queue, engine-image workflow, scoped RustFS policy); merge main
Implemented with Codex. Cluster deploy pending the k3s kubeconfig and the scoped account.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 14:23:04 +02:00
Marius CiepluchandClaude Opus 5.5 8d6921d2df spec(learned-decision-engine): fix-mechanisms plan for review -- learn discharge facts from 710 fix diffs, admitted by held-out pair separation
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 14:11:41 +02:00
Marius CiepluchandClaude Opus 5.5 298bd17780 learn: Joern-slice leak audit on the completed trace run -- no leak; paths differ between pair sides in 2 of 105 both-asked pairs
Coverage 72 path, 231 asked-nothing, 9 failed, 189 unsupported. The engine
almost never distinguishes a vulnerable function from its fix, so a Joern-slice
prompt experiment is not funded; the lever is the engine's knowledge of fix
mechanisms.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 14:08:26 +02:00
Marius CiepluchandClaude Opus 5.5 6a2138fbef merge(ops): lean kind profile -- observability off by default on the server VM
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 12:13:32 +02:00
Marius CiepluchandClaude Opus 5.5 35ded61c85 merge(learn): harvest 3 -- 251 real fix pairs (advisory-fixes-3); complete pairs in memory 459 -> 710
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 12:06:16 +02:00
Marius CiepluchandClaude Opus 5.5 0d78d33fd2 ops(kind): skip Substrate's observability stack by default (930 MiB of 2.3 GB); opt in with OUSAST_KIND_OBSERVABILITY=1
A lean overlay omits otel-collector, prometheus and jaeger and disables the
exporters; on an existing cluster up.sh scales them to 0. Verified live: render
has no otel-system resources, up.sh idempotent (54 s second run), smoke Run
done, kind node 2.66 -> 1.82 GiB. Docs say server VM. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 12:03:19 +02:00
Marius CiepluchandClaude Opus 5.5 cbc252553a learn: 251 more advisory-fix pairs whose function survives the fix (advisory-fixes-3)
Batch 3 of real (vulnerable function, fixed function) pointer pairs, mined at
zero model cost from reviewed GHSA advisories (pip, npm, maven, composer,
published since 2023, newest first). Same rules as batch 2, widened three
ways: a merged PR's single commit when the advisory links no commit, commits
up to 6 files / 240 lines, and two-family advisories with the family read
from the diff. Every candidate's function diff was read: 412 reviewed, 255
kept, 3 one-per-repository, 2 withheld by the reserved-population guard
(bisected with output discarded). Permissive licences only; no repository
named elsewhere under benchmarks/; no WordPress plugin.

Kept 251: injection 56, untrusted_destination 26, path 71, output_encoding
69, access_control 20, deserialization 9; python 83, typescript 48, php 46,
java 37, javascript 37. Only 12 post-cutoff: batch 2 had used up the
post-2026-06-01 pool. Memory 1133 -> 1635 static examples; complete pairs in
the whole memory 459 -> 710. Embeddings: 15 calls, $0.0030. No chat-model
call, no Docker, no Joern. Record:
benchmarks/measurements/2026-10-04-decision-engine-harvest-3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 11:57:34 +02:00
Marius Ciepluch e555ba68dc Merge branch 'joern-leak-audit' 2026-10-04 10:53:23 +02:00
Marius CiepluchandClaude Opus 5.5 231dca1f04 merge: taint-query null-guard fix and trace-enabled engine runner
The query fix stops a class-level call from aborting the whole taint batch on
real PHP code; gate, map_gate and pair_gate outputs are unchanged. Trace parsing
lives with the runner (benchmarks/learn/engine_trace_parse.py).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 10:53:23 +02:00
Marius CiepluchandClaude Opus 5.5 2e6ba93aee learn: Kubernetes executor for the engine trace runner (Jobs, presigned source/result via the S3 store, mixed lanes, --k8s-check)
Implemented with Codex; live cluster check pending.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 10:35:59 +02:00
Marius CiepluchandClaude Opus 5.5 431ab1e738 learn: engine_trace materialises catalog pairs without commits (dedented parseable excerpts, in-function scope), protects labelled files in exports, scopes large repos, records frontend diagnostics, retries the jssrc2cpg engine bug, writes progress.json
Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 08:19:46 +02:00
Marius CiepluchandClaude Opus 5.5 94ec4cef16 learn: Joern-slice leak audit (has-path, steps, sanitizer, guard, sink, asked; both-asked pairs; per-corpus AUC)
Implemented with Codex. The record is written after the trace re-run completes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 23:03:52 +02:00
Marius CiepluchandClaude Opus 5.5 e492ebd16e fix(cpg): null-guard method access in the query scripts -- a class-level call aborted the whole taint batch
On real PHP source (CPS-IT/mailqueue) a call outside any method has a null
.method; inScope/nestedInLabeled dereferenced it (NullPointerException) and the
script exited 1, losing every taint answer for the repository. 16 sites in
taint/dominance/config/vocabulary.sc now treat a null method as out of scope;
non-null behaviour is unchanged (same outputs on the repro graph for the other
scripts). Static regression test added. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 22:53:28 +02:00
Marius CiepluchandClaude Opus 5.5 41f56170bb learn: engine_trace worker renders request params through the production _render; records query stderr
The worker sent tuple params as JSON arrays; taint.sc expects comma-joined
strings (ujson InvalidData, exit 1). Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 22:41:54 +02:00
Marius CiepluchandClaude Opus 5.5 fe2a424018 learn: engine_trace fetches real source at the pinned commit, 2.5 GB JVM heap, one CPG per language
Root causes of the 50 failed pins in trace run 1: advisory pairs were fed as
excerpts (invalid Java/JS fragments), the JVM defaulted to ~768 MB heap in a
3 GB container, and mixed-language pins were refused. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 22:36:23 +02:00
Marius CiepluchandClaude Opus 5.5 36a3355b81 learn: engine_trace guard judges frontends by their CPG, query steps by time and question markers
php2cpg legitimately finishes in ~2 s on small files. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 19:22:37 +02:00
Marius CiepluchandClaude Opus 5.5 34d00391f1 learn: engine_trace container worker imports only the engine; dry-run writes nothing; resume keyed on selection
The 2026-09-14 engine image has no PyYAML or boto3; the worker no longer imports
learn.examples. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 19:19:22 +02:00
Marius CiepluchandClaude Opus 5.5 19c936343b learn: trace-enabled Joern run over labelled units (opt-in trace in taint.sc, trace parsing, resumable runner)
Implemented with Codex. Default scan queries do not request a trace.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 19:10:47 +02:00
Marius CiepluchandClaude Opus 5.5 20777d8863 chore(audit): module audit gains learn.slice
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 17:04:33 +02:00
Marius Ciepluch 77e47ac7d3 Merge branch 'graph-slice' 2026-10-03 16:58:37 +02:00
Marius CiepluchandClaude Opus 5.5 094f6d226c learn: exp-005 graph slice -- inconclusive (within-pair AUC 0.762 -> 0.746, CI [-0.063, 0.032]); unsure rate 0.29 -> 0.24; fill \$1.59
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 16:58:37 +02:00
Marius CiepluchandClaude Opus 5.5 0acbc7ec41 learn: exp-005 graph slice -- slice in the prompt (opt-in), program-arm fill, manifest and units
Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 15:06:08 +02:00
Marius CiepluchandClaude Opus 5.5 1b2c7993a7 learn: in-function def-use slice from the tree-sitter layer + leak audit (\$0)
slice.py renders source -> step -> sink lines with fixed wording for flow,
no-flow and not-analysable; the prompt option is default-off (byte-identical
default prompt). semantic IR: JS/C declarations count as assignments, augmented
assignment keeps its target (gate, map_gate, pair_gate outputs unchanged).
Audit over exp-004's 501 units: 59 flow, 161 no-flow, 281 not analysable;
has-flow within-pair accuracy 0.52, pooled AUC 0.55 -- no leak. Fixes a
numbering bug that rejected every excerpt with a blank line. With Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 14:56:52 +02:00
Marius CiepluchandClaude Opus 5.5 463fe1503a learn: build_features reads v2 _a/_b scans and TypeScript repository-scan witnesses; --dry-run
Population v2 scans are named <case>--<side>_a/_b.json; the script looked only
for <case>--<side>.json and recorded those examples as engine not-run. The two
repeats combine symmetrically for both sides. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 14:41:55 +02:00
Marius CiepluchandClaude Opus 5.5 2836b78819 merge(learn): exp-004 -- inconclusive, pooled top-30 precision 0.90; experiment fill command; registration fix
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 14:29:22 +02:00
Marius CiepluchandClaude Opus 5.5 6d9850578e learn: dry-run estimate from measured cached-response cost; docs: exp-004 result; tidy instrument check
The estimator priced all input at the full rate (\$5.58 estimated vs \$1.24
spent); it now uses the family mean billed cost of cached responses when usage
is recorded. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 14:23:36 +02:00
Marius CiepluchandClaude Opus 5.5 b50f7afccb learn: exp-004 run -- inconclusive, no pooled t_block; fill \$1.24 of a \$2 ceiling, replay \$0; instrument check recorded
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 14:18:17 +02:00
Marius CiepluchandClaude Opus 5.5 90a2642019 learn: registered() returns the frozen registration, not the first row
A pending-then-frozen registration left two rows and rows[0] could be the stale
pending one, so fill and run refused a properly frozen experiment. Implemented
with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 12:42:39 +02:00
Marius CiepluchandClaude Opus 5.5 8aeca972c0 learn: freeze exp-004 units (501 units, 390 paired, 220 groups) after the negatives harvest; audit gains learn.fill
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 12:36:02 +02:00
Marius CiepluchandClaude Opus 5.5 8d14b0c13f learn: experiment fill -- fill a rule experiment's response cache with a hard ceiling, dry-run estimate
Implemented with Codex; review fix: the bounded client keeps the configured
thinking setting so a fill answers as the run would.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 12:35:27 +02:00
Marius CiepluchandClaude Opus 5.5 212a391872 merge(learn): cache fill and exp-004 registration -- pooled block gate with family floor, Req 6.4
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 09:53:10 +02:00
Marius CiepluchandClaude Opus 5.5 da91ef0dd9 exp-003: supplementary all-units replay after the cache fill; exp-004 registered
Supplementary analysis, not a new decision: exp-003's verdict stands.
The 142 units excluded for an uncached response were filled with the same
six incumbent programs, parameters and request keys, on exp-003's
937-example memory (the harvest's 196 rows and its 2026-10-03 embeddings
left out; a zero-cost replay first reproduced exp-003 exactly: 211
decided, 1056 replayed, $0.582876). Fill: MeteredClient, ceiling $0.99
under the approved $1.00, 697 client calls, $0.331759, no provider error.

Replay on all 353 units (244 paired, 146 groups): 0 client calls, $0.0,
1768 responses replayed; no arm flags any unit, the registered rule reads
inconclusive. Calibration holds only for untrusted destination (ECE
0.0454) and arm A still finds no BLOCK point there.
benchmarks/experiments/exp-003-precision-bound-blocking/result-complete-cache.json

exp-004 registered in s3://sast-memory (manifest 87fceb65..., pin
2ad9c1d, units pending): benchmarks/experiments/exp-004-pooled-block-gate/
registration.json. docs/decision-engine.md: both, briefly.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 09:47:20 +02:00
Marius CiepluchandClaude Opus 5.5 2ad9c1db7a learn(exp-004): Req 6.4 pooled BLOCK gate with a family floor; manifest before units
Maintainer decision 2026-10-03: BLOCK precision is shown pooled across
families. Req 6.4 (amended; 7.4 annotated): one t_block on a per-family
Platt-normalised score (calibrate.fit_map, training folds, inner
cross-fit), Wilson 95% LB >= 0.95 over all families' training flags; a
family blocks only with >= F = 10 held-out flags and zero errors, else
advisory; t_quiet per family keeps >= 90% of training paired positives.

evaluate: normalised, pooled_threshold, quiet_threshold,
pooled_block_gate, family_floor, apply_floor. experiments: rule
pooled_block_gate (normalise: platt, floor, recall), decide_pooled on the
pre-floor pooled held-out LB, floor table, quiet_recall and after-floor
totals in analyse_rule; exp-003's analysis is unchanged.

plane/experiments/exp-004-pooled-block-gate.yaml: procedure and decision
rule registered now; units frozen at run time from the post-harvest
memory (units_status pending until then).

Tests (tests/test_learn_rules.py): pooled threshold where no family alone
reaches the bound, no held-out label moves a threshold or the map, a
family with one held-out error cannot block, t_quiet and no blind
silencing, the pooled verdict, the exp-004 manifest.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 09:40:58 +02:00
Marius CiepluchandClaude Opus 5.5 5b2bd9bb91 merge(learn): negatives harvest -- 98 real fix pairs with surviving functions (79 post-cutoff), paired groups 151 -> 249
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:29:32 +02:00
Marius CiepluchandClaude Opus 5.5 50c93edfb8 learn: 98 advisory-fix pairs whose function survives the fix (advisory-fixes-2)
Real (vulnerable function, fixed function) pointer pairs for the six compiled
families, mined at zero model cost from reviewed GHSA advisories (pip, npm,
maven, composer), newest first: 79 of 98 published after 2026-06-01
(post_cutoff = true). Each pair: one linked one-parent fix commit, <= 4 source
files / 120 lines, the function holding the changed code lines declared once on
both sides with a changed body; every function diff read and kept only when the
named function holds the fix (145 reviewed, 100 kept); 2 dropped by the
reserved-population test; permissive licences only, no repository named
elsewhere under benchmarks/.

New label source advisory-fixes-2; the advisory-fix tests run over both
batches. Memory 937 -> 1133 static examples (98 paired groups added);
positives-only groups stay 99 (vibe-py/dev-php/v2 data gaps), their share of
positive groups in the six families 0.40 -> 0.29. Embeddings: 7 calls,
$0.0014. No chat-model call. Record:
benchmarks/measurements/2026-10-03-decision-engine-negatives-harvest.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:23:42 +02:00
Marius CiepluchandClaude Opus 5.5 6ecf0a304e merge(learn): exp-003 precision-bound blocking -- inconclusive; the Wilson bound needs 73 clean flags, injection has 38 paired positives
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:21:27 +02:00
Marius CiepluchandClaude Opus 5.5 b31c85678f exp-003: result -- inconclusive, no BLOCK rule flags a held-out unit
Zero-cost replay of the six slice-2 programs: meter 0 client calls, $0.0;
211 of 353 units scored, 142 excluded for an uncached response (path lost
all 39). Arm A (calibrated) fails calibration (ECE 0.077 access control,
0.1067 injection); arm B (Wilson LB >= 0.95, nested) chooses no threshold on
any fold, as the registration's power note predicted (73 clean flags needed,
38 paired injection positives); secondary arm C flags nothing either. The
instrument check shows real scores; the zero is the rule's.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:15:22 +02:00
Marius CiepluchandClaude Opus 5.5 7bb97ad0bc exp-003: registration, before the run
Registered in s3://sast-memory at dd7ebc1: manifest 5917f9a1, units 338de998
(353 units, 146 groups). Primary metric, decision rule and the threshold
selection are fixed in the manifest; the power note says arm B cannot reach
a Wilson LB of 0.95 on this data (73 clean flags needed, 38 paired
positives at most).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:03:22 +02:00
Marius CiepluchandClaude Opus 5.5 dd7ebc119b learn(exp-003): decision-rule experiments and the precision-bound BLOCK arm
kind: rule experiments score once from the response cache behind a
zero-ceiling meter (a miss is excluded and counted, never asked) and apply
each arm's BLOCK rule to the same held-out scores. Arm B chooses a per-family
raw-score threshold on the other outer folds only (lowest t with Wilson 95%
LB >= 0.95); a test fails if a fold's own labels move its threshold.
Manifest and frozen units (353 units, 146 groups, 244 paired) for exp-003.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:02:51 +02:00
Marius CiepluchandClaude Opus 5.5 b24eeb15f3 merge(plane-on-kubernetes): groups 1-2 -- cluster profile, no host literals, delivery through the S3 store
Kind proofs: records 01 and 02. Task 2.6 (receiver removal) is paused while the
safety net and the decision engine come first. deployment.md conflicts resolved
in the plain-language style; corrects the RuntimeClass claim (Substrate runs
runsc inside its worker pod).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 07:57:13 +02:00
Marius CiepluchandClaude Opus 5.5 c8b418c34c plane(2.5): kind proof and record 02 -- artifacts through the store, one TLS rule
Live on kind with the runner image that carries 2.2
(ousast-runner@sha256:d2a55293...; templates and validation-46 Tasks repinned,
128 lines; kind-images.json updated): the smoke Run on store delivery finished
in 26 s and 24 s; the tar was PUT to runs/ax-e2e-smoke/facts/output.tar
(10240 B) and collected 2 s later; read back from the store its members
(facts.json 451 B, summary.json 238 B, units.jsonl 291 B) equal the extracted
files; facts.json, summary.json and done.json (149 B) re-put, state.json
(350 B) mirrored; a rerun from an empty results root seeded state.json from the
store and finished in 2 s without resubmitting. The actor's egress policy read
back live: one tlsPassthrough rule on 443 naming the store host, nothing else.
`ousast plane status ax-e2e-smoke` printed the table each time. Record:
benchmarks/measurements/2026-10-02-k8s-02-store-delivery/record.json with the
smoke outputs, store read-back and policy beside it. Task 2.6 (receiver
removal) is paused by the maintainer's reprioritisation.

Gates: pytest 2031 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0. (The first run, at 0.97 GiB free disk, failed 11 tests and errored 5
on the FileStore's 1 GiB free-space refusal; rerun after the disk was freed.)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 07:45:19 +02:00
Marius CiepluchandClaude Opus 5.5 432fd2947e merge(pre-push): operational readiness increment 1 -- never block, never hide, quick tier, new files, background engine, install
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 07:29:17 +02:00
Marius CiepluchandClaude Opus 5.5 2ec8071c26 pre-push(17.7): prove it on the audit's three repositories
Rerun records, serve-static and unrolled/secure (same planted commits) with
this build installed from its wheel: 6 replays and 10 hook pushes, all exit 0
in 0.69-1.21 s against the 30 s deadline, input read proven per run. The
planted JS eval in a new file surfaces in the quick tier on the host and, in
one Docker engine run (190.9 s, deadline 300), as two advisory engine findings
classified new/file_added_in_head. The planted Python %-formatted SQL is still
missed (no quick rule, no host engine): recorded as a coverage gap, no rule
edited. Go is named as not covered. The run caught a wrong install hint in
the hook (the package is not on PyPI); corrected here.
Record: benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 23:02:32 +02:00
Marius CiepluchandClaude Opus 5.5 591818d5fc pre-push(17.6): install, Docker wrapper and first-run docs
`ousast pre-push install|uninstall [REPO] [--force] [--docker]` ships in the
wheel with the hook as package data; it honours core.hooksPath, refuses an
existing hook unless --force (which chains it as pre-push.before-ousast), and
uninstall restores the chained hook. ops/ousast-docker is a hook-callable
script: the repository read-only and the artifact directory at their own
paths, stdin passed through, no network, --memory 3g, the GHCR 2.0.1 image by
default, advisory exit unless blocking. Help text for every pre-push flag; a
first-run guide, settings, skip-reason glossary, CI recipe and full --help in
docs/scanning.md; README tiers 1 and 2 pull the published image. The image
was not rebuilt: 1.1 GB free, below the 4 GB threshold. Requirements 7.4, 9.6.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 22:51:22 +02:00
Marius CiepluchandClaude Opus 5.5 66357ffea1 pre-push(17.5): say when the engine cannot finish, and offer it in the background
--engine inline|background|off (hook: OUSAST_PUSH_ENGINE). background returns
after the quick tier and starts one detached explicit replay of the same base
and head (push/background.py), one at a time per artifact directory; its
artifact lands under <artifact dir>/engine/<repo key>/<head>.json and the next
run prints "Engine result for <head> from the previous push" once. A deadline
miss says so in one line naming the deadline and the background option. No
engine speed work. Requirement 9.5.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 22:31:14 +02:00
Marius CiepluchandClaude Opus 5.5 511de4b75f pre-push(17.4): a file new in head is new, not comparison unknown
ChangeContext records the paths Git reports as added. An entailed,
undischarged operation in an added file, with complete head evidence and no
identical operation in any base answer, is classified new
(file_added_in_head) instead of base_incomplete; admission accepts that reason
as change evidence. Unpaired add/delete pushes and every other unknown
comparison keep their vetoes. An engine finding whose novelty is established
and whose only vetoes are capability vetoes is shown as an advisory engine
finding, not an alert. Requirement 9.4.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 22:24:24 +02:00
Marius CiepluchandClaude Opus 5.5 15999a09e1 pre-push(17.3): quick rules as the fast first tier
Before the engine and inside the same deadline, the shipped quick rules run
over the head side of the changed files (push/quick.py) and keep matches on
changed lines or in changed functions. They print as advisory pattern matches,
labeled apart from admitted alerts, are kept in the artifact's quick_tier with
the files and bytes read, and never admit or block. A new branch with no
remote base runs the quick tier over its last commit and says so. The planted
eval in serve-static's new debug.js now surfaces; the planted Python
%-formatted SQL in records does not (a quick-rule coverage gap, no rule
edited). Requirement 9.3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 22:18:14 +02:00
Marius CiepluchandClaude Opus 5.5 f555dee640 pre-push(17.2): name every skipped check in plain words
One terminal line per distinct coverage reason (push/explain.py): hex-encoded
paths decoded, per-question sha256 tokens dropped and counted, unknown reasons
still printed; the JSON artifact keeps the raw values. A changed language with
no engine frontend and no quick rules is recorded and printed as "language not
covered: <lang> (...)", in pre-push and in `ousast scan`. The scan report
renders the eight degradations cli.py recorded but never showed, and any
reason without a sentence. The hook forwards OUSAST_COMPARISON_BASE.
Requirement 9.2.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 22:10:33 +02:00
Marius CiepluchandClaude Opus 5.5 bf07cd4d1d pre-push(17.1): never block a push the user did not choose to block
The installed hook exits 0 with one line when ousast (or OUSAST_COMMAND) is
not on PATH, still chaining a prior hook. In Git hook mode without an explicit
--mode blocking, argument/usage errors (an invalid OUSAST_PUSH_DEADLINE, a cache
directory inside the repository, a refused experimental flag) and unexpected
exceptions print one "check did not run" line and exit 0. Blocking mode keeps
its usage errors. Requirement 9.1.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:57:42 +02:00
Marius CiepluchandClaude Fable 5.1 605b7af5ba plane(2.4): reconciler on Delivery -- the start body carries the URLs, the store completes the task
_execute builds the task's `delivery {put, inputs}` from Delivery, passes it to
router.starter, grants the store's host in the egress policy, awaits
Delivery.delivered (the tar's exact key in the store), then collect()s:
download, extract with the member checks, re-put the declared artifacts, write
done.json; the outcome is the collected summary.json. Three findings from the
tests: a rerun of an unfinished task completed at once on the previous
attempt's tar, so Delivery.reset(task) deletes output.tar and done.json before
every attempt; a worker thread that raised left the run blocked on its queue,
so the worker now turns any exception into a failed task; a consumer may
declare a producer's summary.json without the producer listing it (the loop
Run does), so declared(run) publishes outputs[] plus every artifact a consumer
names. The receiver thread survives behind OUSAST_DELIVERY=receiver for this
group only (no presigned URLs, the http rule, OUSAST_ARTIFACT_URL in the env);
the default path renders no receiver address into any Task. The fake ax PUTs
to the start body's URL and GETs its inputs from theirs; the reconciler tests
run through the S3-shaped fake store and assert the objects, done.json, the
seeded rerun and the receiver mode. reconciler.py: 549 lines (bound 550).

Gates: pytest 2031 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:49:58 +02:00
Marius CiepluchandClaude Opus 5.5 a656fda2f4 pre-push(17): operational-readiness task group and Requirement 9
Append task group 17 "Operational readiness (2026-10-02)" with one task per
maintainer-approved item (never block, name every skip, quick tier, new-file
classification, engine honesty and background mode, install/Docker/docs, proof),
add Requirement 9 (9.1-9.6, marked added 2026-10-02, maintainer-approved), and
check 13.4 and the parent of 14 as the audit found them done.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:46:25 +02:00
Marius CiepluchandClaude Fable 5.1 c49eae4f30 plane(2.3): egress -- one TLS rule over store host, Git hosts and Model hosts; no http rule
policy_for(task, workspaces, model, store_host, receiver_host=None) emits one
tlsPassthrough rule on 443 over the sorted union of the memory store's host,
the bound Workspaces' https Git hosts and the bound Model's egress hosts. There
is no http rule: an http Git repo is refused by name (it could not be reached),
a store given as an address is refused naming S3_ENDPOINT (the gateway allows
hostnames only). receiver_host adds the old receiver's http rule for the path
group 2 keeps until 2.6; the reconciler passes it by keyword (passing the
receiver's address positionally would have raised outside the worker's try and
hung the run on the queue -- 2.4 hardens the worker). Tests: the store-only
policy, the Model-bound policy, the two refusals, the receiver exception, and
every committed template under plane/tasks/ rendering exactly the store plus,
for verify and roles, the provider host.

Gates: pytest 2027 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:29:32 +02:00
Marius CiepluchandClaude Opus 5.5 49e11e0252 docs(readme): one paragraph for the published site and local preview
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:17:44 +02:00
Marius CiepluchandClaude Opus 5.5 f0fcb6686f docs: publish the MkDocs site to GitHub Pages
A Docs workflow builds the site strictly with the docs extra and deploys it to
https://norandom.github.io/OpenUltraSAST/ on pushes to main that touch the docs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:16:49 +02:00
Marius CiepluchandClaude Fable 5.1 9506a18b56 plane(2.2): runner -- delivery {put, inputs} in the start body, PUT to the store
StartGate.offer validates an optional `delivery` beside the credentials: https
URLs to a hostname (never an address, no userinfo; AX_RUNNER_INSECURE_DELIVERY=1
admits http in unit tests only), names ^[A-Z][A-Z0-9_]*$, a 400 that names the
field and echoes no URL. The runner PUTs its output tar to delivery.put and
GETs each declared input from delivery.inputs[NAME] (deliver() gains a method;
log lines and DeliveryError carry the URL without its query, so no presigned
signature reaches a log, a marker or a summary). OUSAST_ARTIFACT_URL/DIAL stay
honoured until task 2.6 so both paths run in the same build; the completion
marker and the retry-only boot are unchanged, and a re-start brings a fresh
URL for the retry. Router.start_task carries `delivery` in the body and
redacts its URLs like credential values. Tests: the gate's acceptance and
refusals, an end-to-end PUT/GET against the S3-shaped fake with no URL in any
log or file, the failed-PUT-then-retry-only boot, deliver() with PUT.

Gates: pytest 2024 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:12:19 +02:00
Marius CiepluchandClaude Fable 5.1 35d8032c0a plane(2.1): Delivery -- presigned store URLs, collection, state mirrored to the store
src/openultrasast/plane/delivery.py: Delivery(store, run, base) mints a presigned
PUT for runs/<run>/<task>/output.tar and a GET per declared input
(runs/<run>/<producer>/<artifact>), expiry OUSAST_TASK_TIMEOUT + 600 s;
delivered() asks the store for the exact key; collect() downloads, extracts
under the run directory with the receiver's member checks (no symlink, hard
link or escape; nothing written on a rejection), re-puts the declared outputs
as single objects and writes done.json; put_input() for inputs no task
produces; save_state()/load_state() mirror state.json. The reconciler opens the
store first (no fallback; the CLI reports a store that cannot open and submits
nothing), mirrors state.json after every change and seeds a rerun with no local
state from the store (the 2.1 bullet's state mirroring is what gives the new
module its importer). The reconciler crossed 500 lines here (517) rather than at
2.4 as 1.6 projected: the bound moves to 550 with the reason in the assertion,
as 1.6 decided. tests/plane_fake_store.py is the S3-shaped fake (objects in a
dict, presigned URLs on a local server); tests/test_plane_delivery.py covers the
body, expiry, polling, rejection, declared-only re-put, state seeding and that
no URL reaches a log. Module-audit manifest regenerated (148/5, no orphan).

Gates: pytest 2020 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:05:48 +02:00
Marius Ciepluch 9833571e94 Merge branch 'worktree-agent-af2e4a098d650d0a2' 2026-10-02 21:03:41 +02:00
Marius Ciepluch c217d1d1d3 Merge branch 'worktree-agent-aed844222b4750ed7' 2026-10-02 21:03:41 +02:00
Marius CiepluchandClaude Opus 5.5 352f8c07aa docs: plain-language pass -- detection techniques, architecture, evaluation, decision engine
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:03:24 +02:00
Marius CiepluchandClaude Opus 5.5 552671ca6a docs: plain-language pass -- README, index, where we stand, scanning, examples, threat model
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:01:10 +02:00
Marius CiepluchandClaude Opus 5.5 7e23379e96 docs: plain-language pass -- deployment, plane, memory, RustFS, token economics
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:00:37 +02:00
Marius CiepluchandClaude Fable 5.1 79296bb4a4 plane(1.8): kind proof and record 01 -- profile, literals, doctor, probe, bound
benchmarks/measurements/2026-10-02-k8s-01-profile/record.json with the live
evidence of group 1: `ousast plane doctor --profile kind` passes every line
(exit 0, 1.4 s; the store check reaches the bucket: verify 0.62 s, probe object
24 B); `--profile k3s` fails on the unreachable context and on the images file
the release has not published yet (two expected lines; the namespaces are not
checked behind an unreachable context); the literal scan has 0 offenders (23
before 1.3) and fails naming workspaces.py:37 when a literal is reinserted;
up.sh reran idempotently in 38 s and the smoke Run finished in 24 s on the
rebuilt image (facts.json 451 B, summary.json 238 B); the placement probe says
`any`; the reconciler bound numbers of 1.6. Doctor, literal, smoke and up.sh
outputs beside it.

Gates: pytest full suite on this tree's src/ and tests/ content: 2010 passed,
42 skipped, exit 0 (the 1.5 run; the 1.8 run reported two module-audit
failures caused by the uncommitted 2.1 module present in the working tree);
ruff check 0; ruff format 0; mypy 0; mkdocs build --strict 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:55:57 +02:00
Marius CiepluchandClaude Fable 5.1 2cca03c1c0 plane(1.7): gvisor tarball reachability documented as planned (manual, k3s)
docs/deployment.md B.1: Substrate's ateom-gvisor worker runs runsc from the
release tarball named in manifests/ate-install/sandboxconfig-gvisor.yaml
(gs://gvisor/releases/nightly/2026-09-02/<arch>/gvisor.tar.zstd, sha256-pinned
per architecture), fetched by the atelet on the node; the node must reach it or
a copied SandboxConfig must name a mirror with the same sha256. The check is the
atelet's fetch line in the worker pod's logs, run by hand on the day k3s exists;
labelled planned until then (design open question 3 stays open).

Gates: docs only; mkdocs build --strict passed with this paragraph present in
the 1.4 gate; no file a test or linter reads changed since the 1.5 gate.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:50:14 +02:00
Marius CiepluchandClaude Fable 5.1 2afa20e2ea plane(1.6): reconciler bound recorded -- 489 lines, bound moves to 550 in 2.4
reconciler.py: 477 lines at 6bd44ca, 489 after 1.2 (the profile parameter of
run() and its docstring). Task 2.4 adds the Delivery wiring (about +25) while
the receiver path is kept for group 2, and 2.6 removes the receiver wiring
(about -20): projected 514 during group 2, 494 after 2.6. The estimate crosses
500 while both paths coexist, so 2.4 moves tests/test_plane_reconciler.py's
bound to 550 with the reason in the assertion message; 2.6 records the final
count. Numbers in benchmarks/measurements/2026-10-02-k8s-01-profile/record.json
(record 01, drafted here, completed by 1.8).

Gates: no file a test or linter reads changed since the 1.5 gate (pytest 2010
passed, 42 skipped, exit 0; ruff 0; format 0; mypy 0).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:49:45 +02:00
Marius CiepluchandClaude Fable 5.1 86ec237483 plane(1.5): placement probe -- Substrate places by any free worker, not by fit
ops/ax/probe-placement.sh scales ousast-pool to one worker, applies probe-pool
(1 x 2 CPU / 3Gi, the already-built ateom-gvisor image, no ko build), runs one
model-free repo-facts Task requesting 2560Mi (limit 3Gi) and one requesting
256Mi, records `kubectl ate get workers -o json` after each resume, and removes
everything it created. Live on kind: both actors landed on the 1536Mi
ousast-pool worker (status.allocated.actors=1 there, 0 on probe-pool) ->
placement: any. design.md section 4 records the outcome and the fallback group
5 must take (an engine atespace with engine-pool as its only pool if Substrate
scopes pools by atespace; else a plain Kubernetes Job on k3s). Evidence under
benchmarks/measurements/2026-10-02-k8s-01-profile/probe/.

Gates (tree identical to 1.4 in src/ and tests/; ops/ gains the probe script,
which the literal scan covers): pytest 2010 passed, 42 skipped, exit 0; ruff
check 0; ruff format 0; mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:49:03 +02:00
Marius CiepluchandClaude Fable 5.1 8099b29deb plane(1.4): up.sh by profile -- context, registry, pool and images file from kind.toml
ops/ax/up.sh reads kube_context, registry, pools.default and the images path
with `python -m openultrasast.plane.profile --print`, writes the runner digest to
ops/k8s/profiles/kind-images.json instead of ~/.cache/.../runner-image, renders
ops/ax/workerpool.yaml.tmpl from the profile's pool (POOL_NAME/REPLICAS/CPU/
MEMORY, WORKER_IMAGE), and keeps an existing pool and an ax CLI newer than its
checkout instead of rebuilding them (ko and go install would refill several GB
of Go cache on this disk). smoke-run.sh takes the runner from `--print
images.runner` and runs the checkout's code with OUSAST_PYTHON. repin_templates
reads the JSON images file (runner key; the one-line pin still reads). `--print
images` returns the file path (the first live rerun fell over this). Templates
and the generated validation-46 Tasks re-pinned to the rebuilt image (128
lines, one digest).

Live: up.sh reran idempotently on kind in 38 s (every step kept except the
runner build); smoke-run.sh done in 24 s on the new image, facts.json 451 B,
summary.json 238 B delivered (record: benchmarks/measurements/2026-10-02-k8s-01-profile/).

Gates: pytest 2010 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0; mkdocs build --strict 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:43:40 +02:00
Marius CiepluchandClaude Fable 5.1 36f5927999 plane(1.3): host-literal scan test over src, plane, ops and docs
tests/test_plane_literals.py scans src/, plane/, ops/ (minus ops/ax/up.sh and
ops/k8s/profiles/) and docs/ for the kind context, the kind registry, the kind
network's gateway, a loopback address with a port and port-forward; router.py
keeps the last two, a line tagged ax-tunnel or loopback is allowed, and a
committed template's digest-pinned image is accepted only while it names the
kind profile's registry (checked by a third test). A self-check reinserts each
literal into a fixture and sees it caught. The runner's health URL and the Joern
session's query URL are tagged loopback; docs/deployment.md and ops/ax/README.md
stop naming this host and describe the kube context and the registry as the
profile's (what 1.2 made configurable).

Gates: pytest 2009 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0; mkdocs build --strict 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:32:51 +02:00
Marius CiepluchandClaude Fable 5.1 77a03bad61 plane(1.2): literals out of doctor, egress, router, generate -- the profile is handed in
doctor(profile) checks the profile's kube context, the two namespaces behind a
reachable context only, the images file's runner reference in the profile's
registry (a local registry over its v2 API, a remote one via docker manifest
inspect) and opens the memory store; its first line reports every configured
address (Req 1.2). Egress(executable, context), open_router(context, url=...),
receiver_cluster_ip(context) and Receiver(..., context) take the profile's
context and build none; generate.increment/render and harvest take the profile's
atespace. `ousast plane doctor|run|workspaces|harvest --profile`. The test fakes
(kubectl-ate, kubectl, ax) record the context they received and the tests assert
it is the profile's. grep "kind-" src/openultrasast/plane: no match.

Gates: pytest 2006 passed, 42 skipped, exit 0; ruff check 0; ruff format 0; mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:26:46 +02:00
Marius CiepluchandClaude Fable 5.1 8eac1b2c69 plane(1.1): PlaneProfile -- one TOML per cluster, env-overridable, no secrets
ops/k8s/profiles/kind.toml and k3s.toml hold the values that differ per cluster
(exec, kube_context, registry, images, router_url, ax_server, memory, atespace,
snapshots_bucket, image_pull_secret, pools); kind-images.json carries the digest
pin ops/ax/up.sh wrote. load_profile() reads OUSAST_PLANE_PROFILE (name or path,
default kind), applies OUSAST_<FIELD> and OUSAST_POOL_<NAME>_<FIELD>, and refuses
an unknown key, a credential-shaped key or value, an exec outside local|remote
and an images file without digest pins. `python -m openultrasast.plane.profile
--print FIELD` serves shell scripts. The module-audit manifest is regenerated
for the new module (counts 147/5, no orphan).

Gates: pytest full suite 2002 passed, 1 failed (the module-audit manifest, fixed
by regenerating it; tests/test_module_audit.py re-run exit 0), ruff check 0,
ruff format 0, mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:16:23 +02:00
187 changed files with 67706 additions and 2259 deletions
+43
View File
@@ -0,0 +1,43 @@
name: Docs
# Builds the MkDocs site and publishes it to GitHub Pages on every push to main that touches it.
on:
push:
branches: [main]
paths: ["docs/**", "mkdocs.yml", "README.md", ".github/workflows/docs.yml"]
workflow_dispatch:
permissions:
contents: read
pages: write
id-token: write
concurrency:
group: pages
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: astral-sh/setup-uv@v7
with:
enable-cache: true
- uses: actions/setup-python@v6
with:
python-version-file: .python-version
- run: uv sync --extra docs
- run: uv run mkdocs build --strict
- uses: actions/upload-pages-artifact@v3
with:
path: site
deploy:
needs: build
runs-on: ubuntu-latest
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- id: deployment
uses: actions/deploy-pages@v4
+63
View File
@@ -0,0 +1,63 @@
name: Engine image
on:
workflow_dispatch:
push:
branches: [main]
paths:
- 'src/**'
- 'benchmarks/learn/engine_trace*.py'
- 'benchmarks/learn/engine_task_entry.py'
- 'benchmarks/learn/engine_worker_service.py'
- 'Dockerfile'
- 'plane/Dockerfile.engine-task'
- 'ops/frontend-retention/install.py'
- 'pyproject.toml'
- '.github/workflows/engine-image.yml'
permissions:
contents: read
packages: write
concurrency:
group: engine-image-${{ github.ref }}
cancel-in-progress: false
jobs:
image:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- package: openultrasast
dockerfile: Dockerfile
- package: ousast-engine-task
dockerfile: plane/Dockerfile.engine-task
steps:
- uses: actions/checkout@v5
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: names
env:
OWNER: ${{ github.repository_owner }}
PACKAGE: ${{ matrix.package }}
run: |
echo "image=ghcr.io/${OWNER,,}/$PACKAGE" >> "$GITHUB_OUTPUT"
echo "sha=sha-${GITHUB_SHA:0:7}" >> "$GITHUB_OUTPUT"
- id: build
uses: docker/build-push-action@v6
with:
context: .
file: ${{ matrix.dockerfile }}
platforms: linux/amd64
push: true
tags: |
${{ steps.names.outputs.image }}:${{ steps.names.outputs.sha }}
${{ steps.names.outputs.image }}:main
- env:
IMAGE: ${{ steps.names.outputs.image }}
DIGEST: ${{ steps.build.outputs.digest }}
run: |
echo "$IMAGE@$DIGEST"
echo "$IMAGE@$DIGEST" >> "$GITHUB_STEP_SUMMARY"
@@ -0,0 +1,80 @@
# Plan: learn fix mechanisms into the engine (draft for review, 2026-10-04)
Status: approved 2026-10-04 by the maintainer. Increment 1 = discharge facts only; coverage additions (Req 7) come later.
Decisions: minimum 2 proposing repositories before a candidate is tried; admission per language (not pooled by callee name).
## Why
The Joern trace run over exp-005's 501 units shows the engine cannot tell a vulnerable function from its
fix. In the 105 pairs where both sides were analysed, a taint path appears on both sides in 12, on the
vulnerable side only in 2, on the fixed side only in 0, and on neither in 91
(`benchmarks/measurements/2026-10-04-joern-slice-leak-audit/record.json`).
So the fix is invisible to the engine. Either the flow survives and the engine does not know the fix's
mechanism (an `allowed_classes` argument, a path normalisation check, an escaping call), or the engine's
sources and sinks miss the function on both sides. A classifier fed this evidence cannot improve on it.
The corpus now holds 710 complete fix pairs (`2026-10-04-decision-engine-harvest-3/record.json`). Each fix
diff shows what was added. That is the material the engine lacks.
## Goal
Grow the engine's discharge knowledge (sanitizers, guards, safe-argument forms) and, second, its source and
sink coverage, from the fix diffs. Admit a fact only when it improves held-out separation. Spend no model
money.
## Requirements (proposed)
1. **Mining.** From each fix pair, extract what the fixed function adds on the vulnerable flow: new calls,
new conditions guarding the sink, new or changed sink arguments. Record each as a candidate fact with its
language, family, form (sanitizer call, guard predicate, safe argument, sink restriction) and the pairs
that propose it. Mining is deterministic (tree-sitter diff of the two function bodies). No model calls.
2. **Generalisation is explicit.** A candidate is a pattern, not a repository name: a callee name, an
argument key, a predicate shape. Candidates naming project-specific identifiers are rejected (Req 8: no
application-specific trees).
3. **Admission by held-out separation.** Folds are by repository, as in the decision engine. A candidate is
admitted for a fold only if, on that fold's held-out pairs, adding it to the fact tables:
- increases pairs where the vulnerable side keeps a path and the fixed side loses it or becomes
sanitized;
- and does not remove any held-out vulnerable-side path (no recall loss).
A candidate proposed by fewer than 2 distinct repositories in the training folds is not tried.
4. **No train-on-test.** Mining and admission never read held-out pairs; the closed-loop leak recorded in
memory (`closed-loop-train-on-test-leak`) is the failure this rule exists to prevent. Population v3 is
never read.
5. **Measurement by the trace runner.** Each admission round re-runs the engine on the affected pins only
(the runner's resume and selection already support this; on k3s the worker Deployment runs them in
parallel). The outcome is the pair-asymmetry table per family, before and after, on held-out folds.
6. **Gate before production.** An admitted fact enters `src/openultrasast/ruleset/semantic/*.toml` only after
the detection gates (`gate`, `map_gate`, `pair_gate`) show no regression, and only as a registered
experiment whose decision rule was fixed before the run.
7. **Coverage second.** After discharge facts, the same loop proposes source and sink additions from pairs
where neither side has a path (91 of 105 today), admitted under the same held-out rule.
## How it relates to what exists
- `corpus-seeded-mechanisms` already has a closed loop that admits and retracts mechanisms by leave-one-out.
This plan reuses its lever and store, and points it at discharge facts mined from real fix diffs instead
of variant search.
- `benchmarks/pairs/mechanisms.toml` (23 mechanisms) labels what kind of bug a pair is. The new candidates
describe what a fix does. They live in the fact tables, not in the mechanism vocabulary.
## Measures of success
- Pair asymmetry on held-out folds: vulnerable-only paths rise from 2 of 105 toward a substantial share,
with zero fixed-only paths.
- No held-out recall loss on vulnerable sides.
- Then the decision engine is re-run with the new engine evidence (exp-005 design), and the pooled block gate
(exp-004 design) is re-measured.
## Cost and time
- Model cost: $0 (mining and admission are deterministic; the classifier re-run afterwards is a separate,
approved spend).
- Engine time: one pass over affected pins per admission round; about 2 hours per full pass on k3s plus the
VM, about 6 hours on the VM alone.
## Open questions for the maintainer
1. Should coverage additions (Req 7) be in the same increment or a later one?
2. Minimum proposing repositories: 2 is the proposal. Higher is safer, lower finds more.
3. Admission per family and language, or pooled across languages for the same callee name?
@@ -119,6 +119,19 @@ rule hits.
2. Without a trained model available the tools fall back to today's behaviour and say so in the report. 2. Without a trained model available the tools fall back to today's behaviour and say so in the report.
3. Hand-written rules remain as signal producers; changing their enabled/shadow status no longer changes what is 3. Hand-written rules remain as signal producers; changing their enabled/shadow status no longer changes what is
reported unless the engine's decision changes. reported unless the engine's decision changes.
4. *(Amended 2026-10-03, maintainer decision; refines the BLOCK rule of 7.3 and 7.4. Tested by exp-004.)* BLOCK
precision is demonstrated pooled across families, not per family:
- **Normalisation**, declared before any run: each family's raw score is put on one scale by a per-family Platt
map (`learn.calibrate.fit_map` with its floors) fitted on training folds only. A family the training folds
cannot map does not block on that fold.
- **t_block**: one threshold on the normalised score for all families, chosen on training folds as the lowest
one whose Wilson 95% lower bound on precision, over the training flags of all families together, is >= 0.95.
It is reported by the Wilson 95% lower bound over all held-out flags of all families, before the floor.
- **Family floor**: a family may block only if its own held-out flags have zero errors and number at least F
(F = 10, reasoned in the exp-004 registration); otherwise that family stays ADVISORY.
- **t_quiet**: per family, the highest threshold keeping >= 90% of the paired positives at or above it (chosen
on training folds, its held-out recall reported); findings below it are silent.
- No held-out label moves a threshold or a normalisation map; candidates whose verdict is `unsure` never block.
### Requirement 7: Built for unseen repositories -- a repo safety net ### Requirement 7: Built for unseen repositories -- a repo safety net
@@ -136,7 +149,8 @@ rule hits.
ADVISORY, trading precision for recall; both reported with out-of-repository intervals. The decision unit is a ADVISORY, trading precision for recall; both reported with out-of-repository intervals. The decision unit is a
candidate in changed code (`pre-push` delta), with whole-repository scans as a secondary mode. candidate in changed code (`pre-push` delta), with whole-repository scans as a secondary mode.
4. Calibration is checked on held-out repositories; where it does not hold, only ADVISORY is offered for that 4. Calibration is checked on held-out repositories; where it does not hold, only ADVISORY is offered for that
family. family. *(Amended 2026-10-03: once exp-004 adopts the pooled gate of 6.4, BLOCK precision is shown by its held-out
bound and family floor, and calibration is reported, not required.)*
5. Any adaptation to a user's own repository (facts reuse, their dismissals) is opt-in, separate from the 5. Any adaptation to a user's own repository (facts reuse, their dismissals) is opt-in, separate from the
generalisation model, and never contributes to its reported numbers. generalisation model, and never contributes to its reported numbers.
@@ -254,6 +254,13 @@ containerd; `Dockerfile:1-7` already argues against it).
a 3 Gi actor lands only on `engine-pool`. If Substrate does not place by resource fit, the fallback is a a 3 Gi actor lands only on `engine-pool`. If Substrate does not place by resource fit, the fallback is a
separate atespace with its own pool if Substrate scopes pools that way; failing that, the engine stays separate atespace with its own pool if Substrate scopes pools that way; failing that, the engine stays
`alerts-engine` on a Docker host and the k3s engine becomes a plain Kubernetes Job outside ax, recorded as such. `alerts-engine` on a Docker host and the k3s engine becomes a plain Kubernetes Job outside ax, recorded as such.
**Probe outcome (task 1.5, 2026-10-02, `benchmarks/measurements/2026-10-02-k8s-01-profile/`): `placement: any`.**
With `ousast-pool` at one 1536Mi worker and a `probe-pool` of one 3Gi worker both ACTIVE, a `repo-facts` actor
requesting 2560Mi (limit 3Gi) was placed on the 1536Mi `ousast-pool` worker (`kubectl ate get workers`: ACTORS
1/1000 there, 0/1000 on `probe-pool`), and so was the 256Mi actor. Substrate does not place by resource fit, so
group 5 takes the fallback: the engine pool must be the only pool its actors can land on. Before 5.2, check whether
Substrate scopes pools by atespace (an `engine` atespace with `engine-pool` as its only pool); if it does not, the
k3s engine runs as a plain Kubernetes Job outside ax and `alerts-engine` stays the kind path.
- **`alerts-engine` stays** for development on a host with Docker (kind profile), its help text and - **`alerts-engine` stays** for development on a host with Docker (kind profile), its help text and
`docs/plane.md` row marked "development only; the plane task is `engine`" (Req 4.3). Generated Runs choose `docs/plane.md` row marked "development only; the plane task is `engine`" (Req 4.3). Generated Runs choose
`engine` when the profile's `images` file has an engine image, else leave `alerts` for `alerts-engine`. `engine` when the profile's `images` file has an engine image, else leave `alerts` for `alerts-engine`.
+13 -13
View File
@@ -10,7 +10,7 @@ its kind proof and the record it writes under `benchmarks/measurements/<date>-k8
go-ahead** with a ceiling; steps that need a hand on a cluster that does not exist yet are marked **manual**. go-ahead** with a ceiling; steps that need a hand on a cluster that does not exist yet are marked **manual**.
- [ ] 1. Profile and literals (design section 1; open-question checks 2, 3, 7) - [ ] 1. Profile and literals (design section 1; open-question checks 2, 3, 7)
- [ ] 1.1 `PlaneProfile` - [x] 1.1 `PlaneProfile`
- `src/openultrasast/plane/profile.py`: frozen dataclass with the fields of design section 1; `load_profile()` - `src/openultrasast/plane/profile.py`: frozen dataclass with the fields of design section 1; `load_profile()`
reads `ops/k8s/profiles/<OUSAST_PLANE_PROFILE>.toml` (a name or a path; default `kind`), then applies reads `ops/k8s/profiles/<OUSAST_PLANE_PROFILE>.toml` (a name or a path; default `kind`), then applies
`OUSAST_<FIELD>` overrides; rejects an unknown key, a secret-looking key (`*_KEY`, `*_SECRET`), an `exec` `OUSAST_<FIELD>` overrides; rejects an unknown key, a secret-looking key (`*_KEY`, `*_SECRET`), an `exec`
@@ -19,7 +19,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
- `tests/test_plane_profile.py`: defaults; file then env precedence; each rejection names the field; both - `tests/test_plane_profile.py`: defaults; file then env precedence; each rejection names the field; both
committed profiles load; no field holds a value matching a credential pattern. committed profiles load; no field holds a value matching a credential pattern.
- _Requirements: 1.1_ - _Requirements: 1.1_
- [ ] 1.2 Literals out of `doctor.py`, `egress.py`, `router.py`, `generate.py` - [x] 1.2 Literals out of `doctor.py`, `egress.py`, `router.py`, `generate.py`
- `doctor(profile)`: context from `profile.kube_context`; the registry check resolves the `images` file's runner - `doctor(profile)`: context from `profile.kube_context`; the registry check resolves the `images` file's runner
reference (`docker manifest inspect` for a remote registry, `/v2/_catalog` for a local one); `open_store` reference (`docker manifest inspect` for a remote registry, `/v2/_catalog` for a local one); `open_store`
verifies `profile.memory`; prints every configured address (Req 1.2). `Egress(context=...)`, verifies `profile.memory`; prints every configured address (Req 1.2). `Egress(context=...)`,
@@ -29,46 +29,46 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
profile's context, never a built one. profile's context, never a built one.
- Evidence: `grep -rn "kind-" src/openultrasast/plane` matches only comments citing the kind profile. - Evidence: `grep -rn "kind-" src/openultrasast/plane` matches only comments citing the kind profile.
- _Requirements: 1.1, 1.2_ - _Requirements: 1.1, 1.2_
- [ ] 1.3 Host-literal scan test - [x] 1.3 Host-literal scan test
- `tests/test_plane_literals.py`: pattern built at runtime (as `tests/test_removed_plane_references.py` does) - `tests/test_plane_literals.py`: pattern built at runtime (as `tests/test_removed_plane_references.py` does)
over `src/`, `plane/`, `ops/` minus `ops/ax/up.sh` and `ops/k8s/profiles/`, and `docs/`, for `kind-ousast`, over `src/`, `plane/`, `ops/` minus `ops/ax/up.sh` and `ops/k8s/profiles/`, and `docs/`, for `kind-ousast`,
`localhost:5001`, `172.19.`, `127.0.0.1:` and `port-forward` outside `router.py` and lines tagged `localhost:5001`, `172.19.`, `127.0.0.1:` and `port-forward` outside `router.py` and lines tagged
`ax-tunnel`; fails listing `path:line`. `ax-tunnel`; fails listing `path:line`.
- Evidence: the test passes on this commit and fails when a literal is reinserted (shown once in the record). - Evidence: the test passes on this commit and fails when a literal is reinserted (shown once in the record).
- _Requirements: 1.3_ - _Requirements: 1.3_
- [ ] 1.4 `up.sh` by profile - [x] 1.4 `up.sh` by profile
- `ops/ax/up.sh`: `KO_DOCKER_REPO` and the context from `kind.toml` (read with `python -m - `ops/ax/up.sh`: `KO_DOCKER_REPO` and the context from `kind.toml` (read with `python -m
openultrasast.plane.profile --print registry|kube_context`); writes `kind-images.json` instead of openultrasast.plane.profile --print registry|kube_context`); writes `kind-images.json` instead of
`runner-image`; `generate.repin_templates` accepts that file (runner key only until task 4.3). `runner-image`; `generate.repin_templates` accepts that file (runner key only until task 4.3).
- Evidence: `ops/ax/up.sh` reruns idempotently on kind; `ops/ax/smoke-run.sh` passes. - Evidence: `ops/ax/up.sh` reruns idempotently on kind; `ops/ax/smoke-run.sh` passes.
- _Requirements: 1.1, 2.4_ - _Requirements: 1.1, 2.4_
- [ ] 1.5 Placement probe (open question 2, decided before the engine work) - [x] 1.5 Placement probe (open question 2, decided before the engine work) -- result: `placement: any`
- `ops/ax/probe-placement.sh`: applies a second WorkerPool `probe-pool` (1 replica, 3 Gi) from the design's - `ops/ax/probe-placement.sh`: applies a second WorkerPool `probe-pool` (1 replica, 3 Gi) from the design's
engine template, then one model-free `repo-facts` Task with `resources.requests.memory: 2560Mi`, and records engine template, then one model-free `repo-facts` Task with `resources.requests.memory: 2560Mi`, and records
with `kubectl ate get workers` which worker took the actor; then the same Task with 256Mi. Removed after. with `kubectl ate get workers` which worker took the actor; then the same Task with 256Mi. Removed after.
- Evidence: `record.json` says whether Substrate places by resource fit (`placement: fit|any`). `any` selects - Evidence: `record.json` says whether Substrate places by resource fit (`placement: fit|any`). `any` selects
the design's fallback for group 5 and is written into `design.md` section 4 before group 5 starts. the design's fallback for group 5 and is written into `design.md` section 4 before group 5 starts.
- _Requirements: 4.1_ - _Requirements: 4.1_
- [ ] 1.6 Reconciler bound (open question 7) - [x] 1.6 Reconciler bound (open question 7) -- 489 lines after 1.2; projected 514 during group 2: the bound moves to 550 in 2.4
- Record `wc -l src/openultrasast/plane/reconciler.py` (477 at 17ac7ad) and the lines task 2.4 may add - Record `wc -l src/openultrasast/plane/reconciler.py` (477 at 17ac7ad) and the lines task 2.4 may add
(receiver wiring removed: `-12`, `Delivery` calls: `+8` estimated). If the estimate crosses 500, the bound (receiver wiring removed: `-12`, `Delivery` calls: `+8` estimated). If the estimate crosses 500, the bound
in `tests/test_plane_reconciler.py:803` moves to 550 in task 2.4 with the reason in the assertion message. in `tests/test_plane_reconciler.py:803` moves to 550 in task 2.4 with the reason in the assertion message.
- Evidence: the numbers in `record.json`. - Evidence: the numbers in `record.json`.
- _Requirements: 2.1_ - _Requirements: 2.1_
- [ ] 1.7 gvisor tarball reachability (open question 3) **manual** - [x] 1.7 gvisor tarball reachability (open question 3) **manual** -- documented as planned in `docs/deployment.md` B.1; the live check waits for k3s
- `docs/deployment.md` (B.1, planned): the k3s node must fetch - `docs/deployment.md` (B.1, planned): the k3s node must fetch
`gs://gvisor/releases/nightly/2026-09-02/<arch>/gvisor.tar.zstd` (`substrate/manifests/ate-install/sandboxconfig-gvisor.yaml`) `gs://gvisor/releases/nightly/2026-09-02/<arch>/gvisor.tar.zstd` (`substrate/manifests/ate-install/sandboxconfig-gvisor.yaml`)
or a mirrored copy named in a copied SandboxConfig; the check is `kubectl -n ate-system logs` of the worker or a mirrored copy named in a copied SandboxConfig; the check is `kubectl -n ate-system logs` of the worker
pod's atelet fetch line, run by hand on the day k3s exists. Labelled planned until then. pod's atelet fetch line, run by hand on the day k3s exists. Labelled planned until then.
- _Requirements: 7.2_ - _Requirements: 7.2_
- [ ] 1.8 Kind proof and record 01 - [x] 1.8 Kind proof and record 01
- `ousast plane doctor --profile kind` passes every line; `ousast plane doctor --profile k3s` fails only on the - `ousast plane doctor --profile kind` passes every line; `ousast plane doctor --profile k3s` fails only on the
unreachable context (the expected line); `test_plane_literals` 0 offenders; the probe of 1.5; the bound of 1.6. unreachable context (the expected line); `test_plane_literals` 0 offenders; the probe of 1.5; the bound of 1.6.
- `benchmarks/measurements/<date>-k8s-01-profile/record.json`. - `benchmarks/measurements/<date>-k8s-01-profile/record.json`.
- _Requirements: 1.1, 1.2, 1.3, 7.1_ - _Requirements: 1.1, 1.2, 1.3, 7.1_
- [ ] 2. Store delivery, receiver-less egress (design section 3) - [ ] 2. Store delivery, receiver-less egress (design section 3)
- [ ] 2.1 Presigned URLs and the store run directory - [x] 2.1 Presigned URLs and the store run directory
- `src/openultrasast/plane/delivery.py`: `Delivery(store, run)` mints presigned PUT for - `src/openultrasast/plane/delivery.py`: `Delivery(store, run)` mints presigned PUT for
`runs/<run>/<task>/output.tar` and GET for `runs/<run>/<producer>/<artifact>` (boto3 `generate_presigned_url` `runs/<run>/<task>/output.tar` and GET for `runs/<run>/<producer>/<artifact>` (boto3 `generate_presigned_url`
on `S3Client.sdk`, expiry `OUSAST_TASK_TIMEOUT + 600`); `delivered(task)` HEADs the tar; `collect(task)` on `S3Client.sdk`, expiry `OUSAST_TASK_TIMEOUT + 600`); `delivered(task)` HEADs the tar; `collect(task)`
@@ -78,7 +78,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
- `tests/test_plane_delivery.py`: against a local S3-shaped fake (PUT/GET/HEAD by key); URLs never appear in - `tests/test_plane_delivery.py`: against a local S3-shaped fake (PUT/GET/HEAD by key); URLs never appear in
logs; expiry; member rejection; declared-only re-put; state seeding. logs; expiry; member rejection; declared-only re-put; state seeding.
- _Requirements: 3.1, 2.1_ - _Requirements: 3.1, 2.1_
- [ ] 2.2 Runner: `delivery` in the start body - [x] 2.2 Runner: `delivery` in the start body
- `src/openultrasast/plane/runner.py`: `StartGate.offer` validates an optional `delivery {put, inputs}` (https, - `src/openultrasast/plane/runner.py`: `StartGate.offer` validates an optional `delivery {put, inputs}` (https,
hostname, no echo); `_deliver_output` PUTs the tar to `delivery.put` (same retry and `DeliveryError` hostname, no echo); `_deliver_output` PUTs the tar to `delivery.put` (same retry and `DeliveryError`
semantics); `fetch_inputs` GETs from `delivery.inputs[name]`; `OUSAST_ARTIFACT_URL` and `OUSAST_ARTIFACT_DIAL` semantics); `fetch_inputs` GETs from `delivery.inputs[name]`; `OUSAST_ARTIFACT_URL` and `OUSAST_ARTIFACT_DIAL`
@@ -86,14 +86,14 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
- `tests/test_plane_runner.py`: accepted body with `delivery`; 400 on an `http` or IP URL; PUT to a local - `tests/test_plane_runner.py`: accepted body with `delivery`; 400 on an `http` or IP URL; PUT to a local
server; inputs fetched from presigned-shaped URLs; the marker and retry-only boot unchanged. server; inputs fetched from presigned-shaped URLs; the marker and retry-only boot unchanged.
- _Requirements: 3.1_ - _Requirements: 3.1_
- [ ] 2.3 Egress without the receiver - [x] 2.3 Egress without the receiver
- `src/openultrasast/plane/egress.py`: `policy_for(task, workspaces, model, store_host)` emits one - `src/openultrasast/plane/egress.py`: `policy_for(task, workspaces, model, store_host)` emits one
`tlsPassthrough` rule on 443 over the sorted union of store host, Git hosts, Model hosts; no `http` rule; an `tlsPassthrough` rule on 443 over the sorted union of store host, Git hosts, Model hosts; no `http` rule; an
IP store host raises naming `S3_ENDPOINT`. IP store host raises naming `S3_ENDPOINT`.
- `tests/test_plane_egress.py`: for every committed template under `plane/tasks/` the rendered policy is the - `tests/test_plane_egress.py`: for every committed template under `plane/tasks/` the rendered policy is the
expected set; the IP refusal. expected set; the IP refusal.
- _Requirements: 3.2_ - _Requirements: 3.2_
- [ ] 2.4 Reconciler on `Delivery` - [x] 2.4 Reconciler on `Delivery`
- `src/openultrasast/plane/reconciler.py`: `run()` opens the store first (no fallback), builds `Delivery`, - `src/openultrasast/plane/reconciler.py`: `run()` opens the store first (no fallback), builds `Delivery`,
passes `delivery` to `router.starter`/`start_task` (body extended in `router.py`), `_await` on passes `delivery` to `router.starter`/`start_task` (body extended in `router.py`), `_await` on
`Delivery.delivered`, outcome from the collected `summary.json`; the receiver thread is kept behind `Delivery.delivered`, outcome from the collected `summary.json`; the receiver thread is kept behind
@@ -101,7 +101,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
- `tests/test_plane_reconciler.py`: the ax fake run completes through the delivery fake; a rerun from an empty - `tests/test_plane_reconciler.py`: the ax fake run completes through the delivery fake; a rerun from an empty
results root resumes from the store's `state.json`. results root resumes from the store's `state.json`.
- _Requirements: 2.1, 3.1_ - _Requirements: 2.1, 3.1_
- [ ] 2.5 Kind proof and record 02 - [x] 2.5 Kind proof and record 02
- `ops/ax/smoke-run.sh` with `OUSAST_DELIVERY=store`: `facts.json` and `summary.json` arrive through the store; - `ops/ax/smoke-run.sh` with `OUSAST_DELIVERY=store`: `facts.json` and `summary.json` arrive through the store;
the actor's policy read back has one TLS rule naming the store host; `ousast plane status ax-e2e-smoke` the actor's policy read back has one TLS rule naming the store host; `ousast plane status ax-e2e-smoke`
prints the table. Instrument: the tar's byte size from the store equals the extracted files' total. prints the table. Instrument: the tar's byte size from the store equals the extracted files' total.
@@ -147,6 +147,43 @@ with Express-style routes and middleware, including vulnerable, fixed and benign
frontend/framework adaptations and new abstract capabilities, and shall report C/C++ envelope or frontend/framework adaptations and new abstract capabilities, and shall report C/C++ envelope or
unsupported-property results separately from demonstrated vulnerability detection. unsupported-property results separately from demonstrated vulnerability detection.
## Requirement 9: Operate as an advisory safety net on any repository
Added 2026-10-02, maintainer-approved, from the operational-readiness audit. The maintainer's
definition: a user installs the hook on ANY repository and gets honest, advisory-only results;
nothing ever blocks a push unless blocking was explicitly chosen.
9.1 (added 2026-10-02, maintainer-approved) When the installed hook cannot run the check, because the
`ousast` executable is missing or an argument or setting is invalid, the hook shall allow the push and
print one line saying the check did not run and how to fix it; only an explicit `--mode blocking`
opt-in may turn an analysis result into a rejected push.
9.2 (added 2026-10-02, maintainer-approved) When a check is skipped or incomplete, the terminal output
shall name each distinct reason in plain words, one line per reason, with repository paths decoded and
no opaque hash tokens; a language with no engine frontend and no quick rules shall be named with its
file count; a first push of a new branch shall say there was no base to compare against and what was
checked instead. The JSON artifact keeps the raw reason values.
9.3 (added 2026-10-02, maintainer-approved) Before the engine, within the same deadline, the Safety Net
shall run the shipped quick pattern rules over the head side of the changed lines and show their
findings as advisory pattern matches, labeled separately from admitted alerts. This amends 3.2 for this
tier only: quick-rule findings are shown, never admitted, and never block.
9.4 (added 2026-10-02, maintainer-approved) When a finding's operation lies in a file that is new in the
head revision, the comparison shall classify it as new rather than unknown; an engine finding whose
novelty is established but whose capability is not yet qualified shall be shown as an advisory engine
finding, not an admitted alert. Genuinely unknown comparisons keep their vetoes.
9.5 (added 2026-10-02, maintainer-approved) When the engine cannot finish within the deadline, one line
shall say so; on explicit request (`--engine background`) the Safety Net shall start the engine
detached after the push, record its result in the artifact directory, and show it on the next run as
the engine result for that head from the previous push.
9.6 (added 2026-10-02, maintainer-approved) The hook installer and remover shall ship in the wheel
(`ousast pre-push install|uninstall`), refuse to overwrite an existing hook unless forced, and a
hook-callable container wrapper shall pass the push input and artifact path through; the documentation
shall give a first-run guide, a glossary of skip reasons, a CI recipe and help text for every flag.
## Accepted experimental acceptance profile ## Accepted experimental acceptance profile
These values are the initial implementation/evaluation profile accepted for task generation, not These values are the initial implementation/evaluation profile accepted for task generation, not
+51 -2
View File
@@ -475,7 +475,7 @@ Preserve all completed task evidence below.
- _Depends: 13.1_ - _Depends: 13.1_
- _Requirements: 6.2, 6.3, 8.1, 8.2_ - _Requirements: 6.2, 6.3, 8.1, 8.2_
- [ ] 14. Ask every declared family whether it establishes anything (precondition for M4) - [x] 14. Ask every declared family whether it establishes anything (precondition for M4)
- Seven families are declared and two were proven end to end. A declared family that has never been asked whether it finds anything is advertised coverage, and M4 cannot qualify a capability that establishes nothing. - Seven families are declared and two were proven end to end. A declared family that has never been asked whether it finds anything is advertised coverage, and M4 cannot qualify a capability that establishes nothing.
- [x] 14.1 Census the seven declared families on NodeGoat and VAmPI - [x] 14.1 Census the seven declared families on NodeGoat and VAmPI
- Run production discovery and scan over each whole checkout and report, per family, the regions that offered it, the questions asked, the questions completed, and what was found with the witness and rung. No comparison, no novelty, no admission: detection only. - Run production discovery and scan over each whole checkout and report, per family, the regions that offered it, the questions asked, the questions completed, and what was found with the witness and rung. No comparison, no novelty, no admission: detection only.
@@ -575,7 +575,7 @@ Preserve all completed task evidence below.
- _Boundary: Evaluation harness, full-run integration_ - _Boundary: Evaluation harness, full-run integration_
- _Depends: 13.2, contributor-scan 7.3, contributor-scan 7.8_ - _Depends: 13.2, contributor-scan 7.3, contributor-scan 7.8_
- _Requirements: 6.2, 6.3, 8.1, 8.2_ - _Requirements: 6.2, 6.3, 8.1, 8.2_
- [ ] 13.4 Let a question inside one partition complete, whatever else the repository holds - [x] 13.4 Let a question inside one partition complete, whatever else the repository holds
- **Diagnosed 2026-09-21, and PHP was never the variable.** A repository holding TWO languages can never complete a question, so it can never admit a finding. `partitions.py` raises `cross_partition_semantics_unresolved` whenever more than one language partition exists; that reason is a member of `GRAPH_INTEGRITY_GAPS`; and a graph-integrity gap demotes every completed outcome in the scan. - **Diagnosed 2026-09-21, and PHP was never the variable.** A repository holding TWO languages can never complete a question, so it can never admit a finding. `partitions.py` raises `cross_partition_semantics_unresolved` whenever more than one language partition exists; that reason is a member of `GRAPH_INTEGRITY_GAPS`; and a graph-integrity gap demotes every completed outcome in the scan.
- The evidence is one column wide: NodeGoat is JavaScript alone and completes 299 of 299, VAmPI is Python alone and completes 35 of 35, PMPro is PHP with 36 regions of bundled JavaScript and completes 0 of 127 and 0 of 520 while establishing 19 and 43 findings. NodeGoat's own `vendor_semantics_unresolved` is a DECLARED exclusion rather than an integrity gap, which is why it does not demote. - The evidence is one column wide: NodeGoat is JavaScript alone and completes 299 of 299, VAmPI is Python alone and completes 35 of 35, PMPro is PHP with 36 regions of bundled JavaScript and completes 0 of 127 and 0 of 520 while establishing 19 and 43 findings. NodeGoat's own `vendor_semantics_unresolved` is a DECLARED exclusion rather than an integrity gap, which is why it does not demote.
- **This is larger than the PHP runtime limit and it lands on the declared v0.1 target.** A WordPress plugin is PHP plus its admin JavaScript, so every one of them is multi-language, so on the target this tool was built for it can detect defects and never admit one. Neither subject that currently passes could reveal it. - **This is larger than the PHP runtime limit and it lands on the declared v0.1 target.** A WordPress plugin is PHP plus its admin JavaScript, so every one of them is multi-language, so on the target this tool was built for it can detect defects and never admit one. Neither subject that currently passes could reveal it.
@@ -587,6 +587,7 @@ Preserve all completed task evidence below.
- _Three causes, not one, and the third was hidden by the record rather than by the code._ The same census gap is raised in TWO passes, arbitration and evidence, and carrying the partition in one but not the other left an unscoped copy that kept demoting everything, so the fix looked applied and was not. It was found by making the census print each outcome's reason instead of only its status, after which three separate causes stopped looking identical. - _Three causes, not one, and the third was hidden by the record rather than by the code._ The same census gap is raised in TWO passes, arbitration and evidence, and carrying the partition in one but not the other left an unscoped copy that kept demoting everything, so the fix looked applied and was not. It was found by making the census print each outcome's reason instead of only its status, after which three separate causes stopped looking identical.
- _Admission is scoped too._ `_blocking_degradations` already exempted declared exclusions; it now also ignores a gap another language partition raised, so a JavaScript build artefact cannot block a PHP finding. A gap naming no partition still reaches everything, and so does every gap when the claim's own language is unknown, because unknown scope is not evidence of independence. Evidence: `benchmarks/measurements/2026-09-21-partition-gap-blast-radius.json`. - _Admission is scoped too._ `_blocking_degradations` already exempted declared exclusions; it now also ignores a gap another language partition raised, so a JavaScript build artefact cannot block a PHP finding. A gap naming no partition still reaches everything, and so does every gap when the claim's own language is unknown, because unknown scope is not evidence of independence. Evidence: `benchmarks/measurements/2026-09-21-partition-gap-blast-radius.json`.
- _Boundary: contributor-scan query/context evidence, Ranker Scope Contract_ - _Boundary: contributor-scan query/context evidence, Ranker Scope Contract_
- _Checked 2026-10-02 per the operational-readiness audit:_ implemented and verified on PMPro (120 of 127, above). 13.3 stays open on its own blocker; this box was left unchecked only because it waits on it.
- _Depends: 13.3_ - _Depends: 13.3_
- _Requirements: 2.1, 2.2, 6.3, 8.1_ - _Requirements: 2.1, 2.2, 6.3, 8.1_
@@ -619,6 +620,54 @@ Preserve all completed task evidence below.
- _Requirements: 8.1, 8.2_ - _Requirements: 8.1, 8.2_
- [x] 17. Operational readiness (2026-10-02)
- **Named by the 2026-10-02 operational-readiness audit**, maintainer-approved as the first increment. The maintainer's definition: a user installs the hook on ANY repository and gets honest, advisory-only results; nothing ever blocks a push. The audit found the plumbing sound (snapshots, resolution, deadline accounting, advisory exit 0, the JSON artifact) and five things in the way: the hook can block a push, quick rules are not part of pre-push, the engine cannot finish a 26-file repository inside 30 s, the terminal hides every skip reason, and there is no supported install path without the source checkout. Before numbers for every item are the audit's live runs on records (Python), serve-static (JavaScript) and unrolled/secure (Go), each with one planted commit.
- This group is plumbing and honesty. It enables no capability, edits no rule for a planted sample, and leaves graph reuse (the engine speed work decided in 11.6) as a later task.
- _Done 2026-10-02:_ all seven subtasks done. The engine still does not fit the 30 s hook deadline (graph reuse, the 11.6 direction, stays a later task), and plane verdicts after the push were not in this increment.
- _Requirements: 9.1–9.6_
- [x] 17.1 Never block a push the user did not choose to block
- The installed hook exits 0 with one line when `ousast` is missing (the check did not run, and how to install). In hook (stdin) mode, an argument or usage error (an invalid `OUSAST_PUSH_DEADLINE`, a cache directory inside the repository) is advisory: one line and exit 0. `--mode blocking` stays the explicit opt-in that can reject a push, and keeps its usage errors. The prior-hook input-retention failure keeps exit 1, because it is the chained hook's verdict that cannot be obtained, not ours.
- Files: `ops/pre-push`, `src/openultrasast/cli.py`. Tests: `tests/test_push_hook.py` (a push with `ousast` absent from PATH succeeds and prints the line; a bad deadline in hook mode exits 0; blocking mode keeps exit 2).
- Evidence: the audit's `hook.sh` push 1 (rejected before) and `fail.sh` (rc=2 before) rerun with exit 0.
- _Done 2026-10-02:_ the hook prints one line and exits 0 (chaining a prior hook if present) when `ousast` (or `OUSAST_COMMAND`) is missing; in hook mode without `--mode blocking`, argparse/usage errors and unexpected exceptions print one line and exit 0. Tests: `test_missing_ousast_never_blocks_the_push`, `test_hook_mode_usage_errors_are_advisory_unless_blocking` (blocking keeps exit 2), and `test_cli_refuses_the_flag_outside_explicit_replay` updated (the experimental flag is still refused in hook mode, now advisory). Gates: 1987 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.1, 5.3, 5.4_
- [x] 17.2 Name every skipped check in plain words
- One terminal line per distinct coverage reason: hex-encoded paths decoded, sha256-suffixed reasons grouped with a count, JSON unchanged. "language not covered: <lang> (<n> files)" for a language with no engine frontend and no quick rules, in both `pre-push` and `scan`. A first push of a new branch says there is no base and what was checked instead. `reports.py` gains lines for the eight scan degradations `cli.py` records and the report did not render (`learning_endpoint_unavailable`, `cpg_unavailable`, `facts_unavailable`, `policy_invalid`, `model_withheld_execution_budget`, `max_findings_exceeded`, `hunter_model_unavailable`, `sandbox_unavailable`).
- Files: `src/openultrasast/push/report.py`, `src/openultrasast/push/explain.py` (new), `src/openultrasast/reports.py`, `src/openultrasast/cli.py`. Tests: `tests/test_push_report.py`, `tests/test_reports.py`.
- Evidence: the three repositories' terminal output names engine-missing, Go not covered and the new branch.
- _Done 2026-10-02:_ new `push/explain.py` turns every raw coverage reason into one plain `Skipped:` line per distinct kind (hex paths decoded, sha256 tokens dropped and counted, unknown reasons still printed); the existing one-line `Notice:` keeps its next action and gains an engine-missing variant. `language_not_covered:<lang>:<changed>:<total>` is recorded for a changed language with no engine frontend and no quick rules; `ousast scan` prints `notice: language not covered: go (N files)` and its report says so. `reports.py` renders the eight previously dropped degradations and prints any reason it has no sentence for. The hook passes `OUSAST_COMPARISON_BASE`. The new-branch line names the base it lacks; what was checked instead lands with the quick tier (17.3). Tests: `tests/test_push_explain.py` (6), two in `tests/test_reports.py`; module-audit manifest regenerated for the new module. Gates: 1995 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.2, 5.2, 8.2_
- [x] 17.3 Run the quick rules as the fast first tier
- Inside the deadline and before the engine, the shipped quick rules run over the head side of the changed files and keep the findings on changed lines or in changed functions. They print as advisory pattern matches, labeled apart from admitted alerts, recorded in the artifact's `quick_tier`, and never block.
- Files: `src/openultrasast/push/quick.py` (new), `src/openultrasast/push/runner.py`, `src/openultrasast/push/report.py`. Tests: `tests/test_push_quick.py` (new).
- Evidence: serve-static's planted `eval` in the new `debug.js` is shown; whether records' planted `%`-formatted SQL is shown is recorded as is, with no rule edits for it.
- _Done 2026-10-02:_ new `push/quick.py` runs the shipped quick rules (no shadow rules, no model) over the changed head files inside `_prepare` under the shared deadline, before the engine, keeping matches on a changed line or inside a changed function (entry-point ranges); it records `files`/`bytes_read` as proof of input. The terminal shows them under "Quick rules: N pattern match(es) on changed code ... Advisory: not verified by the engine."; the artifact keeps `quick_tier`; admission and push disposition are untouched. A new branch runs the quick tier over its last commit's changes and says so. `_analyze` gained `engine="off"`. On the audit's serve-static planted commit the `eval` in the new `debug.js` is now shown (js-eval, js-xss-response at debug.js:2, ~1.1 s on the host); the planted `%`-formatted SQL in records is NOT matched by any quick rule (coverage gap, no rule edited). Tests: `tests/test_push_quick.py` (3). Gates: 1998 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.3_
- [x] 17.4 Classify a file new in head as new, not unknown
- `ChangeContext` records the paths Git reports as added. An operation in an added file, when no add/delete pairing is unresolved and its head evidence is complete, is `new` with reason `file_added_in_head`, and admission accepts that reason as change evidence. An engine finding with established novelty whose only vetoes are capability vetoes renders as an advisory engine finding, not an alert. Every other unknown comparison keeps its veto.
- Files: `src/openultrasast/model/contracts.py`, `src/openultrasast/push/snapshot.py`, `src/openultrasast/push/policy.py`, `src/openultrasast/push/report.py`. Tests: `tests/test_push_policy.py`, `tests/test_push_snapshot.py`, `tests/test_push_report.py`.
- Evidence: unit tests on the comparison and admission; a live engine run if the host allows one (free memory >= 3 GB).
- _Done 2026-10-02:_ `ChangeContext.added_paths` records Git status-A paths (rename targets excluded). In `_compare_evidence`, after every existing context/head/deadline check, an operation in an added file is `new`/`file_added_in_head` when the claim is entailed, undischarged and has change evidence, and no base answer holds the same operation (else `operation_moved_within_change`); an unpaired add/delete still blocks as `change_context_unresolved`. Admission accepts `(new, file_added_in_head)`; with the empty registry such a candidate's only veto is `capability_unavailable`, and the terminal now shows it under "Engine: N finding(s) this push introduced. Advisory ... not an alert." Any comparison, evidence or location veto keeps a candidate out of that line. Tests: `tests/test_push_new_file.py` (9). Live engine evidence depends on host memory (17.7). Gates: 2007 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.4, 3.4_
- [x] 17.5 Say when the engine cannot finish, and offer it in the background
- One line when the engine did not finish (deadline, engine not installed). `--engine background` starts the engine detached after the quick tier, writes its result under the artifact directory and, on the next run, prints "engine result for <head> from the previous push". One background engine at a time; `--engine off` skips it with a stated line. No speed work.
- Files: `src/openultrasast/push/background.py` (new), `src/openultrasast/push/runner.py`, `src/openultrasast/cli.py`, `ops/pre-push`. Tests: `tests/test_push_background.py` (new).
- Evidence: a hook push with `OUSAST_PUSH_ENGINE=background` followed by a second push showing the previous result.
- _Done 2026-10-02:_ `--engine inline|background|off` (hook: `OUSAST_PUSH_ENGINE`) and `--background-deadline` (default 900 s). `off` and `background` run the quick tier and record `engine_off`/`engine_background` with a plain skip line; the deadline line names the deadline and the background option. New `push/background.py` starts one detached explicit replay (`start_new_session`, stdin closed, output to a log) per artifact directory and repository (lock file with a live-pid check: a second start reports busy), writes its artifact under `<artifact dir>/engine/<repo key>/<head>.json`, and the next run prints "Engine result for <head> from the previous push: ..." once, then marks it shown. Not usable through the Docker wrapper (the container exits with the hook); documented in 17.6. Tests: `tests/test_push_background.py` (4, one launching a real detached run). Gates: 2011 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.5, 4.1, 4.2_
- [x] 17.6 Install, Docker and documentation for a first-time user
- `ousast pre-push install|uninstall [--force] [--docker] [REPO]` ships the hook as package data, honours `core.hooksPath`, refuses an existing hook unless `--force` (which chains it as `pre-push.before-ousast`), and works from a pip install. `ops/ousast-docker` is a hook-callable script (not a shell function) that passes stdin and the artifact path through. The `openultrasast:dev` image is rebuilt only if `df -h /` shows at least 4 GB free; otherwise the rebuild command is documented and the record says it was not rebuilt. `docs/scanning.md` gains a first-run guide, a glossary of skip reasons, a GitHub Actions recipe and full `--help` text; README tiers 1 and 2 updated, pulling `ghcr.io/norandom/openultrasast:2.0.1` where the wrapper allows.
- Files: `src/openultrasast/push/hook/pre-push`, `src/openultrasast/push/install.py` (new), `src/openultrasast/cli.py`, `ops/ousast-docker`, `docs/scanning.md`, `README.md`, `ops/README.md`. Tests: `tests/test_push_install.py` (new), `mkdocs build --strict`.
- Evidence: install from the package into a scratch clone, push through it, uninstall restores the prior hook.
- _Done 2026-10-02:_ `ousast pre-push install|uninstall [REPO] [--force] [--docker]` (`push/install.py`) installs the hook shipped as package data (`openultrasast/push/hook/pre-push`, byte-identical to `ops/pre-push`, verified in the built wheel), honours `core.hooksPath` without editing it, refuses an existing hook unless `--force` (ours is replaced; a foreign hook moves to `pre-push.before-ousast` and is chained), and uninstall removes only a hook it wrote and restores the chained one. `ops/ousast-docker` (also package data; `--docker` installs it beside the hook, which then uses it) is a script that mounts the repository read-only and the artifact directory at their own absolute paths, passes stdin with `-i`, runs as the user with no network and `--memory 3g`, defaults to `ghcr.io/norandom/openultrasast:2.0.1`, never pulls inside a hook, and exits 0 unless `OUSAST_PUSH_MODE=blocking`. The hook passes `--engine`/`--comparison-base` only when set, so an older ousast or image still runs. Every pre-push flag has help text; `docs/scanning.md` has the first-run guide, settings table, new-branch and background notes, the artifact keys, a skip-reason glossary (a test keeps it in sync with `push/explain.py`), a GitHub Actions recipe and the full `--help` text; README tiers 1 and 2 and `ops/README.md` updated (pull and tag the GHCR image instead of building). **Image not rebuilt:** `df -h /` showed 1.1 GB free (< 4 GB); rebuild with `docker build -t openultrasast:dev .`. Tests: `tests/test_push_install.py` (11) and the glossary test. Gates: 2022 passed, 42 skipped; ruff, format, mypy clean; `mkdocs build --strict` exit 0.
- _Requirements: 9.6, 7.4_
- [x] 17.7 Prove it on the audit's three repositories
- Rerun records, serve-static and unrolled/secure (same planted commits) with the new build in quick mode and with the hook installed, plus one Docker engine run on serve-static if free memory >= 3 GB and free disk >= 1 GB. Prove each run read its input (files, bytes).
- Files: `benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json`. Evidence: before (the audit) and after: exit codes, wall time against the deadline, findings shown, every skip line printed. A planted sample still missed is a coverage gap, recorded, never a rule edit.
- _Done 2026-10-02:_ `benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json` (before = the audit, after = this build installed from its wheel into a scratch venv). 10 hook pushes and 6 replays, all exit 0, 0.69-1.21 s against 30 s on the host; the quick tier read exactly the changed file and every head snapshot read the full tree (records 43,577 B, serve-static 61,525 B, secure 103,023 B). The planted JS `eval` in the new `debug.js` surfaces: quick tier on the host, and as two advisory engine findings classified `new`/`file_added_in_head` in one Docker engine run (190.9 s against a 300 s deadline; MemAvailable 3.28 GB, 1.16 GB disk). **The planted Python `%`-formatted SQL is still missed**: no quick rule matches it and the host has no engine. Recorded as a coverage gap, no rule edited. Go is named as not covered. The proof caught a wrong install hint (`pip install openultrasast`, not on PyPI), corrected in the hook in this commit.
- _Requirements: 9.1–9.6, 6.3_
## Implementation Notes ## Implementation Notes
- Task5.2 measured PMPro timing out after both snapshots were materialized but before change context/graph analysis. Group6 must account for compatible snapshot/discovery reuse as well as graph reuse; identical-tip hits cannot substitute for the later representative changed-code latency gate. No new detector, scope heuristic or incremental CPG mutation is implied. - Task5.2 measured PMPro timing out after both snapshots were materialized but before change context/graph analysis. Group6 must account for compatible snapshot/discovery reuse as well as graph reuse; identical-tip hits cannot substitute for the later representative changed-code latency gate. No new detector, scope heuristic or incremental CPG mutation is implied.
+1 -1
View File
@@ -52,7 +52,7 @@ COPY benchmarks ./benchmarks
# The core install is zero-dependency by design; `semantic` adds the tree-sitter grammars the candidate # The core install is zero-dependency by design; `semantic` adds the tree-sitter grammars the candidate
# enumerator uses. The LLM endpoint stays optional and is configured by environment, never baked in. # enumerator uses. The LLM endpoint stays optional and is configured by environment, never baked in.
RUN python3 -m venv /venv \ RUN python3 -m venv /venv \
&& /venv/bin/pip install --no-cache-dir -e ".[semantic]" && /venv/bin/pip install --no-cache-dir -e ".[semantic,s3]"
ENV PATH="/venv/bin:${PATH}" ENV PATH="/venv/bin:${PATH}"
# Analysis never needs root. # Analysis never needs root.
+336 -145
View File
@@ -1,38 +1,53 @@
# OpenUltraSAST # OpenUltraSAST
OpenUltraSAST is an experimental static security analyser meant to work as a safety net on OpenUltraSAST is an experimental static security analyser. It is meant to be a safety net on
repositories it has never seen: it scans a checkout, or the commits a `git push` would publish, repositories it has never seen. It scans a checkout, or the commits a `git push` would publish. It
and separates what it can prove from what it merely suspects. A scan combines language-scoped keeps what it can prove apart from what it only suspects.
pattern rules, a tree-sitter overlay, a Joern code property graph and, optionally, a model that is
asked only the question the graph cannot settle; every finding carries the evidence that earned
its place. A scan that could not look at something records a degradation instead of reporting a
clean result. `ousast` is the interface; model work over whole repositories runs on a separate
agentic plane (google/ax), and a learned decision engine is in development. Neither is needed
for a scan.
This file is the front door; each section links to its page of the documentation site (`docs/`, A scan combines up to four parts:
`mkdocs.yml`; `uv sync --extra docs && uv run mkdocs serve`).
- language-scoped pattern rules;
- a tree-sitter overlay (a syntax tree of each file, built by the tree-sitter parser);
- a Joern code property graph (a graph of the program's syntax, control flow and data flow);
- optionally, a model. It is asked only the question the graph cannot settle.
Every finding carries the evidence that earned its place. When a scan could not look at
something, it records a degradation. It does not report a clean result.
`ousast` is the interface. Model work over whole repositories runs on a separate agentic plane
(google/ax). A learned decision engine is in development. A scan needs neither of them.
This file is the front door. Each section links to its page of the documentation site.
The site is published at <https://norandom.github.io/OpenUltraSAST/>. Its sources are in `docs/` and
`mkdocs.yml`; to preview it locally, run `uv sync --extra docs && uv run mkdocs serve`.
## Status ## Status
**v2.0.0, state as of 2026-10-02.** The pre-push safety net is **NO-GO** against every M4 gate: **v2.0.0, state as of 2026-10-02.** The pre-push safety net is **NO-GO** against every M4 gate.
no hook capability is enabled, and the two independent populations spent so far gave engine recall
1 of 11 and then 0 of 17. The learned decision engine ranks but may not block: BLOCK is offered for - No hook capability is enabled.
no family. The project has stopped trying to reach the gates by hand; every detection change now - Two independent populations (case sets frozen before any scan) have been spent so far. Engine
has to win a controlled measurement on held-out repositories. The one page that states all of this recall was 1 of 11 and then 0 of 17.
with the record behind each figure is [docs/where-we-stand.md](docs/where-we-stand.md); the - The learned decision engine ranks but may not block. BLOCK is offered for no family.
release history is in [RELEASE_NOTES.md](RELEASE_NOTES.md).
The project has stopped trying to reach the gates by hand. Every detection change now has to win
a controlled measurement on held-out repositories. One page states all of this, with the record
behind each figure: [docs/where-we-stand.md](docs/where-we-stand.md). The release history is in
[RELEASE_NOTES.md](RELEASE_NOTES.md).
## Install and first scan ## Install and first scan
Which tier you need: a deterministic scan of a checkout or a CI gate is tier 1; the Joern engine Pick the tier you need:
(`standard`, `deep`) is tier 2; running model work over whole repositories on the plane is tier 3,
for maintainers; tier 4 does not exist yet. 1. Tier 1: a deterministic scan of a checkout, or a CI gate.
2. Tier 2: the Joern engine (`standard`, `deep`).
3. Tier 3, for maintainers: model work over whole repositories on the plane.
4. Tier 4 does not exist yet.
### Tier 1: quick scan, no Docker ### Tier 1: quick scan, no Docker
The core install depends only on PyYAML. From a checkout, `uv sync` (or `pip install .`) installs The core install depends only on PyYAML. From a checkout, `uv sync` (or `pip install .`) installs
the `ousast` entry point; `uv sync --group dev` adds the test and lint tools. the `ousast` entry point. `uv sync --group dev` adds the test and lint tools.
```bash ```bash
uv sync uv sync
@@ -41,19 +56,41 @@ uv run ousast scan . --mode quick --fail-on verified # CI: exit 1 on any evide
``` ```
This runs the language-scoped pattern rules, entry-point reachability hints, ranking, This runs the language-scoped pattern rules, entry-point reachability hints, ranking,
verification and scoring: deterministic, reproducible, no network, no keys. What it cannot do: verification and scoring. It is deterministic and reproducible. It needs no network and no keys.
there is no engine, so nothing follows a value across statements or files, and the PHP quick rules
are a low-precision, in-sample development set (per-rule precision lower bounds from 0.03 to 1.0 on The same install gives you the advisory pre-push hook for any repository. It runs the quick rules
the real pins, `benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). The optional on the lines a push changes, names every check it could not run, and never blocks a push:
`semantic` extra (`uv sync --extra semantic`) installs the tree-sitter grammars for the overlay.
```bash
cd /path/to/your/repository && ousast pre-push install # remove: ousast pre-push uninstall
```
What it cannot do:
- There is no engine. Nothing follows a value across statements or files.
- The PHP quick rules are a low-precision, in-sample development set. Per-rule precision lower
bounds run from 0.03 to 1.0 on the real pins
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`).
The optional `semantic` extra (`uv sync --extra semantic`) installs the tree-sitter grammars for
the overlay.
### Tier 2: standard and deep scans (Docker) ### Tier 2: standard and deep scans (Docker)
`standard` and `deep` need Joern. The Docker image `openultrasast:dev` ships the tool, Joern `standard` and `deep` need Joern. The Docker image ships three things in one image: the tool,
v4.0.625 and `php-cli` (needed by the PHP frontend) in one image; it is built locally from the Joern v4.0.625 and `php-cli` (the PHP frontend needs it). Each release publishes it to GHCR. It is
`Dockerfile` (no published image yet), and it is big: a JRE plus about 2 GB of Joern. The shell big: a JRE plus about 2 GB of Joern. Pull it instead of building, and tag it with the name the
wrapper builds it on first use and then runs `ousast` in the container with your directory mounted shell wrapper uses:
read-only and the container network off:
```bash
docker pull ghcr.io/norandom/openultrasast:2.0.1
docker tag ghcr.io/norandom/openultrasast:2.0.1 openultrasast:dev # or build: docker compose build
```
The shell wrapper runs `ousast` in the container. It builds the image only if `openultrasast:dev`
is missing. Your directory is mounted read-only and the container network is off. For the pre-push
hook, use `ousast pre-push install --docker`: Git cannot call a shell function, so it installs the
script `ousast-docker` beside the hook.
```bash ```bash
source /path/to/OpenUltraSAST/ops/shell/ousast.sh # PowerShell: ops/shell/ousast.ps1 source /path/to/OpenUltraSAST/ops/shell/ousast.sh # PowerShell: ops/shell/ousast.ps1
@@ -62,90 +99,132 @@ ousast-with-judge scan . --mode standard # network on, the model is a
``` ```
`docker compose build` and `TARGET=/path/to/repo docker compose run --rm ousast scan /target` do `docker compose build` and `TARGET=/path/to/repo docker compose run --rm ousast scan /target` do
the same without the wrapper. `deep` adds REGRESS, which runs each reproduction snippet with the same without the wrapper.
`docker run` from wherever the scan runs; the engine container has no Docker socket, so run `deep`
from a host install with Joern on the PATH (`pyinfra @local ops/joern.py`, pinned and checksum- `deep` adds REGRESS. REGRESS runs each reproduction snippet with `docker run` from wherever the
verified) and a working `docker`, or accept the recorded `sandbox_unavailable` degradation. scan runs. The engine container has no Docker socket. So run `deep` in one of two ways:
- from a host install with Joern on the PATH (`pyinfra @local ops/joern.py`, pinned and checksum-
verified) and a working `docker`;
- or accept the recorded `sandbox_unavailable` degradation.
Details, including the host install, are in [ops/README.md](ops/README.md). Details, including the host install, are in [ops/README.md](ops/README.md).
Keys and store settings live in `.env` in the working directory (gitignored; `.env.example` lists Keys and store settings live in `.env` in the working directory. The file is gitignored;
the names). `DEEPSEEK_API_KEY` serves every LLM call; `OPENROUTER_API_KEY` is used for embeddings `.env.example` lists the names.
only (`ousast learn memory embed`). `.env` never overrides a variable already exported in your
shell, so `unset` a stale export when a key change seems to have no effect. The plane's memory - `DEEPSEEK_API_KEY` serves every LLM call.
store defaults to a local file store under `~/ousast-results/plane/memory`; an S3-compatible bucket - `OPENROUTER_API_KEY` is used for embeddings only (`ousast learn memory embed`).
is optional (`uv sync --extra s3`, then `OUSAST_MEMORY=s3://<bucket>` with `S3_ENDPOINT` and the - `.env` never overrides a variable already exported in your shell. If a key change seems to have
AWS credentials in `.env`). Nothing in a scan needs it. no effect, `unset` the stale export.
- The plane's memory store defaults to a local file store under `~/ousast-results/plane/memory`.
- An S3-compatible bucket is optional: `uv sync --extra s3`, then `OUSAST_MEMORY=s3://<bucket>`
with `S3_ENDPOINT` and the AWS credentials in `.env`. Nothing in a scan needs it.
### Tier 3: the plane on ax (maintainers) ### Tier 3: the plane on ax (maintainers)
None of this is needed for tier 1 or 2. Today's profile is a single-node kind cluster on the Tier 1 and 2 need none of this. Today's profile is a single-node kind cluster on the maintainer's
maintainer's host. `ops/ax/up.sh` needs `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` on the host.
PATH (it refuses to start when one is missing) and builds Agent Substrate and ax from their
checkouts under `~/.cache/ousast/ax-src/`. Measured idle footprint with two workers: the kind node - `ops/ax/up.sh` needs `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` on the PATH. It refuses to
takes 1.7 GiB of memory (27% of the 7.7 GiB host) and 4.9 GB of disk for images, with several GB start when one is missing.
more of Go caches during the build; the worker pool is sized for that 7 GB host. The verify and - It builds Agent Substrate and ax from their checkouts under `~/.cache/ousast/ax-src/`.
roles tasks need `DEEPSEEK_API_KEY`, and memory lives in the local file store or the S3 store - Measured idle footprint with two workers: the kind node takes 1.7 GiB of memory (27% of the
(RustFS is the tested server). Bring-up, doctor and the smoke Run: [ops/ax/README.md](ops/ax/README.md). 7.7 GiB host) and 4.9 GB of disk for images. The build uses several GB more for Go caches.
- The worker pool is sized for that 7 GB host.
- The verify and roles tasks need `DEEPSEEK_API_KEY`.
- Memory lives in the local file store or the S3 store. RustFS is the tested server.
Bring-up, doctor and the smoke Run: [ops/ax/README.md](ops/ax/README.md).
### Tier 4: Kubernetes (k3s), planned ### Tier 4: Kubernetes (k3s), planned
No deployment to any cluster other than that kind cluster has been made. The target is two Nothing has been deployed to any cluster other than that kind cluster. The target is two profiles
profiles of one code base, `kind` locally and `k3s` in production, where the same `ousast plane run` of one code base: `kind` locally and `k3s` in production. The same `ousast plane run` would
submits a Run in a remote execution mode: it needs a kubeconfig context or the cluster's ingress submit a Run in a remote execution mode. It needs:
(and an `imagePullSecret` on the worker ServiceAccount for a private package), runner and engine
images published to GHCR and pinned by digest, and the S3 store as the artifact path, with no - a kubeconfig context or the cluster's ingress (and an `imagePullSecret` on the worker
receiver on a laptop. The plan, and the code changes it needs with file paths, are in ServiceAccount for a private package);
- runner and engine images published to GHCR and pinned by digest;
- the S3 store as the artifact path, with no receiver on a laptop.
The plan, and the code changes it needs with file paths, are in
[docs/deployment.md](docs/deployment.md#production-topology-ax-on-a-kubernetes-cluster). [docs/deployment.md](docs/deployment.md#production-topology-ax-on-a-kubernetes-cluster).
## Scan modes ## Scan modes
`quick` runs the pattern rules, hints, ranking, verification and score and needs nothing. There are three modes. Each adds to the one before.
`standard` adds the MAP stage: complexity map, semantic overlay, authorization obligations and the
Joern model layer, which builds one code property graph per repository and decides taint, - `quick` runs the pattern rules, hints, ranking, verification and score. It needs nothing.
guard-dominance and configuration questions on it; a model is asked only the residual question - `standard` adds the MAP stage: complexity map, semantic overlay, authorization obligations and
(without a key it is skipped and only the graph's entailed findings are reported). `deep` adds the Joern model layer. The model layer builds one code property graph per repository. It decides
REGRESS: promoted candidates are loaded by a small snippet in a Docker sandbox and get a taint, guard-dominance and configuration questions on that graph. A model is asked only the
`triggerable` or `not_triggerable` verdict. A missing engine is a recorded degradation residual question. Without a key, that question is skipped and only the graph's entailed
(`cpg_unavailable`, `sandbox_unavailable`), never a clean result. findings are reported.
- `deep` adds REGRESS. Promoted candidates are loaded by a small snippet in a Docker sandbox. Each
gets a `triggerable` or `not_triggerable` verdict.
A missing engine is a recorded degradation (`cpg_unavailable`, `sandbox_unavailable`), never a
clean result.
`--fail-on never|findings|verified|worth-fixing` sets the exit code. `--config openultrasast.toml`
loads settings. Every run writes these files under `<target>/.openultrasast/runs/<scan-id>/`:
`manifest.json`, `findings.json`, `verification.json`, `score.json`, `report.md`, `report.sarif`
and `trace/events.jsonl`.
`ousast pre-push` (experimental) analyses the commits a push would publish against their base.
First, the quick rules run on the changed lines. Then the engine compares head with base, and only
new or worsened defects become candidates. It is advisory by default, and it prints one line per
check it could not run. Its capability registry is empty, so it emits no normal alert today. An
engine finding the push introduced is shown as advisory. Install, settings, skip reasons and a CI
recipe: [docs/scanning.md](docs/scanning.md#first-run-install-the-hook-on-any-repository).
`--fail-on never|findings|verified|worth-fixing` sets the exit code and `--config openultrasast.toml`
loads settings. Every run writes `manifest.json`, `findings.json`, `verification.json`,
`score.json`, `report.md`, `report.sarif` and `trace/events.jsonl` under
`<target>/.openultrasast/runs/<scan-id>/`. `ousast pre-push` (experimental) analyses the commits a
push would publish against their base so that only new or worsened defects are candidates; it is
advisory by default and its capability registry is empty, so no normal alert is emitted today.
Flows and the per-language rule table: [docs/scanning.md](docs/scanning.md). Flows and the per-language rule table: [docs/scanning.md](docs/scanning.md).
## How detection works ## How detection works
Three representations, from cheap to expensive: text (regex quick rules, one statement at a time), The tool uses three representations of code, from cheap to expensive:
tree (the tree-sitter CST and an IR per function, for the overlay and the obligations) and graph
(the Joern CPG, the only one that follows a value across statements, calls and files). Taint runs
as fact tables, queries and verdicts on the graph; absence bugs (an operation that should have
been guarded) are found as obligations, dominance and configuration census; a model answers only
the residual question. [docs/detection-techniques.md](docs/detection-techniques.md) explains each
technique with the code it comes from and what it misses;
[docs/architecture.md](docs/architecture.md) gives the stage order, the evidence ladder (a state
machine, not a label a model may assign), the central CWE policy and the project score, and the
self-improving loops (`ousast improve`, with `--memory` for proposals from plane rows).
Quick rules cover C/C++, Python, JavaScript/TypeScript, Java with Groovy templates, and PHP, and 1. Text: regex quick rules, one statement at a time.
run only against files of their language. The detection gate (`uv run python -m openultrasast.gate`) 2. Tree: the tree-sitter CST (concrete syntax tree) and an IR (intermediate representation) per
enforces at least 90% recall and under 10% false positives per language on the bundled cheat-sheet function. The overlay and the obligations use it.
corpora; on this tree it reports 93.62% recall (44/47) and 0.0% false positives. Those corpora show 3. Graph: the Joern CPG. It is the only one that follows a value across statements, calls and
that a rule change did not break a known detection; they are not a recall estimate on real code, files.
and PHP is not in the gate. Framework knowledge (WordPress, Flask, Django, Express, Spring) is
tagged as an optional prior in `src/openultrasast/ruleset/frameworks.toml`, so its contribution can Taint runs as fact tables, queries and verdicts on the graph. Absence bugs are operations that
be measured and switched off. should have been guarded. They are found as obligations, dominance and configuration census. A
model answers only the residual question.
- [docs/detection-techniques.md](docs/detection-techniques.md) explains each technique, with the
code it comes from and what it misses.
- [docs/architecture.md](docs/architecture.md) gives the stage order and the evidence ladder (a
state machine, not a label a model may assign). It also covers the central CWE policy, the
project score and the self-improving loops (`ousast improve`, with `--memory` for proposals from
plane rows).
Quick rules cover C/C++, Python, JavaScript/TypeScript, Java with Groovy templates, and PHP. Each
rule runs only against files of its language.
The detection gate (`uv run python -m openultrasast.gate`) enforces at least 90% recall and under
10% false positives per language on the bundled cheat-sheet corpora. On this tree it reports
93.62% recall (44/47) and 0.0% false positives. Those corpora show that a rule change did not
break a known detection. They are not a recall estimate on real code, and PHP is not in the gate.
Framework knowledge (WordPress, Flask, Django, Express, Spring) is tagged as an optional prior in
`src/openultrasast/ruleset/frameworks.toml`. So its contribution can be measured and switched off.
## The plane on ax ## The plane on ax
Model-driven work over whole repositories runs as a Run manifest, a DAG of ax Tasks, each in its Model-driven work over whole repositories runs as a Run manifest. A Run is a DAG of ax Tasks. Each
own gVisor actor on Agent Substrate. ax is the only agentic executor; there is no local Task runs in its own gVisor actor (an isolated sandbox) on Agent Substrate. ax is the only agentic
subprocess path. Each Task binds at most one Model and its own `usd`/`calls` budget, egress is executor; there is no local subprocess path.
deny-by-default per Task (hostnames only), and the provider key travels only in the Task's start
request, never in a manifest or an image. Measured on the first increment: the plane agreed on 16 - Each Task binds at most one Model and its own `usd`/`calls` budget.
of 20 declared sites at $0.0205 per candidate. - Egress is deny-by-default per Task, by hostname only.
- The provider key travels only in the Task's start request, never in a manifest or an image.
Measured on the first increment: the plane agreed on 16 of 20 declared sites at $0.0205 per
candidate.
```bash ```bash
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
@@ -155,42 +234,61 @@ uv run ousast plane remember validation-46 # ingest the run's rows
``` ```
`workspaces`, `harvest`, `alerts-engine` and `memory-normalise` generate Runs, their inputs and `workspaces`, `harvest`, `alerts-engine` and `memory-normalise` generate Runs, their inputs and
the store's rewrites. The flow and the task catalogue: [docs/plane.md](docs/plane.md); what runs the store's rewrites. The flow and the task catalogue: [docs/plane.md](docs/plane.md). What runs
where today and what a separate cluster would take: [docs/deployment.md](docs/deployment.md). where today, and what a separate cluster would take: [docs/deployment.md](docs/deployment.md).
HarnessX, the earlier agentic extra, was retired 2026-09-30: a `[harnessx]` section is ignored HarnessX, the earlier agentic extra, was retired 2026-09-30. A `[harnessx]` section is ignored
with one warning, and `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail with one warning. `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail and
naming the plane replacement, as do config keys nothing ever read. Fusion is deterministic. name the plane replacement. Config keys that nothing ever read fail the same way. Fusion is
deterministic.
## Memory ## Memory
One store, keyed by repository and pin, holds what plane runs and the decision engine learned: One store holds what plane runs and the decision engine learned. It is keyed by repository and pin.
rows (facts, verdicts, unit costs, alerts, coverage, feature records) and content-addressed blobs It holds two kinds of data:
(excerpts, embeddings, cached responses, compiled programs). `OUSAST_MEMORY` selects the backend:
a local file store, or `s3://<bucket>` on any S3-compatible server with versioning, lifecycle - rows: facts, verdicts, unit costs, alerts, coverage and feature records;
rules, object tags and S3 Select; RustFS is the tested server. An admin sets the bucket up once; - content-addressed blobs: excerpts, embeddings, cached responses and compiled programs.
the store verifies it at every start and refuses to run, naming each missing piece, rather than
fall back. Layout and the read and write paths: [docs/memory.md](docs/memory.md); the bucket `OUSAST_MEMORY` selects the backend. It is either a local file store or `s3://<bucket>` on any
policy and admin steps: [docs/rustfs.md](docs/rustfs.md); how stored rows are meant to turn into S3-compatible server. The S3 server must support versioning, lifecycle rules, object tags and S3
better decisions: [docs/memory-and-detection.md](docs/memory-and-detection.md). Select. RustFS is the tested server.
An admin sets the bucket up once. The store verifies it at every start. If anything is missing, it
refuses to run and names each missing piece. It does not fall back.
- Layout and the read and write paths: [docs/memory.md](docs/memory.md).
- The bucket policy and admin steps: [docs/rustfs.md](docs/rustfs.md).
- How stored rows are meant to turn into better decisions:
[docs/memory-and-detection.md](docs/memory-and-detection.md).
## Decision engine and evaluation ## Decision engine and evaluation
A learned decision engine is being built to replace hand-tuned detection edits: every instrument A learned decision engine is being built. It is meant to replace hand-tuned detection edits. It
produces signals, never verdicts; labels come from ground truth only; a compiled AI classifier with follows three rules:
local memory turns retrieved similar cases and the candidate's signals into a calibrated
probability with BLOCK and ADVISORY operating points. It is **in development, not adopted**: no
scan, pre-push check or report uses it. Measured on six families, on repositories the compiled
program was not shown, the pooled AUC of its score runs from 0.77 (injection) to 0.96 (access
control); the stricter within-pair AUC is 0.72 to 0.93; BLOCK is offered for no family
(`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`). Maintainer surface:
`ousast learn labels|memory|compile|evaluate|curve|experiment|audit-leaks`. Design, slices and
the A/B protocol: [docs/decision-engine.md](docs/decision-engine.md).
How anything is measured on code the tool was not tuned on, the pair corpora, the independent - Every instrument produces signals, never verdicts.
populations and the M4 gates, and the fold discipline that keeps learned decisions out of their - Labels come from ground truth only.
own training data: [docs/evaluation.md](docs/evaluation.md). Population v3 (PHP, 15 cases) is - A compiled AI classifier with local memory turns retrieved similar cases and the candidate's
frozen as the one-time final check and nothing in the tree may read it. signals into a calibrated probability. It has BLOCK and ADVISORY operating points.
It is **in development, not adopted**. No scan, pre-push check or report uses it.
It was measured on six families, on repositories the compiled program was not shown:
- The pooled AUC (area under the ROC curve) of its score runs from 0.77 (injection) to 0.96
(access control).
- The stricter within-pair AUC is 0.72 to 0.93 across the same families.
- BLOCK is offered for no family
(`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`).
Maintainer surface: `ousast learn labels|memory|compile|evaluate|curve|experiment|audit-leaks`.
Design, slices and the A/B protocol: [docs/decision-engine.md](docs/decision-engine.md).
[docs/evaluation.md](docs/evaluation.md) explains how anything is measured on code the tool was not
tuned on. It covers the pair corpora, the independent populations and the M4 gates. It also covers
the fold discipline that keeps learned decisions out of their own training data. Population v3
(PHP, 15 cases) is frozen as the one-time final check. Nothing in the tree may read it.
```bash ```bash
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
@@ -201,19 +299,26 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dr
## Token economics ## Token economics
Model tokens are the one running cost. At runtime the plane routes them: one Model and one budget Model tokens are the one running cost. At runtime the plane routes them:
per Task, a metered client that refuses the next call at the ceiling, repository facts computed
model-free and reused, blobs replayed at $0, triage before the hunt and a tie-break only on - one Model and one budget per Task;
disputed candidates. In development, 85.5% of the maintainers' coding-session tokens were tool - a metered client that refuses the next call at the ceiling;
input and output rather than reasoning (down from 90%), and the remedies in `benchmarks/dev/` - repository facts computed without a model and reused;
target that. Mechanisms, - blobs replayed at $0, with no new model call;
the measured costs and the before-and-after table: [docs/token-ergonomics.md](docs/token-ergonomics.md). - triage before the hunt;
- a tie-break only on disputed candidates.
In development, 85.5% of the maintainers' coding-session tokens were tool input and output rather
than reasoning. That is down from 90%. The remedies in `benchmarks/dev/` target that share.
Mechanisms, the measured costs and the before-and-after table:
[docs/token-ergonomics.md](docs/token-ergonomics.md).
## Examples, threat model ## Examples, threat model
End-to-end walkthroughs, from a quick scan to an MCP client: [docs/examples.md](docs/examples.md). - End-to-end walkthroughs, from a quick scan to an MCP client: [docs/examples.md](docs/examples.md).
What the tool trusts and what it does not (scanned code is untrusted input, secret redaction, - What the tool trusts and what it does not: [docs/threat-model.md](docs/threat-model.md). This
per-task egress, budgets, the `deep` sandbox): [docs/threat-model.md](docs/threat-model.md). covers scanned code as untrusted input, secret redaction, per-task egress, budgets and the
`deep` sandbox.
## Contributing and maintainer tooling ## Contributing and maintainer tooling
@@ -228,25 +333,111 @@ uv run python dagger/ci.py # containerized CI pipeline
``` ```
An agent that can run shell commands needs nothing but the CLI. For [OpenCode](https://opencode.ai) An agent that can run shell commands needs nothing but the CLI. For [OpenCode](https://opencode.ai)
there are two project skills in `.agents/skills/`: `openultrasast-scan` (modes and their there are two project skills in `.agents/skills/`:
prerequisites, `--fail-on`, the evidence ladder, recorded degradations, run artifacts) and
`openultrasast-triage` (false-positive reasons, what an adjudication may feed, fixing and - `openultrasast-scan`: modes and their prerequisites, `--fail-on`, the evidence ladder, recorded
re-scanning). `ousast mcp` runs a narrow MCP server over stdio with ten tools (`openultrasast.scan`, degradations and run artifacts.
`status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`, `explain`, - `openultrasast-triage`: false-positive reasons, what an adjudication may feed, and fixing and
`propose_patch`, `export_report`); no tool runs a shell, Docker or a free-form command: re-scanning.
`ousast mcp` runs a narrow MCP (Model Context Protocol) server over stdio. It has ten tools:
`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`,
`explain`, `propose_patch` and `export_report`. No tool runs a shell, Docker or a free-form
command.
```jsonc ```jsonc
{ "command": "uv", "args": ["run", "ousast", "mcp"] } { "command": "uv", "args": ["run", "ousast", "mcp"] }
``` ```
The `kiro-*` skills, `AGENTS.md` and the Kiro-style specifications under `.kiro/` are the The `kiro-*` skills, `AGENTS.md` and the Kiro-style specifications under `.kiro/` are the
maintainer's development tooling and can be ignored by users. Other maintainer commands: maintainer's development tooling. Users can ignore them. Other maintainer commands:
`ousast repos` (pinned known-vulnerable checkouts) and `ousast model candidates` (what the
candidate enumerator can reach). - `ousast repos`: pinned known-vulnerable checkouts.
- `ousast model candidates`: what the candidate enumerator can reach.
## Licence ## Licence
OpenUltraSAST is licensed under the [Apache License 2.0](LICENSE). Third-party material keeps its OpenUltraSAST is licensed under the [Apache License 2.0](LICENSE). Third-party material keeps its
own terms: Joern (Apache-2.0) is downloaded at image build time, the tree-sitter wheels of the own terms:
`semantic` extra are MIT, and the published benchmark cases were licence-checked one by one (v1's
OpenCVE case is BUSL-1.1, see [RELEASE_NOTES.md](RELEASE_NOTES.md)). - Joern (Apache-2.0) is downloaded at image build time.
- The tree-sitter wheels of the `semantic` extra are MIT.
- The published benchmark cases were licence-checked one by one. v1's OpenCVE case is BUSL-1.1, see [RELEASE_NOTES.md](RELEASE_NOTES.md).
## Engine traces on kube-ax
`benchmarks/learn/engine_trace.py --executor ax` submits AX v0.3.1 Tasks with
`ax apply -f -` and removes them with `ax delete task NAME`. The operator supplies
kubeconfig and AX access; the dispatcher creates no Kubernetes resources or secrets.
The `ousast-engine-task` GHCR package **must be made Public once** in its package
settings so the actors can pull it without registry credentials.
The task image contains the current engine and worker code. Rebuild it after code
changes and pass its digest with `--image`; no analyzer download is needed. The host
materializes each pin as usual and uploads source and prepared questions under
`engine-queue/<run>/<pin>/`. Only object-scoped presigned URLs and deadline parameters
enter Task env. Credentials stay on the coordinator. Configure `OUSAST_MEMORY=s3://…`
and the store credentials in the coordinator's `.env` or environment, with
`S3_ENDPOINT=https://files.because-security.com`. Presigned links expire after the
pin deadline plus ten minutes. IP and HTTP endpoints are rejected: the operator's
actor EgressPolicy must allow **files.because-security.com:443**.
Completion is the result object, even while AX reports Running. Failed phases and
host deadlines produce explicit failure/timeout rows. Task and staging-object cleanup
runs on success and failure; cleanup failure stops further dispatch. Per-pin files,
`--rerun-status`, resume and `progress.json` retain the existing trace-runner behavior.
AX results include `executor: "ax"`, `worker_ip`, and the image digest. Placement is
random; `--parallel` defaults to two AX tasks and cannot exceed two.
AX workers use `--heap-profile cluster` with
`-Xms1700m -Xmx1700m -XX:MaxMetaspaceSize=256m -XX:ReservedCodeCacheSize=128m -XX:MaxDirectMemorySize=256m -Xss512k -XX:ActiveProcessorCount=1 -XX:+UseG1GC`
on every frontend/query JVM; instruments record the flags. A `JVM out of memory`
result is automatically re-queued to one local Docker lane with `--heap-profile vm`
(2560m), recording `fallback: "vm-oom"` and the AX attempt. `mixed-ax` also uses this
one Docker lane for overflow, prioritizing pending OOM fallbacks. Thus a mixed run has
up to two AX tasks plus one Docker container. Use `--docker-image` for the locally
available VM image; Docker never pulls it implicitly. A STOP file prevents new
launches; interrupted fallback queues are recovered on resume.
Coordinator commands, from this worktree (the AX binary and kubeconfig remain
configurable). Build/push the actual uncommitted tree and read its digest:
```bash
cd /home/mc/Source/OpenUltraSAST/.claude/worktrees/engine-ax
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
export PYTHONPATH="$PWD/src"
TASK_TAG="ghcr.io/norandom/ousast-engine-task:sha-$(git rev-parse --short=7 HEAD)"
docker buildx build --platform linux/amd64 -f plane/Dockerfile.engine-task \
-t "$TASK_TAG" -t ghcr.io/norandom/ousast-engine-task:main \
--push --metadata-file /tmp/ousast-engine-task-build.json .
export TASK_IMAGE="ghcr.io/norandom/ousast-engine-task@$(python -c 'import json; print(json.load(open("/tmp/ousast-engine-task-build.json"))["containerimage.digest"])')"
printf '%s\n' "$TASK_IMAGE"
docker build -f Dockerfile -t openultrasast:engine-ax .
```
The Engine image workflow also builds/pushes both images and prints each digest;
CI needs the changes published before dispatch. The local build above includes
uncommitted changes without publishing a git commit.
With `KUBECONFIG` pointing to the operator's kube-ax config and store settings in
`.env`, run a one-pin smoke using the default experiment units and local example
memory (or add `--examples /path/to/frozen-examples.jsonl`):
```bash
: "${KUBECONFIG:?Set KUBECONFIG to the operator kube-ax configuration}"
python benchmarks/learn/engine_trace.py --executor ax \
--ax-bin "$HOME/.local/ax-v0.3.1/ax" --kubeconfig "$KUBECONFIG" \
--atespace default --image "$TASK_IMAGE" --docker-image openultrasast:engine-ax \
--parallel 1 --limit 1 --deadline 900 \
--out "$HOME/ousast-results/plane/engine-ax-smoke"
```
Then run the mixed lanes (repeat to resume; defaults re-run failed/timeout pins):
```bash
python benchmarks/learn/engine_trace.py --executor mixed-ax \
--ax-bin "$HOME/.local/ax-v0.3.1/ax" --kubeconfig "$KUBECONFIG" \
--atespace default --image "$TASK_IMAGE" --docker-image openultrasast:engine-ax \
--parallel 2 --deadline 900 --rerun-status failed,timeout \
--out "$HOME/ousast-results/plane/engine-ax-mixed"
```
@@ -0,0 +1,39 @@
{
"experiment": "exp-003-precision-bound-blocking",
"registered": "2026-10-03",
"store": "s3://sast-memory",
"manifest": "plane/experiments/exp-003-precision-bound-blocking.yaml",
"manifest_digest": "5917f9a12035f09eae76978f3b5827b9fe59c8a9a5275f9ab54c3c1b55e8db3b",
"pin": "dd7ebc119b5ad8e77c05d418eed9cbd46e193587",
"units_status": "frozen",
"units_file": "plane/experiments/exp-003-precision-bound-blocking.units.jsonl",
"units_digest": "338de998b1cfa36857cd793dbc4cf7fd0a13ab517a91f6790335881e941b3006",
"units": {
"total": 353,
"paired": 244,
"groups": 146,
"by_family": {
"access_control": {"units": 41, "positives": 25, "paired": 32, "paired_positives": 16, "groups": 24},
"deserialization": {"units": 48, "positives": 31, "paired": 34, "paired_positives": 17, "groups": 31},
"injection": {"units": 119, "positives": 63, "paired": 76, "paired_positives": 38, "groups": 34},
"output_encoding": {"units": 45, "positives": 29, "paired": 32, "paired_positives": 16, "groups": 27},
"path": {"units": 39, "positives": 23, "paired": 32, "paired_positives": 16, "groups": 23},
"untrusted_destination": {"units": 61, "positives": 37, "paired": 38, "paired_positives": 19, "groups": 36}
},
"note": "frozen model-free on the 937-example static memory; the outer folds (k 5, seed 0, compile fraction 0.25) are the incumbent evaluation's; injection and access_control equal exp-002's 160 units"
},
"kind": "rule: one zero-cost scoring pass of the six slice-2 incumbent programs; arms differ only in the BLOCK rule",
"zero_cost": "responses replayed from the response cache only, behind a meter with zero USD and call ceilings around a refusing client; a unit with any uncached response is excluded and counted; the run is void unless the meter reads 0 calls and $0",
"arms": {
"A": "today's rule: Platt map cross-fitted per family; BLOCK only when the calibration check holds (slope CI contains 1, intercept CI contains 0, ECE <= 0.05); BLOCK point nested on the other folds' calibrated p (Wilson LB >= 0.95)",
"B": "selective blocking: per family and outer fold, threshold chosen on the family's paired units in the other four folds as the lowest raw score t with Wilson 95% LB of flagged precision >= 0.95 (unsure verdicts never flag); none -> no BLOCK on that fold",
"C": "secondary, non-deciding: as B with point precision >= 0.95 over at least 10 training flags"
},
"primary_metric": "held-out (outer-fold) Wilson 95% lower bound of flagged precision, per family, arm B, paired population",
"populations": {"primary": "paired (both sides of the candidate present; excludes every positives-only group and any unpaired unit)", "secondary": ["pooled (every unit)"]},
"secondary_metrics": ["flagged_precision", "coverage (flagged true positives / positives)", "flagged", "repository-cluster bootstrap 95% CI of flagged precision"],
"decision_rule": "REJECT B if in any family B flags held-out paired units at a point precision < 0.95; else ADOPT B if in some family B's held-out paired Wilson LB >= 0.95; else INCONCLUSIVE (the incumbent rule stays)",
"m4_reading": "a family reaches the M4 precision bar on held-out repositories when its held-out paired Wilson LB >= 0.95; point precision reported beside it",
"power": "a Wilson LB of 0.95 needs >= 73 flagged units with no false flag; the largest family holds 38 paired positives over all five folds, so arm B cannot offer a BLOCK in any family by construction; expected verdict inconclusive",
"status": "run and analysed 2026-10-03: inconclusive -- no arm flags any held-out unit (result.json); meter 0 client calls, $0.0; 142 of 353 units excluded for an uncached response"
}
@@ -0,0 +1,981 @@
{
"arms": {
"A": {
"label": "today's rule -- Platt map cross-fitted per family, BLOCK only when the calibration check holds, BLOCK point nested on the other folds' calibrated p",
"rule": {
"name": "calibrated_block"
}
},
"B": {
"label": "selective blocking -- per-family raw-score threshold from the other folds' paired units, lowest t with Wilson 95% LB of flagged precision >= 0.95",
"rule": {
"bound": "wilson",
"name": "precision_bound_block",
"precision": 0.95,
"select_on": "paired"
}
},
"C": {
"label": "secondary, non-deciding -- as B with point precision >= 0.95 over at least 10 training flags",
"rule": {
"bound": "point",
"min_flagged": 10,
"name": "precision_bound_block",
"precision": 0.95,
"select_on": "paired"
}
}
},
"code_commit": "7bb97ad0bca62b5f986c5768297d8b8fca50fae4",
"decision": "inconclusive",
"experiment": "exp-003-precision-bound-blocking",
"families": {
"access_control": {
"A": {
"calibration_holds": false,
"ece": 0.077,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 16,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 16,
"true_flagged": 0,
"units": 32
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 24,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 25,
"true_flagged": 0,
"units": 41
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 16,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 16,
"true_flagged": 0,
"units": 32
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 24,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 25,
"true_flagged": 0,
"units": 41
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"C": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 16,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 16,
"true_flagged": 0,
"units": 32
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 24,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 25,
"true_flagged": 0,
"units": 41
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"capacity": {
"evaluated": 41,
"excluded_replay_miss": 0,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 12,
"outer-1": 13,
"outer-2": 13,
"outer-3": 13,
"outer-4": 13
}
}
},
"deserialization": {
"A": {
"calibration_holds": false,
"ece": null,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 3,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 2,
"true_flagged": 0,
"units": 5
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 12,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 11,
"true_flagged": 0,
"units": 14
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 3,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 2,
"true_flagged": 0,
"units": 5
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 12,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 11,
"true_flagged": 0,
"units": 14
},
"thresholds_by_fold": {
"outer-0": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"C": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 3,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 2,
"true_flagged": 0,
"units": 5
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 12,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 11,
"true_flagged": 0,
"units": 14
},
"thresholds_by_fold": {
"outer-0": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"capacity": {
"evaluated": 14,
"excluded_replay_miss": 34,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 0,
"outer-2": 2,
"outer-3": 2,
"outer-4": 2
}
}
},
"injection": {
"A": {
"calibration_holds": false,
"ece": 0.1067,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 26,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 38,
"true_flagged": 0,
"units": 76
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 34,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 63,
"true_flagged": 0,
"units": 119
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 26,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 38,
"true_flagged": 0,
"units": 76
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 34,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 63,
"true_flagged": 0,
"units": 119
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"C": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 26,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 38,
"true_flagged": 0,
"units": 76
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 34,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 63,
"true_flagged": 0,
"units": 119
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"capacity": {
"evaluated": 119,
"excluded_replay_miss": 0,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 29,
"outer-1": 33,
"outer-2": 33,
"outer-3": 26,
"outer-4": 31
}
}
},
"output_encoding": {
"A": {
"calibration_holds": false,
"ece": null,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 8,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 8,
"true_flagged": 0,
"units": 16
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 15,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 16,
"true_flagged": 0,
"units": 24
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 8,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 8,
"true_flagged": 0,
"units": 16
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 15,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 16,
"true_flagged": 0,
"units": 24
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"C": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 8,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 8,
"true_flagged": 0,
"units": 16
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 15,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 16,
"true_flagged": 0,
"units": 24
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"capacity": {
"evaluated": 24,
"excluded_replay_miss": 21,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 7,
"outer-1": 6,
"outer-2": 6,
"outer-3": 7,
"outer-4": 6
}
}
},
"path": {
"A": {
"paired": {
"coverage": null,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 0,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 0,
"true_flagged": 0,
"units": 0
},
"pooled": {
"coverage": null,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 0,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 0,
"true_flagged": 0,
"units": 0
}
},
"B": {
"paired": {
"coverage": null,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 0,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 0,
"true_flagged": 0,
"units": 0
},
"pooled": {
"coverage": null,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 0,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 0,
"true_flagged": 0,
"units": 0
}
},
"C": {
"paired": {
"coverage": null,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 0,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 0,
"true_flagged": 0,
"units": 0
},
"pooled": {
"coverage": null,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 0,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 0,
"true_flagged": 0,
"units": 0
}
},
"capacity": {
"evaluated": 0,
"excluded_replay_miss": 39,
"not_run": 0,
"paired_positives_by_training_folds": {}
}
},
"untrusted_destination": {
"A": {
"calibration_holds": false,
"ece": null,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 6,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 4,
"true_flagged": 0,
"units": 9
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 9,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 7,
"true_flagged": 0,
"units": 13
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null
}
},
"B": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 6,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 4,
"true_flagged": 0,
"units": 9
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 9,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 7,
"true_flagged": 0,
"units": 13
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null
}
},
"C": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 6,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 4,
"true_flagged": 0,
"units": 9
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 9,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 7,
"true_flagged": 0,
"units": 13
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null
}
},
"capacity": {
"evaluated": 13,
"excluded_replay_miss": 48,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 4,
"outer-1": 3,
"outer-2": 3,
"outer-3": 2
}
}
}
},
"hypothesis": "The calibrated-probability BLOCK rule (Platt map + calibration gate, ECE <= 0.05) is the wrong instrument for M4, which asks for precision on what is flagged. Selective blocking -- a per-family raw-score threshold chosen on training folds only as the lowest one whose Wilson 95% lower bound on flagged precision is >= 0.95 -- reaches the M4 precision bar on held-out repositories where the calibrated rule offers no BLOCK.\n",
"instrument_check": {
"families": {
"access_control": {
"distinct_s": 30,
"paired": {
"blockable": 28,
"positives_before_first_negative": 10
},
"pooled": {
"blockable": 37,
"positives_before_first_negative": 19
},
"verdicts": {
"not_vulnerable": 19,
"unsure": 4,
"vulnerable": 18
}
},
"deserialization": {
"distinct_s": 5,
"paired": {
"blockable": 4,
"positives_before_first_negative": 1
},
"pooled": {
"blockable": 13,
"positives_before_first_negative": 10
},
"verdicts": {
"not_vulnerable": 2,
"unsure": 1,
"vulnerable": 11
}
},
"injection": {
"distinct_s": 48,
"paired": {
"blockable": 50,
"positives_before_first_negative": 3
},
"pooled": {
"blockable": 77,
"positives_before_first_negative": 11
},
"verdicts": {
"not_vulnerable": 46,
"unsure": 42,
"vulnerable": 31
}
},
"output_encoding": {
"distinct_s": 14,
"paired": {
"blockable": 11,
"positives_before_first_negative": 0
},
"pooled": {
"blockable": 19,
"positives_before_first_negative": 8
},
"verdicts": {
"not_vulnerable": 9,
"unsure": 5,
"vulnerable": 10
}
},
"untrusted_destination": {
"distinct_s": 9,
"paired": {
"blockable": 4,
"positives_before_first_negative": 1
},
"pooled": {
"blockable": 8,
"positives_before_first_negative": 4
},
"verdicts": {
"not_vulnerable": 3,
"unsure": 5,
"vulnerable": 5
}
}
},
"note": "descriptive, all folds together, ranking by raw score s among non-unsure verdicts; not nested, chooses nothing. It shows the scores are real and the zero is the rule's, not an unread input."
},
"manifest_digest": "5917f9a12035f09eae76978f3b5827b9fe59c8a9a5275f9ab54c3c1b55e8db3b",
"meter": {
"client_calls": 0,
"decided": 211,
"excluded_replay_miss": 142,
"folds_digest": "a1f78e141a572a4adcc9cd57eb8a784c440c177f0b60d5ef5bed9e94c8753c5f",
"metered_usd": 0.0,
"replay_priced_usd_not_spent": 0.582876,
"replayed_responses": 1056
},
"min_flagged_for_bound": 73,
"overall": {
"A": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 59,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 68,
"true_flagged": 0,
"units": 138
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 89,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 122,
"true_flagged": 0,
"units": 211
}
},
"B": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 59,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 68,
"true_flagged": 0,
"units": 138
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 89,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 122,
"true_flagged": 0,
"units": 211
}
},
"C": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 59,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 68,
"true_flagged": 0,
"units": 138
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 89,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 122,
"true_flagged": 0,
"units": 211
}
}
},
"populations": {
"primary": "paired",
"secondary": [
"pooled"
]
},
"primary": [
{
"name": "flagged_precision_lb",
"target": 0.95
}
],
"reason": "B offers no BLOCK on held-out paired units in any family; the incumbent stays",
"resamples": 2000,
"secondary": [
{
"name": "flagged_precision"
},
{
"name": "coverage"
},
{
"name": "flagged"
}
],
"spend_usd": {
"score": 0.0
},
"target_precision": 0.95,
"units": {
"by_label": {
"0": 89,
"1": 122
},
"evaluated": 211,
"excluded_replay_miss": 142,
"frozen": 353,
"groups": 89,
"not_run": 0,
"paired_evaluated": 138,
"unsure": 57
}
}
@@ -0,0 +1,53 @@
"""Instrument check for exp-004: are the scores real, and how clean is the pooled top of the ranking?
Reads the run's arm_outcome rows from the memory store (OUSAST_MEMORY) and the frozen units file; makes no model call.
usage: python benchmarks/experiments/exp-004-pooled-block-gate/instrument_check.py (from the repository root)
"""
from __future__ import annotations
import collections
import json
from pathlib import Path
from openultrasast.config import load_dotenv
from openultrasast.learn import experiments as ex
from openultrasast.plane.memory import open_store
EXPERIMENT = "exp-004-pooled-block-gate"
UNITS = Path(f"plane/experiments/{EXPERIMENT}.units.jsonl")
def main() -> None:
load_dotenv(Path(".env"))
rows = [r.row for r in open_store(None).rows(repo=ex.experiment_repo(EXPERIMENT), kind="arm_outcome")]
with UNITS.open() as handle:
units = {u["unit"]: u for u in (json.loads(line) for line in handle)}
print("arm_outcome rows", len(rows), "units file", len(units))
by_family: dict[str, list[tuple[float | None, int, str | None, str]]] = collections.defaultdict(list)
for row in rows:
unit = units.get(row.get("unit"))
if unit:
by_family[unit["family"]].append((row.get("s"), unit["label"], unit.get("pair"), str(row.get("verdict") or "")))
for family, xs in sorted(by_family.items()):
scored = [x for x in xs if x[0] is not None]
pairs: dict[str, dict[int, float]] = collections.defaultdict(dict)
for score, label, pair, _ in scored:
if pair:
pairs[pair][label] = score
full = [p for p in pairs.values() if 0 in p and 1 in p]
wins = sum(1 for p in full if p[1] > p[0])
ties = sum(1 for p in full if p[1] == p[0])
top = sorted((x for x in scored if x[2]), key=lambda x: -(x[0] or 0.0))
first_negative = next((i for i, x in enumerate(top) if x[1] == 0), len(top))
print(
f"{family}: n={len(xs)} distinct_scores={len({round(x[0] or 0.0, 4) for x in scored})} "
f"unsure={sum(1 for x in xs if x[3] == 'unsure')} pairs={len(full)} within_pair_win={wins} tie={ties} "
f"pos_above_first_neg={first_negative}"
)
pooled = sorted((x for xs in by_family.values() for x in xs if x[0] is not None and x[2]), key=lambda x: -(x[0] or 0.0))
print("pooled raw top-30 labels:", "".join(str(x[1]) for x in pooled[:30]))
if __name__ == "__main__":
main()
@@ -0,0 +1,81 @@
{
"experiment": "exp-004-pooled-block-gate",
"date": "2026-10-03",
"fill": {
"client_calls": 2073,
"usd": 1.238223,
"ceiling_usd": 2.0,
"still_missing": 0,
"dry_run_estimate_usd": 5.582,
"note": "the dry-run estimator over-priced 4.5x (no prompt-cache pricing); measured spend is the meter's"
},
"run": {
"units": 501,
"client_calls": 0,
"usd": 0.0,
"replayed": 2509,
"replay_miss": 0
},
"verdict": "inconclusive",
"reason": "no t_block on any outer fold, pooled or per family; B offers no BLOCK",
"instrument_check": {
"per_family": {
"access_control": {
"n": 66,
"distinct_scores": 41,
"unsure": 5,
"pairs": 28,
"within_pair_wins": 22,
"ties": 1,
"positives_above_first_negative": 12
},
"deserialization": {
"n": 64,
"distinct_scores": 38,
"unsure": 7,
"pairs": 25,
"within_pair_wins": 18,
"ties": 4,
"positives_above_first_negative": 5
},
"injection": {
"n": 149,
"distinct_scores": 52,
"unsure": 54,
"pairs": 53,
"within_pair_wins": 30,
"ties": 17,
"positives_above_first_negative": 6
},
"output_encoding": {
"n": 69,
"distinct_scores": 31,
"unsure": 32,
"pairs": 28,
"within_pair_wins": 18,
"ties": 6,
"positives_above_first_negative": 2
},
"path": {
"n": 66,
"distinct_scores": 43,
"unsure": 15,
"pairs": 29,
"within_pair_wins": 25,
"ties": 3,
"positives_above_first_negative": 13
},
"untrusted_destination": {
"n": 87,
"distinct_scores": 36,
"unsure": 33,
"pairs": 32,
"within_pair_wins": 18,
"ties": 4,
"positives_above_first_negative": 3
}
},
"pooled_raw_top30_labels": "111111111110111110111111110111",
"reading": "scores are real (31-52 distinct values per family); pooled raw top 30 holds 27 positives (0.90), first negative at rank 12; the Wilson bound needs >= 73 flags with no error, so precision at the top, not data volume alone, now binds"
}
}
@@ -0,0 +1,35 @@
{
"experiment": "exp-004-pooled-block-gate",
"registered": "2026-10-03",
"store": "s3://sast-memory",
"manifest": "plane/experiments/exp-004-pooled-block-gate.yaml",
"manifest_digest": "87fceb6580c6de629d6a6089eb5888cdfc4df0f791576184f987c14c192d85f5",
"pin": "2ad9c1db7ae673c2a2e48a0afadcaa795dfb1ced",
"units_status": "pending",
"units_file": "plane/experiments/exp-004-pooled-block-gate.units.jsonl",
"units_digest": "",
"units": "frozen at run time by `ousast learn experiment units` from the memory after the negatives harvest (merged in 5b2bd9b, 1133 static examples), committed, and registered again before the run; until then the run refuses (check_registered: the units file does not exist)",
"requirement": "learned-decision-engine Req 6.4, amended 2026-10-03 by maintainer decision (refines the BLOCK rule of Req 7.3 and 7.4)",
"kind": "rule: one zero-cost scoring pass of the six slice-2 incumbent programs; arms differ only in the BLOCK rule",
"zero_cost": "the run replays the response cache behind a zero-ceiling meter; a unit with any uncached response is excluded and counted. The harvest changed the memory the programs retrieve from, so a paid cache fill, approved separately with its own ceiling, must precede the run",
"normalisation": {
"choice": "per-family Platt scaling: learn.calibrate.fit_map with its existing floors (own slope and intercept per family at >= 20 positive and negative repository groups, a pooled slope with a per-family intercept at >= 10, no map below that and so no BLOCK for that family on that fold), fitted on the training folds' paired units only",
"why": "a rank-to-quantile map gives every family the same flag rate whatever its precision, so one pooled threshold would flag the top slice of a weak family as readily as of a strong one; Platt puts each family on the scale of its own training precision, which a pooled precision bound needs. The map is a normaliser only: precision is shown by the held-out bound, not by calibration (exp-003: ECE <= 0.05 does not hold per family)"
},
"arms": {
"A": "today's rule: Platt map cross-fitted per family; BLOCK only when the calibration check holds; BLOCK point nested on the other folds' calibrated p (Wilson LB >= 0.95)",
"B": "pooled gate: per outer fold j, training units normalised by an inner cross-fit; t_block = the lowest normalised score whose Wilson 95% LB of precision over the training paired flags of all families together is >= 0.95 (unsure never flags); t_quiet per family = the highest normalised score keeping >= 90% of that family's training paired positives; fold j's units BLOCK at p >= t_block, ADVISORY at p >= t_quiet (or without a map or t_quiet), silent below"
},
"family_floor": {
"F": 10,
"rule": "a family may block only if its own held-out paired flags (all five folds) number >= 10 with zero errors; otherwise its flags are reported as ADVISORY. The floor uses held-out labels to decide which families block and moves no threshold",
"why": "zero errors in 10 flags is unlikely for a family much worse than the bar (probability 0.056 at precision 0.75, 0.107 at 0.80); a larger F is out of reach of the smaller families (16-19 paired positives in exp-003's units over all folds); 10 is exp-003 arm C's training minimum"
},
"primary_metric": "held-out (outer-fold) Wilson 95% lower bound of flagged precision over arm B's paired flags of all families together, before the family floor (so dropping a failing family cannot raise it)",
"populations": {"primary": "paired (both sides of the candidate present)", "secondary": ["pooled (every unit)"]},
"secondary_metrics": ["per-family flagged precision and Wilson LB", "the family-floor table", "quiet_recall per family (held-out positives at or above t_quiet / positives) against 0.90", "after-floor totals (what users would get)", "repository-cluster bootstrap 95% CI of flagged precision", "coverage", "flagged"],
"decision_rule": "on arm B's held-out paired flags of all families together, before the floor: REJECT B if their point precision < 0.95; else ADOPT B if their Wilson 95% LB >= 0.95 and at least one family passes the floor (>= 10 held-out paired flags, zero errors) -- those families block, every other family stays advisory; else INCONCLUSIVE (the incumbent rule stays)",
"power": "the pooled bound needs >= 73 flags with no error (110 with one, 142 with two). On exp-003's units (all 353 scored, result-complete-cache.json) few paired positives rank above the first negative per family on the raw score (access control 10, deserialization 7, injection 2, output encoding 0, path 8, untrusted destination 5); normalisation keeps each family's order, so the pooled top set reaches 73 only if the harvest's pairs separate better. An inconclusive verdict is a plausible outcome",
"tests": "tests/test_learn_rules.py: test_one_pooled_threshold_blocks_where_no_family_alone_reaches_the_bound, test_no_held_out_label_moves_a_threshold_or_the_map, test_a_family_with_one_held_out_error_cannot_block, test_quiet_threshold_keeps_ninety_percent_of_positives_and_nothing_is_silenced_blind, test_the_pooled_verdict_follows_the_registered_rule, test_the_exp004_manifest_registers_the_procedure_before_its_units",
"status": "registered 2026-10-03, units pending, not run"
}
@@ -0,0 +1,822 @@
{
"arms": {
"A": {
"label": "today's rule -- Platt map cross-fitted per family, BLOCK only when the calibration check holds, BLOCK point nested on the other folds' calibrated p",
"rule": {
"name": "calibrated_block"
}
},
"B": {
"label": "pooled gate -- one t_block over all families on a per-family Platt score (training folds, inner cross-fit), t_quiet per family at 90% recall, family floor 10 clean held-out flags",
"rule": {
"floor": 10,
"name": "pooled_block_gate",
"normalise": "platt",
"precision": 0.95,
"recall": 0.9,
"select_on": "paired"
}
}
},
"code_commit": "90a2642019df414cb67b5dcea6e4ec235f35650a",
"decision": "inconclusive",
"experiment": "exp-004-pooled-block-gate",
"families": {
"access_control": {
"A": {
"calibration_holds": false,
"ece": 0.0715,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 28,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 28,
"true_flagged": 0,
"units": 56
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 37,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 38,
"true_flagged": 0,
"units": 66
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"floor": {
"errors": 0,
"flagged": 0,
"floor": 10,
"may_block": false
},
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 28,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 28,
"true_flagged": 0,
"units": 56
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 37,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 38,
"true_flagged": 0,
"units": 66
},
"quiet_recall": {
"paired": 0.8929,
"pooled": 0.9211
},
"t_block_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
},
"t_quiet_by_fold": {
"outer-0": 0.44900641988143913,
"outer-1": 0.3723384028044276,
"outer-2": 0.4202772526614184,
"outer-3": 0.4481849709797749,
"outer-4": 0.36346174463707154
}
},
"capacity": {
"evaluated": 66,
"excluded_replay_miss": 0,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 22,
"outer-1": 22,
"outer-2": 22,
"outer-3": 23,
"outer-4": 23
}
}
},
"deserialization": {
"A": {
"calibration_holds": false,
"ece": 0.0725,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 25,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 25,
"true_flagged": 0,
"units": 50
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 39,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 39,
"true_flagged": 0,
"units": 64
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"floor": {
"errors": 0,
"flagged": 0,
"floor": 10,
"may_block": false
},
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 25,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 25,
"true_flagged": 0,
"units": 50
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 39,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 39,
"true_flagged": 0,
"units": 64
},
"quiet_recall": {
"paired": 0.88,
"pooled": 0.9231
},
"t_block_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
},
"t_quiet_by_fold": {
"outer-0": 0.17507194490500488,
"outer-1": 0.22766488322270723,
"outer-2": 0.2163745870743959,
"outer-3": 0.3112201878893598,
"outer-4": 0.363243722721776
}
},
"capacity": {
"evaluated": 64,
"excluded_replay_miss": 0,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 20,
"outer-1": 20,
"outer-2": 20,
"outer-3": 20,
"outer-4": 20
}
}
},
"injection": {
"A": {
"calibration_holds": false,
"ece": 0.1042,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 41,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 53,
"true_flagged": 0,
"units": 106
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 49,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 78,
"true_flagged": 0,
"units": 149
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"floor": {
"errors": 0,
"flagged": 0,
"floor": 10,
"may_block": false
},
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 41,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 53,
"true_flagged": 0,
"units": 106
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 49,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 78,
"true_flagged": 0,
"units": 149
},
"quiet_recall": {
"paired": 0.8679,
"pooled": 0.8974
},
"t_block_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
},
"t_quiet_by_fold": {
"outer-0": 0.287983504097999,
"outer-1": 0.17524866355153343,
"outer-2": 0.12918182665453967,
"outer-3": 0.14521040470628938,
"outer-4": 0.23005714687697673
}
},
"capacity": {
"evaluated": 149,
"excluded_replay_miss": 0,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 38,
"outer-1": 44,
"outer-2": 45,
"outer-3": 43,
"outer-4": 42
}
}
},
"output_encoding": {
"A": {
"calibration_holds": false,
"ece": 0.0736,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 28,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 28,
"true_flagged": 0,
"units": 56
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 39,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 41,
"true_flagged": 0,
"units": 69
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"floor": {
"errors": 0,
"flagged": 0,
"floor": 10,
"may_block": false
},
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 28,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 28,
"true_flagged": 0,
"units": 56
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 39,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 41,
"true_flagged": 0,
"units": 69
},
"quiet_recall": {
"paired": 0.8571,
"pooled": 0.9024
},
"t_block_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
},
"t_quiet_by_fold": {
"outer-0": 0.3723166587726901,
"outer-1": 0.5329072757249385,
"outer-2": 0.3785066578507705,
"outer-3": 0.4389011844848602,
"outer-4": 0.43855966643440264
}
},
"capacity": {
"evaluated": 69,
"excluded_replay_miss": 0,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 23,
"outer-1": 22,
"outer-2": 22,
"outer-3": 22,
"outer-4": 23
}
}
},
"path": {
"A": {
"calibration_holds": false,
"ece": 0.0621,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 29,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 29,
"true_flagged": 0,
"units": 58
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 37,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 37,
"true_flagged": 0,
"units": 66
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"floor": {
"errors": 0,
"flagged": 0,
"floor": 10,
"may_block": false
},
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 29,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 29,
"true_flagged": 0,
"units": 58
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 37,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 37,
"true_flagged": 0,
"units": 66
},
"quiet_recall": {
"paired": 0.8621,
"pooled": 0.8919
},
"t_block_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
},
"t_quiet_by_fold": {
"outer-0": 0.4680507207548809,
"outer-1": 0.3955717127313657,
"outer-2": 0.43694703358760895,
"outer-3": 0.4450583743716199,
"outer-4": 0.46396657833309796
}
},
"capacity": {
"evaluated": 66,
"excluded_replay_miss": 0,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 23,
"outer-1": 23,
"outer-2": 24,
"outer-3": 23,
"outer-4": 23
}
}
},
"untrusted_destination": {
"A": {
"calibration_holds": false,
"ece": 0.1143,
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 31,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 32,
"true_flagged": 0,
"units": 64
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 49,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 50,
"true_flagged": 0,
"units": 87
},
"reason_by_fold": {
"outer-0": "calibration",
"outer-1": "calibration",
"outer-2": "calibration",
"outer-3": "calibration",
"outer-4": "calibration"
},
"thresholds_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
}
},
"B": {
"floor": {
"errors": 0,
"flagged": 0,
"floor": 10,
"may_block": false
},
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 31,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 32,
"true_flagged": 0,
"units": 64
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 49,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 50,
"true_flagged": 0,
"units": 87
},
"quiet_recall": {
"paired": 1.0,
"pooled": 1.0
},
"t_block_by_fold": {
"outer-0": null,
"outer-1": null,
"outer-2": null,
"outer-3": null,
"outer-4": null
},
"t_quiet_by_fold": {
"outer-0": 0.13676746287025585,
"outer-1": 0.11255775703980551,
"outer-2": 0.13763698150768486,
"outer-3": 0.09787938647374639,
"outer-4": 0.13763698150768486
}
},
"capacity": {
"evaluated": 87,
"excluded_replay_miss": 0,
"not_run": 0,
"paired_positives_by_training_folds": {
"outer-0": 26,
"outer-1": 26,
"outer-2": 24,
"outer-3": 26,
"outer-4": 26
}
}
}
},
"hypothesis": "One BLOCK threshold pooled across families on a per-family Platt-normalised score, chosen on training folds only as the lowest one whose Wilson 95% lower bound on flagged precision is >= 0.95, reaches that bound on held-out repositories' paired units, where per-family thresholds (exp-003) could not for lack of flags; a per-family floor (>= 10 held-out flags, none wrong) keeps a family that the pooling masks from blocking.\n",
"manifest_digest": "87fceb6580c6de629d6a6089eb5888cdfc4df0f791576184f987c14c192d85f5",
"min_flagged_for_bound": 73,
"overall": {
"A": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 182,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 195,
"true_flagged": 0,
"units": 390
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 220,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 283,
"true_flagged": 0,
"units": 501
}
},
"B": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 182,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 195,
"true_flagged": 0,
"units": 390
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 220,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 283,
"true_flagged": 0,
"units": 501
}
}
},
"overall_after_floor": {
"B": {
"paired": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 182,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 195,
"true_flagged": 0,
"units": 390
},
"pooled": {
"coverage": 0.0,
"flagged": 0,
"flagged_precision": null,
"flagged_precision_ci": null,
"flagged_precision_lb": null,
"groups": 220,
"meets_target_lb": false,
"meets_target_point": false,
"positives": 283,
"true_flagged": 0,
"units": 501
}
}
},
"populations": {
"primary": "paired",
"secondary": [
"pooled"
]
},
"primary": [
{
"name": "flagged_precision_lb",
"target": 0.95
}
],
"reason": "B offers no BLOCK on held-out paired units; the incumbent stays",
"resamples": 2000,
"secondary": [
{
"name": "flagged_precision"
},
{
"name": "coverage"
},
{
"name": "flagged"
},
{
"name": "quiet_recall"
}
],
"spend_usd": {
"score": 0.0
},
"target_precision": 0.95,
"units": {
"by_label": {
"0": 218,
"1": 283
},
"evaluated": 501,
"excluded_replay_miss": 0,
"frozen": 501,
"groups": 220,
"not_run": 0,
"paired_evaluated": 390,
"unsure": 146
}
}
@@ -0,0 +1,84 @@
{
"arms": [
"B"
],
"client_calls": 2394,
"experiment": "exp-005-graph-slice",
"families": {
"access_control": {
"cached": 0,
"conditional_retries": 330,
"estimate_method": "estimated",
"estimated_output_tokens": 300,
"estimated_usd": 0.9956892000000007,
"filled": 330,
"missing": 330,
"requests": 330,
"still_missing": 0,
"units": 66
},
"deserialization": {
"cached": 0,
"conditional_retries": 310,
"estimate_method": "estimated",
"estimated_output_tokens": 300,
"estimated_usd": 0.8843295999999988,
"filled": 310,
"missing": 310,
"requests": 310,
"still_missing": 0,
"units": 64
},
"injection": {
"cached": 0,
"conditional_retries": 675,
"estimate_method": "estimated",
"estimated_output_tokens": 300,
"estimated_usd": 2.270278999999999,
"filled": 675,
"missing": 675,
"requests": 675,
"still_missing": 0,
"units": 149
},
"output_encoding": {
"cached": 0,
"conditional_retries": 325,
"estimate_method": "estimated",
"estimated_output_tokens": 300,
"estimated_usd": 1.0014180000000004,
"filled": 328,
"missing": 325,
"requests": 328,
"still_missing": 0,
"units": 69
},
"path": {
"cached": 0,
"conditional_retries": 320,
"estimate_method": "estimated",
"estimated_output_tokens": 300,
"estimated_usd": 0.9399192000000005,
"filled": 325,
"missing": 320,
"requests": 325,
"still_missing": 0,
"units": 66
},
"untrusted_destination": {
"cached": 0,
"conditional_retries": 425,
"estimate_method": "estimated",
"estimated_output_tokens": 300,
"estimated_usd": 1.174316,
"filled": 426,
"missing": 425,
"requests": 426,
"still_missing": 0,
"units": 87
}
},
"reason": "",
"status": "done",
"usd": 1.5890382240000016
}
@@ -0,0 +1,165 @@
{
"alpha": 0.05,
"arms": {
"A": {
"budget_usd": 0.0,
"label": "six slice-2 incumbents unchanged, replaying exp-004's response cache",
"repeats": 1,
"replay_only": true
},
"B": {
"budget_usd": 6.0,
"label": "the same six programs with the in-function slice added to their prompts",
"repeats": 1,
"replay_only": false
}
},
"code_commit": "0acbc7ec41dcaeca805c4720bc3f6d02cb940ee7",
"decision": "inconclusive",
"experiment": "exp-005-graph-slice",
"hypothesis": "Adding the in-function def-use slice to the prompt improves within-pair ranking of the six slice-2 incumbent programs on held-out repositories, at no more than 1.5 times the cost per candidate.\n",
"looks": [
{
"boundary": 1.959964,
"crossed": false,
"estimate": {
"a": 0.7615384615384615,
"b": 0.7461538461538462,
"ci": [
-0.0628019323671497,
0.03155339805825241
],
"diff": -0.01538461538461533,
"groups": 220,
"metric": "within_pair_auc",
"resamples": 10000,
"se": 0.024030780235707972,
"units": 501
},
"fraction": 1.0,
"groups": 220,
"z": -0.6402045723740141
}
],
"manifest_digest": "1b1ec814cde12801491376d2086c8c2d6db70f4a86fc308c0adc5a530398e554",
"mde": 0.0,
"primary": [
{
"name": "within_pair_auc"
}
],
"reason": "the CI of within_pair_auc spans 0; the incumbent stays",
"resamples": 10000,
"secondary": [
{
"name": "top_30_precision"
},
{
"name": "unsure_rate"
},
{
"name": "usd_per_candidate"
}
],
"spend_usd": {
"A": 1.444503,
"B": 1.589038
},
"tests": {
"top_30_precision": {
"ci_vs_zero": "spans",
"direction": 1,
"estimate": {
"a": 1.0,
"b": 1.0,
"ci": [
0.0,
0.0
],
"diff": 0.0,
"groups": 220,
"metric": "top_30_precision",
"resamples": 10000,
"se": 0.0006666666666666664,
"units": 501
},
"mcnemar": null,
"target": null
},
"unsure_rate": {
"ci_vs_zero": "favour",
"direction": -1,
"estimate": {
"a": 0.29141716566866266,
"b": 0.23952095808383234,
"ci": [
-0.08583690987124465,
-0.01996007984031939
],
"diff": -0.05189620758483032,
"groups": 220,
"metric": "unsure_rate",
"resamples": 10000,
"se": 0.01685991406098136,
"units": 501
},
"mcnemar": {
"b": 47,
"c": 21,
"concordant": 433,
"p_value": 0.0021862431776861705
},
"target": null
},
"usd_per_candidate": {
"ci_vs_zero": "against",
"direction": -1,
"estimate": {
"a": 0.002883239520958083,
"b": 0.0031717325349301397,
"ci": [
0.0002458205128205199,
0.0003319800399201574
],
"diff": 0.0002884930139720565,
"groups": 220,
"metric": "usd_per_candidate",
"resamples": 10000,
"se": 2.1746161270931252e-05,
"units": 501
},
"mcnemar": null,
"target": null
},
"within_pair_auc": {
"ci_vs_zero": "spans",
"direction": 1,
"estimate": {
"a": 0.7615384615384615,
"b": 0.7461538461538462,
"ci": [
-0.0628019323671497,
0.03155339805825241
],
"diff": -0.01538461538461533,
"groups": 220,
"metric": "within_pair_auc",
"resamples": 10000,
"se": 0.024030780235707972,
"units": 501
},
"mcnemar": null,
"target": null
}
},
"tost": null,
"units": {
"by_label": {
"0": 218,
"1": 283
},
"groups": 220,
"outcomes": 1002,
"paired": 501
}
}
+132 -17
View File
@@ -7,7 +7,7 @@ from:
- quick rules and deterministic roles, run here on the unit's tree: a pair side materialised from its excerpt, a - quick rules and deterministic roles, run here on the unit's tree: a pair side materialised from its excerpt, a
population or recipe pin exported from its clone or checkout (one at a time, deleted after, disk-guarded); population or recipe pin exported from its clone or checkout (one at a time, deleted after, disk-guarded);
- the engine: ``benchmarks/learn/engine_pairs.py`` records for pairs, the recorded protocol scans - the engine: ``benchmarks/learn/engine_pairs.py`` records for pairs, the recorded protocol scans
(``~/ousast-results/independent-v{1,2}/scans/<case>--<pin role>.json``) for populations; (``~/ousast-results/independent-v{1,2}/scans/<case>--<pin role>[_a|_b].json``) for populations;
- verify passes a/b and ``agree`` from the harvest Run (``harvest-verify``) and, for population v2, the recorded - verify passes a/b and ``agree`` from the harvest Run (``harvest-verify``) and, for population v2, the recorded
``validation-46`` Run; model roles from ``harvest-roles``; ``validation-46`` Run; model roles from ``harvest-roles``;
- callers and entry-point distance on a repository (``not_applicable`` on a pair excerpt: no repository exists). - callers and entry-point distance on a repository (``not_applicable`` on a pair excerpt: no repository exists).
@@ -21,6 +21,11 @@ An instrument that has not run for a unit yet (the engine job still running, a R
memory store (``OUSAST_MEMORY``, default ``~/ousast-results/plane/memory``) as kind ``features``; a rerun replaces memory store (``OUSAST_MEMORY``, default ``~/ousast-results/plane/memory``) as kind ``features``; a rerun replaces
them by id. ``--counts`` writes the counts-only coverage record (no candidate, path or repository name). them by id. ``--counts`` writes the counts-only coverage record (no candidate, path or repository name).
``--dry-run`` instead compares existing local store examples with the corrected engine features, printing
per-source transitions from not-run/none to ran (and recorded failures separately). It reads only the spent
v1/v2 manifests, saved scans and local Git objects; it writes neither the store nor ``--counts`` and does not
rebuild other instruments. No label snapshot, harvest units or scratch directory is needed in this mode.
Usage:: Usage::
python benchmarks/learn/build_features.py --labels labels-<sha>.jsonl --units <harvest plane>/units.json --counts counts.json python benchmarks/learn/build_features.py --labels labels-<sha>.jsonl --units <harvest plane>/units.json --counts counts.json
@@ -30,9 +35,11 @@ from __future__ import annotations
import argparse import argparse
import json import json
import os
import shutil import shutil
import subprocess import subprocess
import tempfile import tempfile
import tomllib
from collections import Counter, defaultdict from collections import Counter, defaultdict
from collections.abc import Mapping from collections.abc import Mapping
from pathlib import Path from pathlib import Path
@@ -61,6 +68,7 @@ from openultrasast.learn.features import (
source_part, source_part,
verify_part, verify_part,
) )
from openultrasast.learn.labels import Clone
from openultrasast.learn.roles import RoleSet, from_model_roles, infer_for_checkout from openultrasast.learn.roles import RoleSet, from_model_roles, infer_for_checkout
from openultrasast.pairs import DEFAULT_CATALOG, _materialize_side, _targets, load_pair_catalog from openultrasast.pairs import DEFAULT_CATALOG, _materialize_side, _targets, load_pair_catalog
from openultrasast.plane import memory from openultrasast.plane import memory
@@ -89,6 +97,110 @@ def read_json(path: Path) -> Any:
return json.loads(path.read_text(encoding="utf-8")) if path.is_file() else None return json.loads(path.read_text(encoding="utf-8")) if path.is_file() else None
def resolve_scan(scans: Path, case: str, side: str) -> dict[str, Any] | None:
"""Read the v1 single scan, otherwise combine the v2 repeats for this side."""
single = read_json(scans / f"{case}--{side}.json")
if single is not None:
return dict(single)
repeats = [read_json(scans / f"{case}--{side}_{repeat}.json") for repeat in ("a", "b")]
available = [r for r in repeats if r is not None]
if not available:
return None
# Identical rule for vulnerable and fixed: a finding in either repeat counts;
# exact duplicates count once. engine_part takes the highest rung and minimum
# witness steps across the union. Completion is sum(completed)/sum(questions).
# A successful repeat survives a failed/missing peer, which marks degradation.
ran = [r for r in available if engine_for(r) is None]
findings = {json.dumps(f, sort_keys=True): f for r in ran for f in r.get("findings", [])}
degradations = sorted({d for r in available for d in r.get("degradations", [])})
if len(ran) != len(repeats):
degradations.append("repeat_missing_or_failed")
return {
"state": "ran" if ran else "none" if all(engine_for(r) == NONE for r in available) else "failed",
"questions": sum(int(r.get("questions") or 0) for r in available),
"completed": sum(int(r.get("completed") or 0) for r in available),
"findings": list(findings.values()),
"degradations": degradations,
"image": "+".join(sorted({str(r.get("image") or "recorded") for r in available})),
}
def recorded_engine(language: str, findings: list, result: Mapping | None, repository: bool) -> Part:
# The language list describes standalone scans. Repository scans also analyze
# TypeScript; recorded function findings take precedence over that list.
if language not in ENGINE_LANGUAGES and not findings and not (repository and language == "typescript" and result is not None):
return NONE
state = engine_for(result)
if state is not None:
return state
result = result or {}
questions, completed = int(result.get("questions") or 0), int(result.get("completed") or 0)
return engine_part(
findings, completion=completed / questions if questions else None, degraded=bool(result.get("degradations")),
vocabulary=Vocabulary.load(language), version=str(result.get("image") or "recorded"),
) # fmt: skip
class PinSourceIndex(SourceIndex):
"""Resolve findings with the same matcher as a checkout, without exporting it."""
def __init__(self, clone: Path, pin: str) -> None:
super().__init__(clone)
self.clone, self.pin = Clone(clone), pin
def lines(self, path: str) -> list[str]:
if path not in self._lines:
self._lines[path] = self.clone.git("show", f"{self.pin}:{path}").splitlines()
return self._lines[path] or []
def dry_run(store: memory.FileStore, results: Path, cache: Path) -> dict[str, Any]:
"""Count existing examples whose engine becomes available; never write rows or blobs.
Only spent v1/v2 populations are read. Counts are examples, not the two
static/plane feature records. Other sources are reported with zero changes.
"""
root = Path(__file__).resolve().parents[2]
cases = {}
for population in ("population-v1", "population-v2"):
manifest = root / "benchmarks" / "independent" / f"{population}.toml"
for case in tomllib.loads(manifest.read_text(encoding="utf-8"))["case"]:
for side in ("vulnerable", "fixed"):
cases[(population, memory.repo_key(case["repo"]), case[side], side)] = case["id"]
rows = [r.row for r in store.rows(kind="example")]
if not rows:
raise ValueError("dry run read no examples from the store")
counts: dict[str, Counter[str]] = defaultdict(Counter)
for row in rows:
count = counts[row["source"]]
count["examples"] += 1
old = row["instruments"]["engine"]
reason = "not_run" if old.get("version") == "missing:not-run" else "none" if old["state"] == "none" else None
if reason is None:
continue
key = (row["split"], row["repo"], row["pin"], row["pin_role"])
case_id = cases.get(key)
if case_id is None:
continue
scans = results / row["split"].replace("population", "independent") / "scans"
result = resolve_scan(scans, case_id, row["pin_role"])
path, _, function = row["candidate"].partition("::")
engine = {}
if engine_for(result) is None:
index = PinSourceIndex(cache / "independent" / case_id, row["pin"])
lines = index.lines(path) # fail loudly if the candidate's input cannot be read
count["source_bytes_read"] += len("\n".join(lines).encode("utf-8"))
selected = dict(result or {})
selected["findings"] = [f for f in selected.get("findings", []) if f["site"].partition(":")[0] == path]
_, engine = _signals(index, [], selected, {})
part = recorded_engine(row["language"], engine.get((path, function, row["family"]), []), result, True)
if part.state == "ran":
count[f"{reason}_to_ran"] += 1
elif result is not None and part.state == "failed":
count[f"{reason}_to_recorded_failed"] += 1
return {source: {**{"not_run_to_ran": 0, "none_to_ran": 0}, **dict(count)} for source, count in sorted(counts.items())}
def done(run: Path, entry: str) -> bool: def done(run: Path, entry: str) -> bool:
state = (read_json(run / "state.json") or {}).get("tasks", {}) state = (read_json(run / "state.json") or {}).get("tasks", {})
return state.get(entry, {}).get("status") == "done" return state.get(entry, {}).get("status") == "done"
@@ -188,17 +300,7 @@ def static_parts(root: Path, index: SourceIndex, quick: Mapping, engine: Mapping
key = (path, function, family) key = (path, function, family)
parts: dict[str, Part] = {} parts: dict[str, Part] = {}
parts["quick"] = quick_part(quick.get(key, []), rules, quick_version) if covers(QUICK_LANGUAGES, language) else NONE parts["quick"] = quick_part(quick.get(key, []), rules, quick_version) if covers(QUICK_LANGUAGES, language) else NONE
if language not in ENGINE_LANGUAGES: parts["engine"] = recorded_engine(language, engine.get(key, []), engine_result, repository)
parts["engine"] = NONE
elif engine_state is not None:
parts["engine"] = engine_state
else:
result = engine_result or {}
questions, completed = int(result.get("questions") or 0), int(result.get("completed") or 0)
parts["engine"] = engine_part(
engine.get(key, []), completion=completed / questions if questions else None, degraded=bool(result.get("degradations")),
vocabulary=Vocabulary.load(language), version=str(result.get("image") or "recorded"),
) # fmt: skip
lines = index.lines(path) lines = index.lines(path)
parts["source"] = source_part(lines, language, function) parts["source"] = source_part(lines, language, function)
parts["roles"] = roles_part(function_of(lines, path, language, function), inferred, language) parts["roles"] = roles_part(function_of(lines, path, language, function), inferred, language)
@@ -237,21 +339,34 @@ def export(clone: Path, pin: str, target: Path) -> bool:
def main() -> int: def main() -> int:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--labels", type=Path, required=True) parser.add_argument("--labels", type=Path)
parser.add_argument("--units", type=Path, required=True, help="units.json of the harvest plane") parser.add_argument("--units", type=Path, help="units.json of the harvest plane")
parser.add_argument("--catalog", type=Path, default=DEFAULT_CATALOG) parser.add_argument("--catalog", type=Path, default=DEFAULT_CATALOG)
parser.add_argument("--cache", type=Path, default=Path.home() / ".cache" / "openultrasast") parser.add_argument("--cache", type=Path, default=Path.home() / ".cache" / "openultrasast")
parser.add_argument("--engine", type=Path, default=RESULTS / "plane" / "harvest-engine") parser.add_argument("--engine", type=Path, default=RESULTS / "plane" / "harvest-engine")
parser.add_argument("--verify-run", type=Path, default=RESULTS / "plane" / "harvest-verify") parser.add_argument("--verify-run", type=Path, default=RESULTS / "plane" / "harvest-verify")
parser.add_argument("--roles-run", type=Path, default=RESULTS / "plane" / "harvest-roles") parser.add_argument("--roles-run", type=Path, default=RESULTS / "plane" / "harvest-roles")
parser.add_argument("--v46", type=Path, default=RESULTS / "plane" / "validation-46") parser.add_argument("--v46", type=Path, default=RESULTS / "plane" / "validation-46")
parser.add_argument("--scratch", type=Path, required=True) parser.add_argument("--scratch", type=Path)
parser.add_argument("--min-free-gb", type=float, default=1.2) parser.add_argument("--min-free-gb", type=float, default=1.2)
parser.add_argument("--counts", type=Path, required=True) parser.add_argument("--counts", type=Path)
parser.add_argument("--store", help="OUSAST_MEMORY spec (default: the environment's)") parser.add_argument("--store", help="OUSAST_MEMORY spec (default: the environment's)")
parser.add_argument("--limit", type=int, default=0, help="stop after this many targets (a test)") parser.add_argument("--limit", type=int, default=0, help="stop after this many targets (a test)")
parser.add_argument("--kinds", default="pairs,pin", help="which targets: pairs, pin (population and recipe pins)") parser.add_argument("--kinds", default="pairs,pin", help="which targets: pairs, pin (population and recipe pins)")
parser.add_argument("--results-root", type=Path, default=RESULTS)
parser.add_argument("--dry-run", action="store_true", help="read local stored examples and print engine changes by source; no writes")
args = parser.parse_args() args = parser.parse_args()
if args.dry_run:
spec = args.store or os.environ.get("OUSAST_MEMORY") or ""
if spec and not spec.startswith("file://"):
parser.error("--dry-run requires a local file:// store (no network)")
store = memory.open_store(spec)
if not isinstance(store, memory.FileStore):
parser.error("--dry-run requires a local file:// store (no network)")
print(json.dumps({"dry_run": True, "by_source": dry_run(store, args.results_root, args.cache)}, sort_keys=True))
return 0
if any(getattr(args, name) is None for name in ("labels", "units", "scratch", "counts")):
parser.error("--labels, --units, --scratch and --counts are required unless --dry-run is used")
labels = [r for r in jsonl(args.labels) if r["unit"] == "pin" and r["source"] not in CONDITIONAL_SOURCES and not r.get("conditional")] labels = [r for r in jsonl(args.labels) if r["unit"] == "pin" and r["source"] not in CONDITIONAL_SOURCES and not r.get("conditional")]
units = json.loads(args.units.read_text(encoding="utf-8")) units = json.loads(args.units.read_text(encoding="utf-8"))
@@ -314,7 +429,7 @@ def main() -> int:
continue continue
split = {r["split"] for r in rows} | {unit["source"]} split = {r["split"] for r in rows} | {unit["source"]}
population = "independent-v1" if "population-v1" in split else "independent-v2" population = "independent-v1" if "population-v1" in split else "independent-v2"
engine_result = read_json(RESULTS / population / "scans" / f"{case_id}--{rows[0]['pin_role']}.json") engine_result = resolve_scan(args.results_root / population / "scans", case_id, rows[0]["pin_role"])
stems = [(args.verify_run, n) for n in names] + [(args.v46, unit["ref"])] stems = [(args.verify_run, n) for n in names] + [(args.v46, unit["ref"])]
owner = next( owner = next(
(n for n in names if f"{n}-roles" in {e for e in (read_json(args.roles_run / "state.json") or {}).get("tasks", {})}), (n for n in names if f"{n}-roles" in {e for e in (read_json(args.roles_run / "state.json") or {}).get("tasks", {})}),
+173
View File
@@ -0,0 +1,173 @@
"""AX task entry: object-scoped transfers and stdlib only; result PUT is completion."""
from __future__ import annotations
import http.client
import json
import os
import shutil
import subprocess
import sys
import tarfile
import time
import urllib.error
import urllib.request
from pathlib import Path
class TaskEntryError(RuntimeError):
"""A task-entry diagnostic containing only locally controlled text."""
def transfer(request, destination=None, *, readiness_budget=None):
"""Wait for actor egress on startup; bound later transfers to three attempts."""
deadline = time.monotonic() + readiness_budget if readiness_budget is not None else None
delay = 0.5
attempt = 0
while True:
remaining = deadline - time.monotonic() if deadline is not None else 60
if remaining <= 0:
raise TaskEntryError("egress not open after 60 s")
attempt += 1
try:
with urllib.request.urlopen(request, timeout=min(60, remaining)) as response:
if destination is not None:
with destination.open("wb") as output:
shutil.copyfileobj(response, output)
return
except urllib.error.HTTPError as exc:
exc.close()
if exc.code != 403 and not 500 <= exc.code < 600:
raise
if deadline is None and attempt >= 3:
raise
except (urllib.error.URLError, OSError, http.client.HTTPException):
if deadline is None and attempt >= 3:
raise
if deadline is not None:
remaining = deadline - time.monotonic()
if remaining <= 0:
raise TaskEntryError("egress not open after 60 s") from None
time.sleep(min(delay, remaining))
delay = min(delay * 2, 8)
else:
time.sleep(attempt)
def download(url, destination, *, readiness_budget=None):
transfer(url, destination, readiness_budget=readiness_budget)
def unpack(source, destination):
"""Reject links and traversal; source archives contain only regular files/directories."""
with tarfile.open(source) as archive:
for member in archive.getmembers():
target = (destination / member.name).resolve()
if not target.is_relative_to(destination.resolve()) or not (member.isfile() or member.isdir()):
raise ValueError("unsafe source archive member")
archive.extractall(destination, filter="data")
def failure_record(pin, reason):
return {
**pin,
"done": True,
"status": "failed",
"reason": reason,
"units": [
{
**unit,
"status": "failed" if unit.get("supported", True) else "unsupported",
"reason": reason,
"instrument": {"files": [], "bytes": 0, "jvm": []},
"questions": {},
"questions_asked": [],
"questions_completed": [],
"witness_rows": [],
"traces": [],
}
for unit in pin.get("units", [])
],
}
def main(*, work_root=Path("/tmp"), worker_script=None, readiness_budget=60):
pin = {}
failed = False
output = work_root / "out"
result = output / "result.json"
step = "prepare output"
try:
output.mkdir(parents=True, exist_ok=True)
result.unlink(missing_ok=True)
questions = work_root / "questions.json"
step = "GET questions"
download(os.environ["QUESTIONS_URL"], questions, readiness_budget=readiness_budget)
step = "parse questions"
pin = json.loads(questions.read_text())
if not isinstance(pin, dict):
pin = {}
raise ValueError("questions object must contain a prepared pin")
source = work_root / "source.tar"
step = "GET source"
download(os.environ["SOURCE_URL"], source)
step = "unpack source"
case = work_root / "case"
case.mkdir(parents=True, exist_ok=True)
unpack(source, case)
step = "configure worker"
deadline = int(os.environ.get("DEADLINE", "900"))
command = [
sys.executable,
str(worker_script or Path(__file__).with_name("engine_trace_worker.py")),
"--worker",
str(questions),
"--root",
str(case),
"--output",
str(result),
"--heap-profile",
"cluster",
"--deadline",
str(deadline),
"--question-deadline",
os.environ.get("QUESTION_DEADLINE", "120"),
]
step = "run worker"
completed = subprocess.run(command, timeout=deadline + 5, check=False)
if completed.returncode:
raise TaskEntryError(f"worker exited {completed.returncode}")
step = "read worker result"
if not result.is_file() or not json.loads(result.read_text()).get("done"):
raise TaskEntryError("worker did not write a completed result")
except Exception as exc:
failed = True
# Avoid including exception text containing a presigned URL in durable output.
if isinstance(exc, urllib.error.HTTPError):
detail = f"HTTP {exc.code}"
else:
detail = str(exc) if isinstance(exc, TaskEntryError) else type(exc).__name__
reason = f"{step}: {detail}"
try:
output.mkdir(parents=True, exist_ok=True)
result.write_text(json.dumps(failure_record(pin, reason)) + "\n")
except Exception:
print("task entry could not write failure result", file=sys.stderr)
step = "pack result"
try:
archive = work_root / "result.tar"
with tarfile.open(archive, "w") as tar:
for path in sorted(output.iterdir()):
tar.add(path, arcname=path.name)
data = archive.read_bytes()
step = "PUT result"
request = urllib.request.Request(os.environ["RESULT_URL"], data=data, method="PUT", headers={"Content-Type": "application/x-tar"})
transfer(request, readiness_budget=readiness_budget if failed else None)
except Exception as exc:
detail = f"HTTP {exc.code}" if isinstance(exc, urllib.error.HTTPError) else type(exc).__name__
print(f"{step}: {detail}; dispatcher deadline remains authoritative", file=sys.stderr)
return 0
if __name__ == "__main__":
raise SystemExit(main())
File diff suppressed because it is too large Load Diff
+319
View File
@@ -0,0 +1,319 @@
"""Host-only AX Task transport; credentials never enter task manifests."""
from __future__ import annotations
import hashlib
import io
import ipaddress
import json
import os
import queue
import re
import shutil
import subprocess
import tarfile
import tempfile
import threading
import time
import uuid
from datetime import timedelta
from pathlib import Path
from urllib.parse import urlsplit
import yaml
from engine_trace_k8s import pack
from openultrasast.config import load_dotenv
from openultrasast.plane.memory import PRESIGNED_ARCHIVE_SUFFIX, S3Store, open_store
class EgressError(ValueError):
"""Safe diagnostic without presigned URL contents."""
class SandboxFailure(Exception):
"""A terminal AX task without an authoritative result object."""
def public_endpoint(url):
"""Reject endpoints the actor's hostname-based HTTPS egress policy cannot reach."""
parts = urlsplit(url)
host = parts.hostname or ""
try:
ipaddress.ip_address(host)
is_ip = True
except ValueError:
is_ip = False
if (
parts.scheme != "https"
or not host
or is_ip
or "." not in host
or host.endswith((".localhost", ".local", ".internal"))
or parts.port not in (None, 443)
or parts.username
or parts.password
):
raise EgressError(
"AX requires a public HTTPS DNS hostname in S3_ENDPOINT and presigned URLs: "
"the egress policy allows the store hostname on port 443 only (no IP or plain HTTP)"
)
return host.lower()
def task_manifest(name, args, urls):
if not re.fullmatch(r"[^\s@]+@sha256:[0-9a-f]{64}", args.image):
raise ValueError("AX --image must be pinned by sha256 digest")
if set(urls) != {"SOURCE_URL", "QUESTIONS_URL", "RESULT_URL"}:
raise ValueError("AX Task accepts only source, questions and result presigned URLs")
for url in urls.values():
public_endpoint(url)
env = [{"name": key, "value": value} for key, value in urls.items()]
env.extend(
{"name": key, "value": str(value)}
for key, value in {"DEADLINE": args.deadline, "QUESTION_DEADLINE": args.question_deadline}.items()
)
if len(json.dumps(env).encode()) >= 32768:
raise ValueError("AX Task env exceeds the 32 KB limit")
return {
"apiVersion": "ax.io/v1alpha1",
"kind": "Task",
"metadata": {"name": name, "atespace": args.atespace},
"spec": {"image": args.image, "env": env, "command": ["python3", "/app/benchmarks/learn/engine_task_entry.py"]},
}
def task_state(text, name):
"""Accept AX YAML/JSON output and its human-readable tasks table."""
try:
doc = yaml.safe_load(text)
except yaml.YAMLError:
doc = None
rows = doc.get("items", doc.get("tasks", [doc])) if isinstance(doc, dict) else doc if isinstance(doc, list) else []
for row in rows:
if not isinstance(row, dict) or row.get("metadata", {}).get("name", row.get("name")) != name:
continue
status = row.get("status", {})
ip = next(
(status.get(key) or row.get(key) for key in ("workerIP", "workerIp", "worker_ip", "ip") if status.get(key) or row.get(key)),
None,
)
worker = status.get("worker")
if not ip and isinstance(worker, dict):
ip = worker.get("ip") or worker.get("address")
return str(status.get("phase", row.get("phase", ""))), ip
lines = [line.split() for line in text.splitlines() if line.strip()]
if lines:
headers = [re.sub(r"[^a-z]", "", cell.lower()) for cell in lines[0]]
for cells in lines[1:]:
if name not in cells:
continue
row = dict(zip(headers, cells, strict=False))
phase = row.get("phase", row.get("status", ""))
ip = row.get("workerip") or row.get("ip")
if not ip:
for cell in cells:
try:
ipaddress.ip_address(cell)
ip = cell
break
except ValueError:
pass
return phase, None if ip in {"<none>", "-", ""} else ip
listed = isinstance(doc, (dict, list)) or bool(lines and "name" in headers and {"phase", "status"}.intersection(headers))
return "Missing" if listed else "", None
def bounded_read(read, timeout):
"""Do not let SDK retries postpone task teardown beyond the pin deadline.
Only the read runs in a daemon thread; it cannot publish a late result to disk.
No executor shutdown waits for an SDK call whose socket has stopped responding.
"""
if timeout <= 0:
raise TimeoutError("AX result deadline")
replies = queue.Queue(maxsize=1)
def fetch():
try:
replies.put((True, read()))
except Exception as exc:
replies.put((False, exc))
threading.Thread(target=fetch, daemon=True, name="engine-ax-result-read").start()
try:
ok, result = replies.get(timeout=timeout)
except queue.Empty:
raise TimeoutError("AX result deadline") from None
if not ok:
raise result
return result
class AX:
def __init__(self, args, store=None, pause=time.sleep, clock=time.monotonic):
self.args = args
self.pause, self.clock = pause, clock
self.run = uuid.uuid4().hex
load_dotenv()
self.host = public_endpoint(os.environ.get("S3_ENDPOINT", ""))
# Validate the image before opening a store, whose constructor probes the bucket.
task_manifest("validate", args, {key: f"https://{self.host}/" for key in ("SOURCE_URL", "QUESTIONS_URL", "RESULT_URL")})
self.store = store if store is not None else open_store()
if store is None and not isinstance(self.store, S3Store):
raise ValueError("AX requires OUSAST_MEMORY=s3://bucket[/prefix]")
def ax(self, *args, manifest=None, timeout=30):
env = dict(os.environ)
if self.args.kubeconfig:
env["KUBECONFIG"] = str(Path(self.args.kubeconfig).expanduser())
done = subprocess.run(
[str(Path(self.args.ax_bin).expanduser()), *args, *(["--atespace", self.args.atespace] if args[0] != "apply" else [])],
input=json.dumps(manifest) if manifest is not None else None,
env=env,
capture_output=True,
text=True,
timeout=max(0.01, timeout),
)
if done.returncode:
# AX errors may echo manifest URLs; report only verb and exit code.
raise ValueError(f"ax {args[0]} failed (exit {done.returncode})")
return done
def receive(self, data, output, pin):
# Validate in isolation: malformed results must never become usable checkpoints.
with tempfile.TemporaryDirectory(prefix="engine-ax-result-") as temporary:
root = Path(temporary)
with tarfile.open(fileobj=io.BytesIO(data)) as archive:
archive.extractall(root, filter="data")
record = json.loads((root / "result.json").read_text())
units = record.get("units", [])
if record.get("done") and record.get("status") == "failed" and not units:
# Questions download can fail before the actor learns any unit identities.
# Keep that explicit failure, filling identities only from the host input.
record = {
**pin,
**record,
"units": [
{
**unit,
"status": "failed" if unit["supported"] else "unsupported",
"reason": record.get("reason", "AX task entry failed"),
"instrument": {"files": [], "bytes": 0, "cpg_path": None, "cpg_bytes": 0, "jvm": []},
"questions": {},
"questions_asked": [],
"questions_completed": [],
"witness_rows": [],
"traces": [],
}
for unit in pin["units"]
],
}
units = record["units"]
(root / "result.json").write_text(json.dumps(record))
expected = [u["unit"] for u in pin["units"]]
actual = [u["unit"] for u in units]
if not record.get("done") or sorted(actual) != sorted(expected):
raise ValueError("AX Task returned incomplete or mismatched units")
output.mkdir(parents=True, exist_ok=True)
# Keep the artifacts without interpreting any remote paths as host destinations.
shutil.copytree(root, output, dirs_exist_ok=True)
def execute(self, checkout, output, pin, pin_id):
attempts = []
try:
for attempt in range(2):
try:
return self._execute_attempt(checkout, output, pin, pin_id, attempt, attempts)
except SandboxFailure:
if attempt == 0:
self.pause(10)
return subprocess.CompletedProcess([], 1, "", "AX sandbox failed twice"), None
finally:
(output / "ax_attempts.json").write_text(json.dumps(attempts))
result_path = output / "result.json"
if result_path.exists():
record = json.loads(result_path.read_text())
record["ax_attempts"] = attempts
result_path.write_text(json.dumps(record))
def _execute_attempt(self, checkout, output, pin, pin_id, attempt, attempts):
digest = hashlib.sha256(f"{self.run}/{pin_id}/attempt-{attempt}".encode()).hexdigest()[:12]
slug = re.sub(r"[^a-z0-9]+", "-", str(pin_id).lower()).strip("-")[:22]
name = f"ousast-engine-{slug or 'pin'}-{digest}"
key_id = hashlib.sha256(str(pin_id).encode()).hexdigest()
prefix = f"engine-queue/{self.run}/{key_id}/attempt-{attempt}/"
keys = [prefix + suffix for suffix in (f"source{PRESIGNED_ARCHIVE_SUFFIX}", "questions.json", f"result{PRESIGNED_ARCHIVE_SUFFIX}")]
expires = timedelta(seconds=self.args.deadline + 600)
worker_ip = None
submitted = False
started = self.clock()
phase = ""
try:
self.store._put(keys[0], pack(checkout))
self.store._put(keys[1], json.dumps(pin).encode())
urls = {
"SOURCE_URL": self.store.presign_get(keys[0], expires),
"QUESTIONS_URL": self.store.presign_get(keys[1], expires),
"RESULT_URL": self.store.presign_put(keys[2], expires),
}
if any(public_endpoint(url) != self.host for url in urls.values()):
raise EgressError("AX presign hostname must match S3_ENDPOINT for the store egress policy")
manifest = task_manifest(name, self.args, urls)
end = self.clock() + self.args.deadline
# An apply timeout can still have created the task; cleanup must cover it.
submitted = True
self.ax("apply", "-f", "-", manifest=manifest, timeout=min(30, self.args.deadline))
while self.clock() < end:
result = bounded_read(lambda: self.store._get(keys[2]), end - self.clock())
if self.clock() >= end:
return None, worker_ip
phase = ""
try:
state = self.ax("get", "tasks", timeout=min(30, end - self.clock())).stdout
phase, ip = task_state(state, name)
worker_ip = ip or worker_ip
except (ValueError, OSError, subprocess.SubprocessError):
# The object is authoritative even when AX status is temporarily unavailable.
pass
if self.clock() >= end:
return None, worker_ip
if result is not None:
self.receive(result[0], output, pin)
return subprocess.CompletedProcess([], 0, "", ""), worker_ip
if phase.lower() in {"failed", "missing"}:
# The worker may have uploaded while we were reading AX status.
result = bounded_read(lambda: self.store._get(keys[2]), end - self.clock())
if self.clock() >= end:
return None, worker_ip
if result is not None:
self.receive(result[0], output, pin)
return subprocess.CompletedProcess([], 0, "", ""), worker_ip
raise SandboxFailure
self.pause(min(2, max(0, end - self.clock())))
return None, worker_ip
except (subprocess.TimeoutExpired, TimeoutError):
return None, worker_ip
except SandboxFailure:
raise
except Exception as exc:
# Never expose exception strings which may contain object-scoped bearer URLs.
detail = str(exc) if isinstance(exc, EgressError) else type(exc).__name__
return subprocess.CompletedProcess([], 1, "", "AX transport failed: " + detail), worker_ip
finally:
attempts.append({"name": name, "phase": phase, "seconds": self.clock() - started})
errors = []
if submitted:
try:
self.ax("delete", "task", name)
except Exception as exc:
# A successful list already confirmed a disappeared task is gone.
if phase != "Missing":
errors.append("task: " + type(exc).__name__)
for key in keys:
try:
self.store._delete(key)
except Exception as exc:
errors.append("object: " + type(exc).__name__)
if errors:
raise RuntimeError("AX cleanup failed; stop dispatching: " + ", ".join(errors))
+259
View File
@@ -0,0 +1,259 @@
"""Host Kubernetes transport; the embedded bootstrap uses only the standard library."""
from __future__ import annotations
import hashlib
import io
import json
import re
import subprocess
import tarfile
import time
import uuid
from datetime import timedelta
from openultrasast.plane.memory import PRESIGNED_ARCHIVE_SUFFIX, S3Store, open_store
IMAGE = "ghcr.io/norandom/openultrasast:2.0.1"
BOOTSTRAP = r"""
import io, json, os, pathlib, subprocess, sys, tarfile, urllib.request
def get(name):
with urllib.request.urlopen(os.environ[name], timeout=120) as response:
return response.read()
def put(data):
request = urllib.request.Request(os.environ['RESULT_URL'], data=data, method='PUT')
with urllib.request.urlopen(request, timeout=120) as response:
response.read()
try:
if os.environ.get('CHECK') == '1':
data = get('SOURCE_URL')
assert len(data) == 1024
subprocess.run([sys.executable, '-c', 'print(42)'], check=True)
put(data)
else:
for name, target in [('SOURCE_URL', '/case'), ('ANALYZER_URL', '/frozen')]:
pathlib.Path(target).mkdir(parents=True, exist_ok=True)
with tarfile.open(fileobj=io.BytesIO(get(name))) as archive:
archive.extractall(target, filter='data')
out = pathlib.Path('/out')
out.mkdir(exist_ok=True)
(out / 'pin.json').write_text(os.environ['PIN'])
env = dict(os.environ, PYTHONPATH='/frozen/src')
for key in ['SOURCE_URL', 'ANALYZER_URL', 'RESULT_URL']:
env.pop(key, None)
with (out / 'container.log').open('w') as log:
done = subprocess.run([sys.executable, '/frozen/benchmarks/learn/engine_trace_worker.py',
'--worker', '/out/pin.json', '--deadline', os.environ['DEADLINE'],
'--question-deadline', os.environ['QUESTION_DEADLINE']], env=env, stdout=log, stderr=log)
(out / 'exit.json').write_text(json.dumps(done.returncode))
buffer = io.BytesIO()
with tarfile.open(fileobj=buffer, mode='w') as archive:
for path in out.iterdir():
archive.add(path, arcname=path.name)
put(buffer.getvalue())
sys.exit(done.returncode)
except Exception as exc:
# urllib exceptions can contain bearer URLs. Never print exception text.
print('engine bootstrap failed: ' + type(exc).__name__, file=sys.stderr)
sys.exit(1)
"""
def pack(path):
buffer = io.BytesIO()
with tarfile.open(fileobj=buffer, mode="w") as archive:
for child in sorted(path.iterdir()):
archive.add(child, arcname=child.name)
return buffer.getvalue()
def manifests(name, run, args, urls, pin, check=False):
labels = {"engine-trace-run": run, "engine-trace-job": name, "engine-trace": "true"}
secret = {
"apiVersion": "v1",
"kind": "Secret",
"metadata": {"name": name, "namespace": args.kube_namespace, "labels": labels},
"type": "Opaque",
"stringData": urls,
}
env = [{"name": key, "valueFrom": {"secretKeyRef": {"name": name, "key": key}}} for key in urls]
env += [
{"name": key, "value": value}
for key, value in {
"PIN": json.dumps(pin),
"DEADLINE": str(args.deadline),
"QUESTION_DEADLINE": str(args.question_deadline),
"CHECK": "1" if check else "0",
}.items()
]
job = {
"apiVersion": "batch/v1",
"kind": "Job",
"metadata": {"name": name, "namespace": args.kube_namespace, "labels": labels},
"spec": {
"activeDeadlineSeconds": args.deadline,
"backoffLimit": 0,
"ttlSecondsAfterFinished": 86400,
"template": {
"metadata": {"labels": labels},
"spec": {
"restartPolicy": "Never",
"automountServiceAccountToken": False,
"securityContext": {"runAsUser": 1000, "runAsGroup": 1000, "fsGroup": 1000},
"volumes": [{"name": name, "emptyDir": {}} for name in ("case", "frozen", "out")],
"affinity": {
"podAntiAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": [
{
"labelSelector": {"matchLabels": {"engine-trace": "true"}},
"topologyKey": "kubernetes.io/hostname",
}
]
}
},
"containers": [
{
"name": "engine",
"image": args.image,
"command": ["python", "-c", BOOTSTRAP],
"env": env,
"volumeMounts": [{"name": name, "mountPath": "/" + name} for name in ("case", "frozen", "out")],
"resources": {"requests": {"memory": "3Gi", "cpu": "1"}, "limits": {"memory": "3Gi"}},
}
],
},
},
},
}
return job, secret
def redact(text):
return re.sub(r"https?://[^\s\"'<>]+", "[redacted URL]", text)
class Kubernetes:
def __init__(self, args, store=None):
self.args = args
self.run = uuid.uuid4().hex
self.store = store if store is not None else open_store()
if not isinstance(self.store, S3Store):
raise ValueError("Kubernetes requires OUSAST_MEMORY=s3://bucket[/prefix]")
def kubectl(self, *args, manifest=None, check=True):
result = subprocess.run(
["kubectl", "--context", self.args.kube_context, "--namespace", self.args.kube_namespace, *args],
input=json.dumps(manifest) if manifest is not None else None,
capture_output=True,
text=True,
timeout=30,
)
if check and result.returncode:
# kubectl can echo submitted Secret data on errors; never expose stderr.
raise ValueError(f"kubectl {args[0]} failed (exit {result.returncode})")
return result
def namespace(self):
self.kubectl("get", "nodes", "-o", "json")
self.kubectl("apply", "-f", "-", manifest={"apiVersion": "v1", "kind": "Namespace", "metadata": {"name": self.args.kube_namespace}})
def execute(self, source, checkout, output, pin, check=False):
name = "engine-" + uuid.uuid4().hex[:24]
pin_key = hashlib.sha256(json.dumps([pin.get("repo"), pin.get("pin")]).encode()).hexdigest()
prefix = f"engine-jobs/{self.run}/{pin_key}/"
keys = [prefix + name + PRESIGNED_ARCHIVE_SUFFIX for name in ("source", "analyzer", "result")]
expiry = timedelta(seconds=self.args.deadline + 900)
node = None
try:
self.store._put(keys[0], b"x" * 1024 if check else pack(checkout))
self.store._put(keys[1], b"" if check else pack(source))
urls = {
"SOURCE_URL": self.store.presign_get(keys[0], expiry),
"ANALYZER_URL": self.store.presign_get(keys[1], expiry),
"RESULT_URL": self.store.presign_put(keys[2], expiry),
}
job, secret = manifests(name, self.run, self.args, urls, pin, check)
# create avoids kubectl's last-applied annotation duplicating Secret values.
created = json.loads(self.kubectl("create", "-f", "-", "-o", "json", manifest=job).stdout)
secret["metadata"]["ownerReferences"] = [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": name,
"uid": created["metadata"]["uid"],
}
]
self.kubectl("create", "-f", "-", manifest=secret)
end = time.monotonic() + self.args.deadline + 60
reason = "host deadline waiting for Job"
while time.monotonic() < end:
state = json.loads(self.kubectl("get", "job", name, "-o", "json").stdout).get("status", {})
pods = json.loads(self.kubectl("get", "pods", "-l", f"engine-trace-job={name}", "-o", "json").stdout)
node = next((p.get("spec", {}).get("nodeName") for p in pods.get("items", []) if p.get("spec", {}).get("nodeName")), node)
result = self.store._get(keys[2])
failed = next((c for c in state.get("conditions", []) if c.get("type") == "Failed" and c.get("status") == "True"), None)
if failed or state.get("failed"):
reason = (failed or {}).get("reason", "Job failed") + ": " + (failed or {}).get("message", "")
break
if state.get("succeeded") and result is not None:
if check:
if result[0] != b"x" * 1024:
raise ValueError("round-trip bytes mismatch")
else:
with tarfile.open(fileobj=io.BytesIO(result[0])) as archive:
archive.extractall(output, filter="data")
if not (output / "result.json").is_file():
raise ValueError("Job produced no result.json")
record = json.loads((output / "result.json").read_text())
if not record.get("done") or {u["unit"] for u in record["units"]} != {u["unit"] for u in pin["units"]}:
raise ValueError("Job returned incomplete or mismatched units")
return subprocess.CompletedProcess([], 0, "", ""), node
time.sleep(2)
logs = self.kubectl("logs", f"job/{name}", "--tail=40", check=False)
reason = redact(reason + "\n" + logs.stdout + logs.stderr)
return subprocess.CompletedProcess([], 1, "", reason), node
except Exception as exc:
reason = "Kubernetes transport failed: " + type(exc).__name__ + ": " + redact(str(exc))
try:
logs = self.kubectl("logs", f"job/{name}", "--tail=40", check=False)
reason += "\n" + redact(logs.stdout + logs.stderr)
except (OSError, ValueError, subprocess.SubprocessError):
pass
return subprocess.CompletedProcess([], 1, "", reason), node
finally:
if not self.args.keep_jobs:
# Attempt every cleanup even if an earlier deletion fails.
errors = []
for kind in ("job", "secret"):
try:
self.kubectl("delete", kind, name, "--ignore-not-found", "--wait=true", "--timeout=30s")
except Exception as exc:
errors.append(type(exc).__name__)
for key in keys:
try:
self.store._delete(key)
except Exception as exc:
errors.append(type(exc).__name__)
if errors:
raise RuntimeError("engine Job cleanup failed: " + ", ".join(errors))
def check(self):
self.namespace()
nodes = json.loads(self.kubectl("get", "nodes", "-o", "json").stdout)
for node in nodes["items"]:
print(json.dumps({"node": node["metadata"]["name"], "allocatable_memory": node["status"]["allocatable"]["memory"]}))
done, node = self.execute(None, None, None, {}, check=True)
print(
json.dumps(
{
"k8s_check": "passed" if done.returncode == 0 else "failed",
"node": node,
"reason": done.stderr,
"round_trip_bytes": 1024 if done.returncode == 0 else 0,
}
)
)
return done.returncode
+51
View File
@@ -0,0 +1,51 @@
"""Lossless parsing of the optional shortest-path trace, with explicit evidence limits."""
from __future__ import annotations
import re
from collections.abc import Mapping
from typing import Any
_STEP = re.compile(r"^(.*):(-?\d+) \[([^\]]+)\] ([\s\S]*)$")
def parse_trace(row: Mapping[str, Any]) -> dict[str, Any]:
"""Preserve row-level discharge evidence; never guess which node sanitized a path.
The query emits basename-only files and truncated code. A bound can be off-path;
it remains in ``guard`` even when no trace step exactly matches it. Invalid steps
raise instead of turning an unreadable trace into an empty path.
"""
raw = row.get("trace", [])
if not isinstance(raw, list):
raise ValueError("trace must be an array")
steps = []
bound = str(row.get("bound") or "")
for index, text in enumerate(raw):
match = _STEP.fullmatch(text) if isinstance(text, str) else None
if match is None:
raise ValueError(f"invalid trace step: {text!r}")
location, line, node_label, code = match.groups()
file, separator, method = location.partition(":")
roles = ["source"] if index == 0 else ["step"]
if index == len(raw) - 1:
roles = ["source", "sink"] if index == 0 else ["sink"]
if bound and code == bound:
roles.append("guard")
steps.append(
{
"file": file,
"method": method if separator else "?",
"line": int(line),
"node_label": node_label,
"code": code,
"roles": roles,
"raw": text,
}
)
return {
"steps": steps,
"sanitized": bool(row.get("sanitized", False)),
"sanitizer": {"on_path": bool(row.get("sanitized", False)), "step": None},
"guard": {"text": bound, "bounded": bool(row.get("bounded", False))},
}
+52
View File
@@ -0,0 +1,52 @@
"""Host queue transport: source and task metadata only, never analyzer code."""
from __future__ import annotations
import json
import subprocess
import time
import uuid
from engine_worker_service import PENDING, client_from_env, pack, put, read, result_key, status
from openultrasast.plane.memory import PRESIGNED_ARCHIVE_SUFFIX
class Dispatcher:
def __init__(self, args, client=None, *, pause=time.sleep, clock=time.monotonic):
self.client = client if client is not None else client_from_env()
self.run = uuid.uuid4().hex
self.args, self.pause, self.clock = args, pause, clock
def execute(self, checkout, output, pin, pin_id):
task_id = f"{self.run}-{pin_id}"
task = dict(
run=self.run,
id=pin_id,
task_id=task_id,
pin=pin,
input=f"engine-queue/inputs/{self.run}/{pin_id}/source{PRESIGNED_ARCHIVE_SUFFIX}",
image=self.args.image,
deadline=self.args.deadline,
question_deadline=self.args.question_deadline,
)
try:
self.client.put(task["input"], pack(checkout), {})
put(self.client, PENDING + task_id, task)
end = self.clock() + self.args.queue_timeout
while self.clock() < end:
result = read(self.client, result_key(task))
if result is not None:
(output / "result.json").write_text(json.dumps(result))
self.client.delete(task["input"])
return subprocess.CompletedProcess([], 0, "", ""), result.get("worker")
self.pause(2)
# Removing pending cancels work that has not been claimed; retain source for a live lease.
self.client.delete(PENDING + task_id)
return None, None
except Exception as exc:
# SDK errors can contain endpoint URLs or credentials. Do not persist their text.
raise OSError("queue transport failed") from exc
def status(self):
return status(self.client)
+644
View File
@@ -0,0 +1,644 @@
"""Container-only Joern trace worker; host prepares all corpus and question metadata.
Keep this module independent of learn/plane and their optional dependencies. The
frozen host invokes this file directly in the shipped, offline analyzer image.
"""
from __future__ import annotations
import argparse
import contextlib
import hashlib
import json
import os
import selectors
import signal
import subprocess
import tempfile
import time
from collections import Counter
from pathlib import Path
from engine_trace_parse import parse_trace
from openultrasast.cpg.backend import DATAFLOW_OVERLAY, JoernBackend, _render, extract_payload
from openultrasast.cpg.session import EngineSession
from openultrasast.model.contracts import ExecutionBudget
def write_json(path, value):
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(".tmp")
tmp.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n")
tmp.replace(path)
def safe_path(root, name):
path = (root / name).resolve()
if not path.is_relative_to(root.resolve()):
raise ValueError(f"path escapes input root: {name}")
return path
FRONTENDS = {"php2cpg", "pysrc2cpg", "jssrc2cpg", "javasrc2cpg", "c2cpg", "joern-parse"}
MIN_CPG_BYTES = 1024
HEAP_MB = 2560
CLUSTER_JVM_FLAGS = (
"-Xms1700m -Xmx1700m -XX:MaxMetaspaceSize=256m -XX:ReservedCodeCacheSize=128m "
"-XX:MaxDirectMemorySize=256m -Xss512k -XX:ActiveProcessorCount=2 -XX:+UseG1GC"
)
ENGINE_BUG = "engine bug: jssrc2cpg ObjectPropertyCallLinker"
EXCLUDED_DIRS = frozenset({"tests", "test", "docs", "node_modules", "vendor", "dist", "build", "templates", "migrations"})
def instrument_failure(proof):
if any(stage.get("oom") for stage in proof.get("jvm", [])):
return "JVM out of memory"
if not proof.get("files") or not proof.get("bytes"):
return "instrument read zero files or zero bytes"
if proof.get("cpg_bytes", 0) <= MIN_CPG_BYTES:
return "frontend wrote no CPG"
for stage in proof.get("jvm", []):
if stage["command"] in FRONTENDS:
if stage.get("cpg_bytes", 0) <= MIN_CPG_BYTES:
return "frontend wrote no CPG"
elif (
stage["command"] == "joern" or (stage["command"] == "joern-session" and stage.get("step") == "start" and stage.get("success"))
) and stage["seconds"] < 5:
return f"implausibly fast JVM step: {stage['command']} in {stage['seconds']:.3f}s (<5s)"
if not proof.get("jvm"):
return "no JVM step recorded"
return ""
class MeasuredBackend(JoernBackend):
"""Use the shipped backend with process and resident-session watchdogs."""
def __init__(self, deadline, question_deadline, checkpoint, heap_profile="vm"):
if heap_profile not in {"vm", "cluster"}:
raise ValueError(f"unknown heap profile: {heap_profile}")
self.heap_profile = heap_profile
super().__init__(
build_timeout=deadline,
query_timeout=deadline,
session_transport=False,
heap_mb=1700 if heap_profile == "cluster" else HEAP_MB,
)
self.deadline = time.monotonic() + deadline
self.question_deadline = question_deadline
self.checkpoint = checkpoint
self.stages = []
self.asked = []
self.answers = {}
self.completion_counts = Counter()
self.timed_out = False
self.fatal = ""
self.labelled_files = []
self.retry_pending = False
self.retries = []
self.session_fallback = ""
def _jvm_env(self):
env = super()._jvm_env()
if self.heap_profile == "cluster":
# Bound every JVM, including the child spawned by the Joern launcher.
env["JAVA_TOOL_OPTIONS"] = CLUSTER_JVM_FLAGS
env.pop("_JAVA_OPTIONS", None)
env.pop("JDK_JAVA_OPTIONS", None)
return env
def _session_stage(self, name, action):
started = time.monotonic()
result = None
try:
result = action()
return result
finally:
self.stages.append(
{
"command": "joern-session",
"step": name,
"success": result is not None and result is not False,
"seconds": time.monotonic() - started,
"heap_mb": self._heap_mb(),
"java_tool_options": self._jvm_env()["JAVA_TOOL_OPTIONS"],
}
)
self.checkpoint()
def _fallback_session(self, reason):
self.session_fallback = reason
if self._session is not None:
detail = self._session.log_tail()
if "OutOfMemoryError" in detail:
self.fatal = "JVM out of memory"
self.stages[-1]["oom"] = True
self.stages[-1]["output_tail"] = detail
self.close_session()
self.checkpoint()
def _session_for(self, cpg_path):
if self.session_fallback or self.fatal or self.timed_out:
return None
try:
if self._session is None:
self._session = EngineSession(
cpg_path.parent / "engine-session",
ExecutionBudget(self.deadline, 5),
self._jvm_env(),
startup_allowance_seconds=min(180, max(20, (self.deadline - time.monotonic()) / 3)),
)
if not self._session_stage("start", self._session.start):
self._fallback_session(self._session.failure or "session_start_failed")
return None
if self._session.loaded_graph != str(cpg_path):
if not self._session_stage("load", lambda: self._session.load(cpg_path, timeout=self.query_timeout)):
self._fallback_session(self._session.failure or "session_load_failed")
return None
self._defined.clear()
return self._session
except (OSError, RuntimeError) as exc:
self._fallback_session(f"session_start_or_load:{exc}")
return None
def _session_payload(self, cpg_path, query, params):
session = self._session_for(cpg_path)
if session is None:
return None
end = min(self.deadline, time.monotonic() + self.query_timeout)
question_end = end
prior_asked = list(self.asked)
prior_answers = dict(self.answers)
prior_counts = self.completion_counts.copy()
consumed = 0
streamed = ""
def observe(output):
nonlocal consumed, question_end, streamed
streamed = output
lines = output[consumed:].splitlines(keepends=True)
for line in lines:
if not line.endswith("\n"):
break
consumed += len(line)
try:
event = json.loads(line)
except ValueError:
continue
if not isinstance(event, dict):
continue
if "__question__" in event:
self.asked.append(event["__question__"])
question_end = min(end, time.monotonic() + self.question_deadline)
if "id" in event and "rows" in event:
self.answers[event["id"]] = event["rows"]
self.completion_counts[event["id"]] += 1
question_end = end
self.checkpoint()
if time.monotonic() >= min(end, question_end):
self.timed_out = True
return False
return True
def run():
source = (self.queries_dir / f"{query}.sc").read_text()
# Same shipped script/parameters as JoernBackend, using the graph loaded above.
if source.count("importCpg(cpgFile)") != 1:
return None
symbol = f"ousast_{query}"
if query not in self._defined:
source = source.replace("importCpg(cpgFile)", "()", 1)
source = source.replace("@main def exec(", f"def {symbol}(", 1)
if session.define(source, timeout=max(0, end - time.monotonic())) is None:
return None
self._defined.add(query)
arguments = ", ".join(f"{name} = {json.dumps(_render(value))}" for name, value in sorted(params.items()))
session.output_observer = observe
try:
answer = session.evaluate(f"{symbol}({arguments})", timeout=max(0, end - time.monotonic()))
finally:
session.output_observer = None
if answer is not None:
observe(answer.stdout)
return answer.stdout
return None
failure = ""
try:
body = self._session_stage(query, run)
except (OSError, RuntimeError) as exc:
body = None
failure = f"session_{query}:{exc}"
if self.timed_out:
self.close_session()
return streamed
parsed = extract_payload(body) if body is not None else None
if not isinstance(parsed, dict):
# Discard markers from an unverified attempt before replaying the script.
self.asked[:] = prior_asked
self.answers = prior_answers
self.completion_counts = prior_counts
self._fallback_session(failure or session.failure or f"session_{query}_payload_missing")
return None
if query == "census":
self.stages[-1]["files"] = int(parsed.get("files", 0))
if self.stages[-1]["files"] <= 0:
self.fatal = "query census read zero files"
return body
def _apply_overlays(self, cpg_path, scratch):
session = self._session_for(cpg_path)
result = False
if session is not None:
body = self._session_payload(cpg_path, "overlay", {"cpgFile": str(cpg_path)})
saved = session.scratch / "workspace" / cpg_path.name / "cpg.bin"
if body is not None and saved.is_file():
import shutil
shutil.copyfile(saved, cpg_path)
result = True
elif body is not None:
self._fallback_session("session_overlay_saved_graph_missing")
if not result:
result = super()._apply_overlays(cpg_path, scratch)
# An optional overlay failure must not prevent the taint-only attempt.
if self.fatal == ENGINE_BUG and not self.timed_out:
self.retry_pending = True
self.fatal = ""
return result
def _batch_once(self, cpg_path, query, requests):
if self.fatal or self.timed_out:
return None
if not self.retry_pending:
answers = super()._batch_once(cpg_path, query, requests)
if self.fatal != ENGINE_BUG:
return answers
if query != "taint" or self.timed_out or self.retries:
self.fatal = ENGINE_BUG
return None
self.retry_pending = False
self.fatal = ""
retry = {"reason": ENGINE_BUG, "mode": "taint-only", "completed": False}
self.retries.append(retry)
# Do not reuse rows/markers from the failed attempt as completion evidence.
self.asked.clear()
self.answers.clear()
self.completion_counts.clear()
original = self.queries_dir
with tempfile.TemporaryDirectory(prefix="trace-taint-", dir=cpg_path.parent) as scratch:
script = (original / "taint.sc").read_text()
marker = "importCpg(cpgFile)"
if script.count(marker) != 1:
self.fatal = ENGINE_BUG
return None
# Loading without enhancement skips the failing JS postprocessing passes.
# A raw graph without dataflow cannot establish a negative answer.
script = script.replace(
marker,
"importCpg(cpgFile, enhance = false)\n"
f' require(cpg.metaData.overlays.l.contains("{DATAFLOW_OVERLAY}"), '
'"taint-only retry requires saved dataflowOss overlay")',
)
self.queries_dir = Path(scratch)
(self.queries_dir / "taint.sc").write_text(script)
try:
answers = super()._batch_once(cpg_path, query, requests)
finally:
self.queries_dir = original
retry["completed"] = answers is not None and not self.fatal and not self.timed_out
if not retry["completed"]:
self.fatal = ENGINE_BUG
return answers
def _run(self, command, *, timeout, cwd=None):
if self.fatal or self.timed_out:
return None
if time.monotonic() >= self.deadline:
self.timed_out = True
return None
started = time.monotonic()
end = min(self.deadline, started + timeout)
question_end = end
binary = Path(command[0]).name
frontend = binary in FRONTENDS
query = binary == "joern" and "--script" in command
output = bytearray()
pending = b""
process = subprocess.Popen(
command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=cwd, env=self._jvm_env(), start_new_session=True
)
try:
with selectors.DefaultSelector() as selector:
selector.register(process.stdout, selectors.EVENT_READ)
while selector.get_map():
if time.monotonic() >= min(end, question_end):
self.timed_out = True
break
for key, _ in selector.select(min(0.2, max(0, min(end, question_end) - time.monotonic()))):
chunk = os.read(key.fileobj.fileno(), 65536)
if not chunk:
selector.unregister(key.fileobj)
continue
output.extend(chunk)
pending += chunk
while b"\n" in pending:
line, pending = pending.split(b"\n", 1)
try:
event = json.loads(line)
except ValueError:
continue
if not isinstance(event, dict):
continue
if "__question__" in event:
self.asked.append(event["__question__"])
question_end = min(end, time.monotonic() + self.question_deadline)
if "id" in event and "rows" in event:
self.answers[event["id"]] = event["rows"]
self.completion_counts[event["id"]] += 1
question_end = end
self.checkpoint()
if not self.timed_out:
process.wait(timeout=max(0.01, end - time.monotonic()))
finally:
with contextlib.suppress(ProcessLookupError):
os.killpg(process.pid, signal.SIGKILL)
process.wait(timeout=5)
process.stdout.close()
elapsed = time.monotonic() - started
if frontend or query:
stage = {
"command": binary,
"argv": command,
"heap_mb": self._heap_mb(),
"java_tool_options": self._jvm_env()["JAVA_TOOL_OPTIONS"],
"seconds": elapsed,
"exit": process.returncode,
"oom": b"OutOfMemoryError" in output,
}
lines = output.decode(errors="replace").splitlines()
stage["labelled_file_logs"] = {
file: [line for line in lines if file in line or Path(file).name in line]
for file in self.labelled_files
if any(file in line or Path(file).name in line for line in lines)
}
self.stages.append(stage)
if query and process.returncode != 0:
# stderr is merged into stdout above; drop JVM frames before taking
# the tail so they cannot crowd out the exception's message.
lines = [
line.strip()
for line in output.decode(errors="replace").splitlines()
if line.strip() and not line.strip().startswith(("at ", "... "))
]
stage["output_tail"] = "\n".join(lines[-20:])
if not self.timed_out:
self.fatal = f"joern query exited {process.returncode}"
if frontend:
# Capture the exact frontend output before build failure cleanup or overlays.
output_arg = next((i + 1 for i, arg in enumerate(command[:-1]) if arg in {"-o", "--output"}), None)
cpg_path = (Path(cwd or Path.cwd()) / command[output_arg]).resolve() if output_arg is not None else None
stage["cpg_path"] = str(cpg_path) if cpg_path is not None else None
stage["cpg_bytes"] = cpg_path.stat().st_size if cpg_path is not None and cpg_path.is_file() else 0
if stage["cpg_bytes"] <= MIN_CPG_BYTES:
self.fatal = "frontend wrote no CPG"
elif elapsed < 5 and not self.fatal:
self.fatal = f"implausibly fast JVM step: {command[0]} in {elapsed:.3f}s (<5s)"
if b"ObjectPropertyCallLinker failed" in output and b"Assignment statement with 3 arguments" in output:
self.fatal = ENGINE_BUG
stage["engine_bug"] = ENGINE_BUG
if b"OutOfMemoryError" in output:
self.fatal = "JVM out of memory"
self.checkpoint()
return subprocess.CompletedProcess(command, process.returncode if not self.fatal else 1, output.decode(errors="replace"), "")
def unit_record(unit, status, reason="", **extra):
return {
**unit,
"status": status,
"reason": reason,
"witness_rows": [],
"traces": [],
"instrument": {"files": [], "bytes": 0, "cpg_path": None, "cpg_bytes": 0, "jvm": []},
"questions": {},
"questions_asked": [],
"questions_completed": [],
**extra,
}
def worker_language(pin, root, out, deadline, question_deadline, persist=write_json, heap_profile="vm"):
proof = {
"files": [],
"bytes": 0,
"cpg_path": None,
"cpg_bytes": 0,
"jvm": [],
"heap_mb": 1700 if heap_profile == "cluster" else HEAP_MB,
"heap_profile": heap_profile,
"java_tool_options": CLUSTER_JVM_FLAGS
if heap_profile == "cluster"
else f"{os.environ.get('JAVA_TOOL_OPTIONS', '').strip()} -Xmx{HEAP_MB}m".strip(),
"excludes": sorted(EXCLUDED_DIRS),
"materialization": pin.get("materialization", {}),
}
record = {
**pin,
"done": False,
"instrument": proof,
"units": [unit_record(u, "unsupported" if not u["supported"] else "timeout", u["reason"]) for u in pin["units"]],
}
def checkpoint():
# Checkpoints retain raw completed rows even if the outer container deadline fires.
# They remain timeout until the finished JVM's instrument proof is validated.
for stage in backend.stages:
if "cpg_path" in stage:
proof["cpg_path"] = stage["cpg_path"]
proof["cpg_bytes"] = stage["cpg_bytes"]
if getattr(backend, "session_fallback", ""):
proof["session_fallback"] = backend.session_fallback
for row in record["units"]:
rid = row["unit"]
row["instrument"] = proof
row["questions_asked"] = [rid] if rid in backend.asked else []
if rid in backend.answers:
row["questions_completed"] = [rid]
row["witness_rows"] = [r for r in backend.answers[rid] if isinstance(r, dict) and "kind" not in r and "sink" in r]
persist(out, record)
backend = MeasuredBackend(deadline, question_deadline, checkpoint, **({"heap_profile": heap_profile} if heap_profile != "vm" else {}))
proof["jvm"] = backend.stages
proof["retries"] = getattr(backend, "retries", [])
backend.labelled_files = sorted({u["file"] for u in pin["units"] if u["supported"]})
cpg = None
try:
active = [u for u in pin["units"] if u["supported"]]
for file in sorted({u["file"] for u in active}):
data = safe_path(root, file).read_bytes()
proof["files"].append({"file": file, "bytes": len(data), "sha256": hashlib.sha256(data).hexdigest()})
proof["bytes"] += len(data)
print(f"read {file}: {len(data)} bytes", flush=True)
if not proof["bytes"]:
raise ValueError("instrument read zero files or zero bytes")
languages = {"javascript" if u["language"] == "typescript" else u["language"] for u in active}
if len(languages) != 1:
raise ValueError(f"one pin has multiple frontend languages: {sorted(languages)}")
language = next(iter(languages))
requests = {rid: {key: _render(value) for key, value in params.items()} for rid, params in pin["questions"].items()}
if set(requests) != {u["unit"] for u in active}:
raise ValueError("prepared questions do not match supported units")
record["questions"] = requests
for unit_row in record["units"]:
unit_row["questions"] = {unit_row["unit"]: requests[unit_row["unit"]]} if unit_row["unit"] in requests else {}
persist(out, record)
cpg = backend.build(root, language=language)
if cpg is None:
raise ValueError(backend.fatal or backend.last_failure or ("pin deadline" if backend.timed_out else "CPG build failed"))
if proof["cpg_path"] is None:
proof["cpg_path"] = str(cpg.cpg_path)
proof["cpg_bytes"] = cpg.cpg_path.stat().st_size if cpg.cpg_path.is_file() else 0
failure = instrument_failure(proof)
if failure:
raise ValueError(failure)
if cpg.run_batch_once is None:
raise ValueError("backend offers no batch operation")
answers = cpg.run_batch_once("taint", requests) or {}
record["census"] = answers.get("__census__", [])
failure = backend.fatal or instrument_failure(proof)
if not backend.asked and not backend.timed_out:
failure = failure or "query produced no markers"
census = record["census"]
if census and int(census[0].get("files", 0)) <= 0:
failure = failure or "query census read zero files"
elif not census and not backend.timed_out:
failure = failure or "query census is missing"
for index, unit in enumerate(pin["units"]):
if not unit["supported"]:
continue
rid = unit["unit"]
extra = {
"questions": {rid: requests[rid]},
"questions_asked": [rid] if rid in backend.asked else [],
"questions_completed": [rid] if rid in answers else [],
"instrument": proof,
}
rows = answers.get(rid)
# A batch across backend shards is complete only when every shard answered.
# The legacy merger can return a union even when one shard was killed.
shards = int(census[0].get("shards", 1)) if census else 1
shard_incomplete = shards > 1 and getattr(backend, "completion_counts", {}).get(rid, 0) < shards
if rows is not None:
raw_flows = [r for r in rows if isinstance(r, dict) and "kind" not in r and "sink" in r]
extra.update(witness_rows=raw_flows, traces=[parse_trace(r) for r in raw_flows])
if shard_incomplete:
extra["questions_completed"] = []
names = census[0].get("file_names", []) if census else []
file_missing = not any(str(p).removeprefix("./") == unit["file"] or str(p).endswith("/" + unit["file"]) for p in names)
dropped = file_missing and rows is not None
if failure or dropped:
status, reason = "failed", failure or "labelled file was dropped or absent from query census"
if dropped:
logs = [line for stage in backend.stages for line in stage.get("labelled_file_logs", {}).get(unit["file"], [])]
reason += "\n" + ("\n".join(dict.fromkeys(logs)) if logs else "frontend emitted no log lines mentioning labelled file")
elif rows is None or shard_incomplete:
status, reason = ("timeout" if backend.timed_out else "failed"), "question did not complete"
elif not rows and rid not in backend.asked:
status, reason = "failed", "query produced no markers"
else:
flows = [r for r in rows if isinstance(r, dict) and "kind" not in r and "sink" in r]
if any(not row.get("trace") for row in flows):
raise ValueError("taint witness returned without requested trace")
extra.update(witness_rows=flows, traces=[parse_trace(r) for r in flows])
status, reason = ("path" if flows else "asked-nothing"), ""
record["units"][index] = unit_record(unit, status, reason, **extra)
except (OSError, ValueError, RuntimeError) as exc:
for index, unit in enumerate(pin["units"]):
if unit["supported"]:
record["units"][index] = unit_record(
unit,
"timeout" if backend.timed_out and not backend.fatal else "failed",
str(exc),
instrument=proof,
questions={unit["unit"]: record["questions"][unit["unit"]]} if unit["unit"] in record.get("questions", {}) else {},
witness_rows=[r for r in backend.answers.get(unit["unit"], []) if isinstance(r, dict) and "sink" in r],
questions_completed=[unit["unit"]] if unit["unit"] in backend.answers else [],
questions_asked=[unit["unit"]] if unit["unit"] in backend.asked else [],
)
finally:
if hasattr(backend, "close_session"):
backend.close_session()
if cpg and cpg.cleanup:
cpg.cleanup()
for stage in backend.stages:
if stage["command"] == "joern" and stage.get("exit") and "output_tail" in stage:
detail = f"joern query exited {stage['exit']}: {stage['output_tail'] or 'no output'}"
for row in record["units"]:
if row["supported"] and row["status"] in {"failed", "timeout"}:
row["reason"] = f"{row['reason']}\n{detail}".strip()
record["questions_asked"] = list(dict.fromkeys(backend.asked))
record["questions_completed"] = [rid for u in record["units"] for rid in u["questions_completed"]]
record["done"] = True
persist(out, record)
return record
def worker(pin, root, out, deadline, question_deadline, heap_profile="vm"):
"""Run language partitions sequentially under one shared container deadline."""
languages = sorted({"javascript" if u["language"] == "typescript" else u["language"] for u in pin["units"] if u["supported"]})
if len(languages) <= 1:
return worker_language(pin, root, out, deadline, question_deadline, heap_profile=heap_profile)
end = time.monotonic() + deadline
record = {
**pin,
"done": False,
"units": [
unit_record(u, "timeout" if u["supported"] else "unsupported", "pin deadline" if u["supported"] else u["reason"])
for u in pin["units"]
],
"instruments": {},
}
def persist(_, partial):
updates = {u["unit"]: u for u in partial["units"]}
record["units"] = [updates.get(u["unit"], u) for u in record["units"]]
record["instruments"][language] = partial["instrument"]
record["questions_asked"] = [rid for u in record["units"] for rid in u["questions_asked"]]
record["questions_completed"] = [rid for u in record["units"] for rid in u["questions_completed"]]
write_json(out, record)
write_json(out, record)
for language in languages:
remaining = end - time.monotonic()
if remaining <= 0:
break
units = [
u for u in pin["units"] if u["supported"] and ("javascript" if u["language"] == "typescript" else u["language"]) == language
]
questions = {u["unit"]: pin["questions"][u["unit"]] for u in units}
worker_language(
{**pin, "units": units, "questions": questions}, root, out, remaining, question_deadline, persist, heap_profile=heap_profile
)
record["done"] = True
write_json(out, record)
return record
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--worker", type=Path, required=True)
parser.add_argument("--deadline", type=int, default=900)
parser.add_argument("--question-deadline", type=int, default=120)
parser.add_argument("--heap-profile", choices=("cluster", "vm"), default="vm")
parser.add_argument("--root", type=Path, default=Path("/case"))
parser.add_argument("--output", type=Path, default=Path("/out/result.json"))
args = parser.parse_args()
if min(args.deadline, args.question_deadline) <= 0:
parser.error("deadlines must be positive")
worker(json.loads(args.worker.read_text()), args.root, args.output, args.deadline, args.question_deadline, args.heap_profile)
return 0
if __name__ == "__main__":
raise SystemExit(main())
+315
View File
@@ -0,0 +1,315 @@
"""Image-resident queue workers. Requires strongly consistent S3 GET/PUT/LIST.
Per-task Lamport bakery tickets use only ordinary object operations. A choosing
marker precedes ticket selection; contenders wait for choosing peers and lower
(ticket, worker) pairs. Late arrivals cannot displace a running owner. Leases
assume bounded clock skew (less than MARGIN / 2) and a worker killed by its deadline.
"""
from __future__ import annotations
import argparse
import contextlib
import ctypes
import io
import json
import logging
import multiprocessing
import os
import signal
import socket
import tarfile
import tempfile
import threading
import time
import uuid
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from engine_trace_worker import unit_record, worker
from openultrasast.plane.memory import S3Client, s3_settings
PENDING = "engine-queue/pending/"
CLAIMED = "engine-queue/claimed/"
RESULTS = "engine-results/"
MARGIN = 60
LOG = logging.getLogger(__name__)
def client_from_env():
settings = s3_settings(os.environ)
bucket = os.environ.get("S3_BUCKET")
if not bucket:
raise ValueError("S3_BUCKET is required")
return S3Client(bucket=bucket, **settings)
def read(client, key):
value = client.get(key)
return json.loads(value[0]) if value else None
def put(client, key, value):
client.put(key, json.dumps(value).encode(), {})
def result_key(task):
return f"{RESULTS}{task['run']}/{task['id']}.json"
def pack(root):
stream = io.BytesIO()
with tarfile.open(fileobj=stream, mode="w") as archive:
archive.add(root, arcname=".")
return stream.getvalue()
def unpack(data, root):
with tarfile.open(fileobj=io.BytesIO(data)) as archive:
# Source snapshots need only regular files and directories, never links/devices.
for member in archive.getmembers():
target = (root / member.name).resolve()
if not target.is_relative_to(root.resolve()) or not (member.isfile() or member.isdir()):
raise ValueError("unsafe source archive")
archive.extractall(root, filter="data")
class Queue:
def __init__(self, client, identity=None, *, now=time.time, pause=time.sleep, stop=None):
self.client = client
self.identity = identity or f"{socket.gethostname()}-{uuid.uuid4().hex}"
self.now, self.pause = now, pause
self.stop = stop or threading.Event()
def markers(self, task_id=None):
result = []
for key in self.client.keys(CLAIMED):
if task_id is None or key.rsplit("/", 1)[-1] == task_id:
marker = read(self.client, key)
if marker:
result.append((key, marker))
return result
def tasks(self):
tasks = {}
for key in self.client.keys(PENDING):
task = read(self.client, key)
if task:
tasks[task["task_id"]] = task
# A dead worker may have removed pending already. Its task remains in the lease.
for _, marker in self.markers():
if marker["expires"] < self.now():
task = marker["task"]
tasks.setdefault(task["task_id"], task)
return list(tasks.values())
def claim(self, task):
if read(self.client, result_key(task)) is not None or self.stop.is_set():
return None
key = f"{CLAIMED}{self.identity}/{task['task_id']}"
marker = dict(task=task, ticket=0, expires=self.now() + task["deadline"] + MARGIN)
put(self.client, key, marker)
try:
live = [m for _, m in self.markers(task["task_id"]) if m["expires"] > self.now()]
marker["ticket"] = 1 + max((m["ticket"] for m in live), default=0)
put(self.client, key, marker)
while not self.stop.is_set():
# Do not start once the execution budget no longer fits within our lease.
if self.now() >= marker["expires"] - task["deadline"] - MARGIN / 2:
break
peers = [(k, m) for k, m in self.markers(task["task_id"]) if k != key and m["expires"] > self.now()]
if any(m["ticket"] == 0 for _, m in peers):
self.pause(0.1)
continue
if any((m["ticket"], k) < (marker["ticket"], key) for k, m in peers):
break # loser backs off and can try another pending task
if read(self.client, key) != marker or read(self.client, result_key(task)) is not None:
break
self.client.delete(PENDING + task["task_id"])
return key
except Exception:
self.client.delete(key)
raise
self.client.delete(key)
return None
def process_one(self, execute=None):
for task in self.tasks():
key = self.claim(task)
if key is None:
continue
try:
lease = read(self.client, key)
if lease is None:
continue
budget = min(task["deadline"], lease["expires"] - self.now() - MARGIN / 2)
if budget <= 0:
continue # Leave the expired lease's task available to recovery.
result = (execute or execute_task)(self.client, {**task, "runtime_budget": budget})
result.update(executor="queue", worker=self.identity, pod=socket.gethostname(), image=os.environ.get("ENGINE_IMAGE", ""))
put(self.client, result_key(task), result)
# Publish the result before removing either the input or the claim.
self.client.delete(task["input"])
for old_key, old in self.markers(task["task_id"]):
if old_key == key or old["expires"] < self.now():
self.client.delete(old_key)
return True
except Exception:
# Keep the lease and task for recovery, without logging credential-bearing errors.
LOG.error("task failed; lease retained for recovery")
return False
return False
def _child(pin, root, out, deadline, question_deadline):
os.setsid()
signal.signal(signal.SIGTERM, signal.SIG_DFL)
worker(pin, root, out, deadline, question_deadline)
def adopt_orphans():
# Linux PR_SET_CHILD_SUBREAPER: escaped Joern sessions reparent here, not
# outside the service. There is exactly one task process tree per worker.
if ctypes.CDLL(None, use_errno=True).prctl(36, 1, 0, 0, 0) != 0:
raise OSError("cannot enable task descendant cleanup")
def kill_children():
"""Freeze parents before walking children, including new-session JVMs."""
seen = set()
def freeze(parent):
for entry in Path("/proc").iterdir():
if not entry.name.isdigit() or int(entry.name) in seen:
continue
try:
# comm can contain spaces or parentheses. Fields after it begin at state.
fields = (entry / "stat").read_text().rsplit(")", 1)[1].split()
if int(fields[1]) != parent:
continue
pid = int(entry.name)
os.kill(pid, signal.SIGSTOP)
seen.add(pid)
freeze(pid)
except (FileNotFoundError, ProcessLookupError, PermissionError):
continue
# A child that exited during discovery can reparent descendants to this
# subreaper. Repeat until every remaining descendant is frozen.
while True:
count = len(seen)
freeze(os.getpid())
if len(seen) == count:
break
for pid in seen:
with contextlib.suppress(ProcessLookupError):
os.kill(pid, signal.SIGKILL)
def execute_task(client, task):
adopt_orphans()
pin = task["pin"]
started = time.monotonic()
budget = task.get("runtime_budget", task["deadline"])
status, reason, code = "failed", "worker produced no result", None
with tempfile.TemporaryDirectory(prefix="engine-task-") as scratch:
root, out = Path(scratch) / "case", Path(scratch) / "result.json"
root.mkdir()
try:
if task["deadline"] > 900 or task["deadline"] <= 0:
raise ValueError("deadline exceeds deployment grace")
if task.get("image") != os.environ.get("ENGINE_IMAGE"):
raise ValueError("task image differs from deployed image")
source = client.get(task["input"])
if source is None:
raise ValueError("missing source")
unpack(source[0], root)
# The shipped worker also checks frontend input and graph output for every language.
files = [p for p in root.rglob("*") if p.is_file()]
if not files:
raise ValueError("empty source")
LOG.info("input proof: first file bytes=%d", len(files[0].read_bytes()))
if time.monotonic() - started >= budget:
status, reason = "timeout", "task input deadline"
raise TimeoutError(reason)
process = multiprocessing.get_context("fork").Process(target=_child, args=(pin, root, out, budget, task["question_deadline"]))
process.start()
try:
process.join(max(0, budget - (time.monotonic() - started)))
if process.is_alive():
status, reason = "timeout", "task deadline"
else:
reason = f"worker exit {process.exitcode}"
code = process.exitcode
finally:
# Joern creates separate sessions. Kill the whole descendant tree,
# not merely the Python wrapper's process group.
kill_children()
process.join()
# Reap adopted JVMs and their launchers after the direct child.
while True:
try:
os.waitpid(-1, 0)
except ChildProcessError:
break
record = json.loads(out.read_text()) if out.exists() else {**pin, "units": []}
except Exception:
record = {**pin, "units": []}
reason = "worker input or execution failed"
if not record.get("done") or code != 0:
previous = {u["unit"]: u for u in record["units"]}
record["units"] = [
previous[u["unit"]]
if code is None and previous.get(u["unit"], {}).get("status") in {"path", "asked-nothing"}
else unit_record(u, status if u["supported"] else "unsupported", reason if u["supported"] else u["reason"])
for u in pin["units"]
]
record.update(done=True, container_exit=code, seconds=time.monotonic() - started)
return record
def status(client):
return {
"pending": len(client.keys(PENDING)),
"claimed": len(client.keys(CLAIMED)),
"done": len([k for k in client.keys(RESULTS) if k.endswith(".json")]),
}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--port", type=int, default=8080)
args = parser.parse_args()
logging.basicConfig(level=logging.INFO)
stop = threading.Event()
signal.signal(signal.SIGTERM, lambda *_: stop.set())
signal.signal(signal.SIGINT, lambda *_: stop.set())
client = client_from_env()
queue = Queue(client, stop=stop)
class Health(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200 if self.path == "/healthz" and not stop.is_set() else 503)
self.end_headers()
def log_message(self, *_):
pass
server = ThreadingHTTPServer(("0.0.0.0", args.port), Health)
threading.Thread(target=server.serve_forever, daemon=True).start()
try:
while not stop.is_set():
try:
if not queue.process_one():
stop.wait(2)
except Exception:
LOG.error("queue unavailable; retrying")
stop.wait(5)
finally:
server.shutdown()
server.server_close()
if __name__ == "__main__":
main()
+239
View File
@@ -0,0 +1,239 @@
"""Offline audit of completed Joern results; never launches an analyzer or reads corpora.
Missing results stay in coverage, not signal denominators. Observed non-path rows
score zero for path features. Sink kinds are one-hot indicators, never ordinal
category IDs. Both signed and best-direction metrics expose reversed shortcuts.
"""
from __future__ import annotations
import argparse
import ast
import hashlib
import json
import re
from collections import Counter
from pathlib import Path
from slice_leak_audit import THRESHOLD, complete_pairs, metrics
ROOT = Path(__file__).resolve().parents[2]
UNITS = ROOT / "plane/experiments/exp-005-graph-slice.units.jsonl"
RESULTS = Path.home() / "ousast-results/plane/engine-trace"
OUTPUT = ROOT / "benchmarks/measurements/2026-10-04-joern-slice-leak-audit/record.json"
STATUSES = ("path", "asked-nothing", "failed", "timeout", "unsupported", "missing")
ASKED = {"path", "asked-nothing"}
SIGNALS = ("has_path", "shortest_path_steps", "sanitized", "guard_present", "asked")
ASYMMETRY = ("vulnerable_only", "fixed_only", "both", "neither", "not_both_asked")
def materialization_version():
# Read the producer's literal without importing its execution dependencies.
tree = ast.parse(Path(__file__).with_name("engine_trace.py").read_bytes())
for node in tree.body:
if isinstance(node, ast.Assign) and any(isinstance(t, ast.Name) and t.id == "MATERIALIZATION_VERSION" for t in node.targets):
return ast.literal_eval(node.value)
raise ValueError("Producer has no materialization version")
def features(row):
path = row["status"] == "path"
traces = row.get("traces", []) if path else []
candidates = [(i, t) for i, t in enumerate(traces) if t.get("steps")]
index, trace = min(candidates, key=lambda it: len(it[1]["steps"])) if candidates else (None, {})
witnesses = row.get("witness_rows", [])
witness = witnesses[index] if index is not None and index < len(witnesses) else {}
guard = trace.get("guard", {})
sink = witness.get("sinkKind")
if not sink:
# Retain only the call target: arguments, literals and code are not kinds.
match = re.search(r"([\w:$>.\\-]+)\s*\(", str(witness.get("sink", "")))
sink = match[1] if match else "unknown"
return {
"has_path": int(path),
"shortest_path_steps": len(trace["steps"]) if trace else None if path else 0,
"sanitized": int(bool(trace.get("sanitized", witness.get("sanitized", False)))) if trace else None if path else 0,
"guard_present": int(
bool(guard.get("text") or guard.get("bounded") or witness.get("bound") or witness.get("bounded"))
or any("guard" in s.get("roles", []) for s in trace.get("steps", []))
)
if trace
else None
if path
else 0,
"sink_kind": str(sink) if path else "no-path",
"asked": int(row["status"] in ASKED),
}
def scored(rows, signal):
eligible = [r for r in rows if r[signal] is not None]
result = metrics(eligible, signal)
result["excluded_rows"] = len(rows) - len(eligible)
result["flagged"] = (result["pairs"] >= 10 and (result["best_direction_within_pair_accuracy"] or 0) >= THRESHOLD) or (
result["rows"] >= 10 and (result["best_direction_pooled_auc"] or 0) >= THRESHOLD
)
return result
def signal_metrics(rows, kinds):
return {
**{s: scored(rows, s) for s in SIGNALS},
"sink_kind": {kind: scored([{**r, "is_kind": int(r["sink_kind"] == kind)} for r in rows], "is_kind") for kind in kinds},
}
def scope_report(rows, kinds):
pairs = complete_pairs(rows)
asymmetry = Counter()
honest = []
for v, f in pairs:
if not (v["asked"] and f["asked"]):
asymmetry["not_both_asked"] += 1
else:
honest.extend((v, f))
asymmetry[
"both"
if v["has_path"] and f["has_path"]
else "vulnerable_only"
if v["has_path"]
else "fixed_only"
if f["has_path"]
else "neither"
] += 1
observed = [r for r in rows if r["status"] != "missing"]
return {
"rows": len(rows),
"coverage": {s: sum(r["status"] == s for r in rows) for s in STATUSES},
"complete_pairs": len(pairs),
"pair_asymmetry": {s: asymmetry[s] for s in ASYMMETRY},
"both_asked_pairs": len(honest) // 2,
"signals": signal_metrics(observed, kinds),
"both_asked_signals": signal_metrics(honest, kinds),
}
def audit(units, results, *, expected_rows=501, expected_pairs=195):
if len(units) != expected_rows or len({u["unit"] for u in units}) != expected_rows:
raise ValueError(f"Expected {expected_rows} unique units")
if len(complete_pairs(units)) != expected_pairs:
raise ValueError(f"Expected {expected_pairs} complete pairs")
if not results.is_dir():
raise ValueError(f"Results directory is unreadable: {results}")
version = materialization_version()
unit_index = {u["unit"]: u for u in units}
matched = {}
files = []
counts = Counter()
for path in sorted(results.glob("*.json")):
raw = path.read_bytes()
record = json.loads(raw) # Corruption fails loudly; producer uses atomic replace.
if not isinstance(record, dict) or "repo" not in record or "pin" not in record:
continue
stale = record.get("materialization_version") != version
unfinished = record.get("done") is not True
counts["stale"] += stale
counts["unfinished"] += unfinished
counts["skipped"] += stale or unfinished
files.append((path.name, len(raw)))
if stale or unfinished:
continue
counts["accepted_files"] += 1
counts["result_rows_read"] += len(record["units"])
for row in record["units"]:
if row["unit"] not in unit_index:
continue
unit = unit_index[row["unit"]]
if any(row.get(k) != unit.get(k) for k in ("family", "label", "pair")):
raise ValueError(f"Result metadata mismatch: {row['unit']}")
if row["status"] not in STATUSES[:-1] or not row.get("source"):
raise ValueError(f"Invalid result status/source: {row['unit']}")
if row["unit"] in matched:
raise ValueError(f"Duplicate current result: {row['unit']}")
matched[row["unit"]] = {
**unit,
"source": row["source"],
"status": row["status"],
"pin_role": row.get("pin_role"),
**features(row),
}
rows = [
matched.get(u["unit"], {**u, "source": u.get("source", "unknown"), "status": "missing", **features({"status": "missing"})})
for u in units
]
kinds = sorted({r["sink_kind"] for r in rows if r["status"] != "missing"})
scopes = {"overall": scope_report(rows, kinds)}
for field in ("family", "source"):
scopes[f"by_{field}"] = {
value: scope_report([r for r in rows if r[field] == value], kinds) for value in sorted({r[field] for r in rows})
}
return {
"cost_usd": 0,
"materialization_version": version,
"threshold": THRESHOLD,
"minimum_samples": 10,
"unit_rows_read": len(units),
"matched_rows": len(matched),
"inputs": {
**{k: counts[k] for k in ("accepted_files", "result_rows_read", "stale", "unfinished", "skipped")},
"files_read": len(files),
"bytes_read": sum(size for _, size in files),
# Hash a canonical inventory; filenames and paths may identify repositories.
"files_sha256": hashlib.sha256(json.dumps(sorted(files), separators=(",", ":")).encode()).hexdigest(),
},
"semantics": {
"signals": "Shortest nonempty trace (first on ties); its sanitizer, guard and corresponding witness sink call target.",
"missing": "Coverage only; absent source metadata is unknown. Skipped result units are never consumed.",
"non_path": "Observed non-path rows score zero; path rows lacking trace evidence have null trace features.",
"asked": "path or asked-nothing=1; failed, timeout, unsupported=0; missing excluded from metrics.",
"asymmetry": "Disjoint categories; not_both_asked takes precedence over path asymmetry.",
"flags": "Best of both directions, >=0.70 with >=10 pairs for pair accuracy or >=10 rows for AUC.",
"sink_kind": "One binary indicator per observed call target, including no-path/unknown; no ordinal encoding.",
"limits": "Exploratory associations, not proof of leakage; running input snapshot, no multiple-testing correction.",
},
**scopes,
"units": [{k: r[k] for k in ("unit", "pair", "family", "label", "fold", "source", "status", *SIGNALS, "sink_kind")} for r in rows],
}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--results", type=Path, default=RESULTS)
parser.add_argument("--units", type=Path, default=UNITS)
parser.add_argument("--output", type=Path, default=OUTPUT)
args = parser.parse_args()
raw = args.units.read_bytes()
units = [json.loads(line) for line in raw.splitlines() if line.strip()]
report = audit(units, args.results)
report["units_input"] = {"bytes": len(raw), "sha256": hashlib.sha256(raw).hexdigest()}
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n")
overall, inputs = report["overall"], report["inputs"]
lines = [
f"Joern leak audit: $0; materialization_version={report['materialization_version']}",
f"Units read: {len(units)}; bytes={len(raw)}; file={args.units}",
f"Result files read: {inputs['files_read']}; bytes={inputs['bytes_read']}; accepted={inputs['accepted_files']}",
f"Skipped: {inputs['skipped']} (stale={inputs['stale']}, unfinished={inputs['unfinished']}; counts overlap)",
f"Result rows read: {inputs['result_rows_read']}; matched={report['matched_rows']}",
f"Coverage: {overall['coverage']}",
f"Pairs: {overall['complete_pairs']}; both asked={overall['both_asked_pairs']}",
f"Asymmetry: {overall['pair_asymmetry']}",
]
for signal in (*SIGNALS, "sink_kind"):
def brief(value, signal=signal):
if signal == "sink_kind":
return {k: brief_metric(m) for k, m in value.items()}
return brief_metric(value)
lines.append(f"{signal}: all={brief(overall['signals'][signal])}; both asked={brief(overall['both_asked_signals'][signal])}")
lines.append(f"Family/source metrics and input inventory digest: {args.output}")
print("\n".join(lines))
def brief_metric(metric):
return {k: metric[k] for k in ("rows", "pairs", "within_pair_accuracy", "pooled_auc", "flagged")}
if __name__ == "__main__":
main()
+177
View File
@@ -0,0 +1,177 @@
"""Read-only, zero-model slice audit. Run: env -u OPENROUTER_API_KEY python benchmarks/learn/slice_leak_audit.py.
Only sources.toml itself is read, never its referenced corpora. Stored source IDs
are the reporting strata (the store merges advisory pair catalogs into `pairs`).
Ties get half credit. Both signed and best-direction scores are recorded; like
learn.leaks, a reversed label shortcut is still a leak. Missing classes/pairs
produce null metrics, not plausible zeros. Unsupported slices score zero for both
indicators and remain separately counted in coverage.
"""
from __future__ import annotations
import hashlib
import json
import os
from collections import Counter, defaultdict
from collections.abc import Mapping, Sequence
from pathlib import Path
from typing import Any
from openultrasast.config import load_dotenv
from openultrasast.learn.examples import load_examples
from openultrasast.learn.labels import load_sources
from openultrasast.learn.slice import function_slice
from openultrasast.plane.memory import MemoryStore, open_store
UNITS = Path("plane/experiments/exp-004-pooled-block-gate.units.jsonl")
OUTPUT = Path("benchmarks/measurements/2026-10-03-graph-slice-leak-audit/record.json")
THRESHOLD = 0.70
def complete_pairs(rows: Sequence[Mapping[str, Any]]) -> list[tuple[Mapping[str, Any], Mapping[str, Any]]]:
groups: dict[tuple[str, str], list[Mapping[str, Any]]] = defaultdict(list)
for row in rows:
if row.get("pair"):
groups[(row["family"], row["pair"])].append(row)
pairs = []
for members in groups.values():
if len(members) == 2 and {m["label"] for m in members} == {0, 1}:
v, f = sorted(members, key=lambda m: -m["label"])
if v["group"] != f["group"] or v["fold"] != f["fold"]:
raise ValueError("Pair crosses group or fold")
pairs.append((v, f))
elif len(members) != 1:
raise ValueError("Ambiguous pair")
return pairs
def metrics(rows: Sequence[Mapping[str, Any]], signal: str) -> dict[str, Any]:
pairs = complete_pairs(rows)
higher = sum(v[signal] > f[signal] for v, f in pairs)
lower = sum(v[signal] < f[signal] for v, f in pairs)
ties = len(pairs) - higher - lower
pair_score = (higher + 0.5 * ties) / len(pairs) if pairs else None
positive = [r[signal] for r in rows if r["label"] == 1]
negative = [r[signal] for r in rows if r["label"] == 0]
auc = (
sum((v > f) + 0.5 * (v == f) for v in positive for f in negative) / (len(positive) * len(negative))
if positive and negative
else None
)
best_pair = max(pair_score, 1 - pair_score) if pair_score is not None else None
best_auc = max(auc, 1 - auc) if auc is not None else None
return {
"rows": len(rows),
"pairs": len(pairs),
"positive": len(positive),
"negative": len(negative),
"higher_on_vulnerable": higher,
"higher_on_fixed": lower,
"ties": ties,
"within_pair_accuracy": pair_score,
"pooled_auc": auc,
"best_direction_within_pair_accuracy": best_pair,
"best_direction_pooled_auc": best_auc,
"flagged": any(v is not None and v >= THRESHOLD for v in (best_pair, best_auc)),
}
def audit(store: MemoryStore, units: Sequence[Mapping[str, Any]], *, expected_rows: int = 501, expected_pairs: int = 195) -> dict[str, Any]:
if len(units) != expected_rows or len({u["unit"] for u in units}) != expected_rows:
raise ValueError(f"Expected {expected_rows} unique units, read {len(units)}")
if len(complete_pairs(units)) != expected_pairs:
raise ValueError(f"Expected {expected_pairs} complete pairs")
examples = load_examples(store)
index = {e.id: e for e in examples}
if len(index) != len(examples):
raise ValueError("Duplicate example IDs in memory")
sources = {s.id for s in load_sources().sources}
coverage: dict[str, Counter[str]] = defaultdict(Counter)
rows = []
excerpt_bytes = 0
blobs: dict[str, str] = {}
for unit in units:
e = index[unit["unit"]] # missing input is fatal, never counted as no-flow
if (e.family, e.label, e.group) != (unit["family"], unit["label"], unit["group"]):
raise ValueError(f"Unit metadata differs from store: {e.id}")
if e.source not in sources:
raise ValueError(f"Source absent from sources.toml: {e.source}")
if e.excerpt_sha not in blobs:
# MemoryStore.get_blob(verify=True) DELETES corrupt blobs: do not use it.
data = store.get_blob("excerpts", e.excerpt_sha, verify=False)
if not data or hashlib.sha256(data).hexdigest() != e.excerpt_sha:
raise ValueError(f"Missing, empty or corrupt excerpt: {e.id}")
excerpt_bytes += len(data)
blobs[e.excerpt_sha] = data.decode("utf-8")
result = function_slice(blobs[e.excerpt_sha], e.language, e.family)
status = "not-analysable" if result is None else "flow" if result.steps else "no-flow"
coverage[e.family][status] += 1
rows.append(
{
**unit,
"source": e.source,
"status": status,
"has_flow": int(status == "flow"),
"slice_length": len(result.steps) if result else 0,
}
)
pairs = complete_pairs(rows)
asymmetry = Counter(
"both" if v["has_flow"] and f["has_flow"] else "vulnerable_only" if v["has_flow"] else "fixed_only" if f["has_flow"] else "neither"
for v, f in pairs
)
scopes = {"all": rows, **{f"source:{s}": [r for r in rows if r["source"] == s] for s in sorted({r["source"] for r in rows})}}
return {
"cost_usd": 0,
"threshold": THRESHOLD,
"unit_rows_read": len(units),
"example_rows_read": len(examples),
"matched_rows": len(rows),
"excerpts_read": len(blobs),
"excerpt_units": len(rows),
"excerpt_bytes": excerpt_bytes,
"complete_pairs": len(pairs),
"coverage": {f: {s: c[s] for s in ("flow", "no-flow", "not-analysable")} for f, c in sorted(coverage.items())},
"pair_asymmetry": {k: asymmetry[k] for k in ("vulnerable_only", "fixed_only", "both", "neither")},
"signals": {
signal: {scope: metrics(members, signal) for scope, members in scopes.items()} for signal in ("has_flow", "slice_length")
},
"source_strata": "Stored source IDs from sources.toml; advisory pair catalogs are merged into pairs by the example builder.",
"limitations": (
"Flat IR: no dominance/guards, textual branch order, potential parameter inputs, conservative call returns; no alias analysis."
),
"units": [
{k: r[k] for k in ("unit", "pair", "family", "label", "fold", "source", "status", "has_flow", "slice_length")} for r in rows
],
}
def main() -> None:
load_dotenv(Path(".env"))
os.environ.pop("OPENROUTER_API_KEY", None) # dotenv must not restore the removed key
raw = UNITS.read_bytes()
units = [json.loads(line) for line in raw.decode().splitlines() if line.strip()]
report = audit(open_store(), units)
report["units_bytes"] = len(raw)
report["units_sha256"] = hashlib.sha256(raw).hexdigest()
OUTPUT.parent.mkdir(parents=True, exist_ok=True)
OUTPUT.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n")
totals = Counter({s: sum(c[s] for c in report["coverage"].values()) for s in ("flow", "no-flow", "not-analysable")})
lines = [
f"Slice leak audit: $0; units bytes={len(raw)}",
f"Rows read: units={len(units)}, examples={report['example_rows_read']}, matched={report['matched_rows']}",
f"Excerpts read: {report['excerpts_read']}; units={report['excerpt_units']}; bytes={report['excerpt_bytes']}",
f"Coverage: {dict(totals)}",
f"Complete pairs: {report['complete_pairs']}",
f"Pair asymmetry: {report['pair_asymmetry']}",
f"Has flow: {report['signals']['has_flow']['all']}",
f"Slice length: {report['signals']['slice_length']['all']}",
f"Source strata: {', '.join(report['signals']['has_flow'])}; threshold={THRESHOLD}",
f"Report: {OUTPUT}",
]
print("\n".join(lines))
if __name__ == "__main__":
main()
@@ -1,7 +1,7 @@
{ {
"modules": 151, "modules": 159,
"counts": { "counts": {
"load_bearing": 146, "load_bearing": 154,
"standalone_capability": 5 "standalone_capability": 5
}, },
"orphaned": [], "orphaned": [],
@@ -229,6 +229,8 @@
"model.report", "model.report",
"pairs", "pairs",
"plane.tasks.alerts", "plane.tasks.alerts",
"push.quick",
"push.runner",
"rank", "rank",
"regress.candidate", "regress.candidate",
"reports", "reports",
@@ -356,12 +358,14 @@
"learn.excerpt", "learn.excerpt",
"learn.experiments", "learn.experiments",
"learn.features", "learn.features",
"learn.fill",
"learn.folds", "learn.folds",
"learn.labels", "learn.labels",
"learn.leaks", "learn.leaks",
"learn.program", "learn.program",
"learn.retrieve", "learn.retrieve",
"learn.roles", "learn.roles",
"learn.slice",
"plane.harvest", "plane.harvest",
"plane.tasks.features", "plane.tasks.features",
"plane.tasks.remember", "plane.tasks.remember",
@@ -410,7 +414,8 @@
"importers": [ "importers": [
"learn.cli", "learn.cli",
"learn.compile", "learn.compile",
"learn.experiments" "learn.experiments",
"learn.fill"
] ]
}, },
{ {
@@ -422,6 +427,7 @@
"learn.compile", "learn.compile",
"learn.evaluate", "learn.evaluate",
"learn.experiments", "learn.experiments",
"learn.fill",
"learn.folds", "learn.folds",
"learn.program", "learn.program",
"learn.retrieve" "learn.retrieve"
@@ -435,6 +441,7 @@
"learn.embeddings", "learn.embeddings",
"learn.examples", "learn.examples",
"learn.labels", "learn.labels",
"learn.slice",
"plane.harvest" "plane.harvest"
] ]
}, },
@@ -443,7 +450,8 @@
"classification": "load_bearing", "classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli", "reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [ "importers": [
"learn.cli" "learn.cli",
"learn.fill"
] ]
}, },
{ {
@@ -455,6 +463,14 @@
"plane.tasks.features" "plane.tasks.features"
] ]
}, },
{
"module": "learn.fill",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"learn.cli"
]
},
{ {
"module": "learn.folds", "module": "learn.folds",
"classification": "load_bearing", "classification": "load_bearing",
@@ -500,7 +516,8 @@
"learn.cli", "learn.cli",
"learn.compile", "learn.compile",
"learn.evaluate", "learn.evaluate",
"learn.experiments" "learn.experiments",
"learn.fill"
] ]
}, },
{ {
@@ -539,6 +556,14 @@
"plane.tasks.roles" "plane.tasks.roles"
] ]
}, },
{
"module": "learn.slice",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"learn.program"
]
},
{ {
"module": "map_gate", "module": "map_gate",
"classification": "load_bearing", "classification": "load_bearing",
@@ -560,6 +585,7 @@
"pairs", "pairs",
"plane.tasks.alerts", "plane.tasks.alerts",
"preprocess", "preprocess",
"push.quick",
"push.runner" "push.runner"
] ]
}, },
@@ -579,9 +605,12 @@
"cli", "cli",
"cpg.backend", "cpg.backend",
"cpg.session", "cpg.session",
"findings",
"learn.cli", "learn.cli",
"learn.features", "learn.features",
"learn.fill",
"learn.labels", "learn.labels",
"learn.slice",
"model.candidates", "model.candidates",
"model.config_value", "model.config_value",
"model.dominance", "model.dominance",
@@ -602,6 +631,7 @@
"push.cache", "push.cache",
"push.contracts", "push.contracts",
"push.policy", "push.policy",
"push.quick",
"push.report", "push.report",
"push.reuse", "push.reuse",
"push.runner", "push.runner",
@@ -649,6 +679,7 @@
"push.cache", "push.cache",
"push.contracts", "push.contracts",
"push.policy", "push.policy",
"push.quick",
"push.report", "push.report",
"push.reuse", "push.reuse",
"push.runner", "push.runner",
@@ -672,6 +703,7 @@
"importers": [ "importers": [
"cli", "cli",
"learn.cli", "learn.cli",
"learn.fill",
"plane.budget", "plane.budget",
"plane.generate", "plane.generate",
"push.assistance", "push.assistance",
@@ -716,7 +748,9 @@
"classification": "load_bearing", "classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli", "reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [ "importers": [
"model.scan" "findings",
"model.scan",
"push.runner"
] ]
}, },
{ {
@@ -775,6 +809,7 @@
"classification": "load_bearing", "classification": "load_bearing",
"reason": "model-grounded-detection Req 7.2-7.4: the security vocabularies the CPG queries are parameterised by", "reason": "model-grounded-detection Req 7.2-7.4: the security vocabularies the CPG queries are parameterised by",
"importers": [ "importers": [
"learn.slice",
"model.config_value", "model.config_value",
"model.dominance", "model.dominance",
"model.pipeline", "model.pipeline",
@@ -846,7 +881,10 @@
"learn.embeddings", "learn.embeddings",
"learn.examples", "learn.examples",
"learn.experiments", "learn.experiments",
"learn.fill",
"learn.program", "learn.program",
"plane.delivery",
"plane.doctor",
"plane.egress", "plane.egress",
"plane.engine_alerts", "plane.engine_alerts",
"plane.generate", "plane.generate",
@@ -871,12 +909,21 @@
"learn.compile", "learn.compile",
"learn.embeddings", "learn.embeddings",
"learn.experiments", "learn.experiments",
"learn.fill",
"learn.program", "learn.program",
"plane.generate", "plane.generate",
"plane.tasks.roles", "plane.tasks.roles",
"plane.tasks.verify" "plane.tasks.verify"
] ]
}, },
{
"module": "plane.delivery",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"plane.reconciler"
]
},
{ {
"module": "plane.doctor", "module": "plane.doctor",
"classification": "load_bearing", "classification": "load_bearing",
@@ -924,6 +971,7 @@
"reason": "ai-service-plane Req 1.1-1.4: ax-shaped Task/Workspace/Model and the Run kind, validated before anything runs", "reason": "ai-service-plane Req 1.1-1.4: ax-shaped Task/Workspace/Model and the Run kind, validated before anything runs",
"importers": [ "importers": [
"learn.cli", "learn.cli",
"plane.delivery",
"plane.egress", "plane.egress",
"plane.generate", "plane.generate",
"plane.harvest", "plane.harvest",
@@ -945,12 +993,27 @@
"learn.embeddings", "learn.embeddings",
"learn.examples", "learn.examples",
"learn.experiments", "learn.experiments",
"learn.fill",
"learn.program", "learn.program",
"plane.delivery",
"plane.doctor",
"plane.generate", "plane.generate",
"plane.reconciler",
"plane.tasks.loop", "plane.tasks.loop",
"plane.tasks.remember" "plane.tasks.remember"
] ]
}, },
{
"module": "plane.profile",
"classification": "load_bearing",
"reason": "python -m openultrasast.plane.profile",
"importers": [
"cli",
"plane.doctor",
"plane.generate",
"plane.reconciler"
]
},
{ {
"module": "plane.reconciler", "module": "plane.reconciler",
"classification": "load_bearing", "classification": "load_bearing",
@@ -1093,6 +1156,7 @@
"improve.evolve", "improve.evolve",
"improve.validator", "improve.validator",
"plane.tasks.loop", "plane.tasks.loop",
"push.quick",
"scoring.project_score", "scoring.project_score",
"semantic.overlay" "semantic.overlay"
] ]
@@ -1132,6 +1196,7 @@
"plane.tasks.alerts", "plane.tasks.alerts",
"plane.tasks.features", "plane.tasks.features",
"plane.tasks.repo_facts", "plane.tasks.repo_facts",
"push.quick",
"push.runner", "push.runner",
"rank", "rank",
"regress.candidate", "regress.candidate",
@@ -1186,6 +1251,7 @@
"importers": [ "importers": [
"cli", "cli",
"push.assistance", "push.assistance",
"push.background",
"push.eligibility", "push.eligibility",
"push.policy", "push.policy",
"push.report", "push.report",
@@ -1203,6 +1269,14 @@
"push.runner" "push.runner"
] ]
}, },
{
"module": "push.background",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"push.runner"
]
},
{ {
"module": "push.cache", "module": "push.cache",
"classification": "load_bearing", "classification": "load_bearing",
@@ -1234,6 +1308,23 @@
"push.runner" "push.runner"
] ]
}, },
{
"module": "push.explain",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"push.report",
"push.runner"
]
},
{
"module": "push.install",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"cli"
]
},
{ {
"module": "push.policy", "module": "push.policy",
"classification": "load_bearing", "classification": "load_bearing",
@@ -1245,12 +1336,21 @@
"push.vetoes" "push.vetoes"
] ]
}, },
{
"module": "push.quick",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [
"push.runner"
]
},
{ {
"module": "push.report", "module": "push.report",
"classification": "load_bearing", "classification": "load_bearing",
"reason": "pre-push-safety-net Req 5.1/5.2/5.5/7.3: compact output and complete artifacts; runner integration pending", "reason": "pre-push-safety-net Req 5.1/5.2/5.5/7.3: compact output and complete artifacts; runner integration pending",
"importers": [ "importers": [
"push.assistance", "push.assistance",
"push.background",
"push.runner" "push.runner"
] ]
}, },
@@ -1315,7 +1415,8 @@
"improve.evolve", "improve.evolve",
"map_gate", "map_gate",
"pairs", "pairs",
"plane.tasks.alerts" "plane.tasks.alerts",
"push.quick"
] ]
}, },
{ {
@@ -1501,6 +1602,7 @@
"hunter_tools", "hunter_tools",
"learn.features", "learn.features",
"learn.roles", "learn.roles",
"learn.slice",
"mapping", "mapping",
"model.candidates", "model.candidates",
"model.layout", "model.layout",
@@ -1557,6 +1659,7 @@
"hunter_tools", "hunter_tools",
"learn.features", "learn.features",
"learn.roles", "learn.roles",
"learn.slice",
"mapping", "mapping",
"model.candidates", "model.candidates",
"model.layout", "model.layout",
@@ -1590,6 +1693,7 @@
"importers": [ "importers": [
"cli", "cli",
"hunter_tools", "hunter_tools",
"learn.slice",
"model.candidates", "model.candidates",
"pairs", "pairs",
"semantic.cst", "semantic.cst",
@@ -1723,6 +1827,7 @@
"classification": "load_bearing", "classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate, cli", "reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [ "importers": [
"learn.slice",
"semantic.overlay" "semantic.overlay"
] ]
}, },
@@ -1748,6 +1853,8 @@
"reason": "reachable from gate, map_gate, pair_gate, cli", "reason": "reachable from gate, map_gate, pair_gate, cli",
"importers": [ "importers": [
"cli", "cli",
"learn.cli",
"learn.fill",
"model.endpoint", "model.endpoint",
"plane.budget", "plane.budget",
"plane.tasks.roles", "plane.tasks.roles",
@@ -0,0 +1,5 @@
[ok] profile k3s: exec=remote kube_context=ousast-k3s registry=ghcr.io/norandom images=/home/mc/Source/OpenUltraSAST/.claude/worktrees/agent-a7d4724e0c0c407a8/ops/k8s/profiles/images.json router_url=(tunnel through the kube context) ax_server=(tunnel through the kube context) memory=s3://sast-memory atespace=default snapshots_bucket=s3://ax-snapshots image_pull_secret=ghcr-pull
[FAIL] cluster ousast-k3s: context ousast-k3s unreachable (error: context "ousast-k3s" does not exist); docs/deployment.md lists the prerequisites
[FAIL] runner image in ghcr.io/norandom: images file /home/mc/Source/OpenUltraSAST/.claude/worktrees/agent-a7d4724e0c0c407a8/ops/k8s/profiles/images.json: No such file or directory; docs/deployment.md lists the prerequisites
[ok] memory store s3://sast-memory: s3://sast-memory opens (an S3 bucket is verified)
exit=1 seconds=1.1
@@ -0,0 +1,7 @@
[ok] profile kind: exec=local kube_context=kind-ousast registry=localhost:5001 images=/home/mc/Source/OpenUltraSAST/.claude/worktrees/agent-a7d4724e0c0c407a8/ops/k8s/profiles/kind-images.json router_url=(tunnel through the kube context) ax_server=(tunnel through the kube context) memory=s3://sast-memory atespace=default snapshots_bucket=s3://ax-snapshots image_pull_secret=(none)
[ok] cluster kind-ousast: reachable
[ok] agent substrate (ate-system): 9 pods ready
[ok] ax controller (ax-system): 4 pods ready
[ok] runner image in localhost:5001: ousast-runner@sha256:95edc7ed5621... present in localhost:5001
[ok] memory store s3://sast-memory: s3://sast-memory opens (an S3 bucket is verified)
exit=0 seconds=1.4
@@ -0,0 +1,19 @@
F.. [100%]
=================================== FAILURES ===================================
________ test_no_host_literal_outside_the_profile_and_the_kind_bring_up ________
def test_no_host_literal_outside_the_profile_and_the_kind_bring_up() -> None:
kind = load_profile("kind", environ={})
found = offenders(_files(), kind.registry)
> assert found == [], "host-only literals (Req 1.3):\n" + "\n".join(found)
E AssertionError: host-only literals (Req 1.3):
E src/openultrasast/plane/workspaces.py:37: [kind context] CONTEXT = "kind-ousast" # reinserted for the record 01 demonstration
E assert ['src/openult...emonstration'] == []
E
E Left contains one more item: 'src/openultrasast/plane/workspaces.py:37: [kind context] CONTEXT = "kind-ousast" # reinserted for the record 01 demonstration'
E Use -v to get more diff
tests/test_plane_literals.py:81: AssertionError
=========================== short test summary info ============================
FAILED tests/test_plane_literals.py::test_no_host_literal_outside_the_profile_and_the_kind_bring_up
1 failed, 2 passed in 0.26s
@@ -0,0 +1,2 @@
... [100%]
3 passed in 0.23s
@@ -0,0 +1,16 @@
apiVersion: ax.io/v1alpha1
kind: Workspace
metadata: {name: ousast-probe-big-ws, atespace: default}
spec:
files:
- {path: "hello.py", content: "def hello():\n return greet()\n"}
---
apiVersion: ax.io/v1alpha1
kind: Task
metadata: {name: ousast-probe-big, atespace: default}
spec:
image: "localhost:5001/ousast-runner@sha256:412db0a1504af8f1b23a395af8d1dcbd152c1bb6d16594f2b733067163a31884"
command: ["repo-facts"]
env: [{name: OUSAST_OUTPUT_DIR, value: "/workspace/.ousast-out/probe"}]
resources: {requests: {cpu: "250m", memory: "2560Mi"}, limits: {cpu: "1", memory: "3Gi"}}
workspaces: [{name: ousast-probe-big-ws, path: "/workspace"}]
@@ -0,0 +1,16 @@
apiVersion: ax.io/v1alpha1
kind: Workspace
metadata: {name: ousast-probe-small-ws, atespace: default}
spec:
files:
- {path: "hello.py", content: "def hello():\n return greet()\n"}
---
apiVersion: ax.io/v1alpha1
kind: Task
metadata: {name: ousast-probe-small, atespace: default}
spec:
image: "localhost:5001/ousast-runner@sha256:412db0a1504af8f1b23a395af8d1dcbd152c1bb6d16594f2b733067163a31884"
command: ["repo-facts"]
env: [{name: OUSAST_OUTPUT_DIR, value: "/workspace/.ousast-out/probe"}]
resources: {requests: {cpu: "250m", memory: "256Mi"}, limits: {cpu: "1", memory: "512Mi"}}
workspaces: [{name: ousast-probe-small-ws, path: "/workspace"}]
@@ -0,0 +1,26 @@
# Warm gVisor workers for ax tasks (Agent Substrate WorkerPool), rendered by ops/ax/up.sh (and by
# ops/ax/probe-placement.sh for the placement probe) from the profile's pool: POOL_NAME, POOL_REPLICAS, POOL_CPU,
# POOL_MEMORY (ops/k8s/profiles/<profile>.toml, [pools.default]) and WORKER_IMAGE (ko:// for a build, or a
# digest-pinned reference of a built worker image). ax's per-task ActorTemplates set no workerSelector, so every
# pool of the matching sandbox class is eligible. An actor occupies a whole worker. Replaced by the renderer of
# plane-on-kubernetes task 6.1 (ops/k8s/base/workerpool-*.yaml.tmpl).
apiVersion: ate.dev/v1alpha1
kind: WorkerPool
metadata:
name: probe-pool
namespace: ax-system
labels:
workload: ousast
spec:
replicas: 1
workerImage: localhost:5001/ateom-gvisor-715889664656de67e44382a8d6ab981d@sha256:220f1cf13ea485b73a2e4270fa544901a18ae826d8f22bd54d42761944a41c5c
template:
nodeSelector:
ate.dev/substrate-version: "4f9c293"
resources:
limits:
cpu: "2"
memory: 3Gi
requests:
cpu: 250m
memory: 3Gi
@@ -0,0 +1,78 @@
{
"workers": [
{
"ip": "10.244.0.22",
"labels": {
"workload": "ousast"
},
"metadata": {
"createTime": "2026-09-29T16:32:08.594663703Z",
"name": "769f7d8b-142a-4ed5-b896-08d6a04bedb6",
"uid": "152cf7c0-3f4d-4634-88c5-9fe501bb403e",
"updateTime": "2026-10-02T05:24:59.069304031Z",
"version": "1758"
},
"nodeName": "ousast-control-plane",
"sandboxClass": "gvisor",
"status": {
"capacity": {
"actors": 1000,
"resources": {
"limits": [
{
"name": "cpu",
"quantity": "1"
},
{
"name": "memory",
"quantity": "1536Mi"
}
]
}
},
"state": "WORKER_STATE_ACTIVE"
},
"workerNamespace": "ax-system",
"workerPod": "ousast-pool-6656b55f5b-9wbrw",
"workerPodUid": "769f7d8b-142a-4ed5-b896-08d6a04bedb6",
"workerPool": "ousast-pool"
},
{
"ip": "10.244.0.28",
"labels": {
"workload": "ousast"
},
"metadata": {
"createTime": "2026-10-02T18:29:30.486026311Z",
"name": "1407a421-7dcf-4ad7-b999-70a4624bf890",
"uid": "2d496d7a-4f32-459d-9b92-256a2c5b4800",
"updateTime": "2026-10-02T18:29:30.551668391Z",
"version": "2"
},
"nodeName": "ousast-control-plane",
"sandboxClass": "gvisor",
"status": {
"capacity": {
"actors": 1000,
"resources": {
"limits": [
{
"name": "cpu",
"quantity": "2"
},
{
"name": "memory",
"quantity": "3Gi"
}
]
}
},
"state": "WORKER_STATE_ACTIVE"
},
"workerNamespace": "ax-system",
"workerPod": "probe-pool-5dffd74bdc-dqlww",
"workerPodUid": "1407a421-7dcf-4ad7-b999-70a4624bf890",
"workerPool": "probe-pool"
}
]
}
@@ -0,0 +1,81 @@
{
"workers": [
{
"ip": "10.244.0.22",
"labels": {
"workload": "ousast"
},
"metadata": {
"createTime": "2026-09-29T16:32:08.594663703Z",
"name": "769f7d8b-142a-4ed5-b896-08d6a04bedb6",
"uid": "152cf7c0-3f4d-4634-88c5-9fe501bb403e",
"updateTime": "2026-10-02T18:29:45.281958347Z",
"version": "1761"
},
"nodeName": "ousast-control-plane",
"sandboxClass": "gvisor",
"status": {
"allocated": {
"actors": 1
},
"capacity": {
"actors": 1000,
"resources": {
"limits": [
{
"name": "cpu",
"quantity": "1"
},
{
"name": "memory",
"quantity": "1536Mi"
}
]
}
},
"state": "WORKER_STATE_ACTIVE"
},
"workerNamespace": "ax-system",
"workerPod": "ousast-pool-6656b55f5b-9wbrw",
"workerPodUid": "769f7d8b-142a-4ed5-b896-08d6a04bedb6",
"workerPool": "ousast-pool"
},
{
"ip": "10.244.0.28",
"labels": {
"workload": "ousast"
},
"metadata": {
"createTime": "2026-10-02T18:29:30.486026311Z",
"name": "1407a421-7dcf-4ad7-b999-70a4624bf890",
"uid": "2d496d7a-4f32-459d-9b92-256a2c5b4800",
"updateTime": "2026-10-02T18:29:30.551668391Z",
"version": "2"
},
"nodeName": "ousast-control-plane",
"sandboxClass": "gvisor",
"status": {
"capacity": {
"actors": 1000,
"resources": {
"limits": [
{
"name": "cpu",
"quantity": "2"
},
{
"name": "memory",
"quantity": "3Gi"
}
]
}
},
"state": "WORKER_STATE_ACTIVE"
},
"workerNamespace": "ax-system",
"workerPod": "probe-pool-5dffd74bdc-dqlww",
"workerPodUid": "1407a421-7dcf-4ad7-b999-70a4624bf890",
"workerPool": "probe-pool"
}
]
}
@@ -0,0 +1,81 @@
{
"workers": [
{
"ip": "10.244.0.22",
"labels": {
"workload": "ousast"
},
"metadata": {
"createTime": "2026-09-29T16:32:08.594663703Z",
"name": "769f7d8b-142a-4ed5-b896-08d6a04bedb6",
"uid": "152cf7c0-3f4d-4634-88c5-9fe501bb403e",
"updateTime": "2026-10-02T18:29:53.001457323Z",
"version": "1763"
},
"nodeName": "ousast-control-plane",
"sandboxClass": "gvisor",
"status": {
"allocated": {
"actors": 1
},
"capacity": {
"actors": 1000,
"resources": {
"limits": [
{
"name": "cpu",
"quantity": "1"
},
{
"name": "memory",
"quantity": "1536Mi"
}
]
}
},
"state": "WORKER_STATE_ACTIVE"
},
"workerNamespace": "ax-system",
"workerPod": "ousast-pool-6656b55f5b-9wbrw",
"workerPodUid": "769f7d8b-142a-4ed5-b896-08d6a04bedb6",
"workerPool": "ousast-pool"
},
{
"ip": "10.244.0.28",
"labels": {
"workload": "ousast"
},
"metadata": {
"createTime": "2026-10-02T18:29:30.486026311Z",
"name": "1407a421-7dcf-4ad7-b999-70a4624bf890",
"uid": "2d496d7a-4f32-459d-9b92-256a2c5b4800",
"updateTime": "2026-10-02T18:30:04.825058843Z",
"version": "4"
},
"nodeName": "ousast-control-plane",
"sandboxClass": "gvisor",
"status": {
"capacity": {
"actors": 1000,
"resources": {
"limits": [
{
"name": "cpu",
"quantity": "2"
},
{
"name": "memory",
"quantity": "3Gi"
}
]
}
},
"state": "WORKER_STATE_ACTIVE"
},
"workerNamespace": "ax-system",
"workerPod": "probe-pool-5dffd74bdc-dqlww",
"workerPodUid": "1407a421-7dcf-4ad7-b999-70a4624bf890",
"workerPool": "probe-pool"
}
]
}
@@ -0,0 +1,132 @@
{
"record": "k8s-01-profile",
"spec": "plane-on-kubernetes",
"tasks": ["1.1", "1.2", "1.3", "1.4", "1.5", "1.6", "1.7", "1.8"],
"date": "2026-10-02",
"host": "the maintainer's laptop: 7.9 GiB, single-node kind cluster, RustFS store reachable over https",
"commits": {
"1.1": "8eac1b2",
"1.2": "77a03ba",
"1.3": "36f5927",
"1.4": "8099b29",
"1.5": "86ec237",
"1.6": "2afa20e",
"1.7": "2cca03c",
"1.8": "the commit that adds this line (git log: plane(1.8))"
},
"images": {
"runner_before": "localhost:5001/ousast-runner@sha256:412db0a1504af8f1b23a395af8d1dcbd152c1bb6d16594f2b733067163a31884",
"runner_after_up_sh": "localhost:5001/ousast-runner@sha256:95edc7ed562189180a472df8115e6a90c8b4fcea47c06593395634bd56041bcb",
"pin_file": "ops/k8s/profiles/kind-images.json (written by ops/ax/up.sh; the templates re-pinned to it, 128 image lines)"
},
"instrument_checks": {
"code_under_test": "PYTHONPATH=<worktree>/src first; `python -c 'import openultrasast; print(openultrasast.__file__)'` printed the worktree path before every gate",
"store_reached": {
"how": "scratch store_probe.py: open_store('s3://sast-memory') through the worktree code (verify_bucket runs), then read the probe object",
"open_and_verify_seconds": 0.62,
"probe_key": "_probe/select.jsonl",
"probe_bytes": 24,
"probe_version_id_present": true
},
"smoke_delivery_sizes_bytes": {"facts.json": 451, "summary.json": 238, "units.jsonl": 291, "task.yaml": 1148},
"registry_resolution": "doctor HEADs /v2/ousast-runner/manifests/<digest> on the kind registry: HTTP 2xx for the new digest"
},
"doctor": {
"kind": {
"command": "ousast plane doctor --profile kind",
"exit": 0,
"seconds": 1.4,
"lines": [
"[ok] profile kind: exec=local kube_context=... registry=... images=... router_url=(tunnel) ax_server=(tunnel) memory=s3://sast-memory atespace=default snapshots_bucket=s3://ax-snapshots image_pull_secret=(none)",
"[ok] cluster <kind context>: reachable",
"[ok] agent substrate (ate-system): 9 pods ready",
"[ok] ax controller (ax-system): 4 pods ready",
"[ok] runner image in <kind registry>: ousast-runner@sha256:95edc7ed5621... present",
"[ok] memory store s3://sast-memory: opens (an S3 bucket is verified)"
],
"file": "doctor-kind.txt"
},
"k3s": {
"command": "ousast plane doctor --profile k3s",
"exit": 1,
"seconds": 1.1,
"failing_lines": [
"cluster ousast-k3s: context does not exist (expected: the cluster does not exist yet)",
"runner image in ghcr.io/norandom: images file ops/k8s/profiles/images.json missing (expected: the release asset is published by task 4.1)"
],
"passing_lines": ["profile k3s: every address reported", "memory store s3://sast-memory: opens"],
"note": "two expected failures, not one: the images file is the release asset and does not exist before group 4; the namespaces are not checked behind an unreachable context, so no third line fails for the same cause",
"file": "doctor-k3s.txt"
}
},
"literal_scan": {
"test": "tests/test_plane_literals.py",
"scope": "src/, plane/, ops/ (minus ops/ax/up.sh, ops/k8s/profiles/), docs/",
"patterns": ["kind context", "kind registry", "kind gateway 172.19.", "loopback with port", "port-forward"],
"exemptions": [
"router.py for loopback-with-port and port-forward",
"lines tagged ax-tunnel or loopback (two: the runner's own health URL, the Joern session's query URL)",
"digest-pinned image: lines under plane/tasks/ whose registry equals the kind profile's (a third test asserts every one does)"
],
"offenders_at_1_3": 0,
"offenders_before_1_3": 23,
"reinsertion_demo": {
"inserted": "a kind-context literal appended to src/openultrasast/plane/workspaces.py",
"pytest_exit_with_literal": 1,
"named": "src/openultrasast/plane/workspaces.py:37: [kind context]",
"restored": "byte for byte (cmp)",
"pytest_exit_restored": 0,
"files": ["literal-reinserted-fail.txt", "literal-restored-pass.txt"]
}
},
"up_sh_rerun": {
"command": "OUSAST_PYTHON=<main venv python> ops/ax/up.sh",
"exit": 0,
"seconds": 38,
"kept": ["cluster", "substrate", "egress gateway", "ax control plane", "worker pool ousast-pool (2/2 ready)", "ax CLI (newer than the checkout)"],
"done": ["receiver Service re-applied", "runner image rebuilt and pushed (new digest above)", "kind-images.json written", "smoke Task applied and resumed"],
"first_attempt_failure": "`profile --print images` answered `no image named ''`; fixed in profile._lookup (an images path with no sub-key is the file path) and covered by test_plane_profile",
"disk_gb_free_before_after": [1.7, 1.5],
"file": "up-sh-rerun.txt"
},
"smoke_run_1_4": {
"command": "ops/ax/smoke-run.sh (receiver delivery: the path group 2 replaces)",
"exit": 0,
"seconds": 24,
"status": "done",
"delivered": "facts.json 451 B, summary.json 238 B, units.jsonl 291 B; summary.image = the new digest, units 2/2",
"file": "smoke-run-1.4.txt"
},
"placement_probe_1_5": {
"script": "ops/ax/probe-placement.sh",
"pools": {"ousast-pool": "scaled to 1 worker, 1 CPU / 1536Mi", "probe-pool": "1 worker, 2 CPU / 3Gi, the already-built ateom-gvisor image (no ko build)"},
"big_actor": {"requests_memory": "2560Mi", "limits_memory": "3Gi", "landed_on": "ousast-pool (1536Mi worker): status.allocated.actors=1; probe-pool 0"},
"small_actor": {"requests_memory": "256Mi", "limits_memory": "512Mi", "landed_on": "ousast-pool: allocated.actors=1; probe-pool 0"},
"placement": "any",
"consequence": "design.md section 4: group 5 takes the fallback (an engine atespace with engine-pool as its only pool if Substrate scopes pools by atespace; else a plain Kubernetes Job on k3s and alerts-engine on kind)",
"resume_latency_seconds": "first resume of each new template succeeded at once (golden snapshot built within the resume deadline); whole probe 46 s",
"cleanup": "probe tasks deleted, probe-pool deleted, ousast-pool back to 2/2 (verified by kubectl get workerpool)",
"files": ["probe/probe.log", "probe/workers-before.json", "probe/workers-ousast-probe-big.json", "probe/workers-ousast-probe-small.json", "probe/ousast-probe-big.yaml", "probe/ousast-probe-small.yaml", "probe/probe-pool.yaml"]
},
"reconciler_bound_1_6": {
"lines_at_spec_commit_6bd44ca": 477,
"lines_after_1_2": 489,
"added_by_1_2": "the profile parameter of run() and its docstring (+12)",
"estimate_2_4": "Delivery wiring: open the store, build Delivery, pass the delivery body to the starter, delivered() and collect() in the await/execute path, state mirroring, the OUSAST_DELIVERY=receiver switch: about +25 while the receiver is kept (group 2 keeps both paths), about -20 when 2.6 removes the receiver wiring",
"projected_during_group_2": 514,
"projected_after_2_6": 494,
"decision": "the estimate crosses 500 while both paths coexist: task 2.4 moves the bound in tests/test_plane_reconciler.py to 550 with the reason in the assertion message, and 2.6 records the final count"
},
"gvisor_tarball_1_7": {
"status": "planned, manual",
"where": "docs/deployment.md B.1",
"url": "gs://gvisor/releases/nightly/2026-09-02/<arch>/gvisor.tar.zstd (sha256-pinned in substrate/manifests/ate-install/sandboxconfig-gvisor.yaml)",
"check": "the atelet's fetch line in `kubectl -n ate-system logs` of a worker pod on the k3s node; labelled planned until read there"
},
"gates": {
"1.1": "pytest 2002 passed + module-audit manifest regenerated (test_module_audit 6 passed), ruff 0, format 0, mypy 0",
"1.2": "pytest 2006 passed, 42 skipped, exit 0; ruff 0; format 0; mypy 0",
"1.3": "pytest 2009 passed, 42 skipped, exit 0; ruff 0; format 0; mypy 0; mkdocs --strict 0",
"1.4": "see the commit message"
}
}
@@ -0,0 +1,9 @@
run finished: done
memory s3://sast-memory: ax-e2e-smoke: 0 tasks ingested, 0 already in the index; rows none
run ax-e2e-smoke: done (/tmp/claude-1000/-home-mc-Source-OpenUltraSAST/c09e30b2-39a4-4564-bc31-6ea71ffaa198/scratchpad/k8s-impl/results/plane/ax-e2e-smoke)
TASK MODEL CALLS PROMPT CACHE-HIT OUTPUT USD
facts - 0 n/a n/a n/a n/a [done]
----------------------------------------------------------------------------------------------------
total - n/a n/a n/a n/a n/a
smoke exit=0 seconds=24
@@ -0,0 +1,58 @@
== kind cluster 'ousast' (context kind-ousast) + registry localhost:5001 (20:35:35)
cluster exists; kept
== agent substrate (ate-system) (20:35:35)
substrate ready; kept
NAME READY STATUS RESTARTS AGE
ate-api-server-647bb6f8b6-c9k5k 1/1 Running 2 (12h ago) 3d
ate-api-server-647bb6f8b6-dznjn 1/1 Running 1 (12h ago) 3d
ate-controller-c95586684-g7nvn 1/1 Running 0 3d2h
atelet-4f9c293-9zpfg 1/1 Running 0 3d2h
atenet-egress-d74b8f6bf-8zmxj 1/1 Running 0 3d
atenet-router-54887bc8f9-rqs6g 2/2 Running 2 (12h ago) 3d2h
postgres-0 2/2 Running 3 (3d2h ago) 3d2h
rustfs-6c4c677cb6-cqvhc 1/1 Running 3 (3d2h ago) 3d2h
rustfs-bucket-init-nvfr6 0/1 Completed 0 3d2h
== egress gateway (atenet-egress, agentgateway variant: one prebuilt image, nothing built) (20:35:35)
gateway present; kept
== ax control plane (ax-system) (20:35:35)
ax ready; kept
NAME READY STATUS RESTARTS AGE
ax-redis-86df786c87-pz5tp 1/1 Running 0 3d2h
ax-server-b479bdc76-hwkqb 1/1 Running 0 3d2h
ousast-pool-6656b55f5b-5brkn 1/1 Running 0 5m23s
ousast-pool-6656b55f5b-9wbrw 1/1 Running 0 3d2h
== artifact receiver Service (ousast-receiver.ax-system -> this host) (20:35:35)
service/ousast-receiver unchanged
endpointslice.discovery.k8s.io/ousast-receiver-host configured
== worker pool (gVisor): pools.default of the profile (20:35:36)
pool ousast-pool exists (2/2 ready); kept
pod/ousast-pool-6656b55f5b-5brkn condition met
pod/ousast-pool-6656b55f5b-9wbrw condition met
== runner image localhost:5001/ousast-runner (20:35:37)
sha256:f004a5ef8f71b1cb8a5880c509389c0569181af7b19170b574958cc3ef75cb1c
localhost:5001/ousast-runner:dev
runner image localhost:5001/ousast-runner@sha256:95edc7ed562189180a472df8115e6a90c8b4fcea47c06593395634bd56041bcb -> /home/mc/Source/OpenUltraSAST/.claude/worktrees/agent-a7d4724e0c0c407a8/ops/k8s/profiles/kind-images.json
re-pin the templates with: ousast plane workspaces ... --runner-image /home/mc/Source/OpenUltraSAST/.claude/worktrees/agent-a7d4724e0c0c407a8/ops/k8s/profiles/kind-images.json (or ousast plane repin, task 4.3)
== ax CLI from the deployed checkout (a release CLI skews from the server) (20:36:05)
ax CLI newer than the checkout; kept
== smoke task (20:36:05)
task.ax.io/ousast-smoke created
workspace.ax.io/ousast-smoke-ws unchanged
task.ax.io/ousast-smoke resumed
NAME ATESPACE PHASE ACTOR WORKER-IP AGE
ousast-smoke default Running ousast-smoke 10.244.0.29 5s
== footprint (20:36:11)
total used free shared buff/cache available
Mem: 7877 4797 225 63 3464 3079
kind-registry 53.77MiB / 7.693GiB 0.00%
up.sh exit 0
@@ -0,0 +1,24 @@
{
"metadata": {
"atespace": "default",
"createTime": "2026-10-02T19:39:19.428045847Z",
"name": "default",
"uid": "741e52b7-aaf1-4a91-b4bf-fc3b20ba4a36",
"updateTime": "2026-10-02T19:39:19.428045847Z",
"version": "1"
},
"rules": [
{
"tlsPassthrough": {
"hostnames": [
"files.because-security.com"
],
"ports": {
"numbers": [
443
]
}
}
}
]
}
@@ -0,0 +1,83 @@
{
"record": "k8s-02-store-delivery",
"spec": "plane-on-kubernetes",
"tasks": ["2.1", "2.2", "2.3", "2.4", "2.5", "2.6"],
"date": "2026-10-02",
"commits": {
"2.1": "35d8032",
"2.2": "9506a18",
"2.3": "c49eae4",
"2.4": "605b7af",
"2.5": "the commit that adds this file (git log: plane(2.5))",
"receiver_removed": "pending: task 2.6 (paused 2026-10-03 by the maintainer's reprioritisation; the receiver path still exists behind OUSAST_DELIVERY=receiver)"
},
"images": {
"runner": "localhost:5001/ousast-runner@sha256:d2a55293999cc34581ef1f02a1ffb0c2dd8fc0cbe001ac952df81600e868f019",
"built_by": "ops/ax/up.sh rerun (file up-sh-rebuild.txt): every step kept but the runner build; the image carries the 2.2 runner",
"disk_gb_free_after_build": 1.3
},
"instrument_checks": {
"code_under_test": "the worktree's src first on PYTHONPATH; `ousast` run as `python -m openultrasast.cli` from the worktree (ops/ax/smoke-run.sh with OUSAST_PYTHON)",
"store_read_back": {
"how": "scratch store_tar_check.py opens the store through the worktree code (verify_bucket runs), lists runs/ax-e2e-smoke/, downloads output.tar, compares its members with the files the reconciler extracted (file store-check.json)",
"tar_bytes": 10240,
"tar_members_bytes": {"facts.json": 451, "summary.json": 238, "units.jsonl": 291},
"extracted_files_bytes": {"facts.json": 451, "summary.json": 238, "units.jsonl": 291},
"members_equal_extracted": true,
"member_bytes_total": 980,
"note": "a tar is 512-byte blocks: 10240 B of tar carry 980 B of files; done.json records the tar's byte size"
},
"no_receiver_in_the_task": "the rendered task.yaml is 996 B against 1148 B on the receiver path: no OUSAST_ARTIFACT_URL/DIAL in the Task's env; the start body carries the URLs"
},
"smoke_runs": {
"store_1": {
"command": "ops/ax/smoke-run.sh (store delivery, the default after 2.4; OUSAST_DELIVERY unset)",
"exit": 0,
"seconds_submit_to_done": 26,
"tar_put_at": "2026-10-02T19:37:24+00:00 (the object's LastModified)",
"collected_at": "2026-10-02T19:37:26+00:00 (done.json finished)",
"put_to_collect_seconds": 2,
"poll_interval_seconds": 3,
"file": "smoke-run-store-1.txt"
},
"seeded_rerun": {
"command": "the same, with the local results root deleted and the store's runs/ax-e2e-smoke/ intact",
"exit": 0,
"seconds": 2,
"what_happened": "state.json seeded from the store: every task done, nothing submitted to ax (plane-on-kubernetes 2.1, design section 2: a rerun from another machine resumes from the store)",
"file": "smoke-run-seeded-rerun.txt"
},
"store_2": {
"command": "the same, after deleting runs/ax-e2e-smoke/ from the store (5 objects) so the task executes again, with the egress policy polled",
"exit": 0,
"seconds_submit_to_done": 24,
"tar_put_at": "2026-10-02T19:39:38+00:00",
"collected_at": "2026-10-02T19:39:40+00:00",
"put_to_collect_seconds": 2,
"file": "smoke-run-store-2.txt"
}
},
"objects_delivered_to_the_store": {
"runs/ax-e2e-smoke/facts/output.tar": 10240,
"runs/ax-e2e-smoke/facts/facts.json": 451,
"runs/ax-e2e-smoke/facts/summary.json": 238,
"runs/ax-e2e-smoke/facts/done.json": 149,
"runs/ax-e2e-smoke/state.json": 350,
"note": "units.jsonl is extracted locally but not a declared output of the smoke Run, so it is not re-put"
},
"egress_policy_read_back": {
"how": "kubectl ate get egress-policy ax-e2e-smoke-facts -a default -o json, polled while the actor lived (file egress-policy-read-back.json)",
"rules": [{"tlsPassthrough": {"hostnames": ["files.because-security.com"], "ports": {"numbers": [443]}}}],
"reading": "one TLS rule naming the store host, no http rule; the model-free smoke task names no Git host and no Model host",
"also": "Egress.apply reads every policy back and raises on a mismatch, so each completed Run is itself the equality check"
},
"status_table": "ousast plane status ax-e2e-smoke printed the attribution table (facts: done, calls 0) in every run; the table is in the smoke-run-*.txt files",
"reconciler_lines": {"after_2_4": 549, "bound": 550, "note": "both paths coexist until 2.6; the receiver wiring goes then"},
"gates": {
"2.1": "pytest 2020 passed, 42 skipped, exit 0; ruff 0; format 0; mypy 0",
"2.2": "pytest 2024 passed, 42 skipped, exit 0; ruff 0; format 0; mypy 0",
"2.3": "pytest 2027 passed, 42 skipped, exit 0; ruff 0; format 0; mypy 0",
"2.4": "pytest 2031 passed, 42 skipped, exit 0; ruff 0; format 0; mypy 0",
"2.5": "pytest 2031 passed, 42 skipped, exit 0; ruff 0; format 0; mypy 0 (first attempt 2026-10-02: 11 failed + 5 errors, all the FileStore 1 GiB free-space refusal at 0.97 GiB free; rerun after the disk was freed)"
}
}
@@ -0,0 +1,9 @@
run finished: done
memory s3://sast-memory: ax-e2e-smoke: 0 tasks ingested, 0 already in the index; rows none
run ax-e2e-smoke: done (/tmp/claude-1000/-home-mc-Source-OpenUltraSAST/c09e30b2-39a4-4564-bc31-6ea71ffaa198/scratchpad/k8s-impl/results/plane/ax-e2e-smoke)
TASK MODEL CALLS PROMPT CACHE-HIT OUTPUT USD
facts - n/a n/a n/a n/a n/a [done]
----------------------------------------------------------------------------------------------------
total - n/a n/a n/a n/a n/a
smoke exit=0 seconds=2
@@ -0,0 +1,9 @@
run finished: done
memory s3://sast-memory: ax-e2e-smoke: 0 tasks ingested, 0 already in the index; rows none
run ax-e2e-smoke: done (/tmp/claude-1000/-home-mc-Source-OpenUltraSAST/c09e30b2-39a4-4564-bc31-6ea71ffaa198/scratchpad/k8s-impl/results/plane/ax-e2e-smoke)
TASK MODEL CALLS PROMPT CACHE-HIT OUTPUT USD
facts - 0 n/a n/a n/a n/a [done]
----------------------------------------------------------------------------------------------------
total - n/a n/a n/a n/a n/a
smoke exit=0 seconds=26
@@ -0,0 +1,9 @@
run finished: done
memory s3://sast-memory: ax-e2e-smoke: 0 tasks ingested, 0 already in the index; rows none
run ax-e2e-smoke: done (/tmp/claude-1000/-home-mc-Source-OpenUltraSAST/c09e30b2-39a4-4564-bc31-6ea71ffaa198/scratchpad/k8s-impl/results/plane/ax-e2e-smoke)
TASK MODEL CALLS PROMPT CACHE-HIT OUTPUT USD
facts - 0 n/a n/a n/a n/a [done]
----------------------------------------------------------------------------------------------------
total - n/a n/a n/a n/a n/a
smoke exit=0 seconds=24
@@ -0,0 +1,36 @@
{
"store": "s3://sast-memory",
"objects_under_run": {
"runs/ax-e2e-smoke/facts/done.json": 149,
"runs/ax-e2e-smoke/facts/facts.json": 451,
"runs/ax-e2e-smoke/facts/output.tar": 10240,
"runs/ax-e2e-smoke/facts/summary.json": 238,
"runs/ax-e2e-smoke/state.json": 350
},
"tar_bytes": 10240,
"tar_members": {
"facts.json": 451,
"summary.json": 238,
"units.jsonl": 291
},
"extracted_files": {
"facts.json": 451,
"units.jsonl": 291,
"summary.json": 238
},
"members_equal_extracted": true,
"member_bytes_total": 980,
"extracted_bytes_total": 980,
"done_json": {
"bytes": 10240,
"files": 3,
"finished": "2026-10-02T19:39:40+00:00",
"published": [
"facts.json",
"summary.json"
],
"status": "done",
"task": "facts"
},
"tar_put_at_last_modified": "2026-10-02T19:39:38+00:00"
}
@@ -0,0 +1,58 @@
== kind cluster 'ousast' (context kind-ousast) + registry localhost:5001 (21:36:18)
cluster exists; kept
== agent substrate (ate-system) (21:36:18)
substrate ready; kept
NAME READY STATUS RESTARTS AGE
ate-api-server-647bb6f8b6-c9k5k 1/1 Running 2 (13h ago) 3d1h
ate-api-server-647bb6f8b6-dznjn 1/1 Running 1 (13h ago) 3d1h
ate-controller-c95586684-g7nvn 1/1 Running 0 3d3h
atelet-4f9c293-9zpfg 1/1 Running 0 3d3h
atenet-egress-d74b8f6bf-8zmxj 1/1 Running 0 3d1h
atenet-router-54887bc8f9-rqs6g 2/2 Running 2 (13h ago) 3d3h
postgres-0 2/2 Running 3 (3d3h ago) 3d3h
rustfs-6c4c677cb6-cqvhc 1/1 Running 3 (3d3h ago) 3d3h
rustfs-bucket-init-nvfr6 0/1 Completed 0 3d3h
== egress gateway (atenet-egress, agentgateway variant: one prebuilt image, nothing built) (21:36:18)
gateway present; kept
== ax control plane (ax-system) (21:36:19)
ax ready; kept
NAME READY STATUS RESTARTS AGE
ax-redis-86df786c87-pz5tp 1/1 Running 0 3d3h
ax-server-b479bdc76-hwkqb 1/1 Running 0 3d3h
ousast-pool-6656b55f5b-5brkn 1/1 Running 0 66m
ousast-pool-6656b55f5b-9wbrw 1/1 Running 0 3d3h
== artifact receiver Service (ousast-receiver.ax-system -> this host) (21:36:19)
service/ousast-receiver unchanged
endpointslice.discovery.k8s.io/ousast-receiver-host configured
== worker pool (gVisor): pools.default of the profile (21:36:20)
pool ousast-pool exists (2/2 ready); kept
pod/ousast-pool-6656b55f5b-5brkn condition met
pod/ousast-pool-6656b55f5b-9wbrw condition met
== runner image localhost:5001/ousast-runner (21:36:21)
sha256:0501baf180b29f7aa7bbb8888bebabfc5d62429b7bdaa7f5a62b5b188f8ad5f6
localhost:5001/ousast-runner:dev
runner image localhost:5001/ousast-runner@sha256:d2a55293999cc34581ef1f02a1ffb0c2dd8fc0cbe001ac952df81600e868f019 -> /home/mc/Source/OpenUltraSAST/.claude/worktrees/agent-a7d4724e0c0c407a8/ops/k8s/profiles/kind-images.json
re-pin the templates with: ousast plane workspaces ... --runner-image /home/mc/Source/OpenUltraSAST/.claude/worktrees/agent-a7d4724e0c0c407a8/ops/k8s/profiles/kind-images.json (or ousast plane repin, task 4.3)
== ax CLI from the deployed checkout (a release CLI skews from the server) (21:36:32)
ax CLI newer than the checkout; kept
== smoke task (21:36:32)
task.ax.io/ousast-smoke created
workspace.ax.io/ousast-smoke-ws unchanged
task.ax.io/ousast-smoke resumed
NAME ATESPACE PHASE ACTOR WORKER-IP AGE
ousast-smoke default Running ousast-smoke 10.244.0.29 4s
== footprint (21:36:38)
total used free shared buff/cache available
Mem: 7877 5202 382 63 2903 2674
kind-registry 32.45MiB / 7.693GiB 0.00%
up.sh exit 0
@@ -0,0 +1,792 @@
{
"id": "2026-10-02-pre-push-operational-1",
"task": "pre-push-safety-net 17.7",
"requirements": [
"9.1",
"9.2",
"9.3",
"9.4",
"9.5",
"9.6",
"6.3"
],
"question": "Does the hook installed on any repository give honest, advisory-only results, and never block a push?",
"subjects": "The audit's three shallow clones (kennethreitz/records, expressjs/serve-static, unrolled/secure), each with the same planted commit c2 on c1, copied fresh from the audit's scratch clones.",
"instrument_proof": "Every replay and hook push records snapshots[].bytes_read and quick_tier files/bytes_read. The head snapshot's bytes equal `git ls-files | xargs cat | wc -c` at head (records 43,577; serve-static 61,525; secure 103,023), and the quick tier reads exactly the changed source files.",
"host": "Joern is not on the host, so host runs cannot run the engine; that is the case a first-time pip user hits, and the output must say so. The Docker engine run uses the 2026-09-14 openultrasast:dev image with the worktree src mounted read-only, as the audit did; the image was not rebuilt (1.1 GB disk free, below the 4 GB threshold).",
"before": {
"source": "/tmp/claude-1000/-home-mc-Source-OpenUltraSAST/c09e30b2-39a4-4564-bc31-6ea71ffaa198/scratchpad/ready/gap-report.md (audit at main 49e11e0)",
"replay_host": {
"records": {
"exit": 0,
"wall_seconds": 1.06,
"files_bytes_read": [
21,
43577
],
"findings_shown": 0,
"terminal": "generic: Some checks could not be completed. Review the comparison details...",
"skip_reasons_named_in_terminal": 0
},
"serve-static": {
"exit": 0,
"wall_seconds": 1.1,
"files_bytes_read": [
26,
61525
],
"findings_shown": 0,
"terminal": "same generic line",
"skip_reasons_named_in_terminal": 0
},
"secure": {
"exit": 0,
"wall_seconds": 1.04,
"files_bytes_read": [
19,
103023
],
"findings_shown": 0,
"terminal": "same generic line; Go not named",
"skip_reasons_named_in_terminal": 0
}
},
"docker_engine_serve_static": {
"deadline_30": {
"exit": 0,
"wall_seconds": 31.3,
"terminal": "Analysis did not finish. Rerun with a longer analysis deadline."
},
"deadline_300": {
"exit": 0,
"wall_seconds": 261.7,
"raw_engine_findings": 2,
"admitted": 0,
"dispositions": "injection and output_encoding at debug.js:2: comparison_unknown/base_incomplete + capability_unavailable",
"terminal": "No actionable defects reported; Details: /out/... (container path)"
}
},
"quick_scan": {
"serve-static": {
"exit": 0,
"wall_seconds": 0.46,
"findings": "js-eval, js-xss-response at debug.js:2"
},
"records": {
"exit": 0,
"wall_seconds": 0.49,
"findings": "2 pre-existing (setup.py os-command); planted %-formatted SQL missed"
},
"secure": {
"exit": 0,
"wall_seconds": 0.48,
"findings": 0,
"terminal": "No quick-mode findings were emitted (silent zero for Go)"
}
},
"hook": {
"ousast_not_on_path": {
"push_exit": "rejected",
"terminal": ".git/hooks/pre-push: 13: exec: ousast: not found"
},
"first_push_new_branch": {
"push_exit": 0,
"wall_seconds": 0.58,
"terminal": "No analysis result is available. (does not say new branch)"
},
"update": {
"push_exit": 0,
"wall_seconds": 1.36,
"terminal": "No actionable defects reported; analysis is incomplete."
},
"bad_OUSAST_PUSH_DEADLINE_or_cache_inside_repo": {
"exit": 2,
"effect": "would block the push"
},
"install": "ops/install-pre-push only (checkout); not in the wheel"
}
},
"verdict": {
"never_blocks": "PASS in this sample: 10 hook pushes across the three repositories all exited 0, including `ousast` absent from PATH (rejected before) and OUSAST_PUSH_DEADLINE=abc (exit 2 before). Blocking stays an explicit opt-in (unit-tested, not exercised here).",
"never_hides": "PASS in this sample: every incomplete run printed one plain `Skipped:` line per reason (2-5 per run), with no hex path or sha256 token. Go is named as not covered (`language not covered: go (1 changed file(s), 10 in the repository)`, and `notice: language not covered: go (10 files)` in `scan --mode quick`). The first push of a new branch says there is no base and what it checked instead.",
"quick_tier_within_deadline": "On the host, every replay and hook push finished in 0.69-1.21 s against the 30 s deadline. The quick tier read exactly the changed source file (records 1 file / 17,449 B; serve-static 1 / 87 B; secure 1 / 193 B).",
"planted_js_eval": "SURFACES. Quick tier: js-eval (CWE-95) and js-xss-response (CWE-79) at debug.js:2 on the host, 0.9 s, in replay and in the hook push. Engine (Docker, deadline 300): injection and output_encoding at debug.js:2, now novelty `new` / `file_added_in_head` (before: `unknown` / `base_incomplete`). The only veto is `capability_unavailable`, so it is shown as an advisory engine finding, not an alert.",
"planted_python_sql": "STILL MISSED. No quick rule matches the %-formatted db.query string in records.py, and the host has no Joern, so nothing in the pre-push path reports it. Recorded as a coverage gap, with no rule edit.",
"planted_go_command": "NOT DETECTED, NOW NAMED: Go has no engine frontend and no quick rules.",
"engine_deadline": "Not solved. The Docker engine took 190.9 s wall time (head 107.6 s, base 80.7 s) against a 300 s deadline. The audit measured 261.7 s on a different image/src combination. A 30 s hook deadline still cannot hold it. --engine background starts it after the push, and the next push showed the earlier result. On the host that result says 'the engine did not run (Joern unavailable)', which is honest."
},
"coverage_gaps": [
"records: the planted Python SQL injection (`db.query(\"... '%s'\" % name)`) is matched by no quick rule, before or after. Recorded as a quick-rule coverage gap; no rule was edited for a planted sample (that would be hand-tuning). On the host the engine cannot run, so nothing in the pre-push path reports it.",
"secure: Go has no engine frontend and no quick rules. It is now named in the terminal (language not covered: go), not detected.",
"The engine still cannot finish inside the 30 s hook deadline (the audit measured 260 s on 26 files); --engine background moves it after the push, graph reuse remains the later speed task."
],
"after": {
"replay_host": {
"records": {
"pre-push --engine inline": {
"exit": 0,
"wall_seconds": 1.206878771,
"deadline_seconds": 30,
"quick_findings_shown": [],
"engine_findings_shown": [],
"skip_lines": [
"changed lines could not be matched one-to-one to base lines in records.py",
"the changed code has no counterpart in the base revision to compare against",
"engine did not run: the code graph could not be built (is Joern installed? 'joern-parse' must be on PATH, or run the hook through ops/ousast-docker)",
"no base revision was analysed, so no engine finding can be called new or worsened"
],
"other_lines": [
"Quick rules: no pattern match on changed code (1 file(s), 17449 bytes read).",
"No actionable defects reported; analysis is incomplete.",
"Notice: Engine checks did not run. Install Joern, or run the hook through ops/ousast-docker. Review the details."
],
"snapshots_files_bytes": [
[
21,
43577
],
[
21,
43482
]
],
"quick_tier_files_bytes": [
[
1,
17449
]
],
"quick_tier_seconds": [
0.041
],
"coverage_reasons_raw": [
"7265636f7264732e7079:ambiguous_line_correspondence",
"base_counterpart_unresolved",
"cpg_build_failed",
"analysis_incomplete",
"base_comparison_unavailable"
],
"admitted_alerts": 0,
"push_disposition": "allow",
"total_seconds_internal": 0.483
},
"pre-push --engine off": {
"exit": 0,
"wall_seconds": 0.954010589,
"deadline_seconds": 30,
"quick_findings_shown": [],
"engine_findings_shown": [],
"skip_lines": [
"changed lines could not be matched one-to-one to base lines in records.py",
"engine checks were switched off for this run (--engine off)",
"no base revision was analysed, so no engine finding can be called new or worsened"
],
"other_lines": [
"Quick rules: no pattern match on changed code (1 file(s), 17449 bytes read).",
"No analysis result is available.",
"Notice: Some checks could not be completed. Review the comparison details and resolve the recorded gaps before retrying."
],
"snapshots_files_bytes": [
[
21,
43577
],
[
21,
43482
]
],
"quick_tier_files_bytes": [
[
1,
17449
]
],
"quick_tier_seconds": [
0.042
],
"coverage_reasons_raw": [
"7265636f7264732e7079:ambiguous_line_correspondence",
"engine_off",
"analysis_incomplete",
"base_comparison_unavailable"
],
"admitted_alerts": 0,
"push_disposition": "allow",
"total_seconds_internal": 0.498
},
"scan --mode quick": {
"exit": 0,
"wall_seconds": 0.407089313,
"terminal": [
"file_targets=9",
"findings=2"
]
}
},
"serve-static": {
"pre-push --engine inline": {
"exit": 0,
"wall_seconds": 0.892543629,
"deadline_seconds": 30,
"quick_findings_shown": [
"js-eval (CWE-95, critical) at debug.js:2: Dynamic JavaScript evaluation needs review",
"js-xss-response (CWE-79, medium) at debug.js:2: Unsanitized HTTP response reflection needs review"
],
"engine_findings_shown": [],
"skip_lines": [
"the changed code has no counterpart in the base revision to compare against",
"engine did not run: the code graph could not be built (is Joern installed? 'joern-parse' must be on PATH, or run the hook through ops/ousast-docker)",
"no base revision was analysed, so no engine finding can be called new or worsened"
],
"other_lines": [
"Quick rules: 2 pattern match(es) on changed code (1 file(s), 87 bytes read). Advisory: not verified by the engine.",
"No actionable defects reported; analysis is incomplete.",
"Notice: Engine checks did not run. Install Joern, or run the hook through ops/ousast-docker. Review the details."
],
"snapshots_files_bytes": [
[
26,
61525
],
[
25,
61438
]
],
"quick_tier_files_bytes": [
[
1,
87
]
],
"quick_tier_seconds": [
0.017
],
"coverage_reasons_raw": [
"base_counterpart_unresolved",
"cpg_build_failed",
"analysis_incomplete",
"base_comparison_unavailable"
],
"admitted_alerts": 0,
"push_disposition": "allow",
"total_seconds_internal": 0.496
},
"pre-push --engine off": {
"exit": 0,
"wall_seconds": 0.878791036,
"deadline_seconds": 30,
"quick_findings_shown": [
"js-eval (CWE-95, critical) at debug.js:2: Dynamic JavaScript evaluation needs review",
"js-xss-response (CWE-79, medium) at debug.js:2: Unsanitized HTTP response reflection needs review"
],
"engine_findings_shown": [],
"skip_lines": [
"engine checks were switched off for this run (--engine off)",
"no base revision was analysed, so no engine finding can be called new or worsened"
],
"other_lines": [
"Quick rules: 2 pattern match(es) on changed code (1 file(s), 87 bytes read). Advisory: not verified by the engine.",
"No analysis result is available.",
"Notice: Some checks could not be completed. Review the comparison details and resolve the recorded gaps before retrying."
],
"snapshots_files_bytes": [
[
26,
61525
],
[
25,
61438
]
],
"quick_tier_files_bytes": [
[
1,
87
]
],
"quick_tier_seconds": [
0.016
],
"coverage_reasons_raw": [
"engine_off",
"analysis_incomplete",
"base_comparison_unavailable"
],
"admitted_alerts": 0,
"push_disposition": "allow",
"total_seconds_internal": 0.471
},
"scan --mode quick": {
"exit": 0,
"wall_seconds": 0.394269229,
"terminal": [
"file_targets=4",
"findings=2"
]
}
},
"secure": {
"pre-push --engine inline": {
"exit": 0,
"wall_seconds": 0.762186729,
"deadline_seconds": 30,
"quick_findings_shown": [],
"engine_findings_shown": [],
"skip_lines": [
"language not covered: go (1 changed file(s), 10 in the repository): no engine frontend and no quick rules, so silence here is not a clean result",
"the changed code has no counterpart in the base revision to compare against",
"engine did not run: the code graph could not be built (is Joern installed? 'joern-parse' must be on PATH, or run the hook through ops/ousast-docker)",
"engine has no frontend for at least one language in this push",
"no base revision was analysed, so no engine finding can be called new or worsened"
],
"other_lines": [
"Quick rules: no pattern match on changed code (1 file(s), 193 bytes read).",
"No actionable defects reported; analysis is incomplete.",
"Notice: Engine checks did not run. Install Joern, or run the hook through ops/ousast-docker. Review the details."
],
"snapshots_files_bytes": [
[
19,
103023
],
[
18,
102830
]
],
"quick_tier_files_bytes": [
[
1,
193
]
],
"quick_tier_seconds": [
0.015
],
"coverage_reasons_raw": [
"language_not_covered:go:1:10",
"base_counterpart_unresolved",
"cpg_build_failed",
"frontend_unsupported",
"analysis_incomplete",
"base_comparison_unavailable"
],
"admitted_alerts": 0,
"push_disposition": "allow",
"total_seconds_internal": 0.352
},
"pre-push --engine off": {
"exit": 0,
"wall_seconds": 0.694374205,
"deadline_seconds": 30,
"quick_findings_shown": [],
"engine_findings_shown": [],
"skip_lines": [
"language not covered: go (1 changed file(s), 10 in the repository): no engine frontend and no quick rules, so silence here is not a clean result",
"engine checks were switched off for this run (--engine off)",
"no base revision was analysed, so no engine finding can be called new or worsened"
],
"other_lines": [
"Quick rules: no pattern match on changed code (1 file(s), 193 bytes read).",
"No analysis result is available.",
"Notice: Some checks could not be completed. Review the comparison details and resolve the recorded gaps before retrying."
],
"snapshots_files_bytes": [
[
19,
103023
],
[
18,
102830
]
],
"quick_tier_files_bytes": [
[
1,
193
]
],
"quick_tier_seconds": [
0.016
],
"coverage_reasons_raw": [
"language_not_covered:go:1:10",
"engine_off",
"analysis_incomplete",
"base_comparison_unavailable"
],
"admitted_alerts": 0,
"push_disposition": "allow",
"total_seconds_internal": 0.311
},
"scan --mode quick": {
"exit": 0,
"wall_seconds": 0.372512766,
"terminal": [
"file_targets=10",
"findings=0",
"notice: language not covered: go (10 files)"
]
}
}
},
"hook_host": {
"records": {
"new-branch": {
"push_exit": 0,
"wall_seconds": 0.873912782,
"quick_findings_shown": [],
"engine_findings_shown": [],
"skip_lines": [
"new branch: the remote has no base to compare against, so the engine comparison did not run (OUSAST_COMPARISON_BASE=<local branch> compares against a local branch instead)",
"no base revision was analysed, so no engine finding can be called new or worsened"
],
"other_lines": [
"New branch: instead, quick rules checked the files its last commit changed (compared with c925575).",
"Quick rules: no pattern match on changed code (2 file(s), 18416 bytes read).",
"No analysis result is available.",
"Notice: Some checks could not be completed. Review the comparison details and resolve the recorded gaps before retrying.",
"To /tmp/claude-1000/-home-mc-Source-OpenUltraSAST/c09e30b2-39a4-4564-bc31-6ea71ffaa198/scratchpad/ready-impl/bare-records.git",
" * [new branch] HEAD~1 -> main"
]
},
"update": {
"push_exit": 0,
"wall_seconds": 0.890046656,
"quick_findings_shown": [],
"engine_findings_shown": [],
"skip_lines": [
"changed lines could not be matched one-to-one to base lines in records.py",
"the changed code has no counterpart in the base revision to compare against",
"engine did not run: the code graph could not be built (is Joern installed? 'joern-parse' must be on PATH, or run the hook through ops/ousast-docker)",
"no base revision was analysed, so no engine finding can be called new or worsened"
],
"other_lines": [
"Quick rules: no pattern match on changed code (1 file(s), 17449 bytes read).",
"No actionable defects reported; analysis is incomplete.",
"Notice: Engine checks did not run. Install Joern, or run the hook through ops/ousast-docker. Review the details.",
"To /tmp/claude-1000/-home-mc-Source-OpenUltraSAST/c09e30b2-39a4-4564-bc31-6ea71ffaa198/scratchpad/ready-impl/bare-records.git",
" ea42736..ad3d7db HEAD -> main"
]
},
"install_uninstall": [
"install rc=0",
"uninstall rc=0"
]
},
"serve-static": {
"new-branch": {
"push_exit": 0,
"wall_seconds": 0.904302643,
"quick_findings_shown": [],
"engine_findings_shown": [],
"skip_lines": [
"new branch: the remote has no base to compare against, so the engine comparison did not run (OUSAST_COMPARISON_BASE=<local branch> compares against a local branch instead)",
"no base revision was analysed, so no engine finding can be called new or worsened"
],
"other_lines": [
"New branch: instead, quick rules checked the files its last commit changed (compared with 76312e1).",
"Quick rules: no pattern match on changed code (0 file(s), 0 bytes read).",
"No analysis result is available.",
"Notice: Some checks could not be completed. Review the comparison details and resolve the recorded gaps before retrying.",
"To /tmp/claude-1000/-home-mc-Source-OpenUltraSAST/c09e30b2-39a4-4564-bc31-6ea71ffaa198/scratchpad/ready-impl/bare-serve-static.git",
" * [new branch] HEAD~1 -> main"
]
},
"update": {
"push_exit": 0,
"wall_seconds": 0.92386555,
"quick_findings_shown": [
"js-eval (CWE-95, critical) at debug.js:2: Dynamic JavaScript evaluation needs review",
"js-xss-response (CWE-79, medium) at debug.js:2: Unsanitized HTTP response reflection needs review"
],
"engine_findings_shown": [],
"skip_lines": [
"the changed code has no counterpart in the base revision to compare against",
"engine did not run: the code graph could not be built (is Joern installed? 'joern-parse' must be on PATH, or run the hook through ops/ousast-docker)",
"no base revision was analysed, so no engine finding can be called new or worsened"
],
"other_lines": [
"Quick rules: 2 pattern match(es) on changed code (1 file(s), 87 bytes read). Advisory: not verified by the engine.",
"No actionable defects reported; analysis is incomplete.",
"Notice: Engine checks did not run. Install Joern, or run the hook through ops/ousast-docker. Review the details.",
"To /tmp/claude-1000/-home-mc-Source-OpenUltraSAST/c09e30b2-39a4-4564-bc31-6ea71ffaa198/scratchpad/ready-impl/bare-serve-static.git",
" 74be78a..468213d HEAD -> main"
]
},
"no-ousast": {
"push_exit": 0,
"wall_seconds": 0.017937758,
"quick_findings_shown": [],
"engine_findings_shown": [],
"skip_lines": [],
"other_lines": [
"ousast: pre-push check did not run: 'ousast' is not on PATH. Install it (pip install git+https://github.com/norandom/OpenUltraSAST) or set OUSAST_COMMAND. Push continues.",
"To /tmp/claude-1000/-home-mc-Source-OpenUltraSAST/c09e30b2-39a4-4564-bc31-6ea71ffaa198/scratchpad/ready-impl/bare-serve-static.git",
" 468213d..aad3dfe HEAD -> main"
]
},
"bad-deadline": {
"push_exit": 0,
"wall_seconds": 0.300776,
"quick_findings_shown": [],
"engine_findings_shown": [],
"skip_lines": [],
"other_lines": [
"usage: ousast pre-push [-h] [--base BASE] [--head HEAD] [--remote NAME URL]",
" [--comparison-base COMPARISON_BASE]",
" [--model-config MODEL_CONFIG] [--prior-hook PRIOR_HOOK]",
" --artifact ARTIFACT [--cache-dir CACHE_DIR]",
" [--deadline DEADLINE]",
" [--cancellation-allowance CANCELLATION_ALLOWANCE]",
" [--max-regions MAX_REGIONS]",
" [--mode {advisory,blocking}]",
" [--incomplete-coverage {allow,block}]",
" [--engine {inline,background,off}]",
" [--background-deadline BACKGROUND_DEADLINE]",
" [--experimental-declarations EXPERIMENTAL_DECLARATIONS]",
" [--experimental-record-vetoes]",
" path",
"ousast pre-push: error: argument --deadline: invalid float value: 'abc'",
"ousast: pre-push check did not run: invalid arguments or settings (message above). Push continues.",
"To /tmp/claude-1000/-home-mc-Source-OpenUltraSAST/c09e30b2-39a4-4564-bc31-6ea71ffaa198/scratchpad/ready-impl/bare-serve-static.git",
" aad3dfe..6fcfec7 HEAD -> main"
]
},
"background-1": {
"push_exit": 0,
"wall_seconds": 0.915950565,
"quick_findings_shown": [],
"engine_findings_shown": [
"Engine: running in the background for c5b6c4d7060f (deadline 900.0 s); the next push shows its result."
],
"skip_lines": [
"engine checks run in the background after this push; the next push shows their result",
"no base revision was analysed, so no engine finding can be called new or worsened"
],
"other_lines": [
"Quick rules: no pattern match on changed code (1 file(s), 93 bytes read).",
"No analysis result is available.",
"Engine: running in the background for c5b6c4d7060f (deadline 900.0 s); the next push shows its result.",
"Notice: Some checks could not be completed. Review the comparison details and resolve the recorded gaps before retrying.",
"To /tmp/claude-1000/-home-mc-Source-OpenUltraSAST/c09e30b2-39a4-4564-bc31-6ea71ffaa198/scratchpad/ready-impl/bare-serve-static.git",
" 6fcfec7..c5b6c4d HEAD -> main"
]
},
"background-2": {
"push_exit": 0,
"wall_seconds": 0.879868436,
"quick_findings_shown": [],
"engine_findings_shown": [
"Engine result for c5b6c4d7060f from the previous push: the engine did not run (Joern unavailable)."
],
"skip_lines": [
"engine checks were switched off for this run (--engine off)",
"no base revision was analysed, so no engine finding can be called new or worsened"
],
"other_lines": [
"Engine result for c5b6c4d7060f from the previous push: the engine did not run (Joern unavailable).",
"Quick rules: no pattern match on changed code (1 file(s), 99 bytes read).",
"No analysis result is available.",
"Notice: Some checks could not be completed. Review the comparison details and resolve the recorded gaps before retrying.",
"To /tmp/claude-1000/-home-mc-Source-OpenUltraSAST/c09e30b2-39a4-4564-bc31-6ea71ffaa198/scratchpad/ready-impl/bare-serve-static.git",
" c5b6c4d..3eee5ef HEAD -> main"
]
},
"install_uninstall": [
"install rc=0",
"uninstall rc=0"
]
},
"secure": {
"new-branch": {
"push_exit": 0,
"wall_seconds": 0.96295861,
"quick_findings_shown": [],
"engine_findings_shown": [],
"skip_lines": [
"new branch: the remote has no base to compare against, so the engine comparison did not run (OUSAST_COMPARISON_BASE=<local branch> compares against a local branch instead)",
"no base revision was analysed, so no engine finding can be called new or worsened"
],
"other_lines": [
"New branch: instead, quick rules checked the files its last commit changed (compared with ba42bc3).",
"Quick rules: no pattern match on changed code (1 file(s), 41228 bytes read).",
"No analysis result is available.",
"Notice: Some checks could not be completed. Review the comparison details and resolve the recorded gaps before retrying.",
"To /tmp/claude-1000/-home-mc-Source-OpenUltraSAST/c09e30b2-39a4-4564-bc31-6ea71ffaa198/scratchpad/ready-impl/bare-secure.git",
" * [new branch] HEAD~1 -> main"
]
},
"update": {
"push_exit": 0,
"wall_seconds": 0.747049238,
"quick_findings_shown": [],
"engine_findings_shown": [],
"skip_lines": [
"language not covered: go (1 changed file(s), 10 in the repository): no engine frontend and no quick rules, so silence here is not a clean result",
"the changed code has no counterpart in the base revision to compare against",
"engine did not run: the code graph could not be built (is Joern installed? 'joern-parse' must be on PATH, or run the hook through ops/ousast-docker)",
"engine has no frontend for at least one language in this push",
"no base revision was analysed, so no engine finding can be called new or worsened"
],
"other_lines": [
"Quick rules: no pattern match on changed code (1 file(s), 193 bytes read).",
"No actionable defects reported; analysis is incomplete.",
"Notice: Engine checks did not run. Install Joern, or run the hook through ops/ousast-docker. Review the details.",
"To /tmp/claude-1000/-home-mc-Source-OpenUltraSAST/c09e30b2-39a4-4564-bc31-6ea71ffaa198/scratchpad/ready-impl/bare-secure.git",
" 740abc8..64739c4 HEAD -> main"
]
},
"install_uninstall": [
"install rc=0",
"uninstall rc=0"
]
}
},
"docker_engine_serve_static": {
"log": [
"MemAvailable=3283MB disk_free=1159MiB",
"base=74be78a8ffad679edfe135c457ce141114d96fcc head=468213d8a737ad03a1f712c792c1db8476e154df",
"exit=0 elapsed=190.895648822s"
],
"exit": 0,
"wall_seconds": 190.895648822,
"deadline_seconds": 300,
"quick_findings_shown": [
"js-eval (CWE-95, critical) at debug.js:2: Dynamic JavaScript evaluation needs review",
"js-xss-response (CWE-79, medium) at debug.js:2: Unsanitized HTTP response reflection needs review"
],
"engine_findings_shown": [
"Engine: 2 finding(s) this push introduced. Advisory: the capability is not yet qualified, so not an alert."
],
"skip_lines": [
"engine findings are not qualified as alerts yet: no capability has passed independent evaluation",
"the capability registry does not match this build (eligibility provenance mismatch), so no capability can be used",
"20 engine check(s) depend on dynamic or external code the engine cannot follow"
],
"other_lines": [
"Engine: 2 finding(s) this push introduced. Advisory: the capability is not yet qualified, so not an alert.",
"Quick rules: 2 pattern match(es) on changed code (1 file(s), 87 bytes read). Advisory: not verified by the engine.",
"No actionable defects reported; analysis is incomplete.",
"Notice: Some checks are not enabled for normal alerts. Review capability coverage in the details."
],
"snapshots_files_bytes": [
[
26,
61525
],
[
25,
61438
]
],
"quick_tier_files_bytes": [
[
1,
87
]
],
"quick_tier_seconds": [
0.038
],
"coverage_reasons_raw": [
"capability_unavailable",
"eligibility:eligibility_provenance_mismatch",
"dynamic_external_or_depth_context_unresolved:contributor-scan:7ba72698b15c888c0835cad443fc1f177f1d2e13ed2658b17e816cc0b81415de",
"dynamic_external_or_depth_context_unresolved:contributor-scan:392beff95d0e245cbaef2676813d7f69d35599fb32977c8f197e6df46738fc75",
"dynamic_external_or_depth_context_unresolved:contributor-scan:ec16a67e984a2f82c3ce9a04dbc44eae1e950393b1c1c4fae88485446f1e2f95",
"dynamic_external_or_depth_context_unresolved:contributor-scan:0a3bd3f145260792e9a5d4af2546020689cfd08d0b1e82264d7fe31dda7210e7",
"dynamic_external_or_depth_context_unresolved:contributor-scan:1551c000ba35ba4e5488013f51171b27ab10df8b80f29509610670192cff8a30",
"dynamic_external_or_depth_context_unresolved:contributor-scan:5b687fa7647cefd89d064b3e1b5e5a62842f8fd23104900e8af8d5b14fc800fd",
"dynamic_external_or_depth_context_unresolved:contributor-scan:f122da538137e0a3a3e598c24e6af57c5e0b90ff48dacb4fd1b9d82ac20c35ef",
"dynamic_external_or_depth_context_unresolved:contributor-scan:8ab9ddcb1f4431fa3a3c1cce4cc3d289bcb5d363d47aac1bcf35bc7db86ce866",
"dynamic_external_or_depth_context_unresolved:contributor-scan:19f1031c2c010d91fb41e23955cd5f18eaa5ede355e68fab9e18c8249c21c2ee",
"dynamic_external_or_depth_context_unresolved:contributor-scan:d31a861124274a09e3cb8f3398831ca299cc0cfad72ea385083c1fd88b7a2c45",
"dynamic_external_or_depth_context_unresolved:contributor-scan:c8a1c8042befd3ff6f296921666949850a1c54d63c77dd44296e7be96e621127",
"dynamic_external_or_depth_context_unresolved:contributor-scan:3bb50c50f77027f1e0347f490e35fdd0a75ae586fdf92775c753877833a9d099",
"dynamic_external_or_depth_context_unresolved:contributor-scan:c32cb93cad379fee7b814900e6673aa5ae470c27acd55d33f855987bccb1ab8e",
"dynamic_external_or_depth_context_unresolved:contributor-scan:300fba1acc39cb0ab32ebbd66a8674daecf496b3d8c675d27f7f71cf9f788c9e",
"dynamic_external_or_depth_context_unresolved:contributor-scan:40c99dfe6d923602dc3aa6353f3b5bc017c0d1a67595c19ddf8206ef431db23d",
"dynamic_external_or_depth_context_unresolved:contributor-scan:0e99666c9be7c4704f6c6f858844d721ed8f0b4e3978d53073bc2c1b64d3f075",
"dynamic_external_or_depth_context_unresolved:contributor-scan:da903214d018e0b1bd7a3a3b635b3e0fb2a23fd44527f41a9483c0bf3913f2df",
"dynamic_external_or_depth_context_unresolved:contributor-scan:79d89b41b86aadcc0423baa786e0518ccff2a5b7853ff05986682ecbd636af52",
"dynamic_external_or_depth_context_unresolved:contributor-scan:7d3331ca3c06d1013c758d3648bfebb1223385955e77df977266fc621f62e733",
"dynamic_external_or_depth_context_unresolved:contributor-scan:54fd93450c8db7e6daef06fe795babb26726402211ca8f6d0a8679a375786041",
"analysis_incomplete"
],
"admitted_alerts": 0,
"push_disposition": "allow",
"total_seconds_internal": 189.327,
"dispositions": [
{
"family": "injection",
"site": "debug.js:2:debug",
"novelty": "new",
"reason": "file_added_in_head",
"vetoes": [
"capability_unavailable"
]
},
{
"family": "output_encoding",
"site": "debug.js:2:debug",
"novelty": "new",
"reason": "file_added_in_head",
"vetoes": [
"capability_unavailable"
]
}
],
"timings": {
"deadline_seconds": 300.0,
"cancellation_allowance_seconds": 2.0,
"resolution_seconds": 0.0,
"provenance_seconds": 0.3,
"snapshot_seconds": 0.3,
"discovery_seconds": 0.3,
"preparation_seconds": 0.6,
"quick_seconds": 0.0,
"head_seconds": 107.6,
"base_seconds": 80.7,
"admission_seconds": 0.0,
"cleanup_seconds": 0.0,
"total_seconds": 189.3
}
}
},
"build": {
"worktree_head": "591818d + hook install-hint fix (this commit)",
"wheel_sha256": "5b725adb052239c2afaec237fecf582f69fc5ea61799dc67f0c07e09e10908c9",
"installed_with": "uv pip install <wheel> into a scratch venv"
},
"inputs": {
"records": {
"language": "python",
"tracked_files": 21,
"tracked_bytes": 43577,
"planted": "records.py: db.query(\"... '%s'\" % name)"
},
"serve-static": {
"language": "javascript",
"tracked_files": 26,
"tracked_bytes": 61525,
"planted": "new debug.js: res.end(String(eval(req.query.code)))"
},
"secure": {
"language": "go",
"tracked_files": 19,
"tracked_bytes": 103023,
"planted": "new run.go: exec.Command(\"sh\", \"-c\", r.URL.Query().Get(\"cmd\"))"
}
},
"notes": [
"The Docker engine run used the worktree src at 591818d mounted over the stale 2026-09-14 openultrasast:dev image (not rebuilt: 1.1 GB disk free, below the 4 GB threshold). `eligibility_provenance_mismatch` still appears; the audit already suspected src/image skew, and it is not investigated here.",
"The run shows host paths after `Details:` because ops/ousast-docker mounts the artifact directory at its own path. Before, it showed the container path /out/...",
"The hook's 'not on PATH' hint first said 'pip install openultrasast', which is not on PyPI. The first proof run caught it. It was corrected to the git URL, and the replay and hook proofs were rerun on the rebuilt wheel recorded under build. The Docker engine run predates the hint fix and does not involve the hook script.",
"Host runs read their input. Each head snapshot's bytes equal `git ls-files | xargs cat | wc -c` at head, and the quick tier lists the files it read."
]
}
@@ -0,0 +1,351 @@
{
"created": "2026-10-03",
"note": "COUNTS ONLY. Real (vulnerable function, fixed function) pairs where the function survives the fix, mined at zero model cost from reviewed GitHub advisories for the six compiled families, added as pointer-pair corpus benchmarks/pairs/advisory-fixes-2 (98 pairs, source `advisory-fixes-2` in learn/sources.toml), labelled, given static feature records and built into the memory store; their excerpts embedded. No chat-model call. Candidate, path, function and repository names stay in the catalog; the two pairs the reserved-population test flagged are withheld (never printed: found by bisection with the test's output discarded).",
"method": {
"enumeration": "gh api /advisories?type=reviewed&ecosystem=<eco>&cwes=<family CWEs>&published=>2025-06-01, per ecosystem (pip, npm, maven, composer, go, rubygems) x family; newest first",
"advisories_enumerated": 6594,
"advisories_published_after_2026_06_01": 2400,
"automated_screen": "single family by CWE; a linked fix commit (at most 3, one repository); repository not named anywhere under benchmarks/ (populations v1/v2, every pair corpus; excluded v3 file names never opened), no WordPress plugin, one pair per repository; SPDX licence permissive; one-parent commit with <= 4 non-test source files and <= 120 changed source lines; the function holding most changed CODE lines (comments and blanks excluded) inside its own matcher span, declared exactly once on both sides, body changed, >= 25% of the file's changed code lines (second pass; the first pass counted lines by nearest declaration above)",
"file_reads": "the two versions of each changed source file through the contents API (metadata + two raw files per file; no clone, no scanner, engine, plane task or model on any of these repositories)",
"manual_review": "every automated candidate's function diff was read (agent, no model call) and kept only when the named function itself holds the fix",
"reserved_population_guard": "tests/test_independent_population.py::test_no_reserved_repository_is_referenced_by_the_development_tree, run with its output discarded; bisection over catalog subsets, 23 runs",
"quotas": "20 per family on the second pass (14 on the first); advisories past a met quota were not examined",
"mechanism": "family default (injection/output_encoding/deserialization source_reaches_sink, path path_join_user_input, access_control missing_auth_guard, untrusted_destination validation_strength for CWE-601 else source_reaches_sink); provenance = agent"
},
"screened": {
"advisories_examined_second_pass": 2983,
"outcomes_second_pass": {
"no fix commit linked": 1247,
"repository already in benchmarks/ (populations v1/v2 or a pair corpus)": 520,
"language outside the feature vocabulary (go/ruby)": 332,
"CWEs span several families": 253,
"repository already kept (one pair per repository)": 140,
"licence not permissive or unrecorded": 135,
"large commit (refactor or release, not the fix)": 125,
"kept": 110,
"no function declared on both sides with a changed body": 51,
"fix spread over several commits or repositories": 25,
"manual review: the chosen function does not hold the fix (comment-only, misattributed, partial or family mismatch)": 13,
"no source file changed": 11,
"merge or root commit": 11,
"language outside the feature vocabulary (go/ruby/c#/...)": 9,
"WordPress plugin (population v3 safety)": 1
},
"candidates_manually_reviewed": 145,
"kept_after_review": 100,
"review_drops": {
"total": 45,
"function misattributed (matcher span overrun or an undetected enclosing declaration)": 13,
"change in the function unrelated to the advisory (refactor, rename, formatting, another advisory)": 11,
"partial: the function only calls or configures the real fix": 7,
"comment or documentation change only": 6,
"one pair per repository (a reviewed pair of the same repository kept)": 4,
"second pass chose another function of an advisory already reviewed": 2,
"family mismatch": 1,
"unsure": 1
},
"reserved_population_drops": 2,
"kept": 98
},
"kept": {
"total": 98,
"by_family": {
"access_control": 17,
"deserialization": 11,
"injection": 20,
"output_encoding": 16,
"path": 17,
"untrusted_destination": 17
},
"post_cutoff_by_family": {
"access_control": 13,
"deserialization": 2,
"injection": 20,
"output_encoding": 16,
"path": 17,
"untrusted_destination": 11
},
"post_cutoff_total": 79,
"by_language": {
"typescript": 34,
"python": 30,
"javascript": 15,
"java": 10,
"php": 9
},
"by_ecosystem": {
"npm": 48,
"pip": 32,
"maven": 9,
"composer": 9
},
"licences": {
"MIT": 67,
"Apache-2.0": 24,
"BSD-3-Clause": 5,
"ISC": 1,
"Unlicense": 1
},
"split": {
"train": 51,
"holdout": 47
},
"published_range": [
"2025-06-06",
"2026-10-02"
],
"verification": {
"verified": 98,
"excerpt_bytes_read": 526134
},
"known_limit": "excerpts inherit the label builder's matcher: some spans run past the function's end into the next declaration; the label is path::function and the changed lines were checked inside the span"
},
"labels": {
"command": "ousast learn labels --root . --out <scratch> --snapshot <scratch>",
"rows": [
6589,
6981
],
"new_pair_rows": 392,
"new_pin_rows_by_label": {
"positive": 98,
"negative": 98
},
"fixed_side_negative": "98 of 98 pairs (absent_fixed_side 0)"
},
"features": {
"command": "benchmarks/learn/build_features.py --labels <the 392 new rows> --units <units.json of `ousast plane harvest --catalog benchmarks/pairs/advisory-fixes-2/catalog.toml` (manifests written, no Run started)> --kinds pairs",
"records": 392,
"instrument_states_plane_all_families": {
"agree:missing": 74,
"agree:none": 122,
"delta:not_applicable": 196,
"engine:missing": 128,
"engine:none": 68,
"entry_points:not_applicable": 196,
"facts:not_applicable": 196,
"language:ran": 196,
"model_sinks:missing": 196,
"quick:ran": 196,
"roles:ran": 196,
"source:ran": 196,
"verify:missing": 74,
"verify:none": 122
},
"note": "quick rules and deterministic roles ran on each side; engine, verify, agree and model sinks never ran for these pairs (missing/none: null features, never zeros)"
},
"memory_build": {
"command": "ousast learn memory build --labels <labels> --units <merged units.json> --profile static --root . (exit 2: adjudications 9 of 43 anonymous functions, unchanged since 2026-10-01; rows kept)",
"examples": [
937,
1133
],
"pairs_examples": [
772,
968
],
"pairs_counts_after": {
"absent_fixed_side": 0,
"examples": 968,
"labels": 1936,
"no_excerpt": 0,
"no_record": 968,
"recorded": 968,
"unread": 0
}
},
"positives_only_groups": {
"how": "the 2026-10-02 gap record's count: static example rows of the store, per family, repository groups with a positive and no negative",
"before": 99,
"after": 99,
"before_note": "the store before this harvest held 937 examples (the 10-02 gap record counted 904 and 98; the 10-02 rebuild added one population-v2 positive without a negative)",
"why_unchanged": "the 99 are data gaps of other corpora (94 vibe-py pairs whose fixed side lost the function, 3 dev-php, 2 population-v2); a new repository group arrives with both labels, so it adds paired groups and leaves those counts as they were",
"paired_groups_total": [
151,
249
],
"positives_only_share_of_positive_groups_six_families": [
0.4034,
0.2857
],
"by_family_before_after": {
"access_control": {
"groups_pos_only": [
12,
12
],
"groups_pos": [
32,
49
],
"groups_both": [
20,
37
],
"examples_neg": [
20,
37
]
},
"config_secrets": {
"groups_pos_only": [
3,
3
],
"groups_pos": [
5,
5
],
"groups_both": [
2,
2
],
"examples_neg": [
13,
13
]
},
"deserialization": {
"groups_pos_only": [
20,
20
],
"groups_pos": [
41,
52
],
"groups_both": [
21,
32
],
"examples_neg": [
21,
32
]
},
"injection": {
"groups_pos_only": [
11,
11
],
"groups_pos": [
44,
64
],
"groups_both": [
33,
53
],
"examples_neg": [
70,
90
]
},
"memory": {
"groups_pos_only": [
0,
0
],
"groups_pos": [
5,
5
],
"groups_both": [
5,
5
],
"examples_neg": [
173,
173
]
},
"output_encoding": {
"groups_pos_only": [
14,
14
],
"groups_pos": [
36,
52
],
"groups_both": [
22,
38
],
"examples_neg": [
61,
77
]
},
"path": {
"groups_pos_only": [
12,
12
],
"groups_pos": [
33,
50
],
"groups_both": [
21,
38
],
"examples_neg": [
21,
38
]
},
"unknown": {
"groups_pos_only": [
0,
0
],
"groups_pos": [
2,
2
],
"groups_both": [
2,
2
],
"examples_neg": [
2,
2
]
},
"untrusted_destination": {
"groups_pos_only": [
27,
27
],
"groups_pos": [
52,
69
],
"groups_both": [
25,
42
],
"examples_neg": [
36,
53
]
}
}
},
"embeddings": {
"command": "ousast learn memory embed --budget-usd 0.20 (dry run first: 66,005 estimated tokens, $0.0013)",
"requested": 1004,
"cached": 781,
"embedded": 223,
"calls": 7,
"tokens": 71163,
"usd": 0.001423,
"note": "embedded = the new pairs' excerpts plus the excerpts the 10-02 rebuild left without a vector; OpenRouter, the embedding model .env names"
},
"spend": {
"chat_model_calls": 0,
"embedding_usd": 0.001423,
"github_api": "metadata and raw contents only"
}
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,326 @@
{
"created": "2026-10-04",
"note": "COUNTS ONLY. Batch 3 of real (vulnerable function, fixed function) pairs whose function survives the fix, mined at zero model cost from reviewed GitHub advisories (pip, npm, maven, composer) and added as pointer-pair corpus benchmarks/pairs/advisory-fixes-3 (source `advisory-fixes-3` in learn/sources.toml); labelled, given static feature records, built into the memory store and embedded. No chat-model call, no Docker, no Joern. Candidate, path, function and repository names stay in the catalog; the pairs the reserved-population guard flagged are withheld (found by bisection with the test output discarded; names never printed).",
"method": {
"enumeration": "gh api /advisories?type=reviewed&ecosystem=<eco>&cwes=<family CWEs>&published=>2023-01-01 per ecosystem (pip, npm, maven, composer) x six compiled families; Go, Ruby and C# not enumerated; newest first",
"advisories_enumerated": 8531,
"advisories_published_after_2026_06_01": 1855,
"automated_screen": "as batch 2 (single family by CWE; linked fix commit, at most 3, one repository; repository not named anywhere under benchmarks/ incl. batches 1-2; excluded v3 file names never opened; no WordPress plugin; one pair per repository; SPDX licence permissive; one-parent commit; the function holding most changed code lines inside its own matcher span, declared once on both sides, body changed, >= 25% of the file's changed code lines) plus three widenings: (a) an advisory linking only a merged pull request uses its single commit or its squash/rebase merge commit; (b) commits of up to 6 non-test source files / 240 changed lines (batch 2: 4 / 120); (c) a second pass over advisories whose CWEs span two families, the family then chosen from the function's diff on review",
"file_reads": "commit metadata and the two versions of each changed source file through the contents API into scratch (no clone, no scanner, engine, plane Run or model on these repositories); scratch deleted after",
"manual_review": "every automated candidate's in-function diff read (agent, no model call); kept only when the named function itself holds the fix; a weak inline check replaced by a stronger helper call counts as the fix, a function that only forwards a flag or calls a fix made elsewhere does not",
"reserved_population_guard": "pytest -q tests/ -k 'reserved or v3' with output discarded, bisected over catalog subsets (18 runs; baseline with an empty batch green); final catalog green",
"rename_check": "canonical full_name of every kept repository from the API: no two kept pairs share a repository, none renamed into a name the benchmarks use",
"mechanism": "family default as batch 2 (injection/output_encoding/deserialization source_reaches_sink, path path_join_user_input, access_control missing_auth_guard, untrusted_destination validation_strength for CWE-601 else source_reaches_sink); provenance = agent"
},
"screened": {
"advisories_examined": 8805,
"outcomes": {
"no fix commit linked": 3480,
"repository already in benchmarks/ (populations v1/v2 or a pair corpus)": 1993,
"licence not permissive or unrecorded": 1249,
"repository already kept (one pair per repository)": 657,
"kept": 398,
"CWEs span several families": 274,
"no function declared on both sides with a changed body": 191,
"large commit (refactor or release, not the fix)": 173,
"fix spread over several commits or repositories": 167,
"merge or root commit": 115,
"no source file changed": 42,
"language outside the feature vocabulary (go/ruby/c#/...)": 32,
"manual review: the chosen function does not hold the fix (comment-only, misattributed, partial or family mismatch)": 19,
"repository unavailable": 9,
"WordPress plugin (population v3 safety)": 4,
"fix commit unavailable": 2
},
"outcomes_note": "single-family pass (screen 2, every advisory examined, no quota met) plus the two-family pass; `kept` = automated candidates; `manual review: the chosen function does not hold the fix (comment-only, misattributed, partial or family mismatch)` = the screen re-meeting a function the 2026-10-03 review had dropped",
"candidates_manually_reviewed": 412,
"kept_after_review": 255,
"review_drops": {
"total": 157,
"change in the function unrelated to the advisory (refactor, rename, formatting, another advisory)": 33,
"function misattributed (matcher span overrun or an undetected enclosing declaration)": 33,
"partial: the function only calls or configures the real fix": 29,
"function diff not small (rewrite or refactor around the fix)": 28,
"unsure": 17,
"family mismatch": 11,
"same fix as a kept pair in a fork or sibling repository (would leak across repository folds)": 3,
"WordPress plugin (population v3 safety; missed by the automated screen)": 2,
"reserved-population guard (flagged by the 2026-10-03 guard run; name not printed)": 1
},
"one_pair_per_repository_drops": 3,
"reserved_population_drops": 2,
"kept": 251
},
"kept": {
"total": 251,
"by_family": {
"access_control": 20,
"deserialization": 9,
"injection": 56,
"output_encoding": 69,
"path": 71,
"untrusted_destination": 26
},
"post_cutoff_by_family": {
"injection": 5,
"output_encoding": 5,
"path": 2
},
"post_cutoff_total": 12,
"by_language": {
"java": 37,
"javascript": 37,
"php": 46,
"python": 83,
"typescript": 48
},
"post_cutoff_by_language": {
"python": 6,
"typescript": 6
},
"by_family_language": {
"access_control:java": 5,
"access_control:javascript": 3,
"access_control:php": 7,
"access_control:python": 2,
"access_control:typescript": 3,
"deserialization:java": 2,
"deserialization:php": 1,
"deserialization:python": 6,
"injection:java": 5,
"injection:javascript": 7,
"injection:php": 9,
"injection:python": 24,
"injection:typescript": 11,
"output_encoding:java": 7,
"output_encoding:javascript": 20,
"output_encoding:php": 12,
"output_encoding:python": 14,
"output_encoding:typescript": 16,
"path:java": 10,
"path:javascript": 4,
"path:php": 10,
"path:python": 34,
"path:typescript": 13,
"untrusted_destination:java": 8,
"untrusted_destination:javascript": 3,
"untrusted_destination:php": 7,
"untrusted_destination:python": 3,
"untrusted_destination:typescript": 5
},
"by_ecosystem": {
"composer": 46,
"maven": 37,
"npm": 81,
"pip": 87
},
"licences": {
"Apache-2.0": 80,
"BSD-2-Clause": 3,
"BSD-3-Clause": 18,
"ISC": 1,
"MIT": 147,
"MPL-2.0": 1,
"Unlicense": 1
},
"split": {
"holdout": 124,
"train": 127
},
"published_by_year": {
"2023": 50,
"2024": 52,
"2025": 66,
"2026": 83
},
"published_range": [
"2023-01-04",
"2026-10-01"
],
"via_pull_request": 13,
"commit_over_120_source_lines": 10,
"two_family_advisory_family_chosen_on_review": 3,
"verification": {
"verified": 251,
"excerpt_bytes_read": 1297516
},
"targets": {
"asked": "injection and untrusted_destination >= 60 each; path, access_control, output_encoding, deserialization >= 30 where advisories allow; >= 250 total",
"met": {
"access_control": false,
"deserialization": false,
"injection": false,
"output_encoding": true,
"path": true,
"untrusted_destination": false
},
"limit": "the reviewed-advisory pool, not the review: every advisory since 2023-01-01 in the four ecosystems was examined; the post-cutoff pool was largely spent by batch 2 (its repositories are now excluded), and untrusted_destination/access_control/deserialization advisories mostly lack a linked single fix commit or sit in repositories the benchmarks already use"
}
},
"labels": {
"command": "ousast learn labels --root . --out <scratch> --snapshot <scratch>",
"rows": [
6981,
7985
],
"pairs_rows": [
1936,
2940
],
"new_pair_rows": 1004
},
"features": {
"command": "benchmarks/learn/build_features.py --labels <the 1004 new rows> --units <merged units.json of `ousast plane harvest --catalog benchmarks/pairs/advisory-fixes-3/catalog.toml` (manifests written, no Run started)> --catalog benchmarks/pairs/advisory-fixes-3/catalog.toml --kinds pairs",
"records": {
"plane:access_control": 40,
"plane:deserialization": 18,
"plane:injection": 112,
"plane:output_encoding": 138,
"plane:path": 142,
"plane:untrusted_destination": 52,
"static:access_control": 40,
"static:deserialization": 18,
"static:injection": 112,
"static:output_encoding": 138,
"static:path": 142,
"static:untrusted_destination": 52
},
"instrument_states_plane_all_families": {
"agree:missing": 164,
"agree:none": 338,
"delta:not_applicable": 502,
"engine:missing": 406,
"engine:none": 96,
"entry_points:not_applicable": 502,
"facts:not_applicable": 502,
"language:ran": 502,
"model_sinks:missing": 502,
"quick:ran": 502,
"roles:ran": 502,
"source:ran": 502,
"verify:missing": 164,
"verify:none": 338
},
"note": "quick rules and deterministic roles ran on each side; engine, verify, agree and model sinks never ran for these pairs (missing/none: null features, never zeros)"
},
"memory_build": {
"command": "ousast learn memory build --labels <labels> --units <merged units.json> --profile static --root . (exit 2: adjudications 9 of 43 anonymous functions, unchanged since 2026-10-01; rows kept)",
"examples": [
1133,
1635
],
"pairs_examples": [
968,
1470
],
"pairs_counts_after": {
"absent_fixed_side": 0,
"examples": 1470,
"labels": 2940,
"no_excerpt": 0,
"no_record": 1470,
"recorded": 1470,
"unread": 0
}
},
"complete_pairs": {
"how": "store example rows (profile static) keyed as learn.experiments.freeze_units pairs them: (repo, candidate, family) holding both labels; whole memory, every family",
"before": 459,
"after": 710,
"by_family": {
"access_control": [
37,
57
],
"config_secrets": [
12,
12
],
"deserialization": [
32,
41
],
"injection": [
68,
124
],
"memory": [
149,
149
],
"output_encoding": [
77,
146
],
"path": [
38,
109
],
"unknown": [
2,
2
],
"untrusted_destination": [
44,
70
]
},
"exp_004_005_units": "195 complete pairs inside exp-004/005's frozen 501 units (six families, evaluation folds); those units are frozen and were not regenerated here"
},
"positives_only_groups": {
"before": 99,
"after": 99,
"groups_both_by_family": {
"access_control": [
37,
57
],
"config_secrets": [
2,
2
],
"deserialization": [
32,
41
],
"injection": [
53,
109
],
"memory": [
5,
5
],
"output_encoding": [
38,
107
],
"path": [
38,
109
],
"unknown": [
2,
2
],
"untrusted_destination": [
42,
68
]
}
},
"embeddings": {
"command": "ousast learn memory embed --budget-usd 0.20 (dry run first: 144,806 estimated tokens, $0.0029)",
"requested": 1479,
"cached": 1004,
"embedded": 475,
"calls": 15,
"tokens": 150339,
"usd": 0.003007
},
"spend": {
"chat_model_calls": 0,
"embedding_usd": 0.003007,
"github_api": "metadata and raw contents only"
}
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,496 @@
{
"pairs": [
{
"name": "prefect-token-validation-e216171",
"ok": true,
"bytes_read": 3697
},
{
"name": "arcadedb-getdatabasegroupsconfiguration-04110c0",
"ok": true,
"bytes_read": 3896
},
{
"name": "langflow-get-flow-by-id-or-endpoint-name-2c9f498",
"ok": true,
"bytes_read": 3948
},
{
"name": "grav-plugin-api-update-26f529c",
"ok": true,
"bytes_read": 5598
},
{
"name": "google-maps-mcp-main-00d8725",
"ok": true,
"bytes_read": 3371
},
{
"name": "camofox-mcp-createmcphttpapp-599f56e",
"ok": true,
"bytes_read": 5587
},
{
"name": "spring-graphql-isoverridefor-63a89ed",
"ok": true,
"bytes_read": 2045
},
{
"name": "jenkins-dofilltimezonenameitems-0586de4",
"ok": true,
"bytes_read": 2093
},
{
"name": "netlicensing-mcp-dispatch-fbbb1d5",
"ok": true,
"bytes_read": 2610
},
{
"name": "starlette-admin-validate-where-af05b45",
"ok": true,
"bytes_read": 2061
},
{
"name": "next-auth-initauth-d008b9b",
"ok": true,
"bytes_read": 8193
},
{
"name": "deepseek-mcp-server-starthttptransport-dab07ed",
"ok": true,
"bytes_read": 2394
},
{
"name": "network-ai-routerequest-a59c13a",
"ok": true,
"bytes_read": 5945
},
{
"name": "device-builder-parse-args-9e294f7",
"ok": true,
"bytes_read": 15243
},
{
"name": "typespec-start-30d6f65",
"ok": true,
"bytes_read": 2109
},
{
"name": "tomcat-findsecurityconstraints-8bafd79",
"ok": true,
"bytes_read": 18199
},
{
"name": "vm2-makebuiltinsfromlegacyoptions-b0f5066",
"ok": true,
"bytes_read": 3539
},
{
"name": "inlong-filtersensitive-ebcd6d3",
"ok": true,
"bytes_read": 6474
},
{
"name": "h2o-3-validatejdbcurl-0298ee3",
"ok": true,
"bytes_read": 4919
},
{
"name": "skops-load-model-29d61ea",
"ok": true,
"bytes_read": 1847
},
{
"name": "picklescan-list-globals-58983e1",
"ok": true,
"bytes_read": 6975
},
{
"name": "sglang-deserialize-49afb3d",
"ok": true,
"bytes_read": 4334
},
{
"name": "pdfminer-six-load-data-b808ee0",
"ok": true,
"bytes_read": 2399
},
{
"name": "reverb-handle-9ec26f8",
"ok": true,
"bytes_read": 3342
},
{
"name": "transformers-is-kernel-a7f8e7f",
"ok": true,
"bytes_read": 1921
},
{
"name": "opentelemetry-java-instrumentation-read-9cf4fba",
"ok": true,
"bytes_read": 2075
},
{
"name": "keras-from-config-55888d3",
"ok": true,
"bytes_read": 3154
},
{
"name": "airflow-handle-event-submit-a1ab065",
"ok": true,
"bytes_read": 4828
},
{
"name": "sequelize-partialescape-5deadd2",
"ok": true,
"bytes_read": 6826
},
{
"name": "electerm-mv-aa77881",
"ok": true,
"bytes_read": 1258
},
{
"name": "systeminformation-checklinuxdchpinterfaces-bbfddde",
"ok": true,
"bytes_read": 2208
},
{
"name": "prompty-buildagent-c27402d",
"ok": true,
"bytes_read": 3198
},
{
"name": "datamodel-code-generator-get-field-extras-17fc235",
"ok": true,
"bytes_read": 2471
},
{
"name": "spring-data-mongodb-bindablevaluefor-24d5afc",
"ok": true,
"bytes_read": 9895
},
{
"name": "spring-ai-emitlucenestring-2b20cfc",
"ok": true,
"bytes_read": 4037
},
{
"name": "aws-cdk-write-baa9e1d",
"ok": true,
"bytes_read": 1461
},
{
"name": "danger-js-localgetfileatsha-087a729",
"ok": true,
"bytes_read": 1724
},
{
"name": "argos-javascript-gitfetch-8355f3a",
"ok": true,
"bytes_read": 1319
},
{
"name": "cakephp-transformfunctionexpression-138f2f6",
"ok": true,
"bytes_read": 6754
},
{
"name": "orval-getroutepath-8ef1bfd",
"ok": true,
"bytes_read": 2607
},
{
"name": "typeorm-handler-41d1c62",
"ok": true,
"bytes_read": 11977
},
{
"name": "velocity-js-isblockedpathkey-f8e47a6",
"ok": true,
"bytes_read": 1539
},
{
"name": "jsonata-evaluategroupexpression-4b217d5",
"ok": true,
"bytes_read": 7399
},
{
"name": "nuxt-getislandcontext-5b60017",
"ok": true,
"bytes_read": 6805
},
{
"name": "gitpython-option-candidates-96a888f",
"ok": true,
"bytes_read": 3421
},
{
"name": "mermaid-addnamespace-12d472c",
"ok": true,
"bytes_read": 9722
},
{
"name": "php-codesniffer-getblamecontent-7a3a6bb",
"ok": true,
"bytes_read": 2040
},
{
"name": "xmldom-serializetostring-1cde3e3",
"ok": true,
"bytes_read": 23297
},
{
"name": "trix-fromjson-9c0a993",
"ok": true,
"bytes_read": 1295
},
{
"name": "oauthlib-prepare-token-revocation-request-c888359",
"ok": true,
"bytes_read": 4800
},
{
"name": "suneditor-submitsrc-9d43a5e",
"ok": true,
"bytes_read": 2319
},
{
"name": "mathlive-atomtomathml-5fe1c46",
"ok": true,
"bytes_read": 32529
},
{
"name": "genql-comment-2dbc80f",
"ok": true,
"bytes_read": 1633
},
{
"name": "angular-visitelementlike-417a407",
"ok": true,
"bytes_read": 8480
},
{
"name": "cyberchef-objtotable-85db3be",
"ok": true,
"bytes_read": 2556
},
{
"name": "mdc-isanchorlinkallowed-61d636c",
"ok": true,
"bytes_read": 1795
},
{
"name": "node-xhtml-purifier-node-21d461a",
"ok": true,
"bytes_read": 7486
},
{
"name": "orpc-init-daabded",
"ok": true,
"bytes_read": 6240
},
{
"name": "mkdocs-material-mountsearchsuggest-52fb6be",
"ok": true,
"bytes_read": 4493
},
{
"name": "jodit-isremovablenode-49a31f4",
"ok": true,
"bytes_read": 3921
},
{
"name": "prowler-transform-6db407e",
"ok": true,
"bytes_read": 5481
},
{
"name": "commonmark-parseattributes-dfcdf45",
"ok": true,
"bytes_read": 6437
},
{
"name": "svgo-fn-3db3ef3",
"ok": true,
"bytes_read": 6895
},
{
"name": "tiptap-mergeattributes-01d7af8",
"ok": true,
"bytes_read": 3690
},
{
"name": "nogginlessdom-matchfilesnapshot-785e6ac",
"ok": true,
"bytes_read": 3023
},
{
"name": "gradio-preprocess-97d541f",
"ok": true,
"bytes_read": 2581
},
{
"name": "gemini-bridge-resolve-path-8f3b85a",
"ok": true,
"bytes_read": 2006
},
{
"name": "libreoffice-convert-convertwithoptions-b78f17d",
"ok": true,
"bytes_read": 8912
},
{
"name": "atomic-agents-stack-do-get-ec474f4",
"ok": true,
"bytes_read": 3493
},
{
"name": "browse-mcp-downloadurl-5352a4a",
"ok": true,
"bytes_read": 5302
},
{
"name": "notebooklm-mcp-makeslug-13828d9",
"ok": true,
"bytes_read": 2103
},
{
"name": "langchain-nvidia-url-to-b64-string-b483c7c",
"ok": true,
"bytes_read": 2318
},
{
"name": "redocly-cli-iteratepathitems-5041204",
"ok": true,
"bytes_read": 4412
},
{
"name": "pip-filename-10dfb6b",
"ok": true,
"bytes_read": 1419
},
{
"name": "lib-get-db-path-d665042",
"ok": true,
"bytes_read": 3879
},
{
"name": "blender-mcp-download-polyhaven-asset-30a3308",
"ok": true,
"bytes_read": 35668
},
{
"name": "datasets-set-feature-f989ef9",
"ok": true,
"bytes_read": 4882
},
{
"name": "postcss-loadfile-7beca13",
"ok": true,
"bytes_read": 1712
},
{
"name": "vitest-configureserver-51edf2b",
"ok": true,
"bytes_read": 3287
},
{
"name": "next-js-getfilepath-968b9fc",
"ok": true,
"bytes_read": 2462
},
{
"name": "composer-install-53b8bb4",
"ok": true,
"bytes_read": 4127
},
{
"name": "vue-cli-mayproxy-72ba750",
"ok": true,
"bytes_read": 1203
},
{
"name": "h3-redirectback-459a1c6",
"ok": true,
"bytes_read": 2750
},
{
"name": "axios-setproxy-fb3befb",
"ok": true,
"bytes_read": 4403
},
{
"name": "rembg-s-command-07ad0d4",
"ok": true,
"bytes_read": 21204
},
{
"name": "http-server-getpreviousurl-2008fb6",
"ok": true,
"bytes_read": 4395
},
{
"name": "authlib-get-authorization-grant-3be0846",
"ok": true,
"bytes_read": 2631
},
{
"name": "java-client-overrideserverurl-2b9cd44",
"ok": true,
"bytes_read": 1382
},
{
"name": "n8n-mcp-validatewebhookurl-9639f75",
"ok": true,
"bytes_read": 13458
},
{
"name": "ufo-withoutbase-5cd9e67",
"ok": true,
"bytes_read": 2154
},
{
"name": "symfony-dogenerate-4b63c3a",
"ok": true,
"bytes_read": 16981
},
{
"name": "php-weasyprint-isoptionurl-9582dcf",
"ok": true,
"bytes_read": 1090
},
{
"name": "compliance-trestle-check-metadata-endpoints-d107cd1",
"ok": true,
"bytes_read": 2412
},
{
"name": "vllm-preprocess-audio-54503ec",
"ok": true,
"bytes_read": 4629
},
{
"name": "phpspreadsheet-webservice-7ef7b25",
"ok": true,
"bytes_read": 4441
},
{
"name": "frontmcp-dofetch-077201e",
"ok": true,
"bytes_read": 3737
},
{
"name": "ip-address-islinklocal-d03e960",
"ok": true,
"bytes_read": 1489
},
{
"name": "pyjwt-fetch-data-0a795b8",
"ok": true,
"bytes_read": 3385
}
],
"verified": 98,
"bytes_read": 526134
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+114 -67
View File
@@ -1,14 +1,22 @@
# Architecture # Architecture
How a scan works, how a finding earns its place, how severity and the project score are This page covers four things:
decided, and how the ruleset improves itself. State as of 2026-10-02; module paths are under
`src/openultrasast/`. - how a scan works;
- how a finding earns its place;
- how severity and the project score are decided;
- how the ruleset improves itself.
State as of 2026-10-02; module paths are under `src/openultrasast/`.
## The scan pipeline ## The scan pipeline
The CLI is thin; `HarnessRuntime` owns the lifecycle and writes an event trace for every stage The CLI is thin. `HarnessRuntime` owns the lifecycle. It writes an event trace for every stage
(`trace/events.jsonl`). The stages a mode requests are `MODE_STAGES` in `stages.py`: (`trace/events.jsonl`). `MODE_STAGES` in `stages.py` lists the stages each mode requests:
`quick` runs STATIC, `standard` adds MAP, `deep` adds REGRESS.
- `quick` runs STATIC;
- `standard` adds MAP;
- `deep` adds REGRESS.
``` ```
policy_load ─▶ ruleset_load ─▶ policy_check (fail loud on an ungoverned CWE) policy_load ─▶ ruleset_load ─▶ policy_check (fail loud on an ungoverned CWE)
@@ -28,14 +36,19 @@ policy_load ─▶ ruleset_load ─▶ policy_check (fail loud on an
``` ```
**The model layer** (`model/`, `cpg/`) is the core of `standard`. It builds one code property **The model layer** (`model/`, `cpg/`) is the core of `standard`. It builds one code property
graph per scan with Joern (`joern-parse`, then batched `joern --script` queries from graph (CPG: a graph of the program's syntax, control flow and data flow) per scan with Joern.
`cpg/queries/*.sc`) and dispatches each question to the arbiter of its spec type: taint It runs `joern-parse`, then batched `joern --script` queries from `cpg/queries/*.sc`. Each
(source to sink with no discharging sanitizer), dominance (does a guard govern the obligated question goes to the arbiter (a checker that decides one kind of question) for its spec type:
operation) and configuration (constant abstraction over a value). Taint facts ship for C,
Java, JavaScript, PHP and Python (`ruleset/semantic/*.toml`). Without Joern the layer records - taint (untrusted input reaching a dangerous call): source to sink with no discharging sanitizer;
`cpg_unavailable` and the scan is otherwise unchanged; the Docker image ships Joern and - dominance: does a guard govern the obligated operation;
`php-cli` (which the PHP frontend needs). The representations and techniques behind each stage - configuration: constant abstraction over a value.
are explained in [Detection techniques](detection-techniques.md).
Taint facts ship for C, Java, JavaScript, PHP and Python (`ruleset/semantic/*.toml`). Without
Joern, the layer records `cpg_unavailable` and the rest of the scan is unchanged. The Docker
image ships Joern and `php-cli`, which the PHP frontend needs. [Detection
techniques](detection-techniques.md) explains the representations and techniques behind each
stage.
The model layer reports on a rung ladder: The model layer reports on a rung ladder:
@@ -47,23 +60,29 @@ model_corroborated the graph found the shape; a judge agreed on the residual
suspicion proposed, nothing corroborated it suspicion proposed, nothing corroborated it
``` ```
The LLM (`model/pipeline.py`, `residual_question`) is asked only the residual question the graph The LLM answers only the residual question: the part the graph cannot settle
cannot settle, for a flow the graph already found; it can confirm or contradict, never create a (`model/pipeline.py`, `residual_question`). It is asked only about a flow the graph already
flow. When the graph resolved nothing, the enumerator's candidates (at most found. It can confirm or contradict that flow, but never create one. When the graph resolved
`MAX_JUDGED_CANDIDATES`, 8) are put to it, and anything it affirms is reported at `suspicion`. nothing, the LLM sees the enumerator's candidates (at most `MAX_JUDGED_CANDIDATES`, 8).
Without a provider key neither question is asked and only the graph's entailed findings are Anything it affirms is reported at `suspicion`. Without a provider key, neither question is
reported. asked. Only the graph's entailed findings are reported then.
**REGRESS** (`regress/`, `sandbox/`) runs only when `docker info` succeeds. Each promoted **REGRESS** (`regress/`, `sandbox/`) runs only when `docker info` succeeds. Each promoted
candidate gets a small language recipe (Python, JavaScript, C) that loads the candidate's candidate gets a small language recipe (Python, JavaScript, C). The recipe loads the
source inside a container with no network, a read-only `/workspace`, a non-root user and the candidate's source inside a container with:
`[sandbox]` limits. A structural deny-list rejects snippets mentioning sockets, curl, host
networking or writes under `/workspace` before anything runs. - no network;
- a read-only `/workspace`;
- a non-root user;
- the `[sandbox]` limits.
Before anything runs, a structural deny-list rejects snippets that mention sockets, curl, host
networking or writes under `/workspace`.
## Evidence ladder: how a false positive is eliminated ## Evidence ladder: how a false positive is eliminated
Evidence level is a state machine, not a label a model may assign. The verifier Evidence level is a state machine, not a label a model may assign. The verifier
(`verification.py`) runs on independent context and enforces: (`verification.py`) runs on independent context and enforces it:
``` ```
suspicion ─▶ static_corroboration ─▶ crash_reproduced ─▶ root_cause_explained suspicion ─▶ static_corroboration ─▶ crash_reproduced ─▶ root_cause_explained
@@ -75,31 +94,36 @@ suspicion ─▶ static_corroboration ─▶ crash_reproduced ─▶ root_cause_
call-graph, route, CLI, parser or dynamic evidence; call-graph, route, CLI, parser or dynamic evidence;
- `static_corroboration` with function-level reachability: `ACCEPTED`. - `static_corroboration` with function-level reachability: `ACCEPTED`.
A rejected finding becomes a **scoped false-positive learning** (`calibration.py`) with a A rejected finding becomes a **scoped false-positive learning** (`calibration.py`). It has a
reason from a fixed taxonomy (`unreachable_path`, `missing_attacker_control`, scope and a reason from a fixed taxonomy: `unreachable_path`, `missing_attacker_control`,
`sanitizer_disproved`, `static_rule_mismatch`, `incorrect_model_assumption`, `duplicate`, `sanitizer_disproved`, `static_rule_mismatch`, `incorrect_model_assumption`, `duplicate`,
`insufficient_impact`, `unsupported`, `contradicted`, `unverified`) and a scope. A rejection in `insufficient_impact`, `unsupported`, `contradicted`, `unverified`. A rejection in `auth/`
`auth/` demotes only `auth/...`, never the whole class demotes only `auth/...`, never the whole class
(`tests/test_calibration.py::test_scoped_false_positive_demotion_does_not_suppress_class_globally`). (`tests/test_calibration.py::test_scoped_false_positive_demotion_does_not_suppress_class_globally`).
## How a false negative is surfaced ## How a false negative is surfaced
Every expected vulnerability that no finding matched becomes a `BenchmarkCalibrationRecord` in An expected vulnerability that no finding matched becomes a `BenchmarkCalibrationRecord` in
`calibration_records.json`, with a `next_improvement_candidate` (a static rule, SARIF source, `calibration_records.json`. Each record names a `next_improvement_candidate`: a static rule,
entry-point mapping, retrieval package, hunter prompt, dynamic reproducer or skill route). SARIF source, entry-point mapping, retrieval package, hunter prompt, dynamic reproducer or
Once a miss is closed the detection gate keeps it closed in CI. skill route. Once a miss is closed, the detection gate keeps it closed in CI.
## Central CWE policy and the project score ## Central CWE policy and the project score
**Policy** (`policy/verycode.py`): a vendored `CWE_Score.tsv` (verycode-policies) is the single **Policy** (`policy/verycode.py`): a vendored `CWE_Score.tsv` (verycode-policies) is the single
source of truth. Each CWE carries a flaw category, a severity (0-5) and `static`/`dynamic` scope source of truth. Each CWE carries:
flags. A finding's severity comes only from policy, keyed on CWE. CWEs that are not `static`
resolve to 0 and are report-only. `assert_rules_resolve` runs as the `policy_check` stage: an - a flaw category;
enabled rule naming an ungoverned CWE aborts the scan. Verycode has no CWE-120, so the C/C++ - a severity (0-5);
- `static`/`dynamic` scope flags.
A finding's severity comes only from policy, keyed on CWE. CWEs that are not `static` resolve
to 0 and are report-only. `assert_rules_resolve` runs as the `policy_check` stage. An enabled
rule that names an ungoverned CWE aborts the scan. Verycode has no CWE-120, so the C/C++
buffer rules carry CWE-121. buffer rules carry CWE-121.
**Score** (`scoring/project_score.py`): each finding's penalty is **Score** (`scoring/project_score.py`): each finding's penalty is
`SEV_WEIGHT[severity] x REACH_MULT[reachability]`, and the score is `100 * e^(-total/k)` with `SEV_WEIGHT[severity] x REACH_MULT[reachability]`. The score is `100 * e^(-total/k)` with
`k = 60`. `k = 60`.
| Severity | Weight | | Reachability | Multiplier | | Severity | Weight | | Reachability | Multiplier |
@@ -109,10 +133,13 @@ buffer rules carry CWE-121.
| 3 | 10 | | `unknown` | 0.4 | | 3 | 10 | | `unknown` | 0.4 |
| 2 / 1 / 0 | 2 / 1 / 0 | | | | | 2 / 1 / 0 | 2 / 1 / 0 | | | |
A confirmed false positive lowers a finding's reachability multiplier instead of deleting the A confirmed false positive lowers a finding's reachability multiplier. The rule is not
rule. A severity-5 reachable finding always fails the score gate; `score < min_score` deleted. The score gate works like this:
(default 80) fails it only with `[score] blocking = true`. The `score` stage writes
`score.json` and merges it into `manifest.json`. - a severity-5 reachable finding always fails it;
- `score < min_score` (default 80) fails it only with `[score] blocking = true`.
The `score` stage writes `score.json` and merges it into `manifest.json`.
## Self-improvement ## Self-improvement
@@ -124,41 +151,61 @@ scan ─▶ verify ─▶ non-accepted findings ─▶ scoped learnings
next scan ─▶ rank ─▶ calibrate (demote rejected scopes) ─▶ findings next scan ─▶ rank ─▶ calibrate (demote rejected scopes) ─▶ findings
``` ```
`record_calibration` (after `verify`) merges this run's non-accepted outcomes into the ledger, 1. `record_calibration` runs after `verify`. It merges this run's non-accepted outcomes into
de-duplicated by finding ID; `calibrate` (after `rank`) demotes the priority of every scope the ledger, de-duplicated by finding ID.
that produced a rejection and writes `applied_calibrations.json`. Demotion is scoped and 2. `calibrate` runs after `rank`. It demotes the priority of every scope that produced a
reversible (`tests/test_pipeline_calibration.py`). rejection and writes `applied_calibrations.json`.
Demotion is scoped and reversible (`tests/test_pipeline_calibration.py`).
### Rule-level loop: `ousast improve` ### Rule-level loop: `ousast improve`
Each round (`improve/evolve.py`): benchmark with the current ledger, per-rule signals Each round (`improve/evolve.py`) runs these steps:
(`rule_signals.json`), bounded proposals, validation (`improve/validator.py`), replay smoke,
re-benchmark, and a hard gate. The levers are rule status (`enabled -> shadow` before 1. benchmark with the current ledger;
`disabled`) and score constants; the loop can never edit pattern text 2. per-rule signals (`rule_signals.json`);
or the authoritative CWE severity. A round is accepted only if recall >= 90%, FP < 10%, the 3. bounded proposals;
project score did not regress and no matched finding was lost; otherwise the ledger is not 4. validation (`improve/validator.py`);
written. With `--pair-catalog`, every round is also gated per provenance profile on the 5. replay smoke;
catalog's holdout pairs (`--no-pair-gate` skips this and is not for accepted ledgers). The same 6. re-benchmark;
detection gate runs in CI (`python -m openultrasast.gate`). 7. a hard gate.
The levers are rule status (`enabled -> shadow` before `disabled`) and score constants. The
loop can never edit pattern text or the authoritative CWE severity. A round is accepted only if
all of these hold:
- recall >= 90%;
- FP < 10%;
- the project score did not regress;
- no matched finding was lost.
Otherwise the ledger is not written. With `--pair-catalog`, each round is also gated per
provenance profile on the catalog's holdout pairs. `--no-pair-gate` skips this check; do not
use it for accepted ledgers. The same detection gate runs in CI (`python -m openultrasast.gate`).
### Proposals from plane memory ### Proposals from plane memory
With `--memory [STORE]`, `ousast improve` also reads the rows plane runs stored per repository With `--memory [STORE]`, `ousast improve` also reads the rows that plane runs stored per
and pin (`improve/memory.py`). Two deterministic rules turn them into rule-status proposals: a repository and pin (`improve/memory.py`). Two deterministic rules turn these rows into
rule whose alerts are repeatedly false across repositories is shadowed, and a shadow rule that rule-status proposals:
keeps hitting agreed, declared vulnerable sites is re-enabled. Rows from holdout pairs, from the
gated manifest's own cases and from any `--qualify-population` are dropped before any rule sees - a rule whose alerts are repeatedly false across repositories is shadowed;
them. Each proposal records the rows it came from (`memory_proposals.jsonl` next to the - a shadow rule that keeps hitting agreed, declared vulnerable sites is re-enabled.
journal) and goes through the same validator and gate.
Some rows are dropped before any rule sees them: rows from holdout pairs, from the gated
manifest's own cases and from any `--qualify-population`. Each proposal records the rows it
came from (`memory_proposals.jsonl` next to the journal). It then goes through the same
validator and gate.
### The loop as a plane Run ### The loop as a plane Run
`ousast plane workspaces <population> --validation-set ... --loop` appends the improvement loop `ousast plane workspaces <population> --validation-set ... --loop` appends the improvement loop
to a generated Run: `alerts` per case (the quick scan on the vulnerable and fixed pins), then to a generated Run. The Run holds `alerts` per case: the quick scan on the vulnerable and fixed
the steps `snapshot` (task `memory-snapshot`), `measure`, `propose` and `improve` (`plane/tasks/loop.py`), each model-free pins. Then come the steps `snapshot` (task `memory-snapshot`), `measure`, `propose` and
with a zero budget. The snapshot applies the same train-on-test guard before any row leaves the `improve` (`plane/tasks/loop.py`). Each step is model-free with a zero budget. The snapshot
store. A recorded live run: `benchmarks/measurements/2026-09-30-harnessx-removal-loop/`. applies the same train-on-test guard before any row leaves the store. A recorded live run:
`benchmarks/measurements/2026-09-30-harnessx-removal-loop/`.
For PHP, which quick mode covered with zero rules when that run was made, `ousast plane When that run was made, quick mode covered PHP with zero rules. So for PHP, `ousast plane
alerts-engine` produces a Run's `alerts` from the engine image on the host instead alerts-engine` produces a Run's `alerts` from the engine image on the host instead
(`benchmarks/measurements/2026-09-30-php-engine-alerts/`). (`benchmarks/measurements/2026-09-30-php-engine-alerts/`).
+256 -67
View File
@@ -1,51 +1,77 @@
# The learned decision engine (in development) # The learned decision engine (in development)
**Status 2026-10-02: in development, not adopted.** No scan, pre-push check or report uses it; **Status 2026-10-02: in development, not adopted.** No scan, pre-push check or report uses it.
every user-facing path is today's deterministic one until a compiled program is adopted. The Every user-facing path stays on today's deterministic one until a compiled program is adopted.
design is the `learned-decision-engine` specification (maintainer tooling); the code is in The design is the `learned-decision-engine` specification (maintainer tooling). The code is in
`src/openultrasast/learn/`. `src/openultrasast/learn/`.
## Why ## Why
Every detection gain before the engine was a hand edit tuned on the cases that exposed it, and Before the engine, every detection gain was a hand edit. Each edit was tuned on the cases that
engine tuning measured on population v1 did not transfer to v2. The reasoning, and the step-by-step exposed the miss. Engine tuning measured on population v1 did not transfer to v2. [Where we
chain that replaced hand edits, are on [Where we stand](where-we-stand.md#2-our-approach-to-detection-step-by-step). stand](where-we-stand.md#2-our-approach-to-detection-step-by-step) gives the reasoning and the
step-by-step chain that replaced hand edits.
## Design ## Design
- **Signals, not verdicts.** Every instrument (quick rules, the Joern engine, model sink and - **Signals, not verdicts.** Every instrument contributes a feature record per candidate
role classification, the plane's verify passes and agreement, repository facts such as `(path, function, family)`. The instruments are quick rules, the Joern engine, model sink and
callers) contributes a feature record per candidate `(path, function, family)`, including an role classification, the plane's verify passes and agreement, and repository facts such as
explicit state when it did not run or failed. None decides alone. callers. The record includes an explicit state when an instrument did not run or failed. No
instrument decides alone.
- **Labels from ground truth.** `ousast learn labels` derives labels only from sources on a - **Labels from ground truth.** `ousast learn labels` derives labels only from sources on a
fail-closed allow-list (`learn/sources.toml`): the vulnerable and fixed sides of pairs, the fail-closed allow-list (`learn/sources.toml`). Fail-closed means a source not on the list is
declared sites and fix ranges of spent populations, benign deltas and recorded adjudications. never used. The sources are:
- the vulnerable and fixed sides of pairs;
- the declared sites and fix ranges of spent populations;
- benign deltas;
- recorded adjudications.
"Rejected by the plane" is never a negative. Reserved populations (v3) are not on the list. "Rejected by the plane" is never a negative. Reserved populations (v3) are not on the list.
- **An AI classifier with local memory, compiled in house in the DSPy style.** The program is an - **An AI classifier with local memory, compiled in house in the DSPy style.** The program has
instruction, a few demonstrations and up to six retrieved labelled examples, then the an instruction, a few demonstrations and up to six retrieved labelled examples. Then come the
candidate's code excerpt and signals; the model (`deepseek-flash`) answers candidate's code excerpt and signals. The model (`deepseek-flash`) answers vulnerable / not
vulnerable / not vulnerable / unsure with a rationale, sampled `k` times, giving a score `s`. vulnerable / unsure with a rationale. It is sampled `k` times, which gives a score `s`.
Compilation bootstraps demonstrations and searches instructions on a compile split, scored by Compilation bootstraps demonstrations and searches instructions on a compile split. A
a pre-registered balanced Brier score (`learn/compile.toml`). pre-registered balanced Brier score (`learn/compile.toml`) scores the candidates. The Brier score
- **Retrieval** from the memory store: nearest neighbours by Gower distance on the signal measures how far predicted probabilities are from the true labels.
profile, re-ranked by cosine similarity of code embeddings (OpenAI `text-embedding-3-small` - **Retrieval** from the memory store works in three steps:
through OpenRouter, cached by excerpt hash), under a label/family/repository balance rule. An 1. find nearest neighbours by Gower distance (a distance over mixed feature types) on the
evaluation-boundary filter keeps a candidate's own repository group out of what it is shown. signal profile;
2. re-rank them by cosine similarity of code embeddings (OpenAI `text-embedding-3-small`
through OpenRouter, cached by excerpt hash);
3. apply a label/family/repository balance rule.
An evaluation-boundary filter keeps a candidate's own repository group out of what it is
shown.
- **Repository-grouped evaluation.** The compile split (25% of repository groups) is never - **Repository-grouped evaluation.** The compile split (25% of repository groups) is never
evaluated. The rest is evaluated by outer grouped 5-fold cross-validation: the Platt evaluated. The rest is evaluated by outer grouped 5-fold cross-validation:
calibration map is cross-fitted on the other folds and the operating points are chosen on the - the Platt calibration map (which turns the score into a probability) is cross-fitted on
other folds (nested). One compile per profile, not one per fold. Intervals are 95% the other folds;
repository-cluster bootstrap. - the operating points are chosen on the other folds (nested).
- **Two operating points.** BLOCK is offered only when calibration holds (slope CI contains 1,
intercept CI contains 0, ECE <= 0.05) and a fold reaches the required precision; otherwise the There is one compile per profile, not one per fold. Intervals are 95% repository-cluster
engine is ADVISORY only. An `unsure` answer can never BLOCK. bootstrap.
- **Two operating points.** BLOCK is offered only when both of these hold:
- calibration holds: slope CI contains 1, intercept CI contains 0, ECE <= 0.05;
- a fold reaches the required precision.
Otherwise the engine is ADVISORY only. An `unsure` answer can never BLOCK. ECE (expected
calibration error) is the average gap between predicted and observed rates.
## First measured slice (2026-10-01) ## First measured slice (2026-10-01)
Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts
only). Family **injection** only; memory of 724 static examples from all families; input profile only). The setup:
v1 with the engine signals and function length withheld (a leak audit found them separating the
sides of a pair by construction: `2026-10-01-decision-engine-leak-audit/`); framework priors off. - family **injection** only;
- memory of 724 static examples from all families;
- input profile v1, with the engine signals and function length withheld;
- framework priors off.
The signals were withheld because a leak audit found them separating the sides of a pair by
construction (`2026-10-01-decision-engine-leak-audit/`). AUC below is the chance the engine
ranks a real bug above a fixed one.
| Measure | Value (95% repository-cluster CI) | | Measure | Value (95% repository-cluster CI) |
| --- | --- | | --- | --- |
@@ -62,11 +88,17 @@ sides of a pair by construction: `2026-10-01-decision-engine-leak-audit/`); fram
## Second slice: six families (2026-10-02) ## Second slice: six families (2026-10-02)
Record: `benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json` (counts only). Record: `benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json` (counts only).
After the advisory-fix harvest the memory holds 904 static examples (724 before). One program was After the advisory-fix harvest, the memory holds 904 static examples (724 before).
compiled per evaluable family (injection recompiled, since the memory changed) and evaluated on
that family's own candidates, out of repository as above: compile split never evaluated, outer One program was compiled per evaluable family. Injection was recompiled, since the memory
grouped 5-fold, calibration cross-fitted, operating points nested, 95% repository-cluster changed. Each program was evaluated on its family's own candidates, out of repository as above:
bootstrap intervals (2,000 resamples); input profile v1, priors off, k = 5.
- compile split never evaluated;
- outer grouped 5-fold;
- calibration cross-fitted;
- operating points nested;
- 95% repository-cluster bootstrap intervals (2,000 resamples);
- input profile v1, priors off, k = 5.
| Family | Candidates (groups; positive) | Pooled AUC of `s` (95% CI) | Within-pair AUC (paired groups) | ADVISORY recall / precision | Canary agreement | | Family | Candidates (groups; positive) | Pooled AUC of `s` (95% CI) | Within-pair AUC (paired groups) | ADVISORY recall / precision | Canary agreement |
| --- | --- | --- | --- | --- | --- | | --- | --- | --- | --- | --- | --- |
@@ -77,45 +109,174 @@ bootstrap intervals (2,000 resamples); input profile v1, priors off, k = 5.
| path | 35 (21; 21) | 0.89 (0.79 - 0.97) | 0.83 (14) | 0.86 / 0.69 | 0.90 | | path | 35 (21; 21) | 0.89 (0.79 - 0.97) | 0.83 (14) | 0.86 / 0.69 | 0.90 |
| untrusted destination | 62 (37; 38) | 0.85 (0.76 - 0.94) | 0.72 (18) | 0.84 / 0.73 | 0.87 | | untrusted destination | 62 (37; 38) | 0.85 (0.76 - 0.94) | 0.72 (18) | 0.84 / 0.73 | 0.87 |
- **BLOCK is offered for no family.** Calibration holds only for output encoding (ECE 0.045), and - **BLOCK is offered for no family.** Calibration holds only for output encoding (ECE 0.045).
there no fold reaches the required precision (`precision_unreachable`: no fold reaches a Wilson There, no fold reaches the required precision (`precision_unreachable`: no fold reaches a
lower bound of 0.95). For the other five families calibration does not hold (ECE 0.071 to 0.119). Wilson lower bound of 0.95). For the other five families, calibration does not hold (ECE
- **The pooled AUC is the more favourable number.** Every family's pool includes candidates that 0.071 to 0.119).
carry only a positive label, with no fixed-side counterpart (7 to 19 per family; there are no - **The pooled AUC is the more favourable number.** Every family's pool includes candidates
single-label negatives), so the pooled AUC is inflated by positives that never had to be told that carry only a positive label, with no fixed-side counterpart (7 to 19 per family; there
apart from their own fix. The within-pair AUC over the paired groups, 0.72 to 0.93, is the are no single-label negatives). These positives never had to be told apart from their own
stricter reading (`paired_groups_check` in the record). fix, so they inflate the pooled AUC. The within-pair AUC over the paired groups, 0.72 to
0.93, is the stricter reading (`paired_groups_check` in the record).
- **Re-run agreement (canary, 30 candidates per family) is below 0.9 for three families:** - **Re-run agreement (canary, 30 candidates per family) is below 0.9 for three families:**
injection 0.77, access control 0.83, untrusted destination 0.87. injection 0.77, access control 0.83, untrusted destination 0.87.
- **One memory-size point for injection** (paired: the same program, candidates and folds, with a - **One memory-size point for injection.** This run is paired: the same program, candidates
retrieval memory of 724 vs 904 examples): AUC 0.76 vs 0.77, within the intervals. No injection and folds, with a retrieval memory of 724 vs 904 examples. AUC was 0.76 vs 0.77, within the
example was added, so the new examples reach an injection prompt only as contrast. intervals. No injection example was added. So the new examples reach an injection prompt
- **Spend:** $2.88 metered at list price for DeepSeek under an $8 ceiling, $0.0009 for only as contrast.
embeddings; $0.0027 to $0.0032 per evaluated candidate. The access-control evaluation replayed - **Spend:** $2.88 metered at list price for DeepSeek under an $8 ceiling, and $0.0009 for
from the response cache at $0 with identical predictions. embeddings. That is $0.0027 to $0.0032 per evaluated candidate. The access-control
- **An instrument fix is recorded:** compiled demonstrations ignored the near-duplicate rule evaluation replayed from the response cache at $0 with identical predictions.
(cosine >= 0.98) until the pre-call boundary assertion stopped the untrusted-destination - **An instrument fix is recorded.** Compiled demonstrations ignored the near-duplicate rule
evaluation; demonstrations are now swapped or dropped like retrieved examples, and the stopped (cosine >= 0.98). The pre-call boundary assertion caught this and stopped the
part was replayed after the fix. untrusted-destination evaluation. Demonstrations are now swapped or dropped like retrieved
examples. The stopped part was replayed after the fix.
**Not adopted.** What is still ahead is listed once, on **Not adopted.** [Where we stand](where-we-stand.md#5-open-items) lists, in one place, what is
[Where we stand](where-we-stand.md#5-open-items). still ahead.
## Experiments ## Experiments
Changes to prompts, models, pass counts, features or sampling are compared by controlled Changes to prompts, models, pass counts, features or sampling are compared by controlled
experiments from a pre-registered manifest (`plane/experiments/<id>.yaml`), so nothing is adopted experiments. Each experiment comes from a pre-registered manifest
on one run's number. Two are registered. exp-001 (known callers) has not run. exp-002 (a retrieval (`plane/experiments/<id>.yaml`). This way nothing is adopted on one run's number.
ensemble against today's temperature sampling) ran on 160 paired candidates in 56 repository
groups and was **rejected**: within-pair AUC 0.815 against 0.741 and $0.0021 against $0.0051 per Four experiments are registered:
candidate, with re-run agreement 0.875 against 0.906 and an interval spanning zero
(`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`). The reading is on - exp-001 (known callers) has not run.
[Where we stand](where-we-stand.md#2-our-approach-to-detection-step-by-step). - exp-002 compared a retrieval ensemble against today's temperature sampling. It ran on 160
paired candidates in 56 repository groups and was **rejected**. Within-pair AUC was 0.815
against 0.741 and the cost was $0.0021 against $0.0051 per candidate. The record shows re-run agreement
0.875 against 0.906 and an interval spanning zero
(`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`).
- exp-003 compared two BLOCK rules on the same scores. It was **inconclusive**. Neither rule flags anything.
- exp-004 tested one pooled BLOCK threshold with a per-family floor. It was **inconclusive**.
### exp-003: precision-bound blocking (2026-10-03)
The question: is the calibration gate what keeps BLOCK out of reach?
Arm A is today's rule: a calibrated probability, offered only when calibration holds.
Arm B picks a raw-score threshold per family, on training folds only.
It takes the lowest threshold whose Wilson 95% lower bound on precision reaches 0.95.
The held-out repository group is then scored with that threshold.
Arm C relaxes the bound to point precision 0.95 over at least 10 flags.
Arm C was registered as secondary and does not decide.
- **Registered first.** The manifest and units were committed and registered before the run.
- **Primary metric:** arm B's held-out Wilson lower bound on flagged precision, per family.
- **Primary population:** paired units only. Positives-only groups are excluded; pooled figures are secondary.
- **Zero cost.** Scores came only from the response cache, behind a zero-ceiling meter.
The meter recorded 0 client calls and $0.0.
- **Exclusions.** 142 of 353 units had an uncached response and were excluded, not asked.
Path lost all 39 units. Injection and access control lost none.
- **Evaluated:** 211 units in 89 repository groups, 138 of them paired.
| Family | Evaluated (excluded) | Paired positives | Flagged A / B / C | Coverage A / B / C |
| --- | --- | --- | --- | --- |
| access control | 41 (0) | 16 | 0 / 0 / 0 | 0.0 / 0.0 / 0.0 |
| deserialization | 14 (34) | 2 | 0 / 0 / 0 | 0.0 / 0.0 / 0.0 |
| injection | 119 (0) | 38 | 0 / 0 / 0 | 0.0 / 0.0 / 0.0 |
| output encoding | 24 (21) | 8 | 0 / 0 / 0 | 0.0 / 0.0 / 0.0 |
| path | 0 (39) | 0 | 0 / 0 / 0 | none |
| untrusted destination | 13 (48) | 4 | 0 / 0 / 0 | 0.0 / 0.0 / 0.0 |
No arm chose a threshold on any fold, so flagged precision is undefined everywhere.
Arm A failed calibration where it was checked: ECE 0.077 (access control), 0.1067 (injection).
The other families had too few units for a calibration check.
**Why B flags nothing.** A Wilson lower bound of 0.95 needs 73 flags without one error.
Injection, the largest family, holds 38 paired positives across all five folds.
The registration predicted this zero before the run.
On this data the bound, not the calibration gate, makes BLOCK unreachable.
**The zero is the rule's, not the instrument's.** The scores are real: 48 distinct values for injection.
In access control, the 10 highest-scored paired units are all positive.
In injection, a negative appears at rank 4 of the paired ranking.
Pooled rankings look cleaner because positives-only units fill the top.
Injection answered `unsure` for 42 of 119 units, and `unsure` never blocks.
Record: `benchmarks/experiments/exp-003-precision-bound-blocking/result.json`.
**Supplementary: all 353 units (2026-10-03).** This is not a new decision; exp-003's verdict stands.
The 142 excluded units had their responses filled with the same programs and request keys.
The fill cost 697 client calls and $0.331759 under a $1.00 ceiling.
The replay then decided all 353 units at zero cost: 244 paired, 146 repository groups.
| Family | Paired units (positives) | Flagged A / B / C | ECE (arm A) |
| --- | --- | --- | --- |
| access control | 32 (16) | 0 / 0 / 0 | 0.077 |
| deserialization | 34 (17) | 0 / 0 / 0 | 0.0596 |
| injection | 76 (38) | 0 / 0 / 0 | 0.1067 |
| output encoding | 32 (16) | 0 / 0 / 0 | 0.0761 |
| path | 32 (16) | 0 / 0 / 0 | 0.062 |
| untrusted destination | 38 (19) | 0 / 0 / 0 | 0.0454 |
Still no arm flags any unit, so the registered rule reads inconclusive.
Calibration holds only for untrusted destination.
There arm A still finds no BLOCK point that reaches the bound.
Record: `benchmarks/experiments/exp-003-precision-bound-blocking/result-complete-cache.json`.
### exp-004: pooled block gate (2026-10-03)
The experiment tested one pooled BLOCK threshold across families.
Each family needed at least 10 clean flags to block.
Each family also had its own quiet threshold.
The run evaluated 501 units across 220 repository groups.
Of these, 390 units were paired.
The cache fill cost $1.238223 for 2,073 calls.
The replay cost $0.0 and made 0 client calls.
The verdict was **inconclusive**.
No fold found a `t_block` threshold.
The instrument check confirmed real scores.
The pooled top 30 contained 27 positives.
The first negative appeared at rank 12.
The bound needs 73 clean flags without an error.
A within-pair win means the vulnerable side scored above its fixed counterpart.
| Family | Within-pair wins | Pairs |
| --- | --- | --- |
| access control | 22 | 28 |
| deserialization | 18 | 25 |
| injection | 30 | 53 |
| output encoding | 18 | 28 |
| path | 25 | 29 |
| untrusted destination | 18 | 32 |
Precision at the top of the ranking now limits blocking.
Data volume alone cannot resolve this.
The next levers are graph evidence in the prompt and the combiner.
Records: `benchmarks/experiments/exp-004-pooled-block-gate/record.json` and `benchmarks/experiments/exp-004-pooled-block-gate/result.json`.
### exp-004: pooled block gate with a family floor (registered, not run)
This registration describes the setup before the run reported above.
The maintainer amended the BLOCK rule on 2026-10-03 (requirements, Req 6.4).
Precision is now shown across all families together, not per family.
Each family's score is first mapped by its own Platt fit, on training folds.
One threshold, t_block, then serves every family.
It is the lowest one whose pooled Wilson lower bound reaches 0.95 on training folds.
A family blocks only with at least 10 held-out flags and no error.
Otherwise its flags stay advisory.
Below a per-family t_quiet, which keeps 90% of positives, findings are silent.
No held-out label moves a threshold.
- **Primary metric:** the pooled held-out Wilson lower bound of flagged precision, paired units, before the floor.
- **Decision:** reject below point precision 0.95; adopt if the bound reaches 0.95 and a family passes the floor.
- **Units:** frozen from the memory after the negatives harvest.
- **Cost:** the run replayed the cache after a paid cache fill.
Manifest: `plane/experiments/exp-004-pooled-block-gate.yaml`.
The reading is on [Where we stand](where-we-stand.md#2-our-approach-to-detection-step-by-step).
## Commands ## Commands
`ousast learn` is a maintainer command group over the engine's data. Paid steps take `ousast learn` is a maintainer command group over the engine's data. Paid steps take
`--budget-usd`; `--replay-only` answers every call from the response cache at no cost. `--budget-usd`. `--replay-only` answers every call from the response cache at no cost.
| Command | Does | | Command | Does |
| --- | --- | | --- | --- |
@@ -127,8 +288,36 @@ candidate, with re-run agreement 0.875 against 0.906 and an interval spanning ze
| `ousast learn curve --program ID` | the metric over memory size on a fixed candidate subset | | `ousast learn curve --program ID` | the metric over memory size on a fixed candidate subset |
| `ousast learn audit-leaks` | how well each signal alone separates the sides of a pair (no model) | | `ousast learn audit-leaks` | how well each signal alone separates the sides of a pair (no model) |
| `ousast learn experiment register\|units\|run\|analyse MANIFEST` | A/B experiments from a pre-registered manifest (`plane/experiments/<id>.yaml`): record its digest, freeze the units (no model), run the arms paired (arm A may replay from the cache at $0), analyse (repository bootstrap, exact McNemar, two looks, the adoption verdict) | | `ousast learn experiment register\|units\|run\|analyse MANIFEST` | A/B experiments from a pre-registered manifest (`plane/experiments/<id>.yaml`): record its digest, freeze the units (no model), run the arms paired (arm A may replay from the cache at $0), analyse (repository bootstrap, exact McNemar, two looks, the adoption verdict) |
| `ousast learn experiment fill MANIFEST --budget-usd X` | Fill missing responses for a registered `kind: rule` experiment's frozen units before its zero-cost run; `--dry-run` inspects and estimates without calls or writes |
`experiment fill` accepts `--memory URL`, `--profile static|plane` (default `static`),
`--manifest-model PATH` (default `plane/models/deepseek-flash.yaml`) and `--out PATH`.
Freeze, commit and register the units first. Fill uses the incumbent programs and the
same retrieval, sampling and cache keys as replay. The first missing request is the
smoke call; a successful smoke response is kept. Any provider failure stops the fill.
The JSON report on stdout gives each family's units, distinct requests, initially
cached and missing responses, newly filled responses and still-missing responses,
plus total client calls and metered USD. `--out` also saves it, except during dry-run,
which writes nothing. Exit codes are 0 for completion or dry-run, 1 for a provider/cache
failure, 2 for invalid setup and 3 for budget exhaustion. Resume a partial fill with a
new invocation and budget; that budget covers only the new invocation, including smoke.
Dry-run uses each family's mean billed cached response cost when recorded usage exists.
Model prices account for cache-hit input, cache-miss input, and output tokens.
Without cached usage, it uses the program's input-token estimate and **300 output tokens**.
Each family's `estimate_method` reports `measured` or `estimated`.
It reports potential retries separately as `conditional_retries`: an unseen answer
may need one parse retry, whose necessity cannot be known without calling the model.
Known invalid cached answers include their retry requests in the inventory.
Fill reserves at least **$0.01** and the largest observed call cost before each call.
For the production client it also reserves a conservative input-byte token bound plus
the program's enforced output-token limit; hidden provider retries are disabled.
Consequently a small budget may stop before making any call, and a fill may finish
with unused budget. Calls without token usage stop as failures, never as free calls.
The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](ops/ax/README.md)). The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](ops/ax/README.md)).
The harvests that produced the plane signals ran as plane Runs generated by The harvests that produced the plane signals ran as plane Runs generated by
`ousast plane harvest` (`benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json` `ousast plane harvest`. Records: `benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json`
and, for the advisory-fix pairs, `2026-10-02-decision-engine-harvest-advisory/record.json`). and, for the advisory-fix pairs, `2026-10-02-decision-engine-harvest-advisory/record.json`.
+403 -244
View File
@@ -1,17 +1,36 @@
# Deployment # Deployment
How the agentic plane is deployed today, and what moving it to a separate Kubernetes cluster This page covers two things. First, how the agentic plane is deployed today. Second, what moving it to a
would take. State as of 2026-10-02 (v2.0.0). A scan needs none of this: `ousast scan` and separate Kubernetes cluster would take. It describes the state as of 2026-10-02 (v2.0.0).
`ousast pre-push` run from the Python package (and the Docker image for Joern, see
[Engine and pre-push](ops/README.md)). The plane is the maintainers' surface for model work over
whole repositories, driven by `ousast plane`.
Two words are used strictly on this page. **Implemented** means the code on `main` does it and it A scan needs none of this. `ousast scan` and `ousast pre-push` run from the Python package. Joern runs
has run on the maintainer's host. **Planned** means it has not been done; the guidance is derived from the Docker image (see [Engine and pre-push](ops/README.md)). The plane is the maintainers' tool for
from the code and the host's lessons, and the code changes it needs are named with their file paths. model work over whole repositories. `ousast plane` drives it.
This page uses two words strictly:
- **Implemented** means the code on `main` does it, and it has run on the maintainer's host.
- **Planned** means it has not been done. The guidance comes from the code and from what the host taught
us. Each code change it needs is named with its file path.
**No deployment of the plane to a cluster other than the maintainer's single-node kind cluster has **No deployment of the plane to a cluster other than the maintainer's single-node kind cluster has
been made.** Section B is guidance, not a record. been made.** Section B is guidance, not a record. (kind runs a Kubernetes cluster inside Docker
containers on one machine.)
The kind profile runs on a 7.7 GB server VM. Observability is off by default
(`kind_observability = false` in `ops/k8s/profiles/kind.toml`). This saves the measured
930 MiB used by `otel-system`, about 40% of the cluster's 2.3 GB. The smoke Run still
completes with those deployments stopped. To enable the stack, run
`OUSAST_KIND_OBSERVABILITY=1 ops/ax/up.sh` or set `kind_observability = true` in the profile.
`up.sh` stages copied Substrate manifests in a temporary checkout view and installs through
`hack/install-ate-kind.sh`. The local `ops/ax/substrate-kind-lean/` overlay removes the collector,
Prometheus, Jaeger and their supporting resources before installation. This preserves the
installer's secret, CRD and image setup. The upstream checkout is unchanged. The shared
`ate-otel-config` disables trace, metric and log export and the OTEL SDK; it removes the old
collector endpoint and metric timing keys. On an existing cluster, `up.sh` scales the three
observability deployments to zero, replaces the ConfigMap data and rolls its consumers when
the configuration changes. Opting in restores the upstream ConfigMap and observability manifests.
## What runs where today (implemented) ## What runs where today (implemented)
@@ -27,11 +46,11 @@ flowchart LR
sub["Agent Substrate (ate-system): atenet-router, atenet-egress"] sub["Agent Substrate (ate-system): atenet-router, atenet-egress"]
act["Task actors in gVisor workers (WorkerPool ousast-pool)"] act["Task actors in gVisor workers (WorkerPool ousast-pool)"]
svc["Service ousast-receiver + EndpointSlice to the host"] svc["Service ousast-receiver + EndpointSlice to the host"]
reg["Registry localhost:5001: runner image by digest"] reg["Registry named by the profile: runner image by digest"]
end end
mem[("Memory store: file store on the host, or an S3 bucket (RustFS tested)")] mem[("Memory store: file store on the host, or an S3 bucket (RustFS tested)")]
rec -- "ax apply, resume, get, delete" --> ax rec -- "ax apply, resume, get, delete" --> ax
rec -- "EgressPolicy per actor; start request via port-forward" --> sub rec -- "EgressPolicy per actor; start request through the kube-context tunnel" --> sub
sub --> act sub --> act
act -- "HTTP by Service name through the gateway" --> svc act -- "HTTP by Service name through the gateway" --> svc
svc --> rcv svc --> rcv
@@ -46,19 +65,26 @@ flowchart LR
| How actors reach the receiver | the Service `ousast-receiver.ax-system` (port 80) with a selector-less EndpointSlice whose endpoint is the kind network's gateway address on the host; the runner dials the ClusterIP with the Service name as `Host`, which the egress gateway allows by hostname | `ops/ax/receiver-service.yaml.tmpl`, rendered by `up.sh` | | How actors reach the receiver | the Service `ousast-receiver.ax-system` (port 80) with a selector-less EndpointSlice whose endpoint is the kind network's gateway address on the host; the runner dials the ClusterIP with the Service name as `Host`, which the egress gateway allows by hostname | `ops/ax/receiver-service.yaml.tmpl`, rendered by `up.sh` |
| ax control plane, Agent Substrate, egress gateway, router | the kind cluster (`ax-system`, `ate-system`) | installed by `ops/ax/up.sh` from the upstream checkouts under `~/.cache/ousast/ax-src/` | | ax control plane, Agent Substrate, egress gateway, router | the kind cluster (`ax-system`, `ate-system`) | installed by `ops/ax/up.sh` from the upstream checkouts under `~/.cache/ousast/ax-src/` |
| Tasks | one actor per Task in a gVisor worker; two warm workers of 1 CPU / 1.5 GiB | `ops/ax/workerpool.yaml.tmpl` | | Tasks | one actor per Task in a gVisor worker; two warm workers of 1 CPU / 1.5 GiB | `ops/ax/workerpool.yaml.tmpl` |
| Runner image | the kind-local registry `localhost:5001`, pinned by digest (`~/.cache/ousast/ax-src/runner-image`) | `plane/Dockerfile.runner`; the `image` of every template under `plane/tasks/` | | Runner image | the registry the profile names (`registry` in `ops/k8s/profiles/kind.toml`: the kind-local one), pinned by digest in the profile's images file (`ops/k8s/profiles/kind-images.json`) | `plane/Dockerfile.runner`; the `image` of every template under `plane/tasks/` |
| Provider keys | read from the operator's environment or `.env` (which never overrides an exported variable), sent only in the start request through a `kubectl port-forward` to `svc/atenet-router` | `src/openultrasast/plane/router.py` | | Provider keys | read from the operator's environment or `.env` (which never overrides an exported variable), sent only in the start request through the tunnel `router.py` opens to `svc/atenet-router` in the profile's kube context | `src/openultrasast/plane/router.py` |
| Memory store | `OUSAST_MEMORY`: a file store under `~/ousast-results/plane/memory` by default, or `s3://<bucket>[/<prefix>]` on any reachable S3-compatible server (RustFS is the tested one) | `src/openultrasast/plane/memory.py`, [Memory](memory.md), [RustFS setup](rustfs.md) | | Memory store | `OUSAST_MEMORY`: a file store under `~/ousast-results/plane/memory` by default, or `s3://<bucket>[/<prefix>]` on any reachable S3-compatible server (RustFS is the tested one) | `src/openultrasast/plane/memory.py`, [Memory](memory.md), [RustFS setup](rustfs.md) |
| Run state and artifacts | `$OUSAST_RESULTS/plane/<run>/` on the host (default `~/ousast-results/plane/`) | `reconciler.py` (`run_dir`) | | Run state and artifacts | `$OUSAST_RESULTS/plane/<run>/` on the host (default `~/ousast-results/plane/`) | `reconciler.py` (`run_dir`) |
Only the reconciler talks to the memory store: a sandboxed task cannot read it, so the loop's Terms in the table:
`memory-snapshot` is seeded by the reconciler before the Run ([The plane on ax](plane.md)).
- **ax** is Google's controller that runs agent tasks on Kubernetes.
- **Agent Substrate** is the layer below ax that owns the machines and runs sandboxes.
- **gVisor** is a sandbox runtime that isolates each actor from the node's kernel.
- An **actor** is one running Task inside a sandbox.
Only the reconciler talks to the memory store. A sandboxed task cannot read it. So the reconciler seeds
the loop's `memory-snapshot` before the Run starts ([The plane on ax](plane.md)).
## A. This host: the local setup (implemented) ## A. This host: the local setup (implemented)
What `ops/ax/up.sh` needs on the PATH: `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` (it `ops/ax/up.sh` needs these tools on the PATH: `docker`, `go`, `kubectl`, `kind`, `ko` and `ax`. It
refuses to start when one is missing). Tools live in `~/go/bin` and `~/.local/bin`; sources and refuses to start when one is missing. Tools live in `~/go/bin` and `~/.local/bin`. Sources and rendered
rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`). files live in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
1. **Bring up the cluster and the plane.** 1. **Bring up the cluster and the plane.**
@@ -66,25 +92,39 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
ops/ax/up.sh ops/ax/up.sh
``` ```
Idempotent: each step is skipped when already done. It creates the kind cluster The script is idempotent: it skips each step that is already done. In order, it:
(`KIND_CLUSTER_NAME`, default `ousast`) with its registry (`KO_DOCKER_REPO`, default
`localhost:5001`), installs Agent Substrate, applies the egress gateway (the agentgateway - creates the kind cluster (`KIND_CLUSTER_NAME`, default `ousast`) with its registry
variant, one prebuilt image) when the install left it out, deploys the ax control plane, applies (`KO_DOCKER_REPO`, default: the `registry` of `ops/k8s/profiles/kind.toml`);
the receiver Service with the host's kind-gateway address as its endpoint, applies the gVisor - installs Agent Substrate;
worker pool, builds and pushes the runner image and writes its digest pin to - applies the egress gateway if the install left it out (the agentgateway variant, one prebuilt image);
`~/.cache/ousast/ax-src/runner-image`, installs the `ax` CLI from the deployed checkout (a release - deploys the ax control plane;
CLI skews from the server), and runs one smoke Task. It ends by printing the host's memory and the - applies the receiver Service, with the host's kind-gateway address as its endpoint;
kind containers' usage. - applies the gVisor worker pool;
- builds and pushes the runner image, and writes its digest pin to `~/.cache/ousast/ax-src/runner-image`;
- installs the `ax` CLI from the deployed checkout (a release CLI does not match the server version);
- runs one smoke Task.
At the end it prints the host's memory and the kind containers' usage.
2. **Check it.** 2. **Check it.**
```bash ```bash
uv run ousast plane doctor uv run ousast plane doctor --profile kind
``` ```
Four checks: the kind context is reachable, every pod in `ate-system` and in `ax-system` is ready, It checks the profile you pass (`--profile`, default `kind`; the profiles are in `ops/k8s/profiles/`):
and the registry at `localhost:5001` holds `ousast-runner`. Each failing line names
`ops/ax/up.sh` as the fix. `doctor` is informational: `ousast plane run` does not call it. - its configured addresses, printed on the first line;
- the kube context is reachable;
- every pod in `ate-system` is ready;
- every pod in `ax-system` is ready;
- the runner image of the profile's images file is in its registry (a local registry over its v2
API, a remote one through `docker manifest inspect`);
- the memory store opens (an S3 bucket is verified).
Each failing line names the fix (`ops/ax/up.sh` for the kind profile). `doctor` is informational:
`ousast plane run` does not call it.
3. **Run the end-to-end smoke Run.** 3. **Run the end-to-end smoke Run.**
@@ -92,23 +132,27 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
ops/ax/smoke-run.sh ops/ax/smoke-run.sh
``` ```
It renders `ops/ax/smoke-run.yaml.tmpl` with the current runner digest and submits the Run It renders `ops/ax/smoke-run.yaml.tmpl` with the current runner digest. It then submits the Run
`ax-e2e-smoke` through the reconciler: one model-free `repo-facts` Task (`budget: {usd: 0, `ax-e2e-smoke` through the reconciler. The Run has one model-free `repo-facts` Task (`budget: {usd: 0,
calls: 0}`) over a files-only Workspace, resumed, started through the router, its `facts.json` and calls: 0}`) over a files-only Workspace, resumed, started through the router, its `facts.json`
`summary.json` delivered to the host, then `ousast plane status ax-e2e-smoke` prints the attribution and `summary.json` delivered to the host. Then `ousast plane status ax-e2e-smoke`
table. The script calls the repository's `.venv/bin/ousast`, so `uv sync` first. prints the attribution table. The script calls the repository's `.venv/bin/ousast`, so run `uv sync`
first.
4. **Keys and the store.** Put `DEEPSEEK_API_KEY` (chat) and `OPENROUTER_API_KEY` (embeddings) in 4. **Keys and the store.**
`.env` or export them; a Task's Model names the variable (`secretKey.key`), and the key travels
only in that Task's start request. Set `OUSAST_MEMORY=s3://<bucket>` with `S3_ENDPOINT`, - Put `DEEPSEEK_API_KEY` (chat) and `OPENROUTER_API_KEY` (embeddings) in `.env`, or export them.
`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (optional `S3_REGION`, `S3_BUCKET`) for the S3 store; - A Task's Model names the variable (`secretKey.key`). The key travels only in that Task's start
the bucket must be set up once by an admin as [RustFS setup](rustfs.md) describes, and the store request.
verifies it at every start. - For the S3 store, set `OUSAST_MEMORY=s3://<bucket>` with `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID` and
`AWS_SECRET_ACCESS_KEY`. `S3_REGION` and `S3_BUCKET` are optional.
- An admin must set up the bucket once, as [RustFS setup](rustfs.md) describes. The store verifies
it at every start.
5. **Run a real Run**, as in [Examples](examples.md#7-running-the-agentic-plane): `ousast plane run 5. **Run a real Run**, as in [Examples](examples.md#7-running-the-agentic-plane): `ousast plane run
plane/runs/validation-46.yaml`, then `status` and `remember`. plane/runs/validation-46.yaml`, then `status` and `remember`.
6. **Tear down** with `ops/ax/down.sh`, which deletes the cluster and its registry. 6. **Tear down** with `ops/ax/down.sh`. It deletes the cluster and its registry.
**Footprint** (measured 2026-09-29, idle, two workers; [ax on this host](ops/ax/README.md#measured-footprint-2026-09-29-idle-two-workers)): **Footprint** (measured 2026-09-29, idle, two workers; [ax on this host](ops/ax/README.md#measured-footprint-2026-09-29-idle-two-workers)):
@@ -119,44 +163,70 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
| node volume (images, containerd) | 4.9 GB on disk | | node volume (images, containerd) | 4.9 GB on disk |
| worker pool | 2 gVisor workers, 1 CPU / 1.5 GiB limit each | | worker pool | 2 gVisor workers, 1 CPU / 1.5 GiB limit each |
Disk is the binding constraint: the Substrate install and the `ko` builds need several GB of Go Disk is the tightest limit, for two reasons:
caches (`go clean -cache -modcache` afterwards), and every Task is its own actor template with a
golden actor (about 24 MB each under `/var/lib/ate/actors` on the node). The failures the host - The Substrate install and the `ko` builds need several GB of Go caches. Run `go clean -cache -modcache`
produced, and what each one taught, are listed in afterwards.
[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run). - Every Task is its own actor template with a golden actor (a saved snapshot the actor boots from). Each
takes about 24 MB under `/var/lib/ate/actors` on the node.
[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run) lists the
failures the host produced and what each one taught.
## B. A separate Kubernetes cluster (planned) ## B. A separate Kubernetes cluster (planned)
The target: a cluster that runs Agent Substrate with its egress gateway and the ax control plane, The target cluster would do four things:
pulls the runner image from a registry of yours, receives task artifacts without a developer
laptop in the path, and keeps its memory in your S3 bucket. The provider key already travels only - run Agent Substrate with its egress gateway and the ax control plane;
in the start request through `atenet-router`, which works the same through a port-forward to any - pull the runner image from a registry of yours;
cluster. Everything else below is either a step you take or a code change that is not made yet. - receive task artifacts without a developer laptop in the path;
- keep its memory in your S3 bucket.
The provider key already travels only in the start request through `atenet-router`. That works the same
through the kube-context tunnel to any cluster. Everything else below is either a step you take or a code
change that is not made yet.
### B.1 Cluster with Agent Substrate, its egress gateway and ax ### B.1 Cluster with Agent Substrate, its egress gateway and ax
- Install Agent Substrate on the cluster following its own instructions. `ops/ax/up.sh` uses - Install Agent Substrate on the cluster with its own instructions. `ops/ax/up.sh` uses Substrate's kind
Substrate's kind helpers (`hack/create-kind-cluster.sh`, `hack/install-ate-kind.sh`), which do helpers (`hack/create-kind-cluster.sh`, `hack/install-ate-kind.sh`). These do not apply to another
not apply to another cluster. cluster.
- Apply the egress gateway. On this host it is the agentgateway variant - Apply the egress gateway. On this host it is the agentgateway variant
(`manifests/ate-install/agentgateway-egress` in the Substrate checkout, applied with (`manifests/ate-install/agentgateway-egress` in the Substrate checkout). It is applied with
`kubectl kustomize --load-restrictor=LoadRestrictionsNone`), which needs no Rust build; the `kubectl kustomize --load-restrictor=LoadRestrictionsNone`. It needs no Rust build; the Envoy variant
Envoy variant does. Without the gateway an actor has no network at all, and the per-task does.
EgressPolicies the reconciler writes (`src/openultrasast/plane/egress.py`) work unchanged - Without the gateway an actor has no network at all.
against the Substrate-installed gateway: deny by default, hostnames only, plain HTTP to the - The reconciler writes one EgressPolicy per task (`src/openultrasast/plane/egress.py`). These work
receiver, TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts. unchanged against the gateway that Substrate installs. The rules are:
- deny by default;
- hostnames only;
- plain HTTP to the receiver;
- TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts.
- Deploy ax (`make deploy AX_IMAGE_REPO=<your registry>` in the ax checkout) **after** B.2. - Deploy ax (`make deploy AX_IMAGE_REPO=<your registry>` in the ax checkout) **after** B.2.
- Apply a WorkerPool (B.6). Substrate installs none by itself ("no free workers"). - Apply a WorkerPool (B.6). Substrate installs none by itself ("no free workers").
- **gVisor on the nodes (planned, checked by hand on the day k3s exists).** Substrate does not use a Kubernetes
`RuntimeClass`: the `ateom-gvisor` worker runs `runsc` itself, from the gVisor release tarball named in
`manifests/ate-install/sandboxconfig-gvisor.yaml` of the Substrate checkout
(`gs://gvisor/releases/nightly/2026-09-02/<arch>/gvisor.tar.zstd`, sha256-pinned per architecture), which
the atelet fetches on the worker node. The node must reach that URL, or a copied `SandboxConfig` must name a
mirrored copy with the same sha256 and the ActorTemplates must name that config. The check is the atelet's
fetch line in `kubectl -n ate-system logs` of the worker pod; it is labelled planned until that log has been
read on a k3s node (plane-on-kubernetes task 1.7, design open question 3).
### B.2 ax's snapshot bucket: `AX_SNAPSHOTS_BUCKET` ### B.2 ax's snapshot bucket: `AX_SNAPSHOTS_BUCKET`
ax's deploy manifest (`deploy/ax-server.yaml` in the ax checkout) sets `AX_SNAPSHOTS_BUCKET` to Point `AX_SNAPSHOTS_BUCKET` at **your own bucket before deploying ax**. ax's deploy manifest
the ax authors' own bucket (`gs://dberkov-gke-dev3/ate-env/` in the checkout used here). Point it (`deploy/ax-server.yaml` in the ax checkout) sets it to the ax authors' own bucket. In the checkout used
at **your own bucket before deploying ax**. On this host the bucket named there did not exist in here that is `gs://dberkov-gke-dev3/ate-env/`.
the S3 server, deleted golden actors piled up in `DELETING` and filled the disk during a 600-task
harvest (2026-10-01); creating the bucket let deletes complete, and even then a deleted actor's This host shows what goes wrong otherwise (2026-10-01):
directory stays on the node, so long Runs need a janitor that removes directories no live actor
owns. 1. The bucket named there did not exist in the S3 server.
2. Deleted golden actors piled up in `DELETING`.
3. They filled the disk during a 600-task harvest.
Creating the bucket let deletes complete. Even then, a deleted actor's directory stays on the node. So
long Runs need a janitor that removes directories no live actor owns.
### B.3 A registry the workers can pull from, the runner image pinned by digest ### B.3 A registry the workers can pull from, the runner image pinned by digest
@@ -168,83 +238,109 @@ owns.
docker inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' <registry>/ousast-runner:dev docker inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' <registry>/ousast-runner:dev
``` ```
- Substrate rejects tags: every Task's `image` must be the `<registry>/ousast-runner@sha256:...` - Substrate rejects tags. Every Task's `image` must be the `<registry>/ousast-runner@sha256:...`
reference. Write it into a file (on this host, `~/.cache/ousast/ax-src/runner-image`) and pass reference.
`--runner-image FILE` to `ousast plane workspaces` and `ousast plane harvest`, which re-pin the - Write that reference into a file. On this host the file is `~/.cache/ousast/ax-src/runner-image`.
Tasks they generate. The committed templates under `plane/tasks/` and the generated - Pass `--runner-image FILE` to `ousast plane workspaces` and `ousast plane harvest`. They re-pin the
`plane/tasks/validation-46.yaml` carry `localhost:5001/ousast-runner@sha256:...` and need the same Tasks they generate.
re-pin (regenerate, or replace the reference) before they run elsewhere. - The committed templates under `plane/tasks/` and the generated `plane/tasks/validation-46.yaml` carry
- Workers must be able to pull from the registry (credentials, if any, are the cluster's concern; the kind profile's registry and digest. They need the same re-pin before they run elsewhere.
ax's default runner image on `gcr.io` needs credentials and is not used here). Regenerate them, or replace the reference.
- **Not yet configurable:** `ousast plane doctor` checks the registry at `localhost:5001` by name - Workers must be able to pull from the registry. Credentials, if any, are the cluster's concern. ax's
(`src/openultrasast/plane/doctor.py:45`), so its fourth check fails on another cluster even when default runner image on `gcr.io` needs credentials and is not used here.
the image is in place. `doctor` is informational, `plane run` does not depend on it. - `ousast plane doctor --profile <name>` looks up the runner reference of the profile's images file in
the profile's `registry` (implemented 2026-10-02, plane-on-kubernetes task 1.2). `doctor` is
informational; `plane run` does not depend on it.
### B.4 One configurable kube context ### B.4 One configurable kube context (implemented 2026-10-02)
The reconciler, the egress client and the router port-forward all use the context The reconciler, the egress client and the router tunnel take the kube context from the profile.
`kind-$KIND_CLUSTER_NAME` (`kind-ousast` by default). **Not yet configurable**: the prefix `kind-`
is hard-coded in `src/openultrasast/plane/doctor.py:39`, `src/openultrasast/plane/egress.py:115` - The setting is `kube_context` in `ops/k8s/profiles/<name>.toml`; `OUSAST_KUBE_CONTEXT` overrides it
and `:148`, and `src/openultrasast/plane/router.py:145`; `ops/ax/up.sh:24` builds the same name. (plane-on-kubernetes task 1.2).
The planned change is one variable (for example `OUSAST_KUBE_CONTEXT`) read in those three - `doctor.py`, `egress.py` and `router.py` are handed the context. They build none themselves.
modules. Until then the only way to drive another cluster from the unchanged code is to give its - `tests/test_plane_literals.py` fails on any module, manifest or page that names this host's cluster,
kubeconfig context a name of that form (`kubectl config rename-context <ctx> kind-<name>` and registry or addresses.
`KIND_CLUSTER_NAME=<name>`), which is a workaround, not a supported configuration. The egress
client also needs `kubectl-ate` (or the executable `OUSAST_KUBECTL_ATE` names) able to use that The egress client also needs `kubectl-ate` (or the executable `OUSAST_KUBECTL_ATE` names). It must be
context. able to use that context.
### B.5 The receiver: in-cluster instead of on the host ### B.5 The receiver: in-cluster instead of on the host
Today the receiver is a thread inside the reconciler process (`egress.py`, `Receiver`), and the Today the receiver is a thread inside the reconciler process (`egress.py`, `Receiver`). The reconciler
reconciler learns of a task's completion by watching that object's `delivered` set in memory learns that a task finished by watching that object's `delivered` set in memory (`reconciler.py`).
(`reconciler.py`). Actors reach it through the Service `ousast-receiver.ax-system` whose Actors reach it through the Service `ousast-receiver.ax-system`. That Service's EndpointSlice points at
EndpointSlice points at the host. Three paths, in order of effort: the host.
1. **Interim, works with the unchanged code:** keep the host receiver and render There are three paths, from least to most effort:
`ops/ax/receiver-service.yaml.tmpl` with an endpoint address the egress gateway pod can reach
(`KIND_GATEWAY` is just an IPv4 address in the template, `OUSAST_ARTIFACT_PORT` the port). The 1. **Interim, works with the unchanged code:** keep the host receiver.
reconciler's host must be reachable from the cluster (a VPN or a routed address), and the Render `ops/ax/receiver-service.yaml.tmpl` with an endpoint address the egress gateway pod can reach.
gateway dials the address the actor connected to, so the Service's ClusterIP must still map to it. In the template, `KIND_GATEWAY` is just an IPv4 address and `OUSAST_ARTIFACT_PORT` is the port.
2. **Planned: the receiver as an in-cluster Deployment** behind the same Service name, with the The cluster must be able to reach the reconciler's host, through a VPN or a routed address.
reconciler reading deliveries and serving inputs from it instead of from its own thread. This The gateway dials the address the actor connected to. So the Service's ClusterIP must still map to it.
needs code: the receiver and its `delivered`/`inputs` state are in-process today, and the 2. **Planned: the receiver as an in-cluster Deployment** behind the same Service name.
runner's `GET /inputs/<producer>/<artifact>` is served by that same process. The reconciler would read deliveries from it and serve inputs from it, instead of from its own thread.
3. **Follow-on: presigned-URL delivery to the S3 store.** The runner would `PUT` its output tar to a This needs code. The receiver and its `delivered`/`inputs` state are in-process today.
presigned URL on the memory store's bucket and fetch inputs the same way, and the reconciler The same process serves the runner's `GET /inputs/<producer>/<artifact>`.
would read from the bucket. Nothing of this exists; the EgressPolicy would then have to allow 3. **Follow-on: presigned-URL delivery to the S3 store.** (A presigned URL grants one upload or download
the store's hostname for the task. without credentials.)
The runner would `PUT` its output tar to a presigned URL on the memory store's bucket.
It would fetch inputs the same way, and the reconciler would read from the bucket.
None of this exists. The EgressPolicy would then have to allow the store's hostname for the task.
### B.6 Worker pool sizing ### B.6 Worker pool sizing
`ops/ax/workerpool.yaml.tmpl` declares two warm gVisor workers of 1 CPU / 1.5 GiB (requests 250m / `ops/ax/workerpool.yaml.tmpl` declares two warm gVisor workers for the 7 GB host. Each has 1 CPU /
1.5 GiB) for the 7 GB host, selected onto nodes by `ate.dev/substrate-version`. An actor occupies a 1.5 GiB (requests 250m / 1.5 GiB). They are placed on nodes by `ate.dev/substrate-version`.
whole worker, so the pool size is the number of tasks that can run at once; set `--workers` on
`ousast plane run` to match. Size the workers above the largest Task's limits (`verify`: requests An actor occupies a whole worker. So the pool size is the number of tasks that can run at once. Set
250m / 384Mi, limits 1 CPU / 1 GiB; `roles` and the others are in `plane/tasks/`), and keep the `--workers` on `ousast plane run` to match.
golden-actor storage on the node in mind (B.2).
Size the workers above the largest Task's limits. For `verify` these are requests 250m / 384Mi and
limits 1 CPU / 1 GiB. `roles` and the others are in `plane/tasks/`. Also keep the golden-actor storage
on the node in mind (B.2).
### B.7 The S3 memory store reachable from where the reconciler runs ### B.7 The S3 memory store reachable from where the reconciler runs
The `s3://` store already works against any reachable S3-compatible server with versioning, The `s3://` store already works against any reachable S3-compatible server. The server must support:
lifecycle rules, object tags and S3 Select (RustFS is the tested one; the bucket setup and the
agent policy are on [RustFS setup](rustfs.md)). Only the reconciler reads and writes it, so today - versioning;
it must be reachable from the operator's host; if the reconciler moves into the cluster, from its - lifecycle rules;
pod. No Task talks to the store directly; the day one does (B.5, path 3), that Task's EgressPolicy - object tags;
must allow the store's hostname (the policies allow hostnames only, never addresses), the way a - S3 Select (a server-side query over JSON objects).
Model declares its hosts today through `openultrasast.io/egress-hosts`.
RustFS is the tested server. The bucket setup and the agent policy are on [RustFS setup](rustfs.md).
Only the reconciler reads and writes the store. So today the operator's host must reach it. If the
reconciler moves into the cluster, its pod must reach it.
No Task talks to the store directly. If one ever does (B.5, path 3), its EgressPolicy must allow the
store's hostname. The policies allow hostnames only, never addresses. A Model already declares its hosts
this way today, through `openultrasast.io/egress-hosts`.
### B.8 Secrets in Kubernetes Secrets, injected into the reconciler's environment ### B.8 Secrets in Kubernetes Secrets, injected into the reconciler's environment
The reconciler reads the variable a Model's `secretKey.key` names (`DEEPSEEK_API_KEY`, The reconciler reads these variables from its own environment or `.env`:
`OPENROUTER_API_KEY`) and the store's `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
(optional `AWS_SESSION_TOKEN`, `S3_REGION`, `S3_BUCKET`) from its own environment or `.env`. When - the variable a Model's `secretKey.key` names (`DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY`);
the reconciler runs in the cluster, keep them in Kubernetes Secrets and inject them into **the - the store's `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`;
reconciler's** pod environment (`envFrom` with `secretRef`, or `valueFrom.secretKeyRef` per - optionally `AWS_SESSION_TOKEN`, `S3_REGION` and `S3_BUCKET`.
variable). Never put a value into a manifest: the Model manifests name a variable, the rendered
Task carries no key, and the key reaches an actor only in the start request, where the runner keeps When the reconciler runs in the cluster, keep these in Kubernetes Secrets. Inject them into **the
it in memory and redacts it from echoed stderr. The `secretKey.name` values in `plane/models/` reconciler's** pod environment. Use `envFrom` with `secretRef`, or `valueFrom.secretKeyRef` per variable.
(`deepseek`, `openrouter`) are the names such Secrets would carry. Running the reconciler
in-cluster is itself planned: today it is a host process, and its state directory Never put a value into a manifest. The key stays out of every manifest:
- the Model manifests name a variable;
- the rendered Task carries no key;
- the key reaches an actor only in the start request;
- the runner keeps it in memory and redacts it from echoed stderr.
The `secretKey.name` values in `plane/models/` (`deepseek`, `openrouter`) are the names such Secrets
would carry.
Running the reconciler in-cluster is itself planned. Today it is a host process. Its state directory
(`OUSAST_RESULTS`) would need a persistent volume. (`OUSAST_RESULTS`) would need a persistent volume.
### What is implemented and what is planned ### What is implemented and what is planned
@@ -255,10 +351,10 @@ in-cluster is itself planned: today it is a host process, and its state director
| Per-task EgressPolicy, credentials only in the start request, per-task budgets and attribution | implemented | | Per-task EgressPolicy, credentials only in the start request, per-task budgets and attribution | implemented |
| S3 memory store against any reachable S3-compatible server, verified at startup | implemented (RustFS tested) | | S3 memory store against any reachable S3-compatible server, verified at startup | implemented (RustFS tested) |
| Local bring-up, doctor, smoke Run (`ops/ax/`) | implemented | | Local bring-up, doctor, smoke Run (`ops/ax/`) | implemented |
| Re-pinning generated Tasks to another registry (`--runner-image`) | implemented; the committed templates still name `localhost:5001` | | Re-pinning generated Tasks to another registry (`--runner-image`) | implemented; the committed templates name the kind profile's registry (`tests/test_plane_literals.py` checks it) |
| Deployment to a cluster other than this host's kind cluster | **not done** | | Deployment to a cluster other than this host's kind cluster | **not done** |
| One configurable kube context | **planned**; `kind-` hard-coded in `doctor.py`, `egress.py`, `router.py` | | One configurable kube context | implemented (2026-10-02): `kube_context` in the profile, `OUSAST_KUBE_CONTEXT` overrides |
| `doctor` registry check against a configurable registry | **planned**; `localhost:5001` hard-coded in `doctor.py:45` | | `doctor` registry check against a configurable registry | implemented (2026-10-02): the profile's `registry` and images file |
| Receiver as an in-cluster Deployment | **planned** (code change) | | Receiver as an in-cluster Deployment | **planned** (code change) |
| Presigned-URL delivery to the S3 store | **follow-on**, not started | | Presigned-URL delivery to the S3 store | **follow-on**, not started |
| Reconciler running in-cluster with Secrets injected into its environment | **planned** | | Reconciler running in-cluster with Secrets injected into its environment | **planned** |
@@ -266,12 +362,20 @@ in-cluster is itself planned: today it is a host process, and its state director
## Production topology: ax on a Kubernetes cluster ## Production topology: ax on a Kubernetes cluster
The maintainer's questions, answered from the code and the upstream checkouts under This section answers the maintainer's questions from the code and the upstream checkouts under
`~/.cache/ousast/ax-src/`: how is it hosted in ax, is there one container, how does ax relate to `~/.cache/ousast/ax-src/`:
Kubernetes, and what scaling the pre-push hook would take. The decisions of 2026-10-02 set the
target: **two profiles of one code base**, `kind` (this laptop, kept as a first-class local - How is it hosted in ax?
execution mode, where every change is proven first) and `k3s` (production). Everything marked - Is there one container?
**planned** is not built; the rest is what runs on this host today. - How does ax relate to Kubernetes?
- What would scaling the pre-push hook take?
The decisions of 2026-10-02 set the target: **two profiles of one code base**.
- `kind` is this server VM. It stays a first-class local execution mode. Every change is proven here first.
- `k3s` is production. (k3s is a small Kubernetes distribution.)
Everything marked **planned** is not built. The rest is what runs on this host today.
```mermaid ```mermaid
flowchart LR flowchart LR
@@ -279,7 +383,7 @@ flowchart LR
hook["ousast pre-push: local, 30 s deadline"] hook["ousast pre-push: local, 30 s deadline"]
cli["ousast plane run, remote mode (planned)"] cli["ousast plane run, remote mode (planned)"]
end end
subgraph k3s ["k3s cluster (the kind profile is the same picture on one laptop)"] subgraph k3s ["k3s cluster (the kind profile is the same picture on one server VM)"]
subgraph axns ["Namespace ax-system"] subgraph axns ["Namespace ax-system"]
axs["ax-server Deployment + Redis"] axs["ax-server Deployment + Redis"]
end end
@@ -287,7 +391,7 @@ flowchart LR
ctl["ate-api-server, ate-controller, atelet"] ctl["ate-api-server, ate-controller, atelet"]
rt["atenet-router"] rt["atenet-router"]
eg["atenet-egress gateway"] eg["atenet-egress gateway"]
wp["WorkerPool: gVisor workers, runsc RuntimeClass, HPA (planned)"] wp["WorkerPool: gVisor workers, runsc in the worker, HPA (planned)"]
ewp["Engine WorkerPool: Joern as a task (planned)"] ewp["Engine WorkerPool: Joern as a task (planned)"]
end end
end end
@@ -308,124 +412,173 @@ flowchart LR
cli -- "polls: status, attribution" --> s3 cli -- "polls: status, attribution" --> s3
``` ```
No inbound path exists in this picture: the CLI reaches ax-server and the router from outside, the This picture has no inbound path:
tasks push their artifacts to the store through the egress gateway, and the CLI reads the store.
Today's EndpointSlice back to a laptop (`ops/ax/receiver-service.yaml.tmpl`) is the kind profile's - the CLI reaches ax-server and the router from outside;
device and does not appear in production. - the tasks push their artifacts to the store through the egress gateway;
- the CLI reads the store.
Today's EndpointSlice back to the server VM (`ops/ax/receiver-service.yaml.tmpl`) belongs to the kind
profile only. It does not appear in production.
### ax, Kubernetes and Agent Substrate in three sentences ### ax, Kubernetes and Agent Substrate in three sentences
**ax is a Kubernetes application**: a Deployment `ax-server` with a Redis beside it in the **ax is a Kubernetes application.**
namespace `ax-system` (`ax/deploy/ax-server.yaml` in the ax checkout; its controller runs with
`--template=default-template --template-atespace=ax-system`), and the `ax` CLI reaches it through a
`kubectl port-forward` to `svc/ax-server` (`ax/internal/tunnel/tunnel.go`), so `ousast plane run`
needs a kube context or an ingress, not a public endpoint. **ax does not run containers itself**:
it turns every Task into an ActorTemplate on Agent Substrate (`ax/internal/substrate/client.go`,
`CreateActorTemplate`), built from the Task's image and environment, and the image must be pinned
by digest (Substrate's `SandboxConfig` validation rejects tags: "All images must include a
digest"). **Agent Substrate is the layer that owns the machines**: in the namespace `ate-system`
it runs `ate-api-server`, `ate-controller`, the `atelet` DaemonSet, `atenet-router`, the
`atenet-egress` gateway, a Postgres and a RustFS (`substrate/manifests/ate-install/`), and three
CRDs under `ate.dev/v1alpha1`, `WorkerPool`, `SandboxConfig` and `CSIDriverConfig`; a
`WorkerPool` keeps warm worker pods (`ko://.../cmd/ateom-gvisor`), and an actor is one gVisor
sandbox inside one of those workers, booted from the golden snapshot that
[The plane on ax](plane.md) describes, suspended and resumed by the controller. Names are limited
to 63 bytes (`MaxNameLength` in `ax/pkg/apis/v1alpha1/types.go`), which is why the generated Task
names are short.
**What k3s changes** (planned; nothing has been applied to a k3s cluster): k3s runs containerd - It is a Deployment `ax-server` with a Redis beside it, in the namespace `ax-system`
with no Docker socket, so the host-Docker path of `ousast plane alerts-engine` does not exist (`ax/deploy/ax-server.yaml` in the ax checkout).
there and the engine must become a task (below); Substrate's gVisor workers need `runsc` and a - Its controller runs with `--template=default-template --template-atespace=ax-system`.
`RuntimeClass` on the nodes they are scheduled to; k3s ships Traefik (the ingress the CLI can use - The `ax` CLI reaches it through a tunnel to `svc/ax-server` over the kube context
instead of a kubeconfig) and local-path storage. Images are built by GitHub Actions from tagged (`ax/internal/tunnel/tunnel.go`).
commits, published to GitHub Container Registry and digest-pinned in the manifests; a private - So `ousast plane run` needs a kube context or an ingress, not a public endpoint.
package needs an `imagePullSecret` on the worker ServiceAccount. Substrate's and ax's own images
are built with `ko` from their checkouts and pushed to the same registry. The kind profile keeps **ax does not run containers itself.**
`localhost:5001` and `ops/ax/up.sh`.
- It turns every Task into an ActorTemplate on Agent Substrate (`ax/internal/substrate/client.go`,
`CreateActorTemplate`).
- The ActorTemplate is built from the Task's image and environment.
- The image must be pinned by digest. Substrate's `SandboxConfig` validation rejects tags: "All images
must include a digest".
**Agent Substrate is the layer that owns the machines.**
- In the namespace `ate-system` it runs `ate-api-server`, `ate-controller`, the `atelet` DaemonSet,
`atenet-router`, the `atenet-egress` gateway, a Postgres and a RustFS
(`substrate/manifests/ate-install/`).
- It defines three CRDs (custom Kubernetes resource types) under `ate.dev/v1alpha1`: `WorkerPool`,
`SandboxConfig` and `CSIDriverConfig`.
- A `WorkerPool` keeps warm worker pods (`ko://.../cmd/ateom-gvisor`).
- An actor is one gVisor sandbox inside one of those workers.
- The actor boots from the golden snapshot that [The plane on ax](plane.md) describes. The controller
suspends and resumes it.
- Names are limited to 63 bytes (`MaxNameLength` in `ax/pkg/apis/v1alpha1/types.go`). That is why the
generated Task names are short.
**What k3s changes** (planned; nothing has been applied to a k3s cluster):
- k3s runs containerd with no Docker socket. So the host-Docker path of `ousast plane alerts-engine`
does not exist there. The engine must become a task (below).
- Substrate's gVisor workers need no Kubernetes `RuntimeClass`: the worker fetches a pinned gVisor
tarball and runs `runsc` inside its own pod. The node checklist is pod security, tarball
reachability and local-path storage (B.1).
- k3s ships Traefik and local-path storage. Traefik is an ingress the CLI can use instead of a kubeconfig.
- GitHub Actions builds the images from tagged commits. They are published to GitHub Container Registry
and digest-pinned in the manifests.
- A private package needs an `imagePullSecret` on the worker ServiceAccount.
- Substrate's and ax's own images are built with `ko` from their checkouts. They are pushed to the same
registry.
- The kind profile keeps its local registry (`ops/k8s/profiles/kind.toml`) and `ops/ax/up.sh`.
### One image, and what concurrency equals ### One image, and what concurrency equals
There is **one container image** for every Task: `plane/Dockerfile.runner` (`python:3.12-slim` There is **one container image** for every Task: `plane/Dockerfile.runner`. It is `python:3.12-slim`
plus this package, PID 1 `/usr/local/bin/ax-task-runner`, which is plus this package. PID 1 is `/usr/local/bin/ax-task-runner`, which is `openultrasast.plane.runner`.
`openultrasast.plane.runner`). Each Task manifest under `plane/tasks/` names the same image by
digest (the pin is kept in `~/.cache/ousast/ax-src/runner-image`) and selects its work with Each Task manifest under `plane/tasks/` names the same image by digest. The pin is kept in
`spec.command`: the runner starts `python -m openultrasast.plane.tasks.<command[0]>` `~/.cache/ousast/ax-src/runner-image`. The manifest picks its work with `spec.command`. The runner
(`runner.py`, `task_module`). `repo-facts`, `verify`, `agree`, `features`, `remember`, `alerts`, starts `python -m openultrasast.plane.tasks.<command[0]>` (`runner.py`, `task_module`).
`loop` and `roles` are modules in that image, not images of their own; the engine would be the
second image (planned). So the production answer to "how many containers" is: one image, one `repo-facts`, `verify`, `agree`, `features`, `remember`, `alerts`, `loop` and `roles` are modules in
actor per Task, and **an actor occupies a whole worker** (its limits are the ActorTemplate's, the that image. They are not images of their own. The engine would be the second image (planned).
worker holds one actor at a time). Concurrency is therefore the number of workers in the
`WorkerPool`, nothing else: `ops/ax/workerpool.yaml.tmpl` declares `replicas: 2` of 1 CPU / So the production answer to "how many containers" is:
1.5 GiB, so at most two Tasks run at once on this host, and `ousast plane run --workers` must not
exceed the pool. Substrate ships an autoscaled pool - one image;
(`substrate/demos/autoscaled-workerpool/README.md`: an HPA on the external metric - one actor per Task;
`ate_workerpool_workers{state=at_capacity}` through prometheus-adapter, writing - **an actor occupies a whole worker**. Its limits are the ActorTemplate's, and the worker holds one
`WorkerPool.spec.replicas`; its README says the demo is kind-only today) and request parking actor at a time.
(`substrate/demos/parking/`: the router holds a resume until a worker frees up). Neither is
applied by `ops/ax/up.sh`; the production profile's pool manifests and autoscaling rule are Concurrency is therefore the number of workers in the `WorkerPool`, nothing else.
`ops/ax/workerpool.yaml.tmpl` declares `replicas: 2` of 1 CPU / 1.5 GiB. So at most two Tasks run at
once on this host. `ousast plane run --workers` must not exceed the pool.
Substrate ships two scaling aids:
- **An autoscaled pool** (`substrate/demos/autoscaled-workerpool/README.md`). An HPA (Horizontal Pod
Autoscaler) reads the external metric `ate_workerpool_workers{state=at_capacity}` through
prometheus-adapter. It writes `WorkerPool.spec.replicas`. Its README says the demo is kind-only today.
- **Request parking** (`substrate/demos/parking/`). The router holds a resume until a worker frees up.
`ops/ax/up.sh` applies neither. The production profile's pool manifests and autoscaling rule are
**planned**. **planned**.
### The three components that are bound to the host today ### The three components that are bound to the host today
| Component | Today (the kind profile) | Production (the k3s profile, planned) | | Component | Today (the kind profile) | Production (the k3s profile, planned) |
| --- | --- | --- | | --- | --- | --- |
| Reconciler (`ousast plane run`, `src/openultrasast/plane/reconciler.py`) | a process on the operator's host; calls `ax` and `kubectl` against `kind-ousast`, holds the Run's state under `OUSAST_RESULTS` | **the reconciler stays the CLI.** An execution mode `local` or `remote`: in `remote`, the same `ousast plane run` from a laptop or a CI job submits the Run to ax-server in k3s (over the cluster's ingress or a kubeconfig), sends the start requests, then polls the store and prints the same `status` and attribution table. Credentials come from the caller's environment as today (CI secrets in a job; B.8's rule, no value in a manifest). An unattended in-cluster reconciler (a Deployment with a submit API, or a Job per Run) is a **later option**, not the first step | | Reconciler (`ousast plane run`, `src/openultrasast/plane/reconciler.py`) | a process on the operator's host; calls `ax` and `kubectl` against the profile's `kube_context`, holds the Run's state under `OUSAST_RESULTS` | **the reconciler stays the CLI.** An execution mode `local` or `remote`: in `remote`, the same `ousast plane run` from a laptop or a CI job submits the Run to ax-server in k3s (over the cluster's ingress or a kubeconfig), sends the start requests, then polls the store and prints the same `status` and attribution table. Credentials come from the caller's environment as today (CI secrets in a job; B.8's rule, no value in a manifest). An unattended in-cluster reconciler (a Deployment with a submit API, or a Job per Run) is a **later option**, not the first step |
| Artifact receiver (`plane/egress.py`, the `Receiver` thread inside the reconciler) | reached through the Service `ousast-receiver.ax-system` whose EndpointSlice points at the kind gateway address on the host (`ops/ax/receiver-service.yaml.tmpl`) | **no receiver in remote mode.** Tasks deliver their artifacts and fetch their inputs through presigned URLs on the S3 memory store (B.5, path 3; the store's hostname enters each Task's EgressPolicy, B.7), and the CLI reads completion from the store. The kind profile keeps the receiver thread; an in-cluster receiver Deployment (B.5, path 2) is not the chosen direction | | Artifact receiver (`plane/egress.py`, the `Receiver` thread inside the reconciler) | reached through the Service `ousast-receiver.ax-system` whose EndpointSlice points at the kind gateway address on the host (`ops/ax/receiver-service.yaml.tmpl`) | **no receiver in remote mode.** Tasks deliver their artifacts and fetch their inputs through presigned URLs on the S3 memory store (B.5, path 3; the store's hostname enters each Task's EgressPolicy, B.7), and the CLI reads completion from the store. The kind profile keeps the receiver thread; an in-cluster receiver Deployment (B.5, path 2) is not the chosen direction |
| Joern engine (`plane/engine_alerts.py`, the host's `openultrasast:dev` container, run with `--memory 3g`) | `ousast plane alerts-engine` runs it on the host, one container at a time, and marks the Run's `alerts` tasks done | **an `engine` task on a second, larger `WorkerPool`** (the engine does not fit a 1.5 GiB worker, and k3s has no Docker socket), from an engine image digest-pinned in GHCR, emitting the same `alerts.jsonl` rows and coverage; `alerts-engine` on the host stays as the development path | | Joern engine (`plane/engine_alerts.py`, the host's `openultrasast:dev` container, run with `--memory 3g`) | `ousast plane alerts-engine` runs it on the host, one container at a time, and marks the Run's `alerts` tasks done | **an `engine` task on a second, larger `WorkerPool`** (the engine does not fit a 1.5 GiB worker, and k3s has no Docker socket), from an engine image digest-pinned in GHCR, emitting the same `alerts.jsonl` rows and coverage; `alerts-engine` on the host stays as the development path |
The S3 memory store (RustFS tested) is already external; in remote mode it is also the artifact The S3 memory store (RustFS tested) is already external. In remote mode it is also the artifact path.
path, so it must be reachable from the CLI's host and, through the egress gateway, from the So the CLI's host must reach it, and the workers must reach it through the egress gateway.
workers.
### Scaling the pre-push hook in production ### Scaling the pre-push hook in production
**Today** `ousast pre-push` is a local, deterministic check: it materialises base and head, **Today** `ousast pre-push` is a local, deterministic check. It does four things:
discovers the changed regions, runs the evidence ranker and arbiter on head and compares with base,
all within `--deadline` (30 s by default) and in `--mode advisory` unless blocking is opted into 1. materialises base and head;
([Pre-push: the delta check](scanning.md#pre-push-the-delta-check-experimental), 2. finds the changed regions;
`src/openultrasast/push/`, [Engine and pre-push](ops/README.md#experimental-pre-push-integration)). 3. runs the evidence ranker and arbiter on head;
**The plane is not in that path**: no hook submits a Run, no model is called without an explicit 4. compares the result with base.
`--model-config`, and nothing leaves the developer's machine.
All of this runs within `--deadline` (30 s by default). It runs in `--mode advisory` unless you opt
into blocking. See [Pre-push: the delta check](scanning.md#pre-push-the-delta-check-experimental),
`src/openultrasast/push/` and [Engine and pre-push](ops/README.md#experimental-pre-push-integration).
**The plane is not in that path.**
- No hook submits a Run.
- No model is called without an explicit `--model-config`.
- Nothing leaves the developer's machine.
**The production shape** (planned, none of it built) splits the work by latency: **The production shape** (planned, none of it built) splits the work by latency:
1. **Blocking, inside the hook's deadline**: what runs today, the quick rules and the delta 1. **Blocking, inside the hook's deadline**: what runs today, the quick rules and the delta engine on
engine on the changed regions. It stays local and deterministic, so the push is never held for the changed regions. It stays local and deterministic. So the push never waits for a model or a
a model or a cluster. cluster.
2. **Asynchronous, on the plane**: the hook or the CI job calls the CLI in remote mode with a 2. **Asynchronous, on the plane**: the hook or the CI job calls the CLI in remote mode with a *delta
*delta Run* whose candidates are the functions the push changed; the Run is the per-case chain Run*.
of [The plane on ax](plane.md#per-case) (`facts`, `verify` a and b, `agree`, `verify` c on the - The Run's candidates are the functions the push changed.
disputed only, `final`). The caller either waits, with its own deadline, for the CLI to print - The Run is the per-case chain of [The plane on ax](plane.md#per-case): `facts`, `verify` a and b,
the status and attribution table, or returns at once and a later `ousast plane status <run>` `agree`, `verify` c on the disputed only, `final`.
reports from the store; a commit status check or a review comment would be built on that. The - The caller can wait, with its own deadline, for the CLI to print the status and attribution table.
worker pool is sized to the push rate or autoscaled on `at_capacity` workers. An unattended - Or the caller returns at once, and a later `ousast plane status <run>` reports from the store.
in-cluster reconciler that accepts submissions without a CLI session is the later option named - A commit status check or a review comment would be built on that.
above. - The worker pool is sized to the push rate, or autoscaled on `at_capacity` workers.
- An unattended in-cluster reconciler that accepts submissions without a CLI session is the later
option named above.
**Cost per push, estimated from measured rates.** Two measured per-candidate rates exist for one **Cost per push, estimated from measured rates.** Two measured per-candidate rates exist for one model
model verdict: `$0.003215` (`spend.cost_per_candidate_evaluate_usd` in verdict:
`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`) and `$0.0021`
(exp-002 arm A, `overall.usd_per_candidate.A` in - `$0.003215` (`spend.cost_per_candidate_evaluate_usd` in
`benchmarks/measurements/2026-10-02-exp-002/record.json`). A push that changes five functions needs `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`);
two verify passes per candidate: 5 x 2 x $0.0021 = $0.021, or 5 x 2 x $0.003215 = $0.032 at the - `$0.0021` (exp-002 arm A, `overall.usd_per_candidate.A` in
higher rate, plus a third pass on the disputed candidates only. That is where "about two cents per `benchmarks/measurements/2026-10-02-exp-002/record.json`).
push" comes from; it is arithmetic over the decision engine's measured rates, not a measurement of
a push Run, and `ousast plane status` would print the real figure per Task. A push that changes five functions needs two verify passes per candidate. That gives 5 x 2 x $0.0021 =
$0.021, or 5 x 2 x $0.003215 = $0.032 at the higher rate. A third pass runs on the disputed candidates
only. That is where "about two cents per push" comes from. It is arithmetic over the decision engine's
measured rates, not a measurement of a push Run. `ousast plane status` would print the real figure per
Task.
**What is not built** (each is a named gap, not a configuration): **What is not built** (each is a named gap, not a configuration):
- The delta-Run command. `ousast plane scan <repo> --base <commit> --head <commit>` **does not - The delta-Run command. `ousast plane scan <repo> --base <commit> --head <commit>` **does not
exist**; `ousast plane` has `run`, `status`, `doctor`, `remember`, `memory-normalise`, exist**. `ousast plane` has `run`, `status`, `doctor`, `remember`, `memory-normalise`,
`workspaces`, `harvest` and `alerts-engine` (`src/openultrasast/cli.py`). `workspaces`, `harvest` and `alerts-engine` (`src/openultrasast/cli.py`).
- A candidates task for an arbitrary repository. `repo-facts` takes its candidates from a - A candidates task for an arbitrary repository. `repo-facts` takes its candidates from a
population's validation set; a step that derives them from a diff is not written. population's validation set. No step derives them from a diff yet.
- Workspace generation from a URL and a commit outside a population. `plane/generate.py` is - Workspace generation from a URL and a commit outside a population. `plane/generate.py` is
population-driven (`read_cases` reads a validation set and a population TOML), so a Run for a population-driven: `read_cases` reads a validation set and a population TOML. So a Run for a
repository that is not in a population cannot be generated today. repository that is not in a population cannot be generated today.
- The remote execution mode itself: the `local`/`remote` switch, configurable cluster context, - The remote execution mode itself:
ax-server and router addresses (the `kind-` prefix is hard-coded, B.4), presigned delivery from - the `local`/`remote` switch;
the runner to the store, and the CLI reading completion and `summary.json` from the store - a configurable cluster context and ax-server and router addresses (the `kind-` prefix is
instead of its receiver thread. hard-coded, B.4);
- presigned delivery from the runner to the store;
- the CLI reading completion and `summary.json` from the store instead of its receiver thread.
- The engine task and its WorkerPool, the GHCR build, the production pool and autoscaling - The engine task and its WorkerPool, the GHCR build, the production pool and autoscaling
manifests, and the k3s runbook and checklist. manifests, and the k3s runbook and checklist.
- The check or comment that carries the asynchronous verdict back to the push. - The check or comment that carries the asynchronous verdict back to the push.
@@ -433,8 +586,7 @@ a push Run, and `ousast plane status` would print the real figure per Task.
### Sizing ### Sizing
The worker is the unit. Every Task's `limits` fit the 1 CPU / 1.5 GiB worker of The worker is the unit. Every Task's `limits` fit the 1 CPU / 1.5 GiB worker of
`ops/ax/workerpool.yaml.tmpl`; the requests are what the Task declares, the worker is what it `ops/ax/workerpool.yaml.tmpl`. The requests are what the Task declares. The worker is what it occupies.
occupies.
| Task (`plane/tasks/*.yaml`) | Requests | Limits | Per push | | Task (`plane/tasks/*.yaml`) | Requests | Limits | Per push |
| --- | --- | --- | --- | | --- | --- | --- | --- |
@@ -446,17 +598,18 @@ occupies.
| `roles` | 250m / 384Mi | 1 CPU / 1Gi | not in a push Run (harvest Runs only) | | `roles` | 250m / 384Mi | 1 CPU / 1Gi | not in a push Run (harvest Runs only) |
| `engine` (planned) | from a measured run | above 3 GiB (the host run uses `--memory 3g`) | its own pool, not the default one | | `engine` (planned) | from a measured run | above 3 GiB (the host run uses `--memory 3g`) | its own pool, not the default one |
- **Actors per push**: 4 (`facts`, `va`, `vb`, `agree`), 6 with a tie-break (`vc`, `final`); - **Actors per push**: 4 (`facts`, `va`, `vb`, `agree`), or 6 with a tie-break (`vc`, `final`).
`features` and `remember` add two more when the verdict is to be kept. `features` and `remember` add two more when the verdict is to be kept.
- **Critical path**: `facts` then `va` and `vb` side by side, then `agree`, then (sometimes) `vc` - **Critical path**: `facts`, then `va` and `vb` side by side, then `agree`, then sometimes `vc`
and `final`: three to five Task boots in series, each a golden-snapshot restore plus the model and `final`. That is three to five Task boots in series. Each boot is a golden-snapshot restore plus
calls of the verify passes. the model calls of the verify passes.
- **One actor per worker**: `agree` asks for 256Mi and still holds a 1.5 GiB worker while it runs, - **One actor per worker**: `agree` asks for 256Mi but still holds a 1.5 GiB worker while it runs. The
because the worker's capacity, not the Task's request, is the unit. A pool of `W` workers runs worker's capacity, not the Task's request, is the unit.
at most `W/2` pushes through their verify phase at once; on this host (`W = 2`) that is one - A pool of `W` workers runs at most `W/2` pushes through their verify phase at once.
push at a time, and every further push waits (the reconciler's `--workers` bound, or Substrate's - On this host (`W = 2`) that is one push at a time.
request parking if the pool is saturated). Autoscaling the pool on `at_capacity` workers is the - Every further push waits, on the reconciler's `--workers` bound or on Substrate's request parking
upstream answer and is **planned** here. if the pool is saturated.
- Autoscaling the pool on `at_capacity` workers is the upstream answer. It is **planned** here.
## C. The manifests under `plane/` ## C. The manifests under `plane/`
@@ -472,6 +625,12 @@ occupies.
| `ops/ax/smoke-task.yaml.tmpl`, `smoke-run.yaml.tmpl` | yes | rendered with `RUNNER_IMAGE` from your pin file | | `ops/ax/smoke-task.yaml.tmpl`, `smoke-run.yaml.tmpl` | yes | rendered with `RUNNER_IMAGE` from your pin file |
| `ops/ax/up.sh`, `down.sh` | no | kind-specific; they do not apply to another cluster | | `ops/ax/up.sh`, `down.sh` | no | kind-specific; they do not apply to another cluster |
The Run and Task contract itself (`plane/runner.py` as PID 1 of the runner image, `AX_TASK_YAML` The Run and Task contract does not depend on where the cluster is. That contract covers:
and `AX_WORKSPACES_YAML`, `/healthz` and `/readyz`, the start request, the tar delivery as
completion) does not depend on where the cluster is: [The plane on ax](plane.md). - `plane/runner.py` as PID 1 of the runner image;
- `AX_TASK_YAML` and `AX_WORKSPACES_YAML`;
- `/healthz` and `/readyz`;
- the start request;
- the tar delivery as completion.
See [The plane on ax](plane.md).
File diff suppressed because it is too large Load Diff
+75 -52
View File
@@ -1,7 +1,8 @@
# Evaluation # Evaluation
What has been measured on code the tool was not tuned on, and how. State as of 2026-10-02. This page shows what has been measured on code the tool was not tuned on, and how. State as of
Every number below is quoted from a committed record; the path is given with it. 2026-10-02. Every number below is quoted from a committed record, and the record's path is
given with it.
## Instruments ## Instruments
@@ -11,32 +12,47 @@ Every number below is quoted from a committed record; the path is given with it.
| Pair corpus | fire on the vulnerable function, stay silent on its fix | `ousast pairs`, `benchmarks/pairs/` | | Pair corpus | fire on the vulnerable function, stay silent on its fix | `ousast pairs`, `benchmarks/pairs/` |
| Independent populations | real repositories at vulnerable, fixed and benign pins, scored under a pre-registered protocol | `benchmarks/independent/` | | Independent populations | real repositories at vulnerable, fixed and benign pins, scored under a pre-registered protocol | `benchmarks/independent/` |
The detection gate and the local pairs are CI gates. They catch regressions; they do not Recall means the share of known bugs found. FP means false positives.
The detection gate and the local pairs are CI gates. They catch regressions. They do not
estimate recall on real code, because the rules were written against them. estimate recall on real code, because the rules were written against them.
## Pair corpora ## Pair corpora
`benchmarks/pairs/catalog.toml` and the slice directories (`local`, `github`, `sast`, `vfc`, A pair is a vulnerable function and its fixed version. You can run the catalog
`vibe-py`, `vfc-js`, `agent-vfc`, `owasp`) are runnable with `ousast pairs --slice <name>`; each `benchmarks/pairs/catalog.toml` and the slice directories with `ousast pairs --slice <name>`.
pair carries a provenance profile and a train/holdout split (`--profile`, `--split`). Pointer The slices are `local`, `github`, `sast`, `vfc`, `vibe-py`, `vfc-js`, `agent-vfc` and `owasp`.
pairs are not vendored: `--pointers` fetches them into `~/.cache/openultrasast/pairs/`.
`benchmarks/pairs/README.md` documents the schema and the review tiers. - Each pair carries a provenance profile and a train/holdout split (`--profile`, `--split`).
- Pointer pairs are not vendored. `--pointers` fetches them into `~/.cache/openultrasast/pairs/`.
- `benchmarks/pairs/README.md` documents the schema and the review tiers.
**`advisory-fixes`** (`benchmarks/pairs/advisory-fixes/`, 2026-10-01) holds 97 pointer pairs, **`advisory-fixes`** (`benchmarks/pairs/advisory-fixes/`, 2026-10-01) holds 97 pointer pairs,
one per repository: a public CVE/GHSA fix commit whose advisory-named function is declared on one per repository. Each is a public CVE/GHSA fix commit. The function the advisory names is
both sides and changed by the fix. It was assembled to give the decision engine real negatives declared on both sides and changed by the fix.
for families the corpus had few of (access control, deserialization, output encoding, path,
untrusted destination). It is a label source of the decision engine It was assembled to give the decision engine real negatives (fixed, non-vulnerable code). It
(`src/openultrasast/learn/sources.toml`), not an `ousast pairs` slice; the research log and targets families the corpus had few of: access control, deserialization, output encoding,
path, untrusted destination. It is a label source of the decision engine
(`src/openultrasast/learn/sources.toml`), not an `ousast pairs` slice. The research log and
every rejection are in `research-2026-10-01.md`. every rejection are in `research-2026-10-01.md`.
## Independent populations and the M4 gates ## Independent populations and the M4 gates
The pre-push safety net's qualification milestone (M4) requires, together: at least 95% M4 is the qualification milestone of the pre-push safety net. It requires all of these
actionable precision, at least 90% supported recall, at least 95% supported-check completion, together:
the runtime gates, and zero fixed-side or benign false alerts. A population is frozen before
any of its cases is scanned, scored once under a pre-registered protocol, and then **spent**: - at least 95% actionable precision;
it may inform changes, but it can no longer qualify anything. - at least 90% supported recall;
- at least 95% supported-check completion;
- the runtime gates;
- zero fixed-side or benign false alerts.
A population is a set of real repositories. Its lifecycle has three steps:
1. It is frozen before any of its cases is scanned.
2. It is scored once under a pre-registered protocol.
3. It is then **spent**: it may inform changes, but it can no longer qualify anything.
| Population | Status | Result | Record | | Population | Status | Result | Record |
| --- | --- | --- | --- | | --- | --- | --- | --- |
@@ -45,9 +61,9 @@ it may inform changes, but it can no longer qualify anything.
| v2, model-driven pipeline | exploratory (v2 already spent) | stopped after 7 of 17 cases when the account emptied: recall 4/7; precision not adjudicated; no gate measurable | `results-v2-model-pipeline.json` | | v2, model-driven pipeline | exploratory (v2 already spent) | stopped after 7 of 17 cases when the account emptied: recall 4/7; precision not adjudicated; no gate measurable | `results-v2-model-pipeline.json` |
| v3 (PHP, 15 cases; one SSRF slot left empty by the selection rule) | frozen, untouched | none: it is the one-time final check, run once under `protocol-v3.md` after a prediction is committed | `population-v3-php.toml`, `freeze-v3-php.json` | | v3 (PHP, 15 cases; one SSRF slot left empty by the selection rule) | frozen, untouched | none: it is the one-time final check, run once under `protocol-v3.md` after a prediction is committed | `population-v3-php.toml`, `freeze-v3-php.json` |
v3 is reserved: development and analysis work never reads its cases, and v3 is reserved. Development and analysis work never reads its cases.
`tests/test_independent_population.py` fails when anything in the development tree (a recipe, `tests/test_independent_population.py` fails when anything in the development tree names one
catalog, manifest or measurement) names one of its repositories. of its repositories. That covers a recipe, catalog, manifest or measurement.
```mermaid ```mermaid
stateDiagram-v2 stateDiagram-v2
@@ -66,9 +82,10 @@ stateDiagram-v2
## Folds: how learned decisions are kept out of their own training data ## Folds: how learned decisions are kept out of their own training data
The decision engine is evaluated on repositories it was not shown (`src/openultrasast/learn/folds.py`). The decision engine is evaluated on repositories it was not shown
Every split is by repository group (the label builder's normalised `owner/name`, merged across (`src/openultrasast/learn/folds.py`). Every split is by repository group. A group is the label
URLs, forks and shared advisories), across all corpora, seeded and deterministic. builder's normalised `owner/name`, merged across URLs, forks and shared advisories. Splits
cover all corpora and are seeded and deterministic.
```mermaid ```mermaid
flowchart TD flowchart TD
@@ -81,44 +98,50 @@ flowchart TD
rest --> lofo["Leave-one-framework-out: frameworks with at least 10 groups"] rest --> lofo["Leave-one-framework-out: frameworks with at least 10 groups"]
``` ```
`assert_disjoint` runs before a compile: the compile split and the evaluation folds share no `assert_disjoint` runs before a compile. It checks that the compile split and the evaluation
group. A framework with fewer than 10 groups gets `insufficient data` instead of a fold. As of folds share no group. A framework with fewer than 10 groups gets `insufficient data` instead of
2026-10-02 the outer folds have been run for the six evaluable families; leave-one-source-out a fold. As of 2026-10-02, the outer folds have been run for the six evaluable families.
and leave-one-framework-out have not Leave-one-source-out and leave-one-framework-out have not
([decision-engine.md](decision-engine.md#second-slice-six-families-2026-10-02)). ([decision-engine.md](decision-engine.md#second-slice-six-families-2026-10-02)).
## The plane increment ## The plane increment
The first increment of the agentic plane ran the model-driven pipeline's validation set (46 The first increment of the agentic plane ran on google/ax. It used the model-driven pipeline's
candidates in 15 cases of v2, 20 declared vulnerable sites) on google/ax and compared it with validation set: 46 candidates in 15 cases of v2, with 20 declared vulnerable sites. It was
the script pipeline on the same cases, candidates, pins and recorded triage. The pre-registered compared with the script pipeline on the same cases, candidates, pins and recorded triage.
gate: at least 16 of 20 declared sites agreed, at a cost per candidate under $0.022.
The pre-registered gate: at least 16 of 20 declared sites agreed, at a cost per candidate under
$0.022.
- **Increment 1** (2026-09-29, `benchmarks/independent/plane-increment-1.json`): 14 of 20 - **Increment 1** (2026-09-29, `benchmarks/independent/plane-increment-1.json`): 14 of 20
declared sites agreed against the reference's 16; $0.0194 per candidate with recorded triage. declared sites agreed, against the reference's 16. Cost was $0.0194 per candidate with
**NOT MET** on recall. recorded triage. **NOT MET** on recall.
*Correction (2026-09-30, in the record):* the first version read agree's *Correction (2026-09-30, in the record):* the first version misread agree's
`declared_sites_matched` (coverage by candidates, 19) as "found in either pass" and concluded `declared_sites_matched`. That field is coverage by candidates (19), but it was read as
detection was unchanged. Recounted from the agreed rows, the plane's passes flagged 17 of 20 "found in either pass". So the first version concluded detection was unchanged. Recounted
declared sites in at least one pass against the reference's 19. from the agreed rows, the plane's passes flagged 17 of 20 declared sites in at least one
- **Increment 2** (2026-09-30, `benchmarks/independent/plane-increment-2.json`): after the design pass. The reference flagged 19.
revisit, a tie-break pass c on the 10 disputed candidates with 2-of-3 agreement. 16 of 20 - **Increment 2** (2026-09-30, `benchmarks/independent/plane-increment-2.json`): came after the
declared sites agreed (reference 16) at $0.0205 per candidate over 46 ($0.02195 over the 43 design revisit. It added a tie-break pass c on the 10 disputed candidates, with 2-of-3
after triage, against the reference's $0.022). **MET**, with two caveats the record carries: agreement. 16 of 20 declared sites agreed (reference 16). Cost was $0.0205 per candidate over
the tie-break was chosen after seeing the first measurement on the same validation set, so 46, and $0.02195 over the 43 after triage (reference: $0.022). **MET**, with two
this compares plane with script and qualifies nothing on an independent population; and the caveats the record carries:
plane's passes still flag fewer declared sites than the reference (17 vs 19). Precision is 1. The tie-break was chosen after seeing the first measurement on the same validation set.
not part of this gate and remains the open problem. So this compares plane with script and qualifies nothing on an independent population.
2. The plane's passes still flag fewer declared sites than the reference (17 vs 19).
Precision is not part of this gate and remains the open problem.
## The decision engine ## The decision engine
Two slices have been measured out of repository (injection, 2026-10-01; six families, Two slices have been measured out of repository: injection (2026-10-01) and six families
2026-10-02). The engine is not adopted. The state against the gates, in one place with the plane (2026-10-02). The engine is not adopted. [Where we
increment and the populations above, is on [Where we stand](where-we-stand.md#1-detection-capability-and-goals-no-go-measured); stand](where-we-stand.md#1-detection-capability-and-goals-no-go-measured) shows the state
the per-family tables are on [decision-engine.md](decision-engine.md). against the gates in one place, with the plane increment and the populations above. The
per-family tables are on [decision-engine.md](decision-engine.md).
## Removal of the earlier agentic extra ## Removal of the earlier agentic extra
The removal of the earlier agentic extra (2026-09-30) was checked for byte-identical The earlier agentic extra was removed on 2026-09-30. The removal was checked for byte-identical
deterministic outputs (pair replay, quick benchmark, standard scan, `improve --dry-run`): deterministic outputs: pair replay, quick benchmark, standard scan, `improve --dry-run`. Records:
`benchmarks/measurements/2026-09-30-harnessx-removal-baseline/` and `-equality/`. `benchmarks/measurements/2026-09-30-harnessx-removal-baseline/` and `-equality/`.
+28 -17
View File
@@ -1,7 +1,7 @@
# End-to-end examples # End-to-end examples
Each example is self-contained. Artifacts land under Each example stands on its own. Artifacts land under `<target>/.openultrasast/runs/<scan-id>/`.
`<target>/.openultrasast/runs/<scan-id>/` and `.openultrasast/` is gitignored. `.openultrasast/` is gitignored.
## 1. Quick scan a local repo (deterministic, no keys) ## 1. Quick scan a local repo (deterministic, no keys)
@@ -21,7 +21,7 @@ ousast scan . --mode standard # Joern model layer, container network off:
ousast-with-judge scan . --mode standard # network on, so the model is asked (needs DEEPSEEK_API_KEY) ousast-with-judge scan . --mode standard # network on, so the model is asked (needs DEEPSEEK_API_KEY)
``` ```
Without a key the graph still decides: its entailed findings are reported, and the LLM's residual Without a key, the graph still decides. Its entailed findings are reported. The LLM's residual
and suspicion questions go unasked. and suspicion questions go unasked.
## 3. CI gate ## 3. CI gate
@@ -52,10 +52,11 @@ uv run ousast pairs --split holdout --profile human # filter by declared split
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
``` ```
Each case is two isolated trees (vuln file vs patched file, same relative path). Each case is two isolated trees: the vuln file and the patched file, at the same relative path.
`pair_correct` means the harness fired on the vuln side and stayed silent on the
fix. Misses and fix-side leaks become `miss`/`fp` signals for `ousast improve`. - `pair_correct` means the harness fired on the vuln side and stayed silent on the fix.
The catalog and public dataset pointers live in `benchmarks/pairs/`. - Misses become `miss` signals and fix-side leaks become `fp` signals for `ousast improve`.
- The catalog and public dataset pointers live in `benchmarks/pairs/`.
## 6. Self-improve the ruleset from benchmark feedback ## 6. Self-improve the ruleset from benchmark feedback
@@ -68,10 +69,11 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml
## 7. Running the agentic plane ## 7. Running the agentic plane
Model work over whole repositories runs as a Run manifest on ax, one isolated actor Model work over whole repositories runs as a Run manifest on ax. Each task gets one isolated
per task (bring-up: [ops/ax/README.md](ops/ax/README.md)). The provider key is actor. Bring-up is in [ops/ax/README.md](ops/ax/README.md).
read from the environment (or `.env`, which never overrides an exported variable) by
`ousast` and sent only in each task's start request. `ousast` reads the provider key from the environment, or from `.env`. `.env` never overrides an
exported variable. The key is sent only in each task's start request.
```bash ```bash
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
@@ -83,16 +85,25 @@ uv run ousast plane remember validation-46 # re-ingest the run's m
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
``` ```
Each task in the Run carries its own Model and `budget: {usd, calls}`; a task that Each task in the Run carries its own Model and `budget: {usd, calls}`.
reaches its ceiling stops as `unfinished` and resumes on a rerun with a larger
budget. `ousast plane status` prints the token attribution table: calls, prompt, - A task that reaches its ceiling stops as `unfinished`. It resumes on a rerun with a larger
cache-hit and output tokens and USD per task. budget.
- `ousast plane status` prints the token attribution table. It shows calls, prompt, cache-hit and
output tokens, and USD per task.
## 8. Drive it from an MCP client (OpenCode / IDE) ## 8. Drive it from an MCP client (OpenCode / IDE)
MCP (Model Context Protocol) lets an editor or agent call the tool's functions.
```jsonc ```jsonc
{ "command": "uv", "args": ["run", "ousast", "mcp"] } { "command": "uv", "args": ["run", "ousast", "mcp"] }
``` ```
Then: `openultrasast.scan {path}` → `run_dir` → `openultrasast.findings {run_dir}` → Then call the tools in this order:
`openultrasast.explain {run_dir, finding_id}`. See the README section "Agent integrations".
1. `openultrasast.scan {path}` returns a `run_dir`.
2. `openultrasast.findings {run_dir}` lists the findings.
3. `openultrasast.explain {run_dir, finding_id}` explains one.
See the README section "Agent integrations".
+24 -18
View File
@@ -1,13 +1,13 @@
# OpenUltraSAST # OpenUltraSAST
OpenUltraSAST is an experimental static security analyser meant to work as a safety net on OpenUltraSAST is an experimental static security analyser. It is meant to be a safety net on
repositories it has never seen: it scans a checkout, or the commits a `git push` would publish, repositories it has never seen. It scans a checkout, or the commits a `git push` would publish. It
and separates what it can prove from what it merely suspects. A scan that could not look at keeps what it can prove apart from what it only suspects. When a scan could not look at
something records a degradation instead of reporting a clean result. something, it records a degradation. It does not report a clean result.
**State as of 2026-10-02 (v2.0.0):** the pre-push safety net is **NO-GO** for rollout. **State as of 2026-10-02 (v2.0.0):** the pre-push safety net is **NO-GO** for rollout.
No hook capability is enabled, and no independent population has passed the qualification gates No hook capability is enabled. No independent population (a case set frozen before any scan) has
(see [Evaluation](evaluation.md)). passed the qualification gates (see [Evaluation](evaluation.md)).
## Three layers ## Three layers
@@ -17,6 +17,9 @@ No hook capability is enabled, and no independent population has passed the qual
| **The agentic plane on ax** | Model-driven work over whole repositories, run as a DAG of tasks on google/ax, one isolated actor per task. What the runs learn is kept in a memory store. Maintainer surface: `ousast plane`. | [The plane on ax](plane.md), [Memory](memory.md) | | **The agentic plane on ax** | Model-driven work over whole repositories, run as a DAG of tasks on google/ax, one isolated actor per task. What the runs learn is kept in a memory store. Maintainer surface: `ousast plane`. | [The plane on ax](plane.md), [Memory](memory.md) |
| **The decision engine** | A learned classifier that weighs every instrument's signals against similar labelled cases from memory. **In development, not adopted**: no scan, pre-push check or report uses it. Maintainer surface: `ousast learn`. | [Memory and detection](memory-and-detection.md), [Decision engine](decision-engine.md) | | **The decision engine** | A learned classifier that weighs every instrument's signals against similar labelled cases from memory. **In development, not adopted**: no scan, pre-push check or report uses it. Maintainer surface: `ousast learn`. | [Memory and detection](memory-and-detection.md), [Decision engine](decision-engine.md) |
A code property graph is a graph of the program's syntax, control flow and data flow. Joern builds
it.
## How they connect ## How they connect
```mermaid ```mermaid
@@ -33,31 +36,34 @@ flowchart LR
engine -. "intended, not adopted: BLOCK / ADVISORY" .-> report engine -. "intended, not adopted: BLOCK / ADVISORY" .-> report
``` ```
- The **scanners** produce findings and, inside plane Runs, the `alerts` that become memory rows. - The **scanners** produce findings. Inside plane Runs, they also produce the `alerts` that become
- The **plane** runs model work (verify passes, roles) and model-free bookkeeping (facts, agreement, memory rows.
feature records) per case; its `remember` task turns a case's artifacts into rows of the - The **plane** runs model work per case: verify passes and roles. It also runs model-free
[memory store](memory.md). bookkeeping per case: facts, agreement and feature records. Its `remember` task turns a case's
- Memory feeds two consumers. `ousast improve --memory` turns stored rows into rule-status artifacts into rows of the [memory store](memory.md).
proposals that pass through the same validator and gate as every other edit. The decision engine - Memory feeds two consumers:
retrieves labelled examples from the same store. 1. `ousast improve --memory` turns stored rows into rule-status proposals. They pass through the
same validator and gate as every other edit.
2. The decision engine retrieves labelled examples from the same store.
- The decision engine's output (dotted line) is the intended path. Nothing user-facing reads it - The decision engine's output (dotted line) is the intended path. Nothing user-facing reads it
today. today.
## Where to go next ## Where to go next
- The current state in one page, every figure with its record: capability against the M4 gates, the - The current state in one page, every figure with its record: [Where we stand](where-we-stand.md).
step-by-step approach, ax on the server, token economics before and after, and the open items: It covers capability against the M4 gates, the step-by-step approach, ax on the server, token
[Where we stand](where-we-stand.md). economics before and after, and the open items.
- Run a scan: [Examples](examples.md), and the commands in [Scanning](scanning.md). - Run a scan: [Examples](examples.md), and the commands in [Scanning](scanning.md).
- Understand a finding's evidence and score: [Architecture](architecture.md). - Understand a finding's evidence and score: [Architecture](architecture.md).
- The program analysis under a scan, what each representation can see and what it misses: - The program analysis under a scan, what each representation can see and what it misses:
[Detection techniques](detection-techniques.md). [Detection techniques](detection-techniques.md).
- Operate the plane and its store: [ax on this host](ops/ax/README.md) and - Operate the plane and its store: [ax on this host](ops/ax/README.md) and
[RustFS setup](rustfs.md). [RustFS setup](rustfs.md).
- Where the plane runs today and what a separate Kubernetes cluster would take (planned, not - Where the plane runs today, and what a separate Kubernetes cluster would take (planned, not
done): [Deployment](deployment.md). done): [Deployment](deployment.md).
- Where the model spend goes, per Task at runtime and in the maintainers' own coding sessions, - Where the model spend goes, per Task at runtime and in the maintainers' own coding sessions,
and what is reused instead of paid again: [Token ergonomics](token-ergonomics.md). and what is reused instead of paid again: [Token ergonomics](token-ergonomics.md).
- Trust boundaries and hardening: [Threat model](threat-model.md). - Trust boundaries and hardening: [Threat model](threat-model.md).
Every number in these pages is quoted from a committed record whose path is given next to it. Every number in these pages is quoted from a committed record. The record's path is given next to
the number.
+139 -85
View File
@@ -1,11 +1,12 @@
# Memory and detection # Memory and detection
How what the system remembers is meant to turn into better decisions on code it has not seen. This page shows how the system's memory is meant to improve decisions on code it has not seen. It
This is the design of the learned decision engine (`src/openultrasast/learn/`), which is **in describes the design of the learned decision engine (`src/openultrasast/learn/`).
development and not adopted**: no scan, pre-push check or report uses it today. Every user-facing
decision is still the deterministic one described in [Scanning](scanning.md) and The engine is **in development and not adopted**. No scan, pre-push check or report uses it today.
Every user-facing decision is still the deterministic one described in [Scanning](scanning.md) and
[Architecture](architecture.md). The engine's commands and status are on [Architecture](architecture.md). The engine's commands and status are on
[Decision engine](decision-engine.md); the store itself is on [Memory](memory.md). [Decision engine](decision-engine.md). The store itself is on [Memory](memory.md).
## From instruments to a decision ## From instruments to a decision
@@ -25,89 +26,128 @@ flowchart TD
op -- "otherwise" --> adv["ADVISORY"] op -- "otherwise" --> adv["ADVISORY"]
``` ```
The intended place for these decisions is changed code: `learn/features.py` builds feature records These decisions are meant for changed code. `learn/features.py` builds feature records in two ways:
for a whole-repository scan or for a base..head delta (`build_for_scan` with `Delta`), the same
delta `ousast pre-push` checks. Integration into scan and pre-push is still ahead. - for a whole-repository scan;
- for a base..head delta (`build_for_scan` with `Delta`). This is the same delta `ousast pre-push` checks.
Integration into scan and pre-push is still ahead.
## 1. Instruments produce signals, not verdicts ## 1. Instruments produce signals, not verdicts
Each instrument contributes one block to a candidate's feature record (`learn/schema.py`, Each instrument contributes one block to a candidate's feature record (`learn/schema.py`,
`INSTRUMENTS`): `language`, `quick` (quick rules), `engine` (the Joern model layer), `facts` (repo `INSTRUMENTS`). The blocks are:
facts, such as callers and function span), `source`, `entry_points`, `roles` (model source, sink
and sanitizer roles), `delta`, `verify` and `agree` (the plane's verify passes and their
agreement), and `model_sinks` (the archived stage-1 classifier, `benchmarks/archive/model_sinks.py`, whose
logic now lives in `plane/tasks/roles.py`). A candidate is `(path, function, family)` of one repository and pin
where at least one instrument emitted a signal of that family (`learn/features.py`).
An instrument that did not run, had no coverage or failed records that state (`none`, `failed`, - `language`;
`not_applicable`), never a zero, so "silent" and "did not look" stay distinguishable. None of the - `quick` (quick rules);
instruments decides alone. - `engine` (the Joern model layer);
- `facts` (repo facts, such as callers and function span);
- `source`;
- `entry_points`;
- `roles` (model source, sink and sanitizer roles);
- `delta`;
- `verify` and `agree` (the plane's verify passes and their agreement);
- `model_sinks` (the archived stage-1 classifier, `benchmarks/archive/model_sinks.py`). Its logic now
lives in `plane/tasks/roles.py`.
**Why profile v1 withholds the engine and function length.** A leak audit A candidate is `(path, function, family)` of one repository and pin. At least one instrument must have
(`ousast learn audit-leaks`, `benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`) emitted a signal of that family there (`learn/features.py`).
found that both separate the vulnerable and fixed sides of a pair by construction: a fix removes
a sink, and a guard lengthens the function. That holds on the pair corpus, not on unseen code, so An instrument may not run, may have no coverage, or may fail. It then records that state (`none`,
`INPUT_PROFILES["v1"]` drops the `engine` block and `facts.function_lines` from prompts and from `failed`, `not_applicable`), never a zero. So "silent" and "did not look" stay distinguishable. No
the retrieval distance. instrument decides alone.
**Why profile v1 withholds the engine and function length.** A leak audit found that both separate the
vulnerable and fixed sides of a pair by construction (`ousast learn audit-leaks`,
`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`):
- a fix removes a sink;
- a guard lengthens the function.
That holds on the pair corpus, not on unseen code. So `INPUT_PROFILES["v1"]` drops the `engine` block
and `facts.function_lines` from prompts and from the retrieval distance.
## 2. Labels come from ground truth only ## 2. Labels come from ground truth only
`ousast learn labels` (`learn/labels.py`) reads only the sources on a fail-closed allow-list `ousast learn labels` (`learn/labels.py`) reads only the sources on a fail-closed allow-list
(`learn/sources.toml`): `population-v1`, `population-v2`, `dev-php` (reviewed recipe sites), (`learn/sources.toml`). The sources are `population-v1`, `population-v2`, `dev-php` (reviewed recipe
`pairs`, `advisory-fixes`, `adjudications` and `assumed_benign`. A file the list does not name is sites), `pairs`, `advisory-fixes`, `adjudications` and `assumed_benign`. A file the list does not name
refused before it is opened; the reserved population v3 is excluded unread. is refused before it is opened. The reserved population v3 is excluded unread.
- **Positives** are declared sites and the vulnerable side of a fix; **negatives** are the same - **Positives** are declared sites and the vulnerable side of a fix.
function on the fixed side where the fix changed it, and recorded `False`/`FP` adjudications. - **Negatives** are two kinds:
A vulnerable-pin candidate matching no site is unlabelled, never negative. - the same function on the fixed side, where the fix changed it;
- **Assumed benign** (maintainer decision 2026-09-30): ordinary non-security commits of the training - recorded `False`/`FP` adjudications.
repositories, at least 90 days from any fix, with no later security commit touching the same - A vulnerable-pin candidate matching no site is unlabelled, never negative.
files. They are their own source with weight 0.5, conditional on a signal at the tip and none at - **Assumed benign** (maintainer decision 2026-09-30) covers ordinary non-security commits of the
the base, and reported separately, never merged with verified negatives. training repositories. Each is at least 90 days from any fix. No later security commit touches the
- **Plane verdicts are features, not labels.** "Agreed", "rejected" or "disputed" is the output of same files.
the verify instrument; it enters the record as the `verify` and `agree` blocks. Using it as a - They are their own source with weight 0.5, conditional on a signal at the tip and none at the base.
label would teach the engine to reproduce the plane, mistakes included, so "rejected by the - They are reported separately, never merged with verified negatives.
plane" is never a negative (`learn/labels.py`: "A plane verdict ... is never a label"). - **Plane verdicts are features, not labels.** "Agreed", "rejected" or "disputed" is the output of the
verify instrument. It enters the record as the `verify` and `agree` blocks. Using it as a label would
teach the engine to reproduce the plane, mistakes included. So "rejected by the plane" is never a
negative (`learn/labels.py`: "A plane verdict ... is never a label").
`ousast learn memory build` joins each label with the candidate's feature record and an excerpt of `ousast learn memory build` joins each label with the candidate's feature record and an excerpt of
its function and stores one `example` row (no rationale) plus the excerpt blob. its function. It stores one `example` row (no rationale) plus the excerpt blob.
## 3. Retrieval of similar labelled cases ## 3. Retrieval of similar labelled cases
`learn/retrieve.py`, no model: `learn/retrieve.py` uses no model. It works in three steps:
1. **Signal profile.** A Gower distance over the allow-listed features of the active profile, 1. **Signal profile.** It computes a Gower distance (a distance for mixed-type features) over the
per instrument block, with missing values explicit (a different state counts as distance 1). allow-listed features of the active profile, per instrument block. Missing values are explicit: a
The nearest 40 examples of the candidate's family and the nearest 40 of other families go on. different state counts as distance 1. The nearest 40 examples of the candidate's family go on. So do
2. **Code embeddings.** A cosine re-rank of the excerpts' embeddings (OpenAI the nearest 40 of other families.
`text-embedding-3-small` through OpenRouter, 1,536 dimensions, cached by excerpt hash): 2. **Code embeddings.** It re-ranks by cosine over the excerpts' embeddings. The embeddings come from
`r = 0.5 (1 - D) + 0.5 cos`. Without the candidate's vector the retrieval is signals only. OpenAI `text-embedding-3-small` through OpenRouter (1,536 dimensions, cached by excerpt hash). The
3. **Balance.** Six examples: at most half per label, at most 2 per repository group, at most 2 formula is `r = 0.5 (1 - D) + 0.5 cos`. Without the candidate's vector, the retrieval uses signals only.
contrast examples from other families. 3. **Balance.** It picks six examples:
- at most half per label;
- at most 2 per repository group;
- at most 2 contrast examples from other families.
**The evaluation boundary.** `eligible()` is the only way an example enters a prompt, for retrieved **The evaluation boundary.** `eligible()` is the only way an example enters a prompt. This holds for
examples and compiled demonstrations alike: never the candidate's own repository group, never a retrieved examples and compiled demonstrations alike. An eligible example is never:
group the fold evaluates, never the held-out source or framework, and (outside deployment) never
a near-duplicate of the candidate (cosine >= 0.98). The program repeats the check on the ids it - from the candidate's own repository group;
actually rendered before any call. - from a group the fold evaluates;
- from the held-out source or framework;
- a near-duplicate of the candidate (cosine >= 0.98), outside deployment.
The program repeats the check on the ids it actually rendered, before any call.
## 4. The AI classifier, compiled ## 4. The AI classifier, compiled
`learn/program.py` and `learn/compile.py`, DSPy-style and in-house. The prompt is the instruction The classifier lives in `learn/program.py` and `learn/compile.py`. It is DSPy-style (DSPy is a
plus the compiled demonstrations (a prefix identical for every candidate, so the provider's prefix framework that compiles prompts from examples) and built in-house.
cache hits), then the retrieved examples, then the candidate's excerpt, signals, roles and family.
The model (`deepseek-flash`) answers `vulnerable`, `not_vulnerable` or `unsure` with a confidence The prompt has this order:
and a rationale that must cite a line of the code. It is sampled `k` times; the score `s` is the
mean over samples, and a majority `unsure` can never BLOCK. Compilation bootstraps demonstrations 1. the instruction plus the compiled demonstrations. This prefix is identical for every candidate, so
and searches instructions on a compile split of repository groups, scored by a pre-registered the provider's prefix cache hits;
balanced Brier score (`learn/compile.toml`); the program is stored as `programs/<id>.json`. 2. the retrieved examples;
3. the candidate's excerpt, signals, roles and family.
The model (`deepseek-flash`) answers `vulnerable`, `not_vulnerable` or `unsure`. It adds a confidence
and a rationale. The rationale must cite a line of the code. The model is sampled `k` times. The score
`s` is the mean over samples. A majority `unsure` can never BLOCK.
Compilation bootstraps demonstrations and searches instructions on a compile split of repository groups.
A pre-registered balanced Brier score scores it (`learn/compile.toml`). The program is stored as
`programs/<id>.json`.
## 5. Calibration and the two operating points ## 5. Calibration and the two operating points
`learn/calibrate.py`: Platt scaling of `s`, cross-fitted so that no map is checked on the fold it `learn/calibrate.py` applies Platt scaling (a logistic map from score to probability) to `s`. It is
was fitted on. Calibration **holds** when the slope interval contains 1, the intercept interval cross-fitted, so no map is checked on the fold it was fitted on.
contains 0 and ECE <= 0.05. **BLOCK** is offered only when calibration holds and a fold reaches the
required precision; otherwise the engine offers **ADVISORY** only. Calibration **holds** when the slope interval contains 1, the intercept interval contains 0 and ECE
(expected calibration error) <= 0.05.
**BLOCK** is offered only when calibration holds and a fold reaches the required precision. Otherwise
the engine offers **ADVISORY** only.
## The loop: memory improves decisions without retraining ## The loop: memory improves decisions without retraining
@@ -125,25 +165,36 @@ flowchart LR
gate -- "accepted round" --> ledger["Rule status in the ledger"] gate -- "accepted round" --> ledger["Rule status in the ledger"]
``` ```
- **Decisions.** The compiled part of the program is its instruction and demonstrations; the - **Decisions.**
examples are retrieved from memory for each candidate. A new `example` row is therefore eligible - The compiled part of the program is its instruction and demonstrations.
for retrieval in the next decision without recompiling the program, subject to the same - The examples are retrieved from memory for each candidate.
evaluation boundary. `ousast learn curve` measures the metric over memory size on a fixed - So a new `example` row is eligible for retrieval in the next decision without recompiling. The same
candidate subset. evaluation boundary applies.
- **Rules.** `ousast improve --memory` reads the run rows (alerts, verdicts, coverage) and proposes - `ousast learn curve` measures the metric over memory size on a fixed candidate subset.
rule-status edits only (shadow a rule that is repeatedly false across repositories; re-enable a - **Rules.** `ousast improve --memory` reads the run rows (alerts, verdicts, coverage). It proposes
shadow rule that keeps hitting agreed declared sites). Rows from holdout pairs, the gated rule-status edits only:
manifest's own cases and any `--qualify-population` are dropped first, and every proposal goes - shadow a rule that is repeatedly false across repositories;
through the same validator and gate as any other edit (`improve/memory.py`). - re-enable a shadow rule that keeps hitting agreed declared sites.
Some rows are dropped first: rows from holdout pairs, the gated manifest's own cases and any
`--qualify-population`. Every proposal goes through the same validator and gate as any other edit
(`improve/memory.py`).
## What the first slice measured ## What the first slice measured
Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts
only). Family **injection**; memory of 724 static examples from all families; input profile v1; only). The setup:
framework priors off; out-of-repository evaluation (compile split never evaluated, outer grouped
5-fold, calibration cross-fitted, operating points nested); 95% repository-cluster bootstrap - family **injection**;
intervals. The slice's spend ($0.003215 per candidate at evaluation, k = 5, and its replay at - memory of 724 static examples from all families;
$0) is on [Token ergonomics](token-ergonomics.md#measured-costs). - input profile v1;
- framework priors off;
- out-of-repository evaluation: compile split never evaluated, outer grouped 5-fold, calibration
cross-fitted, operating points nested;
- 95% repository-cluster bootstrap intervals.
The slice's spend is on [Token ergonomics](token-ergonomics.md#measured-costs). It was $0.003215 per
candidate at evaluation, k = 5, and its replay cost $0.
| Measure | Value | What it means | | Measure | Value | What it means |
| --- | --- | --- | | --- | --- | --- |
@@ -152,9 +203,12 @@ $0) is on [Token ergonomics](token-ergonomics.md#measured-costs).
| ADVISORY | recall 0.90 (0.82 - 0.97), precision 0.54 (0.44 - 0.65) | precision is at the positive share (0.53), so at this threshold ADVISORY is **not discriminating**: it flags nearly as indiscriminately as flagging everything would | | ADVISORY | recall 0.90 (0.82 - 0.97), precision 0.54 (0.44 - 0.65) | precision is at the positive share (0.53), so at this threshold ADVISORY is **not discriminating**: it flags nearly as indiscriminately as flagging everything would |
| BLOCK | not offered | calibration does not hold (ECE 0.064 > 0.05), and every outer fold reported `precision_unreachable` | | BLOCK | not offered | calibration does not hold (ECE 0.064 > 0.05), and every outer fold reported `precision_unreachable` |
**The engine is not adopted.** A second slice (2026-10-02) evaluated the six evaluable families **The engine is not adopted.** A second slice (2026-10-02) evaluated the six evaluable families the
the same way: pooled AUC 0.77 to 0.96 per family, within-pair AUC 0.72 to 0.93, BLOCK offered for same way ([Decision engine](decision-engine.md#second-slice-six-families-2026-10-02)). The results: pooled AUC 0.77 to 0.96 per family, within-pair AUC 0.72 to 0.93, BLOCK offered for
none, re-run agreement below 0.9 for three families none. Re-run agreement was below 0.9 for three families.
([Decision engine](decision-engine.md#second-slice-six-families-2026-10-02)). Leave-one-source-out
and leave-one-framework-out, integration into scan and pre-push, and the one-time check on Still ahead:
population v3 are still ahead.
- leave-one-source-out and leave-one-framework-out;
- integration into scan and pre-push;
- the one-time check on population v3.
+50 -40
View File
@@ -1,8 +1,8 @@
# Memory # Memory
One store, keyed by repository and pin, holds what plane runs and the decision engine learned. One store holds what plane runs and the decision engine learned. It is keyed by repository and pin
The code is `src/openultrasast/plane/memory.py`; the same layout serves both backends. (a fixed commit). The code is `src/openultrasast/plane/memory.py`. The same layout serves both backends.
How the stored rows feed detection is on [Memory and detection](memory-and-detection.md). [Memory and detection](memory-and-detection.md) shows how the stored rows feed detection.
## What is stored ## What is stored
@@ -17,7 +17,7 @@ programs/<program id>.json compiled decision-engine programs
``` ```
**Rows.** Every row carries `id`, `kind`, `repo`, `pin`, `run`, `task`, `population`, `split` **Rows.** Every row carries `id`, `kind`, `repo`, `pin`, `run`, `task`, `population`, `split`
and `image` (`ROW_FIELDS`); `id` is the sha256 of (kind, run, task, subject), so a repeated row and `image` (`ROW_FIELDS`). `id` is the sha256 of (kind, run, task, subject). So a repeated row
replaces itself. The kinds (`KINDS`) and who writes them: replaces itself. The kinds (`KINDS`) and who writes them:
| Kind | Written by | Holds | | Kind | Written by | Holds |
@@ -35,12 +35,15 @@ replaces itself. The kinds (`KINDS`) and who writes them:
| `experiment_result` | `ousast learn experiment analyse --record` | the paired estimates, the looks and the adoption verdict (counts and intervals only) | | `experiment_result` | `ousast learn experiment analyse --record` | the paired estimates, the looks and the adoption verdict (counts and intervals only) |
| `label`, `decision` | reserved for later decision-engine tasks | declared in `KINDS`; not written by the code on main yet | | `label`, `decision` | reserved for later decision-engine tasks | declared in `KINDS`; not written by the code on main yet |
**Blobs** (`put_blob`) are content-addressed: a 64-hex name under a fixed prefix. Excerpts are **Blobs** (`put_blob`) are content-addressed: each has a 64-hex name under a fixed prefix.
written by `learn memory build` (`learn/examples.py`), embeddings by `learn memory embed` Content-addressed means the name is the hash of the content. Who writes each blob:
(`learn/embeddings.py`, keyed by excerpt sha so an excerpt is embedded once), responses by the
program's caller (`learn/program.py`, keyed by the sha256 of model, parameters digest, messages, - Excerpts: `learn memory build` (`learn/examples.py`).
temperature, sample and format, so a rerun replays at no cost), and compiled programs by - Embeddings: `learn memory embed` (`learn/embeddings.py`). They are keyed by excerpt sha, so an
`learn compile` (`learn/compile.py`). excerpt is embedded once.
- Responses: the program's caller (`learn/program.py`). The key is the sha256 of model, parameters
digest, messages, temperature, sample and format. So a rerun replays at no cost.
- Compiled programs: `learn compile` (`learn/compile.py`).
## Write path ## Write path
@@ -60,13 +63,14 @@ flowchart LR
put --> store[("Memory store")] put --> store[("Memory store")]
``` ```
- `ingest` validates every row first: a malformed file fails before anything is written. - `ingest` validates every row first. A malformed file fails before anything is written.
- Rows are merged into their repository + pin object by `id`. Every object is written whole and the - Rows are merged by `id` into their repository + pin object.
index last, so an interrupted ingest leaves nothing a repeat would not repair. - Every object is written whole, and the index last. So a repeat repairs anything an interrupted
ingest left behind.
- Each row object carries tags (`TAG_FIELDS`: `repo`, `pin`, `kind`, `family`, `run`, - Each row object carries tags (`TAG_FIELDS`: `repo`, `pin`, `kind`, `family`, `run`,
`population`, `split`) listing the distinct values of its rows (`*` when they exceed 256 `population`, `split`). The tags list the distinct values of its rows. A tag is `*` when the
characters). values exceed 256 characters.
- The plane's store failures never change a run's result: an ingest can be repeated by hand - The plane's store failures never change a run's result. An ingest can be repeated by hand
(`cli.py`, `_plane_memory`). (`cli.py`, `_plane_memory`).
## Read path ## Read path
@@ -85,14 +89,14 @@ flowchart LR
rules --> gate{"Unchanged validator and gate"} rules --> gate{"Unchanged validator and gate"}
``` ```
- **Seed** (`memory.seed`): a Run task whose `openultrasast.io/memory-key` matches stored facts gets - **Seed** (`memory.seed`).
them and is marked done; a changed pin, candidate set or runner image finds nothing and - A Run task gets stored facts when its `openultrasast.io/memory-key` matches them. It is then marked done.
recomputes. The loop's `memory-snapshot` is seeded the same way, because a sandboxed task cannot - A changed pin, candidate set or runner image finds nothing and recomputes.
read the store. - The loop's `memory-snapshot` is seeded the same way. A sandboxed task cannot read the store.
- **Retrieval** (`learn/retrieve.py`): see - **Retrieval** (`learn/retrieve.py`): see
[Memory and detection](memory-and-detection.md#3-retrieval-of-similar-labelled-cases). [Memory and detection](memory-and-detection.md#3-retrieval-of-similar-labelled-cases).
- **`improve --memory`** (`improve/memory.py`): rows from holdout pairs, from the gated manifest's - **`improve --memory`** (`improve/memory.py`). Some rows are dropped before any rule sees them:
own cases and from any `--qualify-population` are dropped before any rule sees them. See rows from holdout pairs, from the gated manifest's own cases, and from any `--qualify-population`. See
[Architecture](architecture.md#proposals-from-plane-memory). [Architecture](architecture.md#proposals-from-plane-memory).
## Backends: FileStore and the S3 store ## Backends: FileStore and the S3 store
@@ -109,26 +113,32 @@ flowchart LR
### Verify at startup, no fallback ### Verify at startup, no fallback
The S3 store never configures its bucket. Each time an `S3Store` is opened for a bucket and The S3 store never configures its bucket. It only checks it. An `S3Store` is opened for a bucket and
prefix (once per process), `verify_bucket()` checks, using reads plus one small probe object at prefix once per process. Each time, `verify_bucket()` runs these checks. It uses reads plus one small
`<prefix>/_probe/select.jsonl`: probe object at `<prefix>/_probe/select.jsonl`.
1. versioning is `Enabled` (provenance cites object versions); 1. Versioning is `Enabled`. Provenance cites object versions.
2. an enabled lifecycle rule expires `<prefix>/runs/` after a number of days, as a plain prefix 2. An enabled lifecycle rule expires `<prefix>/runs/` after a number of days. It uses a plain prefix
filter, and no enabled rule expires the whole store; filter. No enabled rule expires the whole store.
3. S3 Select answers a probe query over JSON Lines; 3. S3 Select (a server-side query over JSON objects) answers a probe query over JSON Lines.
4. the probe's object tags are readable. 4. The probe's object tags are readable.
Anything missing is named, with the admin command that fixes it, in one `MemoryStoreError`, and If anything is missing, the store does not start. One `MemoryStoreError` names each problem and the
the store does not start. There is no local or fetch-and-filter fallback: a filtered read that admin command that fixes it.
S3 Select cannot answer raises a `MemoryStoreError` instead of returning "no rows". The exact
messages are listed on [RustFS setup](rustfs.md#what-the-store-verifies-at-startup). There is no local or fetch-and-filter fallback. A filtered read that S3 Select cannot answer raises a
`MemoryStoreError`. It does not return "no rows". [RustFS setup](rustfs.md#what-the-store-verifies-at-startup)
lists the exact messages.
### Queryable fields per kind ### Queryable fields per kind
RustFS's Select infers an object's JSON schema from its leading rows, so a `where` field missing RustFS's Select infers an object's JSON schema from its leading rows. This causes two failures
there fails with `EvaluatorBindingDoesNotExist` even when a later row has it, and a column that is (measured 2026-10-02; `memory.py`, `S3Store._select`):
null in the first 1000 rows breaks Select on the whole object (measured 2026-10-02; `memory.py`,
`S3Store._select`). The store therefore writes every row with its kind's full set of queryable - A `where` field missing from the leading rows fails with `EvaluatorBindingDoesNotExist`. This
fields (`QUERY_FIELDS`), using `""` where a field does not apply, and refuses a `where` on an happens even when a later row has the field.
undeclared field. Existing rows are rewritten once with `ousast plane memory-normalise`. - A column that is null in the first 1000 rows breaks Select on the whole object.
So the store writes every row with its kind's full set of queryable fields (`QUERY_FIELDS`). It uses
`""` where a field does not apply. It refuses a `where` on an undeclared field. Existing rows are
rewritten once with `ousast plane memory-normalise`.
+82 -56
View File
@@ -1,23 +1,34 @@
# The plane on ax # The plane on ax
Model-driven work over whole repositories runs on a separate agentic plane: a **Run** Model-driven work over whole repositories runs on a separate agentic plane. The plane has these parts:
(`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG of steps, and each step is
an ax **Task** (`plane/tasks/`) that binds its Workspaces (`plane/workspaces/`), at most one ax - A **Run** (`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG (a graph of steps
**Model** (`plane/models/`: `deepseek-flash` for chat, `openrouter-embedding` for embeddings) and with no cycles).
its own `budget: {usd, calls}`. ax (google/ax over Agent Substrate, in a single-node kind cluster - Each step is an ax **Task** (`plane/tasks/`).
on the maintainer's host) is the only executor; there is no local subprocess path. Bring-up and - A Task binds its Workspaces (`plane/workspaces/`).
the host's lessons are in [ax on this host](ops/ax/README.md); where each part runs today (the - A Task binds at most one ax **Model** (`plane/models/`). `deepseek-flash` serves chat.
reconciler and the receiver on the host, the tasks in gVisor on Agent Substrate) and what a `openrouter-embedding` serves embeddings.
separate Kubernetes cluster would take (planned) are on [Deployment](deployment.md); this page - A Task carries its own `budget: {usd, calls}`.
draws the flow.
ax is the only executor. It is google/ax, Google's controller that runs agent tasks, over Agent Substrate (its
sandboxed actor runtime). It runs in a single-node kind cluster (Kubernetes in Docker) on the maintainer's host.
There is no local subprocess path.
Where to read more:
- Bring-up and the host's lessons are in [ax on this host](ops/ax/README.md).
- [Deployment](deployment.md) shows where each part runs today. The reconciler and the receiver run on the host.
The tasks run in gVisor (a sandboxing kernel) on Agent Substrate. It also covers what a separate Kubernetes
cluster would take (planned).
- This page draws the flow.
Sources: `src/openultrasast/plane/reconciler.py`, `router.py`, `egress.py`, `runner.py`, Sources: `src/openultrasast/plane/reconciler.py`, `router.py`, `egress.py`, `runner.py`,
`generate.py`, `memory.py`, and `src/openultrasast/plane/tasks/`. `generate.py`, `memory.py`, and `src/openultrasast/plane/tasks/`.
## One task, end to end ## One task, end to end
`ousast plane run RUN.yaml` (the reconciler) does this for every ready task; `--workers` bounds `ousast plane run RUN.yaml` (the reconciler) does this for every ready task. `--workers` bounds the
the tasks in flight, and a rerun skips tasks already done. tasks in flight. A rerun skips tasks already done.
```mermaid ```mermaid
sequenceDiagram sequenceDiagram
@@ -48,44 +59,55 @@ sequenceDiagram
What each step guarantees: What each step guarantees:
- **Credentials** (`router.py`, `runner.py`). The reconciler reads the variable the Model's - **Credentials** (`router.py`, `runner.py`).
`secretKey.key` names from the operator's environment or `.env` (which never overrides an - The reconciler reads the variable that the Model's `secretKey.key` names.
exported variable). The key exists only in the start request; the runner keeps it in memory, - It reads it from the operator's environment or `.env`. `.env` never overrides an exported variable.
passes it to the command's environment, redacts it from echoed stderr and never writes it. - The key exists only in the start request.
- **Egress** (`egress.py`). One EgressPolicy per actor, deny by default, hostnames only: plain - The runner keeps it in memory and passes it to the command's environment.
HTTP to the artifact receiver, TLS passthrough to the source hosts of the bound Workspaces and - The runner redacts it from echoed stderr and never writes it.
to the hosts the bound Model declares (`openultrasast.io/egress-hosts`). It is written between - **Egress** (`egress.py`). Egress means outbound network traffic.
`ax apply` and the resume, and deleted with the actor. - Each actor gets one EgressPolicy. It denies by default and names hostnames only.
- **Inputs** (`reconciler.py`). Artifacts other tasks produced are never rendered into the - It allows plain HTTP to the artifact receiver.
manifest; the runner fetches them from the receiver after the start, and the receiver serves a - It allows TLS passthrough to the source hosts of the bound Workspaces.
running task its declared inputs only. - It allows TLS passthrough to the hosts the bound Model declares (`openultrasast.io/egress-hosts`).
- **Completion** (`runner.py`). ax has no Completed phase: the runner's delivery of the output tar - The reconciler writes it between `ax apply` and the resume. It is deleted with the actor.
is the task's completion. A completion marker stops a resumed actor from repeating billed work; - **Inputs** (`reconciler.py`).
when the marker says the delivery failed, a later boot retries the delivery alone. - Artifacts that other tasks produced are never rendered into the manifest.
- **Budgets** (`plane/budget.py`). The metered client refuses the next call once a task's `usd` or - The runner fetches them from the receiver after the start.
`calls` ceiling is reached; the task ends `unfinished` and resumes on a rerun with a larger - The receiver serves a running task its declared inputs only.
budget. Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly. - **Completion** (`runner.py`).
- **Attribution.** `ousast plane status RUN` prints per task its status, `calls`, `prompt`, - ax has no Completed phase. The runner's delivery of the output tar is the task's completion.
`cache_hit` and `output` tokens and `usd` from the tasks' `summary.json`, and writes - A completion marker stops a resumed actor from repeating billed work.
`attribution.json` in the run directory (`$OUSAST_RESULTS/plane/RUN/`, default - If the marker says the delivery failed, a later boot retries only the delivery.
- **Budgets** (`plane/budget.py`).
- The metered client refuses the next call once a task reaches its `usd` or `calls` ceiling.
- The task then ends `unfinished`. It resumes on a rerun with a larger budget.
- Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly.
- **Attribution.**
- `ousast plane status RUN` prints these per task: status, `calls`, `prompt`, `cache_hit` and
`output` tokens, and `usd`. It reads them from the tasks' `summary.json`.
- It writes `attribution.json` in the run directory (`$OUSAST_RESULTS/plane/RUN/`, default
`~/ousast-results/plane/`). `~/ousast-results/plane/`).
!!! note "The golden snapshot" !!! note "The golden snapshot"
Agent Substrate boots every new actor template once as a *golden* actor to snapshot it, with Agent Substrate boots every new actor template once as a *golden* actor and snapshots it. That boot
the same image, Task and environment, and inside the sandbox nothing tells that boot from the uses the same image, Task and environment. Inside the sandbox, nothing tells that boot from the real one.
real one. So the runner's command never starts by itself: it waits for the start request,
which the reconciler sends through `atenet-router` to the task's actor, never to the golden So the runner's command never starts by itself. It waits for the start request. The reconciler sends
one. Workspaces are prepared only after the start, so the golden boot touches no network and a that request through `atenet-router` to the task's actor, never to the golden one.
started actor already runs under its egress policy (a clone at boot failed on the live cluster
while the actor was being restored, 2026-09-29; `runner.py`, `_run_once`). The first resume of Workspaces are prepared only after the start. So the golden boot touches no network. A started actor
a new image can time out while the snapshot is built, which is why the reconciler retries it already runs under its egress policy. A clone at boot once failed on the live cluster while the actor
was being restored (2026-09-29; `runner.py`, `_run_once`).
The first resume of a new image can time out while the snapshot is built. So the reconciler retries it
for up to `OUSAST_RESUME_TIMEOUT` seconds (900). for up to `OUSAST_RESUME_TIMEOUT` seconds (900).
## The Runs: validation and loop ## The Runs: validation and loop
`ousast plane workspaces POPULATION --validation-set ...` (`generate.py`) writes one chain per `ousast plane workspaces POPULATION --validation-set ...` (`generate.py`) writes one chain per case.
case; `--loop` adds the `alerts` step per case and the loop's singleton steps. Every step except `--loop` adds the `alerts` step per case and the loop's singleton steps. Every step is model-free
the verify passes (and `roles` in harvest Runs) is model-free. except the verify passes (and `roles` in harvest Runs).
### Per case ### Per case
@@ -119,23 +141,27 @@ flowchart LR
improve --> outcome["gate.json, journal.json, and rule_policy.json only if accepted"] improve --> outcome["gate.json, journal.json, and rule_policy.json only if accepted"]
``` ```
- **Before the Run** `ousast plane run` seeds from the store (`memory.seed`): a task whose - **Before the Run**, `ousast plane run` seeds from the store (`memory.seed`).
`openultrasast.io/memory-key` (repository, pin, candidates digest, runner image digest) has - Each task has an `openultrasast.io/memory-key`: repository, pin, candidates digest and runner image digest.
stored facts gets them and is marked done, and the sandboxed `memory-snapshot` task gets the - If the store has facts for that key, the task gets them and is marked done.
store's rows after the guard. **After the Run** it ingests every delivered `memory.jsonl` - The sandboxed `memory-snapshot` task gets the store's rows after the guard.
(`ousast plane remember RUN` repeats this by hand). See [Memory](memory.md). - **After the Run**, it ingests every delivered `memory.jsonl`. `ousast plane remember RUN` repeats
- The loop's rules and its guard are described in this by hand. See [Memory](memory.md).
[Architecture](architecture.md#proposals-from-plane-memory). Adopting an accepted ledger stays a - [Architecture](architecture.md#proposals-from-plane-memory) describes the loop's rules and its guard.
maintainer commit (`src/openultrasast/plane/tasks/loop.py`). Adopting an accepted ledger stays a maintainer commit (`src/openultrasast/plane/tasks/loop.py`).
- `ousast plane harvest` writes the decision engine's harvest Runs (verify a/b with agree, and - `ousast plane harvest` writes the decision engine's harvest Runs (verify a/b with agree, and
model `roles`); `ousast plane alerts-engine` produces a Run's `alerts` from the Joern engine model `roles`).
image on the host for PHP and other languages quick mode does not cover. - `ousast plane alerts-engine` produces a Run's `alerts` from the Joern engine image on the host. It
covers PHP and other languages that quick mode does not cover.
## Where this runs in production ## Where this runs in production
ax is a Kubernetes application over Agent Substrate, every Task runs the one runner image as an actor that ax is a Kubernetes application over Agent Substrate. Every Task runs the one runner image as an actor. Each actor
occupies a whole worker, and two profiles of this code base are the target: `kind` on a laptop (today) and `k3s` occupies a whole worker. This code base targets two profiles:
in production (planned), where the same CLI submits Runs remotely and the tasks deliver to the S3 store:
- `kind` on a server VM (today).
- `k3s` in production (planned). There the same CLI submits Runs remotely, and the tasks deliver to the S3 store.
[Production topology: ax on a Kubernetes cluster](deployment.md#production-topology-ax-on-a-kubernetes-cluster) [Production topology: ax on a Kubernetes cluster](deployment.md#production-topology-ax-on-a-kubernetes-cluster)
draws it and sizes the pre-push hook at scale. draws it and sizes the pre-push hook at scale.
+113 -50
View File
@@ -1,16 +1,22 @@
# RustFS setup for the memory store # RustFS setup for the memory store
The plane's memory store can live in an S3-compatible bucket: `OUSAST_MEMORY=s3://sast-memory` The plane's memory store can live in an S3-compatible bucket. RustFS is an S3-compatible object
selects `S3Store` (`src/openultrasast/plane/memory.py`), which talks to the server through boto3. storage server.
Any S3-compatible store with bucket versioning, lifecycle rules, object tags and S3 Select works;
RustFS is the server the contract tests run against. The maintainer's server is RustFS at
`files.because-security.com`, bucket `sast-memory`, store at the bucket root. What the store keeps
there is described on [Memory](memory.md).
**The store never configures its bucket.** An admin sets the bucket up once with admin - `OUSAST_MEMORY=s3://sast-memory` selects `S3Store` (`src/openultrasast/plane/memory.py`).
credentials. The agent account the store runs as can read the bucket's settings but not change - `S3Store` talks to the server through boto3 (the AWS SDK for Python).
them, and reads and writes objects freely. Every time the store opens, it checks the setup and - Any S3-compatible store works if it has bucket versioning, lifecycle rules, object tags and S3 Select
refuses to start if anything is missing. (a server-side query over JSON objects).
- RustFS is the server the contract tests run against.
- The maintainer's server is RustFS at `files.because-security.com`. The bucket is `sast-memory`. The
store sits at the bucket root.
[Memory](memory.md) describes what the store keeps there.
**The store never configures its bucket.** An admin sets the bucket up once with admin credentials.
The store runs as an agent account. That account can read the bucket's settings but not change them.
It reads and writes objects freely. Every time the store opens, it checks the setup. It refuses to start
if anything is missing.
```mermaid ```mermaid
flowchart LR flowchart LR
@@ -35,8 +41,8 @@ aws s3api put-bucket-versioning \
--versioning-configuration Status=Enabled --versioning-configuration Status=Enabled
``` ```
**2. Expire raw run outputs under `runs/` after 30 days.** Everything else (`repos/`, `facts/`, **2. Expire raw run outputs under `runs/` after 30 days.** Everything else is kept: `repos/`, `facts/`,
`index.jsonl` and the blobs) is kept, so the rule must be a plain prefix filter on `runs/`. `index.jsonl` and the blobs. So the rule must be a plain prefix filter on `runs/`.
```bash ```bash
aws s3api put-bucket-lifecycle-configuration \ aws s3api put-bucket-lifecycle-configuration \
@@ -46,9 +52,10 @@ aws s3api put-bucket-lifecycle-configuration \
``` ```
`put-bucket-lifecycle-configuration` **replaces every rule on the bucket**. If the bucket already `put-bucket-lifecycle-configuration` **replaces every rule on the bucket**. If the bucket already
has rules, merge this one into them first. The rule id `ousast-runs-expiry` and the 30 days are the has rules, merge this one into them first.
store's defaults (`RUNS_RULE_ID`, `RUNS_EXPIRE_DAYS`); the store accepts any enabled, untagged
prefix rule on `runs/` with a number of days. The rule id `ousast-runs-expiry` and the 30 days are the store's defaults (`RUNS_RULE_ID`,
`RUNS_EXPIRE_DAYS`). The store accepts any enabled, untagged prefix rule on `runs/` with a number of days.
To read the result back: To read the result back:
@@ -59,8 +66,8 @@ aws s3api get-bucket-lifecycle-configuration --endpoint-url https://files.becaus
## The agent policy ## The agent policy
Attach this policy to the agent account (the account whose keys go into `.env`). It is the Attach this policy to the agent account. That is the account whose keys go into `.env`. It is the
complete policy: nothing else is needed, and it grants no `Put*` on versioning or lifecycle. complete policy, and nothing else is needed. It grants no `Put*` on versioning or lifecycle.
```json ```json
{ {
@@ -102,14 +109,14 @@ complete policy: nothing else is needed, and it grants no `Put*` on versioning o
| `BucketSettingsReadOnly` | `arn:aws:s3:::sast-memory` | The agent can list the bucket and its versions and *read* its location, versioning and lifecycle settings, which is what the startup check needs; it cannot change them. | | `BucketSettingsReadOnly` | `arn:aws:s3:::sast-memory` | The agent can list the bucket and its versions and *read* its location, versioning and lifecycle settings, which is what the startup check needs; it cannot change them. |
| `ObjectsReadWrite` | `arn:aws:s3:::sast-memory/*` | The agent reads, writes and deletes objects and their versions and reads and writes object tags; this covers row objects, blobs, the index, the Select probe, and the `kind` tag the store filters by. | | `ObjectsReadWrite` | `arn:aws:s3:::sast-memory/*` | The agent reads, writes and deletes objects and their versions and reads and writes object tags; this covers row objects, blobs, the index, the Select probe, and the `kind` tag the store filters by. |
So the agent writes objects freely but cannot change bucket settings: turning versioning off or So the agent writes objects freely but cannot change bucket settings. Turning versioning off or
removing the expiry rule needs the admin. removing the expiry rule needs the admin.
## The `.env` settings ## The `.env` settings
The store reads its endpoint and credentials from `.env` in the working directory or from the The store reads its endpoint and credentials from `.env` in the working directory or from the
environment (`s3_settings`), never from a manifest, and never prints them. The names are the ones environment (`s3_settings`). It never reads them from a manifest. It never prints them. The names are
the AWS SDKs and CLI use for credentials, so one `.env` serves both the store and `aws s3api`. the ones the AWS SDKs and CLI use for credentials. So one `.env` serves both the store and `aws s3api`.
| Variable | Value for this server | Notes | | Variable | Value for this server | Notes |
| --- | --- | --- | | --- | --- | --- |
@@ -121,26 +128,33 @@ the AWS SDKs and CLI use for credentials, so one `.env` serves both the store an
| `S3_REGION` | optional (`us-east-1` when unset) | the SigV4 signing region; spares a GetBucketLocation call | | `S3_REGION` | optional (`us-east-1` when unset) | the SigV4 signing region; spares a GetBucketLocation call |
| `S3_BUCKET` | `sast-memory` | the bucket `s3://` names when the URL has none; also the real-server tests' default bucket | | `S3_BUCKET` | `sast-memory` | the bucket `s3://` names when the URL has none; also the real-server tests' default bucket |
The client signs with SigV4 and uses path-style addressing (`https://host/bucket/key`), which The client signs with SigV4 (AWS's request-signing scheme). It uses path-style addressing
RustFS needs. The SDK's own checksum trailers are off, because not every S3-compatible server (`https://host/bucket/key`), which RustFS needs. The SDK's own checksum trailers are off. Not every
accepts them. S3-compatible server accepts them.
- **Never commit `.env`.** It is gitignored (`.gitignore`: `.env`, `.env.*`, except - **Never commit `.env`.** It is gitignored (`.gitignore`: `.env`, `.env.*`, except
`.env.example`). `.env.example` lists these names with empty values. `.env.example`). `.env.example` lists these names with empty values.
- **`.env` never overrides a variable already exported in your shell** (`config.load_dotenv`). - **`.env` never overrides a variable already exported in your shell** (`config.load_dotenv`).
A stale `export AWS_SECRET_ACCESS_KEY=...` in your profile silently wins over the file; `unset` A stale `export AWS_SECRET_ACCESS_KEY=...` in your profile silently wins over the file. `unset`
it when a change in `.env` seems to have no effect. it when a change in `.env` seems to have no effect.
- The store needs the `s3` extra (`uv sync --extra s3`, which installs boto3); without it opening - The store needs the `s3` extra (`uv sync --extra s3`, which installs boto3). These are the
the store fails with `OUSAST_MEMORY=s3://... needs boto3: install openultrasast[s3]`. A missing startup errors:
variable fails with `OUSAST_MEMORY=s3://... needs S3_ENDPOINT, AWS_ACCESS_KEY_ID, - Without the extra, opening the store fails with
AWS_SECRET_ACCESS_KEY in .env or the environment` (naming only the missing ones); an endpoint `OUSAST_MEMORY=s3://... needs boto3: install openultrasast[s3]`.
without a scheme fails with `S3_ENDPOINT must be a URL with a scheme (https://host[:port])`. - A missing variable fails with `OUSAST_MEMORY=s3://... needs S3_ENDPOINT, AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY in .env or the environment`. The message names only the missing ones.
- An endpoint without a scheme fails with
`S3_ENDPOINT must be a URL with a scheme (https://host[:port])`.
## What the store verifies at startup ## What the store verifies at startup
`S3Store.verify_bucket()` runs once per process for each bucket and prefix. It reads the `S3Store.verify_bucket()` runs once per process for each bucket and prefix. It does four things:
versioning status and the lifecycle rules, writes a two-line probe object at
`_probe/select.jsonl` (under the prefix, if any), queries it with S3 Select and reads its tags. 1. It reads the versioning status.
2. It reads the lifecycle rules.
3. It writes a two-line probe object at `_probe/select.jsonl` (under the prefix, if any).
4. It queries the probe with S3 Select and reads its tags.
Every missing piece is collected into one error: Every missing piece is collected into one error:
```text ```text
@@ -148,8 +162,8 @@ memory s3://sast-memory: the bucket is not set up for the store. The store never
- <one line per missing piece> - <one line per missing piece>
``` ```
The lines, exactly as the code writes them for this bucket at the root (`<failure>` is the The table lists the lines exactly as the code writes them for this bucket at the root. `<failure>` is
server's error code and the first 200 characters of its message): the server's error code and the first 200 characters of its message.
| Check | Refusal line | | Check | Refusal line |
| --- | --- | | --- | --- |
@@ -161,32 +175,81 @@ server's error code and the first 200 characters of its message):
| tags unreadable | `object tags cannot be read on _probe/select.jsonl (<failure>): the store filters row objects by their kind tag, so the agent needs s3:GetObjectTagging on arn:aws:s3:::sast-memory/*` | | tags unreadable | `object tags cannot be read on _probe/select.jsonl (<failure>): the store filters row objects by their kind tag, so the agent needs s3:GetObjectTagging on arn:aws:s3:::sast-memory/*` |
| S3 Select missing | `S3 Select (SelectObjectContent over JSON Lines) does not answer on _probe/select.jsonl (<failure>). The store requires it and has no local fallback: use a server with S3 Select (RustFS has it; not every S3-compatible server does) and let the agent s3:PutObject and s3:GetObject` | | S3 Select missing | `S3 Select (SelectObjectContent over JSON Lines) does not answer on _probe/select.jsonl (<failure>). The store requires it and has no local fallback: use a server with S3 Select (RustFS has it; not every S3-compatible server does) and let the agent s3:PutObject and s3:GetObject` |
The admin commands inside the messages use `--endpoint-url "$S3_ENDPOINT"`, which is the full The admin commands inside the messages use `--endpoint-url "$S3_ENDPOINT"`. That is the full URL from
URL from `.env`, so they run as written once `.env` is exported. `.env`. So they run as written once `.env` is exported.
There is **no fallback**: the store does not start in a degraded mode, does not drop to a local There is **no fallback**. The store does not:
store, and does not fetch and filter objects itself.
- start in a degraded mode;
- drop to a local store;
- fetch and filter objects itself.
## The RustFS Select limit and the fixed-schema rule ## The RustFS Select limit and the fixed-schema rule
RustFS's S3 Select infers an object's JSON schema from its leading rows. A `where` field that is RustFS's S3 Select infers an object's JSON schema from its leading rows. A `where` field absent from
absent from those rows is unbound, even if a later row has it: the query fails with those rows is unbound, even if a later row has it. The query then fails with
`EvaluatorBindingDoesNotExist` instead of matching (measured 2026-10-02 with the field first `EvaluatorBindingDoesNotExist` instead of matching (measured 2026-10-02 with the field first
present at row 5000). The store does not read that as "no rows", since that answer could silently present at row 5000).
drop matches; it raises:
The store does not read that failure as "no rows". That answer could silently drop matches. It raises:
```text ```text
memory s3://sast-memory: S3 Select on <key> failed (EvaluatorBindingDoesNotExist: ...) (a field of [<fields>] is absent from the object's leading rows, which the server reads as its schema); the store has no local fallback memory s3://sast-memory: S3 Select on <key> failed (EvaluatorBindingDoesNotExist: ...) (a field of [<fields>] is absent from the object's leading rows, which the server reads as its schema); the store has no local fallback
``` ```
The rule that follows: a field the store filters on is present in every row of its kind from The rule that follows has two parts (see [Memory](memory.md#queryable-fields-per-kind)):
the first row on, as `""` where it does not apply; each row kind has a fixed set of queryable
fields, enforced at write time (see [Memory](memory.md#queryable-fields-per-kind)). - A field the store filters on is present in every row of its kind from the first row on. It is `""`
where it does not apply.
- Each row kind has a fixed set of queryable fields. The store enforces it at write time.
## Checks against the real server ## Checks against the real server
The contract tests that talk to a real server run only with `OUSAST_MEMORY_TEST_S3=1` (bucket The contract tests that talk to a real server run only with `OUSAST_MEMORY_TEST_S3=1`. They use the
`OUSAST_MEMORY_TEST_BUCKET`, else `S3_BUCKET`). They verify the bucket at its root, write only bucket `OUSAST_MEMORY_TEST_BUCKET`, else `S3_BUCKET`. The tests:
under a fresh `contract-<id>/` prefix, and remove every version and delete marker under it
afterwards; they never configure the bucket. More on the host setup: - verify the bucket at its root;
[ax on this host](ops/ax/README.md). - write only under a fresh `contract-<id>/` prefix;
- remove every version and delete marker under it afterwards;
- never configure the bucket.
More on the host setup: [ax on this host](ops/ax/README.md).
## Scoped account for engine workers
Create a **new account** for the k3s engine Deployment. Attach only this policy.
Do not give it the existing memory-store account's policies.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListEnginePrefixes",
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::sast-memory",
"Condition": {
"StringLike": {
"s3:prefix": ["engine-queue/", "engine-queue/*", "engine-results/", "engine-results/*"]
}
}
},
{
"Sid": "EngineObjectsOnly",
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
"Resource": [
"arn:aws:s3:::sast-memory/engine-queue/*",
"arn:aws:s3:::sast-memory/engine-results/*"
]
}
]
}
```
The cluster can read and write only queued engine inputs, claims, and results.
It cannot list or read the rest of the memory store. It cannot change bucket
settings. The host queue dispatcher can use the same scoped account. Store its
keys in the `ousast-engine-s3` Secret; no key values belong in manifests.
Deployment and coordinator commands are in `ops/k8s/engine/README.md`.
+330 -39
View File
@@ -1,13 +1,21 @@
# Scanning # Scanning
`ousast scan PATH --mode quick|standard|deep` scans a checkout; `ousast pre-push` checks only This page shows the scan flows.
what a push (or an explicit `--base`/`--head` pair) changes. The stage-by-stage pipeline,
evidence ladder and score are in [Architecture](architecture.md); this page shows the flows. - `ousast scan PATH --mode quick|standard|deep` scans a checkout.
- `ousast pre-push` checks only what a push changes, or what an explicit `--base`/`--head` pair
changes.
The stage-by-stage pipeline, the evidence ladder and the score are in
[Architecture](architecture.md).
## The three scan modes ## The three scan modes
The stages a mode requests are `MODE_STAGES` in `src/openultrasast/stages.py`: `quick` runs The stages each mode requests are listed in `MODE_STAGES` in `src/openultrasast/stages.py`:
STATIC, `standard` adds MAP, `deep` adds REGRESS.
- `quick` runs STATIC.
- `standard` adds MAP.
- `deep` adds REGRESS.
```mermaid ```mermaid
flowchart TD flowchart TD
@@ -38,7 +46,7 @@ Sources: `README.md` (scan modes), `src/openultrasast/cli.py` (`run_stage` calls
## Quick rules per language ## Quick rules per language
Rules run only against files of their language. The bundled ruleset lives in Each rule runs only against files of its language. The bundled ruleset lives in
`src/openultrasast/ruleset/<language>/rules.toml`. `src/openultrasast/ruleset/<language>/rules.toml`.
| Language | Classes (CWE) | | Language | Classes (CWE) |
@@ -49,21 +57,31 @@ Rules run only against files of their language. The bundled ruleset lives in
| Java + Groovy templates | command injection (78), SQLi (89), weak hash (327), deserialization (502), unescaped template XSS (79) | | Java + Groovy templates | command injection (78), SQLi (89), weak hash (327), deserialization (502), unescaped template XSS (79) |
| PHP | SQLi (89), command injection (78), file inclusion (98), path (22), open redirect (601), SSRF (918); code eval (95), unserialize (502) and echo XSS (79) ship as `shadow` rules | | PHP | SQLi (89), command injection (78), file inclusion (98), path (22), open redirect (601), SSRF (918); code eval (95), unserialize (502) and echo XSS (79) ship as `shadow` rules |
The detection gate (`python -m openultrasast.gate`) enforces at least 90% recall and under 10% The detection gate (`python -m openultrasast.gate`) enforces two limits per language on the
false positives per language on the bundled corpora in `benchmarks/manifests/`. Those corpora are bundled corpora in `benchmarks/manifests/`: at least 90% recall and under 10% false positives.
cheat-sheet fixtures: they catch regressions, they do not estimate recall on real code. **PHP is
not in the gate**; its numbers are in-sample development numbers - Those corpora are cheat-sheet fixtures. They catch regressions. They do not estimate recall on
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). See `README.md` for the real code.
gate's per-language output. - **PHP is not in the gate**. Its numbers are in-sample development numbers
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`).
See `README.md` for the gate's per-language output.
## The Joern engine ## The Joern engine
The model layer (`src/openultrasast/model/`, `src/openultrasast/cpg/`) is the core of `standard`. The model layer (`src/openultrasast/model/`, `src/openultrasast/cpg/`) is the core of `standard`.
It builds one code property graph per scan with Joern (`joern-parse`, then batched It builds one code property graph per scan with Joern. A code property graph is a graph of the
`joern --script` queries from `cpg/queries/*.sc`) and sends each question to the arbiter of its program's syntax, control flow and data flow. The build runs `joern-parse`, then batched
type: **taint** (source to sink with no discharging sanitizer), **dominance** (does a guard govern `joern --script` queries from `cpg/queries/*.sc`.
the obligated operation) and **configuration** (constant abstraction over a value). Taint facts
ship for C, Java, JavaScript, PHP and Python (`src/openultrasast/ruleset/semantic/*.toml`). The layer sends each question to the arbiter (the decider) of its type:
- **taint**: a flow from source to sink with no discharging sanitizer;
- **dominance**: does a guard govern the obligated operation;
- **configuration**: constant abstraction over a value.
Taint facts ship for C, Java, JavaScript, PHP and Python
(`src/openultrasast/ruleset/semantic/*.toml`).
```mermaid ```mermaid
flowchart LR flowchart LR
@@ -79,25 +97,33 @@ flowchart LR
rung -- "residual question only" --> judge["LLM judge: confirm or contradict, never create a flow"] rung -- "residual question only" --> judge["LLM judge: confirm or contradict, never create a flow"]
``` ```
The Docker image ships Joern and `php-cli` (which the PHP frontend needs); without Joern the The Docker image ships Joern and `php-cli`. The PHP frontend needs `php-cli`. Without Joern, the
layer records `cpg_unavailable` and the scan is otherwise unchanged. Engine installation and the layer records `cpg_unavailable`; the rest of the scan is unchanged.
Joern features used are in [Engine and pre-push](ops/README.md).
How the graph is built, queried and refined, with the records behind each refinement, is in - Engine installation and the Joern features used: [Engine and pre-push](ops/README.md).
[Detection techniques](detection-techniques.md). - How the graph is built, queried and refined, with the records behind each refinement:
[Detection techniques](detection-techniques.md).
## Framework priors ## Framework priors
Rule and taint-fact entries that know a framework or library carry a `framework` or `library` tag Some rule and taint-fact entries know a framework or library. They carry a `framework` or
naming a row of `src/openultrasast/ruleset/frameworks.toml` (for example `wordpress`, `flask`, `library` tag. The tag names a row of `src/openultrasast/ruleset/frameworks.toml` (for example
`django`, `express`, `spring`). The loaders take `priors=`: `"all"` (the default and today's `wordpress`, `flask`, `django`, `express`, `spring`).
behaviour), `"off"` (language-level knowledge only) or a set of ids, so the contribution of
framework knowledge can be measured and switched off. The decision engine's first slice ran with The loaders take `priors=` with one of three values:
priors off ([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)).
- `"all"`: the default and today's behaviour;
- `"off"`: language-level knowledge only;
- a set of ids.
So the contribution of framework knowledge can be measured and switched off. The decision
engine's first slice ran with priors off
([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)).
## Pre-push: the delta check (experimental) ## Pre-push: the delta check (experimental)
`ousast pre-push` analyses the commits a `git push` would publish, or an explicit `ousast pre-push` analyses the commits a `git push` would publish, or an explicit
`--base`/`--head` pair, and compares head with base so that only new or worsened defects are `--base`/`--head` pair. It compares head with base. Only new or worsened defects become
candidates. Sources: `ousast pre-push --help`, `src/openultrasast/push/`, candidates. Sources: `ousast pre-push --help`, `src/openultrasast/push/`,
[Engine and pre-push](ops/README.md#explicit-local-replay). [Engine and pre-push](ops/README.md#explicit-local-replay).
@@ -105,23 +131,288 @@ candidates. Sources: `ousast pre-push --help`, `src/openultrasast/push/`,
flowchart TD flowchart TD
input["git push refs on stdin, or --base / --head"] --> resolve["Resolve both commit identities"] input["git push refs on stdin, or --base / --head"] --> resolve["Resolve both commit identities"]
resolve --> snap["Materialise tracked bytes in private scratch storage"] resolve --> snap["Materialise tracked bytes in private scratch storage"]
snap --> regions["Discover changed regions, up to --max-regions"] snap --> quick["Quick rules on the changed head lines (advisory, under 1 s)"]
regions --> engine["Evidence ranker and arbiter on head"] quick --> eng{"--engine"}
eng -- "off" --> art
eng -- "background" --> bg["Detached engine run; result shown on the next push"]
bg --> art
eng -- "inline, the default" --> engine["Engine on head: changed regions up to --max-regions, ranker, arbiter"]
engine --> cmp["Targeted base comparison: new or worsened only"] engine --> cmp["Targeted base comparison: new or worsened only"]
cmp --> admit{"Admission policy: capability registry"} cmp --> admit{"Admission policy: capability registry"}
admit -- "registry empty today" --> diag["Diagnostic candidates in the artifact"] admit -- "registry empty today" --> diag["Advisory engine findings, not alerts"]
admit -- "an enabled capability" --> alert["Normal alert"] admit -- "an enabled capability" --> alert["Normal alert"]
diag --> art["JSON artifact outside the repository"] diag --> art["Terminal lines + JSON artifact outside the repository"]
alert --> art alert --> art
art --> exit{"--mode"} art --> exit{"--mode"}
exit -- "advisory, the default" --> allow["Exit 0: allow"] exit -- "advisory, the default" --> allow["Exit 0: allow"]
exit -- "blocking, explicit opt-in" --> block["May block the push"] exit -- "blocking, explicit opt-in" --> block["May block the push"]
``` ```
- `--deadline` (30 s by default) bounds preparation, both revisions and reporting together; - `--deadline` (30 s by default) bounds preparation, the quick rules, both revisions and reporting together.
`--incomplete-coverage block` opts into failing when coverage is incomplete. - The quick rules run first on the changed lines. Their matches are shown as advisory pattern
- The default capability registry is empty pending independent qualification, so no normal alert matches, never as alerts, and never block.
is emitted today; every candidate stays diagnostic. - The default capability registry is empty until independent qualification passes. So no normal
alert is emitted today. An engine finding the push introduced is shown as an advisory engine
finding; every other candidate stays diagnostic in the artifact.
- A model is called only with an explicit `--model-config`. - A model is called only with an explicit `--model-config`.
- Hook installation (`ops/install-pre-push`) and the private cache (`--cache-dir`) are documented - The private cache (`--cache-dir`) and the manual install recipe are documented in
in [Engine and pre-push](ops/README.md#experimental-pre-push-integration). [Engine and pre-push](ops/README.md#experimental-pre-push-integration).
### First run: install the hook on any repository
The hook is advisory. It never rejects a push unless you set `OUSAST_PUSH_MODE=blocking`. A check
that cannot run says so in one line and lets the push through: a missing `ousast`, a missing
Docker image, an invalid setting, a crash.
1. Install the tool. The package is not on PyPI. Install it from the repository (or from a
release wheel on the GitHub releases page):
```bash
pip install "git+https://github.com/norandom/OpenUltraSAST"
```
2. Install the hook in the repository you want checked. Git's effective hook directory is used,
including `core.hooksPath`, and that setting is never edited:
```bash
cd /path/to/your/repository
ousast pre-push install # refuses if a pre-push hook already exists
ousast pre-push install --force # moves an existing hook to pre-push.before-ousast and chains it
```
3. Push as usual. Without the Joern engine, you get the quick rules and one line saying the
engine did not run. For the engine, choose one:
- a host install with `joern-parse` on the PATH (`pyinfra @local ops/joern.py`, see
[Engine and pre-push](ops/README.md));
- Docker: `docker pull ghcr.io/norandom/openultrasast:2.0.1`, then
`ousast pre-push install --docker`. The wrapper `ousast-docker` is installed beside the
hook and runs the check in the container (repository read-only, no network,
`--memory 3g`). The container runs the code of its image's version: the quick-rule
tier, the plain skip lines and `--engine` need an image built from a version that has them
(`docker build -t openultrasast:dev .`, then `OUSAST_IMAGE=openultrasast:dev`).
4. Remove it with `ousast pre-push uninstall`. Only a hook this tool wrote is removed, and a
chained previous hook is restored.
A push on the 2026-10-02 proof repository (serve-static, a new `debug.js` that evaluates
`req.query.code`, no Joern on the host) prints:
```text
Quick rules: 2 pattern match(es) on changed code (1 file(s), 87 bytes read). Advisory: not verified by the engine.
- js-eval (CWE-95, critical) at debug.js:2: Dynamic JavaScript evaluation needs review
- js-xss-response (CWE-79, medium) at debug.js:2: Unsanitized HTTP response reflection needs review
No actionable defects reported; analysis is incomplete.
Skipped: the changed code has no counterpart in the base revision to compare against
Skipped: engine did not run: the code graph could not be built (is Joern installed? ...)
Skipped: no base revision was analysed, so no engine finding can be called new or worsened
Notice: Engine checks did not run. Install Joern, or run the hook through ops/ousast-docker. Review the details.
Details: /home/you/.cache/openultrasast/push/push-12345.json
```
The hook reads these settings from the environment of `git push`:
| Variable | Default | Effect |
| --- | --- | --- |
| `OUSAST_PUSH_DEADLINE` | `30` | seconds for the whole check |
| `OUSAST_PUSH_MODE` | `advisory` | `blocking` is the only way a result can reject a push |
| `OUSAST_INCOMPLETE_COVERAGE` | `allow` | with `blocking`: `block` also rejects incomplete coverage |
| `OUSAST_PUSH_ENGINE` | `inline` | `background`: return after the quick rules, run the engine detached; `off`: quick rules only |
| `OUSAST_COMPARISON_BASE` | unset | a local ref to compare a new branch against |
| `OUSAST_ARTIFACT_DIR` | `~/.cache/openultrasast/push` | where the JSON results go; outside the repository |
| `OUSAST_COMMAND` | `ousast` | the command the hook runs, for example the Docker wrapper |
| `OUSAST_IMAGE` | `ghcr.io/norandom/openultrasast:2.0.1` | the Docker wrapper's image |
| `OUSAST_DOCKER_MEMORY` | `3g` | the Docker wrapper's memory limit |
**A new branch.** Git gives the hook no remote base for a branch the remote does not have yet.
The check says so ("new branch: the remote has no base to compare against"), runs the quick rules
over the files the branch's last commit changed instead, and says that too. Set
`OUSAST_COMPARISON_BASE=main` to compare a new branch against your local `main`.
**The engine and the deadline.** The engine does not fit a 30 s deadline today. The 2026-10-02
audit measured 260 s on a 26-file repository; graph reuse is the planned fix. When it cannot
finish, one line says so. With `OUSAST_PUSH_ENGINE=background`, the hook returns after the quick
rules and starts one detached engine run (`--background-deadline`, 900 s by default). The next push
in the same repository prints `Engine result for <head> from the previous push: ...` once. Only
one background run happens at a time. The Docker wrapper cannot run the engine in the background,
because the container ends with the hook.
**The artifact.** Every run writes one JSON file, and the terminal prints its path after
`Details:`. It records the following:
- `snapshots[].bytes_read`: proof that the input was read;
- `quick_tier`: the quick-rule matches, with the files and bytes read;
- `admission.coverage_reasons`: the raw skip reasons;
- `admission.dispositions`: every engine candidate and its vetoes;
- `engine_background`: background runs started or shown;
- `timings`.
Files accumulate, one per push; delete old ones when you like.
### Skip reasons
The terminal prints one `Skipped:` line per distinct reason in plain words. The artifact keeps the
raw reason. Repository paths are hex-encoded there, and per-question reasons end in a hash.
| Raw reason (artifact) | Terminal line says | What to do |
| --- | --- | --- |
| `cpg_build_failed`, `cpg_unavailable` | the engine did not run: the code graph could not be built | install Joern, or use the Docker wrapper |
| `cpg_empty` | the engine read no code | check that the changed files are in a supported language |
| `frontend_unsupported` | no engine frontend for a language in this push | none: that language gets the quick rules at most |
| `language_not_covered:<lang>:<changed>:<total>` | language not covered: `<lang>` (n changed files, m in the repository) | none: neither the engine nor the quick rules cover it, so silence is not a clean result |
| `deadline_exhausted` | the engine did not finish within the deadline | raise `OUSAST_PUSH_DEADLINE`, or set `OUSAST_PUSH_ENGINE=background` |
| `change_context_deadline_exhausted` | the deadline ran out while relating the change to the code | as above |
| `report_deadline_exhausted` | the deadline ran out while saving the details | as above |
| `engine_off`, `engine_background` | the engine was switched off, or runs after the push | none |
| `resolution:missing_base` | new branch: the remote has no base to compare against | set `OUSAST_COMPARISON_BASE` |
| `resolution:unsupported_base`, `resolution:no_merge_base`, `resolution:ambiguous_base`, `resolution:missing_target`, `resolution:unsupported_target`, `resolution:resolution_error` | the pushed ref or its remote base could not be resolved locally | fetch the remote, then push again |
| `base_comparison_unavailable`, `base_unavailable` | no base revision was analysed, so nothing can be called new | see the line before it for the cause |
| `base_counterpart_unresolved` | the changed code has no counterpart in the base to compare against | none: expected for new files |
| `<path>:ambiguous_line_correspondence`, `<path>:line_correspondence_unavailable`, `<path>:ambiguous_path_rename_correspondence` | changed lines or a renamed file could not be matched to the base | none: findings there stay diagnostic |
| `unmatched_added_deleted_path_correspondence` | files were both added and deleted, so a move cannot be told from a new file | push the move separately from other edits |
| `declaration_paths_unavailable` | project declaration files could not be listed | rerun |
| `capability_unavailable`, `capability_disabled`, `capability_unevaluated`, `capability_ambiguous` | engine findings are not qualified as alerts yet | none: they show as advisory engine findings |
| `eligibility:<reason>` | the capability registry does not match this build | reinstall a consistent build |
| `comparison_unknown`, `base_incomplete` and the other novelty reasons | whether this push introduced some engine findings could not be established | none: they stay diagnostic |
| `dynamic_external_or_depth_context_unresolved:...` | n engine checks depend on dynamic or external code the engine cannot follow | none |
| `query_failed`, `query_budget_reserved`, `query_too_expensive`, `regions_truncated`, `files_unparsed` | some engine questions were not answered | raise the deadline or `--max-regions` |
| `cross_partition_semantics_unresolved` | flows between languages are not followed | none |
| `dependency_unresolved`, `change_unsupported` | engine findings depend on unresolved code or have no supported link to the change | none |
| `snapshot:<reason>` | some files could not be materialised for analysis | see the reason (symlinks, submodules, LFS) |
| `<stage>_failed:<Error>`, `push_input_failed:<Error>` | the named step failed | rerun; report it if it repeats |
A reason not in this table is still printed as its own words, followed by "see the details file".
### In CI
The same check runs on a pushed range in CI. This GitHub Actions job is advisory: it never fails
the build, and it uploads the artifact.
```yaml
name: ousast pre-push (advisory)
on: push
jobs:
pre-push:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install "git+https://github.com/norandom/OpenUltraSAST"
- name: Check what this push changed
if: github.event.before != '0000000000000000000000000000000000000000'
run: |
ousast pre-push . --base ${{ github.event.before }} --head ${{ github.sha }} \
--artifact "$RUNNER_TEMP/ousast-pre-push.json" --engine off --deadline 120 || true
- uses: actions/upload-artifact@v4
if: always()
with:
name: ousast-pre-push
path: ${{ runner.temp }}/ousast-pre-push.json
```
A branch's first push has no `github.event.before` to compare against, and the job skips it. For
the engine in CI, run the job in the container image instead of `pip install`, and drop
`--engine off`.
### `ousast pre-push --help`
```text
usage: ousast pre-push [-h] [--base BASE] [--head HEAD] [--remote NAME URL]
[--comparison-base COMPARISON_BASE] [--model-config MODEL_CONFIG]
[--prior-hook PRIOR_HOOK] --artifact ARTIFACT [--cache-dir CACHE_DIR]
[--deadline DEADLINE] [--cancellation-allowance CANCELLATION_ALLOWANCE]
[--max-regions MAX_REGIONS] [--mode {advisory,blocking}]
[--incomplete-coverage {allow,block}] [--engine {inline,background,off}]
[--background-deadline BACKGROUND_DEADLINE]
[--experimental-declarations EXPERIMENTAL_DECLARATIONS]
[--experimental-record-vetoes]
path
Check what a push changes: quick rules on the changed lines, then the Joern engine on head and
base within one deadline. Advisory: exit 0 unless --mode blocking. Install the Git hook with
'ousast pre-push install [REPO] [--force] [--docker]'; remove it with 'ousast pre-push uninstall
[REPO]'.
positional arguments:
path the repository to check (the hook passes '.')
options:
-h, --help show this help message and exit
--base BASE explicit replay: the base revision (with --head; instead of --remote)
--head HEAD explicit replay: the head revision to check against --base
--remote NAME URL Git hook mode: the remote name and URL Git passes to pre-push; the push
input is read from stdin
--comparison-base COMPARISON_BASE
hook mode: a local ref to compare a NEW branch against (the remote has no
base for it); the hook reads OUSAST_COMPARISON_BASE
--model-config MODEL_CONFIG
explicit optional witness selection endpoint configuration
--prior-hook PRIOR_HOOK
explicitly chain an existing hook with the same input and arguments
--artifact ARTIFACT where to write the complete JSON result; must be outside the repository
--cache-dir CACHE_DIR
reuse compatible local artifacts in a private directory outside the
repository
--deadline DEADLINE seconds for the whole check: resolution, quick rules, engine and reporting
(default 30; hook: OUSAST_PUSH_DEADLINE)
--cancellation-allowance CANCELLATION_ALLOWANCE
seconds after the deadline to stop work and save the result (default 2)
--max-regions MAX_REGIONS
most code regions the engine examines, highest-ranked first (default 500)
--mode {advisory,blocking}
advisory (default) never blocks; blocking is the explicit opt-in that
rejects a push for an admitted alert (hook: OUSAST_PUSH_MODE)
--incomplete-coverage {allow,block}
with --mode blocking only: also block when coverage is incomplete (hook:
OUSAST_INCOMPLETE_COVERAGE)
--engine {inline,background,off}
inline (default): run the Joern engine inside the deadline; background:
return after the quick rules and run the engine detached, showing its
result on the next run; off: quick rules only
--background-deadline BACKGROUND_DEADLINE
seconds the detached engine run of --engine background may take (default
900)
--experimental-declarations EXPERIMENTAL_DECLARATIONS
explicit --base/--head replay only: unreviewed capability declarations
used to render the experimental evaluation explanation; never consulted by
admission and cannot enable any capability
--experimental-record-vetoes
explicit --base/--head replay only: record each comparison and admission
veto per raw finding instead of applying it; output is labeled
experimental, changes no result and enables no hook capability
```
```text
usage: ousast pre-push install [-h] [--force] [--docker] [repository]
Install the advisory OpenUltraSAST pre-push hook into a repository's effective hook directory
(core.hooksPath is honoured, never edited).
positional arguments:
repository the repository (default: current directory)
options:
-h, --help show this help message and exit
--force install: replace our hook, or move a foreign hook to pre-push.before-ousast and
chain it; uninstall: remove a hook this tool did not write
--docker also install the ousast-docker wrapper beside the hook, so the check runs in the
container image
```
```text
usage: ousast pre-push uninstall [-h] [--force] [repository]
Remove the OpenUltraSAST pre-push hook and restore a chained previous hook.
positional arguments:
repository the repository (default: current directory)
options:
-h, --help show this help message and exit
--force install: replace our hook, or move a foreign hook to pre-push.before-ousast and
chain it; uninstall: remove a hook this tool did not write
```
+74 -51
View File
@@ -1,78 +1,101 @@
# Threat model & hardening # Threat model & hardening
OpenUltraSAST analyzes **untrusted code**. This document states what it trusts, what OpenUltraSAST analyzes **untrusted code**. This page states what it trusts and what it does not.
it does not, and the controls that keep a scan safe to run in CI and safe to share. It also lists the controls that keep a scan safe to run in CI and safe to share.
## Trust boundaries ## Trust boundaries
- **Scanned code is untrusted input.** Quick and standard modes never import, build or run - **Scanned code is untrusted input.** Quick and standard modes never import, build or run the
the target: quick reads source as text, and standard additionally parses it into a code target.
property graph with Joern (`joern-parse`, a parser, not a build) and may send code excerpts - Quick mode reads source as text.
to the configured model. Only `deep` mode executes target-derived code: its REGRESS stage - Standard mode also parses it into a code property graph (a graph of the program's syntax,
loads promoted candidates inside the Docker sandbox below, and it runs only when `docker control flow and data flow) with Joern. It uses `joern-parse`, a parser, not a build.
Standard mode may send code excerpts to the configured model.
- Only `deep` mode executes target-derived code. Its REGRESS stage loads promoted candidates
inside the Docker sandbox described below. It runs only when `docker
info` succeeds (otherwise the stage is skipped and recorded as `sandbox_unavailable`). info` succeeds (otherwise the stage is skipped and recorded as `sandbox_unavailable`).
- **The ruleset and CWE policy are trusted, governed data.** They change only through - **The ruleset and CWE policy are trusted, governed data.** They change in only two ways:
reviewed commits or the bounded self-improvement loop (`ousast improve`), which can through reviewed commits, or through the bounded self-improvement loop (`ousast improve`). The
flip rule *status* and tune score constants but can never edit pattern text or the loop can flip rule *status* and tune score constants. It can never edit pattern text or the
authoritative 0–5 severity. See [architecture.md](architecture.md#self-improvement). authoritative 0–5 severity. See [architecture.md](architecture.md#self-improvement).
- **Provider credentials are the operator's.** The core install (PyYAML only) makes no - **Provider credentials are the operator's.** The core install (PyYAML only) makes no network
network calls on its own. A local scan calls a model only in standard or deep mode, and calls on its own.
only when a key is available: the model layer asks the residual question the graph cannot - A local scan calls a model only in standard or deep mode, and only when a key is available.
settle, and `[models] hunter` enables the tool hunter. LLM calls go to DeepSeek The model layer asks the residual question the graph cannot settle. `[models] hunter`
(`DEEPSEEK_API_KEY`); OpenRouter (`OPENROUTER_API_KEY`) serves embeddings. Keys come from enables the tool hunter.
the environment or a gitignored `.env`, which never overrides an exported variable. The - LLM calls go to DeepSeek (`DEEPSEEK_API_KEY`). OpenRouter (`OPENROUTER_API_KEY`) serves
pre-push check calls a model only with an explicit `--model-config`. Agentic work runs on the plane (`ousast plane run`): `ousast` reads embeddings.
the variable a task's Model names (`secretKey.key`) from the operator's environment or - Keys come from the environment or a gitignored `.env`. `.env` never overrides an exported
`.env` and sends the key only in that task's start request, through Agent Substrate's variable.
router to that task's actor. It is not baked into an image or task template, and it is - The pre-push check calls a model only with an explicit `--model-config`.
never logged or written to artifacts. - Agentic work runs on the plane (`ousast plane run`). `ousast` reads the variable a task's
Model names (`secretKey.key`) from the operator's environment or `.env`. It sends the key
only in that task's start request. The request goes through Agent Substrate's router to that
task's actor. The key is not baked into an image or task template. It is never logged or
written to artifacts.
## Controls ## Controls
### Secret redaction ### Secret redaction
Scan artifacts can quote source that contains live credentials. `redaction.py` masks Scan artifacts can quote source that contains live credentials. `redaction.py` masks
recognizable secret shapes (provider API keys, AWS/GitHub/Slack/Google tokens, bearer recognizable secret shapes before traces (`trace/events.jsonl`) and the markdown report are
tokens, URL-embedded credentials, PEM private keys, and `key = value` assignments for written. It masks:
sensitive names) before traces (`trace/events.jsonl`) and the markdown report are
written. On by default; disable with `[hardening] redact_secrets = false`. - provider API keys;
- AWS/GitHub/Slack/Google tokens;
- bearer tokens;
- URL-embedded credentials;
- PEM private keys;
- `key = value` assignments for sensitive names.
Redaction is on by default. Turn it off with `[hardening] redact_secrets = false`.
### Plane egress ### Plane egress
Each plane task runs in its own actor behind Agent Substrate's egress gateway, which Each plane task runs in its own actor behind Agent Substrate's egress gateway. The gateway denies
denies by default. The task's EgressPolicy (`src/openultrasast/plane/egress.py`) allows exactly three by default. The task's EgressPolicy (`src/openultrasast/plane/egress.py`) allows exactly three
things: plain HTTP to the artifact receiver, and TLS passthrough to the Git hosts of the things:
task's bound Workspaces and to the hosts its bound Model declares
(`openultrasast.io/egress-hosts`). There are no wildcards, no IP addresses and no TLS 1. plain HTTP to the artifact receiver;
interception. The policy is deleted with the actor. 2. TLS passthrough to the Git hosts of the task's bound Workspaces;
3. TLS passthrough to the hosts its bound Model declares (`openultrasast.io/egress-hosts`).
There are no wildcards, no IP addresses and no TLS interception. The policy is deleted with the
actor.
### Cost & CI budgets ### Cost & CI budgets
- **Agentic spend** is bounded per plane task by the Run's `budget: {usd, calls}`. The - **Agentic spend** is bounded per plane task by the Run's `budget: {usd, calls}`.
metered client refuses the next call once either ceiling is reached (the task is - The metered client refuses the next call once either ceiling is reached. The task is then
`unfinished` and resumes on a rerun with a larger budget). A usd budget on a Model `unfinished`. It resumes on a rerun with a larger budget.
without prices is refused rather than metered as zero, and an account or - A usd budget on a Model without prices is refused. It is not metered as zero.
authentication error fails the task and starts nothing further. The standard-mode tool - An account or authentication error fails the task and starts nothing further.
hunter is bounded by the per-tier hunter budgets. - The standard-mode tool hunter is bounded by the per-tier hunter budgets.
- **Output size** is bounded by `[hardening] max_findings` (0 = unlimited). Truncation is - **Output size** is bounded by `[hardening] max_findings` (0 = unlimited). Truncation is
severity-ordered and disclosed as a `budget` degradation in the manifest. severity-ordered. It is disclosed as a `budget` degradation in the manifest.
### Provider reliability ### Provider reliability
LLM/embedding calls retry transient failures (HTTP 429/5xx, connection errors, timeouts) LLM/embedding calls retry transient failures with exponential backoff. Transient failures are
with exponential backoff; non-transient errors (e.g. 4xx, malformed JSON) fail fast. HTTP 429/5xx, connection errors and timeouts. Non-transient errors fail fast, for example 4xx or
malformed JSON.
### Visible degradation & determinism ### Visible degradation & determinism
When an optional engine is missing (Joern: `cpg_unavailable`; Docker: `sandbox_unavailable`), A scan is never silently downgraded. When an optional engine is missing, the scan falls back to
the scan falls back to its deterministic equivalent and records a `degradations` entry in the its deterministic equivalent. It records a `degradations` entry in the manifest (Joern:
manifest — it is never silently downgraded. Without a provider key the model layer asks no `cpg_unavailable`; Docker: `sandbox_unavailable`).
question and reports only what the graph entailed. Runs are reproducible: fixed config,
artifact manifests, prompt hashes, and model identifiers are recorded. Without a provider key, the model layer asks no question. It reports only what the graph
entailed.
Runs are reproducible. The fixed config, artifact manifests, prompt hashes and model identifiers
are recorded.
### MCP surface ### MCP surface
`ousast mcp` exposes only the ten narrow project tools. No tool runs arbitrary shell, `ousast mcp` exposes only the ten narrow project tools. No tool runs arbitrary shell, Docker or
Docker, or internal hunter tools, and no tool accepts a free-form command argument. internal hunter tools. No tool accepts a free-form command argument.
## Sandbox (deep mode) ## Sandbox (deep mode)
The REGRESS stage (`regress/`, `sandbox/runner.py`) runs each snippet with `docker run` The REGRESS stage (`regress/`, `sandbox/runner.py`) runs each snippet with `docker run`. It never
and never mounts the host Docker socket: mounts the host Docker socket.
| Control | Setting | | Control | Setting |
|---|---| |---|---|
@@ -84,5 +107,5 @@ and never mounts the host Docker socket:
| pids | `[sandbox] pids_limit`, default 512 | | pids | `[sandbox] pids_limit`, default 512 |
| timeout | `[sandbox] timeout_seconds`, default 300 s | | timeout | `[sandbox] timeout_seconds`, default 300 s |
Before anything runs, a structural deny-list (`regress/safety.py`) rejects a snippet that A structural deny-list (`regress/safety.py`) checks each snippet before anything runs. It rejects
mentions sockets, curl, host networking or writes under `/workspace`. a snippet that mentions sockets, curl, host networking or writes under `/workspace`.
+200 -128
View File
@@ -1,18 +1,26 @@
# Token ergonomics # Token ergonomics
Model tokens are the one running cost of this project, and they are spent in two places: by the Model tokens are the one running cost of this project. They are spent in two places:
**plane** when a Run verifies candidates or compiles the decision engine, and by the maintainers'
own **agentic coding sessions**. The maintainer's framing (2026-09-29): "the token saving strategy
is a routing concern. ax can handle that and structure our services in small components." So the
runtime half is routing, one Model per Task with its own budget, and reuse of what was already
paid for; the development half is tooling that keeps bulk file contents out of the session.
Every number on this page comes from a committed record, named next to it. Records are counts - by the **plane** (the system that runs scans as small tasks) when a Run verifies candidates
only; none holds code, prompts or responses. or compiles the decision engine;
- by the maintainers' own **agentic coding sessions**.
The before-and-after summary (script pipeline against plane, and the development share) is on The maintainer's framing (2026-09-29): "the token saving strategy is a routing concern. ax can
[Where we stand](where-we-stand.md#4-token-economics-before-and-after); this page holds the handle that and structure our services in small components." Here ax is Google's controller
mechanisms and the detail table. that runs agent tasks on Kubernetes. So the two halves look like this:
- **Runtime** is routing: one Model per Task with its own budget, and reuse of what was
already paid for.
- **Development** is tooling that keeps bulk file contents out of the session.
Every number on this page comes from a committed record, named next to it. Records hold counts
only. None holds code, prompts or responses.
The before-and-after summary is on
[Where we stand](where-we-stand.md#4-token-economics-before-and-after). It compares the script
pipeline with the plane and gives the development share. This page holds the mechanisms and the
detail table.
## A. Runtime: the plane's token economy ## A. Runtime: the plane's token economy
@@ -37,46 +45,67 @@ flowchart LR
### One Model per Task, one budget per Task ### One Model per Task, one budget per Task
A Run is a DAG of ax Tasks; each Task binds at most one Model and its own `budget: {usd, calls}` A Run is a DAG (a graph of steps with no cycles) of ax Tasks. Each Task binds at most one Model.
([The plane on ax](plane.md)). The budget is enforced by the metered client Each Task has its own `budget: {usd, calls}` ([The plane on ax](plane.md)).
(`src/openultrasast/plane/budget.py`, `MeteredClient`): `check()` runs before every call and
raises `BudgetExhausted` once the spent `usd` or the call count reaches the ceiling, so the call
that crosses the ceiling completes and the next one is refused. The task then ends `unfinished`
and resumes from its `units.jsonl` on a rerun with a larger budget; an HTTP 401/402 or
"Insufficient Balance" is an `AccountError`, which fails the task and starts nothing further.
Usage is attributed per call: the wrapped client appends one usage row per call
(`prompt_tokens`, `prompt_cache_hit_tokens`, `completion_tokens`), rows are absorbed even when
the call raises, and they are priced by the Model's price list. An unpriced Model cannot run
under a `usd` budget, and `summary()` reports `usd: None` for it, never 0. Model-free tasks run
with `{usd: 0, calls: 0}`, so a stray call fails loudly.
`ousast plane status RUN` is the attribution table: one row per task with `calls`, `prompt`, The metered client enforces the budget (`src/openultrasast/plane/budget.py`, `MeteredClient`):
`cache_hit`, `output` and `usd` from each task's `summary.json`, a `subtotal` row per Model and a
`total` (a missing value makes a sum `n/a`, never a smaller number); `--units` adds the per-unit 1. `check()` runs before every call.
rows. The same table is written as `attribution.json` in the run directory 2. It raises `BudgetExhausted` once the spent `usd` or the call count reaches the ceiling.
(`src/openultrasast/plane/reconciler.py`, `attribution()`/`status()`). Provider credentials 3. So the call that crosses the ceiling completes, and the next one is refused.
reach a task only in the start request: `router.py` reads the variable the Model's 4. The task then ends `unfinished`. A rerun with a larger budget resumes from its `units.jsonl`.
`secretKey.key` names, the runner keeps the value in memory, passes it to the command's
environment and redacts it from echoed stderr; it never appears in a manifest, a rendered Task An HTTP 401/402 or "Insufficient Balance" is an `AccountError`. It fails the task and starts
or a file. nothing further.
Usage is attributed per call:
- The wrapped client appends one usage row per call (`prompt_tokens`,
`prompt_cache_hit_tokens`, `completion_tokens`).
- Rows are absorbed even when the call raises.
- Rows are priced by the Model's price list.
- An unpriced Model cannot run under a `usd` budget. `summary()` reports `usd: None` for it,
never 0.
- Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly.
`ousast plane status RUN` prints the attribution table. It reads each task's `summary.json`
and shows:
- one row per task with `calls`, `prompt`, `cache_hit`, `output` and `usd`;
- a `subtotal` row per Model;
- a `total`. A missing value makes a sum `n/a`, never a smaller number.
`--units` adds the per-unit rows. The same table is written as `attribution.json` in the run
directory (`src/openultrasast/plane/reconciler.py`, `attribution()`/`status()`).
Provider credentials reach a task only in the start request:
- `router.py` reads the variable that the Model's `secretKey.key` names.
- The runner keeps the value in memory and passes it to the command's environment.
- The runner redacts it from echoed stderr.
- It never appears in a manifest, a rendered Task or a file.
### Pay once, reuse ### Pay once, reuse
- **Repo facts are computed model-free and reused.** `repo-facts` reads source text only (no - **Repo facts are computed model-free and reused.**
model, no git) and its `facts.json` is byte-identical across runs over the same tree. The - `repo-facts` reads source text only (no model, no git).
memory store keeps a facts entry under the key `(repo, pin, candidates digest, runner image - Its `facts.json` is byte-identical across runs over the same tree.
digest)` (`src/openultrasast/plane/memory.py`, `memory_key()`); when a later Run's task - The memory store keeps a facts entry under the key `(repo, pin, candidates digest, runner image
carries the same key, `seed()` writes the stored `facts.json`, a `done` summary with digest)` (`src/openultrasast/plane/memory.py`, `memory_key()`).
`calls: 0` and no `usd`, and marks the task done without starting an actor. A changed pin, - A later Run's task may carry the same key. Then `seed()` writes the stored `facts.json` and a
candidate set or image finds nothing and recomputes. `done` summary with `calls: 0` and no `usd`. It marks the task done without starting an
- **Content-addressed memory** (`src/openultrasast/learn/`). Excerpts are stored as actor.
`excerpts/<sha256 of the normalised text>.txt`; an embedding lives at - A changed pin, candidate set or image finds nothing and recomputes.
`embeddings/<model>/<excerpt sha>.json` and "an excerpt is embedded once; a rerun makes no - **Content-addressed memory** (`src/openultrasast/learn/`). Each item is stored under a hash
call" (`embeddings.py`). Model responses are cached under the sha256 of of its content.
`{model, messages, params, temperature, sample}` (`program.py`, `request_key()`), and a - Excerpts are stored as `excerpts/<sha256 of the normalised text>.txt`.
compiled program is `programs/<sha256 of its artifact>.json` with the digests of the memory - An embedding lives at `embeddings/<model>/<excerpt sha>.json`. The source says: "an excerpt
snapshot and of the response cache it was built from (`compile.py`). is embedded once; a rerun makes no call" (`embeddings.py`).
- **Replay is free.** With no client, the response cache replays at $0 and a miss raises - Model responses are cached under the sha256 of `{model, messages, params, temperature, sample}`
(`program.py`, `request_key()`).
- A compiled program is `programs/<sha256 of its artifact>.json`. It records the digests of
the memory snapshot and of the response cache it was built from (`compile.py`).
- **Replay is free.** With no client, the response cache replays at $0. A miss raises
`ReplayMiss` instead of calling. The injection-slice record states it: "evaluate re-run `ReplayMiss` instead of calling. The injection-slice record states it: "evaluate re-run
replay-only on the committed code: 550 responses replayed, $0, identical overall and replay-only on the committed code: 550 responses replayed, $0, identical overall and
calibration" (`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`, calibration" (`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`,
@@ -84,26 +113,35 @@ or a file.
### Cheap before expensive ### Cheap before expensive
- **Triage first.** Verification runs only over the candidates the recorded triage kept; the - **Triage first.** Verification runs only over the candidates the recorded triage kept.
plane applies that recording without a model call and prices it as a separate line - The plane applies that recording without a model call.
(`recorded_triage_usd`), so every per-candidate cost below is reported both with and without - It prices the recording as a separate line (`recorded_triage_usd`).
it (`src/openultrasast/plane/tasks/agree.py`). - So every per-candidate cost below is reported both with and without it
(`src/openultrasast/plane/tasks/agree.py`).
- **Hunts are batched per file.** `verify` runs one tool hunt per file for up to `PER_HUNT = 6` - **Hunts are batched per file.** `verify` runs one tool hunt per file for up to `PER_HUNT = 6`
candidates, with the candidates' known callers (capped at 8) in the prompt candidates. The prompt includes the candidates' known callers, capped at 8
(`src/openultrasast/plane/tasks/verify.py`, `units_of()`). (`src/openultrasast/plane/tasks/verify.py`, `units_of()`).
- **The tie-break pass runs only on disputes.** `agree` compares passes a and b; pass c is - **The tie-break pass runs only on disputes.**
started with `inputs.only` set to the case's `disputed.json`, an empty list ends it `done` - `agree` compares passes a and b.
with no model call, and the final verdict is 2-of-3 ("pass c alone never decides", - Pass c starts with `inputs.only` set to the case's `disputed.json`.
`agree.py`). - An empty list ends it `done` with no model call.
- **Cache-friendly prompt order.** The decision engine's prompt is the compiled prefix - The final verdict is 2-of-3 ("pass c alone never decides", `agree.py`).
(instruction and fixed demonstrations, identical for every candidate so the provider's - **Cache-friendly prompt order.** The decision engine's prompt has three parts, in this order
prefix cache hits), then the retrieved examples, then the candidate (`learn/program.py`). (`learn/program.py`):
Its self-consistency `k` is tied to that cache: "k = 5 when samples 2..k hit the prefix cache 1. The compiled prefix: instruction and fixed demonstrations. It is identical for every
for >= 0.8 of their prompt tokens"; the measured repeat-hit share was 0.96 (injection-slice candidate, so the provider's prefix cache hits.
record, `smoke.k_rule`, `smoke.k5_repeat_hit_share_measured`). The verify hunts share a prefix 2. The retrieved examples.
too: in the first plane increment 2,385,152 of 3,820,395 prompt tokens were cache hits 3. The candidate.
(`benchmarks/independent/plane-increment-1.json`, `detail.cache_hit_tokens`,
`detail.prompt_tokens`; its reading: "Cost fell 17% (62% of prompt tokens were cache hits)"). The self-consistency `k` (the number of samples per candidate) is tied to that cache: "k = 5
when samples 2..k hit the prefix cache for >= 0.8 of their prompt tokens". The measured
repeat-hit share was 0.96 (injection-slice record, `smoke.k_rule`,
`smoke.k5_repeat_hit_share_measured`).
The verify hunts share a prefix too. In the first plane increment, 2,385,152 of 3,820,395
prompt tokens were cache hits (`benchmarks/independent/plane-increment-1.json`,
`detail.cache_hit_tokens`, `detail.prompt_tokens`). The record's reading: "Cost fell 17% (62%
of prompt tokens were cache hits)".
### Measured costs ### Measured costs
@@ -116,37 +154,41 @@ or a file.
| Decision engine harvest: `verify` part | $2.5373 over 1,961 calls, 474 tasks done, 10,921,167 prompt tokens of which 6,275,060 cache hits, 306,941 output tokens, task budgets summing to $8.624 under a $10 ceiling (`verify`) | `benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json` | | Decision engine harvest: `verify` part | $2.5373 over 1,961 calls, 474 tasks done, 10,921,167 prompt tokens of which 6,275,060 cache hits, 306,941 output tokens, task budgets summing to $8.624 under a $10 ceiling (`verify`) | `benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json` |
| Decision engine harvest: `roles` part | $1.4036 over 1,034 calls, 151 tasks done, 2,398,669 prompt tokens of which 329,600 cache hits, 370,155 output tokens, task budgets summing to $4.917 under a $10 ceiling (`roles`) | same record | | Decision engine harvest: `roles` part | $1.4036 over 1,034 calls, 151 tasks done, 2,398,669 prompt tokens of which 329,600 cache hits, 370,155 output tokens, task budgets summing to $4.917 under a $10 ceiling (`roles`) | same record |
The increment-2 record's note on the two denominators: "the reference divides by the 46 The increment-2 record notes that the two denominators differ: "the reference divides by the 46
candidates before triage; on that basis the plane is 7% cheaper, on the 43 after triage it is candidates before triage; on that basis the plane is 7% cheaper, on the 43 after triage it is
0.2% under". 0.2% under".
!!! note "The meter prices at list; the account moved less" !!! note "The meter prices at list; the account moved less"
The attribution table prices every call at the Model's list price The attribution table prices every call at the Model's list price
(`plane/models/deepseek-flash.yaml`). The harvest record: "Spend is the plane's attribution (`plane/models/deepseek-flash.yaml`). The harvest record says: "Spend is the plane's
table (`ousast plane status`, task meters priced at plane/models/deepseek-flash.yaml); the attribution table (`ousast plane status`, task meters priced at
DeepSeek account moved less (balance below)", with the balance at $19.63 before, $18.75 plane/models/deepseek-flash.yaml); the DeepSeek account moved less (balance below)".
after `verify` and $18.23 after `roles`. The injection-slice record names the ratio: "the The balance was $19.63 before, $18.75 after `verify` and $18.23 after `roles`.
provider charged ~1/3 of the list-price meter (balance moved $0.30 for $0.91 metered): The injection-slice record names the ratio: "the provider charged ~1/3 of the list-price
DeepSeek's off-peak discount window, presumably; the meter prices at list" meter (balance moved $0.30 for $0.91 metered): DeepSeek's off-peak discount window,
(`spend.balance_note`, balance $18.2 to $17.9). Budgets and gates are set against the presumably; the meter prices at list" (`spend.balance_note`, balance $18.2 to $17.9).
meter, never against the balance. Budgets and gates are set against the meter, never against the balance.
### Not done yet ### Not done yet
- **No model-side token budget in ax itself.** The plane's requirements state what ax does - **No model-side token budget in ax itself.**
not yet carry: "task dependencies, artifacts, token budgets (on ax's roadmap), a DeepSeek - The plane's requirements state what ax does not yet carry: "task dependencies, artifacts,
provider"; the thin `Run` layer carries them and dissolves into ax as ax gains them. Until token budgets (on ax's roadmap), a DeepSeek provider".
then the ceiling lives in the metered client, not in the executor. - The thin `Run` layer carries them. It dissolves into ax as ax gains them.
- **Triage is applied, not run.** The plane has no triage task; it filters by the recorded - Until then the ceiling lives in the metered client, not in the executor.
triage and adds that recording's cost as its own line. A triage task is planned as a later - **Triage is applied, not run.**
specification. - The plane has no triage task.
- **`k` and the contrast examples are not knobs.** `k` is a compiled setting in {1, 3, 5} - It filters by the recorded triage and adds that recording's cost as its own line.
(`learn/compile.toml`, "k and lam remain A/B variants"), changed only through the decision - A triage task is planned as a later specification.
engine's A/B experiments, its requirement that "changes to prompts, models, pass counts, rule - **`k` and the contrast examples are not knobs.**
sets and features [are] compared by controlled experiments, so that nothing is adopted on one - `k` is a compiled setting in {1, 3, 5} (`learn/compile.toml`, "k and lam remain A/B
run's number" (Requirement 4 of its specification). The retrieval cap of at most two variants").
contrast examples from other families is a module constant (`learn/retrieve.py`, - It changes only through the decision engine's A/B experiments. Requirement 4 of its
`CONTRAST = 2`), a maintainer decision, not an experiment variant yet. specification asks that "changes to prompts, models, pass counts, rule sets and features
[are] compared by controlled experiments, so that nothing is adopted on one run's number".
- Retrieval adds at most two contrast examples from other families. This cap is a module
constant (`learn/retrieve.py`, `CONTRAST = 2`).
- The cap is a maintainer decision, not an experiment variant yet.
## B. Development: the maintainers' own sessions ## B. Development: the maintainers' own sessions
@@ -165,28 +207,47 @@ flowchart LR
### The measurement: tool I/O dominates ### The measurement: tool I/O dominates
`benchmarks/dev/token_report.py` reads a Claude Code session transcript (`.jsonl`; without an `benchmarks/dev/token_report.py` reads a Claude Code session transcript (`.jsonl`). Without an
argument, the newest under `~/.claude/projects/`) and reports, per content kind, characters and argument it reads the newest one under `~/.claude/projects/`.
share: `tool_use_input` (the tool call's input as JSON), `tool_result`, `thinking`,
`assistant:text`, `user:text`, plus the tool-call count and the top five tools. Tokens are It reports characters and share per content kind:
approximated at four characters each. It prints to stdout and writes nothing; what gets
committed is the counts-only summary. The one in `benchmarks/independent/plane-increment-1.json` - `tool_use_input` (the tool call's input as JSON);
(`dev_token_report`, for "the whole development session (c09e30b2), not the increment alone"): - `tool_result`;
7.3M unique characters (~1.81M tokens), `tool_use_input_pct` 45.4, `tool_result_pct` 40.1, - `thinking`;
`tool_io_pct` 85.5 against `previous_tool_io_pct` 90, over 2,701 tool calls. The reasoning the - `assistant:text`;
session is paid for (assistant text and thinking) is the small remainder. That is why the - `user:text`.
remedies below target file contents, not prose.
It also reports the tool-call count and the top five tools. Tokens are approximated at four
characters each. It prints to stdout and writes nothing. Only the counts-only summary is
committed.
The committed summary is in `benchmarks/independent/plane-increment-1.json`
(`dev_token_report`). It covers "the whole development session (c09e30b2), not the increment
alone". It counts 7.3M unique characters (~1.81M tokens). The shares
are `tool_use_input_pct` 45.4, `tool_result_pct` 40.1, and `tool_io_pct` 85.5 against
`previous_tool_io_pct` 90, over 2,701 tool calls.
The reasoning the session is paid for (assistant text and thinking) is the small remainder.
That is why the remedies below target file contents, not prose.
### The remedies in `benchmarks/dev/` ### The remedies in `benchmarks/dev/`
**`read_guard.py`**, a `PreToolUse` hook for agentic coding sessions. It reads the hook event on **`read_guard.py`** is a `PreToolUse` hook (a script that runs before each tool call) for
stdin and, for the `Read` tool only, denies a whole-file read of a file longer than 350 lines agentic coding sessions.
(`LIMIT = 350`); a read with `offset` or `limit`, a missing path, an unreadable file or a file
within the limit passes. It always exits 0 ("never fails the tool"). A denial is one JSON line - It reads the hook event on stdin.
whose `permissionDecisionReason` tells the agent to read a range or to ask the bulk reader. The - For the `Read` tool only, it denies a whole-file read of a file longer than 350 lines
script's docstring records that it is wired in `.claude/settings.local.json` (a local, untracked (`LIMIT = 350`).
file) as `PreToolUse` with matcher `Read`; the setting has this shape, with the command - These pass: a read with `offset` or `limit`, a missing path, an unreadable file, or a file
repo-relative because hooks run from the project directory: within the limit.
- It always exits 0 ("never fails the tool").
- A denial is one JSON line. Its `permissionDecisionReason` tells the agent to read a range or
to ask the bulk reader.
The script's docstring records how it is wired. It sits in `.claude/settings.local.json` (a
local, untracked file) as `PreToolUse` with matcher `Read`. The setting has this shape. The
command is repo-relative because hooks run from the project directory:
```json ```json
{ {
@@ -203,37 +264,48 @@ repo-relative because hooks run from the project directory:
} }
``` ```
**`bulk_read.py`**, the portal pattern: a cheap model reads the files and answers one question, **`bulk_read.py`** follows the portal pattern. A cheap model reads the files and answers one
citing `path:line` for every claim; it reports what the files say and decides nothing. question. It cites `path:line` for every claim. It reports what the files say and decides
nothing.
```text ```text
.venv/bin/python benchmarks/dev/bulk_read.py "<question>" FILE [FILE ...] [--max-usd 0.05] .venv/bin/python benchmarks/dev/bulk_read.py "<question>" FILE [FILE ...] [--max-usd 0.05]
``` ```
Files go to the model in 400-line chunks with a `--- path (lines a-b)` header and numbered How it works:
lines; the client is the project's detector client (`deepseek-flash` through `DEEPSEEK_API_KEY`,
otherwise OpenRouter), each call has a 90 s timeout and no tools. `--max-usd` (default 0.05) is - Files go to the model in 400-line chunks, with a `--- path (lines a-b)` header and numbered
checked before every chunk; at the ceiling the answer ends with lines.
`[stopped at the $X ceiling before: <chunk header>]`. The answer goes to stdout, the summary - The client is the project's detector client: `deepseek-flash` through `DEEPSEEK_API_KEY`,
`[bulk_read: N chunk(s), $X.XXXX]` to stderr. Without a chat client it exits 2. otherwise OpenRouter.
- Each call has a 90 s timeout and no tools.
- `--max-usd` (default 0.05) is checked before every chunk. At the ceiling the answer ends with
`[stopped at the $X ceiling before: <chunk header>]`.
- The answer goes to stdout. The summary `[bulk_read: N chunk(s), $X.XXXX]` goes to stderr.
- Without a chat client it exits 2.
**`token_report.py`**: `python benchmarks/dev/token_report.py [TRANSCRIPT.jsonl]`, described **`token_report.py`**: `python benchmarks/dev/token_report.py [TRANSCRIPT.jsonl]`, described
above. above.
### Practices ### Practices
- **Scripts into files, not heredocs.** A heredoc is `tool_use_input` and its output - **Scripts into files, not heredocs.** A heredoc (a script typed inline in a shell command) is
`tool_result`; both are paid again at every later turn of the session. A script on disk is `tool_use_input`, and its output is `tool_result`. Both are paid again at every later turn of
read once, by the interpreter. the session. A script on disk is read once, by the interpreter.
- **Terse outputs.** Commands print counts, exit codes and the lines that matter; a harness - **Terse outputs.** Commands print counts, exit codes and the lines that matter. A harness
fails loudly with the exit code rather than printing a table that has to be read back. fails loudly with the exit code. It does not print a table that has to be read back.
- **Subagents with worktrees** for work whose output need not enter the main context: a - **Subagents with worktrees** for work whose output need not enter the main context.
subagent's reads and command output stay in its own context and only its conclusion comes - A subagent's reads and command output stay in its own context.
back; a git worktree keeps its edits off the main checkout. - Only its conclusion comes back.
- A git worktree (a separate checkout of the same repository) keeps its edits off the main
checkout.
- **Counts-only measurement records.** A record under `benchmarks/measurements/` or - **Counts-only measurement records.** A record under `benchmarks/measurements/` or
`benchmarks/independent/` holds numbers and their keys, never code, prompts or responses, so `benchmarks/independent/` holds numbers and their keys. It never holds code, prompts or
it can be quoted on a page like this one and reread by a model at a few hundred tokens. responses. So it can be quoted on a page like this one. A model can reread it at a few
hundred tokens.
Related: [The plane on ax](plane.md) for the Run and Task contract, [Memory](memory.md) for Related:
the store the reuse keys live in, [Evaluation](evaluation.md) for the gates these costs are
measured against. - [The plane on ax](plane.md) for the Run and Task contract;
- [Memory](memory.md) for the store the reuse keys live in;
- [Evaluation](evaluation.md) for the gates these costs are measured against.
+259 -174
View File
@@ -1,33 +1,50 @@
# Where we stand # Where we stand
State as of 2026-10-02 (v2.0.0). This page answers five questions in one place. How far is State as of 2026-10-02 (v2.0.0). This page answers five questions in one place:
detection from its gates. How does the project now try to close that gap. How does the model work
run on ax. What did the tokens cost before and after that move. What is still open. Every figure
is quoted from a committed record whose path stands next to it; detail tables stay on the pages
linked from each section.
The goal has not changed: a safety net on repositories the tool has never seen, which says 1. How far is detection from its gates?
BLOCK or ADVISORY about the code a push would change and stays silent on fixes and on benign 2. How does the project now try to close that gap?
commits. Two independent populations have been spent against that goal, with engine recall 1 of 3. How does the model work run on ax?
4. What did the tokens cost before and after that move?
5. What is still open?
Every figure is quoted from a committed record. The record's path stands next to the figure.
Detail tables stay on the pages linked from each section.
The goal has not changed. The tool should be a safety net on repositories it has never seen. It
should say BLOCK or ADVISORY about the code a push would change. It should stay silent on fixes
and on benign commits.
Two independent populations have been spent against that goal. An independent population is a
set of cases frozen before any scan. Engine recall was 1 of
11 and then 0 of 17. The hand edits that closed misses on the first population did not carry to 11 and then 0 of 17. The hand edits that closed misses on the first population did not carry to
the second. So the question is no longer which rule to add next. It is how a change to detection the second. So the question is no longer which rule to add next. The question is this: how can a
can be shown to work on code it was not tuned on, at a cost per candidate that can be paid for change to detection be shown to work on code it was not tuned on? And can that be done at a cost
every push. per candidate that can be paid for every push?
The answer this page records: the pre-push safety net is NO-GO against every M4 gate, and the The answer this page records:
project has stopped trying to reach the gates by hand. Every detection change now has to win a
controlled measurement on held-out repositories, run on an agentic plane whose cost per candidate - The pre-push safety net is NO-GO against every M4 gate.
is known. The pieces of that method exist and have been measured. None of them has produced a - The project has stopped trying to reach the gates by hand.
BLOCK decision yet. - Every detection change now has to win a controlled measurement on held-out repositories. The
measurement runs on an agentic plane whose cost per candidate is known.
- The pieces of that method exist and have been measured. None of them has produced a BLOCK
decision yet.
## 1. Detection capability and goals: NO-GO, measured ## 1. Detection capability and goals: NO-GO, measured
No M4 gate is met on an independent population. The qualification milestone for the pre-push No M4 gate is met on an independent population. M4 is the qualification milestone for the
safety net (M4) requires, together: at least 95% actionable precision, at least 90% supported pre-push safety net. It requires all of these together:
recall, at least 95% supported-check completion, the runtime gates, and zero fixed-side or benign
false alerts. A population is frozen before any - at least 95% actionable precision;
case is scanned, scored once under a pre-registered protocol, then spent; it can inform changes - at least 90% supported recall;
but can qualify nothing afterwards ([Evaluation](evaluation.md#independent-populations-and-the-m4-gates)). - at least 95% supported-check completion;
- the runtime gates;
- zero fixed-side or benign false alerts.
A population is frozen before any case is scanned. It is scored once under a pre-registered
protocol, then spent. After that it can inform changes but can qualify nothing
([Evaluation](evaluation.md#independent-populations-and-the-m4-gates)).
| Gate | Required | Latest independent result | Record | | Gate | Required | Latest independent result | Record |
| --- | --- | --- | --- | | --- | --- | --- | --- |
@@ -36,156 +53,217 @@ but can qualify nothing afterwards ([Evaluation](evaluation.md#independent-popul
| Supported-check completion | >= 95% | 13,206 of 108,374 | `results-v2.json` | | Supported-check completion | >= 95% | 13,206 of 108,374 | `results-v2.json` |
| Fixed-side and benign alerts | zero | 0 fixed-side, which the record calls met trivially since nothing was detected; 3 benign alerts | `results-v2.json` | | Fixed-side and benign alerts | zero | 0 fixed-side, which the record calls met trivially since nothing was detected; 3 benign alerts | `results-v2.json` |
Three later measurements narrow the picture without changing the verdict. Three later measurements narrow the picture. They do not change the verdict.
- The plane reproduces the script pipeline's agreement at a known cost. On the model-driven - **The plane reproduces the script pipeline's agreement at a known cost.** The test used the
pipeline's validation set (46 candidates, 15 cases of v2, 20 declared vulnerable sites), the model-driven pipeline's validation set: 46 candidates, 15 cases of v2, 20 declared vulnerable
plane's verify passes a and b plus a tie-break on the 10 disputed candidates agreed on 16 of 20 sites. The plane ran verify passes a and b, plus a tie-break on the 10 disputed candidates. It
declared sites. The script reference agreed on 16 as well. The cost was $0.0205 per candidate agreed on 16 of 20 declared sites. The script reference agreed on 16 as well. The cost was
(`benchmarks/independent/plane-increment-2.json`). The record carries two caveats. The tie-break $0.0205 per candidate (`benchmarks/independent/plane-increment-2.json`). The record carries two
was chosen after seeing the first measurement on the same set, so this compares plane with caveats:
script and qualifies nothing. And the passes flag fewer declared sites than the reference in 1. The tie-break was chosen after seeing the first measurement on the same set. So this compares
any pass, 17 against 19; that count is a correction in `plane-increment-1.json`, whose first plane with script and qualifies nothing.
reading took agree's `declared_sites_matched` (19, coverage by candidates) for "found in either 2. The passes flag fewer declared sites than the reference in any pass: 17 against 19 (passes against reference). That
pass". count is a correction in `plane-increment-1.json`. Its first reading took agree's
- The learned decision engine ranks, but may not block. On six families, evaluated on `declared_sites_matched` (19, coverage by candidates) to mean "found in either pass".
repositories the compiled program was not shown, the pooled AUC of its score runs from 0.77 - **The learned decision engine ranks, but may not block.** It was evaluated on six families, on
(injection) to 0.96 (access control). The stricter reading is the within-pair AUC, which asks repositories the compiled program was not shown.
whether the vulnerable side of a fix scores above its own fixed side: 0.72 to 0.93. BLOCK is - The pooled AUC (area under the ROC curve) of its score runs from 0.77 (injection) to 0.96
offered for no family; calibration holds only for output encoding, and there no fold reaches (access control).
the required precision (`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`). - The stricter reading is the within-pair AUC. It asks whether the vulnerable side of a fix
The per-family table is on [Decision engine](decision-engine.md#second-slice-six-families-2026-10-02). scores above its own fixed side: 0.72 to 0.93.
- PHP coverage is in-sample. The PHP quick rules were written and tightened while looking at - BLOCK is offered for no family. Calibration holds only for output encoding. There, no fold
the pins and fixtures they are scored on, so the record says so itself (`provenance.in_sample` in reaches the required precision
`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`): a development check, not (`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`).
a recall estimate. On real PHP code the Joern engine is the instrument. - The per-family table is on
[Decision engine](decision-engine.md#second-slice-six-families-2026-10-02).
- **PHP coverage is in-sample.** The PHP quick rules were written and tightened while looking at
the pins and fixtures they are scored on. The record says so itself (`provenance.in_sample` in
`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). It is a development
check, not a recall estimate. On real PHP code, the Joern engine is the instrument.
What has not been touched: population v3, 15 PHP cases, frozen as the one-time final check. No What has not been touched: population v3, 15 PHP cases, frozen as the one-time final check.
development or analysis work reads its cases, a test fails when anything in the tree names one
of its repositories, and it runs once, after a prediction is committed. - No development or analysis work reads its cases.
- A test fails when anything in the tree names one of its repositories.
- It runs once, after a prediction is committed.
## 2. Our approach to detection, step by step ## 2. Our approach to detection, step by step
The path changed because hand edits did not transfer. Every gain before 2026-09-30 was a hand The path changed because hand edits did not transfer. Every gain before 2026-09-30 was a hand
edit after a miss: a rule, a taint vocabulary entry, a guard, a prompt line, a tie-break. Each was edit after a miss: a rule, a taint vocabulary entry, a guard, a prompt line, a tie-break. Each was
tuned on the cases that exposed it, and the tuning measured on population v1 did not carry to v2 tuned on the cases that exposed it. The tuning measured on population v1 did not carry to v2
(1 of 11, then 0 of 17). The conclusions drawn: hand edits do not transfer. The data that could (1 of 11, then 0 of 17).
teach a better decision cannot be produced by hand at the scale needed, so it has to be harvested
and labelled by machinery with a boundary around it. Framework knowledge (which function is a sink in which
framework) is useful as a prior but must stay optional, or the engine learns the vocabulary
instead of the code. A reader might object that a classifier is slower to show gains than the
next rule. It is. The difference is that its gains are measured where rules were not: on
repositories it never saw.
The chain, one job per step. The design is on [Memory and detection](memory-and-detection.md); The conclusions drawn:
this is the order of operations.
1. Instruments produce signals, never verdicts. Quick rules, the Joern engine, model sink - Hand edits do not transfer.
classification, the plane's verify passes a and b with the tie-break c, repository facts - The data that could teach a better decision cannot be produced by hand at the scale needed. It
(callers, spans) and model source/sink/sanitizer roles each write one block of a candidate's has to be harvested and labelled by machinery with a boundary around it.
record. None decides alone. - Framework knowledge (which function is a sink in which framework) is useful as a prior. But it
2. Feature records are allow-listed, and missing is not zero. A candidate is `(path, function, must stay optional. Otherwise the engine learns the vocabulary instead of the code.
family)` of one repository and pin. An instrument that did not run, had no coverage or failed
records that state; "silent" and "did not look" stay distinguishable. Input profile v1 withholds A reader might object that a classifier is slower to show gains than the next rule. It is. The
the engine block and the function length, which a leak audit found separate the two sides of a difference is where its gains are measured: on repositories it never saw. Rules were not measured
pair by construction (`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`), and there.
since 2026-10-02 the plane's `verify.flag_b` and `ms.operation`, flagged the same way by the
audit of the plane profile (`2026-10-02-decision-engine-harvest-advisory/record.json`, The chain has one job per step. The design is on [Memory and detection](memory-and-detection.md).
This list is the order of operations.
1. **Instruments produce signals, never verdicts.** Each instrument writes one block of a
candidate's record. None decides alone. The instruments are:
- quick rules;
- the Joern engine;
- model sink classification;
- the plane's verify passes a and b, with the tie-break c;
- repository facts (callers, spans);
- model source/sink/sanitizer roles.
2. **Feature records are allow-listed, and missing is not zero.** A candidate is `(path, function,
family)` of one repository and pin.
- An instrument that did not run, had no coverage or failed records that state. "Silent" and
"did not look" stay distinguishable.
- Input profile v1 withholds the engine block and the function length. A leak audit found that
they separate the two sides of a pair by construction
(`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`).
- Since 2026-10-02 it also withholds the plane's `verify.flag_b` and `ms.operation`. The audit
of the plane profile flagged them the same way
(`2026-10-02-decision-engine-harvest-advisory/record.json`,
`src/openultrasast/learn/schema.py`). `src/openultrasast/learn/schema.py`).
3. Labels come from ground truth only. A fail-closed allow-list (`src/openultrasast/learn/sources.toml`) 3. **Labels come from ground truth only.** A fail-closed allow-list
names the sources: the spent populations' declared sites and fix ranges, the vulnerable and (`src/openultrasast/learn/sources.toml`) names the sources:
fixed sides of pairs, including the 97 real public fixes in `benchmarks/pairs/advisory-fixes/`, - the spent populations' declared sites and fix ranges;
benign controls and recorded adjudications. Assumed-benign commits are their own source, - the vulnerable and fixed sides of pairs, including the 97 real public fixes in
weighted 0.5 and reported separately. A plane verdict is a feature, never a label: "rejected by `benchmarks/pairs/advisory-fixes/`;
the plane" is not a negative, or the engine would learn to reproduce the plane's mistakes. - benign controls;
4. Retrieval finds similar labelled cases. A Gower distance over the signal profile, re-ranked - recorded adjudications.
by cosine similarity of code embeddings (OpenAI `text-embedding-3-small` through OpenRouter,
cached by excerpt hash). The evaluation boundary is the only door into a prompt: never the Assumed-benign commits are their own source. They are weighted 0.5 and reported separately. A
candidate's own repository group, never a group the fold evaluates. plane verdict is a feature, never a label. "Rejected by the plane" is not a negative.
5. A classifier compiled in house, DSPy-style. `retrieve -> classify(k)`: an instruction and Otherwise the engine would learn to reproduce the plane's mistakes.
compiled demonstrations (a fixed prefix, so the provider's cache hits), the retrieved examples, 4. **Retrieval finds similar labelled cases.** It uses a Gower distance (a distance for mixed
then the candidate's excerpt and signals. The model answers vulnerable, not vulnerable or numeric and categorical data) over the signal profile. Results are re-ranked by cosine
unsure with a rationale that cites a line; `k` samples give a score. One program is compiled similarity of code embeddings. The embeddings are OpenAI `text-embedding-3-small` through
per profile and family on a compile split that is never evaluated. OpenRouter, cached by excerpt hash. The evaluation boundary is the only door into a prompt. It
6. Calibration on held-out repositories. Platt scaling of the score, cross-fitted by never admits the candidate's own repository group, or a group the fold evaluates.
repository group so no map is checked on the fold that fitted it. Calibration holds when the 5. **A classifier compiled in house, DSPy-style.** The program is `retrieve -> classify(k)`. The
slope interval contains 1, the intercept interval contains 0 and ECE <= 0.05. prompt holds, in order:
7. Two operating points. BLOCK only where calibration holds and a fold reaches the required - an instruction and compiled demonstrations (a fixed prefix, so the provider's cache hits);
precision; otherwise ADVISORY. An unsure majority can never BLOCK. - the retrieved examples;
8. A/B experiments are the only way a change is adopted. Prompts, models, pass counts, - the candidate's excerpt and signals.
features and sampling are compared under a pre-registered manifest with frozen units, paired
arms, a repository bootstrap and two looks. exp-002 is the first to finish: a retrieval The model answers vulnerable, not vulnerable or unsure, with a rationale that cites a line. `k`
ensemble (five samples over disjoint example sets at temperature 0) against today's samples give a score. One program is compiled per profile and family, on a compile split that
temperature sampling. It raised re-run agreement a little (0.906 against 0.875, interval is never evaluated.
spanning zero) and lowered the within-pair AUC from 0.815 to 0.741 at 2.4 times the cost per 6. **Calibration on held-out repositories.** Platt scaling (a logistic fit) maps the score to a
candidate ($0.0051 against $0.0021). Decision: REJECTED probability. It is cross-fitted by repository group, so no map is checked on the fold that
(`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`). fitted it. Calibration holds when three conditions are met: the slope interval contains 1, the
9. Extrapolation with intervals. Every number carries a 95% repository-cluster bootstrap intercept interval contains 0 and ECE <= 0.05.
interval; the pooled and the within-pair readings are both reported, and the stricter one is 7. **Two operating points.** BLOCK applies only where calibration holds and a fold reaches the
required precision. Otherwise the point is ADVISORY. An unsure majority can never BLOCK.
8. **A/B experiments are the only way a change is adopted.** Prompts, models, pass counts,
features and sampling are compared under a pre-registered manifest. The manifest fixes frozen
units, paired arms, a repository bootstrap and two looks. exp-002 is the first to finish:
- It compared a retrieval ensemble (five samples over disjoint example sets at temperature 0)
with today's temperature sampling.
- It raised re-run agreement a little (0.906 against 0.875, interval spanning zero).
- It lowered the within-pair AUC from 0.815 to 0.741 and cost 2.4 times as much per candidate
($0.0051 against $0.0021).
- Decision: REJECTED (`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`).
9. **Extrapolation with intervals.** Every number carries a 95% repository-cluster bootstrap
interval. Both the pooled and the within-pair readings are reported. The stricter one is
named. named.
10. The one-time v3 check. When a program is adopted and a prediction is committed, population 10. **The one-time v3 check.** It happens when a program is adopted and a prediction is
v3 is scored once under `benchmarks/independent/protocol-v3.md`. committed. Then population v3 is scored once under `benchmarks/independent/protocol-v3.md`.
## 3. AI integration: ax on the server ## 3. AI integration: ax on the server
All model work runs as Tasks on google/ax over Agent Substrate, in a single-node kind cluster on All model work runs as Tasks on google/ax over Agent Substrate. They run in a single-node kind
the maintainer's host, each Task in its own gVisor sandbox. There is no local subprocess path. cluster on the maintainer's host. Each Task runs in its own gVisor sandbox. There is no local
The reconciler (`ousast plane run`) is a host process: it applies a Run's Workspaces and Tasks, subprocess path.
resumes each Task, sends its start signal, collects its artifacts and records state. Flow and
contract: [The plane on ax](plane.md); bring-up and the host's lessons: [ax on this host](ops/ax/README.md). The reconciler (`ousast plane run`) is a host process. It does five things:
1. applies a Run's Workspaces and Tasks;
2. resumes each Task;
3. sends its start signal;
4. collects its artifacts;
5. records state.
Flow and contract: [The plane on ax](plane.md). Bring-up and the host's lessons:
[ax on this host](ops/ax/README.md).
What each Run guarantees: What each Run guarantees:
- One Model and one budget per Task. A Task binds at most one ax Model (`deepseek-flash` for - **One Model and one budget per Task.** A Task binds at most one ax Model and
chat, `openrouter-embedding` for embeddings) and `budget: {usd, calls}`. The metered client `budget: {usd, calls}`. The Models are `deepseek-flash` for chat and `openrouter-embedding` for
refuses the next call at the ceiling; the task ends unfinished and resumes on a rerun. Model-free embeddings. The metered client refuses the next call at the ceiling. The task ends unfinished
tasks run with a zero budget, so a stray call fails loudly. and resumes on a rerun. Model-free tasks run with a zero budget, so a stray call fails loudly.
- Credentials exist only in the start request. Agent Substrate boots each template once as a - **Credentials exist only in the start request.** Agent Substrate boots each template once as a
golden actor, so the runner never starts by itself. It waits for `POST /ousast/v1/start`, which golden actor. So the runner never starts by itself. It waits for `POST /ousast/v1/start`. The
the reconciler sends through `atenet-router` to that actor alone. The key is read from the reconciler sends that request through `atenet-router` to that actor alone.
operator's environment or `.env`, kept in memory by the runner, passed to the command and - The key is read from the operator's environment or `.env`.
redacted from echoed stderr; no manifest, rendered Task or file carries it. - The runner keeps it in memory and passes it to the command.
- Per-task egress. One EgressPolicy per actor through the `atenet-egress` gateway, deny by - It is redacted from echoed stderr.
default, hostnames only: plain HTTP to the artifact receiver, TLS passthrough to the Workspaces' - No manifest, rendered Task or file carries it.
Git hosts and the Model's declared hosts. Deleted with the actor. - **Per-task egress.** Each actor gets one EgressPolicy through the `atenet-egress` gateway. It is
- Artifacts come back through the gateway. The runner posts its output directory as a tar to deny by default and uses hostnames only. It allows plain HTTP to the artifact receiver. It allows
the receiver, an HTTP server inside the reconciler process on the host, reached through a Service TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts. It is deleted with
with an EndpointSlice to the host. That delivery is the task's completion; ax has no Completed the actor.
phase. - **Artifacts come back through the gateway.** The runner posts its output directory as a tar to
- Memory on S3-compatible RustFS. `OUSAST_MEMORY=s3://sast-memory` on a RustFS server. The the receiver. The receiver is an HTTP server inside the reconciler process on the host. It is
store verifies the bucket at every start (versioning, lifecycle, tags, S3 Select) and refuses to reached through a Service with an EndpointSlice to the host. That delivery is the task's
start when anything is missing; there is no local or fetch-and-filter fallback. Each row kind completion; ax has no Completed phase.
has a fixed set of queryable fields, enforced at write time - **Memory on S3-compatible RustFS.** The setting is `OUSAST_MEMORY=s3://sast-memory` on a RustFS
server.
- The store verifies the bucket at every start: versioning, lifecycle, tags, S3 Select.
- It refuses to start when anything is missing. There is no local or fetch-and-filter fallback.
- Each row kind has a fixed set of queryable fields, enforced at write time
([Memory](memory.md#verify-at-startup-no-fallback), [RustFS setup](rustfs.md)). ([Memory](memory.md#verify-at-startup-no-fallback), [RustFS setup](rustfs.md)).
Lessons the host taught, each from a failed live run (the full list is on Lessons the host taught, each from a failed live run (the full list is on
[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run)): [ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run)):
- Substrate installs no WorkerPool, and actor images must be pinned by digest. - Substrate installs no WorkerPool, and actor images must be pinned by digest.
- A release `ax` CLI skews from the server; build it from the deployed checkout. - A release `ax` CLI skews from the server. Build it from the deployed checkout.
- ax has no Completed phase, so completion is our artifact delivery. - ax has no Completed phase, so completion is our artifact delivery.
- The golden boot means the runner must wait for a start request. - The golden boot means the runner must wait for a start request.
- The egress gateway dials the address the actor connected to, so the receiver is a Service - The egress gateway dials the address the actor connected to. So the receiver is a Service,
dialled by ClusterIP with its name as Host. dialled by ClusterIP with its name as Host.
- ax's snapshot bucket must exist in your S3 server. When it did not, deleted golden actors piled - ax's snapshot bucket must exist in your S3 server. When it did not, deleted golden actors piled
up in `DELETING` and filled the disk during a 600-task harvest on 2026-10-01. up in `DELETING`. They filled the disk during a 600-task harvest on 2026-10-01.
Deployment status, in the two words [Deployment](deployment.md) uses strictly. Implemented: Deployment status uses two words, as [Deployment](deployment.md) does, strictly.
ax as the only executor, per-task EgressPolicy, credentials only in the start request, per-task
budgets and attribution, the S3 store verified at startup, local bring-up, doctor and smoke Run, Implemented:
re-pinning generated Tasks to another registry. Planned, not done: deployment to any cluster
other than this host's kind cluster, a configurable kube context (`kind-` is hard-coded in - ax as the only executor;
`doctor.py`, `egress.py`, `router.py`), the receiver as an in-cluster Deployment, the - per-task EgressPolicy;
reconciler in-cluster with Secrets injected, presigned-URL delivery to the store. - credentials only in the start request;
- per-task budgets and attribution;
- the S3 store verified at startup;
- local bring-up, doctor and smoke Run;
- re-pinning generated Tasks to another registry.
Planned, not done:
- deployment to any cluster other than this host's kind cluster;
- a configurable kube context (`kind-` is hard-coded in `doctor.py`, `egress.py`, `router.py`);
- the receiver as an in-cluster Deployment;
- the reconciler in-cluster with Secrets injected;
- presigned-URL delivery to the store.
## 4. Token economics, before and after ## 4. Token economics, before and after
Before the plane, the script pipeline cost $0.022 per candidate for two verify passes Before the plane, two costs stood out:
(`benchmarks/independent/verifier-batched-check-2026-09-29.json`), and the maintainers' own
coding sessions spent about 90% of their tokens on tool input and output rather than reasoning. - The script pipeline cost $0.022 per candidate for two verify passes
After it, the per-candidate costs below are what the records show. The mechanisms (one budget (`benchmarks/independent/verifier-batched-check-2026-09-29.json`).
per Task, pay-once reuse, cheap-before-expensive ordering, replay at $0) are on - The maintainers' own coding sessions spent about 90% of their tokens on tool input and output
[Token ergonomics](token-ergonomics.md); this table is the before and after. rather than reasoning.
After it, the per-candidate costs below are what the records show. The mechanisms are on
[Token ergonomics](token-ergonomics.md): one budget per Task, pay-once reuse,
cheap-before-expensive ordering and replay at $0. This table is the before and after.
| Item | Before | After | Record | | Item | Before | After | Record |
| --- | --- | --- | --- | | --- | --- | --- | --- |
@@ -195,63 +273,70 @@ per Task, pay-once reuse, cheap-before-expensive ordering, replay at $0) are on
| Harvest of plane signals for the engine | none existed | `verify` $2.5373 over 1,961 calls and 474 tasks; `roles` $1.4036 over 1,034 calls and 151 tasks | `2026-10-01-decision-engine-harvest/record.json` | | Harvest of plane signals for the engine | none existed | `verify` $2.5373 over 1,961 calls and 474 tasks; `roles` $1.4036 over 1,034 calls and 151 tasks | `2026-10-01-decision-engine-harvest/record.json` |
| Development transcript, tool I/O share | 90% | 85.5% over 2,701 tool calls (`dev_token_report`) | `plane-increment-1.json` | | Development transcript, tool I/O share | 90% | 85.5% over 2,701 tool calls (`dev_token_report`) | `plane-increment-1.json` |
Two notes from the records. The meter prices every call at the Model's list price Two notes from the records:
(`plane/models/deepseek-flash.yaml`); the DeepSeek account moved less, about one third of the
metered amount on the injection slice ($0.30 for $0.91 metered), and budgets and gates are set 1. The meter prices every call at the Model's list price (`plane/models/deepseek-flash.yaml`). The
against the meter, never against the balance. And the development share barely moved: 90 to 85.5 DeepSeek account moved less: about one third of the metered amount on the injection slice
is a small gain, and the remedies (a read guard, a bulk reader, scripts in files) target file ($0.30 for $0.91 metered). Budgets and gates are set against the meter, never against the
contents because that is where the tokens go. balance.
2. The development share barely moved: 90 to 85.5 is a small gain. The remedies are a read guard,
a bulk reader and scripts in files. They target file contents, because that is where the
tokens go.
## 5. Open items ## 5. Open items
Five groups, each item with its record: the engine's own gates, data, measurement debt, There are five groups. Each item has its record. The groups are the engine's own gates, data,
infrastructure, and instrument gaps. measurement debt, infrastructure, and instrument gaps.
The engine's gates. The engine's gates.
- BLOCK is reachable for no family: calibration fails for five (ECE 0.071 to 0.119) and the - BLOCK is reachable for no family. Calibration fails for five (ECE 0.071 to 0.119). The sixth
sixth reaches no fold with a Wilson lower bound of 0.95 on precision (slice-2 record). reaches no fold with a Wilson lower bound of 0.95 on precision (slice-2 record).
- Re-run agreement is below 0.9 for three families: injection 0.77, access control 0.83, - Re-run agreement is below 0.9 for three families: injection 0.77, access control 0.83,
untrusted destination 0.87. exp-002 tried the obvious variance fix and lost on the ranking. untrusted destination 0.87. exp-002 tried the obvious variance fix and lost on the ranking.
Data. Data.
- Positives-only groups: 98 (family, group) pairs in the 904-example memory hold positive examples - Positives-only groups: 98 (family, group) pairs in the 904-example memory hold positive examples
and no negative. 94 of them exist because the fixed side no longer has the function (the vibe-py and no negative.
pairs). No advisory-fixes group is among them - 94 of them exist because the fixed side no longer has the function (the vibe-py pairs).
(`benchmarks/measurements/2026-10-02-decision-engine-negatives-gap/record.json`). There are no - No advisory-fixes group is among them
single-label negatives, so every family's pooled AUC is inflated by positives that never had to (`benchmarks/measurements/2026-10-02-decision-engine-negatives-gap/record.json`).
be told apart from their own fix (slice-2 record, `paired_groups_check`). - There are no single-label negatives. So every family's pooled AUC is inflated by positives
- The model has seen public fixes. A verify pass named the advisory's CVE id for a public fix, an that never had to be told apart from their own fix (slice-2 record, `paired_groups_check`).
observation the engine's design specification (maintainer tooling) records as visible - The model has seen public fixes. A verify pass named the advisory's CVE id for a public fix.
memorisation. Excerpts now redact advisory ids and security-worded comments The engine's design specification (maintainer tooling) records this as visible memorisation.
(`src/openultrasast/learn/excerpt.py`), but labels drawn from public advisories are partly in the Excerpts now redact advisory ids and security-worded comments
model's training data, and the within-pair numbers carry that. (`src/openultrasast/learn/excerpt.py`). But labels drawn from public advisories are partly in
- PHP is covered by the engine on real code and by in-sample quick rules; the population that the model's training data, and the within-pair numbers carry that.
- PHP is covered by the engine on real code and by in-sample quick rules. The population that
would measure it (v3) is reserved. would measure it (v3) is reserved.
Measurement debt. Measurement debt.
- exp-001 (known callers) is registered and has not run; the model-free candidate generation on - exp-001 (known callers) is registered and has not run. Its units need model-free candidate
the 41 development repositories that its units need is not built generation on the 41 development repositories. That generation is not built
(`benchmarks/experiments/exp-001-known-callers/registration.json`). (`benchmarks/experiments/exp-001-known-callers/registration.json`).
- Leave-one-source-out, leave-one-framework-out and the memory-size curve points (0, 25, 50%) - Leave-one-source-out, leave-one-framework-out and the memory-size curve points (0, 25, 50%)
have not run; one paired point exists (724 against 904 examples, injection, AUC 0.76 against 0.77). have not run. One paired point exists (724 against 904 examples, injection, AUC 0.76 against 0.77).
- The plane-profile leak audit flagged `verify.flag_b` and `ms.operation`; they are withheld from - The plane-profile leak audit flagged `verify.flag_b` and `ms.operation`. They are withheld from
profile v1, and whether they track the label on unseen code has to be decided before any profile v1. Before any plane-profile program runs, someone must decide whether they track the
plane-profile program runs (`2026-10-02-decision-engine-harvest-advisory/record.json`). label on unseen code (`2026-10-02-decision-engine-harvest-advisory/record.json`).
- Population v3 waits for `benchmarks/independent/prediction-v3.json`, which does not exist. It is - Population v3 waits for `benchmarks/independent/prediction-v3.json`, which does not exist. It is
written only after a program is adopted. written only after a program is adopted.
Infrastructure. Infrastructure.
- Deployment to a separate Kubernetes cluster is planned, not done; the code changes it needs are - Deployment to a separate Kubernetes cluster is planned, not done. The code changes it needs are
listed with file paths on [Deployment](deployment.md#what-is-implemented-and-what-is-planned). listed with file paths on [Deployment](deployment.md#what-is-implemented-and-what-is-planned).
Instrument gaps. Instrument gaps.
- A missing provider key is a quiet path in a scan. The plane refuses to start a Task whose Model - A missing provider key is a quiet path in a scan.
names an unset variable (`plane/router.py`, `StartError`). A scan without a key asks the model - The plane refuses to start a Task whose Model names an unset variable (`plane/router.py`,
nothing, reports only the graph's entailed findings, and records `learning_endpoint_unavailable` `StartError`).
in `manifest.json` (`cli.py`, after `resolve_chat_endpoint` returns nothing); `report.md` has no - A scan without a key asks the model nothing and reports only the graph's entailed findings.
line for that reason yet, so the report does not say the model layer was skipped. It records `learning_endpoint_unavailable` in `manifest.json` (`cli.py`, after
`resolve_chat_endpoint` returns nothing).
- `report.md` has no line for that reason yet. So the report does not say the model layer was
skipped.
+1
View File
@@ -1,5 +1,6 @@
site_name: OpenUltraSAST site_name: OpenUltraSAST
site_description: An experimental static security analyser and repository safety net, with an agentic plane and a learned decision engine in development. site_description: An experimental static security analyser and repository safety net, with an agentic plane and a learned decision engine in development.
site_url: https://norandom.github.io/OpenUltraSAST/
repo_url: https://github.com/norandom/OpenUltraSAST repo_url: https://github.com/norandom/OpenUltraSAST
edit_uri: edit/main/docs/ edit_uri: edit/main/docs/
docs_dir: docs docs_dir: docs
+9 -2
View File
@@ -291,13 +291,20 @@ The default capability registry is empty pending independent qualification. Inst
this interface does not make the project rollout-ready. Advisory is the default; this interface does not make the project rollout-ready. Advisory is the default;
blocking and strict incomplete-coverage handling are separate explicit choices. blocking and strict incomplete-coverage handling are separate explicit choices.
Put the installed `ousast` executable on the Git process's PATH. From this tool's The supported install is `ousast pre-push install [REPO] [--force] [--docker]`, shipped in
checkout, explicitly install into the desired repository: the wheel; `ousast pre-push uninstall [REPO]` removes it. The first-run guide, settings, skip
reasons and a CI recipe are in [Scanning](../scanning.md#first-run-install-the-hook-on-any-repository).
If `ousast` is not on the Git process's PATH, the hook prints one line and lets the push through.
From this tool's checkout, the script installer remains available:
```sh ```sh
ops/install-pre-push /absolute/path/to/repository ops/install-pre-push /absolute/path/to/repository
``` ```
`ops/ousast-docker` is the hook-callable Docker wrapper (`OUSAST_COMMAND=/path/to/ops/ousast-docker`,
or installed beside the hook by `--docker`). It is a script, not a shell function, so Git can run
it. It passes the push input through and mounts the artifact directory at its own path.
The installer resolves Git's effective hook directory (including `core.hooksPath`), The installer resolves Git's effective hook directory (including `core.hooksPath`),
leaves that configuration untouched, and refuses to overwrite any existing hook. leaves that configuration untouched, and refuses to overwrite any existing hook.
For an existing hook, inspect it and explicitly move it aside **before** installation: For an existing hook, inspect it and explicitly move it aside **before** installation:
+25 -7
View File
@@ -7,12 +7,30 @@ single-node kind cluster. ax is the only executor; `ousast plane run` submits, r
ops/ax/up.sh # idempotent: kind + registry, Substrate, ax, gVisor worker pool, egress gateway, ops/ax/up.sh # idempotent: kind + registry, Substrate, ax, gVisor worker pool, egress gateway,
# receiver Service, runner image (digest pinned), ax CLI, smoke Task # receiver Service, runner image (digest pinned), ax CLI, smoke Task
ousast plane doctor # kind, Substrate, ax controller, runner image ousast plane doctor # the profile's cluster (ops/k8s/profiles/kind.toml), Substrate, ax, runner image, memory store
ops/ax/smoke-run.sh # end to end: repo-facts on ax, artifacts back on the host, attribution table ops/ax/smoke-run.sh # end to end: repo-facts on ax, artifacts back on the host, attribution table
ops/ax/down.sh # deletes the cluster and its registry ops/ax/down.sh # deletes the cluster and its registry
Tools live in `~/go/bin` (kind, ax, ko, kubectl-ate) and `~/.local/bin` (kubectl); sources and rendered files Tools live in `~/go/bin` (kind, ax, ko, kubectl-ate) and `~/.local/bin` (kubectl); sources and rendered files
in `~/.cache/ousast/ax-src/` (the runner digest pin is `runner-image` there). in `~/.cache/ousast/ax-src/`. The cluster's context, registry and pool sizes come from the kind profile
(`ops/k8s/profiles/kind.toml`); `up.sh` writes the runner's digest pin to that profile's images file
(`ops/k8s/profiles/kind-images.json`), which the templates are re-pinned from and which is committed with them.
`up.sh` and `smoke-run.sh` run the checkout's code with `OUSAST_PYTHON` (default `.venv/bin/python`).
The kind profile runs on a 7.7 GB server VM. Observability is off by default
(`kind_observability = false` in `ops/k8s/profiles/kind.toml`). This saves the measured
930 MiB used by `otel-system`, about 40% of the cluster's 2.3 GB. The smoke Run still
completes with those deployments stopped. To enable the stack, run
`OUSAST_KIND_OBSERVABILITY=1 ops/ax/up.sh` or set `kind_observability = true` in the profile.
`up.sh` stages copied Substrate manifests in a temporary checkout view and installs through
`hack/install-ate-kind.sh`. The local `ops/ax/substrate-kind-lean/` overlay removes the collector,
Prometheus, Jaeger and their supporting resources before installation. This preserves the
installer's secret, CRD and image setup. The upstream checkout is unchanged. The shared
`ate-otel-config` disables trace, metric and log export and the OTEL SDK; it removes the old
collector endpoint and metric timing keys. On an existing cluster, `up.sh` scales the three
observability deployments to zero, replaces the ConfigMap data and rolls its consumers when
the configuration changes. Opting in restores the upstream ConfigMap and observability manifests.
## How a Run executes ## How a Run executes
@@ -188,20 +206,20 @@ configure the bucket.
No deployment outside this host's kind cluster has been made. The step-by-step guidance, with what is implemented No deployment outside this host's kind cluster has been made. The step-by-step guidance, with what is implemented
and what is planned, is `docs/deployment.md` (the site page "Deployment"). The Model and Workspace manifests under and what is planned, is `docs/deployment.md` (the site page "Deployment"). The Model and Workspace manifests under
`plane/` move as they are and the Task templates need their image reference re-pinned to your registry; these parts `plane/` move as they are and the Task templates need their image reference re-pinned to your registry; these parts
assume this laptop and must change first (checked 2026-10-02): assume this server VM and must change first (checked 2026-10-02):
| Assumption | Where | Needed in a real cluster | | Assumption | Where | Needed in a real cluster |
|---|---|---| |---|---|---|
| Artifact receiver runs on the developer host, reached through the `ousast-receiver` Service with an EndpointSlice to the kind gateway `172.19.0.1:18090` | `egress.py`, `receiver-service.yaml.tmpl` | the receiver as an in-cluster Deployment (or object storage, e.g. Substrate's S3-compatible store), with the reconciler reading from it | | Artifact receiver runs on the developer host, reached through the `ousast-receiver` Service with an EndpointSlice to the kind network's gateway address on `OUSAST_ARTIFACT_PORT` | `egress.py`, `receiver-service.yaml.tmpl` | the receiver as an in-cluster Deployment (or object storage, e.g. Substrate's S3-compatible store), with the reconciler reading from it |
| Registry `localhost:5001`, rewritten by Substrate for kind | `up.sh` (`KO_DOCKER_REPO`), `doctor.py:45` (hard-coded check), the runner digest pin, the `image` of every template under `plane/tasks/` | a registry the workers can pull from; keep digest pins (Substrate rejects tags); `--runner-image FILE` re-pins generated Tasks | | The kind-local registry (`registry` in `ops/k8s/profiles/kind.toml`), rewritten by Substrate for kind | `up.sh` (`KO_DOCKER_REPO`), the profile's images file, the `image` of every template under `plane/tasks/` | the `k3s` profile's registry; keep digest pins (Substrate rejects tags); `--runner-image FILE` re-pins generated Tasks |
| kubectl context `kind-$KIND_CLUSTER_NAME` | `doctor.py:39`, `egress.py:115` and `:148`, `router.py:145`, `up.sh:24` | one configurable context (for example `OUSAST_KUBE_CONTEXT`); not yet configurable | | kubectl context | `kube_context` in the profile, read by `doctor.py`, `egress.py`, `router.py` (plane-on-kubernetes 1.2); `up.sh` reads the same profile | the `k3s` profile's context; nothing else changes |
| ax's snapshot bucket: `AX_SNAPSHOTS_BUCKET` in ax's `deploy/ax-server.yaml` points at the ax authors' GCS bucket | ax deploy manifest | your own bucket, set before deploying ax | | ax's snapshot bucket: `AX_SNAPSHOTS_BUCKET` in ax's `deploy/ax-server.yaml` points at the ax authors' GCS bucket | ax deploy manifest | your own bucket, set before deploying ax |
| Egress gateway applied by hand (agentgateway variant, no Rust build) | this README | the Substrate-installed gateway; per-task EgressPolicies work unchanged | | Egress gateway applied by hand (agentgateway variant, no Rust build) | this README | the Substrate-installed gateway; per-task EgressPolicies work unchanged |
| Worker pool of 2 x 1 CPU / 1.5 GiB for the 7 GB host | `workerpool.yaml.tmpl` | sized to the cluster; `--workers` to match | | Worker pool of 2 x 1 CPU / 1.5 GiB for the 7 GB host | `workerpool.yaml.tmpl` | sized to the cluster; `--workers` to match |
| Plane memory store and results under `~/ousast-results/` on the host | `reconciler.py` (`OUSAST_RESULTS`), `memory.py` (`OUSAST_MEMORY`) | a persistent volume or bucket shared by the reconciler; the `s3://` store already works against any reachable S3-compatible server | | Plane memory store and results under `~/ousast-results/` on the host | `reconciler.py` (`OUSAST_RESULTS`), `memory.py` (`OUSAST_MEMORY`) | a persistent volume or bucket shared by the reconciler; the `s3://` store already works against any reachable S3-compatible server |
The provider key already travels only in the start request through `atenet-router`, which works the same through The provider key already travels only in the start request through `atenet-router`, which works the same through
a port-forward to any cluster. the kube-context tunnel to any cluster.
## What a larger ax deployment needs ## What a larger ax deployment needs
+43
View File
@@ -0,0 +1,43 @@
#!/usr/bin/env python3
"""Stage an installer checkout with lean kind manifests; never modify the source checkout.
Usage: python ops/ax/prepare-substrate.py SOURCE EMPTY_DESTINATION
The destination lives only for an install/render. Source/build inputs are symlinked;
manifests are copied because the installer uses fixed paths (including the OTEL ConfigMap).
"""
from __future__ import annotations
import shutil
import sys
from pathlib import Path
import yaml
def prepare(source: Path, destination: Path) -> None:
source = source.resolve()
destination.mkdir(parents=True, exist_ok=True)
if any(destination.iterdir()):
raise ValueError(f"staging directory must be empty: {destination}")
for entry in source.iterdir():
if entry.name != "manifests":
(destination / entry.name).symlink_to(entry, target_is_directory=entry.is_dir())
shutil.copytree(source / "manifests", destination / "manifests")
install = destination / "manifests/ate-install"
shutil.copytree(install / "kind", install / "kind-upstream")
overlay = Path(__file__).resolve().parent / "substrate-kind-lean"
shutil.copytree(overlay, install / "kind", dirs_exist_ok=True)
# The installer applies this file directly before it renders the bundle.
# Keep the strategic-merge directive only in the overlay's patch file.
config = yaml.safe_load((overlay / "ate-otel-config.yaml").read_text())
config["data"].pop("$patch")
(install / "kind/disabled-otel-config.yaml").write_text(yaml.safe_dump(config))
(install / "kind/ate-otel-config.yaml").rename(install / "kind/otel-config-patch.yaml")
(install / "kind/disabled-otel-config.yaml").rename(install / "kind/ate-otel-config.yaml")
kustomization = install / "kind/kustomization.yaml"
kustomization.write_text(kustomization.read_text().replace("path: ate-otel-config.yaml", "path: otel-config-patch.yaml"))
if __name__ == "__main__":
prepare(Path(sys.argv[1]), Path(sys.argv[2]))
+81
View File
@@ -0,0 +1,81 @@
#!/usr/bin/env bash
# Placement probe (plane-on-kubernetes task 1.5, design open question 2): does Agent Substrate place an actor on a
# worker that fits its memory request, or on any free worker of the sandbox class?
#
# Applies a second WorkerPool `probe-pool` (1 replica, 2 CPU / 3Gi) from ops/ax/workerpool.yaml.tmpl with the worker
# image the existing pool already built (no ko build), scales the default pool to one worker so the 7 GB host holds
# both, then runs one model-free repo-facts Task asking 2560Mi (limit 3Gi) and one asking 256Mi, recording after
# each resume which worker holds the actor (`kubectl ate get workers -o json`). Everything it created is removed
# on exit and the default pool is scaled back. Usage: ops/ax/probe-placement.sh OUT_DIR (the JSON evidence lands there).
set -euo pipefail
export PATH="$HOME/go/bin:$HOME/.local/bin:$PATH"
OUT="${1:?usage: probe-placement.sh OUT_DIR}"
mkdir -p "$OUT"
REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
PY="${OUSAST_PYTHON:-$REPO_ROOT/.venv/bin/python}"
profile() { PYTHONPATH="$REPO_ROOT/src" "$PY" -m openultrasast.plane.profile --profile "${OUSAST_PLANE_PROFILE:-kind}" --print "$@"; }
CTX="$(profile kube_context)"
RUNNER_IMAGE="$(profile images.runner)"
DEFAULT_REPLICAS="$(profile pools.default.replicas)"
K() { kubectl --context "$CTX" -n ax-system "$@"; }
log() { printf '%s %s\n' "$(date +%H:%M:%S)" "$*" | tee -a "$OUT/probe.log"; }
cleanup() {
rc=$?
log "cleanup (exit $rc): delete the probe tasks and probe-pool, scale ousast-pool back to $DEFAULT_REPLICAS"
for t in ousast-probe-big ousast-probe-small; do ax delete task "$t" >/dev/null 2>&1 || true; done
K delete workerpool probe-pool --ignore-not-found >/dev/null 2>&1 || true
K patch workerpool ousast-pool --type merge -p "{\"spec\":{\"replicas\":$DEFAULT_REPLICAS}}" >/dev/null 2>&1 || true
echo "probe exit $rc"
}
trap cleanup EXIT
workers() { kubectl ate --context "$CTX" get workers -o json > "$OUT/$1.json"; kubectl ate --context "$CTX" get workers | tee -a "$OUT/probe.log"; }
export POOL_NAME=probe-pool POOL_REPLICAS=1 POOL_CPU=2 POOL_MEMORY=3Gi
WORKER_IMAGE="$(K get workerpool ousast-pool -o jsonpath='{.spec.workerImage}')" # already built and pulled by the node
SUBSTRATE_VERSION="$(kubectl --context "$CTX" get nodes -o jsonpath='{.items[0].metadata.labels.ate\.dev/substrate-version}')"
export WORKER_IMAGE SUBSTRATE_VERSION
log "default pool to 1 replica (room for a 3Gi worker on this host)"
K patch workerpool ousast-pool --type merge -p '{"spec":{"replicas":1}}'
for i in $(seq 1 60); do [ "$(K get pods -l ate.dev/worker-pool=ousast-pool --no-headers 2>/dev/null | wc -l)" = 1 ] && break; sleep 2; done
log "apply probe-pool: $POOL_REPLICAS x $POOL_CPU CPU / $POOL_MEMORY, image $WORKER_IMAGE"
envsubst < "$(dirname "$0")/workerpool.yaml.tmpl" | tee "$OUT/probe-pool.yaml" | K apply -f -
K wait --for=condition=Ready pod -l ate.dev/worker-pool=probe-pool --timeout=600s
workers workers-before
task() { # name, request memory, limit memory
cat > "$OUT/$1.yaml" <<EOF
apiVersion: ax.io/v1alpha1
kind: Workspace
metadata: {name: $1-ws, atespace: default}
spec:
files:
- {path: "hello.py", content: "def hello():\n return greet()\n"}
---
apiVersion: ax.io/v1alpha1
kind: Task
metadata: {name: $1, atespace: default}
spec:
image: "$RUNNER_IMAGE"
command: ["repo-facts"]
env: [{name: OUSAST_OUTPUT_DIR, value: "/workspace/.ousast-out/probe"}]
resources: {requests: {cpu: "250m", memory: "$2"}, limits: {cpu: "1", memory: "$3"}}
workspaces: [{name: $1-ws, path: "/workspace"}]
EOF
ax delete task "$1" >/dev/null 2>&1 || true
log "apply $1 (requests $2, limits $3)"
ax apply -f "$OUT/$1.yaml"
for i in 1 2 3 4 5 6 7 8 9; do
if ax resume task "$1" 2>&1 | tee -a "$OUT/probe.log"; then break; fi # the first resume may wait for the golden snapshot
sleep 20
done
sleep 15
ax get task "$1" | tee "$OUT/$1.status.yaml" | grep -E "phase|message" || true
workers "workers-$1"
ax delete task "$1" >/dev/null 2>&1 || true
for i in $(seq 1 30); do kubectl ate --context "$CTX" get workers | grep -q " 1/1000 " || break; sleep 2; done
}
task ousast-probe-big 2560Mi 3Gi
task ousast-probe-small 256Mi 512Mi
log "done; evidence under $OUT"
+9 -5
View File
@@ -1,14 +1,18 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Render the end-to-end smoke Run with the current runner digest and execute it through the reconciler on ax. # Render the end-to-end smoke Run with the profile's runner digest and execute it through the reconciler on ax.
# The profile (OUSAST_PLANE_PROFILE, default kind; ops/k8s/profiles/) names the images file up.sh wrote.
set -euo pipefail set -euo pipefail
export PATH="$HOME/go/bin:$HOME/.local/bin:$PATH" export PATH="$HOME/go/bin:$HOME/.local/bin:$PATH"
SRC="${OUSAST_AX_SRC:-$HOME/.cache/ousast/ax-src}" SRC="${OUSAST_AX_SRC:-$HOME/.cache/ousast/ax-src}"
RUNNER_IMAGE="$(cat "$SRC/runner-image")"; export RUNNER_IMAGE REPO="$(cd "$(dirname "$0")/../.." && pwd)"
PY="${OUSAST_PYTHON:-$REPO/.venv/bin/python}"
RUNNER_IMAGE="$(PYTHONPATH="$REPO/src" "$PY" -m openultrasast.plane.profile --profile "${OUSAST_PLANE_PROFILE:-kind}" --print images.runner)"
export RUNNER_IMAGE
# Actors reach the receiver by its Service name through the egress gateway (ops/ax/receiver-service.yaml.tmpl, # Actors reach the receiver by its Service name through the egress gateway (ops/ax/receiver-service.yaml.tmpl,
# applied by up.sh, endpoint this host at OUSAST_ARTIFACT_PORT): the runner dials the Service's ClusterIP on port # applied by up.sh, endpoint this host at OUSAST_ARTIFACT_PORT): the runner dials the Service's ClusterIP on port
# 80 with that name as Host; the reconciler looks the ClusterIP up unless OUSAST_ARTIFACT_DIAL is set. # 80 with that name as Host; the reconciler looks the ClusterIP up unless OUSAST_ARTIFACT_DIAL is set.
export OUSAST_ARTIFACT_PORT="${OUSAST_ARTIFACT_PORT:-18090}" export OUSAST_ARTIFACT_PORT="${OUSAST_ARTIFACT_PORT:-18090}"
envsubst < "$(dirname "$0")/smoke-run.yaml.tmpl" > "$SRC/smoke-run.yaml" envsubst < "$(dirname "$0")/smoke-run.yaml.tmpl" > "$SRC/smoke-run.yaml"
REPO="$(cd "$(dirname "$0")/../.." && pwd)" ousast() { PYTHONPATH="$REPO/src" "$PY" -m openultrasast.cli "$@"; } # this checkout's code, whatever python runs it
"$REPO/.venv/bin/ousast" plane run "$SRC/smoke-run.yaml" "$@" ousast plane run "$SRC/smoke-run.yaml" "$@"
"$REPO/.venv/bin/ousast" plane status ax-e2e-smoke ousast plane status ax-e2e-smoke
@@ -0,0 +1,11 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: ate-otel-config
namespace: ate-system
data:
$patch: replace
OTEL_TRACES_EXPORTER: none
OTEL_METRICS_EXPORTER: none
OTEL_LOGS_EXPORTER: none
OTEL_SDK_DISABLED: "true"
@@ -0,0 +1,44 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# prepare-substrate.py stages the upstream kind overlay beside this directory.
resources:
- ../kind-upstream
patches:
- target:
namespace: otel-system
patch: |-
$patch: delete
apiVersion: v1
kind: ConfigMap
metadata:
name: unused
- target:
kind: Namespace
name: otel-system
patch: |-
$patch: delete
apiVersion: v1
kind: Namespace
metadata:
name: otel-system
- target:
kind: ClusterRole
name: ate-prometheus
patch: |-
$patch: delete
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ate-prometheus
- target:
kind: RoleBinding
name: ate-prometheus
namespace: ate-system
patch: |-
$patch: delete
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ate-prometheus
namespace: ate-system
- path: ate-otel-config.yaml
+91 -13
View File
@@ -1,13 +1,22 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Bring up ax on this host: kind cluster + local registry, Agent Substrate, the ax control plane, one smoke # Bring up ax on this host: kind cluster + local registry, Agent Substrate, the ax control plane, one smoke
# Task. Idempotent enough to rerun; `down.sh` removes everything. Requires docker, go, kubectl, kind, ko, ax. # Task. Idempotent: every step is skipped when already done; `down.sh` removes everything. Requires docker, go,
# kubectl, kind, ko, ax. The cluster's context, the registry and the pool sizes come from the kind profile
# (ops/k8s/profiles/kind.toml; OUSAST_PLANE_PROFILE selects another file with exec = "local"). This script may
# name the kind cluster because it creates it; the code reads the profile (tests/test_plane_literals.py).
set -euo pipefail set -euo pipefail
trap 'rc=$?; echo "up.sh FAILED at line $LINENO (exit $rc)"' ERR trap 'rc=$?; echo "up.sh FAILED at line $LINENO (exit $rc)"' ERR
trap 'echo "up.sh exit $?"' EXIT trap 'echo "up.sh exit $?"' EXIT
export PATH="$HOME/go/bin:$HOME/.local/bin:$PATH" export PATH="$HOME/go/bin:$HOME/.local/bin:$PATH"
export GOTOOLCHAIN=auto export GOTOOLCHAIN=auto
export KIND_CLUSTER_NAME="${KIND_CLUSTER_NAME:-ousast}" REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
export KO_DOCKER_REPO="${KO_DOCKER_REPO:-localhost:5001}" PY="${OUSAST_PYTHON:-$REPO_ROOT/.venv/bin/python}"
profile() { PYTHONPATH="$REPO_ROOT/src" "$PY" -m openultrasast.plane.profile --profile "${OUSAST_PLANE_PROFILE:-kind}" --print "$@"; }
CTX="$(profile kube_context)" # kind names its context kind-<cluster>
export KIND_CLUSTER_NAME="${CTX#kind-}"
export KO_DOCKER_REPO="${KO_DOCKER_REPO:-$(profile registry)}"
OBSERVABILITY="$(profile kind_observability)"
IMAGES_FILE="$(profile images)" # the digest pins the reconciler and the templates read
SRC="${OUSAST_AX_SRC:-$HOME/.cache/ousast/ax-src}" SRC="${OUSAST_AX_SRC:-$HOME/.cache/ousast/ax-src}"
for t in docker go kubectl kind ko ax; do for t in docker go kubectl kind ko ax; do
@@ -19,9 +28,8 @@ clone substrate https://github.com/agent-substrate/substrate.git
clone ax https://github.com/google/ax.git clone ax https://github.com/google/ax.git
step() { printf '\n== %s (%s)\n' "$1" "$(date +%H:%M:%S)"; } step() { printf '\n== %s (%s)\n' "$1" "$(date +%H:%M:%S)"; }
step "kind cluster '$KIND_CLUSTER_NAME' + registry $KO_DOCKER_REPO" step "kind cluster '$KIND_CLUSTER_NAME' (context $CTX) + registry $KO_DOCKER_REPO"
# Each step is skipped when already done: the substrate helper DELETES an existing cluster before creating one. # Each step is skipped when already done: the substrate helper DELETES an existing cluster before creating one.
CTX="kind-$KIND_CLUSTER_NAME"
ready() { kubectl --context "$CTX" -n "$1" get pods --no-headers 2>/dev/null | awk '$2 ~ /^[0-9]+\/[0-9]+$/ {n++; split($2,a,"/"); if (a[1]!=a[2] && $3!="Completed") bad++} END {exit !(n>0 && bad==0)}'; } ready() { kubectl --context "$CTX" -n "$1" get pods --no-headers 2>/dev/null | awk '$2 ~ /^[0-9]+\/[0-9]+$/ {n++; split($2,a,"/"); if (a[1]!=a[2] && $3!="Completed") bad++} END {exit !(n>0 && bad==0)}'; }
if kind get clusters 2>/dev/null | grep -qx "$KIND_CLUSTER_NAME"; then if kind get clusters 2>/dev/null | grep -qx "$KIND_CLUSTER_NAME"; then
echo "cluster exists; kept" echo "cluster exists; kept"
@@ -30,9 +38,68 @@ else
fi fi
step "agent substrate (ate-system)" step "agent substrate (ate-system)"
# The upstream installer has fixed manifest paths and also creates secrets/CRDs.
# Give it a temporary checkout view so it never installs the observability resources.
install_substrate() (
if [ "$OBSERVABILITY" = true ]; then
cd "$SRC/substrate"
else
# Symlinked build inputs must not make the staged view a different binary version.
export VERSION="${VERSION:-$(git -C "$SRC/substrate" describe --tags --always --dirty)}"
staged="$(mktemp -d)"
trap 'rm -rf "$staged"' EXIT
"$PY" "$REPO_ROOT/ops/ax/prepare-substrate.py" "$SRC/substrate" "$staged"
cd "$staged"
fi
KUBECTL_CONTEXT="$CTX" hack/install-ate-kind.sh --deploy-ate-system
)
if ready ate-system; then echo "substrate ready; kept"; else if ready ate-system; then echo "substrate ready; kept"; else
(cd "$SRC/substrate" && hack/install-ate-kind.sh --deploy-ate-system) install_substrate
fi fi
# Reconcile the choice even when the existing control plane is already ready.
if [ "$OBSERVABILITY" = true ]; then
for manifest in otel-collector prometheus; do
kubectl --context "$CTX" apply -f "$SRC/substrate/manifests/ate-install/kind/$manifest.yaml"
done
otel_config="$SRC/substrate/manifests/ate-install/kind/ate-otel-config.yaml"
else
otel_config="$REPO_ROOT/ops/ax/substrate-kind-lean/ate-otel-config.yaml"
for deployment in opentelemetry-collector prometheus jaeger; do
existing="$(kubectl --context "$CTX" -n otel-system get deployment "$deployment" --ignore-not-found -o name)"
if [ -n "$existing" ]; then
kubectl --context "$CTX" -n otel-system scale "$existing" --replicas=0
fi
done
echo "kind observability off: existing otel-system deployments scaled to 0; opt in with OUSAST_KIND_OBSERVABILITY=1 ops/ax/up.sh"
fi
# Replace data to remove stale exporter/endpoint keys in either direction. ConfigMap
# envFrom is read at pod start; a content annotation triggers only changed rollouts.
otel_data="$("$PY" - "$otel_config" <<'PYCONFIG'
import json
import sys
import yaml
config = yaml.safe_load(open(sys.argv[1]))
config["data"].pop("$patch", None)
print(json.dumps(config["data"], sort_keys=True))
PYCONFIG
)"
kubectl --context "$CTX" -n ate-system patch configmap ate-otel-config --type=json \
-p "[{\"op\":\"replace\",\"path\":\"/data\",\"value\":$otel_data}]"
otel_hash="$(printf '%s' "$otel_data" | sha256sum | cut -d ' ' -f1)"
otel_workloads="$(kubectl --context "$CTX" -n ate-system get deployments,daemonsets -o json | "$PY" -c '
import json, sys
for item in json.load(sys.stdin)["items"]:
spec = item["spec"]["template"]["spec"]
containers = spec.get("containers", []) + spec.get("initContainers", [])
if any(env.get("configMapRef", {}).get("name") == "ate-otel-config"
for container in containers for env in container.get("envFrom", [])):
print(item["kind"].lower() + "/" + item["metadata"]["name"])
')"
for workload in $otel_workloads; do
kubectl --context "$CTX" -n ate-system patch "$workload" --type=merge \
-p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"ousast.io/otel-config\":\"$otel_hash\"}}}}}"
kubectl --context "$CTX" -n ate-system rollout status "$workload" --timeout=300s
done
kubectl --context "$CTX" -n ate-system get pods kubectl --context "$CTX" -n ate-system get pods
step "egress gateway (atenet-egress, agentgateway variant: one prebuilt image, nothing built)" step "egress gateway (atenet-egress, agentgateway variant: one prebuilt image, nothing built)"
@@ -56,25 +123,36 @@ KIND_GATEWAY="$(docker network inspect kind --format '{{range .IPAM.Config}}{{if
export KIND_GATEWAY OUSAST_ARTIFACT_PORT="${OUSAST_ARTIFACT_PORT:-18090}" # the reconciler's fixed receiver port export KIND_GATEWAY OUSAST_ARTIFACT_PORT="${OUSAST_ARTIFACT_PORT:-18090}" # the reconciler's fixed receiver port
envsubst < "$(dirname "$0")/receiver-service.yaml.tmpl" | kubectl --context "$CTX" apply -f - envsubst < "$(dirname "$0")/receiver-service.yaml.tmpl" | kubectl --context "$CTX" apply -f -
step "worker pool (gVisor)" step "worker pool (gVisor): pools.default of the profile"
# `ko apply` builds the worker image from the Substrate checkout (several GB of Go caches): only when the pool
# is absent. A size change in the profile is applied with `kubectl patch` or by deleting the pool first.
export POOL_NAME=ousast-pool POOL_REPLICAS="$(profile pools.default.replicas)" POOL_CPU="$(profile pools.default.cpu)" POOL_MEMORY="$(profile pools.default.memory)"
export WORKER_IMAGE="ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor"
SUBSTRATE_VERSION="$(kubectl --context "$CTX" get nodes -o jsonpath='{.items[0].metadata.labels.ate\.dev/substrate-version}')" SUBSTRATE_VERSION="$(kubectl --context "$CTX" get nodes -o jsonpath='{.items[0].metadata.labels.ate\.dev/substrate-version}')"
export SUBSTRATE_VERSION export SUBSTRATE_VERSION
envsubst < "$(dirname "$0")/workerpool.yaml.tmpl" > "$SRC/workerpool.yaml" envsubst < "$(dirname "$0")/workerpool.yaml.tmpl" > "$SRC/workerpool.yaml"
(cd "$SRC/substrate" && KO_DOCKER_REPO="$KO_DOCKER_REPO" ko apply -f "$SRC/workerpool.yaml") if kubectl --context "$CTX" -n ax-system get workerpool "$POOL_NAME" >/dev/null 2>&1; then
kubectl --context "$CTX" -n ax-system wait --for=condition=Ready pod -l ate.dev/worker-pool=ousast-pool --timeout=600s \ echo "pool $POOL_NAME exists ($(kubectl --context "$CTX" -n ax-system get workerpool "$POOL_NAME" -o jsonpath='{.status.readyReplicas}/{.spec.replicas}') ready); kept"
else
(cd "$SRC/substrate" && KO_DOCKER_REPO="$KO_DOCKER_REPO" ko apply -f "$SRC/workerpool.yaml")
fi
kubectl --context "$CTX" -n ax-system wait --for=condition=Ready pod -l "ate.dev/worker-pool=$POOL_NAME" --timeout=600s \
|| kubectl --context "$CTX" -n ax-system get pods || kubectl --context "$CTX" -n ax-system get pods
step "runner image $KO_DOCKER_REPO/ousast-runner" step "runner image $KO_DOCKER_REPO/ousast-runner"
REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
docker build -q -f "$REPO_ROOT/plane/Dockerfile.runner" -t "$KO_DOCKER_REPO/ousast-runner:dev" "$REPO_ROOT" docker build -q -f "$REPO_ROOT/plane/Dockerfile.runner" -t "$KO_DOCKER_REPO/ousast-runner:dev" "$REPO_ROOT"
docker push -q "$KO_DOCKER_REPO/ousast-runner:dev" docker push -q "$KO_DOCKER_REPO/ousast-runner:dev"
RUNNER_IMAGE="$(docker inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' "$KO_DOCKER_REPO/ousast-runner:dev" | grep "^$KO_DOCKER_REPO/ousast-runner@" | head -1)" RUNNER_IMAGE="$(docker inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' "$KO_DOCKER_REPO/ousast-runner:dev" | grep "^$KO_DOCKER_REPO/ousast-runner@" | head -1)"
[ -n "$RUNNER_IMAGE" ] || { echo "no digest for the runner image" >&2; exit 2; } [ -n "$RUNNER_IMAGE" ] || { echo "no digest for the runner image" >&2; exit 2; }
export RUNNER_IMAGE; echo "$RUNNER_IMAGE" > "$SRC/runner-image" # the reconciler reads this pin export RUNNER_IMAGE
echo "runner image $RUNNER_IMAGE" printf '{\n "runner": "%s"\n}\n' "$RUNNER_IMAGE" > "$IMAGES_FILE" # the profile's images file: commit it with the templates
echo "runner image $RUNNER_IMAGE -> $IMAGES_FILE"
echo "re-pin the templates with: ousast plane workspaces ... --runner-image $IMAGES_FILE (or ousast plane repin, task 4.3)"
step "ax CLI from the deployed checkout (a release CLI skews from the server)" step "ax CLI from the deployed checkout (a release CLI skews from the server)"
(cd "$SRC/ax" && go install ./cmd/ax) if [ -x "$HOME/go/bin/ax" ] && [ "$HOME/go/bin/ax" -nt "$SRC/ax/.git/HEAD" ]; then echo "ax CLI newer than the checkout; kept"; else
(cd "$SRC/ax" && go install ./cmd/ax)
fi
step "smoke task" step "smoke task"
ax delete task ousast-smoke >/dev/null 2>&1 || true ax delete task ousast-smoke >/dev/null 2>&1 || true
+12 -9
View File
@@ -1,23 +1,26 @@
# Warm gVisor workers for ax tasks on this host (Agent Substrate WorkerPool). ax's per-task ActorTemplates # Warm gVisor workers for ax tasks (Agent Substrate WorkerPool), rendered by ops/ax/up.sh (and by
# set no workerSelector, so every pool of the matching sandbox class is eligible. Two workers of 1 CPU / # ops/ax/probe-placement.sh for the placement probe) from the profile's pool: POOL_NAME, POOL_REPLICAS, POOL_CPU,
# 1.5 GiB fit the 7 GB host beside the control planes; an actor occupies a whole worker. # POOL_MEMORY (ops/k8s/profiles/<profile>.toml, [pools.default]) and WORKER_IMAGE (ko:// for a build, or a
# digest-pinned reference of a built worker image). ax's per-task ActorTemplates set no workerSelector, so every
# pool of the matching sandbox class is eligible. An actor occupies a whole worker. Replaced by the renderer of
# plane-on-kubernetes task 6.1 (ops/k8s/base/workerpool-*.yaml.tmpl).
apiVersion: ate.dev/v1alpha1 apiVersion: ate.dev/v1alpha1
kind: WorkerPool kind: WorkerPool
metadata: metadata:
name: ousast-pool name: ${POOL_NAME}
namespace: ax-system namespace: ax-system
labels: labels:
workload: ousast workload: ousast
spec: spec:
replicas: 2 replicas: ${POOL_REPLICAS}
workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor workerImage: ${WORKER_IMAGE}
template: template:
nodeSelector: nodeSelector:
ate.dev/substrate-version: "${SUBSTRATE_VERSION}" ate.dev/substrate-version: "${SUBSTRATE_VERSION}"
resources: resources:
limits: limits:
cpu: "1" cpu: "${POOL_CPU}"
memory: 1536Mi memory: ${POOL_MEMORY}
requests: requests:
cpu: 250m cpu: 250m
memory: 1536Mi memory: ${POOL_MEMORY}
+85
View File
@@ -0,0 +1,85 @@
# Engine workers on k3s
Build the new **Engine image** workflow on main (or dispatch it), then copy its
`ghcr.io/<owner>/openultrasast@sha256:...` job-summary value. The old September
image does not include boto3 or this service. The release workflow still builds
version tags. The Dockerfile copies the worker and parser under `/app/benchmarks/learn`
and installs `.[semantic,s3]`; no analyzer tarball is sent by the dispatcher.
Use two schedulable nodes with at least 3 GiB available each. Required hostname
anti-affinity places one worker on each node. Updates use Recreate so a third
worker does not wait forever for a third node. Local scratch uses the container's
filesystem, with no host mount or Kubernetes API access.
Create a **new scoped account** using the policy in `docs/rustfs.md`. Export its
`S3_ENDPOINT`, `S3_BUCKET=sast-memory`, `AWS_ACCESS_KEY_ID`, and
`AWS_SECRET_ACCESS_KEY` in the coordinator shell. Do not use the memory-store
account. The following commands are for the coordinator; they were not run by the
implementation agent.
```sh
export ENGINE_IMAGE='ghcr.io/<owner>/openultrasast@sha256:<digest-from-workflow>'
# From the repository root, edit the digest parameter without changing the template:
export ENGINE_OVERLAY="$(mktemp -d /tmp/ousast-engine.XXXXXX)"
cp ops/k8s/engine/{namespace,deployment,kustomization}.yaml "$ENGINE_OVERLAY/"
python - <<'PY'
import os
from pathlib import Path
image, digest = os.environ['ENGINE_IMAGE'].split('@', 1)
p = Path(os.environ['ENGINE_OVERLAY']) / 'kustomization.yaml'
p.write_text(p.read_text().replace('ghcr.io/OWNER/openultrasast', image)
.replace('sha256:REPLACE_WITH_ENGINE_IMAGE_DIGEST', digest))
PY
kubectl apply -f ops/k8s/engine/namespace.yaml
kubectl -n ousast-engine create secret generic ousast-engine-s3 \
--from-literal=S3_ENDPOINT="$S3_ENDPOINT" \
--from-literal=S3_BUCKET="$S3_BUCKET" \
--from-literal=AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \
--from-literal=AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY"
kubectl apply -k "$ENGINE_OVERLAY"
kubectl -n ousast-engine rollout status deployment/ousast-engine-worker
kubectl -n ousast-engine get pods -o wide
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
export PYTHONPATH="$PWD/src"
python benchmarks/learn/engine_trace.py --queue-status
python benchmarks/learn/engine_trace.py --executor queue \
--image "$ENGINE_IMAGE" --parallel 2 --deadline 900 --queue-timeout 3600 \
--units plane/experiments/exp-005-graph-slice.units.jsonl \
--out "$HOME/ousast-results/plane/engine-trace-queue"
```
The dispatcher uses the existing local memory examples by default. Add
`--examples /path/to/examples.jsonl` for a frozen example snapshot. Materialization
uses locally cached source pins, just as the Docker executor does. `--executor
k8s-jobs` preserves the old Job lane (`k8s` is its legacy alias); `queue` requires
no kubectl access. Resume skips completed local units and retries failed or timed
out units. `progress.json`, `STOP`, filtering, and `--summary` retain their existing
meaning. Each invocation uses a new run ID, so an old remote result cannot satisfy
a new retry. The dispatcher records worker and pod identity in each pin JSON.
Tasks contain the pin, units, prepared questions, source object key, and deadlines.
Results live at `engine-results/<run>/<pin-id>.json`; inputs are removed after
publication. The Docker lane retains JSON and a log, not a result tar, so the queue
also returns JSON. Queue timeout records timeout locally and removes pending;
an already claimed task finishes remotely and cleans its input. Orphaned inputs
from a failed submission can be removed under `engine-queue/inputs/` after checking
that no pending task or claim references them.
Claims use per-worker choosing markers followed by ordered tickets. A worker waits
for choosing peers and every lower ticket before removing pending and starting.
This requires strongly consistent GET/PUT/LIST from RustFS, including paginated
LIST, and synchronized node clocks with skew below 30 seconds. Half of the 60-second lease margin is reserved for clock
skew; the remaining execution budget is checked again after claiming.
It does not require conditional writes, object tags, S3 Select, or bucket-admin
permissions. A dead worker's task is recovered from its expired claim. Execution
runs in a child process. Linux subreaping and a frozen descendant walk kill its
entire tree, including Joern's separate sessions, at the task deadline before the
lease expires. As with any unfenced lease, a node suspended beyond its lease must be
terminated before it rejoins; the protocol cannot guarantee exclusivity against
arbitrary process suspension or an inconsistent object store.
SIGTERM stops new claims and lets the current task finish. The 900-second
termination grace matches the maximum task deadline. Keep these values aligned
if changing the deployment. `/healthz` serves both probes and becomes unready on
shutdown. Errors omit SDK exception text to avoid exposing credentials.
+63
View File
@@ -0,0 +1,63 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: ousast-engine-worker
spec:
replicas: 2
strategy:
type: Recreate
selector:
matchLabels:
app: ousast-engine-worker
template:
metadata:
labels:
app: ousast-engine-worker
spec:
automountServiceAccountToken: false
terminationGracePeriodSeconds: 900
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: ousast-engine-worker
topologyKey: kubernetes.io/hostname
containers:
- name: worker
image: engine-image
command: [python, /app/benchmarks/learn/engine_worker_service.py]
envFrom:
- secretRef:
name: ousast-engine-s3
env:
- name: ENGINE_IMAGE
value: engine-image
resources:
requests:
memory: 3Gi
cpu: '1'
limits:
memory: 3Gi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
ports:
- name: health
containerPort: 8080
readinessProbe:
httpGet:
path: /healthz
port: health
periodSeconds: 5
livenessProbe:
httpGet:
path: /healthz
port: health
initialDelaySeconds: 15
periodSeconds: 10
+22
View File
@@ -0,0 +1,22 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ousast-engine
resources:
- namespace.yaml
- deployment.yaml
# Deliberately non-pullable until the coordinator supplies the workflow's digest.
images:
- name: engine-image
newName: ghcr.io/OWNER/openultrasast
digest: sha256:REPLACE_WITH_ENGINE_IMAGE_DIGEST
replacements:
- source:
kind: Deployment
name: ousast-engine-worker
fieldPath: spec.template.spec.containers.0.image
targets:
- select:
kind: Deployment
name: ousast-engine-worker
fieldPaths:
- spec.template.spec.containers.0.env.0.value
+4
View File
@@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: ousast-engine
+9
View File
@@ -0,0 +1,9 @@
# Reference only; excluded from kustomization. Create using the README command.
apiVersion: v1
kind: Secret
metadata:
name: ousast-engine-s3
namespace: ousast-engine
type: Opaque
stringData: {}
# Required keys: S3_ENDPOINT, S3_BUCKET, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
+26
View File
@@ -0,0 +1,26 @@
# The k3s profile: the plane on the maintainer's production cluster (plane-on-kubernetes design section 1;
# requirements "Target Environment"). Documented values, not secrets: the kubeconfig for the context, the GHCR
# pull secret's content and the store's keys live outside this file. Every field is overridable by OUSAST_<FIELD>,
# a pool by OUSAST_POOL_<NAME>_<FIELD>. Nothing has been applied to a k3s cluster yet; docs/deployment.md says
# which items are planned.
exec = "remote" # the same CLI from a laptop or a CI job, through the cluster's kube context
kube_context = "ousast-k3s" # the maintainer's context name for the k3s cluster
registry = "ghcr.io/norandom" # images built by GitHub Actions from tagged commits (design section 8)
images = "images.json" # the release asset {"runner": "ghcr.io/norandom/ousast-runner@sha256:...", "engine": ...}
router_url = "" # empty: port-forward to atenet-router through the kube context (ax-tunnel)
ax_server = "" # empty: the ax CLI's own tunnel through the kube context (ax-tunnel)
memory = "s3://sast-memory" # the memory store; S3_ENDPOINT and the keys come from the CI secrets or .env
atespace = "default"
snapshots_bucket = "s3://ax-snapshots" # ax's AX_SNAPSHOTS_BUCKET, ours (rendered into the ax-server patch)
image_pull_secret = "ghcr-pull" # only needed while the GHCR packages are private
[pools.default] # production sizing: four verify-class workers; the HPA (design section 5) scales it
replicas = 4
cpu = "1"
memory = "1536Mi"
[pools.engine] # two engine workers sized from the measured 2.1 GiB peak under a 3 GiB limit
replicas = 2
cpu = "2"
memory = "4Gi"
+3
View File
@@ -0,0 +1,3 @@
{
"runner": "localhost:5001/ousast-runner@sha256:784bc6afc869c679c4966d04d3f9653a8d379726b899809590cc83327f4038e3"
}
+26
View File
@@ -0,0 +1,26 @@
# The kind profile: the plane on this server VM's single-node kind cluster, which ops/ax/up.sh creates and which is
# the development default (plane-on-kubernetes design section 1). Values only; a credential never goes here
# (keys stay in the environment or .env). Every field is overridable by OUSAST_<FIELD> in the environment, a pool
# by OUSAST_POOL_<NAME>_<FIELD>. This file may name the kind cluster; the code must not (tests/test_plane_literals.py).
kind_observability = false # opt in with OUSAST_KIND_OBSERVABILITY=1
exec = "local" # the reconciler runs on this host against the cluster below
kube_context = "kind-ousast" # kind's context for KIND_CLUSTER_NAME=ousast
registry = "localhost:5001" # the kind-local registry up.sh creates (Substrate rewrites it for the node)
images = "kind-images.json" # digest-pinned image references, written by up.sh after each build
router_url = "" # empty: port-forward to atenet-router through the kube context (ax-tunnel)
ax_server = "" # empty: the ax CLI's own tunnel through the kube context (ax-tunnel)
memory = "s3://sast-memory" # the memory store; S3_ENDPOINT and the keys come from .env
atespace = "default"
snapshots_bucket = "s3://ax-snapshots" # ax's AX_SNAPSHOTS_BUCKET, ours (rendered into the ax-server patch)
image_pull_secret = "" # the kind registry needs none
[pools.default] # ousast-pool: two gVisor workers fit the 7 GB host beside the control planes
replicas = 2
cpu = "1"
memory = "1536Mi"
[pools.engine] # engine-pool (design section 4): one worker for the Joern engine task
replicas = 1
cpu = "2"
memory = "4Gi"
+61
View File
@@ -0,0 +1,61 @@
#!/bin/sh
# OpenUltraSAST in Docker, callable from a Git hook. A script, not a shell function: Git runs
# hooks non-interactively and never sees functions from ~/.bashrc.
#
# OUSAST_COMMAND=/path/to/ousast-docker git push # or: ousast pre-push install --docker
#
# The repository is mounted READ-ONLY at its own absolute path and the artifact directory
# read-write at its own path, so every path in the arguments and in the output ("Details: ...")
# means the same thing inside and outside the container. The push input on stdin is passed
# through (-i). The container runs as your user, with no network and --memory 3g.
#
# Advisory like the hook: when Docker or the image is missing, it says the check did not run
# and exits 0. Pull the image once first; a hook never downloads ~3 GB on its own:
#
# docker pull ghcr.io/norandom/openultrasast:2.0.1
#
# --engine background does not work through this wrapper: the container ends with the hook.
image=${OUSAST_IMAGE:-ghcr.io/norandom/openultrasast:2.0.1}
if ! command -v docker >/dev/null 2>&1; then
echo "ousast-docker: check did not run: docker is not installed. Push continues."
exit 0
fi
if ! docker image inspect "$image" >/dev/null 2>&1; then
echo "ousast-docker: check did not run: image $image is not present." \
"Run 'docker pull $image' once. Push continues."
exit 0
fi
repo=$(pwd -P)
artifact_dir=${OUSAST_ARTIFACT_DIR:-${XDG_CACHE_HOME:-$HOME/.cache}/openultrasast/push}
previous=
for argument in "$@"; do
if [ "$previous" = "--artifact" ]; then
artifact_dir=$(dirname -- "$argument")
fi
previous=$argument
done
mkdir -p -- "$artifact_dir" || exit 0
artifact_dir=$(cd -- "$artifact_dir" && pwd -P)
# A linked worktree keeps its objects in the main repository's git directory: mount that too.
common=$(git rev-parse --path-format=absolute --git-common-dir 2>/dev/null || echo "$repo/.git")
case "$common" in
"$repo"/*) extra_mount= ;;
*) extra_mount="-v $common:$common:ro" ;;
esac
# shellcheck disable=SC2086 # OUSAST_DOCKER_ARGS and extra_mount are deliberately word-split
docker run --rm -i --network none --memory "${OUSAST_DOCKER_MEMORY:-3g}" \
--user "$(id -u):$(id -g)" -e HOME=/tmp -e TMPDIR=/work --tmpfs /work:size=4g \
-v "$repo:$repo:ro" -w "$repo" -v "$artifact_dir:$artifact_dir" $extra_mount \
${OUSAST_DOCKER_ARGS:-} "$image" "$@"
status=$?
if [ "$status" -ge 125 ]; then
echo "ousast-docker: check did not run: the container could not start (docker exit $status). Push continues."
exit 0
fi
# Only an explicit blocking opt-in may turn the container's exit into a rejected push; an older
# image that rejects a newer argument must not block either.
if [ "$status" -ne 0 ] && [ "${OUSAST_PUSH_MODE:-advisory}" != blocking ]; then
echo "ousast-docker: the check exited with status $status; push continues (advisory)."
exit 0
fi
exit "$status"
+32 -11
View File
@@ -1,17 +1,38 @@
#!/bin/sh #!/bin/sh
# Explicitly copy using the non-overwriting recipe in ops/README.md. # OpenUltraSAST pre-push safety net. Advisory: nothing here rejects a push unless
# A prior hook is chained only after you explicitly move it to this sibling path. # OUSAST_PUSH_MODE=blocking is set explicitly. Install with `ousast pre-push install`
# (or ops/install-pre-push from a checkout); remove with `ousast pre-push uninstall`.
# A prior hook is chained only after it was moved to the sibling path "$0.before-ousast".
artifact_dir=${OUSAST_ARTIFACT_DIR:-${XDG_CACHE_HOME:-$HOME/.cache}/openultrasast/push} artifact_dir=${OUSAST_ARTIFACT_DIR:-${XDG_CACHE_HOME:-$HOME/.cache}/openultrasast/push}
prior="$0.before-ousast" prior="$0.before-ousast"
if [ -e "$prior" ] || [ -L "$prior" ]; then # The Docker wrapper installed beside this hook (`ousast pre-push install --docker`) is used
exec ousast pre-push . --artifact "$artifact_dir/push-$$.json" \ # unless OUSAST_COMMAND names another command; otherwise `ousast` from PATH.
--deadline "${OUSAST_PUSH_DEADLINE:-30}" \ if [ -z "${OUSAST_COMMAND:-}" ] && [ -x "$(dirname -- "$0")/ousast-docker" ]; then
--mode "${OUSAST_PUSH_MODE:-advisory}" \ OUSAST_COMMAND="$(dirname -- "$0")/ousast-docker"
--incomplete-coverage "${OUSAST_INCOMPLETE_COVERAGE:-allow}" \
--prior-hook "$prior" --remote "$1" "$2"
fi fi
exec ousast pre-push . --artifact "$artifact_dir/push-$$.json" \ command=${OUSAST_COMMAND:-ousast}
if ! command -v "$command" >/dev/null 2>&1; then
echo "ousast: pre-push check did not run: '$command' is not on PATH." \
"Install it (pip install git+https://github.com/norandom/OpenUltraSAST) or set OUSAST_COMMAND. Push continues."
if [ -e "$prior" ] || [ -L "$prior" ]; then
exec "$prior" "$@"
fi
exit 0
fi
remote_name=$1
remote_url=$2
set -- pre-push . --artifact "$artifact_dir/push-$$.json" \
--deadline "${OUSAST_PUSH_DEADLINE:-30}" \ --deadline "${OUSAST_PUSH_DEADLINE:-30}" \
--mode "${OUSAST_PUSH_MODE:-advisory}" \ --mode "${OUSAST_PUSH_MODE:-advisory}" \
--incomplete-coverage "${OUSAST_INCOMPLETE_COVERAGE:-allow}" \ --incomplete-coverage "${OUSAST_INCOMPLETE_COVERAGE:-allow}"
--remote "$1" "$2" # Optional settings are passed only when set, so an older ousast (or image) still runs.
if [ -n "${OUSAST_PUSH_ENGINE:-}" ]; then
set -- "$@" --engine "$OUSAST_PUSH_ENGINE"
fi
if [ -n "${OUSAST_COMPARISON_BASE:-}" ]; then
set -- "$@" --comparison-base "$OUSAST_COMPARISON_BASE"
fi
if [ -e "$prior" ] || [ -L "$prior" ]; then
set -- "$@" --prior-hook "$prior"
fi
exec "$command" "$@" --remote "$remote_name" "$remote_url"
+52
View File
@@ -0,0 +1,52 @@
# Build from the repository root. AX v0.3.1 starts spec.command immediately;
# preserve its runner as PID 1 and put the engine command in the Task manifest.
FROM eclipse-temurin:21-jre-jammy AS jre
FROM ghcr.io/norandom/ax-task-runner:v0.3.1
LABEL org.opencontainers.image.licenses="Apache-2.0" \
org.opencontainers.image.source="https://github.com/norandom/OpenUltraSAST"
USER root
ARG JOERN_VERSION=v4.0.625
ARG ARCHIVE=joern-cli-linux-x86_64.zip
# The AX runner is Python 3.12 / Debian slim. Copy the Java 21 runtime from
# Jammy (older glibc than Debian bookworm) instead of relying on distro JRE versions.
COPY --from=jre /opt/java/openjdk /opt/java/openjdk
ENV JAVA_HOME=/opt/java/openjdk
ENV PATH="/venv/bin:/opt/java/openjdk/bin:/opt/joern-cli:${PATH}"
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
php-cli unzip curl ca-certificates libstdc++6 zlib1g \
&& rm -rf /var/lib/apt/lists/*
RUN set -eux; \
cd /tmp; \
curl -fsSL -O "https://github.com/joernio/joern/releases/download/${JOERN_VERSION}/${ARCHIVE}"; \
curl -fsSL -O "https://github.com/joernio/joern/releases/download/${JOERN_VERSION}/${ARCHIVE}.sha512"; \
sed 's@ target/@ @' "${ARCHIVE}.sha512" > checksum.sha512; \
sha512sum -c checksum.sha512; \
unzip -q "${ARCHIVE}" -d /opt; \
rm -f "${ARCHIVE}" "${ARCHIVE}.sha512" checksum.sha512; \
/opt/joern-cli/joern-parse --help > /dev/null
RUN if [ ! -e /opt/joern-cli/jssrc2cpg ] && [ -x /opt/joern-cli/jssrc2cpg.sh ]; then \
ln -s jssrc2cpg.sh /opt/joern-cli/jssrc2cpg; \
fi
COPY ops/frontend-retention/install.py /tmp/frontend-retention-install.py
RUN python3 /tmp/frontend-retention-install.py /opt/joern-cli \
&& rm /tmp/frontend-retention-install.py
WORKDIR /app
COPY pyproject.toml README.md LICENSE ./
COPY src ./src
COPY benchmarks/learn/engine_task_entry.py \
benchmarks/learn/engine_trace_worker.py \
benchmarks/learn/engine_trace_parse.py ./benchmarks/learn/
# Worker imports are standard-library plus engine modules. Isolate them from
# the AX runner's Python environment and omit PyYAML, boto3 and all extras.
RUN python3 -m venv /venv \
&& /venv/bin/pip install --no-cache-dir --no-deps . \
&& /venv/bin/python -I -c 'import pathlib, runpy, sys; sys.modules.update(yaml=None, boto3=None); sys.path.insert(0, "/app/benchmarks/learn"); p = pathlib.Path("/app/benchmarks/learn/engine_trace_worker.py"); print("Worker import smoke bytes:", len(p.read_bytes())); runpy.run_path(str(p), run_name="engine_worker_import_smoke")'
# Do not set ENTRYPOINT or CMD: both belong to the upstream AX runner.
@@ -0,0 +1,353 @@
{"family": "deserialization", "fold": "outer-1", "group": "aio-libs/aiohttp", "label": 1, "pair": "ee577ee4b6076fab", "unit": "2523f55d035d1a772a842090bc335e5d755b2fe15b376dd5b6b9e30b9dae82d1"}
{"family": "deserialization", "fold": "outer-1", "group": "aio-libs/aiohttp", "label": 0, "pair": "ee577ee4b6076fab", "unit": "ca0f6c160d7839204f6fcc1709bcbc749a02a0b3b8bb648e2a238eacbf22ed78"}
{"family": "access_control", "fold": "outer-0", "group": "alchemycms/alchemy_cms", "label": 1, "pair": "c95b6b01d0c2db2a", "unit": "15540c23be0c42472a49039658ec2ec0bbd8c6b318e35dd040a9d427370e0ef0"}
{"family": "access_control", "fold": "outer-0", "group": "alchemycms/alchemy_cms", "label": 0, "pair": "c95b6b01d0c2db2a", "unit": "542471980417e964182d7d3db59a05fe12bdd072449f3bd77f7daa5515dbc909"}
{"family": "access_control", "fold": "outer-1", "group": "anotherik/threatbyte", "label": 1, "pair": "", "unit": "bcc636b23dfb5f570d8864b17fe07b6b4fb52b65a40988c32011f1422f74d504"}
{"family": "access_control", "fold": "outer-1", "group": "anotherik/threatbyte", "label": 1, "pair": "", "unit": "dbcf1e67a1be7ccf7d51701f9b8339339164055135443044c23a446b16b3d934"}
{"family": "injection", "fold": "outer-1", "group": "anotherik/threatbyte", "label": 1, "pair": "", "unit": "50408f68114ecb4de3bebd836d52d9afad86b22dd1843641ac8a49b617c09e33"}
{"family": "injection", "fold": "outer-0", "group": "anxolerd/dvpwa", "label": 1, "pair": "", "unit": "cafb0f00f80dc6beeba40eb95c764bf9a3569193070ba321c28677dfa425ec88"}
{"family": "output_encoding", "fold": "outer-3", "group": "apache/logging-log4net", "label": 1, "pair": "fe0e104d111c2555", "unit": "088036fab9962be2b96f9162ea4f32190575678d3c5ba96bdfeac178bc22b2c4"}
{"family": "output_encoding", "fold": "outer-3", "group": "apache/logging-log4net", "label": 0, "pair": "fe0e104d111c2555", "unit": "7fd2b56a816e25358477eee64487fb2c77f34c652703ad0a7120c368492017a3"}
{"family": "path", "fold": "outer-0", "group": "apache/wicket", "label": 1, "pair": "b6742570f7f0e026", "unit": "7c6c1d27aee6557f1b6bf371b8209003f0d2d1122cfa35b6629eb43f38769cf4"}
{"family": "path", "fold": "outer-0", "group": "apache/wicket", "label": 0, "pair": "b6742570f7f0e026", "unit": "df8300d84dd784d0be46969ea432ef97b8ce611bae7a896ed4f349bd4d6897db"}
{"family": "access_control", "fold": "outer-0", "group": "apolloconfig/apollo", "label": 1, "pair": "0f584bb97a154488", "unit": "222a84ffc7c58d45e8f10df221762b23bf20d0287ecd20f375558134928886ca"}
{"family": "access_control", "fold": "outer-0", "group": "apolloconfig/apollo", "label": 0, "pair": "0f584bb97a154488", "unit": "25e4c9d2941f59f072385a7bde3fd0bd76f9e986ed24b06ce81962ab447f5094"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "archivy/archivy", "label": 0, "pair": "bb3d7baed62e6bb7", "unit": "e8ae88fd2b677e3f64ac94bddb012e3b903afa294ecc87774d10087c0d50489c"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "archivy/archivy", "label": 1, "pair": "bb3d7baed62e6bb7", "unit": "f6b0d241891dc1d897d74d98b6ff7ada2c1f1ce7874ca32097630c06ca53e389"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "axllent/mailpit", "label": 1, "pair": "143b900b99a46cf4", "unit": "200f96cd784ff5c39d8bba22883f22bc8a7c87c6970a7c871802f48a45548337"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "axllent/mailpit", "label": 0, "pair": "143b900b99a46cf4", "unit": "7a0d62ec424849ec4681f4333fbb077e8239054d1e964bd99b40847ad7ccdb01"}
{"family": "injection", "fold": "outer-2", "group": "browserify/static-eval", "label": 1, "pair": "c12dbc8a50f7fe9b", "unit": "04beb8a0fd054b92c2dd0ed83907330fdee6124d86114df948ac72a297a7d51f"}
{"family": "injection", "fold": "outer-2", "group": "browserify/static-eval", "label": 0, "pair": "c12dbc8a50f7fe9b", "unit": "c5029016a6fd2e3f9df59dba0ce3c173bbbb0d2b1de22d151086e01e1b272119"}
{"family": "injection", "fold": "outer-3", "group": "budibase/budibase", "label": 0, "pair": "", "unit": "0e213aeabacf4205db024ea3c557e720c3aba911424555784917e967d6404867"}
{"family": "injection", "fold": "outer-3", "group": "budibase/budibase", "label": 0, "pair": "e8bc65415bd42e43", "unit": "4920230ef773595a9fd9a92a090dff42b48499a1dc5fc8831f941dae55cb2538"}
{"family": "injection", "fold": "outer-3", "group": "budibase/budibase", "label": 1, "pair": "e8bc65415bd42e43", "unit": "49209a810f207eb61382e2ceeab7076936838a5652546f21c187d066e220160b"}
{"family": "injection", "fold": "outer-3", "group": "budibase/budibase", "label": 0, "pair": "", "unit": "9e2964884cdf281fa3f18ee3481aa62f9a8a0a195d605da2df5bd71876fe311a"}
{"family": "injection", "fold": "outer-3", "group": "budibase/budibase", "label": 1, "pair": "", "unit": "f9cc6f1bbcd59f5dc0de8f96248f722ea0679f8afa47c53f70832151ce7616d3"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "cakephp/authentication", "label": 0, "pair": "e69463f945f9a0f4", "unit": "5feace7b28e260175f80c682e9009025aea6fae3dc75409cabbb955b560d0372"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "cakephp/authentication", "label": 1, "pair": "e69463f945f9a0f4", "unit": "799a4f5a5b9324e86c90752f0239f059984f44e8ff1ec0dd518d0d9413e34c66"}
{"family": "deserialization", "fold": "outer-2", "group": "cesargb/laravel-magiclink", "label": 0, "pair": "768fc9db8a75a351", "unit": "adc6527469618f72b1dd7776814717cbd0056a0c625e05bef74199171ce73f1a"}
{"family": "deserialization", "fold": "outer-2", "group": "cesargb/laravel-magiclink", "label": 1, "pair": "768fc9db8a75a351", "unit": "da47663a57e370a75cc65b810b8a380aa26d12dcabb1d9b3c02641398ff7d348"}
{"family": "path", "fold": "outer-0", "group": "christopy/serve-here.js", "label": 0, "pair": "fab4ce020049724b", "unit": "b9be90242be627aa46462143ac85940a906232ebc0b47776b5729475b27d215e"}
{"family": "path", "fold": "outer-0", "group": "christopy/serve-here.js", "label": 1, "pair": "fab4ce020049724b", "unit": "ece12ea7d87b4a12c4da30faed9eb530b51fc879c271b68ba5304b230cda8e79"}
{"family": "output_encoding", "fold": "outer-1", "group": "chrome-php/chrome", "label": 1, "pair": "9126ba83905eb8b4", "unit": "d8dc50b7bb40bbcfc3f3d4f1f8d3a4d17eb1c662d5f759efd18656a482dd2e32"}
{"family": "output_encoding", "fold": "outer-1", "group": "chrome-php/chrome", "label": 0, "pair": "9126ba83905eb8b4", "unit": "f426f9307621b3f2d6a8d1669cdb4c19e962b94e34fbb394d4660cbde5921c88"}
{"family": "deserialization", "fold": "outer-3", "group": "commenthol/serialize-to-js", "label": 0, "pair": "7ac5e21d9d8f4c47", "unit": "1d1efa2fd7323c264290d4d314c957c853e1c7a9c877a54d6f03ab7d2af2ca86"}
{"family": "deserialization", "fold": "outer-3", "group": "commenthol/serialize-to-js", "label": 1, "pair": "7ac5e21d9d8f4c47", "unit": "b5dc78c7ac0b76803dc98904142c1139083fb9daafb5bff40e7b5c3f14ee03f5"}
{"family": "deserialization", "fold": "outer-0", "group": "contrast-security-oss/djangoat", "label": 1, "pair": "", "unit": "0894a7212280bab755cd9d93b1f0ede2c51e4fca76b76e5ed9e24ef8f8cf0b9c"}
{"family": "injection", "fold": "outer-0", "group": "contrast-security-oss/djangoat", "label": 1, "pair": "", "unit": "02694b5fb85e9e52bf5eaaab4e0b628f1f44ff3a8d5ad026cc10afdbc4222eaf"}
{"family": "injection", "fold": "outer-0", "group": "contrast-security-oss/djangoat", "label": 1, "pair": "", "unit": "db9a3e888780501dd3c2c125af2f9d47f68ebf02f0e0db8370b2ea09c6f9d070"}
{"family": "injection", "fold": "outer-1", "group": "contrast-security-oss/vulnpy", "label": 1, "pair": "", "unit": "9af96b099dedcc4005e5fcd19ad78f73864e1cfbafbe6953b4edb6260957943d"}
{"family": "injection", "fold": "outer-1", "group": "contrast-security-oss/vulnpy", "label": 1, "pair": "", "unit": "9e7c2a786068a936a6c664cc229d6f39cea1fe57caf0806306f0cfc44515460c"}
{"family": "deserialization", "fold": "outer-4", "group": "cps-it/mailqueue", "label": 1, "pair": "07d1dc4d299a7700", "unit": "0e664e274dfb438d2f721edc81f8469ac48387dbd5c1b43973cd7e651db46db3"}
{"family": "deserialization", "fold": "outer-4", "group": "cps-it/mailqueue", "label": 0, "pair": "07d1dc4d299a7700", "unit": "da55f60378c67e91683b9720f2fa0e6659e28c7b392c050278b40713e47c84a3"}
{"family": "deserialization", "fold": "outer-0", "group": "datapizza-labs/datapizza-ai", "label": 1, "pair": "5a91049be21aa611", "unit": "1ab9936627aa59c3a60ea44074ad44b68aea4f9c04f14c1949dd9c77ca13ff0a"}
{"family": "deserialization", "fold": "outer-0", "group": "datapizza-labs/datapizza-ai", "label": 0, "pair": "5a91049be21aa611", "unit": "8df72b9eab3f2fd858c40312d0a806cb67c620bd62873826c8732ba8795d9cfa"}
{"family": "injection", "fold": "outer-0", "group": "davila7/claude-code-templates", "label": 0, "pair": "", "unit": "115d87f37bf1d4ab91fa3010303e81b127f84b817268b60b8fd4b9934efeae4f"}
{"family": "injection", "fold": "outer-0", "group": "davila7/claude-code-templates", "label": 0, "pair": "2b8101c89c839614", "unit": "7bb7d858a670fc51f81e08c141c2bca3b8b7524254ab679e33b22ce79536e895"}
{"family": "injection", "fold": "outer-0", "group": "davila7/claude-code-templates", "label": 0, "pair": "c23acf4f38945093", "unit": "82bdc0b6bb0c19e342efa63af0c9014a607461d0682cdd14bd5fea33516fbd50"}
{"family": "injection", "fold": "outer-0", "group": "davila7/claude-code-templates", "label": 1, "pair": "2b8101c89c839614", "unit": "886285661b88f14e00ff8e863b31cf7b7afb303ad4aa4aa13b1cffbf1272d5e0"}
{"family": "injection", "fold": "outer-0", "group": "davila7/claude-code-templates", "label": 1, "pair": "c23acf4f38945093", "unit": "bf74aae5a48156f183426a80cafca6da219ab13b2ae4b40fa88fca75acb72980"}
{"family": "access_control", "fold": "outer-2", "group": "delmaredigital/payload-puck", "label": 0, "pair": "d7cc37a3a54315cb", "unit": "4aecd43c6e5b81daf7aadd6a6e22f22c17c08b652a2f5121a49320ca1057b186"}
{"family": "access_control", "fold": "outer-2", "group": "delmaredigital/payload-puck", "label": 1, "pair": "d7cc37a3a54315cb", "unit": "88123ae860be2fdc28fd6a289cf26c3e0af0689f455cf8c15098e5dc76e5c8a8"}
{"family": "output_encoding", "fold": "outer-4", "group": "devbridge/jquery-autocomplete", "label": 0, "pair": "81caad96b221c2b8", "unit": "3c2dc720ed159e55629ba737665d93859def9790427289567a513c83c133e236"}
{"family": "output_encoding", "fold": "outer-4", "group": "devbridge/jquery-autocomplete", "label": 1, "pair": "81caad96b221c2b8", "unit": "e8b9595a320776c8e663fd0b168193c1e64e36a9052784d1219f2c4a91d647f2"}
{"family": "path", "fold": "outer-3", "group": "dgtlmoon/changedetection.io", "label": 0, "pair": "f9b6599018ef3b60", "unit": "7558d6ee39a311c5459e35fe6f7b12917b7f843b21ab1fbb5e485a07f53b7d78"}
{"family": "path", "fold": "outer-3", "group": "dgtlmoon/changedetection.io", "label": 1, "pair": "f9b6599018ef3b60", "unit": "b1a6cf450a074788ffca2bebd880435abac2c376e89cb328d7a22d277fdccad8"}
{"family": "output_encoding", "fold": "outer-0", "group": "dicebear/dicebear", "label": 1, "pair": "eaf2f03db2c502b5", "unit": "38dbfbf745d7cbcc52c6fff83248d84663273fe6a1f1b014eac7776c4bb8f074"}
{"family": "output_encoding", "fold": "outer-0", "group": "dicebear/dicebear", "label": 0, "pair": "eaf2f03db2c502b5", "unit": "a84d848ef56b72fc15872d88acaae1743c9cdf88095d13b1869abbec6a0d8a8f"}
{"family": "deserialization", "fold": "outer-1", "group": "docling-project/docling-core", "label": 0, "pair": "7df48bca2c9fabfc", "unit": "7b76ce8f55d61071adb367018cc3e50f04a05d08c7c4919c3806fc1cc8bf41a7"}
{"family": "deserialization", "fold": "outer-1", "group": "docling-project/docling-core", "label": 1, "pair": "7df48bca2c9fabfc", "unit": "c82ea0adc9c8366aa2b77ce73772038676fc37ff50879e873ff397958fdd7ec1"}
{"family": "injection", "fold": "outer-4", "group": "dolevf/damn-vulnerable-graphql-application", "label": 1, "pair": "", "unit": "7f02a09b8d8f2321f2cdade74c2fa1d52442e006149a749a8be768daaa62195d"}
{"family": "injection", "fold": "outer-4", "group": "dolevf/damn-vulnerable-graphql-application", "label": 1, "pair": "", "unit": "9b5e439943ce5c2b04d6201b7a843634aa1b13b726813c23450ff54ce03f3dd6"}
{"family": "injection", "fold": "outer-4", "group": "dolevf/damn-vulnerable-graphql-application", "label": 1, "pair": "5b836889b4da4921", "unit": "a84c2ce4a4eadf265d4481ad7f6864edf1d7e3084c6ee32cbbfe454bb2946bfe"}
{"family": "injection", "fold": "outer-4", "group": "dolevf/damn-vulnerable-graphql-application", "label": 0, "pair": "5b836889b4da4921", "unit": "caeed7f097a3311fe2e84efd372fe4b4c140f63a80a9f07ba0b61966dbab15e0"}
{"family": "access_control", "fold": "outer-2", "group": "erev0s/vampi", "label": 1, "pair": "", "unit": "c2a2c2d85114a97f892fb1e5500fbd98622457a8c46451e1f9e958ab08a40318"}
{"family": "injection", "fold": "outer-2", "group": "erev0s/vampi", "label": 1, "pair": "", "unit": "29d8be8206c88ba7efd86b56a7dcf8b655e33b4a792c2d18b028c92c6252daab"}
{"family": "output_encoding", "fold": "outer-2", "group": "etaminstudio/prosemirror_to_html", "label": 0, "pair": "f96f0fb245948915", "unit": "0954e6931373915c204b9ec8d2b8ae6297a95667bfd41aadf9035625ad93967f"}
{"family": "output_encoding", "fold": "outer-2", "group": "etaminstudio/prosemirror_to_html", "label": 1, "pair": "f96f0fb245948915", "unit": "5e09cdae2504e834d591b6f92a808427f07550364b5419899f7e6f76084fc377"}
{"family": "output_encoding", "fold": "outer-4", "group": "ether/etherpad", "label": 0, "pair": "bf11f138fe39675e", "unit": "56f0d0b156992af56141d6368fdb0c5376fc7a44a03991d754f97202b9d77008"}
{"family": "output_encoding", "fold": "outer-4", "group": "ether/etherpad", "label": 1, "pair": "bf11f138fe39675e", "unit": "8271af5017894167556893da2070188060d06eebc87f27db45c46234c359631d"}
{"family": "path", "fold": "outer-2", "group": "fastify/fastify-static", "label": 0, "pair": "9d937c2dffb4aadc", "unit": "01ac12bb06e660d0aeef8b1e26dddc5c99d52cc333f7af6d1981eee2818e64fd"}
{"family": "path", "fold": "outer-2", "group": "fastify/fastify-static", "label": 1, "pair": "9d937c2dffb4aadc", "unit": "9612edf0f927048335e83e9925f01be9aea9b772bdbdc568ad9cc0a17055662a"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "feathersjs/feathers", "label": 1, "pair": "d4f648048e45f2ea", "unit": "08807c8110b8c1b8635a6883c229fb91f2b9566db8b6c252448a0db7b2ddd47e"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "feathersjs/feathers", "label": 0, "pair": "d4f648048e45f2ea", "unit": "10869c931053bf77ecaa98a47f56dea27d643d1abed2a4758bcf48a78547ccc9"}
{"family": "injection", "fold": "outer-1", "group": "fecommunity/reactpress", "label": 0, "pair": "662e1f36c230d7f1", "unit": "0b73332b25413902c26d5b5dee6767465eed99ff71a814582988ffe0d8ec1a30"}
{"family": "injection", "fold": "outer-1", "group": "fecommunity/reactpress", "label": 0, "pair": "", "unit": "8c8e46d0c7d361b784baff9d70f38744b2805ef1ee0aaacaa616bc20a2c6f32d"}
{"family": "injection", "fold": "outer-1", "group": "fecommunity/reactpress", "label": 0, "pair": "", "unit": "aca938682c196fd4c85b39b89eeedb97c35f6a5a017651fe9c03c5b727cab7ea"}
{"family": "injection", "fold": "outer-1", "group": "fecommunity/reactpress", "label": 0, "pair": "", "unit": "cec00d1509a129364da6ab8537fb3f433fd847e26d073c90311ccdaf7a1b39e7"}
{"family": "injection", "fold": "outer-1", "group": "fecommunity/reactpress", "label": 1, "pair": "662e1f36c230d7f1", "unit": "e0f58a7ac5627de3e7b6cfe5add41d18b0f64b998c1cc788d42eb2cdbbc3fe41"}
{"family": "access_control", "fold": "outer-0", "group": "filebrowser/filebrowser", "label": 0, "pair": "2c251a645094f389", "unit": "0d41cc634771048f72d07c4aa9feaed669d0bfe3844d61dcdc525a0f899617e2"}
{"family": "access_control", "fold": "outer-0", "group": "filebrowser/filebrowser", "label": 1, "pair": "2c251a645094f389", "unit": "ca8288d0b8b94dc6b7e4c681936ea610d295fe949cba5f64f09b6de134ef9875"}
{"family": "access_control", "fold": "outer-2", "group": "forgekeep/nebula-mesh", "label": 1, "pair": "2ac5ec5011615c9b", "unit": "2f427f403af3cb58d776437736c2389805543413fdb91aad98979158c6d00de3"}
{"family": "access_control", "fold": "outer-2", "group": "forgekeep/nebula-mesh", "label": 0, "pair": "2ac5ec5011615c9b", "unit": "cddc8c5bed0841b80e7b20d0b378e858f8e4ac37fd7b890ce76611d989857f92"}
{"family": "injection", "fold": "outer-2", "group": "frangoteam/fuxa", "label": 1, "pair": "", "unit": "0c7830c2912ff285855d8a3c8e9b47db4ca4af840baa66d4b5d3f4422f46f95d"}
{"family": "injection", "fold": "outer-4", "group": "froxlor/froxlor", "label": 0, "pair": "", "unit": "5383a8c2a78fabd3d17a5b3021125953b5ebbee61f8900c30d016c56b1a3bad6"}
{"family": "injection", "fold": "outer-4", "group": "froxlor/froxlor", "label": 0, "pair": "360d8bfd2f116e98", "unit": "7c1989a4ec3622bdaebcf1161de850f22cfed041cb38830ca1ded4fa99d40466"}
{"family": "injection", "fold": "outer-4", "group": "froxlor/froxlor", "label": 1, "pair": "ec8c706cc2b20f36", "unit": "848688dcd0fe8d46e6762cc8aee1d82e8fad8a2a044c94b547aea11049c85bfc"}
{"family": "injection", "fold": "outer-4", "group": "froxlor/froxlor", "label": 0, "pair": "ec8c706cc2b20f36", "unit": "983cc9231f91baee4e2d626193f3b88aa5b09907f6faab870949bdc9c336b816"}
{"family": "injection", "fold": "outer-4", "group": "froxlor/froxlor", "label": 0, "pair": "e4ad86416d5c6668", "unit": "98ff1dff2afe80709b8e61a9c95cfeea12b2110c13b7120478268b4762e2c992"}
{"family": "injection", "fold": "outer-4", "group": "froxlor/froxlor", "label": 1, "pair": "360d8bfd2f116e98", "unit": "a1a825d23e34757c8e961e43ae791253e62682e130acdb846b58118588006a98"}
{"family": "injection", "fold": "outer-4", "group": "froxlor/froxlor", "label": 0, "pair": "", "unit": "a7fa4207508b3989baabc7a1f0ff2ea5229b366c1cb106e7fc945134b8b106c8"}
{"family": "injection", "fold": "outer-4", "group": "froxlor/froxlor", "label": 1, "pair": "e4ad86416d5c6668", "unit": "ef5c53f3114d0057e694df13dc07cbcb532c3b0d59150eda269b87b021d7a6e8"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "getgrav/grav", "label": 1, "pair": "9ae575048504c690", "unit": "176acf077fd58c3e6f940753179ffa3e8316661f355d94acb5e0345b51925ca4"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "getgrav/grav", "label": 0, "pair": "9ae575048504c690", "unit": "228b34ebc2090b5239c7821d28366ddc663e7118d015a79b45dca6618d5490fb"}
{"family": "output_encoding", "fold": "outer-0", "group": "gofiber/template", "label": 0, "pair": "a3fdf9ce3441ec86", "unit": "653236018929ab8534c27704a20ec5d358047bc66e451620d2ac32081c91d333"}
{"family": "output_encoding", "fold": "outer-0", "group": "gofiber/template", "label": 1, "pair": "a3fdf9ce3441ec86", "unit": "b70ca2c43f5929e7b885bd728349e721fd85d7605e2f722a7e8cfe0f0d477b0c"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "groundhoggwp/groundhogg", "label": 1, "pair": "", "unit": "92400e83890882a66ffdea1a405e2354a195442e176d85fb707b0442c44d45ca"}
{"family": "path", "fold": "outer-1", "group": "gtsteffaniak/filebrowser", "label": 0, "pair": "f9dcf3a0fb0f0a14", "unit": "7ca6dd4f673976b000a657f885af3463740419ce41e14f972af0c5483cb75a7d"}
{"family": "path", "fold": "outer-1", "group": "gtsteffaniak/filebrowser", "label": 1, "pair": "f9dcf3a0fb0f0a14", "unit": "86c1db3a5e9f843d6546146449171e0378ac61d6c8f44c6dd72fd379cdbb188a"}
{"family": "injection", "fold": "outer-4", "group": "handsontable/formula-parser", "label": 0, "pair": "6144baeb491f4a56", "unit": "f5c22ce104e9593e5272b6e8b3c6f3b61fa4578552ccea80e6021ee90bdb3ffc"}
{"family": "injection", "fold": "outer-4", "group": "handsontable/formula-parser", "label": 1, "pair": "6144baeb491f4a56", "unit": "f687e7e95dbe47f602a1e0002a44f94fd48e5b1c33f77ecebc6c2d814fcbf450"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "heartcombo/devise", "label": 0, "pair": "2ef4471c66e5e4ab", "unit": "ad7c8e368fe1699a044e058d5c10b41bd9fc0f91c3f3c384c5dac40af39e71f6"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "heartcombo/devise", "label": 1, "pair": "2ef4471c66e5e4ab", "unit": "d888baf7194bbe7158cb3932f58b47a9a235351d388d10df82a130ddd54c2d46"}
{"family": "output_encoding", "fold": "outer-1", "group": "honojs/hono", "label": 1, "pair": "5628c54152216646", "unit": "47d081c9f6060218967902f694e61c88f64f57c65bec6fc3073afeb0b77d117c"}
{"family": "output_encoding", "fold": "outer-1", "group": "honojs/hono", "label": 0, "pair": "5628c54152216646", "unit": "e1297bccf9c3606e0b41d762a69ff5d0a33ae4f7f1506071af6b9cb4a177ff56"}
{"family": "path", "fold": "outer-2", "group": "hs-web/hsweb-framework", "label": 1, "pair": "e52aa814c36ace1f", "unit": "2d6d74364f1982e91f27f1c995b8699df1fc8f8dce6e1e2cf274bf01b6e9494e"}
{"family": "path", "fold": "outer-2", "group": "hs-web/hsweb-framework", "label": 0, "pair": "e52aa814c36ace1f", "unit": "4727471fea2cf0a92d43b4b1503f1339a7dde87f872ba8064dacbf855c57705b"}
{"family": "injection", "fold": "outer-1", "group": "ibm/mcp-context-forge", "label": 0, "pair": "6548e1527e82bd99", "unit": "5542613f6d2eb9421b8213b8bb864894f0ecf9fe64945681019230ef0d75cfaa"}
{"family": "injection", "fold": "outer-1", "group": "ibm/mcp-context-forge", "label": 0, "pair": "", "unit": "6c2adee34e9e8384535ec1073eda60e2ab528ac77c36747802c303e1f5e9ffe8"}
{"family": "injection", "fold": "outer-1", "group": "ibm/mcp-context-forge", "label": 1, "pair": "", "unit": "7ce70e9b20107cc7ebb87d173da765fc53ffd268e521749a06a9a1f38d746845"}
{"family": "injection", "fold": "outer-1", "group": "ibm/mcp-context-forge", "label": 1, "pair": "6548e1527e82bd99", "unit": "c3e54975464323e58ec96ee4d83e7bc08baf94553d553a8dc3fb28cd9c70ff21"}
{"family": "injection", "fold": "outer-1", "group": "ibm/mcp-context-forge", "label": 1, "pair": "", "unit": "d3173c1e43eb6402a71ea2dec1592913dc078a2194ba1c1d509ab129a3180f6f"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "internlm/lmdeploy", "label": 1, "pair": "0924befc49c7bb18", "unit": "e7e4872db40f0420547ed622fbdd7a072b64a7ba466d1ccfff68e691cee6ad99"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "internlm/lmdeploy", "label": 0, "pair": "0924befc49c7bb18", "unit": "f7d95f43fb2df20ebc4c37fceabdd00ffd8159f404ca384481f1764252382388"}
{"family": "path", "fold": "outer-1", "group": "jarofghosts/glance", "label": 1, "pair": "772e18821701d60f", "unit": "89a0496c7663bbfd67d660c9f481c12420da2dd1bdb29dc079735ee7a0a49626"}
{"family": "path", "fold": "outer-1", "group": "jarofghosts/glance", "label": 0, "pair": "772e18821701d60f", "unit": "d3241ceaa50a7d23a4650e6a2a55372b08ec96b35b0bede844bc40e16e818ebc"}
{"family": "injection", "fold": "outer-0", "group": "jas-/node-libnmap", "label": 1, "pair": "250de1302c7e541d", "unit": "5d03535f2df2acdbc319d9e6b20da65d51f3e810b80f3c8f89b5074a971a3f1c"}
{"family": "injection", "fold": "outer-0", "group": "jas-/node-libnmap", "label": 0, "pair": "250de1302c7e541d", "unit": "78f977cfc096e34b4eada7f798ad2973ef45728d0d01bfab6ac9e2a3257827ef"}
{"family": "injection", "fold": "outer-0", "group": "jashkenas/underscore", "label": 1, "pair": "c10bfb006f2da55a", "unit": "1ea4b86dbd84581b1d7b7cabbab9f28c8b213416a60778879666ff42d23fd7f1"}
{"family": "injection", "fold": "outer-0", "group": "jashkenas/underscore", "label": 0, "pair": "c10bfb006f2da55a", "unit": "7647021f742da24c660b2d407886771786e957076c4cf336def560a570dd8778"}
{"family": "access_control", "fold": "outer-0", "group": "jellyfin/jellyfin", "label": 1, "pair": "a73421efb106ab0a", "unit": "1f6f207a758582de0c0e6b331b19fe119abe18727d45ccbbe09aabb795f3a292"}
{"family": "access_control", "fold": "outer-0", "group": "jellyfin/jellyfin", "label": 0, "pair": "a73421efb106ab0a", "unit": "32fbdad39eaf5177db4ad0a11d53e785b30bc7b2376ad9d460f29a08429bd0c7"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "jenkinsci/cas-plugin", "label": 0, "pair": "0458e3fd444409f9", "unit": "b521216bb4106738b3571ad3902d08f6230aabd88cf3a8f9f26f1c8ec01304cb"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "jenkinsci/cas-plugin", "label": 1, "pair": "0458e3fd444409f9", "unit": "c87d2871907798fa2e73d31381e9d63089b0388033f096d247373fec07c2cb36"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "jenkinsci/google-login-plugin", "label": 0, "pair": "6bdccabcf5b9fdc9", "unit": "3d6ddca5eb264d88411b86755909f1d2abddd09c129cabd54c96ff26b4105316"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "jenkinsci/google-login-plugin", "label": 1, "pair": "6bdccabcf5b9fdc9", "unit": "e055b898802279bc63204cc601e29f7f1018efad7228cab0cce00ea5866ecc7b"}
{"family": "access_control", "fold": "outer-2", "group": "jenkinsci/mcp-server-plugin", "label": 1, "pair": "91bc832e70680c92", "unit": "1060c63ff0fae3af567448f3d4c60d9c159b61ccc358964243eb152a9ac38865"}
{"family": "access_control", "fold": "outer-2", "group": "jenkinsci/mcp-server-plugin", "label": 0, "pair": "91bc832e70680c92", "unit": "369dffd1e730d7997813adfbebf1481cc68987cae0d0085e47b91a26d5330cf8"}
{"family": "path", "fold": "outer-4", "group": "jknack/handlebars.java", "label": 1, "pair": "a4144258dffb3b8d", "unit": "7ab301e1614979f44c353e665540582493f78ef4bfc41209562c19b900710f8c"}
{"family": "path", "fold": "outer-4", "group": "jknack/handlebars.java", "label": 0, "pair": "a4144258dffb3b8d", "unit": "ea95e8387dff32ded9dc7b04c11a76a545b4e2ea6db0076b9b13971a59904bb3"}
{"family": "deserialization", "fold": "outer-2", "group": "jmespath/jmespath.rb", "label": 1, "pair": "858330e01d3cf9bb", "unit": "033cda2b7cffa5a5fa6abf0b7f2cd27ba78d916eb3cab8c79decccd8776f4a3f"}
{"family": "deserialization", "fold": "outer-2", "group": "jmespath/jmespath.rb", "label": 0, "pair": "858330e01d3cf9bb", "unit": "d44c4555a5ae19aa17b468ec5cff56b33a0b221df52e04d8abcb656399499dee"}
{"family": "deserialization", "fold": "outer-2", "group": "jooby-project/jooby", "label": 1, "pair": "c6a49339d4f0c798", "unit": "dc2e4cd5bd963643702d2b832ae170dd25744162e4076e36687256a09e69d089"}
{"family": "deserialization", "fold": "outer-2", "group": "jooby-project/jooby", "label": 0, "pair": "c6a49339d4f0c798", "unit": "f83cd137b85e651ca5299c5a42176b797ed4964cbef31418c96d63c52fdb65f5"}
{"family": "access_control", "fold": "outer-1", "group": "jsonbored/gittensory", "label": 1, "pair": "5ec37c0f475001ca", "unit": "5d8175fb21a4649bdc8730358e9b87adb90b2aafc4da5fee203829663e4f46a0"}
{"family": "access_control", "fold": "outer-1", "group": "jsonbored/gittensory", "label": 0, "pair": "5ec37c0f475001ca", "unit": "af845e9e332555096dd350401977b2f07015433f3db8a040891d9470f3f9e355"}
{"family": "deserialization", "fold": "outer-0", "group": "kevin1024/vcrpy", "label": 1, "pair": "c74525e844650de2", "unit": "2e444ad34ee5f47d73f0cfad0a10c70363337b4ddf9067ecd1b8072d8232652c"}
{"family": "deserialization", "fold": "outer-0", "group": "kevin1024/vcrpy", "label": 0, "pair": "c74525e844650de2", "unit": "c492d325f5ff82b5f817aa4be1440cec8595054dd0719e07a58ac5501b2ae7eb"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "koajs/koa", "label": 1, "pair": "111a5a80a3abc2c9", "unit": "c7ae403742450deb2bd051ad1231121d2117ba56fa72a038c82d58d207cb3870"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "koajs/koa", "label": 0, "pair": "111a5a80a3abc2c9", "unit": "debe34831afca6f8220b9952036a2d4570babdb5e321a2f8dbaec661a016bd50"}
{"family": "access_control", "fold": "outer-0", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-crm-saas-django", "label": 1, "pair": "", "unit": "9950b85412b021ebabb11eea4c490a660ad380d14c673a87c3fb568ba4bcfac9"}
{"family": "path", "fold": "outer-0", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-crm-saas-django", "label": 1, "pair": "", "unit": "656df5ff8e69dcdb1f9ba1ed52d734966eb29773bc745587483a355743b8eb5a"}
{"family": "access_control", "fold": "outer-3", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-hr-payroll-django", "label": 1, "pair": "", "unit": "91652a4e776fbce70cde9e4a99c6da5e17da7f8d6d0404867c61eec04161b6cf"}
{"family": "path", "fold": "outer-3", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-hr-payroll-django", "label": 1, "pair": "", "unit": "e53bb3a217dc4737e8dd03671de4f6a895599024070a985a268c65a034295e7c"}
{"family": "deserialization", "fold": "outer-0", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-legal-case-django", "label": 1, "pair": "", "unit": "3d7aba22444811a337abcad0e4fc62990bb704d3a10e72d893342fa94699026b"}
{"family": "output_encoding", "fold": "outer-0", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-legal-case-django", "label": 1, "pair": "", "unit": "5f18a3026a51f7140b7aaa3aa8563e8b6fdda6065d95e19e25b1dac4b9ce0e00"}
{"family": "deserialization", "fold": "outer-4", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-logistics-dispatch-fastapi", "label": 1, "pair": "", "unit": "0fccc306ce80cb4cbfdcde0a2e56d637bcefa9dd30423d6c54a4f77860afe740"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-logistics-dispatch-fastapi", "label": 1, "pair": "", "unit": "1e2eaa45661345f1b6759d968d7696dd0d96d6fba1350a00e4d0a2ebaaa2e391"}
{"family": "output_encoding", "fold": "outer-4", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-marketplace-commerce-fastapi", "label": 1, "pair": "", "unit": "8b9a8ff0e32a327df75eb8a1811464d5f34690079d5930139dd917063bdd2809"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-marketplace-commerce-fastapi", "label": 1, "pair": "", "unit": "245d0e939b7c1ccf098c798492009e3a73ea17400fa613dd865cd369d2e7f383"}
{"family": "access_control", "fold": "outer-4", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-property-management-fastapi", "label": 1, "pair": "", "unit": "653c48df493f31c2d01c973d1e379a180fbe6a6edc83a500bc62509112175f41"}
{"family": "path", "fold": "outer-4", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-property-management-fastapi", "label": 1, "pair": "", "unit": "3719fe21d45b50b35458bcd2e263e4baee67adbd3c98adf0eaab6bc6183c5651"}
{"family": "output_encoding", "fold": "outer-0", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-support-desk-fastapi", "label": 1, "pair": "", "unit": "1209a6cc8c1548ed83d4a9a09d6a27f706a23e69cf0c261fc5e3a27536a83306"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-support-desk-fastapi", "label": 1, "pair": "", "unit": "362c90495dc468bb732c71f62d71ec308854161de31ad18d2a870056d947ddb6"}
{"family": "access_control", "fold": "outer-1", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-crm-saas-django", "label": 1, "pair": "", "unit": "b52e53fec9707b2b442c4dc7a41d376cf76946d2c45c2eb600c9477fc6baebaa"}
{"family": "path", "fold": "outer-1", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-crm-saas-django", "label": 1, "pair": "", "unit": "477e3b90b92e622888dbe0e9d3a23b31271ddfa3104a5ab326eb9e9e765aa01d"}
{"family": "output_encoding", "fold": "outer-2", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-education-lms-django", "label": 1, "pair": "", "unit": "660c63c4a73bfe64088ae3249e40d9c5d69a6f8a505ff671217c5708a345c5b0"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-education-lms-django", "label": 1, "pair": "", "unit": "4d60c386c801bedb61c1461fced29a3cfb50b09d6435e8617c60398343719d93"}
{"family": "deserialization", "fold": "outer-1", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-fintech-lending-fastapi", "label": 1, "pair": "", "unit": "f1fddcc1354a29bef55610ee2b73eaa440192f2f18a0d22d08057b9d1e1b61b8"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-fintech-lending-fastapi", "label": 1, "pair": "", "unit": "6d9886c38221b2f9aacf83749218ac5ebc354a6fcc4170e6c78bedb92b88296a"}
{"family": "deserialization", "fold": "outer-4", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-healthcare-clinic-django", "label": 1, "pair": "", "unit": "5550fec4a81c4c9cb22d7312d8a63e1fb2b03c4aed0c1f316d44c1eb27511bac"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-healthcare-clinic-django", "label": 1, "pair": "", "unit": "8ff87dd04140f0137ea1891f220b1276156ce10a58be5a49d0c8e07be531ffb2"}
{"family": "deserialization", "fold": "outer-3", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-legal-case-django", "label": 1, "pair": "", "unit": "c6c9bde6630678377d250701073c5c223c7892e6ff8f2f8e0293dea22c5b5e7e"}
{"family": "output_encoding", "fold": "outer-3", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-legal-case-django", "label": 1, "pair": "", "unit": "b9b15958973c83f8d7b58372ab5ba58eb191c400125579d5ae8c5b52d5d1368e"}
{"family": "output_encoding", "fold": "outer-2", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-marketplace-commerce-fastapi", "label": 1, "pair": "", "unit": "b95ec12e4e126ad558e944e2c260116ed6a63f5f3ab97fb5cbcdca75a931ad73"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-marketplace-commerce-fastapi", "label": 1, "pair": "", "unit": "38a5545ac9ef67e15db720d1396a2d5915f9d06d6f4772012e5bc95f33770376"}
{"family": "access_control", "fold": "outer-2", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-property-management-fastapi", "label": 1, "pair": "", "unit": "c8bbe9be91c976be97b7ca387048a7d4537d9f6fcc6cbad0a541d1080629d5a6"}
{"family": "path", "fold": "outer-2", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-property-management-fastapi", "label": 1, "pair": "", "unit": "ffacd53f1bd75155e63b318fb9cba27ec03e754841e1eb39d1d4fa0b2c3dfca2"}
{"family": "output_encoding", "fold": "outer-1", "group": "kolega-ai-dev/vc-codex-seeded-v2-healthcare-clinic-django", "label": 1, "pair": "", "unit": "8fd920175fc43b91b4397acffacb10ab8f7fd0ee38bcf2d2f3ed06a3fad0dfe9"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "kolega-ai-dev/vc-codex-seeded-v2-healthcare-clinic-django", "label": 1, "pair": "", "unit": "86c9598ebe3dcda261a69ae545513f6ce73101f3980066e4506ece99bebea84f"}
{"family": "deserialization", "fold": "outer-2", "group": "kolega-ai-dev/vc-codex-seeded-v2-hr-payroll-django", "label": 1, "pair": "", "unit": "e5190fcc57505116f79a31c51aed69f8a1eea13a8ae3e7265f44c72418277ef9"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "kolega-ai-dev/vc-codex-seeded-v2-hr-payroll-django", "label": 1, "pair": "", "unit": "2b807ff54e6c47e941124c16d0c27c691cef54f2203060a3a6ce812d478c5261"}
{"family": "deserialization", "fold": "outer-3", "group": "kolega-ai-dev/vc-codex-seeded-v2-legal-case-django", "label": 1, "pair": "", "unit": "401aefb4b4791301ace2d770d321c6b26743b0aab7b96f8b410254c704be6d2e"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "kolega-ai-dev/vc-codex-seeded-v2-legal-case-django", "label": 1, "pair": "", "unit": "22d186c93498e21418b6c1d0547a73374b13041d8afc89a80d9540fa658b6b42"}
{"family": "output_encoding", "fold": "outer-1", "group": "kolega-ai-dev/vc-codex-seeded-v2-marketplace-commerce-fastapi", "label": 1, "pair": "", "unit": "dbfb6df14fe37f8c0efda5f16643297fb0550dfc5ad5805d4e4b844d967773b2"}
{"family": "output_encoding", "fold": "outer-1", "group": "kolega-ai-dev/vc-codex-seeded-v2-marketplace-commerce-fastapi", "label": 1, "pair": "", "unit": "f0da8cfe4ac1e586aa7903d72e67b44500bbd5306d72b4b342d10613efc69222"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "kolega-ai-dev/vc-codex-seeded-v2-support-desk-fastapi", "label": 1, "pair": "", "unit": "2b8fd73943f456da266434470333a7775f55d9a41953e97a000fdaa3257b6a30"}
{"family": "deserialization", "fold": "outer-1", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-crm-saas-django", "label": 1, "pair": "", "unit": "a612091f452e60f28f44eefec641175170dc5db04456a39a099cbb13c50b4d2c"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-crm-saas-django", "label": 1, "pair": "", "unit": "4c6600facd83bbf234a32d9e28ac494fb516df6e3d9e9dc28d720ff35252d067"}
{"family": "deserialization", "fold": "outer-0", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-education-lms-django", "label": 1, "pair": "", "unit": "fb0ea8cab8c9430d22d1b0fdd65a69146fa95fbff94e720b4861b96538d992c6"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-education-lms-django", "label": 1, "pair": "", "unit": "e81f91a56058ac1473579767abea57064c53d7d007797a854314814e3c35ee74"}
{"family": "access_control", "fold": "outer-3", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-fintech-lending-fastapi", "label": 1, "pair": "", "unit": "5851ecd96a449f98b05f243bfda0a7cf3af8612b27f53c414ae1f4ea67242a7f"}
{"family": "path", "fold": "outer-3", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-fintech-lending-fastapi", "label": 1, "pair": "", "unit": "328621ef2a94632e5d37ea6267d17b766233384cde35c07e29a4eb672d40ba28"}
{"family": "output_encoding", "fold": "outer-3", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-healthcare-clinic-django", "label": 1, "pair": "", "unit": "04b2eae2128608dad460ba8275142a41a44e32c4243a29e2a6fb2c9ebf3d8f1a"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-healthcare-clinic-django", "label": 1, "pair": "", "unit": "33337002ff5f5a189610b042522ceea254c9d107c1f24771b4f616a0f96f636e"}
{"family": "deserialization", "fold": "outer-4", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-hr-payroll-django", "label": 1, "pair": "", "unit": "02c09a783cbf058856fc2b448406f3ffa6fbfbacf9b70f4e5e22fd741a5c7699"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-hr-payroll-django", "label": 1, "pair": "", "unit": "956260164cb21a47c499e5598491031ef303418943f5096f225a87162b588574"}
{"family": "deserialization", "fold": "outer-0", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-legal-case-django", "label": 1, "pair": "", "unit": "8bd8c5aed69eae3019cc2d3654b36bdf2b31435e6a020a389cd103696a2f874c"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-legal-case-django", "label": 1, "pair": "", "unit": "1d8397ef4a9fde7b48e2e1e871bba0f960a802cd7823ecfc10e9dfaddb955ccc"}
{"family": "output_encoding", "fold": "outer-3", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-marketplace-commerce-fastapi", "label": 1, "pair": "", "unit": "c9db92476032eb4296b950a1eff8fa4aaa26f11542a01a5cf10c7b2e32dc6433"}
{"family": "output_encoding", "fold": "outer-3", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-marketplace-commerce-fastapi", "label": 1, "pair": "", "unit": "d3e0d0f9b0f4b723080989b5486395cda0a5b186a674022ea70e7fa7ab561323"}
{"family": "deserialization", "fold": "outer-2", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-property-management-fastapi", "label": 1, "pair": "", "unit": "44ff3b3e61c02744830c4abea872821ca7e0f94950c082740c5c48ef13604e94"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-property-management-fastapi", "label": 1, "pair": "", "unit": "4103e979a99233c3e71fdfe9f1cdcbab34a2c60b5351b75c5ff5dfc287b962d0"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "kozea/weasyprint", "label": 1, "pair": "6768bfae38c6859d", "unit": "f376519889a00950f5af67eb15c639306ea70da7235d3be4a94dbc19cb096cf5"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "kozea/weasyprint", "label": 0, "pair": "6768bfae38c6859d", "unit": "f3fea4e334f49ef6fe6b17121fef553eb8e33a9ffa13e6d6cbc4c448da996a7c"}
{"family": "access_control", "fold": "outer-1", "group": "kubev2v/migration-planner", "label": 1, "pair": "d670253fa209671c", "unit": "1aeebe2e440bdd70525c31f93180f33be93f820db5ba46a2e59c371ef83ee794"}
{"family": "access_control", "fold": "outer-1", "group": "kubev2v/migration-planner", "label": 0, "pair": "d670253fa209671c", "unit": "b11c8a8a24d3809c0507228568fe7cc05ca0a22ed28cb19019adb13cb619d63f"}
{"family": "injection", "fold": "outer-0", "group": "kylefarris/clamscan", "label": 0, "pair": "33260f3e6c8ea6d0", "unit": "7ca409c7dbc1c38ff005624e212efc6d424cb421e27b482dc8dc0d6881f4c11a"}
{"family": "injection", "fold": "outer-0", "group": "kylefarris/clamscan", "label": 1, "pair": "33260f3e6c8ea6d0", "unit": "fc364d7ed77c7c0508da00e5ba847ec0792f3bc373d8b4297747189ef42d1ae0"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 1, "pair": "", "unit": "6cce78d7569ad8db7ca0a61d7078198f666502d26ea6c0afb03173a8a4bb16ef"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 1, "pair": "82047c0513e43a83", "unit": "6d2dbed1ab1c9d8ac259dd4227be69641fa59d52e816a8dc547ade01d0b9b0c4"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 1, "pair": "98ee23cd6bd31cd3", "unit": "ab4bb39be535d70cd95e9035bb958bfc7418576a6ff40a47551894dd140317bb"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 0, "pair": "98ee23cd6bd31cd3", "unit": "b70068cbc646bd01d3e10986e002f648a74e9d798900dc59cf3039b749c95c18"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 1, "pair": "ff168d95d8d9496e", "unit": "c6ec9478c125ed780d7013c2807f2cec1878f44253667dfed319c28cbfd17d4d"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 0, "pair": "b1bfef46dcd3b9a5", "unit": "cd4d095d8380a2fb7ca02fc6abdbae5249cc06455db2929875533f4aa8b886dc"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 0, "pair": "82047c0513e43a83", "unit": "d1bade217be8d345cf5d0f0442ee31924187acbb98779dada95274a77462f696"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 0, "pair": "ff168d95d8d9496e", "unit": "d561db250b892b91feed7ae877900e1e25963444f73c7699576e010f06da73cf"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 1, "pair": "b1bfef46dcd3b9a5", "unit": "ed78fd5a7ab30bcd6e8ddd27c706a19cfd89750de4fbf81c0b567946eb94db83"}
{"family": "output_encoding", "fold": "outer-2", "group": "lepture/mistune", "label": 0, "pair": "25f75d77a669b54e", "unit": "0f2825568461562f2a915a4f16ef8a4931290fd20a9801470aeb12677e66e3f1"}
{"family": "output_encoding", "fold": "outer-2", "group": "lepture/mistune", "label": 1, "pair": "25f75d77a669b54e", "unit": "bf22bb44bcff4a5baf5d550c9d3e83e5f3ca607475fb51deaf4fb6fccff3d73a"}
{"family": "output_encoding", "fold": "outer-4", "group": "linuxuser255/vulnerable_python_apps", "label": 1, "pair": "", "unit": "cd80750447cf7af67ad809279718d98106c71705fef9ff7254728bfff4ece5b1"}
{"family": "path", "fold": "outer-4", "group": "linuxuser255/vulnerable_python_apps", "label": 1, "pair": "", "unit": "2ce605ab5ce7ad3a1fb983cd6bddf3924e5da3010d2252a0a4640aab4e5921a0"}
{"family": "deserialization", "fold": "outer-1", "group": "litedb-org/litedb", "label": 1, "pair": "b0a899116dc36b85", "unit": "510ad48af03ba4a1a2a06dd254b135311dddb62d9fe5a2e8e8b12627586e6b13"}
{"family": "deserialization", "fold": "outer-1", "group": "litedb-org/litedb", "label": 0, "pair": "b0a899116dc36b85", "unit": "b414718a8a5f79739b88db24da6ad9da8e48bcb6b5e2dc99158848cdaa226767"}
{"family": "deserialization", "fold": "outer-4", "group": "manuelz120/extremely-vulnerable-flask-app", "label": 1, "pair": "", "unit": "8e22ee30441b20484090659b0c97423c711f5d10219041bf3a399623fc056a07"}
{"family": "injection", "fold": "outer-4", "group": "manuelz120/extremely-vulnerable-flask-app", "label": 1, "pair": "", "unit": "0f9070df9dc334f981cb01008c260b8bff78dbdeb48ce672830ed95caa8406fb"}
{"family": "injection", "fold": "outer-4", "group": "manuelz120/extremely-vulnerable-flask-app", "label": 1, "pair": "", "unit": "5e352f797b1422b0df477bce4730a9ce33cbd0837c3d63da94e4d8801a9a4b64"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "medplum/medplum", "label": 1, "pair": "dadccb128ac0e120", "unit": "505097494da5939404ad7d5b4af2b1c4a10efe94048084492d6ce60ebaa04c97"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "medplum/medplum", "label": 0, "pair": "dadccb128ac0e120", "unit": "84ff8ca2d98a1b5d1ca4bfaabe7d19e03a396042b228c8d18d1185a178389702"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "medplum/medplum", "label": 0, "pair": "", "unit": "91e261c54bd2c3c3a9ac04e48894ecabaca5b13e7db4fbd5578f9d266098117a"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "medplum/medplum", "label": 0, "pair": "", "unit": "a308dfd59bc398824e2454bf6cf6cfd5b7880e5b948a82424f14753d2b6fcdbe"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "medplum/medplum", "label": 0, "pair": "", "unit": "ba24eb554458bc97314ed561a7b2f4a4e643dde1205c32de4c8fcdd8c46e575b"}
{"family": "output_encoding", "fold": "outer-0", "group": "mganss/htmlsanitizer", "label": 0, "pair": "24d8f1840b67fbe5", "unit": "7ffa4a229a62266788fbf64890f81f20358b3eb8c35d3ab87d0c339d5bb4b507"}
{"family": "output_encoding", "fold": "outer-0", "group": "mganss/htmlsanitizer", "label": 1, "pair": "24d8f1840b67fbe5", "unit": "d19493ed7941e1cd25a76dc50504d97398770d07ebc302dd66d83441e946c468"}
{"family": "path", "fold": "outer-3", "group": "mguinness/elfinder.aspnet", "label": 0, "pair": "9f8a60eb48d74bc8", "unit": "80fce34f2377731e3de4355c7ea33bb20fc2fc00a5f388565b23018c7a3ed008"}
{"family": "path", "fold": "outer-3", "group": "mguinness/elfinder.aspnet", "label": 1, "pair": "9f8a60eb48d74bc8", "unit": "b68e62722a327c9d5ce6774eb96527451729410768efc0b5c7ba604ec468052d"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "miniflux/v2", "label": 0, "pair": "997f9d2efc32302d", "unit": "b587bb499945a9f5f0749eb4d57cc1027cfd21c8292bd32be4d4e1cc1b1db261"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "miniflux/v2", "label": 1, "pair": "997f9d2efc32302d", "unit": "f8d9c9a7c12797b3caf042e6bfdb89a5072c4eae1fd36be6196f9fa326cdf257"}
{"family": "output_encoding", "fold": "outer-4", "group": "mixxorz/slippers", "label": 0, "pair": "8e6b4b4e28981590", "unit": "3cce3e8f9ff3f1d261f9b96c844c1e4da07bae72d5393897b3a929be25372edc"}
{"family": "output_encoding", "fold": "outer-4", "group": "mixxorz/slippers", "label": 1, "pair": "8e6b4b4e28981590", "unit": "b105689ac24ed334294c434a77bb6d56d5d598830ffe8a5d60ab3be5290b47ad"}
{"family": "injection", "fold": "outer-3", "group": "mongo-express/mongo-express", "label": 1, "pair": "", "unit": "0a36cd5177f3b2c20b433db6bfffe5ec5b7e74cacc34a86ac581f7a0a559432a"}
{"family": "injection", "fold": "outer-3", "group": "mongo-express/mongo-express", "label": 1, "pair": "e7fc97d83013c9ff", "unit": "5ab444e64297fc19065478104c7ebdabe23cc415f22adafd0cf82c98e4c5cae3"}
{"family": "injection", "fold": "outer-3", "group": "mongo-express/mongo-express", "label": 0, "pair": "e7fc97d83013c9ff", "unit": "6b462ea30c1afda572600a10324249223932d36a032eacafac8c837bccbdce87"}
{"family": "injection", "fold": "outer-3", "group": "mongo-express/mongo-express", "label": 1, "pair": "84d6a4202a082e0d", "unit": "8175082f40d8d61d44b54579c84a808e25b89110be5d8fd82feedf7dd5e0d5ab"}
{"family": "injection", "fold": "outer-3", "group": "mongo-express/mongo-express", "label": 1, "pair": "", "unit": "8abefca43ee4ae2c797033da31f85c17bfa7be6b35413d70b0216415c5db9a09"}
{"family": "injection", "fold": "outer-3", "group": "mongo-express/mongo-express", "label": 1, "pair": "", "unit": "91d213946859f33d3f243c43aa8285cc6e8ecb3db1de938f32af775a70607f8e"}
{"family": "injection", "fold": "outer-3", "group": "mongo-express/mongo-express", "label": 1, "pair": "", "unit": "a244d9f010ddb43a09e82363dfabb0e406d411fb5de08ab7d6c9ee367f2a15c3"}
{"family": "injection", "fold": "outer-3", "group": "mongo-express/mongo-express", "label": 1, "pair": "", "unit": "d27d174d6e8e5f1682a1c42723705e89f0d18c95ecedd0567e313daa9a0aae44"}
{"family": "injection", "fold": "outer-3", "group": "mongo-express/mongo-express", "label": 0, "pair": "84d6a4202a082e0d", "unit": "f45608ed47fa11ca25093641f6021ec06076c18676abac9b809f31b3b7713de8"}
{"family": "access_control", "fold": "outer-4", "group": "moquette-io/moquette", "label": 0, "pair": "55d1ce4f29dd1583", "unit": "a877ce3462da316bcdd82ce9db1d9887e887eaa83157922edb1663413edae1d6"}
{"family": "access_control", "fold": "outer-4", "group": "moquette-io/moquette", "label": 1, "pair": "55d1ce4f29dd1583", "unit": "aee67be0ba31f99c7a24faa1bad6178c6d37361a308cf5aa10138e6ce4bbcfeb"}
{"family": "path", "fold": "outer-0", "group": "naorsabag/openhop", "label": 1, "pair": "16e529fe8c5486b2", "unit": "1d21acd0171726775d3da4589b660645fcd43cc026042a2b27e0e0e8374eceb7"}
{"family": "path", "fold": "outer-0", "group": "naorsabag/openhop", "label": 0, "pair": "16e529fe8c5486b2", "unit": "f8e004b67ac98dc3392ea4115051e6bd10ce22eb638a762f0f16e1d4200bf129"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "nuxt-modules/og-image", "label": 1, "pair": "ef603436079b0005", "unit": "77f843b5044bb5da935df4b2620bf2abe9e7dceb3147439e883927112eaec22f"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "nuxt-modules/og-image", "label": 0, "pair": "ef603436079b0005", "unit": "f2ce95f0e44e73986784b63a702ffa4035c423056cac3667a63140a7e8d795c2"}
{"family": "deserialization", "fold": "outer-3", "group": "opensearch-project/opensearch-ruby", "label": 1, "pair": "1945f28a3fc5bd55", "unit": "64c0b7ccad09192eeaf590befe0de0087d5b48a65a4f6cef41415f3f54a8f536"}
{"family": "deserialization", "fold": "outer-3", "group": "opensearch-project/opensearch-ruby", "label": 0, "pair": "1945f28a3fc5bd55", "unit": "b55cd6759d4b7861e589052aa534bd0554586db46a329ed3beed7244d579e8c4"}
{"family": "injection", "fold": "outer-3", "group": "openssl/openssl", "label": 0, "pair": "1c9e47835232f644", "unit": "90e220d3540b88dd9efc8e66c8ab519d4cc78805f16bed585b795c78cfbb4d58"}
{"family": "injection", "fold": "outer-3", "group": "openssl/openssl", "label": 1, "pair": "1c9e47835232f644", "unit": "b5a185b05235f70f6cf27e2adba919f4d8393dbdf2d99de095548a21946aeaf3"}
{"family": "access_control", "fold": "outer-3", "group": "openwisp/openwisp-ipam", "label": 1, "pair": "9b2b34cf62a76ea3", "unit": "91a3984e47f17512db8523e04f863525c67db18631d8f11776cf61ca7221c91b"}
{"family": "access_control", "fold": "outer-3", "group": "openwisp/openwisp-ipam", "label": 0, "pair": "9b2b34cf62a76ea3", "unit": "d91f1e14282a0c3c8da57c6935a1625e881374bd205b5baa9e4a341d9109ae21"}
{"family": "access_control", "fold": "outer-3", "group": "oqtane/oqtane.framework", "label": 0, "pair": "a20cdf9dd208720d", "unit": "18a26df68e4b81d411cfbcc8155a7e6f382753d6e1ad95eece7ed24fa8c44324"}
{"family": "access_control", "fold": "outer-3", "group": "oqtane/oqtane.framework", "label": 1, "pair": "a20cdf9dd208720d", "unit": "e6fb74441fb585d7ef5855317fba795ef0f0a0e6eae9970a49556ce3070c3254"}
{"family": "path", "fold": "outer-4", "group": "owen2345/camaleon-cms", "label": 1, "pair": "d6930fe7e752b7ec", "unit": "06f0341dfbf1aa9504083737df5d9be3cedfd4e3a198799d8075a29990776800"}
{"family": "path", "fold": "outer-4", "group": "owen2345/camaleon-cms", "label": 0, "pair": "d6930fe7e752b7ec", "unit": "30e2cd8bd381d7568b78eaf982f562d278a6a8d942d5282466532c1441bdf0ad"}
{"family": "access_control", "fold": "outer-4", "group": "perses/perses", "label": 0, "pair": "2d482951d7380c4f", "unit": "5c1a721da6f85d1d5dfc8cbe5e9809b093a09c059e5abeebed2cbbee55fbd498"}
{"family": "access_control", "fold": "outer-4", "group": "perses/perses", "label": 1, "pair": "2d482951d7380c4f", "unit": "6cabd1c4b5aa94cafdd07c0c6a3f9abfdc03761ddc5b0a2bebc72a508d020b8b"}
{"family": "injection", "fold": "outer-2", "group": "pgadmin-org/pgadmin4", "label": 0, "pair": "", "unit": "2300c097374b9098671dcbcd42d02d08172e0f3064a8323223d0fe79a6106aab"}
{"family": "injection", "fold": "outer-2", "group": "pgadmin-org/pgadmin4", "label": 1, "pair": "0664148b0b192456", "unit": "d702617aaf4d19811baaa18386a3c8b0eb35cd3fd72e8d90611d34850e990858"}
{"family": "injection", "fold": "outer-2", "group": "pgadmin-org/pgadmin4", "label": 0, "pair": "0664148b0b192456", "unit": "f5ed13d811c5b16d46cbd4a3b9edb8f7d2db49acbd49d95f39f33eaab38f58f5"}
{"family": "access_control", "fold": "outer-1", "group": "publify/publify", "label": 0, "pair": "1972ea9f72b178b6", "unit": "3ddabb356d78e7c65ec8f88b6da8e4fcabc055ae73f2f6d92c50efddc5d4fcbe"}
{"family": "access_control", "fold": "outer-1", "group": "publify/publify", "label": 1, "pair": "1972ea9f72b178b6", "unit": "6875dd5e9c0ff8a1b693d29a109cb2e62e6ff4081df40868aca70e4770476757"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 1, "pair": "04aa777130c02a06", "unit": "16aa43c87956c44a4b115245140eefed704095d37cec0cc6410377403cd2fa7a"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 0, "pair": "", "unit": "23744137e0485a36f686f5f1b5666780fbed3ca522e8c6d35687a391ef504498"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 0, "pair": "67565d99fe28c68f", "unit": "2692316792e6f02fb51578cf30d71bba6c73bf5420c36f709681c8eaa69807d4"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 0, "pair": "", "unit": "2e36f73002c416a68d077bffb0335d82346c7507f2478353aeac005fce5a9cbc"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 0, "pair": "04aa777130c02a06", "unit": "67721f67055eb6fed955ff22bfe4060344662eccb369431835a2116ad120c4f8"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 0, "pair": "", "unit": "8fc1c96fa6faa4d3966ecc42876189b18ddab779d58f47cdf28a1155f48a6ac4"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 1, "pair": "67565d99fe28c68f", "unit": "ab535098a8e709b022afee1da15dd1fc371d389056938ee4fa4971b2966c823a"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 0, "pair": "", "unit": "d46f94235bc462360097e7a22e1d442d95f049306631489a4d8b25f647a95bbe"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 0, "pair": "", "unit": "edd6dd4364442f66384d14e9317ef2f6234e32ab78327a45741b8a816b4332ee"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 0, "pair": "", "unit": "fc7df6887ac719d4615934c0e8c3296728c2f6089f233158c59989027c802d72"}
{"family": "path", "fold": "outer-2", "group": "rack/rack", "label": 1, "pair": "c0412509f15fe3e1", "unit": "63bda2a71ad7155ad2f59aa9086efef3fd3710dc82c57f4b32ec8bf561460239"}
{"family": "path", "fold": "outer-2", "group": "rack/rack", "label": 0, "pair": "c0412509f15fe3e1", "unit": "b1c9a070c93764655786c3de5fde536c150e319efe47548614cc544b73a8ee2b"}
{"family": "deserialization", "fold": "outer-3", "group": "rails/kredis", "label": 1, "pair": "feed5f7de11289a4", "unit": "71531567b2b8609ae0d31cb177b8faea4f5fc61923e513b0154a2c941287ce2b"}
{"family": "deserialization", "fold": "outer-3", "group": "rails/kredis", "label": 0, "pair": "feed5f7de11289a4", "unit": "d6be6d397a301a07faa761163d3d9e270d71d23d3bdfd084eeafb1e62b91dd71"}
{"family": "deserialization", "fold": "outer-2", "group": "redisson/redisson", "label": 1, "pair": "ffd4b04a7bebfaf8", "unit": "54f7519d3d59b6d8d69a5f8f7bd78e360d3fabaae00b7587c37e388719169370"}
{"family": "deserialization", "fold": "outer-2", "group": "redisson/redisson", "label": 0, "pair": "ffd4b04a7bebfaf8", "unit": "c5e42c9ab2ee54e46700ab0b64e239b567fbc75ccaceaf6cbf20687813663332"}
{"family": "deserialization", "fold": "outer-0", "group": "remix-run/react-router", "label": 1, "pair": "73f2a014c6a699ed", "unit": "44ba9d9d8e75769c1444ebf37a64941a845b4bcae678ec79d64dc84e5cc067a0"}
{"family": "deserialization", "fold": "outer-0", "group": "remix-run/react-router", "label": 0, "pair": "73f2a014c6a699ed", "unit": "b430d488c486cef26b1c661f773e7e9e4a177eeea73537a033eec98d532e4eed"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "remram44/taguette", "label": 1, "pair": "cb0c638282db0d37", "unit": "a49e6ad2e7c2a5ad0d68d5d69c263306b501fc0dc9382e701fb5fe680611196b"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "remram44/taguette", "label": 0, "pair": "cb0c638282db0d37", "unit": "dee28099d3b37cbd0ee14412122ecc5f8488ed6bec01863520b87706a040fba1"}
{"family": "injection", "fold": "outer-4", "group": "roest01/node-pdf-image", "label": 0, "pair": "09bce271e8530fbc", "unit": "35eb19b4c6176b37287e2123bb98ec4045ce6bde7cc3864dde317c024b0935d8"}
{"family": "injection", "fold": "outer-4", "group": "roest01/node-pdf-image", "label": 1, "pair": "09bce271e8530fbc", "unit": "dd33291cc494bbc77d44e9b13d16f171d32c9746637506f315f375e8a9ddee95"}
{"family": "injection", "fold": "outer-2", "group": "rrainn/portprocesses", "label": 1, "pair": "40d8d81a764c508f", "unit": "7deeed73c29c74ccbed29be049e109d363d41e4b5eea245ee57780b85f0e137a"}
{"family": "injection", "fold": "outer-2", "group": "rrainn/portprocesses", "label": 0, "pair": "40d8d81a764c508f", "unit": "b5cdadf848e67910550666ba2a6ac67b24112e501c02dba1865766a316317a23"}
{"family": "injection", "fold": "outer-3", "group": "saltcorn/saltcorn", "label": 1, "pair": "", "unit": "15dbea384b3255457a44d9382efc6944dd27f1d411d903344a5d5e5fd194b72f"}
{"family": "injection", "fold": "outer-3", "group": "saltcorn/saltcorn", "label": 1, "pair": "b2ac284bd70dd684", "unit": "225890ba6694dc8f4fbab46e9c851dff8c095e906cf14db15d01ebce0891aaab"}
{"family": "injection", "fold": "outer-3", "group": "saltcorn/saltcorn", "label": 1, "pair": "b6eca35de93a5be8", "unit": "42eb8427415704e060e73b5ce0943b014ddf380ca756d5ec47c558e59a79895a"}
{"family": "injection", "fold": "outer-3", "group": "saltcorn/saltcorn", "label": 0, "pair": "b6eca35de93a5be8", "unit": "4c9cdcc11a18641ef41e9308712c5e8b830ec9ad5ebc3a956c8701e4f77b39c0"}
{"family": "injection", "fold": "outer-3", "group": "saltcorn/saltcorn", "label": 0, "pair": "6a53939838afbdd7", "unit": "5794809e595311c86909d673a570d003aa025e4dcd6abc5641b641d935beb0b1"}
{"family": "injection", "fold": "outer-3", "group": "saltcorn/saltcorn", "label": 0, "pair": "46d3ec7d49cb8ca7", "unit": "5f7a63ec650915962ee5255f5416df63ab882d6107f3b540be36d481798a8944"}
{"family": "injection", "fold": "outer-3", "group": "saltcorn/saltcorn", "label": 1, "pair": "6a53939838afbdd7", "unit": "97a7018535ab511622e3ade9e53d570ffb1c80633eaa372fb76e749b3a586c8f"}
{"family": "injection", "fold": "outer-3", "group": "saltcorn/saltcorn", "label": 1, "pair": "46d3ec7d49cb8ca7", "unit": "a84e9ab47f4f2b584684dbb81575758d9032fe3f2ed53bef478cb9dcfedaf677"}
{"family": "injection", "fold": "outer-3", "group": "saltcorn/saltcorn", "label": 0, "pair": "b2ac284bd70dd684", "unit": "aeceb1ebc863ebbc3b69ae56c5b78d2dba005132b75cbd6af929bb0bfe25da9b"}
{"family": "injection", "fold": "outer-3", "group": "saltcorn/saltcorn", "label": 1, "pair": "f42cab375ace1739", "unit": "e329d4a6ac757680bf3d292a81f1f74f8a8a9b482d1651efe9f503f6a4de8796"}
{"family": "injection", "fold": "outer-3", "group": "saltcorn/saltcorn", "label": 0, "pair": "f42cab375ace1739", "unit": "e73c5aac41c0b93072223a3097fb175548ce893e0678ff3a22bc7571745a17e0"}
{"family": "injection", "fold": "outer-2", "group": "sgabe/dsvpwa", "label": 1, "pair": "", "unit": "52354ccdbc16f209e4692d97d5237e04ed030cf4a5e71f77b5a767962b355b4a"}
{"family": "injection", "fold": "outer-2", "group": "sgabe/dsvpwa", "label": 1, "pair": "7d23be667536afff", "unit": "b5016a5a53409491ba805ed8718599b84250b7b6914da0434b0882b0b3578dc8"}
{"family": "injection", "fold": "outer-2", "group": "sgabe/dsvpwa", "label": 0, "pair": "7d23be667536afff", "unit": "f1b01102968568d3ba1b2fdc70f4ae281f433fb20d29eff2a55fc45555ddbfcd"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "signalk/signalk-server", "label": 0, "pair": "1199de75f426940b", "unit": "10fdc236f43a02b7e291eb8bf6ada297a2ae98086a63b171057bc7bfc37a3377"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "signalk/signalk-server", "label": 0, "pair": "22112c31603bbe9f", "unit": "5eb3b7523b5a196dcaacba08d68457f9a45e31503ea02745ca3a7d0542f9881c"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "signalk/signalk-server", "label": 1, "pair": "22112c31603bbe9f", "unit": "6bd5bdfd81b0a3374d59cfb2c1f7fa2b27887f4672f4485e77eb5234b569983f"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "signalk/signalk-server", "label": 1, "pair": "1199de75f426940b", "unit": "c09dffd04708540845c8d1dd77a2bd989bd35f264abf652a2db55f8cb1faf7a9"}
{"family": "output_encoding", "fold": "outer-1", "group": "silverstripe/silverstripe-versioned", "label": 0, "pair": "7aab025ea7b97019", "unit": "7487a34cdfdb532a5096764a47a3e565fa509605c1df0e2b105eb9c751fc5307"}
{"family": "output_encoding", "fold": "outer-1", "group": "silverstripe/silverstripe-versioned", "label": 1, "pair": "7aab025ea7b97019", "unit": "9c0be2a58741f095fd0a9cdc60d4ca4039de1529699d5701dc4d0bcf7d5bf90c"}
{"family": "path", "fold": "outer-0", "group": "simonh1000/angular-http-server", "label": 0, "pair": "4de5441f1776f083", "unit": "a3a35d573da308cd1d0614a26616bbd37b360d9ea416894cdeffdb59abd691d4"}
{"family": "path", "fold": "outer-0", "group": "simonh1000/angular-http-server", "label": 1, "pair": "4de5441f1776f083", "unit": "dc33fc26c1bf7c67e5439ddfdad06c0faf16c796e0fe8a46b6e6a541e708676d"}
{"family": "access_control", "fold": "outer-3", "group": "smithyhq/sqladmin", "label": 1, "pair": "d76d5232a8587127", "unit": "1ec32d427822d1ddd3b0d7245f5771ddeb56c146ac8ff9959dd85760ff659a20"}
{"family": "access_control", "fold": "outer-3", "group": "smithyhq/sqladmin", "label": 0, "pair": "d76d5232a8587127", "unit": "5143b19a1a1e6b9162b80594ed7fb5c0c57ed0b8abfd793bfc71089552d57638"}
{"family": "injection", "fold": "outer-1", "group": "soyuka/pidusage", "label": 0, "pair": "5f6e814444b4acd8", "unit": "2e8aa07b1aa4bfc5349b0d4464f3f2a6500aa55ff765b0139e9901ed34151ffa"}
{"family": "injection", "fold": "outer-1", "group": "soyuka/pidusage", "label": 1, "pair": "5f6e814444b4acd8", "unit": "e26c726c08be761b22fcc6ec17fc1fa7aba4769f08957f024e5545906df62797"}
{"family": "output_encoding", "fold": "outer-2", "group": "spatie/schema-org", "label": 1, "pair": "888e955fb6cfb4fd", "unit": "6f1a431ca97afc34ceec48393eb134a8928a312e6f761eb907c6b5855cac6208"}
{"family": "output_encoding", "fold": "outer-2", "group": "spatie/schema-org", "label": 0, "pair": "888e955fb6cfb4fd", "unit": "7b62415d17569924d16238a4aa28ab1af2db6561f56be9f59d8075fd1bc4a941"}
{"family": "injection", "fold": "outer-3", "group": "strangerstudios/paid-memberships-pro", "label": 1, "pair": "", "unit": "ac905d12b92954263c046bd0138a682cea86cf8476336d6baa664ade28d34ac6"}
{"family": "injection", "fold": "outer-3", "group": "sulu/sulu", "label": 1, "pair": "218a45d869bd7e39", "unit": "309ebac468b7ab054c8396ed495770385ae3590caba43709745ba2d1db4b6d54"}
{"family": "injection", "fold": "outer-3", "group": "sulu/sulu", "label": 1, "pair": "", "unit": "3cc8613813dec13cb1b7b45b296688da883f0b36cd2c4fc1da6e81703fb59ab9"}
{"family": "injection", "fold": "outer-3", "group": "sulu/sulu", "label": 0, "pair": "218a45d869bd7e39", "unit": "9bd2331dbc260451355307e4c37bae17ad6676654c66181934e1b35fb174cc5a"}
{"family": "injection", "fold": "outer-3", "group": "sulu/sulu", "label": 0, "pair": "", "unit": "b087ac4d0989e0b72698a9644ef4eac8073b4a15bb21730d63e4730b5cc58a60"}
{"family": "injection", "fold": "outer-3", "group": "sulu/sulu", "label": 0, "pair": "", "unit": "c5d1e8d19560c997018fdca12eb60f1799be76c33be058b88472f5a2fc6709be"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "themegrill/user-registration", "label": 0, "pair": "", "unit": "22befc17a75a688e2f9358d2ddf9a7b197da44cd95dbdeae5e0191b113ee54d3"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "themegrill/user-registration", "label": 0, "pair": "63f496d9ff53e39b", "unit": "6a98f8145776c78793b37156e9a81757841bb6de80016ab4fa575209d724db3f"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "themegrill/user-registration", "label": 1, "pair": "63f496d9ff53e39b", "unit": "7ac60c96c15f8517e089cc2240aedcf1fc30e54a41e35fe03c6b965bc393fbd6"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "themegrill/user-registration", "label": 0, "pair": "", "unit": "ae39cb55ef2a87c5347e0ddc40975f142450e82de4909e93ba62a7ed0efb4134"}
{"family": "injection", "fold": "outer-4", "group": "thenables/thenify", "label": 1, "pair": "d1b19898f8deaa41", "unit": "3763c7b407ad471820b5f8fa25bde0ba6d91e15c07c56681de4b41add418b0b1"}
{"family": "injection", "fold": "outer-4", "group": "thenables/thenify", "label": 0, "pair": "d1b19898f8deaa41", "unit": "bc8b934938c30687f3d4f982392d2cfa9504a916b463ccb5c46fe8a30eede488"}
{"family": "path", "fold": "outer-1", "group": "thumbor/thumbor", "label": 1, "pair": "f5f33fa6fe7b8492", "unit": "938bbf146a9037e687190bcee82c047492dffbe90efea40a49a647f81a0f1ed0"}
{"family": "path", "fold": "outer-1", "group": "thumbor/thumbor", "label": 0, "pair": "f5f33fa6fe7b8492", "unit": "b90c9f7e0011685f1a8e853a9ed0ae37fd437ef6d6eb0ba17c240aec2c22b603"}
{"family": "deserialization", "fold": "outer-3", "group": "tomasnorre/crawler", "label": 0, "pair": "f74364dc8e35c8ba", "unit": "11e760e99c5c852676c9288e18a250febb3854e9047b51e8b9ecc9f3b423b6f5"}
{"family": "deserialization", "fold": "outer-3", "group": "tomasnorre/crawler", "label": 1, "pair": "f74364dc8e35c8ba", "unit": "5fc980194040f4c4d803075d4085618cc0260ae8f3454b26b9b704d30200ffea"}
{"family": "output_encoding", "fold": "outer-3", "group": "tornadoweb/tornado", "label": 0, "pair": "db1c51833994ec55", "unit": "c2ce8320c778dd55434d7413844797b68e6fcabc4dc509458a9d60ca70b88c60"}
{"family": "output_encoding", "fold": "outer-3", "group": "tornadoweb/tornado", "label": 1, "pair": "db1c51833994ec55", "unit": "f2d6843547af1276ae8a4dff3a7786913fa79dc69540d4fbb95c4c091e27c7f2"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "truelayer/truelayer-dotnet", "label": 1, "pair": "f4690679bf9d80aa", "unit": "1a4e24df8d08672ecde4709a77ea5c437b2198fecff8bfff373b63c9eaed6e63"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "truelayer/truelayer-dotnet", "label": 0, "pair": "f4690679bf9d80aa", "unit": "78711973ec84867671314dd7878769092b32f59860eafed50a0f6979daee76fc"}
{"family": "injection", "fold": "outer-2", "group": "ultimatemember/ultimatemember", "label": 0, "pair": "85c15e3fc2b370b2", "unit": "0990d8f41cb07c1cf217812e51a7025381de4ba68d1be3799e26241d81dcea67"}
{"family": "injection", "fold": "outer-2", "group": "ultimatemember/ultimatemember", "label": 1, "pair": "85c15e3fc2b370b2", "unit": "9da1d63f0c4da270f66eb87bf9c342ace883a0aef78c4643e254159b5da8fce3"}
{"family": "output_encoding", "fold": "outer-0", "group": "viewcomponent/view_component", "label": 0, "pair": "8f5a6abedd742bbe", "unit": "71e3ced643a244100cc3fde234f5b4b27655214c7ae0ff67ab61e3e48a8fdf78"}
{"family": "output_encoding", "fold": "outer-0", "group": "viewcomponent/view_component", "label": 1, "pair": "8f5a6abedd742bbe", "unit": "7c8cbfae87127042f9f43621345b4f7958bde27a3ff9db76254063e80735f3ce"}
{"family": "injection", "fold": "outer-1", "group": "vincentmorneau/apex-publish-static-files", "label": 1, "pair": "c9538342f2d7efca", "unit": "2b8db57d65ef1d67c457ff5a336e8a542d03387395d69d5e16815c4f00f40723"}
{"family": "injection", "fold": "outer-1", "group": "vincentmorneau/apex-publish-static-files", "label": 0, "pair": "c9538342f2d7efca", "unit": "e2ed113dc21ad5f7caa71ba4a5ece8f8f900a97fd0d00dc85f7ac79ff5bed19a"}
{"family": "injection", "fold": "outer-1", "group": "vvo/node-whereis", "label": 1, "pair": "8acd426c5771b37c", "unit": "983e146a60e061254b15dc8dd573cbba57f8dd0358f23ea1b9c313ef69069af5"}
{"family": "injection", "fold": "outer-1", "group": "vvo/node-whereis", "label": 0, "pair": "8acd426c5771b37c", "unit": "f80dfec45231e4778078cb10f00128ed445177cac0a062fc2df5ced5c0171f21"}
{"family": "deserialization", "fold": "outer-4", "group": "waterfutures/epyt-flow", "label": 0, "pair": "9b84800ce5c395cb", "unit": "9b7dfdceae03fd86ad3260a03f85bcdcff8f32f10c265d23743ff8dc98eeefb8"}
{"family": "deserialization", "fold": "outer-4", "group": "waterfutures/epyt-flow", "label": 1, "pair": "9b84800ce5c395cb", "unit": "b99dda150c0da3defa5f4c527ef6c402f5589dfb2f2f03f3e5e424910eb48c53"}
{"family": "path", "fold": "outer-4", "group": "webpack/webpack-dev-middleware", "label": 1, "pair": "ad64e8fac4ac3ee0", "unit": "36dfd01915a553d86576e8ecd3b737b8e2b297aa542f58cc83b448060e74398a"}
{"family": "path", "fold": "outer-4", "group": "webpack/webpack-dev-middleware", "label": 0, "pair": "ad64e8fac4ac3ee0", "unit": "c2b6d40f130d35bbfc16fab9411c664fa06321c59fb1d901a1987a69cb09c72e"}
{"family": "access_control", "fold": "outer-4", "group": "wintercms/winter", "label": 0, "pair": "d48c08b19f560848", "unit": "03d02c9e25eec0f380b125279fce35e8867065bf6e1ec75f9f39729ce6efdc77"}
{"family": "access_control", "fold": "outer-4", "group": "wintercms/winter", "label": 1, "pair": "d48c08b19f560848", "unit": "abbea69b7d0fa68c3ace442c25597e128cde354b417787b5913fd10d666c0d19"}
{"family": "output_encoding", "fold": "outer-3", "group": "xyproto/algernon", "label": 0, "pair": "3844ed3d703d7ef4", "unit": "445efda074e4d9f6c1634225eb08a3c3eb1257311c995ce891df2c791a6f5c8b"}
{"family": "output_encoding", "fold": "outer-3", "group": "xyproto/algernon", "label": 1, "pair": "3844ed3d703d7ef4", "unit": "8ac27b08549353ada819caaf99a5180d2f2f515b03aa8cd99a5d37b527ea3a5c"}
{"family": "path", "fold": "outer-3", "group": "yiisoft/yii2", "label": 1, "pair": "a63eae7d621a985e", "unit": "04ca0b583b09c3b4f67bd2496a4607ecb134d24b7bdaf8960fa151e40ebcc447"}
{"family": "path", "fold": "outer-3", "group": "yiisoft/yii2", "label": 0, "pair": "a63eae7d621a985e", "unit": "ecfa2a67726ca9d3e2621a4d8be60145095b76aa18ecf0268212341d63752871"}
{"family": "deserialization", "fold": "outer-4", "group": "zumba/json-serializer", "label": 0, "pair": "0d9f529b7f0084b6", "unit": "35aa369236f510634b3aa7b54b04e9da0f80877e515786a888e760f8fe33c524"}
{"family": "deserialization", "fold": "outer-4", "group": "zumba/json-serializer", "label": 1, "pair": "0d9f529b7f0084b6", "unit": "ef4f8613c0914eb815504e00cd6ee05df38352fd3b27826798e6623594017042"}
@@ -0,0 +1,65 @@
# Experiment exp-003 (learned-decision-engine Req 6, Req 7.3/7.4): the BLOCK rule. Registered before it runs;
# `ousast learn experiment register` records this file's sha256, and a changed file is a new experiment.
# kind: rule -- both arms decide from the SAME scores: one scoring pass of the six slice-2 incumbent programs, replayed
# from the response cache only (zero cost by construction: a zero-ceiling meter around a refusing client; a unit with any
# uncached response is excluded and counted, never asked; the meter's calls and usd must read 0 or the run is void).
#
# Threshold selection (arm B), fixed here: per family and per outer fold j, the candidate thresholds are the raw scores
# s of the training rows, i.e. the family's paired units in the OTHER four outer folds (each of those predictions is
# itself out of repository: retrieval and demonstrations never cross its own fold's boundary). The chosen threshold is
# the lowest t such that, among those training rows with s >= t whose majority verdict is not `unsure`, the Wilson 95%
# lower bound of precision is >= 0.95. No such t -> fold j gets no BLOCK. Fold j's units are then flagged when not
# `unsure` and s >= t. Fold j's labels never enter its own threshold (tests/test_learn_rules.py).
#
# Pre-registered expectation (power): a Wilson lower bound of 0.95 needs at least 73 flagged units with no false flag.
# The frozen units (353 in 146 repository groups, 244 paired) hold 38 paired positives for injection and 16-19 for each
# other family, all five folds together; the training rows of any one fold hold fewer. So arm B cannot offer a BLOCK in
# any family by construction, and the expected verdict is inconclusive with arm B flagging nothing. That is the finding
# this run would confirm: on this data the bound, not the calibration gate, is what makes BLOCK unreachable. Arm C (point precision >= 0.95 over >= 10 training flags) is a secondary,
# non-deciding arm that measures how far the held-out precision is from the bar when the bound is relaxed.
id: exp-003-precision-bound-blocking
hypothesis: >
The calibrated-probability BLOCK rule (Platt map + calibration gate, ECE <= 0.05) is the wrong instrument for M4,
which asks for precision on what is flagged. Selective blocking -- a per-family raw-score threshold chosen on training
folds only as the lowest one whose Wilson 95% lower bound on flagged precision is >= 0.95 -- reaches the M4 precision
bar on held-out repositories where the calibrated rule offers no BLOCK.
kind: rule
programs: # the slice-2 incumbents (benchmarks/measurements/2026-10-02-decision-engine-slice-2, program.id)
access_control: 745cb1d76bbb09dc1574d333d2118489be62e39886517894d3699fa565b0e8c2
deserialization: 3c0c2831be7d78198a62351661c37867363c501cb40e3ed7b0119c155af09654
injection: b207a0710e0abf053b3b9397b25f9ee1cab4679144865e3a82c1b9be2c7b4982
output_encoding: 5d768f16724562c381c09949ff3df9cbb11d58cbb7a9c0515d0dc3f858d3818e
path: aea396680a4874dcb6de70e240ca0480c83424debdd3710bc3f1ea6a612661f3
untrusted_destination: 1242b3973fc89056aa08e00843043d68235df217d8e3ef8a0ea0e0b81c142753
arms:
A:
label: today's rule -- Platt map cross-fitted per family, BLOCK only when the calibration check holds, BLOCK point nested on the other folds' calibrated p
rule: {name: calibrated_block}
B:
label: selective blocking -- per-family raw-score threshold from the other folds' paired units, lowest t with Wilson 95% LB of flagged precision >= 0.95
rule: {name: precision_bound_block, bound: wilson, precision: 0.95, select_on: paired}
C:
label: secondary, non-deciding -- as B with point precision >= 0.95 over at least 10 training flags
rule: {name: precision_bound_block, bound: point, precision: 0.95, min_flagged: 10, select_on: paired}
unit: candidate
pairing: paired # every unit scored once; every arm decides every unit
cluster: repository
order: {randomise: none, seed: 3}
# primary population: paired units (both sides of the candidate present). It excludes every positives-only group and any
# unpaired unit, whose positives never had to be told apart from their own fix. Pooled (every unit) is reported beside it.
populations: {primary: paired, secondary: [pooled]}
metric:
primary:
- {name: flagged_precision_lb, target: 0.95} # held-out (outer-fold) Wilson 95% LB of flagged precision, per family, arm B, paired
secondary: [flagged_precision, coverage, flagged] # coverage = flagged true positives / positives
# Decision rule (decide_rule): REJECT B if in any family B flags held-out paired units at a point precision < 0.95;
# else ADOPT B if in some family B's held-out paired Wilson LB >= 0.95; else INCONCLUSIVE (the incumbent rule stays).
# A family "reaches the M4 precision bar on held-out repositories" when its held-out paired Wilson LB >= 0.95 (the
# point precision is reported beside it).
decision: {arm: B, target_precision: 0.95}
test: {resamples: 2000} # repository-cluster bootstrap of the held-out flagged precision, beside the Wilson bound
mde: 0.0 # not a paired difference test; see the power note above
budget_usd: {total: 0} # zero cost is a hard rule
families: [access_control, deserialization, injection, output_encoding, path, untrusted_destination]
folds: {kind: outer, k: 5, seed: 0, compile_fraction: 0.25} # the incumbent evaluation's folds (learn/compile.toml seed)
units_file: plane/experiments/exp-003-precision-bound-blocking.units.jsonl # frozen by `ousast learn experiment units`
@@ -0,0 +1,501 @@
{"family": "untrusted_destination", "fold": "outer-3", "group": "adonisjs/http-server", "label": 0, "pair": "6b0b3cab2df3397b", "unit": "0ac37b7154242dfa637140af643fb4595e85510a17ab1a12c111441b9d293238"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "adonisjs/http-server", "label": 1, "pair": "6b0b3cab2df3397b", "unit": "6e5e274946d0e8f13a7ef4e37f6c7ed1828d057304a9262db0311413f18e1731"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "agentfront/frontmcp", "label": 0, "pair": "c0d0c91757f5ba95", "unit": "285891b16ee6870af50b36dec3c2ca9a90ee15d9e14851c61e66ad443e6eab59"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "agentfront/frontmcp", "label": 1, "pair": "c0d0c91757f5ba95", "unit": "f33fb2413edf1499a524c4592b3b2f93d166be845f5b65291969a086a4bec334"}
{"family": "path", "fold": "outer-2", "group": "ahujasid/blender-mcp", "label": 1, "pair": "f56f2a107ae591be", "unit": "350149bf9cafbb547f500b714a164734d2361b22e86fc9819c7bad3507d8458a"}
{"family": "path", "fold": "outer-2", "group": "ahujasid/blender-mcp", "label": 0, "pair": "f56f2a107ae591be", "unit": "ad125319dd2d4c9d2d99a1eb34bb4cfab9f099a96e7547d0d59a8ecf1c990347"}
{"family": "deserialization", "fold": "outer-3", "group": "aio-libs/aiohttp", "label": 1, "pair": "ee577ee4b6076fab", "unit": "2523f55d035d1a772a842090bc335e5d755b2fe15b376dd5b6b9e30b9dae82d1"}
{"family": "deserialization", "fold": "outer-3", "group": "aio-libs/aiohttp", "label": 0, "pair": "ee577ee4b6076fab", "unit": "ca0f6c160d7839204f6fcc1709bcbc749a02a0b3b8bb648e2a238eacbf22ed78"}
{"family": "access_control", "fold": "outer-4", "group": "alchemycms/alchemy_cms", "label": 1, "pair": "c95b6b01d0c2db2a", "unit": "15540c23be0c42472a49039658ec2ec0bbd8c6b318e35dd040a9d427370e0ef0"}
{"family": "access_control", "fold": "outer-4", "group": "alchemycms/alchemy_cms", "label": 0, "pair": "c95b6b01d0c2db2a", "unit": "542471980417e964182d7d3db59a05fe12bdd072449f3bd77f7daa5515dbc909"}
{"family": "output_encoding", "fold": "outer-0", "group": "angular/angular", "label": 0, "pair": "29252934ef4bcfcf", "unit": "63bae9240b31f9cf1dfe00d9b01d4d0ffda95931c9e7d040df1fd718f04d828b"}
{"family": "output_encoding", "fold": "outer-0", "group": "angular/angular", "label": 1, "pair": "29252934ef4bcfcf", "unit": "cce2af1743834dcc3df58827ac263a2d63296ca1b2a2f9bdb811954fb5767c33"}
{"family": "access_control", "fold": "outer-3", "group": "anotherik/threatbyte", "label": 1, "pair": "", "unit": "bcc636b23dfb5f570d8864b17fe07b6b4fb52b65a40988c32011f1422f74d504"}
{"family": "access_control", "fold": "outer-3", "group": "anotherik/threatbyte", "label": 1, "pair": "", "unit": "dbcf1e67a1be7ccf7d51701f9b8339339164055135443044c23a446b16b3d934"}
{"family": "injection", "fold": "outer-3", "group": "anotherik/threatbyte", "label": 1, "pair": "", "unit": "50408f68114ecb4de3bebd836d52d9afad86b22dd1843641ac8a49b617c09e33"}
{"family": "injection", "fold": "outer-3", "group": "anxolerd/dvpwa", "label": 1, "pair": "", "unit": "cafb0f00f80dc6beeba40eb95c764bf9a3569193070ba321c28677dfa425ec88"}
{"family": "deserialization", "fold": "outer-2", "group": "apache/airflow", "label": 0, "pair": "07317a2258292656", "unit": "a9f292d7cb467def582d9c0b31a397b13cdaefa77380ce971bfe9fbf535bb660"}
{"family": "deserialization", "fold": "outer-2", "group": "apache/airflow", "label": 1, "pair": "07317a2258292656", "unit": "ed363d0dfa620707be477b90eb4b83a901bfd402040b918aeee6f06b2c5b795c"}
{"family": "deserialization", "fold": "outer-3", "group": "apache/inlong", "label": 1, "pair": "1544b6dfa2299d60", "unit": "d45018fc4ce3dfd715c7087d4d197ad4108c0c8b6625c02c4d4da0adb9f6a140"}
{"family": "deserialization", "fold": "outer-3", "group": "apache/inlong", "label": 0, "pair": "1544b6dfa2299d60", "unit": "f295c7dfb095595a4aa985c58688e4757c29be9e06b7a4ca55035ac5d06b1ef8"}
{"family": "output_encoding", "fold": "outer-4", "group": "apache/logging-log4net", "label": 1, "pair": "fe0e104d111c2555", "unit": "088036fab9962be2b96f9162ea4f32190575678d3c5ba96bdfeac178bc22b2c4"}
{"family": "output_encoding", "fold": "outer-4", "group": "apache/logging-log4net", "label": 0, "pair": "fe0e104d111c2555", "unit": "7fd2b56a816e25358477eee64487fb2c77f34c652703ad0a7120c368492017a3"}
{"family": "path", "fold": "outer-2", "group": "apache/wicket", "label": 1, "pair": "b6742570f7f0e026", "unit": "7c6c1d27aee6557f1b6bf371b8209003f0d2d1122cfa35b6629eb43f38769cf4"}
{"family": "path", "fold": "outer-2", "group": "apache/wicket", "label": 0, "pair": "b6742570f7f0e026", "unit": "df8300d84dd784d0be46969ea432ef97b8ce611bae7a896ed4f349bd4d6897db"}
{"family": "access_control", "fold": "outer-4", "group": "apolloconfig/apollo", "label": 1, "pair": "0f584bb97a154488", "unit": "222a84ffc7c58d45e8f10df221762b23bf20d0287ecd20f375558134928886ca"}
{"family": "access_control", "fold": "outer-4", "group": "apolloconfig/apollo", "label": 0, "pair": "0f584bb97a154488", "unit": "25e4c9d2941f59f072385a7bde3fd0bd76f9e986ed24b06ce81962ab447f5094"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "appium/java-client", "label": 0, "pair": "b15b924b4f7b1ae1", "unit": "2fa95fce0751c97db1a12a80ddb249ff5c54ee31c7dc3cd47765099c7b673e52"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "appium/java-client", "label": 1, "pair": "b15b924b4f7b1ae1", "unit": "52be8128053102eafbee67c2d3aac88c83ca290b943ab2f2cfdcd2630601c7d9"}
{"family": "access_control", "fold": "outer-3", "group": "arcadedata/arcadedb", "label": 0, "pair": "d31ca58f2a1d27d1", "unit": "2ae40ea61d691060e7b9f6c8e910e3aa0edf16bc6f88bec81aa9a7dae5cf5603"}
{"family": "access_control", "fold": "outer-3", "group": "arcadedata/arcadedb", "label": 1, "pair": "d31ca58f2a1d27d1", "unit": "4876728701698045e65eeae6e11e133c30f43b09029d7777d6d780f7d0855885"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "archivy/archivy", "label": 0, "pair": "bb3d7baed62e6bb7", "unit": "e8ae88fd2b677e3f64ac94bddb012e3b903afa294ecc87774d10087c0d50489c"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "archivy/archivy", "label": 1, "pair": "bb3d7baed62e6bb7", "unit": "f6b0d241891dc1d897d74d98b6ff7ada2c1f1ce7874ca32097630c06ca53e389"}
{"family": "injection", "fold": "outer-1", "group": "argos-ci/argos-javascript", "label": 0, "pair": "d9d8cfa38981847f", "unit": "864f1800617f5218a2300ce390ba7922fe1bd0a182de115952342b61d627bb16"}
{"family": "injection", "fold": "outer-1", "group": "argos-ci/argos-javascript", "label": 1, "pair": "d9d8cfa38981847f", "unit": "fdcda2307e20ef1b9247b7082a85337f457dd1f4e648bb8264077bf983c03271"}
{"family": "access_control", "fold": "outer-2", "group": "arikusi/deepseek-mcp-server", "label": 0, "pair": "1f73ad2576d52a64", "unit": "0bc9634940734cf7f41628f220032fb7e2fd2ff5b584afc519875e90ea0be5a4"}
{"family": "access_control", "fold": "outer-2", "group": "arikusi/deepseek-mcp-server", "label": 1, "pair": "1f73ad2576d52a64", "unit": "ddb59b2dc39715281a6d48b32e7ca4ec0354b85b953fff55139579f83905b7d6"}
{"family": "access_control", "fold": "outer-0", "group": "arthurkatcher/google-maps-mcp", "label": 1, "pair": "a7326e6daa507b79", "unit": "793ebf4c8e0f46d8ce3b3d090d2f6290d4167931a3b574f8269b90008e3dcde7"}
{"family": "access_control", "fold": "outer-0", "group": "arthurkatcher/google-maps-mcp", "label": 0, "pair": "a7326e6daa507b79", "unit": "b8d25f337fad527544c33c82f473015e08399457276fc480477f3cfcfc9c5b8a"}
{"family": "path", "fold": "outer-1", "group": "asymmetric-effort/nogginlessdom", "label": 0, "pair": "211ba072bdfa67b7", "unit": "3217f9d672a4a1d78005c8fb66e1bc60a542cb26bd8c46903402c765c123d60a"}
{"family": "path", "fold": "outer-1", "group": "asymmetric-effort/nogginlessdom", "label": 1, "pair": "211ba072bdfa67b7", "unit": "8370f9624baa0d144e56b8087eea897b98e5b2afcd6eb3a029d03e5b4b69d584"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "authlib/authlib", "label": 1, "pair": "baccf350d5cc9911", "unit": "50a3ca70700c8b41874c43ba94e6b7ffe0aa28b8930ff941a40767e1057065b5"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "authlib/authlib", "label": 0, "pair": "baccf350d5cc9911", "unit": "62927a118d3662c81450e19d49f4146b66f29a58d55ec0d1606ed521b1883f12"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "axios/axios", "label": 1, "pair": "697f34e19bda57f3", "unit": "775b6d900ec1ba71e4082ed49bb94439f38066628eed91461eecf0658caee78a"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "axios/axios", "label": 0, "pair": "697f34e19bda57f3", "unit": "9e855586c685b3c62ffe32407a82966661d5839e2cff742f2676acaba645a97a"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "axllent/mailpit", "label": 1, "pair": "143b900b99a46cf4", "unit": "200f96cd784ff5c39d8bba22883f22bc8a7c87c6970a7c871802f48a45548337"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "axllent/mailpit", "label": 0, "pair": "143b900b99a46cf4", "unit": "7a0d62ec424849ec4681f4333fbb077e8239054d1e964bd99b40847ad7ccdb01"}
{"family": "output_encoding", "fold": "outer-3", "group": "basecamp/trix", "label": 1, "pair": "4b4e81797ca48cbf", "unit": "a4d17d85973d024a615b89334ad5a2f46afea660ef5d0bcf2b2196c998b1caf2"}
{"family": "output_encoding", "fold": "outer-3", "group": "basecamp/trix", "label": 0, "pair": "4b4e81797ca48cbf", "unit": "d604e460d8f5291124ba6fab5f9c94c7dfda61ea5f6808e512162334d6d6db4f"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "beaugunderson/ip-address", "label": 1, "pair": "bcbd06e53147628b", "unit": "0b5cfe2d64771e30e50ff92cc409b8dd5e30079c252918ba03774cb5198a73b5"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "beaugunderson/ip-address", "label": 0, "pair": "bcbd06e53147628b", "unit": "b805976c35330452af6c9f814ff4617b7b9ad0e82e1fa4440a443f93f0772c4a"}
{"family": "injection", "fold": "outer-3", "group": "browserify/static-eval", "label": 1, "pair": "c12dbc8a50f7fe9b", "unit": "04beb8a0fd054b92c2dd0ed83907330fdee6124d86114df948ac72a297a7d51f"}
{"family": "injection", "fold": "outer-3", "group": "browserify/static-eval", "label": 0, "pair": "c12dbc8a50f7fe9b", "unit": "c5029016a6fd2e3f9df59dba0ce3c173bbbb0d2b1de22d151086e01e1b272119"}
{"family": "injection", "fold": "outer-4", "group": "budibase/budibase", "label": 0, "pair": "", "unit": "0e213aeabacf4205db024ea3c557e720c3aba911424555784917e967d6404867"}
{"family": "injection", "fold": "outer-4", "group": "budibase/budibase", "label": 0, "pair": "e8bc65415bd42e43", "unit": "4920230ef773595a9fd9a92a090dff42b48499a1dc5fc8831f941dae55cb2538"}
{"family": "injection", "fold": "outer-4", "group": "budibase/budibase", "label": 1, "pair": "e8bc65415bd42e43", "unit": "49209a810f207eb61382e2ceeab7076936838a5652546f21c187d066e220160b"}
{"family": "injection", "fold": "outer-4", "group": "budibase/budibase", "label": 0, "pair": "", "unit": "9e2964884cdf281fa3f18ee3481aa62f9a8a0a195d605da2df5bd71876fe311a"}
{"family": "injection", "fold": "outer-4", "group": "budibase/budibase", "label": 1, "pair": "", "unit": "f9cc6f1bbcd59f5dc0de8f96248f722ea0679f8afa47c53f70832151ce7616d3"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "cakephp/authentication", "label": 0, "pair": "e69463f945f9a0f4", "unit": "5feace7b28e260175f80c682e9009025aea6fae3dc75409cabbb955b560d0372"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "cakephp/authentication", "label": 1, "pair": "e69463f945f9a0f4", "unit": "799a4f5a5b9324e86c90752f0239f059984f44e8ff1ec0dd518d0d9413e34c66"}
{"family": "injection", "fold": "outer-3", "group": "cakephp/cakephp", "label": 0, "pair": "71f56810b8ee6947", "unit": "c659c0e717b7dc1084e1178288f6c9b2e87fbf0ef518ba3bbaedb0fbadae15c1"}
{"family": "injection", "fold": "outer-3", "group": "cakephp/cakephp", "label": 1, "pair": "71f56810b8ee6947", "unit": "c8d2e035a6099fb775c8b4ecf00196c1488dbddd9eed3d2d268a4f1d2567b194"}
{"family": "deserialization", "fold": "outer-1", "group": "cesargb/laravel-magiclink", "label": 0, "pair": "768fc9db8a75a351", "unit": "adc6527469618f72b1dd7776814717cbd0056a0c625e05bef74199171ce73f1a"}
{"family": "deserialization", "fold": "outer-1", "group": "cesargb/laravel-magiclink", "label": 1, "pair": "768fc9db8a75a351", "unit": "da47663a57e370a75cc65b810b8a380aa26d12dcabb1d9b3c02641398ff7d348"}
{"family": "path", "fold": "outer-0", "group": "christopy/serve-here.js", "label": 0, "pair": "fab4ce020049724b", "unit": "b9be90242be627aa46462143ac85940a906232ebc0b47776b5729475b27d215e"}
{"family": "path", "fold": "outer-0", "group": "christopy/serve-here.js", "label": 1, "pair": "fab4ce020049724b", "unit": "ece12ea7d87b4a12c4da30faed9eb530b51fc879c271b68ba5304b230cda8e79"}
{"family": "output_encoding", "fold": "outer-1", "group": "chrome-php/chrome", "label": 1, "pair": "9126ba83905eb8b4", "unit": "d8dc50b7bb40bbcfc3f3d4f1f8d3a4d17eb1c662d5f759efd18656a482dd2e32"}
{"family": "output_encoding", "fold": "outer-1", "group": "chrome-php/chrome", "label": 0, "pair": "9126ba83905eb8b4", "unit": "f426f9307621b3f2d6a8d1669cdb4c19e962b94e34fbb394d4660cbde5921c88"}
{"family": "deserialization", "fold": "outer-1", "group": "commenthol/serialize-to-js", "label": 0, "pair": "7ac5e21d9d8f4c47", "unit": "1d1efa2fd7323c264290d4d314c957c853e1c7a9c877a54d6f03ab7d2af2ca86"}
{"family": "deserialization", "fold": "outer-1", "group": "commenthol/serialize-to-js", "label": 1, "pair": "7ac5e21d9d8f4c47", "unit": "b5dc78c7ac0b76803dc98904142c1139083fb9daafb5bff40e7b5c3f14ee03f5"}
{"family": "deserialization", "fold": "outer-1", "group": "contrast-security-oss/djangoat", "label": 1, "pair": "", "unit": "0894a7212280bab755cd9d93b1f0ede2c51e4fca76b76e5ed9e24ef8f8cf0b9c"}
{"family": "injection", "fold": "outer-1", "group": "contrast-security-oss/djangoat", "label": 1, "pair": "", "unit": "02694b5fb85e9e52bf5eaaab4e0b628f1f44ff3a8d5ad026cc10afdbc4222eaf"}
{"family": "injection", "fold": "outer-1", "group": "contrast-security-oss/djangoat", "label": 1, "pair": "", "unit": "db9a3e888780501dd3c2c125af2f9d47f68ebf02f0e0db8370b2ea09c6f9d070"}
{"family": "injection", "fold": "outer-2", "group": "contrast-security-oss/vulnpy", "label": 1, "pair": "", "unit": "9af96b099dedcc4005e5fcd19ad78f73864e1cfbafbe6953b4edb6260957943d"}
{"family": "injection", "fold": "outer-2", "group": "contrast-security-oss/vulnpy", "label": 1, "pair": "", "unit": "9e7c2a786068a936a6c664cc229d6f39cea1fe57caf0806306f0cfc44515460c"}
{"family": "deserialization", "fold": "outer-3", "group": "cps-it/mailqueue", "label": 1, "pair": "07d1dc4d299a7700", "unit": "0e664e274dfb438d2f721edc81f8469ac48387dbd5c1b43973cd7e651db46db3"}
{"family": "deserialization", "fold": "outer-3", "group": "cps-it/mailqueue", "label": 0, "pair": "07d1dc4d299a7700", "unit": "da55f60378c67e91683b9720f2fa0e6659e28c7b392c050278b40713e47c84a3"}
{"family": "output_encoding", "fold": "outer-3", "group": "cstigler/node-xhtml-purifier", "label": 1, "pair": "749eee05a507fe93", "unit": "24fbb1dffcca083382a23e48db232e0d83bbbf2ff2b325f0dc0f4c72dd1a16bc"}
{"family": "output_encoding", "fold": "outer-3", "group": "cstigler/node-xhtml-purifier", "label": 0, "pair": "749eee05a507fe93", "unit": "640e1626f8d7d9d5c2478ede48cc7b2780370bda98ae114bae102bba892da90c"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "czlonkowski/n8n-mcp", "label": 0, "pair": "e36bbaf8f3cfe65c", "unit": "14f7c65fb5621ac5e2068c8695ddbf482103e4320ec3b36886f6e1f08cbc312d"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "czlonkowski/n8n-mcp", "label": 1, "pair": "e36bbaf8f3cfe65c", "unit": "c96dd379dec2ecfd99051d7a20e6ef5601821f93c46eb34ed087daf5fbed2d47"}
{"family": "injection", "fold": "outer-4", "group": "danger/danger-js", "label": 0, "pair": "b81c8c63cae892f6", "unit": "5982fe7fa1ea6bfc1c8dcb8c71cd4410de475d25cafa324147649f26a1ae1324"}
{"family": "injection", "fold": "outer-4", "group": "danger/danger-js", "label": 1, "pair": "b81c8c63cae892f6", "unit": "59916deeb8ba14922eebac61c9eac122e15d1f7590ee3cb1b97f019585fb798f"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "danielgatis/rembg", "label": 1, "pair": "bab3a167ab5ca505", "unit": "5ebea23aee96c80fa0d2d9239964d99614613c4ae36f709811bc19e61ebc720c"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "danielgatis/rembg", "label": 0, "pair": "bab3a167ab5ca505", "unit": "9bfea8d7a6ffa6d79d16f77fc355fc1bb605189513ebad561ee523d577a1498b"}
{"family": "deserialization", "fold": "outer-3", "group": "datapizza-labs/datapizza-ai", "label": 1, "pair": "5a91049be21aa611", "unit": "1ab9936627aa59c3a60ea44074ad44b68aea4f9c04f14c1949dd9c77ca13ff0a"}
{"family": "deserialization", "fold": "outer-3", "group": "datapizza-labs/datapizza-ai", "label": 0, "pair": "5a91049be21aa611", "unit": "8df72b9eab3f2fd858c40312d0a806cb67c620bd62873826c8732ba8795d9cfa"}
{"family": "injection", "fold": "outer-3", "group": "davila7/claude-code-templates", "label": 0, "pair": "", "unit": "115d87f37bf1d4ab91fa3010303e81b127f84b817268b60b8fd4b9934efeae4f"}
{"family": "injection", "fold": "outer-3", "group": "davila7/claude-code-templates", "label": 0, "pair": "2b8101c89c839614", "unit": "7bb7d858a670fc51f81e08c141c2bca3b8b7524254ab679e33b22ce79536e895"}
{"family": "injection", "fold": "outer-3", "group": "davila7/claude-code-templates", "label": 0, "pair": "c23acf4f38945093", "unit": "82bdc0b6bb0c19e342efa63af0c9014a607461d0682cdd14bd5fea33516fbd50"}
{"family": "injection", "fold": "outer-3", "group": "davila7/claude-code-templates", "label": 1, "pair": "2b8101c89c839614", "unit": "886285661b88f14e00ff8e863b31cf7b7afb303ad4aa4aa13b1cffbf1272d5e0"}
{"family": "injection", "fold": "outer-3", "group": "davila7/claude-code-templates", "label": 1, "pair": "c23acf4f38945093", "unit": "bf74aae5a48156f183426a80cafca6da219ab13b2ae4b40fa88fca75acb72980"}
{"family": "access_control", "fold": "outer-1", "group": "delmaredigital/payload-puck", "label": 0, "pair": "d7cc37a3a54315cb", "unit": "4aecd43c6e5b81daf7aadd6a6e22f22c17c08b652a2f5121a49320ca1057b186"}
{"family": "access_control", "fold": "outer-1", "group": "delmaredigital/payload-puck", "label": 1, "pair": "d7cc37a3a54315cb", "unit": "88123ae860be2fdc28fd6a289cf26c3e0af0689f455cf8c15098e5dc76e5c8a8"}
{"family": "path", "fold": "outer-4", "group": "dep0we/atomic-agents-stack", "label": 1, "pair": "5011004f06f1e217", "unit": "6c48682d1b02ff2559d92346d192cc9dbe6a0bbe2ab2c146653110420105ff1d"}
{"family": "path", "fold": "outer-4", "group": "dep0we/atomic-agents-stack", "label": 0, "pair": "5011004f06f1e217", "unit": "d3592260108a838246e0cfd5ab6d48bceb0ef870e223565745dcfd68f1826840"}
{"family": "output_encoding", "fold": "outer-1", "group": "devbridge/jquery-autocomplete", "label": 0, "pair": "81caad96b221c2b8", "unit": "3c2dc720ed159e55629ba737665d93859def9790427289567a513c83c133e236"}
{"family": "output_encoding", "fold": "outer-1", "group": "devbridge/jquery-autocomplete", "label": 1, "pair": "81caad96b221c2b8", "unit": "e8b9595a320776c8e663fd0b168193c1e64e36a9052784d1219f2c4a91d647f2"}
{"family": "path", "fold": "outer-4", "group": "dgtlmoon/changedetection.io", "label": 0, "pair": "f9b6599018ef3b60", "unit": "7558d6ee39a311c5459e35fe6f7b12917b7f843b21ab1fbb5e485a07f53b7d78"}
{"family": "path", "fold": "outer-4", "group": "dgtlmoon/changedetection.io", "label": 1, "pair": "f9b6599018ef3b60", "unit": "b1a6cf450a074788ffca2bebd880435abac2c376e89cb328d7a22d277fdccad8"}
{"family": "output_encoding", "fold": "outer-4", "group": "dicebear/dicebear", "label": 1, "pair": "eaf2f03db2c502b5", "unit": "38dbfbf745d7cbcc52c6fff83248d84663273fe6a1f1b014eac7776c4bb8f074"}
{"family": "output_encoding", "fold": "outer-4", "group": "dicebear/dicebear", "label": 0, "pair": "eaf2f03db2c502b5", "unit": "a84d848ef56b72fc15872d88acaae1743c9cdf88095d13b1869abbec6a0d8a8f"}
{"family": "deserialization", "fold": "outer-0", "group": "docling-project/docling-core", "label": 0, "pair": "7df48bca2c9fabfc", "unit": "7b76ce8f55d61071adb367018cc3e50f04a05d08c7c4919c3806fc1cc8bf41a7"}
{"family": "deserialization", "fold": "outer-0", "group": "docling-project/docling-core", "label": 1, "pair": "7df48bca2c9fabfc", "unit": "c82ea0adc9c8366aa2b77ce73772038676fc37ff50879e873ff397958fdd7ec1"}
{"family": "injection", "fold": "outer-4", "group": "dolevf/damn-vulnerable-graphql-application", "label": 1, "pair": "", "unit": "7f02a09b8d8f2321f2cdade74c2fa1d52442e006149a749a8be768daaa62195d"}
{"family": "injection", "fold": "outer-4", "group": "dolevf/damn-vulnerable-graphql-application", "label": 1, "pair": "", "unit": "9b5e439943ce5c2b04d6201b7a843634aa1b13b726813c23450ff54ce03f3dd6"}
{"family": "injection", "fold": "outer-4", "group": "dolevf/damn-vulnerable-graphql-application", "label": 1, "pair": "5b836889b4da4921", "unit": "a84c2ce4a4eadf265d4481ad7f6864edf1d7e3084c6ee32cbbfe454bb2946bfe"}
{"family": "injection", "fold": "outer-4", "group": "dolevf/damn-vulnerable-graphql-application", "label": 0, "pair": "5b836889b4da4921", "unit": "caeed7f097a3311fe2e84efd372fe4b4c140f63a80a9f07ba0b61966dbab15e0"}
{"family": "injection", "fold": "outer-4", "group": "electerm/electerm", "label": 0, "pair": "ac526f457067fad8", "unit": "50d8bafd5189fab0429bd4135d9ed400cc2fc14706226a2de98d65746ebfdcc7"}
{"family": "injection", "fold": "outer-4", "group": "electerm/electerm", "label": 1, "pair": "ac526f457067fad8", "unit": "9532b70b0df76a34170b57c74deafaba54d5d2f16a729c39b0dfd3eb2022d899"}
{"family": "path", "fold": "outer-1", "group": "elyin/gemini-bridge", "label": 1, "pair": "c97a971a18f85244", "unit": "95c80c503ed43fe9a0370f89ea935d47badee6a9625548b9dca78e04275003dd"}
{"family": "path", "fold": "outer-1", "group": "elyin/gemini-bridge", "label": 0, "pair": "c97a971a18f85244", "unit": "e9810872beb9c158f54048392142c2f701314f854d6dfc748553ca36ed6c08d0"}
{"family": "access_control", "fold": "outer-4", "group": "erev0s/vampi", "label": 1, "pair": "", "unit": "c2a2c2d85114a97f892fb1e5500fbd98622457a8c46451e1f9e958ab08a40318"}
{"family": "injection", "fold": "outer-4", "group": "erev0s/vampi", "label": 1, "pair": "", "unit": "29d8be8206c88ba7efd86b56a7dcf8b655e33b4a792c2d18b028c92c6252daab"}
{"family": "access_control", "fold": "outer-2", "group": "esphome/device-builder", "label": 0, "pair": "6a4e0bd705331747", "unit": "6db80439d89b54f60f836c415dd403fd09d21cf630da81435d4d2f9cef5d883c"}
{"family": "access_control", "fold": "outer-2", "group": "esphome/device-builder", "label": 1, "pair": "6a4e0bd705331747", "unit": "bf257e4f0812012be330fe119d07d29253d291161f435262787f491762c6bfb1"}
{"family": "output_encoding", "fold": "outer-2", "group": "etaminstudio/prosemirror_to_html", "label": 0, "pair": "f96f0fb245948915", "unit": "0954e6931373915c204b9ec8d2b8ae6297a95667bfd41aadf9035625ad93967f"}
{"family": "output_encoding", "fold": "outer-2", "group": "etaminstudio/prosemirror_to_html", "label": 1, "pair": "f96f0fb245948915", "unit": "5e09cdae2504e834d591b6f92a808427f07550364b5419899f7e6f76084fc377"}
{"family": "output_encoding", "fold": "outer-0", "group": "ether/etherpad", "label": 0, "pair": "bf11f138fe39675e", "unit": "56f0d0b156992af56141d6368fdb0c5376fc7a44a03991d754f97202b9d77008"}
{"family": "output_encoding", "fold": "outer-0", "group": "ether/etherpad", "label": 1, "pair": "bf11f138fe39675e", "unit": "8271af5017894167556893da2070188060d06eebc87f27db45c46234c359631d"}
{"family": "path", "fold": "outer-0", "group": "fastify/fastify-static", "label": 0, "pair": "9d937c2dffb4aadc", "unit": "01ac12bb06e660d0aeef8b1e26dddc5c99d52cc333f7af6d1981eee2818e64fd"}
{"family": "path", "fold": "outer-0", "group": "fastify/fastify-static", "label": 1, "pair": "9d937c2dffb4aadc", "unit": "9612edf0f927048335e83e9925f01be9aea9b772bdbdc568ad9cc0a17055662a"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "feathersjs/feathers", "label": 1, "pair": "d4f648048e45f2ea", "unit": "08807c8110b8c1b8635a6883c229fb91f2b9566db8b6c252448a0db7b2ddd47e"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "feathersjs/feathers", "label": 0, "pair": "d4f648048e45f2ea", "unit": "10869c931053bf77ecaa98a47f56dea27d643d1abed2a4758bcf48a78547ccc9"}
{"family": "injection", "fold": "outer-1", "group": "fecommunity/reactpress", "label": 0, "pair": "662e1f36c230d7f1", "unit": "0b73332b25413902c26d5b5dee6767465eed99ff71a814582988ffe0d8ec1a30"}
{"family": "injection", "fold": "outer-1", "group": "fecommunity/reactpress", "label": 0, "pair": "", "unit": "8c8e46d0c7d361b784baff9d70f38744b2805ef1ee0aaacaa616bc20a2c6f32d"}
{"family": "injection", "fold": "outer-1", "group": "fecommunity/reactpress", "label": 0, "pair": "", "unit": "aca938682c196fd4c85b39b89eeedb97c35f6a5a017651fe9c03c5b727cab7ea"}
{"family": "injection", "fold": "outer-1", "group": "fecommunity/reactpress", "label": 0, "pair": "", "unit": "cec00d1509a129364da6ab8537fb3f433fd847e26d073c90311ccdaf7a1b39e7"}
{"family": "injection", "fold": "outer-1", "group": "fecommunity/reactpress", "label": 1, "pair": "662e1f36c230d7f1", "unit": "e0f58a7ac5627de3e7b6cfe5add41d18b0f64b998c1cc788d42eb2cdbbc3fe41"}
{"family": "access_control", "fold": "outer-0", "group": "filebrowser/filebrowser", "label": 0, "pair": "2c251a645094f389", "unit": "0d41cc634771048f72d07c4aa9feaed669d0bfe3844d61dcdc525a0f899617e2"}
{"family": "access_control", "fold": "outer-0", "group": "filebrowser/filebrowser", "label": 1, "pair": "2c251a645094f389", "unit": "ca8288d0b8b94dc6b7e4c681936ea610d295fe949cba5f64f09b6de134ef9875"}
{"family": "access_control", "fold": "outer-4", "group": "forgekeep/nebula-mesh", "label": 1, "pair": "2ac5ec5011615c9b", "unit": "2f427f403af3cb58d776437736c2389805543413fdb91aad98979158c6d00de3"}
{"family": "access_control", "fold": "outer-4", "group": "forgekeep/nebula-mesh", "label": 0, "pair": "2ac5ec5011615c9b", "unit": "cddc8c5bed0841b80e7b20d0b378e858f8e4ac37fd7b890ce76611d989857f92"}
{"family": "injection", "fold": "outer-3", "group": "frangoteam/fuxa", "label": 1, "pair": "", "unit": "0c7830c2912ff285855d8a3c8e9b47db4ca4af840baa66d4b5d3f4422f46f95d"}
{"family": "injection", "fold": "outer-4", "group": "froxlor/froxlor", "label": 0, "pair": "", "unit": "5383a8c2a78fabd3d17a5b3021125953b5ebbee61f8900c30d016c56b1a3bad6"}
{"family": "injection", "fold": "outer-4", "group": "froxlor/froxlor", "label": 0, "pair": "360d8bfd2f116e98", "unit": "7c1989a4ec3622bdaebcf1161de850f22cfed041cb38830ca1ded4fa99d40466"}
{"family": "injection", "fold": "outer-4", "group": "froxlor/froxlor", "label": 1, "pair": "ec8c706cc2b20f36", "unit": "848688dcd0fe8d46e6762cc8aee1d82e8fad8a2a044c94b547aea11049c85bfc"}
{"family": "injection", "fold": "outer-4", "group": "froxlor/froxlor", "label": 0, "pair": "ec8c706cc2b20f36", "unit": "983cc9231f91baee4e2d626193f3b88aa5b09907f6faab870949bdc9c336b816"}
{"family": "injection", "fold": "outer-4", "group": "froxlor/froxlor", "label": 0, "pair": "e4ad86416d5c6668", "unit": "98ff1dff2afe80709b8e61a9c95cfeea12b2110c13b7120478268b4762e2c992"}
{"family": "injection", "fold": "outer-4", "group": "froxlor/froxlor", "label": 1, "pair": "360d8bfd2f116e98", "unit": "a1a825d23e34757c8e961e43ae791253e62682e130acdb846b58118588006a98"}
{"family": "injection", "fold": "outer-4", "group": "froxlor/froxlor", "label": 0, "pair": "", "unit": "a7fa4207508b3989baabc7a1f0ff2ea5229b366c1cb106e7fc945134b8b106c8"}
{"family": "injection", "fold": "outer-4", "group": "froxlor/froxlor", "label": 1, "pair": "e4ad86416d5c6668", "unit": "ef5c53f3114d0057e694df13dc07cbcb532c3b0d59150eda269b87b021d7a6e8"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "getgrav/grav", "label": 1, "pair": "9ae575048504c690", "unit": "176acf077fd58c3e6f940753179ffa3e8316661f355d94acb5e0345b51925ca4"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "getgrav/grav", "label": 0, "pair": "9ae575048504c690", "unit": "228b34ebc2090b5239c7821d28366ddc663e7118d015a79b45dca6618d5490fb"}
{"family": "access_control", "fold": "outer-1", "group": "getgrav/grav-plugin-api", "label": 0, "pair": "d6edb5496b3fdba2", "unit": "9acb8017ebadc480515a664181c427f37359edcc1dc6f344f368a87e0c191b3f"}
{"family": "access_control", "fold": "outer-1", "group": "getgrav/grav-plugin-api", "label": 1, "pair": "d6edb5496b3fdba2", "unit": "c1bfe63148bc700c812fde95ad9f83d456e4fa04e10c6f5394072a08f73174c6"}
{"family": "injection", "fold": "outer-0", "group": "gitpython-developers/gitpython", "label": 0, "pair": "27e1c8e8ee93a892", "unit": "47fb89b132d6de48674c37965579e727d922bef1796db70f8b9f862ac47cb9a5"}
{"family": "injection", "fold": "outer-0", "group": "gitpython-developers/gitpython", "label": 1, "pair": "27e1c8e8ee93a892", "unit": "79887f0ca8b05d6599d55167d8a9faf67bdb2254d17c143141da44971077002d"}
{"family": "output_encoding", "fold": "outer-3", "group": "gofiber/template", "label": 0, "pair": "a3fdf9ce3441ec86", "unit": "653236018929ab8534c27704a20ec5d358047bc66e451620d2ac32081c91d333"}
{"family": "output_encoding", "fold": "outer-3", "group": "gofiber/template", "label": 1, "pair": "a3fdf9ce3441ec86", "unit": "b70ca2c43f5929e7b885bd728349e721fd85d7605e2f722a7e8cfe0f0d477b0c"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "groundhoggwp/groundhogg", "label": 1, "pair": "", "unit": "92400e83890882a66ffdea1a405e2354a195442e176d85fb707b0442c44d45ca"}
{"family": "path", "fold": "outer-4", "group": "gtsteffaniak/filebrowser", "label": 0, "pair": "f9dcf3a0fb0f0a14", "unit": "7ca6dd4f673976b000a657f885af3463740419ce41e14f972af0c5483cb75a7d"}
{"family": "path", "fold": "outer-4", "group": "gtsteffaniak/filebrowser", "label": 1, "pair": "f9dcf3a0fb0f0a14", "unit": "86c1db3a5e9f843d6546146449171e0378ac61d6c8f44c6dd72fd379cdbb188a"}
{"family": "deserialization", "fold": "outer-4", "group": "h2oai/h2o-3", "label": 1, "pair": "3e71f79348a6b770", "unit": "200250ebceadde4cbdfec2802a4f9fbfa220dbd7fa4c2dd5ed717fb1f3ea7447"}
{"family": "deserialization", "fold": "outer-4", "group": "h2oai/h2o-3", "label": 0, "pair": "3e71f79348a6b770", "unit": "b9faa5007c8cbb67ada3f1f026ca59e0ace6fbb9c3d1f1369d4521a7e323e7da"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "h3js/h3", "label": 0, "pair": "8972a0f56ee57477", "unit": "6b987df2960828dcb1b2b3f190a903a40361d58117d63321d671aecc6c2cb570"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "h3js/h3", "label": 1, "pair": "8972a0f56ee57477", "unit": "ca3fa87dd37435c01a7121119912c1a1e261259c11805c0b66e7d4b315640945"}
{"family": "injection", "fold": "outer-1", "group": "handsontable/formula-parser", "label": 0, "pair": "6144baeb491f4a56", "unit": "f5c22ce104e9593e5272b6e8b3c6f3b61fa4578552ccea80e6021ee90bdb3ffc"}
{"family": "injection", "fold": "outer-1", "group": "handsontable/formula-parser", "label": 1, "pair": "6144baeb491f4a56", "unit": "f687e7e95dbe47f602a1e0002a44f94fd48e5b1c33f77ecebc6c2d814fcbf450"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "heartcombo/devise", "label": 0, "pair": "2ef4471c66e5e4ab", "unit": "ad7c8e368fe1699a044e058d5c10b41bd9fc0f91c3f3c384c5dac40af39e71f6"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "heartcombo/devise", "label": 1, "pair": "2ef4471c66e5e4ab", "unit": "d888baf7194bbe7158cb3932f58b47a9a235351d388d10df82a130ddd54c2d46"}
{"family": "output_encoding", "fold": "outer-2", "group": "honojs/hono", "label": 1, "pair": "5628c54152216646", "unit": "47d081c9f6060218967902f694e61c88f64f57c65bec6fc3073afeb0b77d117c"}
{"family": "output_encoding", "fold": "outer-2", "group": "honojs/hono", "label": 0, "pair": "5628c54152216646", "unit": "e1297bccf9c3606e0b41d762a69ff5d0a33ae4f7f1506071af6b9cb4a177ff56"}
{"family": "path", "fold": "outer-4", "group": "hs-web/hsweb-framework", "label": 1, "pair": "e52aa814c36ace1f", "unit": "2d6d74364f1982e91f27f1c995b8699df1fc8f8dce6e1e2cf274bf01b6e9494e"}
{"family": "path", "fold": "outer-4", "group": "hs-web/hsweb-framework", "label": 0, "pair": "e52aa814c36ace1f", "unit": "4727471fea2cf0a92d43b4b1503f1339a7dde87f872ba8064dacbf855c57705b"}
{"family": "path", "fold": "outer-2", "group": "huggingface/datasets", "label": 1, "pair": "a07929b3fc1cfc28", "unit": "0637029194bd7378237b4dc944dc5cf9f2b63db9796f27410a74a9d567b66dc4"}
{"family": "path", "fold": "outer-2", "group": "huggingface/datasets", "label": 0, "pair": "a07929b3fc1cfc28", "unit": "ffd286c8588a77929639a89e4a742e15306b642af14aa906c721165df804ccd3"}
{"family": "injection", "fold": "outer-2", "group": "ibm/mcp-context-forge", "label": 0, "pair": "6548e1527e82bd99", "unit": "5542613f6d2eb9421b8213b8bb864894f0ecf9fe64945681019230ef0d75cfaa"}
{"family": "injection", "fold": "outer-2", "group": "ibm/mcp-context-forge", "label": 0, "pair": "", "unit": "6c2adee34e9e8384535ec1073eda60e2ab528ac77c36747802c303e1f5e9ffe8"}
{"family": "injection", "fold": "outer-2", "group": "ibm/mcp-context-forge", "label": 1, "pair": "", "unit": "7ce70e9b20107cc7ebb87d173da765fc53ffd268e521749a06a9a1f38d746845"}
{"family": "injection", "fold": "outer-2", "group": "ibm/mcp-context-forge", "label": 1, "pair": "6548e1527e82bd99", "unit": "c3e54975464323e58ec96ee4d83e7bc08baf94553d553a8dc3fb28cd9c70ff21"}
{"family": "injection", "fold": "outer-2", "group": "ibm/mcp-context-forge", "label": 1, "pair": "", "unit": "d3173c1e43eb6402a71ea2dec1592913dc078a2194ba1c1d509ab129a3180f6f"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "internlm/lmdeploy", "label": 1, "pair": "0924befc49c7bb18", "unit": "e7e4872db40f0420547ed622fbdd7a072b64a7ba466d1ccfff68e691cee6ad99"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "internlm/lmdeploy", "label": 0, "pair": "0924befc49c7bb18", "unit": "f7d95f43fb2df20ebc4c37fceabdd00ffd8159f404ca384481f1764252382388"}
{"family": "injection", "fold": "outer-3", "group": "iximiuz/node-diskusage-ng", "label": 0, "pair": "d4ee6cfeba71e398", "unit": "e79582de3031bada2617a0667ef7ff5cd7183d4aa885f55a665cf0ec88d17624"}
{"family": "injection", "fold": "outer-3", "group": "iximiuz/node-diskusage-ng", "label": 1, "pair": "d4ee6cfeba71e398", "unit": "fef35272853e7f3692efd54daa01c12862953352fa6ccbbb5e3f7fea70c63d78"}
{"family": "path", "fold": "outer-0", "group": "jarofghosts/glance", "label": 1, "pair": "772e18821701d60f", "unit": "89a0496c7663bbfd67d660c9f481c12420da2dd1bdb29dc079735ee7a0a49626"}
{"family": "path", "fold": "outer-0", "group": "jarofghosts/glance", "label": 0, "pair": "772e18821701d60f", "unit": "d3241ceaa50a7d23a4650e6a2a55372b08ec96b35b0bede844bc40e16e818ebc"}
{"family": "injection", "fold": "outer-1", "group": "jas-/node-libnmap", "label": 1, "pair": "250de1302c7e541d", "unit": "5d03535f2df2acdbc319d9e6b20da65d51f3e810b80f3c8f89b5074a971a3f1c"}
{"family": "injection", "fold": "outer-1", "group": "jas-/node-libnmap", "label": 0, "pair": "250de1302c7e541d", "unit": "78f977cfc096e34b4eada7f798ad2973ef45728d0d01bfab6ac9e2a3257827ef"}
{"family": "injection", "fold": "outer-2", "group": "jashkenas/underscore", "label": 1, "pair": "c10bfb006f2da55a", "unit": "1ea4b86dbd84581b1d7b7cabbab9f28c8b213416a60778879666ff42d23fd7f1"}
{"family": "injection", "fold": "outer-2", "group": "jashkenas/underscore", "label": 0, "pair": "c10bfb006f2da55a", "unit": "7647021f742da24c660b2d407886771786e957076c4cf336def560a570dd8778"}
{"family": "access_control", "fold": "outer-2", "group": "jellyfin/jellyfin", "label": 1, "pair": "a73421efb106ab0a", "unit": "1f6f207a758582de0c0e6b331b19fe119abe18727d45ccbbe09aabb795f3a292"}
{"family": "access_control", "fold": "outer-2", "group": "jellyfin/jellyfin", "label": 0, "pair": "a73421efb106ab0a", "unit": "32fbdad39eaf5177db4ad0a11d53e785b30bc7b2376ad9d460f29a08429bd0c7"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "jenkinsci/cas-plugin", "label": 0, "pair": "0458e3fd444409f9", "unit": "b521216bb4106738b3571ad3902d08f6230aabd88cf3a8f9f26f1c8ec01304cb"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "jenkinsci/cas-plugin", "label": 1, "pair": "0458e3fd444409f9", "unit": "c87d2871907798fa2e73d31381e9d63089b0388033f096d247373fec07c2cb36"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "jenkinsci/google-login-plugin", "label": 0, "pair": "6bdccabcf5b9fdc9", "unit": "3d6ddca5eb264d88411b86755909f1d2abddd09c129cabd54c96ff26b4105316"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "jenkinsci/google-login-plugin", "label": 1, "pair": "6bdccabcf5b9fdc9", "unit": "e055b898802279bc63204cc601e29f7f1018efad7228cab0cce00ea5866ecc7b"}
{"family": "access_control", "fold": "outer-3", "group": "jenkinsci/mcp-server-plugin", "label": 1, "pair": "91bc832e70680c92", "unit": "1060c63ff0fae3af567448f3d4c60d9c159b61ccc358964243eb152a9ac38865"}
{"family": "access_control", "fold": "outer-3", "group": "jenkinsci/mcp-server-plugin", "label": 0, "pair": "91bc832e70680c92", "unit": "369dffd1e730d7997813adfbebf1481cc68987cae0d0085e47b91a26d5330cf8"}
{"family": "output_encoding", "fold": "outer-0", "group": "jihong88/suneditor", "label": 0, "pair": "137fb3496a3a75f4", "unit": "7e8b6c484999229b23d50d9f5a4c4c5b2a8d954a5d64230981b1c6705b760fbf"}
{"family": "output_encoding", "fold": "outer-0", "group": "jihong88/suneditor", "label": 1, "pair": "137fb3496a3a75f4", "unit": "c461154e4dbd74e35a4ed2e85de7b076f0582beeec459be28104c38f616c7565"}
{"family": "path", "fold": "outer-4", "group": "jknack/handlebars.java", "label": 1, "pair": "a4144258dffb3b8d", "unit": "7ab301e1614979f44c353e665540582493f78ef4bfc41209562c19b900710f8c"}
{"family": "path", "fold": "outer-4", "group": "jknack/handlebars.java", "label": 0, "pair": "a4144258dffb3b8d", "unit": "ea95e8387dff32ded9dc7b04c11a76a545b4e2ea6db0076b9b13971a59904bb3"}
{"family": "deserialization", "fold": "outer-4", "group": "jmespath/jmespath.rb", "label": 1, "pair": "858330e01d3cf9bb", "unit": "033cda2b7cffa5a5fa6abf0b7f2cd27ba78d916eb3cab8c79decccd8776f4a3f"}
{"family": "deserialization", "fold": "outer-4", "group": "jmespath/jmespath.rb", "label": 0, "pair": "858330e01d3cf9bb", "unit": "d44c4555a5ae19aa17b468ec5cff56b33a0b221df52e04d8abcb656399499dee"}
{"family": "deserialization", "fold": "outer-2", "group": "jooby-project/jooby", "label": 1, "pair": "c6a49339d4f0c798", "unit": "dc2e4cd5bd963643702d2b832ae170dd25744162e4076e36687256a09e69d089"}
{"family": "deserialization", "fold": "outer-2", "group": "jooby-project/jooby", "label": 0, "pair": "c6a49339d4f0c798", "unit": "f83cd137b85e651ca5299c5a42176b797ed4964cbef31418c96d63c52fdb65f5"}
{"family": "access_control", "fold": "outer-1", "group": "jowilf/starlette-admin", "label": 1, "pair": "4b6526d132e9ac86", "unit": "8ec3596444f3045684242840e9a1df19879383f2733aca8f8af6f6cf8a96d388"}
{"family": "access_control", "fold": "outer-1", "group": "jowilf/starlette-admin", "label": 0, "pair": "4b6526d132e9ac86", "unit": "f3d39162fa14446b207aabe7bf3c913dc32a9637ac83352ea5cb2fdbb43720a8"}
{"family": "injection", "fold": "outer-4", "group": "jsonata-js/jsonata", "label": 0, "pair": "80abf1d5c4a3f6a5", "unit": "3df3c6d9941352dd2861427da56b94ee335f308d0a2a27c6f745aef60349c80f"}
{"family": "injection", "fold": "outer-4", "group": "jsonata-js/jsonata", "label": 1, "pair": "80abf1d5c4a3f6a5", "unit": "4fde61a570906e658d9b14db0e5d1f5acc0c7489951421bff2afde25033ff73c"}
{"family": "access_control", "fold": "outer-2", "group": "jsonbored/gittensory", "label": 1, "pair": "5ec37c0f475001ca", "unit": "5d8175fb21a4649bdc8730358e9b87adb90b2aafc4da5fee203829663e4f46a0"}
{"family": "access_control", "fold": "outer-2", "group": "jsonbored/gittensory", "label": 0, "pair": "5ec37c0f475001ca", "unit": "af845e9e332555096dd350401977b2f07015433f3db8a040891d9470f3f9e355"}
{"family": "deserialization", "fold": "outer-0", "group": "keras-team/keras", "label": 1, "pair": "68d44ef338ef3470", "unit": "3020ac6c3186d7974953b1c4da65b80141b81aca5203436ae7a2d7aca0870944"}
{"family": "deserialization", "fold": "outer-0", "group": "keras-team/keras", "label": 0, "pair": "68d44ef338ef3470", "unit": "8fdb85a6bfec89f0715ea81e83fd91616f92c59954c092e79272154f202b7488"}
{"family": "deserialization", "fold": "outer-4", "group": "kevin1024/vcrpy", "label": 1, "pair": "c74525e844650de2", "unit": "2e444ad34ee5f47d73f0cfad0a10c70363337b4ddf9067ecd1b8072d8232652c"}
{"family": "deserialization", "fold": "outer-4", "group": "kevin1024/vcrpy", "label": 0, "pair": "c74525e844650de2", "unit": "c492d325f5ff82b5f817aa4be1440cec8595054dd0719e07a58ac5501b2ae7eb"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "koajs/koa", "label": 1, "pair": "111a5a80a3abc2c9", "unit": "c7ae403742450deb2bd051ad1231121d2117ba56fa72a038c82d58d207cb3870"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "koajs/koa", "label": 0, "pair": "111a5a80a3abc2c9", "unit": "debe34831afca6f8220b9952036a2d4570babdb5e321a2f8dbaec661a016bd50"}
{"family": "access_control", "fold": "outer-3", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-crm-saas-django", "label": 1, "pair": "", "unit": "9950b85412b021ebabb11eea4c490a660ad380d14c673a87c3fb568ba4bcfac9"}
{"family": "path", "fold": "outer-3", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-crm-saas-django", "label": 1, "pair": "", "unit": "656df5ff8e69dcdb1f9ba1ed52d734966eb29773bc745587483a355743b8eb5a"}
{"family": "access_control", "fold": "outer-1", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-hr-payroll-django", "label": 1, "pair": "", "unit": "91652a4e776fbce70cde9e4a99c6da5e17da7f8d6d0404867c61eec04161b6cf"}
{"family": "path", "fold": "outer-1", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-hr-payroll-django", "label": 1, "pair": "", "unit": "e53bb3a217dc4737e8dd03671de4f6a895599024070a985a268c65a034295e7c"}
{"family": "deserialization", "fold": "outer-1", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-legal-case-django", "label": 1, "pair": "", "unit": "3d7aba22444811a337abcad0e4fc62990bb704d3a10e72d893342fa94699026b"}
{"family": "output_encoding", "fold": "outer-1", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-legal-case-django", "label": 1, "pair": "", "unit": "5f18a3026a51f7140b7aaa3aa8563e8b6fdda6065d95e19e25b1dac4b9ce0e00"}
{"family": "deserialization", "fold": "outer-0", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-logistics-dispatch-fastapi", "label": 1, "pair": "", "unit": "0fccc306ce80cb4cbfdcde0a2e56d637bcefa9dd30423d6c54a4f77860afe740"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-logistics-dispatch-fastapi", "label": 1, "pair": "", "unit": "1e2eaa45661345f1b6759d968d7696dd0d96d6fba1350a00e4d0a2ebaaa2e391"}
{"family": "output_encoding", "fold": "outer-2", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-marketplace-commerce-fastapi", "label": 1, "pair": "", "unit": "8b9a8ff0e32a327df75eb8a1811464d5f34690079d5930139dd917063bdd2809"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-marketplace-commerce-fastapi", "label": 1, "pair": "", "unit": "245d0e939b7c1ccf098c798492009e3a73ea17400fa613dd865cd369d2e7f383"}
{"family": "access_control", "fold": "outer-2", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-property-management-fastapi", "label": 1, "pair": "", "unit": "653c48df493f31c2d01c973d1e379a180fbe6a6edc83a500bc62509112175f41"}
{"family": "path", "fold": "outer-2", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-property-management-fastapi", "label": 1, "pair": "", "unit": "3719fe21d45b50b35458bcd2e263e4baee67adbd3c98adf0eaab6bc6183c5651"}
{"family": "output_encoding", "fold": "outer-3", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-support-desk-fastapi", "label": 1, "pair": "", "unit": "1209a6cc8c1548ed83d4a9a09d6a27f706a23e69cf0c261fc5e3a27536a83306"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "kolega-ai-dev/vc-claude-code-seeded-v2-support-desk-fastapi", "label": 1, "pair": "", "unit": "362c90495dc468bb732c71f62d71ec308854161de31ad18d2a870056d947ddb6"}
{"family": "access_control", "fold": "outer-4", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-crm-saas-django", "label": 1, "pair": "", "unit": "b52e53fec9707b2b442c4dc7a41d376cf76946d2c45c2eb600c9477fc6baebaa"}
{"family": "path", "fold": "outer-4", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-crm-saas-django", "label": 1, "pair": "", "unit": "477e3b90b92e622888dbe0e9d3a23b31271ddfa3104a5ab326eb9e9e765aa01d"}
{"family": "output_encoding", "fold": "outer-0", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-education-lms-django", "label": 1, "pair": "", "unit": "660c63c4a73bfe64088ae3249e40d9c5d69a6f8a505ff671217c5708a345c5b0"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-education-lms-django", "label": 1, "pair": "", "unit": "4d60c386c801bedb61c1461fced29a3cfb50b09d6435e8617c60398343719d93"}
{"family": "deserialization", "fold": "outer-2", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-fintech-lending-fastapi", "label": 1, "pair": "", "unit": "f1fddcc1354a29bef55610ee2b73eaa440192f2f18a0d22d08057b9d1e1b61b8"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-fintech-lending-fastapi", "label": 1, "pair": "", "unit": "6d9886c38221b2f9aacf83749218ac5ebc354a6fcc4170e6c78bedb92b88296a"}
{"family": "deserialization", "fold": "outer-0", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-healthcare-clinic-django", "label": 1, "pair": "", "unit": "5550fec4a81c4c9cb22d7312d8a63e1fb2b03c4aed0c1f316d44c1eb27511bac"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-healthcare-clinic-django", "label": 1, "pair": "", "unit": "8ff87dd04140f0137ea1891f220b1276156ce10a58be5a49d0c8e07be531ffb2"}
{"family": "access_control", "fold": "outer-1", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-hr-payroll-django", "label": 1, "pair": "", "unit": "f24b19cc0f67247eccb274521a62741fef8f4729f0ddc9b5c1fa561a4947182a"}
{"family": "path", "fold": "outer-1", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-hr-payroll-django", "label": 1, "pair": "", "unit": "83523c9d83380d4cdae593a593d361493e00d18e61740f84e762aad41b991e18"}
{"family": "deserialization", "fold": "outer-4", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-legal-case-django", "label": 1, "pair": "", "unit": "c6c9bde6630678377d250701073c5c223c7892e6ff8f2f8e0293dea22c5b5e7e"}
{"family": "output_encoding", "fold": "outer-4", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-legal-case-django", "label": 1, "pair": "", "unit": "b9b15958973c83f8d7b58372ab5ba58eb191c400125579d5ae8c5b52d5d1368e"}
{"family": "output_encoding", "fold": "outer-0", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-marketplace-commerce-fastapi", "label": 1, "pair": "", "unit": "b95ec12e4e126ad558e944e2c260116ed6a63f5f3ab97fb5cbcdca75a931ad73"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-marketplace-commerce-fastapi", "label": 1, "pair": "", "unit": "38a5545ac9ef67e15db720d1396a2d5915f9d06d6f4772012e5bc95f33770376"}
{"family": "access_control", "fold": "outer-0", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-property-management-fastapi", "label": 1, "pair": "", "unit": "c8bbe9be91c976be97b7ca387048a7d4537d9f6fcc6cbad0a541d1080629d5a6"}
{"family": "path", "fold": "outer-0", "group": "kolega-ai-dev/vc-codex-high-seeded-v2-property-management-fastapi", "label": 1, "pair": "", "unit": "ffacd53f1bd75155e63b318fb9cba27ec03e754841e1eb39d1d4fa0b2c3dfca2"}
{"family": "output_encoding", "fold": "outer-4", "group": "kolega-ai-dev/vc-codex-seeded-v2-healthcare-clinic-django", "label": 1, "pair": "", "unit": "8fd920175fc43b91b4397acffacb10ab8f7fd0ee38bcf2d2f3ed06a3fad0dfe9"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "kolega-ai-dev/vc-codex-seeded-v2-healthcare-clinic-django", "label": 1, "pair": "", "unit": "86c9598ebe3dcda261a69ae545513f6ce73101f3980066e4506ece99bebea84f"}
{"family": "deserialization", "fold": "outer-3", "group": "kolega-ai-dev/vc-codex-seeded-v2-hr-payroll-django", "label": 1, "pair": "", "unit": "e5190fcc57505116f79a31c51aed69f8a1eea13a8ae3e7265f44c72418277ef9"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "kolega-ai-dev/vc-codex-seeded-v2-hr-payroll-django", "label": 1, "pair": "", "unit": "2b807ff54e6c47e941124c16d0c27c691cef54f2203060a3a6ce812d478c5261"}
{"family": "deserialization", "fold": "outer-4", "group": "kolega-ai-dev/vc-codex-seeded-v2-legal-case-django", "label": 1, "pair": "", "unit": "401aefb4b4791301ace2d770d321c6b26743b0aab7b96f8b410254c704be6d2e"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "kolega-ai-dev/vc-codex-seeded-v2-legal-case-django", "label": 1, "pair": "", "unit": "22d186c93498e21418b6c1d0547a73374b13041d8afc89a80d9540fa658b6b42"}
{"family": "output_encoding", "fold": "outer-4", "group": "kolega-ai-dev/vc-codex-seeded-v2-marketplace-commerce-fastapi", "label": 1, "pair": "", "unit": "dbfb6df14fe37f8c0efda5f16643297fb0550dfc5ad5805d4e4b844d967773b2"}
{"family": "output_encoding", "fold": "outer-4", "group": "kolega-ai-dev/vc-codex-seeded-v2-marketplace-commerce-fastapi", "label": 1, "pair": "", "unit": "f0da8cfe4ac1e586aa7903d72e67b44500bbd5306d72b4b342d10613efc69222"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "kolega-ai-dev/vc-codex-seeded-v2-support-desk-fastapi", "label": 1, "pair": "", "unit": "2b8fd73943f456da266434470333a7775f55d9a41953e97a000fdaa3257b6a30"}
{"family": "deserialization", "fold": "outer-2", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-crm-saas-django", "label": 1, "pair": "", "unit": "a612091f452e60f28f44eefec641175170dc5db04456a39a099cbb13c50b4d2c"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-crm-saas-django", "label": 1, "pair": "", "unit": "4c6600facd83bbf234a32d9e28ac494fb516df6e3d9e9dc28d720ff35252d067"}
{"family": "deserialization", "fold": "outer-1", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-education-lms-django", "label": 1, "pair": "", "unit": "fb0ea8cab8c9430d22d1b0fdd65a69146fa95fbff94e720b4861b96538d992c6"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-education-lms-django", "label": 1, "pair": "", "unit": "e81f91a56058ac1473579767abea57064c53d7d007797a854314814e3c35ee74"}
{"family": "access_control", "fold": "outer-0", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-fintech-lending-fastapi", "label": 1, "pair": "", "unit": "5851ecd96a449f98b05f243bfda0a7cf3af8612b27f53c414ae1f4ea67242a7f"}
{"family": "path", "fold": "outer-0", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-fintech-lending-fastapi", "label": 1, "pair": "", "unit": "328621ef2a94632e5d37ea6267d17b766233384cde35c07e29a4eb672d40ba28"}
{"family": "output_encoding", "fold": "outer-1", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-healthcare-clinic-django", "label": 1, "pair": "", "unit": "04b2eae2128608dad460ba8275142a41a44e32c4243a29e2a6fb2c9ebf3d8f1a"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-healthcare-clinic-django", "label": 1, "pair": "", "unit": "33337002ff5f5a189610b042522ceea254c9d107c1f24771b4f616a0f96f636e"}
{"family": "deserialization", "fold": "outer-0", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-hr-payroll-django", "label": 1, "pair": "", "unit": "02c09a783cbf058856fc2b448406f3ffa6fbfbacf9b70f4e5e22fd741a5c7699"}
{"family": "untrusted_destination", "fold": "outer-0", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-hr-payroll-django", "label": 1, "pair": "", "unit": "956260164cb21a47c499e5598491031ef303418943f5096f225a87162b588574"}
{"family": "deserialization", "fold": "outer-1", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-legal-case-django", "label": 1, "pair": "", "unit": "8bd8c5aed69eae3019cc2d3654b36bdf2b31435e6a020a389cd103696a2f874c"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-legal-case-django", "label": 1, "pair": "", "unit": "1d8397ef4a9fde7b48e2e1e871bba0f960a802cd7823ecfc10e9dfaddb955ccc"}
{"family": "output_encoding", "fold": "outer-1", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-marketplace-commerce-fastapi", "label": 1, "pair": "", "unit": "c9db92476032eb4296b950a1eff8fa4aaa26f11542a01a5cf10c7b2e32dc6433"}
{"family": "output_encoding", "fold": "outer-1", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-marketplace-commerce-fastapi", "label": 1, "pair": "", "unit": "d3e0d0f9b0f4b723080989b5486395cda0a5b186a674022ea70e7fa7ab561323"}
{"family": "deserialization", "fold": "outer-3", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-property-management-fastapi", "label": 1, "pair": "", "unit": "44ff3b3e61c02744830c4abea872821ca7e0f94950c082740c5c48ef13604e94"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "kolega-ai-dev/vc-kimi-code-seeded-v2-property-management-fastapi", "label": 1, "pair": "", "unit": "4103e979a99233c3e71fdfe9f1cdcbab34a2c60b5351b75c5ff5dfc287b962d0"}
{"family": "injection", "fold": "outer-0", "group": "koxudaxi/datamodel-code-generator", "label": 0, "pair": "f73ecbfda2bcadef", "unit": "53e596dabf90b2992d5b3e50394791859b393beedc8f84e00484e37498627ff0"}
{"family": "injection", "fold": "outer-0", "group": "koxudaxi/datamodel-code-generator", "label": 1, "pair": "f73ecbfda2bcadef", "unit": "6daaccf64706e876dd505c65a62aeacc1357f3470a5cef7b1ac874032a6fcf99"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "kozea/weasyprint", "label": 1, "pair": "6768bfae38c6859d", "unit": "f376519889a00950f5af67eb15c639306ea70da7235d3be4a94dbc19cb096cf5"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "kozea/weasyprint", "label": 0, "pair": "6768bfae38c6859d", "unit": "f3fea4e334f49ef6fe6b17121fef553eb8e33a9ffa13e6d6cbc4c448da996a7c"}
{"family": "access_control", "fold": "outer-0", "group": "kubev2v/migration-planner", "label": 1, "pair": "d670253fa209671c", "unit": "1aeebe2e440bdd70525c31f93180f33be93f820db5ba46a2e59c371ef83ee794"}
{"family": "access_control", "fold": "outer-0", "group": "kubev2v/migration-planner", "label": 0, "pair": "d670253fa209671c", "unit": "b11c8a8a24d3809c0507228568fe7cc05ca0a22ed28cb19019adb13cb619d63f"}
{"family": "injection", "fold": "outer-0", "group": "kylefarris/clamscan", "label": 0, "pair": "33260f3e6c8ea6d0", "unit": "7ca409c7dbc1c38ff005624e212efc6d424cb421e27b482dc8dc0d6881f4c11a"}
{"family": "injection", "fold": "outer-0", "group": "kylefarris/clamscan", "label": 1, "pair": "33260f3e6c8ea6d0", "unit": "fc364d7ed77c7c0508da00e5ba847ec0792f3bc373d8b4297747189ef42d1ae0"}
{"family": "access_control", "fold": "outer-1", "group": "labs64/netlicensing-mcp", "label": 0, "pair": "32ef8148a2c08f58", "unit": "2fb2c9314a483787168a450f50430721f9527f913aa0eae07298d5334cf4f137"}
{"family": "access_control", "fold": "outer-1", "group": "labs64/netlicensing-mcp", "label": 1, "pair": "32ef8148a2c08f58", "unit": "78dd291bb2ef83ea3a87948c0ac0027c6d667b9044093e766e2bc91bebe83936"}
{"family": "path", "fold": "outer-3", "group": "langchain-ai/langchain-nvidia", "label": 0, "pair": "54586b5a24655adf", "unit": "2ece75a0240ed36a811c381841aa6d440af2b9774ddacb1d7a31a67ec267b0b2"}
{"family": "path", "fold": "outer-3", "group": "langchain-ai/langchain-nvidia", "label": 1, "pair": "54586b5a24655adf", "unit": "e60f6b788a196db9bb671d621189b79a5526302a6d09c2396d67fbe0b9ab3c24"}
{"family": "access_control", "fold": "outer-2", "group": "langflow-ai/langflow", "label": 0, "pair": "474f96e2a9b3d202", "unit": "4a5e8e8ebd7ecb0e252280af0cb38a1d7794757728b912525322ac26df5ee223"}
{"family": "access_control", "fold": "outer-2", "group": "langflow-ai/langflow", "label": 1, "pair": "474f96e2a9b3d202", "unit": "6a888540443061d3b37c06f0cebc70cffdfa70e0aa940626e86cbff9829310fe"}
{"family": "deserialization", "fold": "outer-1", "group": "laravel/reverb", "label": 1, "pair": "a09692dc94164db8", "unit": "139cf84e79c4efd026a064b2ca0a3d7ffff87b54eae1275a07c0d69ef4981e27"}
{"family": "deserialization", "fold": "outer-1", "group": "laravel/reverb", "label": 0, "pair": "a09692dc94164db8", "unit": "828e88b656245a1d0865c79b2ed023a49e78607a4ac81d70ccd3690f4419d2d1"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 1, "pair": "", "unit": "6cce78d7569ad8db7ca0a61d7078198f666502d26ea6c0afb03173a8a4bb16ef"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 1, "pair": "82047c0513e43a83", "unit": "6d2dbed1ab1c9d8ac259dd4227be69641fa59d52e816a8dc547ade01d0b9b0c4"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 1, "pair": "98ee23cd6bd31cd3", "unit": "ab4bb39be535d70cd95e9035bb958bfc7418576a6ff40a47551894dd140317bb"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 0, "pair": "98ee23cd6bd31cd3", "unit": "b70068cbc646bd01d3e10986e002f648a74e9d798900dc59cf3039b749c95c18"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 1, "pair": "ff168d95d8d9496e", "unit": "c6ec9478c125ed780d7013c2807f2cec1878f44253667dfed319c28cbfd17d4d"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 0, "pair": "b1bfef46dcd3b9a5", "unit": "cd4d095d8380a2fb7ca02fc6abdbae5249cc06455db2929875533f4aa8b886dc"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 0, "pair": "82047c0513e43a83", "unit": "d1bade217be8d345cf5d0f0442ee31924187acbb98779dada95274a77462f696"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 0, "pair": "ff168d95d8d9496e", "unit": "d561db250b892b91feed7ae877900e1e25963444f73c7699576e010f06da73cf"}
{"family": "injection", "fold": "outer-0", "group": "learnpress/learnpress", "label": 1, "pair": "b1bfef46dcd3b9a5", "unit": "ed78fd5a7ab30bcd6e8ddd27c706a19cfd89750de4fbf81c0b567946eb94db83"}
{"family": "output_encoding", "fold": "outer-0", "group": "lepture/mistune", "label": 0, "pair": "25f75d77a669b54e", "unit": "0f2825568461562f2a915a4f16ef8a4931290fd20a9801470aeb12677e66e3f1"}
{"family": "output_encoding", "fold": "outer-0", "group": "lepture/mistune", "label": 1, "pair": "25f75d77a669b54e", "unit": "bf22bb44bcff4a5baf5d550c9d3e83e5f3ca607475fb51deaf4fb6fccff3d73a"}
{"family": "path", "fold": "outer-1", "group": "linuxfabrik/lib", "label": 0, "pair": "47069df6bfb3dc00", "unit": "444c1f41fc23ad656a8bf98dae58b36d0f1daa214ab030682f26111dd389e58d"}
{"family": "path", "fold": "outer-1", "group": "linuxfabrik/lib", "label": 1, "pair": "47069df6bfb3dc00", "unit": "c0c8b5cca070673ea4b4bb8f08dfff86b7781780e300720c4051aded035052fe"}
{"family": "output_encoding", "fold": "outer-2", "group": "linuxuser255/vulnerable_python_apps", "label": 1, "pair": "", "unit": "cd80750447cf7af67ad809279718d98106c71705fef9ff7254728bfff4ece5b1"}
{"family": "path", "fold": "outer-2", "group": "linuxuser255/vulnerable_python_apps", "label": 1, "pair": "", "unit": "2ce605ab5ce7ad3a1fb983cd6bddf3924e5da3010d2252a0a4640aab4e5921a0"}
{"family": "deserialization", "fold": "outer-0", "group": "litedb-org/litedb", "label": 1, "pair": "b0a899116dc36b85", "unit": "510ad48af03ba4a1a2a06dd254b135311dddb62d9fe5a2e8e8b12627586e6b13"}
{"family": "deserialization", "fold": "outer-0", "group": "litedb-org/litedb", "label": 0, "pair": "b0a899116dc36b85", "unit": "b414718a8a5f79739b88db24da6ad9da8e48bcb6b5e2dc99158848cdaa226767"}
{"family": "deserialization", "fold": "outer-0", "group": "manuelz120/extremely-vulnerable-flask-app", "label": 1, "pair": "", "unit": "8e22ee30441b20484090659b0c97423c711f5d10219041bf3a399623fc056a07"}
{"family": "injection", "fold": "outer-0", "group": "manuelz120/extremely-vulnerable-flask-app", "label": 1, "pair": "", "unit": "0f9070df9dc334f981cb01008c260b8bff78dbdeb48ce672830ed95caa8406fb"}
{"family": "injection", "fold": "outer-0", "group": "manuelz120/extremely-vulnerable-flask-app", "label": 1, "pair": "", "unit": "5e352f797b1422b0df477bce4730a9ce33cbd0837c3d63da94e4d8801a9a4b64"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "medplum/medplum", "label": 1, "pair": "dadccb128ac0e120", "unit": "505097494da5939404ad7d5b4af2b1c4a10efe94048084492d6ce60ebaa04c97"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "medplum/medplum", "label": 0, "pair": "dadccb128ac0e120", "unit": "84ff8ca2d98a1b5d1ca4bfaabe7d19e03a396042b228c8d18d1185a178389702"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "medplum/medplum", "label": 0, "pair": "", "unit": "91e261c54bd2c3c3a9ac04e48894ecabaca5b13e7db4fbd5578f9d266098117a"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "medplum/medplum", "label": 0, "pair": "", "unit": "a308dfd59bc398824e2454bf6cf6cfd5b7880e5b948a82424f14753d2b6fcdbe"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "medplum/medplum", "label": 0, "pair": "", "unit": "ba24eb554458bc97314ed561a7b2f4a4e643dde1205c32de4c8fcdd8c46e575b"}
{"family": "output_encoding", "fold": "outer-1", "group": "mganss/htmlsanitizer", "label": 0, "pair": "24d8f1840b67fbe5", "unit": "7ffa4a229a62266788fbf64890f81f20358b3eb8c35d3ab87d0c339d5bb4b507"}
{"family": "output_encoding", "fold": "outer-1", "group": "mganss/htmlsanitizer", "label": 1, "pair": "24d8f1840b67fbe5", "unit": "d19493ed7941e1cd25a76dc50504d97398770d07ebc302dd66d83441e946c468"}
{"family": "path", "fold": "outer-3", "group": "mguinness/elfinder.aspnet", "label": 0, "pair": "9f8a60eb48d74bc8", "unit": "80fce34f2377731e3de4355c7ea33bb20fc2fc00a5f388565b23018c7a3ed008"}
{"family": "path", "fold": "outer-3", "group": "mguinness/elfinder.aspnet", "label": 1, "pair": "9f8a60eb48d74bc8", "unit": "b68e62722a327c9d5ce6774eb96527451729410768efc0b5c7ba604ec468052d"}
{"family": "injection", "fold": "outer-4", "group": "microsoft/prompty", "label": 0, "pair": "2c857235d0764986", "unit": "33bdfe911cd1b512e5994ee7a3e3ab2ad1f302cab29df3b011c0d825c23536a3"}
{"family": "injection", "fold": "outer-4", "group": "microsoft/prompty", "label": 1, "pair": "2c857235d0764986", "unit": "b5efdcb84a057f3cd9fd3a30a32239dc06c13536a4e84a66c7a17cb372050c97"}
{"family": "output_encoding", "fold": "outer-0", "group": "middleapi/orpc", "label": 1, "pair": "9e84429d629c0b6b", "unit": "774cfdecac67499bb35a872931d4698d1b92ebeeb42d21790469e418889954e5"}
{"family": "output_encoding", "fold": "outer-0", "group": "middleapi/orpc", "label": 0, "pair": "9e84429d629c0b6b", "unit": "a6c5c15ca1815f83dfb47428c4a8afd4e60fa1c027b9f33c668cb9a0d77c18af"}
{"family": "output_encoding", "fold": "outer-2", "group": "mixxorz/slippers", "label": 0, "pair": "8e6b4b4e28981590", "unit": "3cce3e8f9ff3f1d261f9b96c844c1e4da07bae72d5393897b3a929be25372edc"}
{"family": "output_encoding", "fold": "outer-2", "group": "mixxorz/slippers", "label": 1, "pair": "8e6b4b4e28981590", "unit": "b105689ac24ed334294c434a77bb6d56d5d598830ffe8a5d60ab3be5290b47ad"}
{"family": "injection", "fold": "outer-1", "group": "mongo-express/mongo-express", "label": 1, "pair": "", "unit": "0a36cd5177f3b2c20b433db6bfffe5ec5b7e74cacc34a86ac581f7a0a559432a"}
{"family": "injection", "fold": "outer-1", "group": "mongo-express/mongo-express", "label": 1, "pair": "e7fc97d83013c9ff", "unit": "5ab444e64297fc19065478104c7ebdabe23cc415f22adafd0cf82c98e4c5cae3"}
{"family": "injection", "fold": "outer-1", "group": "mongo-express/mongo-express", "label": 0, "pair": "e7fc97d83013c9ff", "unit": "6b462ea30c1afda572600a10324249223932d36a032eacafac8c837bccbdce87"}
{"family": "injection", "fold": "outer-1", "group": "mongo-express/mongo-express", "label": 1, "pair": "84d6a4202a082e0d", "unit": "8175082f40d8d61d44b54579c84a808e25b89110be5d8fd82feedf7dd5e0d5ab"}
{"family": "injection", "fold": "outer-1", "group": "mongo-express/mongo-express", "label": 1, "pair": "", "unit": "8abefca43ee4ae2c797033da31f85c17bfa7be6b35413d70b0216415c5db9a09"}
{"family": "injection", "fold": "outer-1", "group": "mongo-express/mongo-express", "label": 1, "pair": "", "unit": "91d213946859f33d3f243c43aa8285cc6e8ecb3db1de938f32af775a70607f8e"}
{"family": "injection", "fold": "outer-1", "group": "mongo-express/mongo-express", "label": 1, "pair": "", "unit": "a244d9f010ddb43a09e82363dfabb0e406d411fb5de08ab7d6c9ee367f2a15c3"}
{"family": "injection", "fold": "outer-1", "group": "mongo-express/mongo-express", "label": 1, "pair": "", "unit": "d27d174d6e8e5f1682a1c42723705e89f0d18c95ecedd0567e313daa9a0aae44"}
{"family": "injection", "fold": "outer-1", "group": "mongo-express/mongo-express", "label": 0, "pair": "84d6a4202a082e0d", "unit": "f45608ed47fa11ca25093641f6021ec06076c18676abac9b809f31b3b7713de8"}
{"family": "access_control", "fold": "outer-3", "group": "moquette-io/moquette", "label": 0, "pair": "55d1ce4f29dd1583", "unit": "a877ce3462da316bcdd82ce9db1d9887e887eaa83157922edb1663413edae1d6"}
{"family": "access_control", "fold": "outer-3", "group": "moquette-io/moquette", "label": 1, "pair": "55d1ce4f29dd1583", "unit": "aee67be0ba31f99c7a24faa1bad6178c6d37361a308cf5aa10138e6ce4bbcfeb"}
{"family": "path", "fold": "outer-4", "group": "naorsabag/openhop", "label": 1, "pair": "16e529fe8c5486b2", "unit": "1d21acd0171726775d3da4589b660645fcd43cc026042a2b27e0e0e8374eceb7"}
{"family": "path", "fold": "outer-4", "group": "naorsabag/openhop", "label": 0, "pair": "16e529fe8c5486b2", "unit": "f8e004b67ac98dc3392ea4115051e6bd10ce22eb638a762f0f16e1d4200bf129"}
{"family": "access_control", "fold": "outer-1", "group": "nextauthjs/next-auth", "label": 1, "pair": "0b34438e6e0ba63d", "unit": "bf2608b00f0f784cc42e4c607f0b1ef50903283779bfb2eec91d2ec187856d14"}
{"family": "access_control", "fold": "outer-1", "group": "nextauthjs/next-auth", "label": 0, "pair": "0b34438e6e0ba63d", "unit": "c5104cbf723fa0d55675963d8aa6802780ce3d274efe0012290fc49164c3fcc4"}
{"family": "output_encoding", "fold": "outer-3", "group": "nuxt-content/mdc", "label": 0, "pair": "b0210cd069757ad2", "unit": "d1d40c8b48d039c1999ff957a015273c30652ca0c45c4eb088052991a2a92f28"}
{"family": "output_encoding", "fold": "outer-3", "group": "nuxt-content/mdc", "label": 1, "pair": "b0210cd069757ad2", "unit": "e2e804652a885f640ebf341627f3279d6c965b47ee2ce792e55045ab4d1e1dee"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "nuxt-modules/og-image", "label": 1, "pair": "ef603436079b0005", "unit": "77f843b5044bb5da935df4b2620bf2abe9e7dceb3147439e883927112eaec22f"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "nuxt-modules/og-image", "label": 0, "pair": "ef603436079b0005", "unit": "f2ce95f0e44e73986784b63a702ffa4035c423056cac3667a63140a7e8d795c2"}
{"family": "injection", "fold": "outer-2", "group": "nuxt/nuxt", "label": 0, "pair": "a023056d2704c727", "unit": "6028fe87bea019646fff2b31c1b2d7bd0a95eb9214fe437e2008bc5d2dfabb7d"}
{"family": "injection", "fold": "outer-2", "group": "nuxt/nuxt", "label": 1, "pair": "a023056d2704c727", "unit": "dd93069189c5f242db611e4bfba4341c22f283b65af666fc31343854358de87e"}
{"family": "output_encoding", "fold": "outer-2", "group": "oauthlib/oauthlib", "label": 0, "pair": "abc80e34d5f72367", "unit": "87d4a5b81bbfdd9628109a5f820895b6fddb00eb96b0f7bbe30a8d640457bcc2"}
{"family": "output_encoding", "fold": "outer-2", "group": "oauthlib/oauthlib", "label": 1, "pair": "abc80e34d5f72367", "unit": "b171c77b52637bc84c487d7f8bce9a54229f9dbd6605e5089e1e88943ced989c"}
{"family": "deserialization", "fold": "outer-4", "group": "opensearch-project/opensearch-ruby", "label": 1, "pair": "1945f28a3fc5bd55", "unit": "64c0b7ccad09192eeaf590befe0de0087d5b48a65a4f6cef41415f3f54a8f536"}
{"family": "deserialization", "fold": "outer-4", "group": "opensearch-project/opensearch-ruby", "label": 0, "pair": "1945f28a3fc5bd55", "unit": "b55cd6759d4b7861e589052aa534bd0554586db46a329ed3beed7244d579e8c4"}
{"family": "injection", "fold": "outer-4", "group": "openssl/openssl", "label": 0, "pair": "1c9e47835232f644", "unit": "90e220d3540b88dd9efc8e66c8ab519d4cc78805f16bed585b795c78cfbb4d58"}
{"family": "injection", "fold": "outer-4", "group": "openssl/openssl", "label": 1, "pair": "1c9e47835232f644", "unit": "b5a185b05235f70f6cf27e2adba919f4d8393dbdf2d99de095548a21946aeaf3"}
{"family": "access_control", "fold": "outer-0", "group": "openwisp/openwisp-ipam", "label": 1, "pair": "9b2b34cf62a76ea3", "unit": "91a3984e47f17512db8523e04f863525c67db18631d8f11776cf61ca7221c91b"}
{"family": "access_control", "fold": "outer-0", "group": "openwisp/openwisp-ipam", "label": 0, "pair": "9b2b34cf62a76ea3", "unit": "d91f1e14282a0c3c8da57c6935a1625e881374bd205b5baa9e4a341d9109ae21"}
{"family": "access_control", "fold": "outer-4", "group": "oqtane/oqtane.framework", "label": 0, "pair": "a20cdf9dd208720d", "unit": "18a26df68e4b81d411cfbcc8155a7e6f382753d6e1ad95eece7ed24fa8c44324"}
{"family": "access_control", "fold": "outer-4", "group": "oqtane/oqtane.framework", "label": 1, "pair": "a20cdf9dd208720d", "unit": "e6fb74441fb585d7ef5855317fba795ef0f0a0e6eae9970a49556ce3070c3254"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "oscal-compass/compliance-trestle", "label": 0, "pair": "b2e833b60a0fa396", "unit": "08ccacb7434e9f8b8b05c7bb0996be0609e34860d8fc418db329aaf8d7910f2f"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "oscal-compass/compliance-trestle", "label": 1, "pair": "b2e833b60a0fa396", "unit": "2ca7f80f0badc049ce725718e49bd60da72d316589399126a5f4c5308cf83868"}
{"family": "path", "fold": "outer-0", "group": "owen2345/camaleon-cms", "label": 1, "pair": "d6930fe7e752b7ec", "unit": "06f0341dfbf1aa9504083737df5d9be3cedfd4e3a198799d8075a29990776800"}
{"family": "path", "fold": "outer-0", "group": "owen2345/camaleon-cms", "label": 0, "pair": "d6930fe7e752b7ec", "unit": "30e2cd8bd381d7568b78eaf982f562d278a6a8d942d5282466532c1441bdf0ad"}
{"family": "access_control", "fold": "outer-4", "group": "patriksimek/vm2", "label": 0, "pair": "e9c34a6a4bf2d333", "unit": "0ae20c7334fb27889ef8551f34c65a847dffd6b37f84f722f7e8ed64fd82e95b"}
{"family": "access_control", "fold": "outer-4", "group": "patriksimek/vm2", "label": 1, "pair": "e9c34a6a4bf2d333", "unit": "59ab1924056704605b22b7c8360f40f745e7f488450be3647a632bf9a3b22699"}
{"family": "deserialization", "fold": "outer-1", "group": "pdfminer/pdfminer.six", "label": 0, "pair": "053a9e02feda3960", "unit": "34e18fe343b5c25ec582fc05ab091fd9ba160f658ba7678066c56db4991a951b"}
{"family": "deserialization", "fold": "outer-1", "group": "pdfminer/pdfminer.six", "label": 1, "pair": "053a9e02feda3960", "unit": "8fcb18a2a8654b6594cb2ec078e2a018d20f73d2b69a2e625338ffa80b3f72c3"}
{"family": "access_control", "fold": "outer-3", "group": "perses/perses", "label": 0, "pair": "2d482951d7380c4f", "unit": "5c1a721da6f85d1d5dfc8cbe5e9809b093a09c059e5abeebed2cbbee55fbd498"}
{"family": "access_control", "fold": "outer-3", "group": "perses/perses", "label": 1, "pair": "2d482951d7380c4f", "unit": "6cabd1c4b5aa94cafdd07c0c6a3f9abfdc03761ddc5b0a2bebc72a508d020b8b"}
{"family": "injection", "fold": "outer-0", "group": "pgadmin-org/pgadmin4", "label": 0, "pair": "", "unit": "2300c097374b9098671dcbcd42d02d08172e0f3064a8323223d0fe79a6106aab"}
{"family": "injection", "fold": "outer-0", "group": "pgadmin-org/pgadmin4", "label": 1, "pair": "0664148b0b192456", "unit": "d702617aaf4d19811baaa18386a3c8b0eb35cd3fd72e8d90611d34850e990858"}
{"family": "injection", "fold": "outer-0", "group": "pgadmin-org/pgadmin4", "label": 0, "pair": "0664148b0b192456", "unit": "f5ed13d811c5b16d46cbd4a3b9edb8f7d2db49acbd49d95f39f33eaab38f58f5"}
{"family": "injection", "fold": "outer-1", "group": "phpcsstandards/php_codesniffer", "label": 0, "pair": "fd5781ba332ab81e", "unit": "59eca69b2bd99ba282eb8056d7bc789e9c691eebf62a4b42594e34f5516833ae"}
{"family": "injection", "fold": "outer-1", "group": "phpcsstandards/php_codesniffer", "label": 1, "pair": "fd5781ba332ab81e", "unit": "7a7d774a6de9e4a95fe97fff822652f1f1383922f2f0a4dcb12b8e964ca330f5"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "phpoffice/phpspreadsheet", "label": 0, "pair": "366dc5ddcfe598dd", "unit": "0053d866cec1e99f8a4a8754cf8ece3c46df16e002c86703c68937d5f3a09b08"}
{"family": "untrusted_destination", "fold": "outer-3", "group": "phpoffice/phpspreadsheet", "label": 1, "pair": "366dc5ddcfe598dd", "unit": "027dbc5b9245fb804a1e2ba4445b8afa180aa2c2b3293db107ffe0ca4900ebb4"}
{"family": "path", "fold": "outer-1", "group": "postcss/postcss", "label": 0, "pair": "d0a34bf90e99e794", "unit": "4efbb75cc4105149522d2b2220987fd4373a8ef645c2c544546da540f14109eb"}
{"family": "path", "fold": "outer-1", "group": "postcss/postcss", "label": 1, "pair": "d0a34bf90e99e794", "unit": "fd790f533103fd1a5646a9374ae895cf6e36d578fc747db2b0d0249a54a85e9f"}
{"family": "deserialization", "fold": "outer-0", "group": "powsybl/powsybl-core", "label": 0, "pair": "a697aa902d141a6a", "unit": "341e8fbb46f00167a32a2d653ba09a205794084eff65afddb70f9cfff42d697e"}
{"family": "deserialization", "fold": "outer-0", "group": "powsybl/powsybl-core", "label": 1, "pair": "a697aa902d141a6a", "unit": "e29414494b275afc15f3a028f62402161df7f6da5123c302149e23845177f7f6"}
{"family": "output_encoding", "fold": "outer-4", "group": "prowler-cloud/prowler", "label": 0, "pair": "8c1fc8417190a134", "unit": "2125fb101d6108627e77f61c05cadddddfebd1cddce0e66376d9b822a2eecbb4"}
{"family": "output_encoding", "fold": "outer-4", "group": "prowler-cloud/prowler", "label": 1, "pair": "8c1fc8417190a134", "unit": "ae7bbc58026fb71f91b477946a27bbe3599be90501da9237755f3cf0fb289794"}
{"family": "access_control", "fold": "outer-3", "group": "publify/publify", "label": 0, "pair": "1972ea9f72b178b6", "unit": "3ddabb356d78e7c65ec8f88b6da8e4fcabc055ae73f2f6d92c50efddc5d4fcbe"}
{"family": "access_control", "fold": "outer-3", "group": "publify/publify", "label": 1, "pair": "1972ea9f72b178b6", "unit": "6875dd5e9c0ff8a1b693d29a109cb2e62e6ff4081df40868aca70e4770476757"}
{"family": "path", "fold": "outer-0", "group": "pypa/pip", "label": 1, "pair": "2b007a2377d633dd", "unit": "bdd71a783efb4eda9a82afbd30a16bd7608153830dbc599e65c453c18d1644b2"}
{"family": "path", "fold": "outer-0", "group": "pypa/pip", "label": 0, "pair": "2b007a2377d633dd", "unit": "f398ef33900624219e59de08e1cc101c43082a81c37d6d03a916cc220727ca67"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 1, "pair": "04aa777130c02a06", "unit": "16aa43c87956c44a4b115245140eefed704095d37cec0cc6410377403cd2fa7a"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 0, "pair": "", "unit": "23744137e0485a36f686f5f1b5666780fbed3ca522e8c6d35687a391ef504498"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 0, "pair": "67565d99fe28c68f", "unit": "2692316792e6f02fb51578cf30d71bba6c73bf5420c36f709681c8eaa69807d4"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 0, "pair": "", "unit": "2e36f73002c416a68d077bffb0335d82346c7507f2478353aeac005fce5a9cbc"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 0, "pair": "04aa777130c02a06", "unit": "67721f67055eb6fed955ff22bfe4060344662eccb369431835a2116ad120c4f8"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 0, "pair": "", "unit": "8fc1c96fa6faa4d3966ecc42876189b18ddab779d58f47cdf28a1155f48a6ac4"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 1, "pair": "67565d99fe28c68f", "unit": "ab535098a8e709b022afee1da15dd1fc371d389056938ee4fa4971b2966c823a"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 0, "pair": "", "unit": "d46f94235bc462360097e7a22e1d442d95f049306631489a4d8b25f647a95bbe"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 0, "pair": "", "unit": "edd6dd4364442f66384d14e9317ef2f6234e32ab78327a45741b8a816b4332ee"}
{"family": "injection", "fold": "outer-3", "group": "qwed-ai/qwed-verification", "label": 0, "pair": "", "unit": "fc7df6887ac719d4615934c0e8c3296728c2f6089f233158c59989027c802d72"}
{"family": "path", "fold": "outer-3", "group": "rack/rack", "label": 1, "pair": "c0412509f15fe3e1", "unit": "63bda2a71ad7155ad2f59aa9086efef3fd3710dc82c57f4b32ec8bf561460239"}
{"family": "path", "fold": "outer-3", "group": "rack/rack", "label": 0, "pair": "c0412509f15fe3e1", "unit": "b1c9a070c93764655786c3de5fde536c150e319efe47548614cc544b73a8ee2b"}
{"family": "deserialization", "fold": "outer-4", "group": "rails/kredis", "label": 1, "pair": "feed5f7de11289a4", "unit": "71531567b2b8609ae0d31cb177b8faea4f5fc61923e513b0154a2c941287ce2b"}
{"family": "deserialization", "fold": "outer-4", "group": "rails/kredis", "label": 0, "pair": "feed5f7de11289a4", "unit": "d6be6d397a301a07faa761163d3d9e270d71d23d3bdfd084eeafb1e62b91dd71"}
{"family": "access_control", "fold": "outer-1", "group": "redf0x1/camofox-mcp", "label": 1, "pair": "31c259b4e9dccd82", "unit": "1f1ddb8a5b415461638056c70756d2d0e836de71e824c0e46474f20646eaa4d5"}
{"family": "access_control", "fold": "outer-1", "group": "redf0x1/camofox-mcp", "label": 0, "pair": "31c259b4e9dccd82", "unit": "a6ec568d4eef212a842dded7fdef2284524d367e5d24473ddb34e6374be9fad7"}
{"family": "deserialization", "fold": "outer-0", "group": "redisson/redisson", "label": 1, "pair": "ffd4b04a7bebfaf8", "unit": "54f7519d3d59b6d8d69a5f8f7bd78e360d3fabaae00b7587c37e388719169370"}
{"family": "deserialization", "fold": "outer-0", "group": "redisson/redisson", "label": 0, "pair": "ffd4b04a7bebfaf8", "unit": "c5e42c9ab2ee54e46700ab0b64e239b567fbc75ccaceaf6cbf20687813663332"}
{"family": "path", "fold": "outer-3", "group": "redocly/redocly-cli", "label": 1, "pair": "2ffec9490ef1d482", "unit": "0ab883d3561162b75ad9d161f79c0d17521a23317f2f574c910465f288119483"}
{"family": "path", "fold": "outer-3", "group": "redocly/redocly-cli", "label": 0, "pair": "2ffec9490ef1d482", "unit": "b0af86c1642c14047248f1134181be0ec3839900917dffa360c5014119f8632c"}
{"family": "deserialization", "fold": "outer-2", "group": "remix-run/react-router", "label": 1, "pair": "73f2a014c6a699ed", "unit": "44ba9d9d8e75769c1444ebf37a64941a845b4bcae678ec79d64dc84e5cc067a0"}
{"family": "deserialization", "fold": "outer-2", "group": "remix-run/react-router", "label": 0, "pair": "73f2a014c6a699ed", "unit": "b430d488c486cef26b1c661f773e7e9e4a177eeea73537a033eec98d532e4eed"}
{"family": "output_encoding", "fold": "outer-3", "group": "remorses/genql", "label": 0, "pair": "738e4c22edb32172", "unit": "932d7312b2bb4b830179104bca076f587505c2d4ce881b2f04f992defba6dd9e"}
{"family": "output_encoding", "fold": "outer-3", "group": "remorses/genql", "label": 1, "pair": "738e4c22edb32172", "unit": "eee5136489c3baa3653ededbf6385d0237e2bd8b5384286780646b88ed34fb91"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "remram44/taguette", "label": 1, "pair": "cb0c638282db0d37", "unit": "a49e6ad2e7c2a5ad0d68d5d69c263306b501fc0dc9382e701fb5fe680611196b"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "remram44/taguette", "label": 0, "pair": "cb0c638282db0d37", "unit": "dee28099d3b37cbd0ee14412122ecc5f8488ed6bec01863520b87706a040fba1"}
{"family": "injection", "fold": "outer-1", "group": "roest01/node-pdf-image", "label": 0, "pair": "09bce271e8530fbc", "unit": "35eb19b4c6176b37287e2123bb98ec4045ce6bde7cc3864dde317c024b0935d8"}
{"family": "injection", "fold": "outer-1", "group": "roest01/node-pdf-image", "label": 1, "pair": "09bce271e8530fbc", "unit": "dd33291cc494bbc77d44e9b13d16f171d32c9746637506f315f375e8a9ddee95"}
{"family": "path", "fold": "outer-0", "group": "roomi-fields/notebooklm-mcp", "label": 1, "pair": "a856cd13668353a6", "unit": "6149fb0132b981620a90a23a9af8d4656d0d827403e63b48258443cacd0a9c81"}
{"family": "path", "fold": "outer-0", "group": "roomi-fields/notebooklm-mcp", "label": 0, "pair": "a856cd13668353a6", "unit": "b5aecdd6a87f8d57d28f54a013b1f96332460fe1bb44bff4ed28e166f18f9abe"}
{"family": "injection", "fold": "outer-2", "group": "rrainn/portprocesses", "label": 1, "pair": "40d8d81a764c508f", "unit": "7deeed73c29c74ccbed29be049e109d363d41e4b5eea245ee57780b85f0e137a"}
{"family": "injection", "fold": "outer-2", "group": "rrainn/portprocesses", "label": 0, "pair": "40d8d81a764c508f", "unit": "b5cdadf848e67910550666ba2a6ac67b24112e501c02dba1865766a316317a23"}
{"family": "injection", "fold": "outer-0", "group": "saltcorn/saltcorn", "label": 1, "pair": "", "unit": "15dbea384b3255457a44d9382efc6944dd27f1d411d903344a5d5e5fd194b72f"}
{"family": "injection", "fold": "outer-0", "group": "saltcorn/saltcorn", "label": 1, "pair": "b2ac284bd70dd684", "unit": "225890ba6694dc8f4fbab46e9c851dff8c095e906cf14db15d01ebce0891aaab"}
{"family": "injection", "fold": "outer-0", "group": "saltcorn/saltcorn", "label": 1, "pair": "b6eca35de93a5be8", "unit": "42eb8427415704e060e73b5ce0943b014ddf380ca756d5ec47c558e59a79895a"}
{"family": "injection", "fold": "outer-0", "group": "saltcorn/saltcorn", "label": 0, "pair": "b6eca35de93a5be8", "unit": "4c9cdcc11a18641ef41e9308712c5e8b830ec9ad5ebc3a956c8701e4f77b39c0"}
{"family": "injection", "fold": "outer-0", "group": "saltcorn/saltcorn", "label": 0, "pair": "6a53939838afbdd7", "unit": "5794809e595311c86909d673a570d003aa025e4dcd6abc5641b641d935beb0b1"}
{"family": "injection", "fold": "outer-0", "group": "saltcorn/saltcorn", "label": 0, "pair": "46d3ec7d49cb8ca7", "unit": "5f7a63ec650915962ee5255f5416df63ab882d6107f3b540be36d481798a8944"}
{"family": "injection", "fold": "outer-0", "group": "saltcorn/saltcorn", "label": 1, "pair": "6a53939838afbdd7", "unit": "97a7018535ab511622e3ade9e53d570ffb1c80633eaa372fb76e749b3a586c8f"}
{"family": "injection", "fold": "outer-0", "group": "saltcorn/saltcorn", "label": 1, "pair": "46d3ec7d49cb8ca7", "unit": "a84e9ab47f4f2b584684dbb81575758d9032fe3f2ed53bef478cb9dcfedaf677"}
{"family": "injection", "fold": "outer-0", "group": "saltcorn/saltcorn", "label": 0, "pair": "b2ac284bd70dd684", "unit": "aeceb1ebc863ebbc3b69ae56c5b78d2dba005132b75cbd6af929bb0bfe25da9b"}
{"family": "injection", "fold": "outer-0", "group": "saltcorn/saltcorn", "label": 1, "pair": "f42cab375ace1739", "unit": "e329d4a6ac757680bf3d292a81f1f74f8a8a9b482d1651efe9f503f6a4de8796"}
{"family": "injection", "fold": "outer-0", "group": "saltcorn/saltcorn", "label": 0, "pair": "f42cab375ace1739", "unit": "e73c5aac41c0b93072223a3097fb175548ce893e0678ff3a22bc7571745a17e0"}
{"family": "injection", "fold": "outer-1", "group": "sebhildebrandt/systeminformation", "label": 1, "pair": "1041a18cd1aec5f8", "unit": "c301b8c362aee3fe5ea38e18f858166bc4162ad8ace44c5fb2df192afc81da2a"}
{"family": "injection", "fold": "outer-1", "group": "sebhildebrandt/systeminformation", "label": 0, "pair": "1041a18cd1aec5f8", "unit": "ca8e58ffcfd60bdc8626db8b9eb4f253c1b8b8affcb793c6d963b7a3ec91906b"}
{"family": "injection", "fold": "outer-4", "group": "sgabe/dsvpwa", "label": 1, "pair": "", "unit": "52354ccdbc16f209e4692d97d5237e04ed030cf4a5e71f77b5a767962b355b4a"}
{"family": "injection", "fold": "outer-4", "group": "sgabe/dsvpwa", "label": 1, "pair": "7d23be667536afff", "unit": "b5016a5a53409491ba805ed8718599b84250b7b6914da0434b0882b0b3578dc8"}
{"family": "injection", "fold": "outer-4", "group": "sgabe/dsvpwa", "label": 0, "pair": "7d23be667536afff", "unit": "f1b01102968568d3ba1b2fdc70f4ae281f433fb20d29eff2a55fc45555ddbfcd"}
{"family": "injection", "fold": "outer-0", "group": "shepherdwind/velocity.js", "label": 0, "pair": "f21df949f8aca4fa", "unit": "15d23ab3aea4c878dd73bbae579f3ab473e8363477a97d28164ed5e1b94cb510"}
{"family": "injection", "fold": "outer-0", "group": "shepherdwind/velocity.js", "label": 1, "pair": "f21df949f8aca4fa", "unit": "787bc98c651ed3db21be4c28e9c560c366ffd6ca508823d21e5d83eeaf1269db"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "signalk/signalk-server", "label": 0, "pair": "1199de75f426940b", "unit": "10fdc236f43a02b7e291eb8bf6ada297a2ae98086a63b171057bc7bfc37a3377"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "signalk/signalk-server", "label": 0, "pair": "22112c31603bbe9f", "unit": "5eb3b7523b5a196dcaacba08d68457f9a45e31503ea02745ca3a7d0542f9881c"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "signalk/signalk-server", "label": 1, "pair": "22112c31603bbe9f", "unit": "6bd5bdfd81b0a3374d59cfb2c1f7fa2b27887f4672f4485e77eb5234b569983f"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "signalk/signalk-server", "label": 1, "pair": "1199de75f426940b", "unit": "c09dffd04708540845c8d1dd77a2bd989bd35f264abf652a2db55f8cb1faf7a9"}
{"family": "output_encoding", "fold": "outer-1", "group": "silverstripe/silverstripe-versioned", "label": 0, "pair": "7aab025ea7b97019", "unit": "7487a34cdfdb532a5096764a47a3e565fa509605c1df0e2b105eb9c751fc5307"}
{"family": "output_encoding", "fold": "outer-1", "group": "silverstripe/silverstripe-versioned", "label": 1, "pair": "7aab025ea7b97019", "unit": "9c0be2a58741f095fd0a9cdc60d4ca4039de1529699d5701dc4d0bcf7d5bf90c"}
{"family": "path", "fold": "outer-2", "group": "simonh1000/angular-http-server", "label": 0, "pair": "4de5441f1776f083", "unit": "a3a35d573da308cd1d0614a26616bbd37b360d9ea416894cdeffdb59abd691d4"}
{"family": "path", "fold": "outer-2", "group": "simonh1000/angular-http-server", "label": 1, "pair": "4de5441f1776f083", "unit": "dc33fc26c1bf7c67e5439ddfdad06c0faf16c796e0fe8a46b6e6a541e708676d"}
{"family": "deserialization", "fold": "outer-3", "group": "skops-dev/skops", "label": 0, "pair": "1972a9a06d7585ad", "unit": "1f472e842e894c9907536c6eb6c9c922bbb3b21c27999114588ec2a4345ab665"}
{"family": "deserialization", "fold": "outer-3", "group": "skops-dev/skops", "label": 1, "pair": "1972a9a06d7585ad", "unit": "7e38e24f1cc4d287627b7cd6cb870888865bb87fa87bba1c4fbf84eabb213121"}
{"family": "access_control", "fold": "outer-2", "group": "smithyhq/sqladmin", "label": 1, "pair": "d76d5232a8587127", "unit": "1ec32d427822d1ddd3b0d7245f5771ddeb56c146ac8ff9959dd85760ff659a20"}
{"family": "access_control", "fold": "outer-2", "group": "smithyhq/sqladmin", "label": 0, "pair": "d76d5232a8587127", "unit": "5143b19a1a1e6b9162b80594ed7fb5c0c57ed0b8abfd793bfc71089552d57638"}
{"family": "injection", "fold": "outer-2", "group": "soyuka/pidusage", "label": 0, "pair": "5f6e814444b4acd8", "unit": "2e8aa07b1aa4bfc5349b0d4464f3f2a6500aa55ff765b0139e9901ed34151ffa"}
{"family": "injection", "fold": "outer-2", "group": "soyuka/pidusage", "label": 1, "pair": "5f6e814444b4acd8", "unit": "e26c726c08be761b22fcc6ec17fc1fa7aba4769f08957f024e5545906df62797"}
{"family": "output_encoding", "fold": "outer-3", "group": "spatie/schema-org", "label": 1, "pair": "888e955fb6cfb4fd", "unit": "6f1a431ca97afc34ceec48393eb134a8928a312e6f761eb907c6b5855cac6208"}
{"family": "output_encoding", "fold": "outer-3", "group": "spatie/schema-org", "label": 0, "pair": "888e955fb6cfb4fd", "unit": "7b62415d17569924d16238a4aa28ab1af2db6561f56be9f59d8075fd1bc4a941"}
{"family": "injection", "fold": "outer-2", "group": "spring-projects/spring-data-mongodb", "label": 1, "pair": "a5691395e641fd4c", "unit": "40339c123107f1231d6aa58de876cded1dfefd5b31dadc4fc86eedc6a687165f"}
{"family": "injection", "fold": "outer-2", "group": "spring-projects/spring-data-mongodb", "label": 0, "pair": "a5691395e641fd4c", "unit": "ea3828e2f59cc6f7b415a101532b566ecbd47ab123c3c7593c23c52f6773a2e3"}
{"family": "access_control", "fold": "outer-0", "group": "spring-projects/spring-graphql", "label": 1, "pair": "f1ebdacbd647e39d", "unit": "79628ef3d6c9e12a94922cdca548ff77d3dbdd6b595c1fec5958fcf23b609095"}
{"family": "access_control", "fold": "outer-0", "group": "spring-projects/spring-graphql", "label": 0, "pair": "f1ebdacbd647e39d", "unit": "f910e010c3ff2c8c17f0f80fbe89a934e4f792b601febbf318359e6dc3afc18e"}
{"family": "output_encoding", "fold": "outer-4", "group": "squidfunk/mkdocs-material", "label": 0, "pair": "2dd5384f097711b5", "unit": "212fba6b5e50db045114af369a03ffbdf09ec898c10eeff3c45bfc01a5017b4c"}
{"family": "output_encoding", "fold": "outer-4", "group": "squidfunk/mkdocs-material", "label": 1, "pair": "2dd5384f097711b5", "unit": "370ba3a1d0e354e882b7be92c1ebbf2e2903bdf29d442048ea3dc1f80922467f"}
{"family": "injection", "fold": "outer-4", "group": "strangerstudios/paid-memberships-pro", "label": 1, "pair": "", "unit": "ac905d12b92954263c046bd0138a682cea86cf8476336d6baa664ade28d34ac6"}
{"family": "injection", "fold": "outer-0", "group": "sulu/sulu", "label": 1, "pair": "218a45d869bd7e39", "unit": "309ebac468b7ab054c8396ed495770385ae3590caba43709745ba2d1db4b6d54"}
{"family": "injection", "fold": "outer-0", "group": "sulu/sulu", "label": 1, "pair": "", "unit": "3cc8613813dec13cb1b7b45b296688da883f0b36cd2c4fc1da6e81703fb59ab9"}
{"family": "injection", "fold": "outer-0", "group": "sulu/sulu", "label": 0, "pair": "218a45d869bd7e39", "unit": "9bd2331dbc260451355307e4c37bae17ad6676654c66181934e1b35fb174cc5a"}
{"family": "injection", "fold": "outer-0", "group": "sulu/sulu", "label": 0, "pair": "", "unit": "b087ac4d0989e0b72698a9644ef4eac8073b4a15bb21730d63e4730b5cc58a60"}
{"family": "injection", "fold": "outer-0", "group": "sulu/sulu", "label": 0, "pair": "", "unit": "c5d1e8d19560c997018fdca12eb60f1799be76c33be058b88472f5a2fc6709be"}
{"family": "output_encoding", "fold": "outer-2", "group": "svg/svgo", "label": 0, "pair": "c863054be488a284", "unit": "425e1d9961b79bbc011c447cd050910dd796a32b248962a5b0acc3f5f19b7345"}
{"family": "output_encoding", "fold": "outer-2", "group": "svg/svgo", "label": 1, "pair": "c863054be488a284", "unit": "552925bba199119c7db5457b9d3c535bc9f6061cecfd2ab2f72cde690a536e1e"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "themegrill/user-registration", "label": 0, "pair": "", "unit": "22befc17a75a688e2f9358d2ddf9a7b197da44cd95dbdeae5e0191b113ee54d3"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "themegrill/user-registration", "label": 0, "pair": "63f496d9ff53e39b", "unit": "6a98f8145776c78793b37156e9a81757841bb6de80016ab4fa575209d724db3f"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "themegrill/user-registration", "label": 1, "pair": "63f496d9ff53e39b", "unit": "7ac60c96c15f8517e089cc2240aedcf1fc30e54a41e35fe03c6b965bc393fbd6"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "themegrill/user-registration", "label": 0, "pair": "", "unit": "ae39cb55ef2a87c5347e0ddc40975f142450e82de4909e93ba62a7ed0efb4134"}
{"family": "injection", "fold": "outer-3", "group": "thenables/thenify", "label": 1, "pair": "d1b19898f8deaa41", "unit": "3763c7b407ad471820b5f8fa25bde0ba6d91e15c07c56681de4b41add418b0b1"}
{"family": "injection", "fold": "outer-3", "group": "thenables/thenify", "label": 0, "pair": "d1b19898f8deaa41", "unit": "bc8b934938c30687f3d4f982392d2cfa9504a916b463ccb5c46fe8a30eede488"}
{"family": "path", "fold": "outer-3", "group": "thumbor/thumbor", "label": 1, "pair": "f5f33fa6fe7b8492", "unit": "938bbf146a9037e687190bcee82c047492dffbe90efea40a49a647f81a0f1ed0"}
{"family": "path", "fold": "outer-3", "group": "thumbor/thumbor", "label": 0, "pair": "f5f33fa6fe7b8492", "unit": "b90c9f7e0011685f1a8e853a9ed0ae37fd437ef6d6eb0ba17c240aec2c22b603"}
{"family": "deserialization", "fold": "outer-2", "group": "tomasnorre/crawler", "label": 0, "pair": "f74364dc8e35c8ba", "unit": "11e760e99c5c852676c9288e18a250febb3854e9047b51e8b9ecc9f3b423b6f5"}
{"family": "deserialization", "fold": "outer-2", "group": "tomasnorre/crawler", "label": 1, "pair": "f74364dc8e35c8ba", "unit": "5fc980194040f4c4d803075d4085618cc0260ae8f3454b26b9b704d30200ffea"}
{"family": "output_encoding", "fold": "outer-1", "group": "tornadoweb/tornado", "label": 0, "pair": "db1c51833994ec55", "unit": "c2ce8320c778dd55434d7413844797b68e6fcabc4dc509458a9d60ca70b88c60"}
{"family": "output_encoding", "fold": "outer-1", "group": "tornadoweb/tornado", "label": 1, "pair": "db1c51833994ec55", "unit": "f2d6843547af1276ae8a4dff3a7786913fa79dc69540d4fbb95c4c091e27c7f2"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "truelayer/truelayer-dotnet", "label": 1, "pair": "f4690679bf9d80aa", "unit": "1a4e24df8d08672ecde4709a77ea5c437b2198fecff8bfff373b63c9eaed6e63"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "truelayer/truelayer-dotnet", "label": 0, "pair": "f4690679bf9d80aa", "unit": "78711973ec84867671314dd7878769092b32f59860eafed50a0f6979daee76fc"}
{"family": "injection", "fold": "outer-2", "group": "typeorm/typeorm", "label": 1, "pair": "d6a42c18151efb48", "unit": "01ae2279707361bfc06a005335d41b85066d0a163802ca81c031f5dbded5bb46"}
{"family": "injection", "fold": "outer-2", "group": "typeorm/typeorm", "label": 0, "pair": "d6a42c18151efb48", "unit": "a50dc15c313a9078badea01365987c66c5cee1ad48cea65c35c8bc2cbba25e8e"}
{"family": "injection", "fold": "outer-3", "group": "ultimatemember/ultimatemember", "label": 0, "pair": "85c15e3fc2b370b2", "unit": "0990d8f41cb07c1cf217812e51a7025381de4ba68d1be3799e26241d81dcea67"}
{"family": "injection", "fold": "outer-3", "group": "ultimatemember/ultimatemember", "label": 1, "pair": "85c15e3fc2b370b2", "unit": "9da1d63f0c4da270f66eb87bf9c342ace883a0aef78c4643e254159b5da8fce3"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "unjs/ufo", "label": 1, "pair": "c7f4f4ebebfae831", "unit": "b5cbc1490bca76679adfbe5e4d9f8067d27a226d595ba401b551b00c3ce4d45a"}
{"family": "untrusted_destination", "fold": "outer-1", "group": "unjs/ufo", "label": 0, "pair": "c7f4f4ebebfae831", "unit": "ba2c1808eebdbe3f54c488b09bcb3d69e521916d1a3a0b5b14e02d5bc88bf016"}
{"family": "path", "fold": "outer-1", "group": "vercel/next.js", "label": 1, "pair": "1507c8e00ed4b859", "unit": "620e9b5ed143c0bcec178db359fa601acb1d3a134b3e85c2e838b7dcf3892d85"}
{"family": "path", "fold": "outer-1", "group": "vercel/next.js", "label": 0, "pair": "1507c8e00ed4b859", "unit": "6f5ce4008dac0471260310c60e1e2a9f76a7e4eec6ebd039835f0488cf3b19aa"}
{"family": "output_encoding", "fold": "outer-1", "group": "viewcomponent/view_component", "label": 0, "pair": "8f5a6abedd742bbe", "unit": "71e3ced643a244100cc3fde234f5b4b27655214c7ae0ff67ab61e3e48a8fdf78"}
{"family": "output_encoding", "fold": "outer-1", "group": "viewcomponent/view_component", "label": 1, "pair": "8f5a6abedd742bbe", "unit": "7c8cbfae87127042f9f43621345b4f7958bde27a3ff9db76254063e80735f3ce"}
{"family": "injection", "fold": "outer-3", "group": "vincentmorneau/apex-publish-static-files", "label": 1, "pair": "c9538342f2d7efca", "unit": "2b8db57d65ef1d67c457ff5a336e8a542d03387395d69d5e16815c4f00f40723"}
{"family": "injection", "fold": "outer-3", "group": "vincentmorneau/apex-publish-static-files", "label": 0, "pair": "c9538342f2d7efca", "unit": "e2ed113dc21ad5f7caa71ba4a5ece8f8f900a97fd0d00dc85f7ac79ff5bed19a"}
{"family": "path", "fold": "outer-2", "group": "vitest-dev/vitest", "label": 0, "pair": "494dd5d5e5cdf4b8", "unit": "3b8011f94ec721f68dd29afc6c8b01117c86a8b0c31f3650a6ce0a8d7c0739ff"}
{"family": "path", "fold": "outer-2", "group": "vitest-dev/vitest", "label": 1, "pair": "494dd5d5e5cdf4b8", "unit": "f7e2280574e9ae51daa64d8adb2955407ff6d1e156f0e2854b6c047e0a733c78"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "vllm-project/vllm", "label": 1, "pair": "51eae488182a123a", "unit": "20b039866ba796c34efa0e92468fb49a314ad7fe2914d9ff8b823e3e35ab26f9"}
{"family": "untrusted_destination", "fold": "outer-4", "group": "vllm-project/vllm", "label": 0, "pair": "51eae488182a123a", "unit": "2ab3fef05def768ef7b4165bcedcf7d200282dba78a32e19b90a0df3aee7e3e5"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "vuejs/vue-cli", "label": 1, "pair": "6f5a650865e08571", "unit": "e2051cc42e4f8b026c7807f966d492367310c90326e60fef1b0ee8c509c805a9"}
{"family": "untrusted_destination", "fold": "outer-2", "group": "vuejs/vue-cli", "label": 0, "pair": "6f5a650865e08571", "unit": "f5364d68ffe80b578e788f37f08784b0162826393b49b4a3affd70d7cc4c9e01"}
{"family": "injection", "fold": "outer-2", "group": "vvo/node-whereis", "label": 1, "pair": "8acd426c5771b37c", "unit": "983e146a60e061254b15dc8dd573cbba57f8dd0358f23ea1b9c313ef69069af5"}
{"family": "injection", "fold": "outer-2", "group": "vvo/node-whereis", "label": 0, "pair": "8acd426c5771b37c", "unit": "f80dfec45231e4778078cb10f00128ed445177cac0a062fc2df5ced5c0171f21"}
{"family": "deserialization", "fold": "outer-2", "group": "waterfutures/epyt-flow", "label": 0, "pair": "9b84800ce5c395cb", "unit": "9b7dfdceae03fd86ad3260a03f85bcdcff8f32f10c265d23743ff8dc98eeefb8"}
{"family": "deserialization", "fold": "outer-2", "group": "waterfutures/epyt-flow", "label": 1, "pair": "9b84800ce5c395cb", "unit": "b99dda150c0da3defa5f4c527ef6c402f5589dfb2f2f03f3e5e424910eb48c53"}
{"family": "path", "fold": "outer-1", "group": "webpack/webpack-dev-middleware", "label": 1, "pair": "ad64e8fac4ac3ee0", "unit": "36dfd01915a553d86576e8ecd3b737b8e2b297aa542f58cc83b448060e74398a"}
{"family": "path", "fold": "outer-1", "group": "webpack/webpack-dev-middleware", "label": 0, "pair": "ad64e8fac4ac3ee0", "unit": "c2b6d40f130d35bbfc16fab9411c664fa06321c59fb1d901a1987a69cb09c72e"}
{"family": "access_control", "fold": "outer-0", "group": "wintercms/winter", "label": 0, "pair": "d48c08b19f560848", "unit": "03d02c9e25eec0f380b125279fce35e8867065bf6e1ec75f9f39729ce6efdc77"}
{"family": "access_control", "fold": "outer-0", "group": "wintercms/winter", "label": 1, "pair": "d48c08b19f560848", "unit": "abbea69b7d0fa68c3ace442c25597e128cde354b417787b5913fd10d666c0d19"}
{"family": "output_encoding", "fold": "outer-2", "group": "xdan/jodit", "label": 0, "pair": "bfc8188bbb673e3b", "unit": "a7cff0da55c0cc1d5c838e11fc6ce6c4b22c54b9f8e101688b218d6928eca504"}
{"family": "output_encoding", "fold": "outer-2", "group": "xdan/jodit", "label": 1, "pair": "bfc8188bbb673e3b", "unit": "f741c56815230b9c30e2c8ef8952b895e9480e4780a89ff2cf6daf60d1fce7b6"}
{"family": "output_encoding", "fold": "outer-4", "group": "xyproto/algernon", "label": 0, "pair": "3844ed3d703d7ef4", "unit": "445efda074e4d9f6c1634225eb08a3c3eb1257311c995ce891df2c791a6f5c8b"}
{"family": "output_encoding", "fold": "outer-4", "group": "xyproto/algernon", "label": 1, "pair": "3844ed3d703d7ef4", "unit": "8ac27b08549353ada819caaf99a5180d2f2f515b03aa8cd99a5d37b527ea3a5c"}
{"family": "path", "fold": "outer-3", "group": "yiisoft/yii2", "label": 1, "pair": "a63eae7d621a985e", "unit": "04ca0b583b09c3b4f67bd2496a4607ecb134d24b7bdaf8960fa151e40ebcc447"}
{"family": "path", "fold": "outer-3", "group": "yiisoft/yii2", "label": 0, "pair": "a63eae7d621a985e", "unit": "ecfa2a67726ca9d3e2621a4d8be60145095b76aa18ecf0268212341d63752871"}
{"family": "deserialization", "fold": "outer-1", "group": "zumba/json-serializer", "label": 0, "pair": "0d9f529b7f0084b6", "unit": "35aa369236f510634b3aa7b54b04e9da0f80877e515786a888e760f8fe33c524"}
{"family": "deserialization", "fold": "outer-1", "group": "zumba/json-serializer", "label": 1, "pair": "0d9f529b7f0084b6", "unit": "ef4f8613c0914eb815504e00cd6ee05df38352fd3b27826798e6623594017042"}
@@ -0,0 +1,90 @@
# Experiment exp-004 (learned-decision-engine Req 6.4, amended 2026-10-03 by maintainer decision): the pooled BLOCK
# gate with a family floor. Registered before it runs; `ousast learn experiment register` records this file's sha256,
# and a changed file is a new experiment. The procedure and the decision rule are fixed here; the units are NOT: they
# are frozen at run time (`ousast learn experiment units`) from the memory after the negatives harvest (merged in
# 5b2bd9b, 1133 static examples), committed, and registered again before the run. Until then the registration row
# says `units_status: pending` and the run refuses.
#
# Scoring: kind rule, zero cost by construction -- one scoring pass of the six slice-2 incumbent programs, replayed from
# the response cache behind a zero-ceiling meter; a unit with any uncached response is excluded and counted. The
# harvest changed the memory the programs retrieve from, so most prompts are new: a paid cache fill (separately
# approved, its own ceiling, as for exp-003's supplementary analysis) must precede the run, or the exclusions will be
# most of the units.
#
# Normalisation (declared before the run): per-family Platt scaling, learn.calibrate.fit_map with its existing floors
# (own slope and intercept per family at >= 20 positive and negative repository groups; a pooled slope with a
# per-family intercept at >= 10; below that no map, so the family cannot block on that fold). Fitted on the training
# folds' paired units only. Why Platt and not rank-to-quantile: a quantile map gives every family the same flag rate
# whatever its precision, so one pooled threshold would flag the top slice of a weak family as readily as of a strong
# one. Platt puts each family on the scale of its own training precision, which is what a pooled precision bound needs.
# The map is a normaliser only: exp-003 showed calibration (ECE <= 0.05) does not hold per family, and precision is
# demonstrated here by the bound on held-out flags, not by calibration.
#
# Thresholds, per outer fold j, from the other four folds only (fold j's labels never enter a threshold or a map;
# tests/test_learn_rules.py):
# - the training units are normalised by an inner cross-fit (each training fold k by a map fitted on the training
# folds other than k), so t_block is chosen on out-of-sample scores;
# - t_block: the lowest normalised score at which the training paired units of ALL families together with p >= t
# and a verdict other than `unsure` have a Wilson 95% lower bound of precision >= 0.95; none -> no BLOCK on fold j;
# - t_quiet, per family: the highest normalised score keeping >= 90% of that family's training paired positives at
# or above it;
# - fold j's units, normalised by the map fitted on all its training units: BLOCK when not `unsure` and p >= t_block;
# ADVISORY when p >= t_quiet (or the family has no map or no t_quiet: nothing is silenced blind); silent below.
#
# Family floor F = 10: a family may block only if its own held-out paired flags (all five folds) number >= 10 with zero
# errors; otherwise its flags are reported as ADVISORY. Why 10: zero errors in 10 flags is unlikely for a family that is
# much worse than the bar (a family at 0.75 precision produces 10 clean flags with probability 0.056, at 0.80 with
# 0.107); a larger F is out of reach of the smaller families (16-19 paired positives in exp-003's units, all folds); and
# 10 is exp-003 arm C's training minimum. The floor uses held-out labels to decide which families block; it moves no
# threshold. The primary metric is computed BEFORE the floor, over every family's held-out flags, so dropping a failing
# family cannot raise it.
#
# Power note: the pooled bound needs >= 73 flagged units with no error (110 with one, 142 with two). exp-003's
# instrument check found few positives above the first negative per family on the raw score (paired: 10, 1, 3, 0, 1);
# per-family normalisation keeps each family's order, so the pooled top set can reach 73 only if the harvest's pairs
# separate better. An inconclusive verdict is a plausible outcome and will be reported as such.
id: exp-004-pooled-block-gate
hypothesis: >
One BLOCK threshold pooled across families on a per-family Platt-normalised score, chosen on training folds only as
the lowest one whose Wilson 95% lower bound on flagged precision is >= 0.95, reaches that bound on held-out
repositories' paired units, where per-family thresholds (exp-003) could not for lack of flags; a per-family floor
(>= 10 held-out flags, none wrong) keeps a family that the pooling masks from blocking.
kind: rule
programs: # the slice-2 incumbents, as exp-003
access_control: 745cb1d76bbb09dc1574d333d2118489be62e39886517894d3699fa565b0e8c2
deserialization: 3c0c2831be7d78198a62351661c37867363c501cb40e3ed7b0119c155af09654
injection: b207a0710e0abf053b3b9397b25f9ee1cab4679144865e3a82c1b9be2c7b4982
output_encoding: 5d768f16724562c381c09949ff3df9cbb11d58cbb7a9c0515d0dc3f858d3818e
path: aea396680a4874dcb6de70e240ca0480c83424debdd3710bc3f1ea6a612661f3
untrusted_destination: 1242b3973fc89056aa08e00843043d68235df217d8e3ef8a0ea0e0b81c142753
arms:
A:
label: today's rule -- Platt map cross-fitted per family, BLOCK only when the calibration check holds, BLOCK point nested on the other folds' calibrated p
rule: {name: calibrated_block}
B:
label: pooled gate -- one t_block over all families on a per-family Platt score (training folds, inner cross-fit), t_quiet per family at 90% recall, family floor 10 clean held-out flags
rule: {name: pooled_block_gate, precision: 0.95, recall: 0.9, floor: 10, normalise: platt, select_on: paired}
unit: candidate
pairing: paired # every unit scored once; every arm decides every unit
cluster: repository
order: {randomise: none, seed: 4}
# primary population: paired units (both sides of the candidate present); pooled (every unit) is reported beside it
populations: {primary: paired, secondary: [pooled]}
metric:
primary:
- {name: flagged_precision_lb, target: 0.95} # held-out Wilson 95% LB of flagged precision, all families pooled, arm B, paired, before the floor
secondary: [flagged_precision, coverage, flagged, quiet_recall] # quiet_recall: held-out positives at or above t_quiet / positives
# Decision rule (decide_pooled), on arm B's held-out paired flags of all families together, before the floor:
# REJECT B if their point precision < 0.95;
# else ADOPT B if their Wilson 95% LB >= 0.95 and at least one family passes the floor (>= 10 held-out paired
# flags, zero errors) -- those families block, every other family stays advisory;
# else INCONCLUSIVE (the incumbent rule stays).
# Reported, not deciding: per-family held-out precision and LB, the floor table, quiet_recall per family against 0.90,
# the after-floor totals (what users would get), the repository-cluster bootstrap interval.
decision: {arm: B, target_precision: 0.95, pooled: true}
test: {resamples: 2000} # repository-cluster bootstrap of the held-out flagged precision, beside the Wilson bound
mde: 0.0 # not a paired difference test; see the power note above
budget_usd: {total: 0} # the run itself is zero cost; the cache fill before it is a separate, approved spend
families: [access_control, deserialization, injection, output_encoding, path, untrusted_destination]
folds: {kind: outer, k: 5, seed: 0, compile_fraction: 0.25} # the incumbent evaluation's folds (learn/compile.toml seed)
units_file: plane/experiments/exp-004-pooled-block-gate.units.jsonl # frozen at run time, after the negatives harvest

Some files were not shown because too many files have changed in this diff Show More