learn/sources.toml is the fail-closed source list: every read goes through one guard
that refuses an unlisted path or an [[excluded]] file name before opening it, and a
source whose evaluation record is missing cannot label. Population v3 is named only
as excluded, by file name, and is never opened (tested with a v3-named fixture and
open/read_text tracked).
learn/labels.py and `ousast learn labels` apply the design's rules: v2 declared
sites and v1 fix-range functions positive, the same function on the fixed pin
negative only when the fix changed it, introduce/repair delta units, benign control
pushes as conditional negatives; pairs vulnerable/fixed (unscorable and title tier
out, title as a sensitivity arm); dev-php recipe known functions; recorded
adjudications with their reasons. v2 rows are labels marked design_informed, as the
design says. Repository groups: normalised owner/name, merged by shared CVE/GHSA,
shared fix commits and [[alias]] rows.
assumed_benign (maintainer decision): ordinary commits of the population
repositories, >= 90 days from any fix, no later security commit touching their
files, weight 0.5, conditional, reported separately and never merged with verified
negatives. The security filter is multilingual (CJK, Korean, Russian, European and
other languages) plus CVE/GHSA/CWE references and dependency bumps; widened after
reading accepted pushes (validation, passwords, guards).
First snapshot, counts only (benchmarks/measurements/2026-09-30-decision-engine-
labels/labels.json): verified pin labels in positive/negative repository groups
injection 44/41, untrusted_destination 35/34, deserialization 20/20 (per-family),
output_encoding 15/15, path 15/14, access_control 12/12 (pooled), config_secrets 5/5,
memory 5/5 (insufficient). assumed_benign: 969 pushes in 27 groups from 36,000
commits scanned.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Wrapper inference (learn/roles.py, no model): a function whose parameter reaches a
call already in the role set becomes a derived sink (or a sanitizer when it returns
the cleaned value) of the same operation, at depth + 1; a function returning a
source's value becomes a derived source. The role set starts from language-level
entries only (priors off by default, optional), grows to depth 3, and reads
vendored dependency source when present, so a vendored database class reaching
mysqli makes `->query` a sink without a framework table. A name the repository
declares more than once becomes a role only when every definition derives it.
Model roles (plane/tasks/roles.py): the model_sinks.py classification moved into
the package and extended to sink/source/sanitizer/guard/dispatch, per file chunk,
metered and resumable, client injected; a failed chunk is unclassified, never
"no roles". vocabulary_overlay merges inferred and model roles over the
language-level facts in the shape taint_specs takes, origin in the entry id.
Features: the roles (static) and model_sinks (plane) instruments are filled.
On the recorded validation-46 facts (15 cases, 14,368 product files, 87 MB read):
1,999 wrapper roles (1,753 sinks, 157 sources, 89 sanitizers; depth 1/2/3 =
973/651/375), 14 of 15 cases with at least one; the WordPress gopay plugin has
none with priors off, which is what model roles are for.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ruleset/frameworks.toml lists 19 framework and library ids (wordpress, flask, django, express,
spring, jakarta-servlet, react, vue and 11 libraries), each with its languages, dependency
package names and a symbol lexicon used only by the lint and for stratification.
Every framework/library entry of section 2 is tagged `framework`/`library`, and mixed entries
are split so a tag covers a whole entry (split pieces keep the entry's id and flags, in place):
semantic facts 38 tagged entries (wordpress 6, django 6, flask 4, express 4, jakarta-servlet 3,
pyyaml 2, one each for spring, sqlalchemy, requests, httpx, lodash, node-http-clients,
escape-html, sanitize-html, dompurify, esapi, commons-text, react, vue); the dispatch fact
behind the hook-as-source path is wordpress; the PHP layout row is renamed php_layout and stays
untagged (a product filter). Quick rules: 9 tagged (wordpress 6, flask, pyyaml,
node-http-clients); php-unserialize is split into php-unserialize and php-maybe-unserialize;
the lint also caught WordPress escapers in exclusions and axios/yaml.load in mixed rules, so
those keep their ids as tagged rules and gain a `fallback_for` language-level rule (6) that is
active only when the prior is off.
load_facts and load_ruleset take priors ("all" by default -- today's scan --, "off", or a set
of ids). tests/test_semantic_priors.py: the lint (no untagged entry contains a lexicon symbol
of its language), "off" removes exactly the tagged entries and activates the fallbacks, split
entries keep their language-level calls. Rule and layout counts in three existing tests follow the split and the rename.
Unchanged defaults: test_benchmark_output_is_byte_identical_to_golden_baseline passes, and the
quick findings of all 15 benchmark manifests (full StaticFinding records) are byte-identical
before and after. priors=off on the same manifests: 61 -> 57 findings; the 4 lost are js-ssrf
(axios), php-wpdb-sql-composition, python-ssti and python yaml.load; 5 PHP and 1 Python finding
move to their fallback rule at the same location. Semantic calls with priors off: php 75/92,
python 32/77, javascript 42/70, java 24/41, c 27/27 (the engine-side effect needs Joern and was
not measured).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/schema.py is the closed, typed allow-list of candidate features: 66 features over
11 instruments (language one-hot, quick rules with 23 mechanism buckets and a prior count,
engine, repo-facts, source span, entry-point distance, roles, delta, verify, agree, model
sinks), two profiles (static 57; plane adds verify, agree and model sinks for 66), schema_version and a feature-set digest.
No repository, path, file, function, rule or commit identity is a feature; site_match,
in_fix_range, fixed-pin alerts, case.json fields and costs are refused. Missing is not zero:
each record carries per-instrument state (ran | none | failed | not_applicable), and every
feature of an instrument that did not run is null.
learn/features.py builds records on the host (build_for_scan, whole repository or base..head
delta); plane/tasks/features.py builds them per case in a Run from the delivered artifacts,
dropping site_match and in_fix_range on read and ignoring fixed-pin alerts. The generator
adds a model-free `features` entry per case before `remember`, which stores the records as
`features` rows (re-validated). memory KINDS gains features, label, decision, experiment,
arm_outcome and experiment_result. The committed validation-46 manifests are regenerated
(entries appended; the first Run's entries are unchanged). The runner image must be rebuilt
and re-pinned before a Run executes the new task.
Proof on real artifacts (scratch store, not committed): the recorded validation-46 run gives
75 feature records over 15 cases (43 verify candidates plus vulnerable-pin alerts), each
case's vulnerable pin exported and read (0.5k-9k files); remember stored 75 features rows
beside 43 verdict, 84 alert, 61 unit_cost, 15 facts and 8 coverage rows.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Three instruments scored separately at one analyzer commit (quick PHP rules
with a fixed rule-to-family table, the Joern engine, the plane pipeline
triage -> verify a/b -> tie-break -> 2-of-3), protocol v2's six pins, one
matching rule for all six families (declared site or fix range, same family;
evaluate.matches_v2 unchanged), per-family meaning of a match, the M4 gates
copied from protocol v2 and applied per instrument, a $60 total / $6 per case
model-spend ceiling, and the retune rule.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
freeze.py, extended for v3 and still reading only git history, file lists and
fix diffs:
- a file the advisory names without a function resolves to the function(s)
the fix's diff changes at the vulnerable pin (maintainer's decision 2),
recorded with its basis (advisory-named / resolved-from-fix-diff); a new
helper inserted at file scope adds no site;
- an inferred fix must change a declared site function, else the case drops;
- a single-language population gets a strict benign proposal (PHP change,
merges judged with all they bring in, pre-vulnerable history as fallback).
All 15 cases verified, none dropped (pagible and podlove fixes confirmed).
Nine sites in eight cases resolved from diffs; five benign proposals
replaced by hand.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer's selection 2026-09-30, "Recommended 16, minus v2 mentions": the
research's rule-based 16 with Winter, elFinder and Easy!Appointments replaced
by the next candidate in their family by the same rule (named fix first, then
newest first), v2-mentioned alternates (Grav, Backpack CRUD, Koel) skipped.
Winter -> mantisbt-tokenizer-eval-rce; elFinder -> pagible (inferred fix, to
be confirmed at freeze); Easy!Appointments -> empty slot (no remaining
untrusted_destination candidate), so 15 cases are reserved.
The reservation guard now covers v3: all six families, a declared-language
population, status "reserved".
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Quick mode had no PHP rules, so a PHP quick scan read the files and reported a plausible zero.
ruleset/php/rules.toml adds 13 line/statement rules built on semantic/php.toml's sinks, sources and
sanitizers (sql/wpdb, command, code_eval, unserialize, echo, file_inclusion, filesystem,
outbound_request, redirect). 9 are enabled and 4 are shadow (sql-call-composition, code-eval,
unserialize, xss-echo-input). A rule is shadow when it fired on the corpus but never at a known site.
precision_estimate is the measured lower bound.
Measured in-sample on the development corpus only (pmpro introduce/repair/benign, population-v1 PHP
cases; population-v2 was not read). Known site hit on the vulnerable pin in 4 of 5 cases (LearnPress
missed). The hit disappears on the fixed pin for pmpro and YesWiki. Numbers, per-pin files/bytes read
and the engine comparison are in benchmarks/measurements/2026-09-30-php-quick-rules/.
- fixtures php-vulnerable / php-benign plus manifests: ousast benchmark covers PHP
(9/14 reported, 13/14 including shadow; the benign twin records 2 known false alerts)
- PHP `#` comments are comments; `#[...]` attributes are not
- the alerts task now reports PHP as quick coverage; alerts-engine keeps running the engine for PHP
next to the quick rules (ENGINE_ALONGSIDE_QUICK)
- other languages' benchmark findings and metrics are unchanged
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Quick mode has rules only for c/cpp, groovy, java, javascript/typescript and
python, so every PHP case's "0 alerts" read as clean. The `alerts` task now
records per language of the files read whether quick mode covers it
(derived from the bundled ruleset, not a list): coverage quick | none, with
file counts per pin, and `uncovered`. `remember` turns that into `coverage`
memory rows (a new kind; the Run passes the alerts summary.json as input),
loop-measure reports coverage per language, and improve/memory.py's M1/M2
skip a repository+pin whose coverage for the rule's language is only
`none` (counted under `coverage_none` in the proposal's excluded).
`ousast plane alerts-engine <Run.yaml> --source <frozen export>`
(plane/engine_alerts.py) does the alerts task on the host for cases whose
files include a language the engine covers but quick mode does not (PHP):
exports both pins from the case cache, runs the quick scan and the
openultrasast:dev engine container (finding_dump.py, --network none,
--memory 3g, one at a time, per-pin deadline), writes engine findings in
the alerts.jsonl row shape (rule_id engine:<family>, source engine,
in_fix_range from case.json) with a summary (files read, questions,
completed, seconds, degradations per pin), and marks the task done via
reconciler.mark_done. No result, no files read, or questions == 0 fails
loudly and leaves the run untouched. Refuses a git checkout as source.
Runner image rebuilt and re-pinned (sha256:ebb40a9a...); validation-46
regenerated (digest only). Live, recorded in
benchmarks/measurements/2026-09-30-php-engine-alerts/: froxlor 366 files,
130/130 questions per pin, 6 alerts per pin, 0 in the fix range; wp-gopay
19 files, 101/101 questions, 0 alerts. The loop: 32 run rows (coverage 8,
php=engine), 0 proposals, gate no_proposals, $0.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reran benchmarks/harnessx_removal_equality.py record from a git archive
export of b9afb66 (tasks 1-6 landed; same venv freeze hash as the
baseline, harnessx not importable, no API keys, empty config) and
compared it against 2026-09-30-harnessx-removal-baseline:
equal=True differences=0
All 13 output files (pairs.json, benchmark, standard scan, improve
--dry-run) are raw-byte identical to the baseline. The suite map
(suite-map.json) accounts for 29 deleted ids, 2 renamed ids and 83
added ids; each deleted or renamed id cites the design's Tests table and
commit cbadc38 or b9afb66. Every surviving test id keeps its outcome
(1740 passed, 38 skipped incl. the README xfail).
Instrument note: a first record failed test_push_hook twice because the
recording shell's PATH lacked .venv/bin (the hook execs `ousast`); the
same two tests fail at b9afb66 without it and pass with it. The
committed record has .venv/bin on PATH.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Deletes harness_ext.py and hunter_harness.py with their HarnessX-only
tests (the judge path, verify_judge.py, went in task 5). The harnessx
extra, its mypy override and its uv.lock entries are removed (uv lock
drops only packages; no remaining version changes). The conftest
fixture becomes assert_cold_import; test_gate's two never-import checks
become one allow-list (stdlib, openultrasast, yaml) over the governance
planes' module-level imports, and the skipped live-extra tolerance test
is deleted. Audit rows for the two modules are removed, stage_processors
is reworded, and the module-audit manifest is regenerated (no orphan).
test_removed_plane_references implements Req 2.4 over src, tests and
pyproject.toml: a match must carry a `retired 2026-..` tag (the spec's
own slug is not counted). The README part is a strict xfail until task 8
rewrites the README's plane sections.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The standard scan now runs run_hunter_pool, verify_findings and the
deterministic fuse_findings directly; the HarnessX hunter orchestrator,
the LLM judge dispatcher (verify_judge.py, deleted with its test) and the
LLM fusion panels are gone, and so are their extra-absent degradations.
host_under_harnessx is dropped from stage_processors.
config: [models] verifier and [fusion] panel_model/decider_model raise
RetiredConfigError naming the plane replacement (the CLI prints it and
exits 2); a [harnessx] section loads with one warning and is otherwise
ignored. harness.json keeps "verifier": null. Tests that asserted the
fallback are renamed to the behaviour they assert; the module-audit
manifest drops verify_judge. Nothing outside the two modules deleted in
task 6 imports harness_ext or hunter_harness.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
`ousast plane workspaces --validation-set ... --loop` appends the loop to a Run:
per case `alerts` (plane/tasks/alerts.py: the quick-mode scan on the vulnerable
and the fixed pin, a fixed-pin Workspace per case, alerts attributed to their
enclosing function by the repo-facts patterns, `in_fix_range` from the fix's
old-side and new-side ranges, failing on an unread tree) feeding `remember`,
then `memory-snapshot` -> `loop-measure` -> `loop-propose` -> `loop-improve`
(plane/tasks/loop.py). Every new task binds no Model and has budget
{usd: 0, calls: 0}. The snapshot is the store's rows after the train-on-test
guard, written on the host by memory.seed (a sandboxed run of it fails loudly);
propose runs M1/M2 against this repository at the commit under test; improve
runs one unchanged evolve.run_round and delivers gate.json, journal.json, the
sidecar, rule_policy.json only when accepted, and proposal_outcome rows that
the host ingests. Adoption stays a maintainer commit.
`remember` is now a Run entry per case (the image contains it), with the facts
row keyed by the image the facts were computed on (repo-facts records it in
its summary; unknown -> the row is omitted, never guessed). Alert rows carry
their own pin; improve/memory.py counts a repeated alert once.
Runner image rebuilt (+0.39 MB: only the new modules; the quick scan needs no
extra and the rulesets were already in the package), pushed, plane/ re-pinned
and regenerated (validation-46: 105 entries). Fixture test runs the whole loop
graph through the fake ax, whose new `exec` mode runs the real task modules.
Live: one loop Run over three validation-46 cases on the local ax, recorded in
benchmarks/measurements/2026-09-30-harnessx-removal-loop/ (all done, $0 new
spend, 0 proposals, gate verdict no_proposals).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
improve/memory.py: rules M1 (enabled->shadow on repeated false alerts across
>=2 repositories, no agreed or site_match hit) and M2 (shadow->enabled on
>=2 agreed declared sites no enabled rule catches); M3 disputed families are
advisory (signals.json) and never reach the validator. The train-on-test
guard drops rows of --qualify-population, gated benchmark cases and holdout
pair repositories before any rule sees a row. run_round(proposals=...) adds
memory edits after the benchmark's through the unchanged validator and gate;
provenance (row ids, object keys and versions, counts, exclusions, index
digest) goes to memory_proposals.jsonl and the journal's evidence field;
proposal_outcome rows are written back after a non-dry run.
ousast improve --memory is opt-in; default improve output is byte-identical
to the task-1 baseline.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
plane/memory.py: the MemoryStore interface with FileStore (index.jsonl, facts by
content hash, repos/<host>__<owner>__<name>/<pin>.jsonl, idempotent ingest,
deterministic row ids, refusal below 1 GiB free naming the path) and MinioStore
(object metadata and tags, S3 Select pushdown probed once per process with a
logged local fallback, object version ids on every row read, versioning and
runs/ lifecycle helper, presigned URLs). OUSAST_MEMORY selects the backend;
MINIO_* settings come from .env; the minio SDK is the optional extra `minio`.
plane/tasks/remember.py: model-free rows (facts, verdict, unit_cost, alert)
from a case's artifacts; remember_run computes them on the host for a recorded
run without writing into it. reconciler.mark_done (477 lines total), seed
before and ingest after `ousast plane run`, `ousast plane remember <run>`.
The generator annotates each repo-facts Task with openultrasast.io/memory-key
(repo, pin, candidates digest, image digest) and the Run with its population
and split; plane/ regenerated.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Equality script (design section 3) records suite outcomes, `ousast pairs --json`,
the python-vulnerable quick benchmark, one standard scan and `improve --dry-run`
from a frozen export with the harnessx extra absent and no model reachable.
Two independent records are byte-identical (determinism.json).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A third verify pass on the 10 disputed candidates, 2-of-3 agreement (Req 6.4),
run on ax by extending the same Run: 16/20 declared sites agreed (reference 16),
$0.0205 per candidate with recorded triage (reference $0.022). Caveats recorded:
a design change on the same set; 17 vs 19 sites flagged in any pass; three
non-declared pgadmin candidates promoted.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
agree's declared_sites_matched counts candidate coverage, not detection. Recounted
from agreed.json rows, plane passes flagged 17 of 20 declared sites against the
reference's 19 (budibase read and fuxa getDaqValue missed by both passes). The
earlier "detection unchanged" reading was wrong.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
validation-46 ran done on google/ax: 60 tasks, 357 DeepSeek calls, $0.83 with
recorded triage ($0.0194 per candidate, reference $0.022: cost met). Declared
sites agreed 14/20 against the reference's 16/20: recall criterion not met.
Both runs find 19/20 sites in at least one pass; the two-pass agreement
flicker moved three sites. Design revisit before task 9.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Agent Substrate's egress gateway denies an actor without an EgressPolicy, so
no task could deliver its artifacts or reach a model. egress.py grants each
task exactly: http/80 to ousast-receiver.ax-system.svc.cluster.local, and
TLS passthrough/443 to the bound Workspaces' https Git hosts and the bound
Model's openultrasast.io/egress-hosts (api.deepseek.com for deepseek-flash).
No IPs, no wildcards, no https (MITM) rules.
The reconciler writes the policy with kubectl-ate between ax apply and
resume, reads it back, waits OUSAST_EGRESS_SETTLE (11 s, the gateway's
cached deny), and after ax delete task confirms the policy went with its
actor (kubectl-ate has no delete verb; the store cascades).
The receiver is a selector-less Service (ops/ax/receiver-service.yaml.tmpl,
applied by up.sh together with the no-build agentgateway egress gateway)
on port 18090 of the host. Measured live: the gateway checks the policy on
Host but connects to the address the actor dialled, so the runner dials the
Service's ClusterIP (OUSAST_ARTIFACT_DIAL) with the receiver's name as Host.
The runner now labels its delivery with the Run's task name (OUSAST_TASK),
which is what the receiver accepts; the ax Task name was refused with 403.
The Receiver moved to egress.py to keep reconciler.py at 460 lines.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Agent Substrate's golden-snapshot boot runs the same image with the same task env,
so the runner no longer starts its command by itself: after the workspaces are
ready it waits for POST /ousast/v1/start (run/task checked, once per boot, refused
after a delivered run). The reconciler sends it through svc/atenet-router with
ate-target-actor, which never addresses the golden actor (plane/router.py), and the
bound Model's secretKey variable travels only in that body, into the child's env.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The first live run on the local ax cluster and ax's source (docs/runner.md,
docs/sandbox.md, internal/controller/reconciler.go, internal/server/server.go)
showed a lifecycle the task-3/4 code did not model:
- `ax apply` creates a Task Suspended (Ready=False, TaskSuspended); it runs only
after `ax resume task`. The first resume of a new image can fail with
DeadlineExceeded while Agent Substrate builds the golden snapshot, and ax's
server then stores phase Failed; a later resume succeeds.
- ax has no Completed phase and never reads a command's exit back.
- The runner is PID 1 and must keep serving after the command exits; stop and
suspend deliver SIGTERM, to be forwarded to the command's process group.
Our runner imported the task in-process and exited, which stopped the
container ("Container guest is stopped") and made every resume time out.
- ax's controller probes /readyz?check=workspace; the query string made our
handler answer 404.
Runner (Req 4.1-4.4): spec.command runs as a child
`python -m openultrasast.plane.tasks.<name>` in its own process group, cwd the
first workspace, AX_METADATA_URL set; ax's metadata paths are served. After the
child exits the runner reads summary.json (a crash without one becomes failed
with the exit code and stderr tail), delivers the output directory, writes
/workspace/.ousast-state/<task>.done.json and serves until SIGTERM (forwarded
to the group, SIGKILL after OUSAST_TERM_GRACE, exit 0). A boot with the marker
never reruns the command; a failed delivery is retried alone. Workspaces are
prepared on the first boot only. AX_RUNNER_EXIT_AFTER_COMMAND=1 is the
unit-test escape.
Reconciler (Req 2.2, 2.3, 3.1-3.3): `ax resume task` after apply, retrying
DeadlineExceeded/Unavailable with backoff up to OUSAST_RESUME_TIMEOUT (900 s);
completion is the artifact delivery; Failed after a resume went through, a
vanished task, or no delivery within OUSAST_TASK_TIMEOUT (7200 s) fail the
task with ax's condition message; `ax delete task` follows. Ax gains resume()
and phase() returns the condition message. doctor moves to plane/doctor.py
(tasks.md task 0 allows the sibling) to keep reconciler.py under 500 lines.
Tests: the fake ax follows the real lifecycle (Suspended, a failing first
resume that leaves Failed, Running forever, delivery only after a resume went
through), plus Failed-with-message, vanish, resume-timeout and task-timeout
scenarios; runner tests cover the child's cwd/group/metadata URL, stay-up and
SIGTERM in a real subprocess, SIGKILL of a SIGTERM-ignoring group, and the
marker. The opt-in live test now checks resume, Running, suspend and a second
resume instead of a Completed phase. design.md and tasks.md state the
lifecycle; the module audit manifest counts plane.doctor.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Model deepseek-flash (declared provider extension, endpoint prices in
parameters, secretKey naming the DEEPSEEK_API_KEY secret), digest-pinned
Task templates repo-facts / verify / agree, and a generator
(plane/generate.py, `ousast plane workspaces --validation-set ...`) that
emits per case the vulnerable-pin Workspace, a files-only inputs Workspace
(recorded candidates after the recorded triage, declared sites, triage),
per-case Tasks and one Run validation-46: facts -> verify a, verify b ->
agree for the 15 cases, 46 candidates (43 kept), budgets 1.5x the recorded
per-pass share. A Run entry cannot add a binding or env, and Run.inputs
names only task artifacts, so the case is fixed by per-case Tasks.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ai-service-plane task 3, Requirements 4.1-4.4 (with 2.2 and 2.3 honoured by the
exit-code contract):
- src/openultrasast/plane/runner.py: the ax-task-runner entrypoint. Reads the Task
from AX_TASK_YAML and the bound Workspaces from AX_WORKSPACES_YAML (task-1 loaders),
serves /healthz and /readyz on port 80 (AX_RUNNER_PORT; 0 picks a free port,
AX_RUNNER_HTTP=0 disables it for unit tests), materialises each workspace at its
bound path (spec.files verbatim; spec.git at <path>/<name>, exported from the
OUSAST_CASE_CACHE checkout whose origin matches, else a shallow clone at the pin;
a repo already present on a resumed volume is kept) and only then answers 200 on
/readyz. Runs spec.command by importing openultrasast.plane.tasks.<name> with the
remaining elements as argv; sets OUSAST_WORKSPACE_DIR from the first binding when
the Task's env leaves it unset. A crash writes summary.json with status failed and
the traceback. When OUSAST_ARTIFACT_URL is set, the whole OUSAST_OUTPUT_DIR is
POSTed as an uncompressed tar (Content-Type application/x-tar, X-Ousast-Task,
X-Ousast-Run) with three attempts and backoff; a failed delivery rewrites
summary.json to failed and exits 2. Exit codes: done 0, failed 2, unfinished 3.
- plane/Dockerfile.runner: python:3.12-slim + git, `pip install .` (no extras), a
shell wrapper at /usr/local/bin/ax-task-runner exec'ing the module; built and
loaded into kind by ops/ax/up.sh, not by the tests.
- tests/test_plane_runner.py (+ tests/plane_stub_task.py registered as the task
module): env contract on a files workspace, readyz 503 then 200, delivery to a
local receiver with the headers and tar checked, delivery failure -> failed and
exit 2 after three attempts, crash -> failed with traceback, exit codes per
status, case-cache export at the pin, shallow-clone fallback, resume keeps the
checkout.
- model/audit.py: SPEC_OWNED row for plane.runner; module-audit measurement
regenerated (122 modules, no orphans, note kept).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Requirements 3.1, 3.2, 3.3, 3.4, 3.5, 2.2, 7.1, 7.3 of ai-service-plane.
- plane/reconciler.py (492 lines, test-counted): a Run's tasks submitted to ax
in dependency order through the injectable `ax` CLI (apply / get task /
delete task), serialize labels never overlapping, --workers, state.json with
pending|running|suspended|done|failed|unfinished, a PID lock per run, done
tasks skipped on rerun, unfinished tasks holding their dependants, a failed
task halting the run. Budgets, inputs, output dir, artifact URL and the bound
Model (annotation openultrasast.io/model; identity and priced parameters) go
to the task as env in the rendered Task; consumed artifacts travel as files
of a generated Workspace bound under /workspace. A threaded HTTP receiver
takes each task's tar delivery, validating run/task and rejecting traversal.
Outcome comes from the delivered summary.json (done only when every unit
finished), never from ax's phase alone. status prints the token attribution
table per task and Model from the summaries' usage fields (n/a, never an
estimate) and writes attribution.json; doctor runs the four host checks.
- plane/workspaces.py: one Workspace manifest per case pin of a population.
- CLI: ousast plane run|status|doctor|workspaces.
- tests/test_plane_reconciler.py: a fake ax executable drives the DAG;
tests/test_plane_ax_live.py (marker ax) applies ops/ax/smoke-task.yaml on a
live cluster and skips unless doctor passes.
- Module audit rows and regenerated manifest.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Requirements 5.3, 2.1, 2.2, 2.3, 7.2 of ai-service-plane.
- plane/tasks/verify.py: the batched hunt of verify_batched.py moved into
the package (system prompt, one hunt per file, up to 6 candidates, 6
steps, candidate-anchored sites). New against the reference: each
candidate's known callers from facts.json in the prompt (capped at 8),
tool turns and usage per hunt in units.jsonl, per-unit resume (rows
with "error" are re-asked), pass label from env, MeteredClient wrapper:
BudgetExhausted -> unfinished (exit 3), AccountError -> failed with the
provider message (exit 2); summary.json is written last, names the bound
Model and the summed usage, and is done only when every unit finished.
- plane/tasks/agree.py: verify_batched.agreement over two pass outputs,
agreed.json / disputed.json, per-candidate cost and turns, declared-site
match via benchmarks/independent/evaluate.matches_v2 loaded by file;
metrics block in summary.json; no model calls.
- tests/test_plane_verify.py: a scripted usage-recording hunter client;
callers line, resume, 402, budget stop and resume, model and summed
usage, unpriced usd None, both main() env contracts, agreement metrics.
- model/audit.py + module-audit manifest: SPEC_OWNED rows for plane.tasks,
plane.tasks.verify, plane.tasks.agree; manifest regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
plane/tasks/repo_facts.py computes, from source text only and without a
model or git, the product files of a workspace (tests, vendored trees,
node_modules, dist/build, minified bundles and dot-directories excluded via
the layout facts plus the model_sinks.py filter), the functions each file
declares (Python, PHP, JavaScript/TypeScript, Java, Go, Ruby declaration
patterns extended from sink_candidates.py) and, per candidate function, the
call sites in other product files with path, 1-based line and enclosing
function, capped at 40 and sorted (Requirement 5.1). Output is byte-identical
across runs over the same tree (Requirement 5.2).
Under the task contract the unit is one file: a row goes to units.jsonl as
each file finishes and a restart reuses the rows computed for the same
candidate set (Requirement 2.2); facts.json is assembled from the rows and
summary.json is written last, refusing `done` unless every unit finished.
The entrypoint reads OUSAST_WORKSPACE_DIR, OUSAST_OUTPUT_DIR and optional
OUSAST_INPUT_CANDIDATES only (Requirement 2.1) and exits 0 on done, 2 on
failed with the traceback recorded in summary.json.
tests/test_plane_repo_facts.py: a three-file tree with a tests/ file, a
vendored file, a node_modules bundle and a dot-directory; cross-file callers
with line and enclosing function, same-file and declaration lines excluded,
the 40 cap, byte-identical second run, resume that recomputes only the
missing unit, a crash that leaves `failed` with the unit counts and resumes
to `done`, and the declaration patterns per language.
The two new modules are registered as spec-owned in the module audit (their
importer, the runner, lands in task 3) and the audit manifest is regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Add openultrasast.plane.manifests: frozen dataclasses and loaders for ax's
Task, Workspace and Model kinds (ax.io/v1alpha1, ax's fields only) and the
Run kind (openultrasast.io/v1alpha1: tasks, dependsOn, inputs, outputs,
budget, serialize). Every violation raises ManifestError naming the kind,
metadata.name and field path: unknown fields at any depth, wrong apiVersion
or kind, a provider outside google/anthropic unless deepseek is declared via
the openultrasast.io/provider-extension annotation, a Run with a dependency
cycle, an input naming no task, and a Run task naming no loaded Task.
load_manifests reads multi-document YAML files and indexes by kind and name.
PyYAML becomes the one declared core dependency; the two zero-dependency
guards now pin exactly that, and the module audit registers the plane
package under the ai-service-plane spec.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
MeteredClient wraps a ChatClient, sums the provider's usage rows per call,
prices them from a Model manifest's parameters (cache-hit, input, output per
million), and raises BudgetExhausted before a call once the usd or calls
ceiling is reached. A 401/402 or "Insufficient Balance" failure becomes
AccountError with the provider's message. An unpriced model reports usd None,
never zero, and refuses a usd budget; summary() gives the spend block of a
task's summary.json.
The two plane modules are registered as spec-owned in the module audit (their
consumers land in tasks 3-4) and the audit manifest is regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Measured on the main session: 90% of development tokens were tool I/O. Three
tools from the Portal pattern:
- read_guard.py: PreToolUse hook that denies a whole-file Read over 350 lines and
points to a ranged read or the bulk reader (pipe-tested: deny, range, small).
- bulk_read.py: answers a question about files with the cheap chat model, citing
path:line, under a spend ceiling.
- token_report.py: where a session's tokens go, from its transcript.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Exploratory, 7 of 17 cases, stopped when the account emptied:
- source-only scope: a model reading every product file puts the vulnerable
function among the candidates in 15/17 (ranker 1/17, vocabulary 2/17);
- per-file triage + per-file 6-step hunts + two-pass agreement: 12/15 candidate
cases on the validation set; 4/7 on the full run, two more in one pass of two;
- precision: 5 pgAdmin agreed findings read against source, 3 true (privileged),
2 false with self-contradicting rationales; agreement cannot catch an error
both passes make;
- cost: $0.009-0.022 per candidate for two passes.
Four instrument defects fixed on the way (model id, provider routing, JSON-mode
final turn, unbounded tool output) void the earlier hunter number. No gate is
met; qualification needs population v3 and a protocol that states the judge
stage, the pass rule and a config-family question.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
- verify_batched.py --fixed asks the agreed candidates again at the fixed pin
(no triage); a function the fix removed is recorded as removed.
- score_batched.py applies protocol v2's matching to the batched output
(vulnerable pin, and the fixed pin where rechecked) and draws the precision
sample.
- A finding's site is anchored to the candidate operation asked about; the
model's own reported location is kept in `reported_at`. ReactPress's confirmed
findAll was reported at the controller file and failed to match its declared
site. The scorer anchors older records the same way.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
verify_batched.py replaces the per-candidate verifier's design:
- triage per file (no tools): where does the operation's data originate;
internal/constant candidates are not verified;
- one 6-step hunt per file for up to 6 candidates, a verdict per operation;
- two independent passes; only agreement is reported, the rest is "disputed".
On the 46-candidate validation set ($1.01):
- declared sites: 16 of 20 agreed, 12 of 15 cases (the per-candidate design
confirmed 11). Budibase and Froxlor are new; Unleash dropped to one pass of two.
winml is triaged out as "constant": a CORS literal, which the reachability
question does not fit.
- the 26 pgAdmin candidates that flickered between single runs: 2 triaged out,
9 agreed in both passes, 4 disputed, 13 quiet.
- $0.022 per candidate for two passes vs $0.033 under the old design; on a file
with several candidates, 2.3x cheaper per pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Declared vulnerable sites: 11 of the 15 that were candidates are confirmed (8 at 3
steps, 3 more at 6). Missed: winml (a CORS config; the reachability question does
not fit), FUXA, Budibase, Froxlor.
Restart-based escalation is strictly worse than a straight 6-step budget: 88% of
pgAdmin candidates used every step, so they paid 3 + 6 ($0.0201 per candidate vs
$0.0139 baseline). Defaults are now 6 steps, no escalation. The cache order change
did not lower the per-call cost ($0.002): the conversation prefix was already
cached; 56% of prompt tokens are served from cache.
The larger finding: on 180 pgAdmin candidates verified twice, 15 and 13 were
flagged with only 2 in common. Verdicts on borderline candidates flip run to run,
while the declared sites were found consistently. Precision needs agreement
across runs or a second judgement, not a single pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
--declared-sites-only, --candidates-from <jsonl> and --results-suffix, so the
efficiency change can be validated on the 180 already-verified pgAdmin candidates
and the 15 declared sites without touching the main results.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Token efficiency, measured on 180 verified pgAdmin candidates: stage 2 costs
$0.0139 per candidate, 90% of the spend, because the conversation is re-sent on
every turn and grows with each tool result.
- Context files now precede the question in the hunter's first message. DeepSeek
bills a cached prefix at 1/30 of a fresh one, by prefix; with the question first,
every candidate asked about the same file (about 5 per file) broke the prefix on
its first line and paid for the file again. A test pins the order.
- The verifier runs each candidate at 3 steps and repeats it once at 6 only when
the hunt spent every step on tool calls without a verdict. On the seven declared
v2 sites, 3 steps found 5 and 8 steps found 6; most candidates conclude in 3.
- Per-candidate token usage (prompt, cached, completion, calls) and escalation
are recorded, and each case reports its cache share, so the next efficiency
decision is made on numbers.
Validation on the 180 pgAdmin candidates plus the 22 known sites (about $1)
waits for account credit: DeepSeek returns 402 at the moment.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The rerun hit DeepSeek HTTP 402 ("Insufficient Balance") after 180 pgAdmin
candidates. Every later call failed at $0, and the runner wrote sixteen cases as
complete with nothing asked, because an unverified candidate counted toward
completion. That is a clean-looking zero from an instrument that could not call
the model.
Now:
- 401/402 or "Insufficient Balance" stops the run at once, exit 2, nothing
recorded;
- only verified candidates count as done, so unverified ones are asked again on
resume;
- a case is complete only when every candidate was verified.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
