Scope+bound (change-first ordering, adaptive budget-fitted portioning,
_context_pass scoped to changed files, evidence_pass_bounded degradation) is
preserved here but does NOT unblock analysis: query_seconds stayed 0 across every
live re-verify. Uncontended measurement proved why -- the evidence pass is ~150s
fixed (first-query graph-load + census) + ~93ms/pair, so ~200s fixed overhead
(with the 48s build) precedes any analysis and a 300s budget leaves ~100s.
Pair-count scoping only trims the ~86s per-pair tail. The warm-session env
(OUSAST_ENGINE_SESSION=1) times out at startup under contention and, even working,
spares only subsequent batches -- not the first evidence query's load.
Parked per decision to finish the G0 baseline first; resume on a clean host.
Branch unmerged. 95 model tests pass; no push-suite regression.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Measured: on an unseen push the evidence-tier pass is the whole cost -- 238s of a
291s bedrock run, 99.9% of the non-build/non-query time -- because it tiers every
region of the repo to rank them, and a cold large CPG answers each pair ~50-100x
slower than the design's assumed few ms. The budget is gone before one flow query
runs: selected=0, deferred=1365 all deadline_exhausted, query_seconds=0.
When the push carries a change context AND an execution budget, the evidence pass
is now spent CHANGE-FIRST and BOUNDED:
- evidence candidates are ordered so the changed regions tier before the rest;
- the pass stops once it has taken EVIDENCE_BUDGET_FRACTION (0.5) of the remaining
budget, in BOUNDED_EVIDENCE_PORTION (256) chunks so the stop is granular;
- the CPG stays whole-repo, so a flow INTO a changed region from unchanged code is
still traced -- only the question population's long unchanged tail may go untiered.
Honesty: _portioned_evidence returns requests ATTEMPTED, so _evidence_pass splits a
genuine query failure (engine asked, no answer) from a budget skip; the skipped tail
is recorded as evidence_pass_bounded, never read as a clean repository.
Preserves the tested invariants: with no execution budget (the unit tests) the path
is inert, so change context still does not alter the ranker's selection and a changed
dependency still reaches its unchanged ranked question.
Tests: _portioned_evidence bounds at the deadline keeping the first portion; the scan
sets the sub-deadline + bounded portion + change-first order; the bounded tail is
recorded and is not a query failure. 128 model/change-context tests pass; no push-suite
regression (Joern-gated engine_trace/cpg_budget failures pre-exist on HEAD).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Saved-artifact run: total=291s build=52s query=0s arbitrate=238.75s, and
query_seconds_by_kind[evidence]=238.54s. The evidence-tier pass is the whole
middle; affected_context + _scope_work are 0.21s. It tiered 1015 pairs at
~235ms/pair (contended) and exhausted the budget -> selected=0, deferred=1365
all deadline_exhausted. Secondary anomaly: ~50-100x the design's assumed
2.5ms/pair -- the evidence query is pathologically slow on a cold large CPG.
Increment 2's single target is the evidence pass (scan.py:338-359). Fork
recorded: (i) scope+bound the pass change-first, (ii) two-phase neighbourhood
pass, (iii) per-pair speedup. Must preserve reach-into-unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A first scoping attempt (restrict analyzed regions to changed files) was reverted:
it breaks a tested design invariant. The engine deliberately selects an unchanged
region's question when a changed region reaches it
(test_changed_dependency_reaches_unchanged_ranked_question;
test_change_does_not_replace_ranker_selection). _changed_files feeds _evidence_pass
only as contextFilter/contextPaths (itemise vs summarise), not to shrink the tier
population. Any scope must keep changed regions PLUS the regions they reach.
Also: increment 1 over-claimed. Increment 0 measured build=67s/query=0s but NOT
the split of the ~236s middle (evidence-tier pass vs affected_context/context
resolution vs _scope_work enumeration). Increment 2 must capture per_kind /
other_seconds / affected_context timing and cut the dominant phase while
preserving reach-into-unchanged -- instrument before fixing.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The 4095 candidates are not an in-scan artifact: push/runner.py:345 discovers
head_regions across the whole head tree, then runner.py:397 passes them wholesale
to scan_repository with change_context threaded in but unused for scoping. The
scan enumerates all regions x families (scan.py:167/364) and, with
order_by_evidence=True, evidence-tiers every pair before analysis -> the ~236s
from increment 0 -> deadline at query_seconds=0.
Fix locus (increment 2): scope the analyzed region set to context.changed_paths +
dataflow neighbourhood (regions.py:279 primitive exists; compare_targeted_base is
precedent). Keep the CPG whole-repo so taint can trace into the change from
unchanged code; only the question set is change-scoped.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Timed live bedrock replay: build_seconds=66.69, query_seconds=0.0, total 303s,
4095 candidate questions all analysis_not_started=deadline_exhausted, selected=[],
regions_scanned=0. ~67s builds the CPG; ~236s enumerates/ranks/prepares 4095
repo-wide candidates for a 3-file change, then the deadline hits before any query
runs. The CPG build is not the bottleneck; the over-broad candidate set is ->
change-scoping is the confirmed lever (and compounds with the correspondence fix,
since per-candidate prep includes correspondence work).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Track the engine-detection lever found while driving G0. Live diagnosis on the
frozen pool: the engine's "no finding" on unseen changes is usually
analysis-never-runs -- it ranks repo-wide candidates and defers them all
(deadline_exhausted) before analyzing the pushed change (bedrock injection,
303s, selected=[], ranked sites unrelated to the change). Lever: seed analysis
from the change's declarations + dataflow neighbourhood (rank.py/mapping.py) so
it runs within budget.
- .kiro/specs/engine-change-scoped-analysis/brief.md: problem (measured),
hypothesis, anchors, and checkable increments (0 confirm build-vs-analysis
cost split -> 1 locate selection/budget -> 2 seed from change -> 3 soundness
+ tests -> 4 live re-test -> 5 measure delta on the engine arm).
- roadmap.md G2: active workstreams (change-scoped analysis; the done-but-
necessary-not-sufficient line-correspondence fix), for progress tracking.
Honest scope: G2, gated on the pool metric; not part of the G0 quick-rule
record-07. Coverage also needs the capability registry (Task 8.3); arm-scale
measurement needs the kube lane.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The correspondence mapped an equal-region base<->head line pair only when the
line's content was GLOBALLY unique in the file, so common lines (`}`, blanks,
imports) tripped `ambiguous_line_correspondence` and the finding's anchor was
discarded -> the operation came back `unmapped` -> novelty `unknown` ->
`comparison_unknown` -> the engine finding was suppressed (a direct cause of
"near-blind to fixes").
Replace the global-uniqueness gate with run-aware positional anchoring: segment
the equal-region pairs into contiguous runs (strictly +1/+1, as _equal_line_pairs
emits), and map a pair when its content occupies exactly one old run and one new
run. Duplicates WITHIN a contiguous run are positionally forced and now map;
duplicates ACROSS runs split by an edit are aligned arbitrarily by git and stay
withheld + flagged (soundness preserved -- the moved-repeated-operation guard
still passes, so no false `new`/identity claims).
Tests: update test_change_context_unknown_base_binary_and_ambiguous_lines_are_
unresolved (its two in-run sinks now anchor); keep the moved-operation soundness
test; add intra-run-maps, cross-run-withheld, and distinctive-line-anchors cases.
163 push tests pass, ruff clean.
Honest scope (design.md amendment): this removes one real cause but is
necessary-not-sufficient -- live-testing a messy 14-file Java change showed
coverage stays `incomplete` due to other, dominant blockers (capability_unavailable
until Task 8.3; the engine produced no actionable defect; eligibility/projection
gaps). Engine coverage on unseen diffs needs those too, not just correspondence.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Fold the night's learnings into the roadmap, steering and the
unseen-repo-evaluation spec so the process is reproducible and the
project is "ready now":
- roadmap.md: a 2026-10-11 state table (pool p1 frozen; harvest hardened
+ bucket-first; replay-image provenance pinned; baseline-on-kube next)
and a 5-step readiness plan to record 07.
- storage-streaming.md: RustFS-over-local as the default (local disk is
the scarce resource); the implemented ousast-corpus layout; replay
outputs belong on RustFS; the AX store endpoint is operator-owned.
- design.md: an "Amendments (2026-10-11)" section (bucket-first
persistence, ~2.2% rate-bound yield, machine-label record 07, pinned
replay image provenance).
- tasks.md: a folded-in amendment block (pool p1 frozen + freeze recipe;
bucket-first; CI replay image; machine-label baseline; kube lane +
ax/store as operator-owned infra).
Facts stated are verified (measured yield, freeze digest, GHCR anon pull,
batch.py's endpoint check); infra capability/policy is left to the
operator, not assumed. The frozen pool artifacts themselves are not
committed here (a separate samples/licensing decision).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The real bottleneck was not enumeration or extraction: canonicalize_used
resolves the entire exclusion set (measured: 4,499 names) via serial
GitHub get_repo on every launch, before any harvesting -- ~30 min of
network-latency-bound work that also burns ~4,499 of GitHub's 5,000/hr
budget before real work starts (hence the low CPU and the "never
finishes"). The earlier resolution-cache only banked via
BankedGitHub.resolve, but canonicalize_used calls eligibility.resolve ->
get_repo directly, so it was never cached and re-ran every time.
Cache get_repo at the real choke point. CorpusBank gains cache_repo /
get_cached_repo storing one object per normalized name at
<prefix>/repos/<sha256(name)>.json -- NOT seed-scoped, since repo
metadata is seed-independent, so every seed/run shares it. 404s are
cached as None too (a found-but-None hit) so known-missing repos are not
re-fetched. BankedGitHub.get_repo now checks the local journal, then the
bucket, and only then GitHub (banking the result). canonicalize_used,
advisory_index and candidate resolution all funnel through get_repo, so
all of them are covered.
Effect: the first run pays the lookups once and banks them; every later
launch reads them from the bucket (no GitHub, no rate-limit spend), so
the rate budget goes to actual candidate harvesting and restarts are
cheap. Proven live against ousast-corpus (repos/ fills 1:1 with
resolutions). 206 unseen tests + new repo-cache test pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The decision bank made extraction durable, but every restart still
re-paid the ~26-minute advisory enumeration + per-advisory resolution,
since those lived only in the local journal. Mirror them into the bucket
so a restart reads them back and goes essentially straight to any
remaining extraction.
bank.py: bank_resolution/resolved at <prefix>/<seed>/resolved/<sha256(ghsa)>.json
(streamed one-by-one as each advisory is decided), cache_advisories/
cached_advisories at <prefix>/<seed>/advisories/list.json, and
bank_repository/repositories (caches the advisory_index repo-resolution
pass, preserving renames and 404s). All gated by RESOLUTION_CONTRACT="v1"
(extract.py) -- bumped only when enumeration/eligibility/source.candidate
output changes; recent fetch/parse/pool changes do not bump it.
draft.py run(): with --bank, advisories load from the cache (else
enumerate then cache); eligibility decisions load from the bucket
(contract-matched) to skip re-resolution; new resolutions stream to the
bucket before the local checkpoint write. Repository lookups reuse the
repositories/ cache. No --bank => local-journal behavior unchanged.
Net: advisories + resolutions + extraction all resume from ousast-corpus;
restarts no longer re-enumerate or re-resolve. Fresh and resumed runs
produce byte-identical manifests at 1 and 4 workers. Bank methods proven
live against ousast-corpus. 356 tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The draw lost all progress on any restart because the local journal
invalidates on a code change and is the only cross-run state. Make the
bucket the source of truth: stream every per-repo accept/reject to
ousast-corpus as it is decided, keyed by sha256(repository+"\n"+fix) so
writes are idempotent (content dedup at rest), and resume by listing the
bucket and skipping already-decided repos. No supervisor/watchdog needed
-- a crash or a code change just re-runs and resumes.
New benchmarks/unseen/bank.py: corpus_settings (CORPUS_S3_* from .env,
missing key named, value never printed), CorpusObjects (lazy boto3,
path-style + SigV4 verified live against ousast-corpus @ 10.0.0.54:9001),
CorpusBank.bank/decided with one-retry put/get, split_decided. Decisions
live at <prefix>/<seed>/decisions/<key>.json with repository, fix,
accept/reject, reason, entry, exclusions, counts, contract, seed, ts.
draft.py: --bank [prefix] opens the bank on the coordinator (never shipped
to fork workers); run() resumes from bank.decided() before extracting,
reusing accepted entries across contracts (deterministic from repo+commit)
and rejections only when contract == EXTRACTION_CONTRACT ("v1", in
extract.py); collect() streams each decision one-by-one. Seed-order
acceptance, fork/used-set dedup, and duplicate-advisory reuse preserved.
Without --bank, local-journal behavior is unchanged.
CorpusBank proven live end-to-end (bank accept+reject, idempotent re-bank,
decided/split roundtrip). 325 tests pass offline, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The harvester cloned --filter=blob:none --no-checkout, then blame/log/diff
triggered the promisor to fetch each needed blob ON DEMAND -- hundreds of
round-trips per repo, the dominant cost that made the draw unable to
converge. The blobless clone was a premature optimization that backfired.
Drop --filter=blob:none from the clone and the two commit fetches. A
full-objects --no-checkout clone brings the whole commit graph AND every
blob in one pack, so blame/log/diff read locally; --no-checkout still
skips the working tree so disk stays bounded (plus the existing 500 MiB
clone_size_cap and the disk guard). With no promisor remote, a genuinely
missing object now errors loudly instead of silently round-tripping --
the instrument guarantee that the bulk fetch was complete.
Extraction semantics are unchanged (same SZZ introducing-commit positive,
same stratified ordinary negatives, byte-identical entries); only the
object-transfer strategy changes. Test inverted to assert no blob:none
filter on the clone; README updated. 145 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Git subprocess calls were deadline-bounded (Git.command_timeout uses the
remaining per-repo budget), but a pure-Python parse between git calls
(repository_changes -> history/blame/diff on a giant file) had no
deadline: when it hung, no further git call fired the git deadline, so
the worker stalled indefinitely (observed >720s with zero git
processes). An external watchdog was papering over it at ~20 min/stall.
Add extract.process_deadline(seconds): a SIGALRM/setitimer wall-clock
alarm installed on the worker's main thread that raises
ExtractionTimeout("wallclock_timeout"), with no-op guards (None/<=0, off
the main thread, or no SIGALRM support) and strict timer-cancel +
handler-restore in finally. Wrap the whole extract_repository body in it
so the timeout propagates to the existing `except source.Rejected` and
journals as a clean rejection, not a worker failure.
Budget = --extract-deadline (git, 240s) + --parse-deadline (new, 120s)
= 360s total ceiling, so git-bound repos keep their full budget (yield
unchanged) while a hung parse is rejected well before any external
watchdog. Record parse_deadline_seconds in the report.
295 unseen tests pass (incl. timeout + handler-restore + no-op cases),
ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The draw hardcoded 300 repos / 3 slices. At the measured lazy-fetch rate
the full 300 projects ~2.5 days; a 100-repo single slice (slice 1, all
the G0 baseline needs) is ~20h. Add --pool-size (default 300) and
--slices (default 3), validated (>=1, slices<=pool_size, evenly
divisible). Generalize assign_slices' stratified allocation from a
literal 3 to `slices`, the run() break / insufficient check / report
(changes = pool_size*13) to pool_size, and record both in the journal
inputs. Remove bisect's three-slice assumption; freeze reads the size
from the draft. Defaults (300/3) reproduce prior output exactly.
With --pool-size 100 --slices 1 the pool is the first 100 accepted repos
in seed order, all in slice 1 (deterministic, byte-identical across
worker counts). 280 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Git.materialize enumerated every object reachable from HEAD and tried to
`git fetch --no-filter` them by SHA. Git rejects that for real repos
("did not send all necessary objects" / "bad revision"), exiting 1 ->
InstrumentFailure(git_exit_1) that failed the whole repository. Measured
in a live draw: 326/670 attempts, ~49% of all failures, which froze the
accepted pool at 44/300. Reproduced on pallets/flask (26221 reachable,
9287 missing, fetch exit 1).
Git cannot batch-fetch arbitrary tree/blob objects by SHA, so remove
materialize and restore the original lazy path: the blob:none clone keeps
the commit graph and the introducing/blame/ordinary-history reads fetch
their blobs on demand through Git's promisor. The per-fix tolerant fetch
loop, the process-pool parallelism, the 25-with-zero-entries guard, the
per-repo deadline and resume are unchanged. Extraction content is
byte-identical (only the prefetch is gone).
217 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two defects stalled the draw at 40/300:
1. The systemic guard used an absolute `instrument_failures > 200` cap.
At ~10% extraction yield, reaching 300 entries legitimately produces
far more than 200 per-candidate failures, and the counter is rebuilt
from the journal on resume -- so once past 200 the guard tripped
immediately on every restart (permanent stall; the supervisor's
no-progress guard then stopped). Drop the absolute cap; keep only
`>=25 failures with zero entries`, which still catches a genuinely
broken instrument (it yields no entries) without aborting a working
low-yield draw. Bounded anyway by the population / max-hours.
2. repository_changes fetched all fixes in one `git fetch origin *fixes`,
so a single unfetchable secondary fix ref failed the whole repo
(extract_git_exit_1 was 166 of ~200 extract_ failures -- the dominant
loss). Fetch the extra advisory fixes per-fix, best-effort, skipping a
git_exit_ failure; candidate.fix stays strict (already fetched in
extract_repository). Only resolved fixes are materialized and
fix-site-indexed.
218 unseen tests pass (incl. 1000 failures after a success not tripping
the guard, and fetch-error regressions), ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The ProcessPool spawned its workers but ordered_extractions blocked the
main thread on the OLDEST future, so while one slow repo finished the
other workers sat idle: live profiling showed 1 of 4 cores busy, ~67%
idle, ~1 attempt/min.
Separate the seed-order acceptance cursor (`pending`: row+decision+key,
no futures) from the worker set (`in_flight`: future->key). Keep up to
`workers` jobs running; journal each result as it COMPLETES via
wait(FIRST_COMPLETED) in any order and refill freed slots, never waiting
on the oldest. A candidate is accepted/rejected only when its result is
in the journal, advanced strictly in seed order -- so the pool runs
out of order while the accepted pool stays byte-identical to a 1-worker
run. Fork-network dedup and the systemic guard still apply at the cursor.
New real-process test proves later extractions are journaled before an
earlier slow one completes (completion order != seed order) while
accepted entries stay in seed order; the old loop fails it. 210 unseen
tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Local disk is the scarce resource (host near full; the draw stops at
<2G free). Record the direction to stream durable run artifacts to RustFS
instead of hoarding on local disk, split across purpose-specific buckets
(sast-memory durable, sast-pools durable archive, sast-runs transient
with aggressive lifecycle), each with a lifecycle tuned to its artifact
-- the lesson from the engine-queue versioning bloat. Extends the
existing S3Store pattern; new buckets are an operator admin action.
Planned as a follow-up spec, not retrofitted into an in-flight run.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Group 3 was unimplemented, so there was no way to turn the draft into a
frozen pool. Add:
- bisect_guard.py: stages the cache draft and bisects it against the
reserved-repository guard, run as a black box
(pytest -k referenced, stdout/stderr discarded, exit code only; an exit
other than 0/1 aborts as an instrument failure). Drops guard-tripping
candidates and reports the dropped COUNT only -- never a name or URL.
Guard runner is injectable for offline tests.
- freeze.py: writes canonical pool-p1.toml (+ private counterpart,
reusing draft's public/private split and private_digest) and
freeze-p1.json (used-set digest, per-source counts, guard result,
dropped count, label-check counts, freeze digest = sha256 of the
canonical public manifest). A matching retry is a no-op; any changed
frozen artifact requires a new pool id (never an overwrite).
- protocol-p1.md: the frozen scoring/statistics protocol from design
sections 3 and 4.
Offline only; the guard subprocess is the sole subprocess and is mocked
in tests. 42 new tests pass (incl the group-1.3 guard digest check on a
fixture pool); neighbouring unseen suites green; ruff clean. New files
only -- does not touch the journal-pinned draft/extract modules, so a
running draw is unaffected.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Extraction is CPU-bound Python (git-output parsing, HUNK regex,
ast.parse of hundreds of source files per repo). A ThreadPoolExecutor
cannot parallelize it: live profiling showed one thread pegging a core
at 99.7% while three cores sat idle and the others waited on the GIL, so
six "workers" barely beat serial (~0.25 entries/min).
Switch ordered_extractions to a ProcessPoolExecutor with a fork context
(copy-on-write memory; lets the existing in-memory test fakes cross into
children without pickling). Effective workers are clamped to the core
count. The parent still owns the journal, REST (finish_license), decide,
counters, the fork-network dedup, strict seed-order acceptance, the
systemic guard, and the graceful drain. Children run extract_repository
with github=None on picklable args and return their own count deltas
(threads had shared a Counter; processes return counts in the result).
A broken pool is replaced; a collateral worker_crash is a skip outside
the guard, a task-specific extract_worker_failure counts toward it.
Determinism is unchanged and tested with REAL processes: workers=6 ==
workers=1 byte-identical (fork collisions, out-of-order completion,
resume). 208 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Serial extraction at ~147s/repo made a 300-repo pool take days, and
~626 lazy per-file blob faults per clone pushed slow repos past the
240s deadline (a draw reached 21 entries from 347 attempts, 160 of them
extraction_timeout). Both fixed.
Parallel: a bounded ThreadPoolExecutor (--extract-workers, default 6)
runs clones concurrently, each into its own clone with a private counter.
The main thread alone owns the journal, REST (license via finish_license),
counters, and acceptance. Entries are accepted in STRICT SEED ORDER with
the fork-network dedup applied at acceptance, so the pool is byte-
identical to the serial draw (tested: workers=6 == workers=1, including
fork collisions, out-of-order completion, and resume). The systemic guard
and resilient skips are unchanged; a graceful stop drains started work to
the journal for resume.
Batched blobs: repository_changes fetches the fixes together and
Git.materialize fetches all missing blobs for the read revisions in one
`fetch --no-filter --stdin` (rev-list --missing=print), instead of per-
file promisor faults. Content is byte-identical (tested; >=12 fetches
collapsed to 1).
202 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
ObjectStoreExecutor.submit re-PUT the identical command.json on every
0.2s result poll, so a task running minutes wrote ~1,500 versions of one
engine-queue/search-exec/<id>/command.json. On the versioned memory
bucket that was the dominant growth: 109,715 versions, 10.3 GB (91% of
the bucket). retry_mailbox already retries a lost PUT acknowledgement
internally, so one upload per task is the correct delivery guarantee.
Guard the PUT with an `uploaded` flag: upload once (still retried on
transient failure), then poll only the result. One command.json version
per task instead of ~1,500. All submit semantics preserved.
59 executor tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Record the policy: OpenUltraSAST notifies the system/AI operator on
Discord about its own processes and results, and especially when a
process is stuck or impaired (stall, insufficient result, abort,
instrument_failure, timeout, budget/resource limit). Operator-facing,
not developer debugging, and explicitly not k3s/ax infrastructure.
Credentials by name in gitignored .env (DISCORD_WEBHOOK_URL,
DISCORD_OPERATOR_ID), listed blank in .env.example; off when unset.
Messages carry status/counts/digests only, under the hardening and
measurement-record redaction (no payloads, secrets, or repo content).
Notifier implementation to be wired/specced; this states the behavior.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The resilient draw only covered extraction. A per-candidate transport
failure in the decide/eligibility phase -- get_commit on an unresolvable
fix ref raising ValueError("http_failure") (observed: ljharb/shell-quote
commit 1518179) -- still aborted the whole draft, because decide only
caught source.Rejected and eligibility.RepositoryNotFound.
decide now catches that http_failure and returns a "decide_http_failure"
skip, and the run loop feeds decide + extract per-candidate instrument
failures into one systemic guard (25 with zero successes, or >200), so a
single bad commit lookup skips while a broken instrument (e.g. a network
outage failing every lookup) still aborts loudly.
Full unseen draft suite passes (48), ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Individual repositories in the advisory population raise per-candidate
extract.InstrumentFailure during extraction (git_exit_128 on an
unfetchable fix commit, clone_size_cap, and others), and each one
aborted the whole 300-repo draft. The draw is healthy; one bad repo
should not reset it.
extract_repository now also catches extract.InstrumentFailure and
OSError and records them as "extract_<reason>" rejections, so the draw
skips that repo and continues (the reason, e.g. extract_git_exit_128,
shows in the rejections counter). ExtractionTimeout keeps its existing
extraction_timeout skip; any other exception still propagates as a real
abort, so genuine code bugs are not masked.
A systemic guard preserves the instrument rule: run() raises
InstrumentFailure("systemic_extraction_failure") if 25 extraction
attempts fail with zero successes (e.g. git missing, auth broken) or if
extraction failures exceed 200 overall. A handful of bad repos with many
successes never trips it.
190 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A pathologically slow repository (observed: tensorzero, 183s to clone,
then `git blame -C` past the 180s command limit) raised
subprocess.TimeoutExpired, which Git.run mislabeled as a fatal
InstrumentFailure("git_launch_failed") and aborted the whole 300-repo
draft. The same slowness stalled throughput (a 54-minute no-progress
gap).
Give each clone a per-repo wall-clock budget (--extract-deadline,
default 240s, plumbed Clones -> Git, counting the initial clone). A
timeout -- a command exceeding the remaining budget, or the budget
elapsing -- raises ExtractionTimeout(Rejected), which the existing
rejection handler turns into an "extraction_timeout" skip so the draw
moves to the next candidate. Genuine launch failures and non-zero git
exits stay fatal; timeouts and launch failures are now distinct. The
deadline is recorded on the report only, not the journal inputs, so
changing it does not refuse resume.
181 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Update the front door and Status to the current state: static
abstract-interpretation blocking is NO-GO by hand, so the work is a
dynamic proof arm that builds and runs the app and demonstrates a
finding by a differential (triggers on the vulnerable revision, not the
fixed one), earning a block by execution rather than calibration. It is
pilot, not shipped. The release gate is a measured catch/false-alarm
number on unseen repositories, now being produced. Bump v2.0.0 -> v2.0.1,
link the developer guides.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
disk_bytes() summed the clone tree with rglob + stat() as a size cap
before every git command. On a blob:none clone, `git show` triggers an
on-demand blob fetch that writes and deletes transient
.git/objects/pack/*.rev files, so a path rglob had just listed could be
gone by the stat() call -> FileNotFoundError aborted the whole draft as
an instrument_failure. The race can hit any candidate, so it was a
non-deterministic draft-killer (observed extracting a real npm repo).
Skip a file that raises OSError mid-walk; the size cap is unaffected.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The GitHub /advisories endpoint ignores the `page` query parameter: it
paginates with the Link header's rel="next" cursor. The page-number
pager re-fetched the same first 100 rows forever and never terminated,
so enumeration never completed and no draft converged (a run fetched
5137 "pip pages" holding only 100 unique advisories).
Surface the successful response headers from JSONTransport, and walk the
rel="next" cursor per ecosystem until it is absent. The ecosystem /
published>=2019 / bare-number cwes filters are preserved on the first
request. Journal keys stay ecosystem:n (now cursor-walk positions) with
the saved next URL for resume; journal_version 4 refuses older journals.
Live smoke via the production pager: 300 unique advisories pulled in 3
API calls, advancing past the first page. Offline: 170 unseen tests
pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
enumerate_advisories paged the entire global GitHub reviewed-advisory
database (every ecosystem, every year, every CWE) before extraction, so
a 300-repo draft never converged -- a live run reached 21,962 advisories
in 4.5h with extraction never started.
Push the existing candidate() rejections into the query: page per
ecosystem, published>=2019-01-01, and cwes restricted to the seven
families. The cwes filter uses bare CWE numbers (89), which is what the
GitHub advisories API matches -- "CWE-89" silently returns zero. The
whole cwes filter is omitted if the taxonomy ever holds a non-exact
token, so a selectable advisory is never dropped. Journal version 3
(ecosystem:page keys) refuses incompatible older journals.
Live page-1 smoke (pip, npm): 100/100 rows carry a family CWE, none
published before 2019. Offline: 166 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A live G0 sourcing run failed on both: a transient connection/decode error aborted
the whole enumeration (the bare except raised http_failure on the first blip), and
the advisory journal stored full raw advisory JSON and reached 1.77 GB on a near-full
host. Fix: the JSON transport now retries URLError/timeout/ConnectionError/IncompleteRead/
JSONDecodeError through the same attempt-4 backoff as HTTP 5xx/429 (fatal only after the
cap or on a genuinely non-retryable error); enumerate_advisories journals only the fields
resolution/extraction consume via source.reduce_advisory (ghsa id, published/withdrawn,
type, CWE ids, package ecosystem/name for OSV, fix-commit links), with journal_version
bumped to 2 so an old full-format journal is rejected and resume-skip stays intact.
Offline: test_unseen_source/draft/extract 133 passed; ruff + format clean; mypy unchanged
(pre-existing verify.py lambda only).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Maintainer direction: the same agentic endpoint authors the PoC, writes a candidate
fix against the repair direction + failing PoC, and hands it to the differential
verifier, which re-runs the PoC and confirms absence. Closes the no-auto-patch gap:
the proof is the acceptance test for the fix (a wrong fix is caught), the fix stays a
developer-reviewed proposal, and the green proof is the objective signal. Specced
alongside proof authoring, after qualification.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The loop: catch a regression in the diff, prove it with a reproducible check,
the dev/agent fixes it (tool gives a repair direction + the failing proof, not an
auto-patch), then the same proof re-runs and no longer triggers = absence
demonstrated by execution; the proof becomes a committed regression test. Example
GitHub Actions workflow uses the real 'ousast pre-push . --base --head --deadline
--mode --artifact' entry; PR-comment formatter marked as a placeholder. Honest
status separates what exists (re-verify by construction, --base/--head) from
delivery work (PR formatter, auto-landing the regression test) and the open
large-repo latency.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Revise the developer guide for engineers with full backlogs and no spare schedule:
a vague warning dies in the backlog, so the value of the demonstrated proof is the
prioritization conversation -- reproducible evidence (input, behavior, the one-line
fix difference) a dev takes to a product owner or the business to win capacity,
dropped into the ticket system as a location, repair direction, and reproduction.
mcp-shift-left-agent brief: the agent's return should be ticket-ready evidence.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
docs/pre-push-for-developers.md: COBESY-structured guide for engineers who are
strong at dev but not security. Leads with the conclusion (zero-setup, no
annotations, diff-scoped, proof-backed), frames the agent-era case (a pre-push
check is how you oversee coding agents), states control (advisory vs blocking,
budget, lane, models), and is honest about limits with an explicit disproof
condition (coverage, large-repo latency, generalization being measured by G0).
.kiro/specs/mcp-shift-left-agent/: follow-up brief for an MCP agent + skills that
give devs and coding agents early shift-left feedback and a required pre-push
security step for agent oversight; builds on the existing ousast mcp server;
sequenced after the safety net is qualified.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
verification-architecture.md documents the existing two-stage design: the Cairn-inspired
search (brain + model calls on the VM) and the DAST proof (gVisor AX tasks in kube-ax),
the trust boundary (secrets VM-only, cluster tasks secret-free with restricted egress),
and the three verify lanes (ax=cluster default, vm=local Bubblewrap for $0 proofs,
docker=replay). Clarifies the proof is dynamic, not abstract interpretation: AI triage
stays the static backbone, proof earns the block by execution.
proof-arm-opencode/ is a follow-up brief: drive proof-authoring with OpenCode MCP, an
abliterated/GLM proof model (DeepSeek stays the search model), DSPy-optimized from
OpenObserve traces, to beat the safety-classifier refusals and nondeterministic wiring.
Flags the trust-domain decision that must come first (A: scoped key + egress inside the
cluster; B: brain on VM, only DAST in gVisor — recommended to start). Credentials live in
.env by name only (ABLITERATION_*, OPENOBSERVE_*); no secret value is in any tracked file.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Lockless old npm projects resolve era-appropriate deps (fixes glance filed->mime@4 crash);
npm ci untouched; git-failure falls back to plain install.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A lockless npm project (e.g. glance, 2015) resolved an unpinned transitive range to the
latest major ([email protected] -> mime@4, which dropped .lookup()), so the built app crashed
serving files -- a build-reproducibility defect, not an app or agent fault. The executor
now, for the npm-install (no-lock) case only and only when .git is present, resolves
dependencies as of the checkout's HEAD commit date via `npm install --before <date>`,
giving era-appropriate, compatible versions. npm ci (lockfile present) is untouched, and
any git/date failure falls back to plain install. demo.py also recognizes
npm-shrinkwrap.json as a lock for the ci-vs-install choice.
Tests: executor + smoke 75 passed, demo 107 passed; ruff clean; mypy clean on executor.py.
Executor-side change -> image rebuild required.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Feed the milestone back into the spec and steering. HANDOFF: the open "first demonstrated"
question is RESOLVED -- xiaomusic (path, prefix-check-bypass) demonstrated on the vulnerable
side (3/3 affected, 0/3 safe) and the blind fixed-side search produced 0 false proofs; pair
total $1.22. Records the six-fix chain that enabled it (parity, verify-time-only, fair-share
budget, served-root guidance, oracle prefix-bypass coverage, control-signal feedback) plus the
keystone. tasks.md: implementation note; pilot (6.2/6.3) in progress. spec.json: pilot status.
safety-net steering: proof-based BLOCK mechanism reached first execution-backed demonstration
(feasibility-pilot scope, not release readiness; pilot exit + 73-negative qualification open;
classifier stays triage). search README: status note. Payload-free; demos/boards stay gitignored.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
search-with-proof produced its first demonstrated finding on a real CVE fix-pair end to
end on kube-ax: vuln observes, fixed does not, 0 false proofs, $0.37, 55 calls. Enabled
by the full session fix chain (parity, verify-time-only, fair-share budget, served-root
guidance, oracle prefix-bypass coverage, control-signal feedback). Payload-free record.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Non-blocking advisory hint when the control shows the app didn't serve ${served_root},
so the agent self-corrects its served-directory wiring. Host-side; verdicts unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
When a path demo does not serve ${served_root}, the keystone control fails
(SideRecord.control_observed False) but the search only reported a generic
inconclusive, so the agent kept making the same wiring mistake. The pilot now
surfaces the affected side's control_observed in the verify result, and the
coordinator appends one advisory, deduplicated "served-root" hint (visible in the
board snapshot the agent reads) telling it to set the served-directory config to the
${served_root} placeholder. Advisory only: verdicts, _end, the oracle, the differential
and the demo validator are unchanged; the hint fires only when control_observed is False.
Tests: coordinator + pilot_run suites 78 passed; ruff clean; mypy unchanged (pre-existing
benchmarks errors only). Host-side change, no image rebuild.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Oracle broadening works in the agent: demo now targets the prefix-sibling vector. Still
inconclusive (0 false proofs, $0.29): the agent set the served-dir env var to its own
value, not ${served_root}, so the sibling canary was outside the served tree. Served-root
wiring is nondeterministic (a prior run wired it). Last-mile agent-reliability gap.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Plant a sibling canary so prefix-bypass path traversals (startswith without separator)
are demonstrable; parent canary + differential unchanged; glance still demonstrates.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The path oracle only modelled classic parent traversal (a `canary` file in the parent of
${served_root}). A common real path-traversal class is the prefix-check bypass:
startswith(base) without a trailing separator, which the fix closes with base + os.sep;
it reaches a SIBLING whose path string-extends the served directory, not the parent.
PathOracle now also plants a sibling canary `public_leak` beside the served `public`
(same nonce, 0600), so a prefix-bypass demo leaks it on the vulnerable side and the fix
blocks it. ORACLE_GUIDE names the second target and tells the agent to pick the vector
its code shows. One shared nonce stays correct: a canary leaks only when the demo's own
request reaches it, and each demo targets one vector whose fix blocks it on the fixed side.
observe()/capture()/the differential/SideRecord are UNCHANGED. Local glance ceiling proof
still demonstrates (parent canary, 3/3 vuln, 0/3 fixed, control on both sides). Gate:
worker+control+verify suites 106 passed / 4 socket-skips; ruff clean; mypy clean on oracles.py.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Served-root guidance validated: authored demo now references ${served_root}. Still
inconclusive (0 false proofs); agent spread budget across vectors and hit run_ceiling
(164 calls, $1.47/$1.50) without converging, and the committed traversal did not
differential. Remaining = per-app exploit-authoring + convergence (the layer the $0 local
glance proof did by hand).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Agent guidance: point the app's served directory at the ${served_root} placeholder via
arg/env/config so the planted observable is reachable. Prompt strings only; 37 worker tests pass.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Path searches authored and verified many demos but all came back with no differential:
the demo never pointed the app's served directory at ${served_root}, so the observable
(which lives relative to ${served_root}) was outside the served tree and unreachable. The
agent had set an app env var to a value of its own; the guide only showed the --dir form.
ORACLE_GUIDE now states the agent MUST set the app's served/root to the ${served_root}
placeholder via whichever mechanism the app exposes (command-line argument, environment
variable, or config written to start.files), not only --dir, and that serving elsewhere
makes the observable unreachable. Verify-time-only clarification, observable names, and all
prohibitions are kept. Prompt strings only.
Tests: tests/test_search_worker.py 37 passed; ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Fair-share budget validated: the Python agent completed the full loop and verified 18
demos within $0.74 of $1.50 (0 verify tasks before). Still inconclusive, 0 false proofs.
Root cause of no-differential: the authored demo never references ${served_root}; the
agent pointed the app's served dir at its own value, so the observable was outside the
served tree. Next: guidance to wire the served dir to ${served_root} for the path oracle.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Each pilot search gets a fair share of --ceiling-usd with settled rollover, so one
expensive search can no longer starve the others; total stays within the ceiling.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
--ceiling-usd was one shared SpendBudget passed to every search, so an expensive
search drained the pool and starved later ones: in the last 2-pair/$2 run the first
search spent $1.60 and the second hit the ceiling at $0.34 after finishing its analysis.
The driver now allocates a dynamic fair share per search (remaining_ceiling /
searches_not_yet_started), rolls unused settled budget forward, caps each search at its
share, and keeps total settled <= ceiling. A search now gets room to author and verify
after it finds the path.
Driver-only change (benchmarks/search/pilot_run.py) plus tests. tests/test_search_pilot_run.py
39 passed; ruff clean; mypy unchanged (pre-existing benchmarks errors only).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2 path searches (ts/py), vulnerable, cluster, $1.95 of $2. The trap fix is validated
behaviourally: ts pair 9 now authors a demo (verify_tasks 0->2; fails could_not_build on
entrypoint), py pair 13 drops 118->36 calls and its board shows a complete route-to-sink
+ entrypoint trace but hits the per-run budget ceiling before authoring/verifying. Still
0 demonstrations, 0 false proofs. Payload-free aggregate; raw stays gitignored. Next
levers: per-run budget/step allocation for demo-write+verify after analysis; demo
entrypoint/build (start.path) for compiled/served apps.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Agent guidance so path searches stop trying to confirm the oracle observable during
exploration (where it is absent) and instead author a demo for the differential verifier.
Prompt strings only; validators untouched; 37 worker tests pass.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The parity fix named each oracle's observable, but path searches then spent their
whole budget without authoring a demo: the agent tried to confirm reachability during
exploration, where the oracle fixture does not exist (it is injected only in the
verifier's sandbox at verify time), concluded "unreachable", and never committed a demo.
ORACLE_GUIDE now states the fixture/observable is injected at VERIFY time only and is
absent during exploration; its explore-time absence never means unreachable or justifies
ending the search. The agent confirms the mechanism (sink reachability, access to data the
app should not expose) from app code and its own sandbox test data, targets the observable's
known verify-time location in the demo, and relies on differential verification as the only
proof. All prior prohibitions (no out-of-band/off-disk access, no input capture, no revision
sniffing) and the parity observable names are kept. Adjacent wording shortened to fit.
Tests: tests/test_search_worker.py 37 passed; ruff clean; mypy unchanged (pre-existing
verify.py lambda note only). Prompt strings only; validators untouched.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
3 path searches (php/ts/py), vulnerable side, cluster verify lane, $2.95 of $3.
Payload-free aggregate; raw boards/demos stay gitignored. Finding: the parity fix
produced directed, target-aware search (the agent reasons about reaching the named
observable and inspects the app's serving guard), but no demonstration landed -- all
three spent their per-pair allowance before a differential, and two wrote zero verify
demos (budget consumed in reason/explore). Next levers: allowance/budget split toward
demo-writing, entrypoint discovery for php, and the explore-vs-verify path layout the
py agent flagged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The path family's agent guidance named no target, while command (${marker_dir}/marker)
and ssrf (${callback_url}) disclose theirs, so path searches could not author a demo
that surfaces the observable through the app. ORACLE_GUIDE now names it (canary in the
parent of ${served_root}, matching PathOracle.prepare). The reason/explore constraints
are refined to forbid only out-of-band access (direct disk reads, non-app channels),
input capture, and revision sniffing, while allowing the app to disclose it through the
vulnerability under test. Prompt strings only; the demo validator and its _PRIVATE guard
are untouched, so direct references to verifier paths remain blocked.
Tests: tests/test_search_worker.py 37 passed; ruff clean; mypy unchanged (pre-existing
verify.py lambda note only).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Outcome-only record of the first end-to-end demonstrated verdict and the two
masking defects (regenerated npm lock pinning an incompatible mime; agent demo
targeting the wrong file). Payloads stay in the gitignored private area.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Honest serve signal for the HTTP verifier: SideRecord.control_observed records
whether a benign verifier-planted in-root marker came back through the app's
interface, as its own guarded app start that never changes the verdict. A
2xx-based block was rejected (any HTTP status counts as alive). Gate green;
ceiling re-proof on glance still demonstrated with control_observed on both
sides. The search task image needs a rebuild before the next cluster pilot.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The HTTP verifier could not distinguish a mis-wired or non-serving app from a
genuine no-effect: both yielded `inconclusive`. This adds a positive control for
the path oracle.
- PathOracle plants a benign in-root marker file (random token from os.urandom,
distinct from the secret canary) and exposes it via a new Oracle.control()
hook (default None; only the path oracle implements it today).
- On the first run of an HTTP side, _side runs a benign GET for that marker as
its own fully-guarded app start. The marker returning proves the interface
serves the fixture; its absence — or any control failure — is recorded in the
new SideRecord.control_observed field but never changes the verdict, since a
valid app may answer only its own routes. A crash is still caught by the
existing could_not_run path.
A 2xx-based "nothing served" block was rejected: the verifier deliberately
treats any HTTP status as the app being alive (a 503-only app is a valid
observed side), so status-based blocking would false-fail valid apps. Soundly
converting a mis-wired-but-responding app into could_not_run requires a
demo-declared control, which is a separate follow-up.
Tests: new tests/test_search_control.py covers marker observed, absent marker
not blocking, control failure not blocking a working attack, and crash still
could_not_run. Full search-verify/probe/worker suites pass; ruff clean; mypy
unchanged (only the pre-existing file_bindings lambda note).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Replays a stored demo through the executor task and the verifier lane for one
side or both, with per-phase timings, so verifier changes are checked on the
exact image before any paid search.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The pip recipe ran under the system Python 3.13 while apps started under the
venv 3.12, so compiled extensions never loaded. Both now use sys.executable.
Package-style starts see the checkout root; NODE_PATH lists the project
node_modules under the checkout before the products mount.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Demos may reference ${served_root}, ${database_path}, ${marker_dir} or
${callback_url}; the verifier substitutes them per side, so apps that take a
directory or database argument can be wired to the oracle. Python apps start
under the verifier interpreter (the one pip built the products for): the image
carries a second system Python whose ABI tag broke compiled extensions.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Archives are gzipped and uploaded in parts of at most 90 MB with per-part
digests (edge request limit); pip installs ignore the task venv so transitive
dependencies land in products; the verifier sets NODE_PATH and the Java
classpath from products when starting the app.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
HTTP start waits up to 90 s (CLI 30 s) with TCP then HTTP probing; app stdout
and stderr tails are recorded; tool-call turns get 4096 output tokens with one
retry on truncation; tasks get 40 steps with reminders.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The kind default "pin" had produced ousast-engine-pin-search-verify-* names,
which the operator per-host egress rules do not match.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Built projects spool to /workspace without package caches (cap 1 GiB, both
sides); a plain-text model reply is fed back once; mailbox PUT/GET retry for up
to 90 s with idempotent re-send by seq; executor tasks are deleted on every exit
path.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Demos may set a bounded environment (loader variables denied) and up to 8 small
data files under .demo/; the reason step sees conclusions and may not re-propose
them; a task spend limit concludes its intent, not the search; finish wins over
stray tool calls.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
All five oracles demonstrated, gaming demos inconclusive, 38 s, ~500 MB RSS.
The probe dispatcher accepts a skipped isolation check in task-boundary mode.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The agent passes a manifest file or project directory; pip uses -r for
requirements files and the directory for pyproject/setup, npm ci/install runs in
the directory, composer/maven/gradle likewise. Build and verify failures carry
phase, exit code and a sanitised stderr tail.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
On kube-ax the probe detected task-boundary and then failed the bwrap preflight it
no longer needs; now it skips the preflight, runs every oracle pair through the
verifier path a task uses, and reports ok only if every real demo is
demonstrated and no gaming demo is. Fixes a double fixture-path translation.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Demos declare their oracle within the family allowed set (injection: sql or
command; output_encoding: xss; untrusted_destination: ssrf; path: path);
builds and app starts use the search memory budget instead of the 256 MB tool
limit (npm died with SIGTRAP); malformed tool-call JSON is fed back once.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
budget_spent carries end_detail (spend, task_spend, reason_rounds, ...); a task
whose allowance cannot cover the next call finishes with its facts; per-task
metrics in pilot records; input reservation bytes/2 + 1024 per message (measured
3-4x over-reservation before).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
One base (Temurin 21 JDK, node/npm/tsc, php/composer, maven, sqlite, bwrap)
built only when its Dockerfile or dependencies change, plus a browser variant;
engine/search/VM images FROM base@sha256 with the code in the last layer; the
image workflow runs on dispatch, tags, Dockerfile or pin changes, with a build
cache. Executor NPROC bound relative to the current process count.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Settled ~$0.003 per model call; worst-case reservations exhausted the $0.05 task
allowance after 2-4 calls. Now $0.40 per task, $2 per search (maintainer decision).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The operator keys per-host egress rules on the task-name prefix; engine passes and
replays had none. Names stay within 49 bytes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Zero running workers behind "no free workers available" now raises
ClusterUnavailable instead of waiting out the capacity deadline; CapacityDeadline
carries its message; cleanup failures no longer mask the original exception; the
pilot driver aborts the run on infrastructure failure before any model call.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
First live run: the reason step got ~190 tokens and returned {} -> silent
no_evidence. Now: a bootstrap intent reads the candidate and its entry point; the
reason prompt states the goal, family oracles, demo schema and must propose
intents when none are open; empty replies retry once then end as reason_refused;
failure records keep messages.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
One side per run over selected pilot pairs: board, brain on the VM with the
project DeepSeek client, executor tasks on kube-ax, verification on the VM lane
(AX verify lane switchable). Hard run ceiling with reservation before every call,
per-search and step caps; fixed side never sees the vulnerable revision; records
carry digests only. Dry run with the stub model reaches demonstrated.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Operator 2026-10-05: Substrate boots a ~1 s golden-snapshot actor per new task on the
other worker; retry apply after 2-5 s for the first minute, then 15-30 s.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Reasons carry type plus a sanitised message tail; command failures carry phase,
exit code and bounded stderr. Scratch walk tolerates directories pip removes
mid-walk; writable /workspace/home and runtime paths. Capacity retries delete a
possibly created task before resubmitting. Docker smoke checks the image first.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The official Python base sets GPG_KEY (public CPython signing fingerprint); the
executor rightly refused it as a credential-like name on kube-ax.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
20 development pairs, one per repository, 4 per language, 5 per family; identities
in a gitignored manifest, counts and digest committed. Executor smoke drives one
executor task end to end without a model and fails loudly on an unread clone.
kube-ax operator constraints as steering.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Demos are data the verifier executes (fixed build recipes, runtime+path start,
HTTP/CLI steps with captures; the canary is never referable). The agent loop with
the key never runs repository code: executor tasks without a key exchange commands
and results through presigned objects. AX capacity errors are back-pressure with at
most 2 tasks in flight; caches and builds under /workspace with a 2 GiB guard;
verifiers take one presigned archive per side. Launcher fix: private /dev in bwrap.
190 tests on the host; probe 5/5 demonstrated.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Replaces the accepted key-exposure exception: the key never shares a sandbox with
repository code; executor AX tasks exchange commands and results via presigned
objects. Firecracker executors only if the pilot build-failure rate requires it.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
kube-ax probe: AX tasks have no capabilities (no namespaces, no setuid), so no
in-task isolation. Demos become data the verifier executes; the explore key risk is
accepted and bounded by cap and rotation.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The kube-ax probe showed bwrap cannot map uids in gVisor as an unprivileged user.
As root (gVisor is the outer boundary) the sandbox uses pid/ipc/uts/net/mount
namespaces without a user namespace and drops the artefact to its own uid via
setpriv; the mode is recorded. The probe reports each namespace kind separately.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Probe runs each oracle toy pair with a real demo and two gaming demos, records
image facts, RSS and disk, fails loudly if isolation is unavailable; dispatchable
as an AX task. Worker: reason step without tools, explore tool loop whose run tool
goes through the sandbox with a scrubbed env (model key absent), spend reserved
before each call. Host probe: 5/5 real demos demonstrated, 10/10 gaming demos
inconclusive. 89 tests.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Board as immutable versioned blobs (coordinator sole writer); budget reserve/settle
before model calls; per-task and run caps; coordinator loop over an injected
executor; batch.py accepts a fixed allow-list of search variables; XSS browser
oracle; ousast-search-task image (Python, Node/TypeScript, PHP, Java 17, sqlite,
bwrap, chromium) built by CI. 103 tests on the host.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The verifier, not the artefact, observes the effect: SQL, path, command and SSRF
oracles owned by the verifier (XSS needs a browser executor, else no_oracle);
artefact exit code and output ignored; immutable checkout and dependencies;
3 runs per side; one-sided build or readiness failure is inconclusive. An
isolation failure raises IsolationUnavailable instead of reading as
could_not_build (the NPROC limit now applies inside the user namespace).
35 tests, including gaming attempts and 300 host processes.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Req 6.5 (maintainer decision 2026-10-05, with search-with-proof): BLOCK comes from
verified demonstrations; the engine is judged by top-K recall per push and advisory
false alarms; the pooled block gate governs only families without an oracle.
Task 17 adds the metric and re-scores existing arms at no cost.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
False alarms count every visible finding; the verifier owns per-family oracles;
inconclusive on one-sided build failure; feasibility pilot separated from a
73-negative BLOCK qualification; coverage ceilings; spend reserved before calls;
key isolated from executed code; cluster model calls supersede the pool VM-only
rule for this arm only. Requirements approval reopened.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
After reviewing Cairn: detection so far classified excerpts and hit a precision
wall; this adds directed search with an objective goal (a re-verified
demonstration), a fact/intent board, and reason/explore steps as AX Tasks on
kube-ax. Design only learns from Cairn (AGPL); no code copied.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
gVisor replays exhausted the hook's 30 s deadline in the snapshot step on a
1,725-file repository; cluster replays now use 30 s x a measured slowdown
factor, verified on a native sample. The engine runner deletes each pin's fetched
source (1.8 GB had accumulated), caps the cache at 500 MB and refuses below
1.5 GB free. Proof 4.4: two engine pins via the generic dispatcher match the VM
baseline exactly. Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Implemented with Codex; task 1.5 run by the coordinator against the real
sources (S3 memory read-only).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Decisions: long-deadline arm (300 s), adoption needs no unseen regression
(Req 5.4), permissive repositories by name only, two AX lanes. Also adds the
trained-classifier and joint-optimisation arms to learned-decision-engine.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Maintainer 2026-10-04: the safety net is for unseen code and the A/B tests exist
to prove the approach generalises; infrastructure work had drifted from that.
Runs are designed for AX on kube-ax, kind is the fallback.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Two engine actors died at gVisor level during a 12-pin run (AX phase Failed, no
result). Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Through Cloudflare, signed requests for keys ending .tar/.gz/.zip/.bin reached
RustFS altered and failed with 403 (cf-cache-status BYPASS); the same signed
request straight to RustFS on the LAN returned 200. A Cloudflare cache-bypass
rule on files.because-security.com fixes it (all extensions 200, DYNAMIC); the
engine executors keep .dat as a safeguard, the plane keeps output.tar.
Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The Joern leak-audit record named spent population v1/v2 repositories, which
the reserved-repository guard forbids outside benchmarks/independent/ (no v3
repository was involved: 0 matches). Records now carry counts and SHA-256
digests. With Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Coverage 72 path, 231 asked-nothing, 9 failed, 189 unsupported. The engine
almost never distinguishes a vulnerable function from its fix, so a Joern-slice
prompt experiment is not funded; the lever is the engine's knowledge of fix
mechanisms.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A lean overlay omits otel-collector, prometheus and jaeger and disables the
exporters; on an existing cluster up.sh scales them to 0. Verified live: render
has no otel-system resources, up.sh idempotent (54 s second run), smoke Run
done, kind node 2.66 -> 1.82 GiB. Docs say server VM. Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Batch 3 of real (vulnerable function, fixed function) pointer pairs, mined at
zero model cost from reviewed GHSA advisories (pip, npm, maven, composer,
published since 2023, newest first). Same rules as batch 2, widened three
ways: a merged PR's single commit when the advisory links no commit, commits
up to 6 files / 240 lines, and two-family advisories with the family read
from the diff. Every candidate's function diff was read: 412 reviewed, 255
kept, 3 one-per-repository, 2 withheld by the reserved-population guard
(bisected with output discarded). Permissive licences only; no repository
named elsewhere under benchmarks/; no WordPress plugin.
Kept 251: injection 56, untrusted_destination 26, path 71, output_encoding
69, access_control 20, deserialization 9; python 83, typescript 48, php 46,
java 37, javascript 37. Only 12 post-cutoff: batch 2 had used up the
post-2026-06-01 pool. Memory 1133 -> 1635 static examples; complete pairs in
the whole memory 459 -> 710. Embeddings: 15 calls, $0.0030. No chat-model
call, no Docker, no Joern. Record:
benchmarks/measurements/2026-10-04-decision-engine-harvest-3.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The query fix stops a class-level call from aborting the whole taint batch on
real PHP code; gate, map_gate and pair_gate outputs are unchanged. Trace parsing
lives with the runner (benchmarks/learn/engine_trace_parse.py).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
On real PHP source (CPS-IT/mailqueue) a call outside any method has a null
.method; inScope/nestedInLabeled dereferenced it (NullPointerException) and the
script exited 1, losing every taint answer for the repository. 16 sites in
taint/dominance/config/vocabulary.sc now treat a null method as out of scope;
non-null behaviour is unchanged (same outputs on the repro graph for the other
scripts). Static regression test added. Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The worker sent tuple params as JSON arrays; taint.sc expects comma-joined
strings (ujson InvalidData, exit 1). Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Root causes of the 50 failed pins in trace run 1: advisory pairs were fed as
excerpts (invalid Java/JS fragments), the JVM defaulted to ~768 MB heap in a
3 GB container, and mixed-language pins were refused. Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The 2026-09-14 engine image has no PyYAML or boto3; the worker no longer imports
learn.examples. Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
slice.py renders source -> step -> sink lines with fixed wording for flow,
no-flow and not-analysable; the prompt option is default-off (byte-identical
default prompt). semantic IR: JS/C declarations count as assignments, augmented
assignment keeps its target (gate, map_gate, pair_gate outputs unchanged).
Audit over exp-004's 501 units: 59 flow, 161 no-flow, 281 not analysable;
has-flow within-pair accuracy 0.52, pooled AUC 0.55 -- no leak. Fixes a
numbering bug that rejected every excerpt with a blank line. With Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Population v2 scans are named <case>--<side>_a/_b.json; the script looked only
for <case>--<side>.json and recorded those examples as engine not-run. The two
repeats combine symmetrically for both sides. Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The estimator priced all input at the full rate (\$5.58 estimated vs \$1.24
spent); it now uses the family mean billed cost of cached responses when usage
is recorded. Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A pending-then-frozen registration left two rows and rows[0] could be the stale
pending one, so fill and run refused a properly frozen experiment. Implemented
with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Implemented with Codex; review fix: the bounded client keeps the configured
thinking setting so a fill answers as the run would.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Supplementary analysis, not a new decision: exp-003's verdict stands.
The 142 units excluded for an uncached response were filled with the same
six incumbent programs, parameters and request keys, on exp-003's
937-example memory (the harvest's 196 rows and its 2026-10-03 embeddings
left out; a zero-cost replay first reproduced exp-003 exactly: 211
decided, 1056 replayed, $0.582876). Fill: MeteredClient, ceiling $0.99
under the approved $1.00, 697 client calls, $0.331759, no provider error.
Replay on all 353 units (244 paired, 146 groups): 0 client calls, $0.0,
1768 responses replayed; no arm flags any unit, the registered rule reads
inconclusive. Calibration holds only for untrusted destination (ECE
0.0454) and arm A still finds no BLOCK point there.
benchmarks/experiments/exp-003-precision-bound-blocking/result-complete-cache.json
exp-004 registered in s3://sast-memory (manifest 87fceb65..., pin
2ad9c1d, units pending): benchmarks/experiments/exp-004-pooled-block-gate/
registration.json. docs/decision-engine.md: both, briefly.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer decision 2026-10-03: BLOCK precision is shown pooled across
families. Req 6.4 (amended; 7.4 annotated): one t_block on a per-family
Platt-normalised score (calibrate.fit_map, training folds, inner
cross-fit), Wilson 95% LB >= 0.95 over all families' training flags; a
family blocks only with >= F = 10 held-out flags and zero errors, else
advisory; t_quiet per family keeps >= 90% of training paired positives.
evaluate: normalised, pooled_threshold, quiet_threshold,
pooled_block_gate, family_floor, apply_floor. experiments: rule
pooled_block_gate (normalise: platt, floor, recall), decide_pooled on the
pre-floor pooled held-out LB, floor table, quiet_recall and after-floor
totals in analyse_rule; exp-003's analysis is unchanged.
plane/experiments/exp-004-pooled-block-gate.yaml: procedure and decision
rule registered now; units frozen at run time from the post-harvest
memory (units_status pending until then).
Tests (tests/test_learn_rules.py): pooled threshold where no family alone
reaches the bound, no held-out label moves a threshold or the map, a
family with one held-out error cannot block, t_quiet and no blind
silencing, the pooled verdict, the exp-004 manifest.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Real (vulnerable function, fixed function) pointer pairs for the six compiled
families, mined at zero model cost from reviewed GHSA advisories (pip, npm,
maven, composer), newest first: 79 of 98 published after 2026-06-01
(post_cutoff = true). Each pair: one linked one-parent fix commit, <= 4 source
files / 120 lines, the function holding the changed code lines declared once on
both sides with a changed body; every function diff read and kept only when the
named function holds the fix (145 reviewed, 100 kept); 2 dropped by the
reserved-population test; permissive licences only, no repository named
elsewhere under benchmarks/.
New label source advisory-fixes-2; the advisory-fix tests run over both
batches. Memory 937 -> 1133 static examples (98 paired groups added);
positives-only groups stay 99 (vibe-py/dev-php/v2 data gaps), their share of
positive groups in the six families 0.40 -> 0.29. Embeddings: 7 calls,
$0.0014. No chat-model call. Record:
benchmarks/measurements/2026-10-03-decision-engine-negatives-harvest.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Zero-cost replay of the six slice-2 programs: meter 0 client calls, $0.0;
211 of 353 units scored, 142 excluded for an uncached response (path lost
all 39). Arm A (calibrated) fails calibration (ECE 0.077 access control,
0.1067 injection); arm B (Wilson LB >= 0.95, nested) chooses no threshold on
any fold, as the registration's power note predicted (73 clean flags needed,
38 paired injection positives); secondary arm C flags nothing either. The
instrument check shows real scores; the zero is the rule's.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Registered in s3://sast-memory at dd7ebc1: manifest 5917f9a1, units 338de998
(353 units, 146 groups). Primary metric, decision rule and the threshold
selection are fixed in the manifest; the power note says arm B cannot reach
a Wilson LB of 0.95 on this data (73 clean flags needed, 38 paired
positives at most).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
kind: rule experiments score once from the response cache behind a
zero-ceiling meter (a miss is excluded and counted, never asked) and apply
each arm's BLOCK rule to the same held-out scores. Arm B chooses a per-family
raw-score threshold on the other outer folds only (lowest t with Wilson 95%
LB >= 0.95); a test fails if a fold's own labels move its threshold.
Manifest and frozen units (353 units, 146 groups, 244 paired) for exp-003.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Kind proofs: records 01 and 02. Task 2.6 (receiver removal) is paused while the
safety net and the decision engine come first. deployment.md conflicts resolved
in the plain-language style; corrects the RuntimeClass claim (Substrate runs
runsc inside its worker pod).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Live on kind with the runner image that carries 2.2
(ousast-runner@sha256:d2a55293...; templates and validation-46 Tasks repinned,
128 lines; kind-images.json updated): the smoke Run on store delivery finished
in 26 s and 24 s; the tar was PUT to runs/ax-e2e-smoke/facts/output.tar
(10240 B) and collected 2 s later; read back from the store its members
(facts.json 451 B, summary.json 238 B, units.jsonl 291 B) equal the extracted
files; facts.json, summary.json and done.json (149 B) re-put, state.json
(350 B) mirrored; a rerun from an empty results root seeded state.json from the
store and finished in 2 s without resubmitting. The actor's egress policy read
back live: one tlsPassthrough rule on 443 naming the store host, nothing else.
`ousast plane status ax-e2e-smoke` printed the table each time. Record:
benchmarks/measurements/2026-10-02-k8s-02-store-delivery/record.json with the
smoke outputs, store read-back and policy beside it. Task 2.6 (receiver
removal) is paused by the maintainer's reprioritisation.
Gates: pytest 2031 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0. (The first run, at 0.97 GiB free disk, failed 11 tests and errored 5
on the FileStore's 1 GiB free-space refusal; rerun after the disk was freed.)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Rerun records, serve-static and unrolled/secure (same planted commits) with
this build installed from its wheel: 6 replays and 10 hook pushes, all exit 0
in 0.69-1.21 s against the 30 s deadline, input read proven per run. The
planted JS eval in a new file surfaces in the quick tier on the host and, in
one Docker engine run (190.9 s, deadline 300), as two advisory engine findings
classified new/file_added_in_head. The planted Python %-formatted SQL is still
missed (no quick rule, no host engine): recorded as a coverage gap, no rule
edited. Go is named as not covered. The run caught a wrong install hint in
the hook (the package is not on PyPI); corrected here.
Record: benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
`ousast pre-push install|uninstall [REPO] [--force] [--docker]` ships in the
wheel with the hook as package data; it honours core.hooksPath, refuses an
existing hook unless --force (which chains it as pre-push.before-ousast), and
uninstall restores the chained hook. ops/ousast-docker is a hook-callable
script: the repository read-only and the artifact directory at their own
paths, stdin passed through, no network, --memory 3g, the GHCR 2.0.1 image by
default, advisory exit unless blocking. Help text for every pre-push flag; a
first-run guide, settings, skip-reason glossary, CI recipe and full --help in
docs/scanning.md; README tiers 1 and 2 pull the published image. The image
was not rebuilt: 1.1 GB free, below the 4 GB threshold. Requirements 7.4, 9.6.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
--engine inline|background|off (hook: OUSAST_PUSH_ENGINE). background returns
after the quick tier and starts one detached explicit replay of the same base
and head (push/background.py), one at a time per artifact directory; its
artifact lands under <artifact dir>/engine/<repo key>/<head>.json and the next
run prints "Engine result for <head> from the previous push" once. A deadline
miss says so in one line naming the deadline and the background option. No
engine speed work. Requirement 9.5.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ChangeContext records the paths Git reports as added. An entailed,
undischarged operation in an added file, with complete head evidence and no
identical operation in any base answer, is classified new
(file_added_in_head) instead of base_incomplete; admission accepts that reason
as change evidence. Unpaired add/delete pushes and every other unknown
comparison keep their vetoes. An engine finding whose novelty is established
and whose only vetoes are capability vetoes is shown as an advisory engine
finding, not an alert. Requirement 9.4.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Before the engine and inside the same deadline, the shipped quick rules run
over the head side of the changed files (push/quick.py) and keep matches on
changed lines or in changed functions. They print as advisory pattern matches,
labeled apart from admitted alerts, are kept in the artifact's quick_tier with
the files and bytes read, and never admit or block. A new branch with no
remote base runs the quick tier over its last commit and says so. The planted
eval in serve-static's new debug.js now surfaces; the planted Python
%-formatted SQL in records does not (a quick-rule coverage gap, no rule
edited). Requirement 9.3.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
One terminal line per distinct coverage reason (push/explain.py): hex-encoded
paths decoded, per-question sha256 tokens dropped and counted, unknown reasons
still printed; the JSON artifact keeps the raw values. A changed language with
no engine frontend and no quick rules is recorded and printed as "language not
covered: <lang> (...)", in pre-push and in `ousast scan`. The scan report
renders the eight degradations cli.py recorded but never showed, and any
reason without a sentence. The hook forwards OUSAST_COMPARISON_BASE.
Requirement 9.2.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The installed hook exits 0 with one line when ousast (or OUSAST_COMMAND) is
not on PATH, still chaining a prior hook. In Git hook mode without an explicit
--mode blocking, argument/usage errors (an invalid OUSAST_PUSH_DEADLINE, a cache
directory inside the repository, a refused experimental flag) and unexpected
exceptions print one "check did not run" line and exit 0. Blocking mode keeps
its usage errors. Requirement 9.1.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
_execute builds the task's `delivery {put, inputs}` from Delivery, passes it to
router.starter, grants the store's host in the egress policy, awaits
Delivery.delivered (the tar's exact key in the store), then collect()s:
download, extract with the member checks, re-put the declared artifacts, write
done.json; the outcome is the collected summary.json. Three findings from the
tests: a rerun of an unfinished task completed at once on the previous
attempt's tar, so Delivery.reset(task) deletes output.tar and done.json before
every attempt; a worker thread that raised left the run blocked on its queue,
so the worker now turns any exception into a failed task; a consumer may
declare a producer's summary.json without the producer listing it (the loop
Run does), so declared(run) publishes outputs[] plus every artifact a consumer
names. The receiver thread survives behind OUSAST_DELIVERY=receiver for this
group only (no presigned URLs, the http rule, OUSAST_ARTIFACT_URL in the env);
the default path renders no receiver address into any Task. The fake ax PUTs
to the start body's URL and GETs its inputs from theirs; the reconciler tests
run through the S3-shaped fake store and assert the objects, done.json, the
seeded rerun and the receiver mode. reconciler.py: 549 lines (bound 550).
Gates: pytest 2031 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Append task group 17 "Operational readiness (2026-10-02)" with one task per
maintainer-approved item (never block, name every skip, quick tier, new-file
classification, engine honesty and background mode, install/Docker/docs, proof),
add Requirement 9 (9.1-9.6, marked added 2026-10-02, maintainer-approved), and
check 13.4 and the parent of 14 as the audit found them done.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
policy_for(task, workspaces, model, store_host, receiver_host=None) emits one
tlsPassthrough rule on 443 over the sorted union of the memory store's host,
the bound Workspaces' https Git hosts and the bound Model's egress hosts. There
is no http rule: an http Git repo is refused by name (it could not be reached),
a store given as an address is refused naming S3_ENDPOINT (the gateway allows
hostnames only). receiver_host adds the old receiver's http rule for the path
group 2 keeps until 2.6; the reconciler passes it by keyword (passing the
receiver's address positionally would have raised outside the worker's try and
hung the run on the queue -- 2.4 hardens the worker). Tests: the store-only
policy, the Model-bound policy, the two refusals, the receiver exception, and
every committed template under plane/tasks/ rendering exactly the store plus,
for verify and roles, the provider host.
Gates: pytest 2027 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
StartGate.offer validates an optional `delivery` beside the credentials: https
URLs to a hostname (never an address, no userinfo; AX_RUNNER_INSECURE_DELIVERY=1
admits http in unit tests only), names ^[A-Z][A-Z0-9_]*$, a 400 that names the
field and echoes no URL. The runner PUTs its output tar to delivery.put and
GETs each declared input from delivery.inputs[NAME] (deliver() gains a method;
log lines and DeliveryError carry the URL without its query, so no presigned
signature reaches a log, a marker or a summary). OUSAST_ARTIFACT_URL/DIAL stay
honoured until task 2.6 so both paths run in the same build; the completion
marker and the retry-only boot are unchanged, and a re-start brings a fresh
URL for the retry. Router.start_task carries `delivery` in the body and
redacts its URLs like credential values. Tests: the gate's acceptance and
refusals, an end-to-end PUT/GET against the S3-shaped fake with no URL in any
log or file, the failed-PUT-then-retry-only boot, deliver() with PUT.
Gates: pytest 2024 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
src/openultrasast/plane/delivery.py: Delivery(store, run, base) mints a presigned
PUT for runs/<run>/<task>/output.tar and a GET per declared input
(runs/<run>/<producer>/<artifact>), expiry OUSAST_TASK_TIMEOUT + 600 s;
delivered() asks the store for the exact key; collect() downloads, extracts
under the run directory with the receiver's member checks (no symlink, hard
link or escape; nothing written on a rejection), re-puts the declared outputs
as single objects and writes done.json; put_input() for inputs no task
produces; save_state()/load_state() mirror state.json. The reconciler opens the
store first (no fallback; the CLI reports a store that cannot open and submits
nothing), mirrors state.json after every change and seeds a rerun with no local
state from the store (the 2.1 bullet's state mirroring is what gives the new
module its importer). The reconciler crossed 500 lines here (517) rather than at
2.4 as 1.6 projected: the bound moves to 550 with the reason in the assertion,
as 1.6 decided. tests/plane_fake_store.py is the S3-shaped fake (objects in a
dict, presigned URLs on a local server); tests/test_plane_delivery.py covers the
body, expiry, polling, rejection, declared-only re-put, state seeding and that
no URL reaches a log. Module-audit manifest regenerated (148/5, no orphan).
Gates: pytest 2020 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
benchmarks/measurements/2026-10-02-k8s-01-profile/record.json with the live
evidence of group 1: `ousast plane doctor --profile kind` passes every line
(exit 0, 1.4 s; the store check reaches the bucket: verify 0.62 s, probe object
24 B); `--profile k3s` fails on the unreachable context and on the images file
the release has not published yet (two expected lines; the namespaces are not
checked behind an unreachable context); the literal scan has 0 offenders (23
before 1.3) and fails naming workspaces.py:37 when a literal is reinserted;
up.sh reran idempotently in 38 s and the smoke Run finished in 24 s on the
rebuilt image (facts.json 451 B, summary.json 238 B); the placement probe says
`any`; the reconciler bound numbers of 1.6. Doctor, literal, smoke and up.sh
outputs beside it.
Gates: pytest full suite on this tree's src/ and tests/ content: 2010 passed,
42 skipped, exit 0 (the 1.5 run; the 1.8 run reported two module-audit
failures caused by the uncommitted 2.1 module present in the working tree);
ruff check 0; ruff format 0; mypy 0; mkdocs build --strict 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
docs/deployment.md B.1: Substrate's ateom-gvisor worker runs runsc from the
release tarball named in manifests/ate-install/sandboxconfig-gvisor.yaml
(gs://gvisor/releases/nightly/2026-09-02/<arch>/gvisor.tar.zstd, sha256-pinned
per architecture), fetched by the atelet on the node; the node must reach it or
a copied SandboxConfig must name a mirror with the same sha256. The check is the
atelet's fetch line in the worker pod's logs, run by hand on the day k3s exists;
labelled planned until then (design open question 3 stays open).
Gates: docs only; mkdocs build --strict passed with this paragraph present in
the 1.4 gate; no file a test or linter reads changed since the 1.5 gate.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
reconciler.py: 477 lines at 6bd44ca, 489 after 1.2 (the profile parameter of
run() and its docstring). Task 2.4 adds the Delivery wiring (about +25) while
the receiver path is kept for group 2, and 2.6 removes the receiver wiring
(about -20): projected 514 during group 2, 494 after 2.6. The estimate crosses
500 while both paths coexist, so 2.4 moves tests/test_plane_reconciler.py's
bound to 550 with the reason in the assertion message; 2.6 records the final
count. Numbers in benchmarks/measurements/2026-10-02-k8s-01-profile/record.json
(record 01, drafted here, completed by 1.8).
Gates: no file a test or linter reads changed since the 1.5 gate (pytest 2010
passed, 42 skipped, exit 0; ruff 0; format 0; mypy 0).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ops/ax/probe-placement.sh scales ousast-pool to one worker, applies probe-pool
(1 x 2 CPU / 3Gi, the already-built ateom-gvisor image, no ko build), runs one
model-free repo-facts Task requesting 2560Mi (limit 3Gi) and one requesting
256Mi, records `kubectl ate get workers -o json` after each resume, and removes
everything it created. Live on kind: both actors landed on the 1536Mi
ousast-pool worker (status.allocated.actors=1 there, 0 on probe-pool) ->
placement: any. design.md section 4 records the outcome and the fallback group
5 must take (an engine atespace with engine-pool as its only pool if Substrate
scopes pools by atespace; else a plain Kubernetes Job on k3s). Evidence under
benchmarks/measurements/2026-10-02-k8s-01-profile/probe/.
Gates (tree identical to 1.4 in src/ and tests/; ops/ gains the probe script,
which the literal scan covers): pytest 2010 passed, 42 skipped, exit 0; ruff
check 0; ruff format 0; mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ops/ax/up.sh reads kube_context, registry, pools.default and the images path
with `python -m openultrasast.plane.profile --print`, writes the runner digest to
ops/k8s/profiles/kind-images.json instead of ~/.cache/.../runner-image, renders
ops/ax/workerpool.yaml.tmpl from the profile's pool (POOL_NAME/REPLICAS/CPU/
MEMORY, WORKER_IMAGE), and keeps an existing pool and an ax CLI newer than its
checkout instead of rebuilding them (ko and go install would refill several GB
of Go cache on this disk). smoke-run.sh takes the runner from `--print
images.runner` and runs the checkout's code with OUSAST_PYTHON. repin_templates
reads the JSON images file (runner key; the one-line pin still reads). `--print
images` returns the file path (the first live rerun fell over this). Templates
and the generated validation-46 Tasks re-pinned to the rebuilt image (128
lines, one digest).
Live: up.sh reran idempotently on kind in 38 s (every step kept except the
runner build); smoke-run.sh done in 24 s on the new image, facts.json 451 B,
summary.json 238 B delivered (record: benchmarks/measurements/2026-10-02-k8s-01-profile/).
Gates: pytest 2010 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0; mkdocs build --strict 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
tests/test_plane_literals.py scans src/, plane/, ops/ (minus ops/ax/up.sh and
ops/k8s/profiles/) and docs/ for the kind context, the kind registry, the kind
network's gateway, a loopback address with a port and port-forward; router.py
keeps the last two, a line tagged ax-tunnel or loopback is allowed, and a
committed template's digest-pinned image is accepted only while it names the
kind profile's registry (checked by a third test). A self-check reinserts each
literal into a fixture and sees it caught. The runner's health URL and the Joern
session's query URL are tagged loopback; docs/deployment.md and ops/ax/README.md
stop naming this host and describe the kube context and the registry as the
profile's (what 1.2 made configurable).
Gates: pytest 2009 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0; mkdocs build --strict 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
doctor(profile) checks the profile's kube context, the two namespaces behind a
reachable context only, the images file's runner reference in the profile's
registry (a local registry over its v2 API, a remote one via docker manifest
inspect) and opens the memory store; its first line reports every configured
address (Req 1.2). Egress(executable, context), open_router(context, url=...),
receiver_cluster_ip(context) and Receiver(..., context) take the profile's
context and build none; generate.increment/render and harvest take the profile's
atespace. `ousast plane doctor|run|workspaces|harvest --profile`. The test fakes
(kubectl-ate, kubectl, ax) record the context they received and the tests assert
it is the profile's. grep "kind-" src/openultrasast/plane: no match.
Gates: pytest 2006 passed, 42 skipped, exit 0; ruff check 0; ruff format 0; mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ops/k8s/profiles/kind.toml and k3s.toml hold the values that differ per cluster
(exec, kube_context, registry, images, router_url, ax_server, memory, atespace,
snapshots_bucket, image_pull_secret, pools); kind-images.json carries the digest
pin ops/ax/up.sh wrote. load_profile() reads OUSAST_PLANE_PROFILE (name or path,
default kind), applies OUSAST_<FIELD> and OUSAST_POOL_<NAME>_<FIELD>, and refuses
an unknown key, a credential-shaped key or value, an exec outside local|remote
and an images file without digest pins. `python -m openultrasast.plane.profile
--print FIELD` serves shell scripts. The module-audit manifest is regenerated
for the new module (counts 147/5, no orphan).
Gates: pytest full suite 2002 passed, 1 failed (the module-audit manifest, fixed
by regenerating it; tests/test_module_audit.py re-run exit 0), ruff check 0,
ruff format 0, mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
CI and the release job check `ruff format --check .` and `ruff check .` over the
whole repository, while the local gate checked only src and tests, so eight
benchmark scripts drifted and the v2.0.0 release job stopped before building.
The fixes are formatting, two noqa comments and one renamed loop variable; no
script computes anything differently. The local gate now checks the whole tree.
The release workflow gains an images job: Dockerfile -> ghcr.io/<owner>/openultrasast,
plane/Dockerfile.runner -> ghcr.io/<owner>/openultrasast-runner, tagged with the
version, and the digests written to the job summary.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Rewrite README.md as the front door of the docs site: a short "What it is", the
Status (NO-GO against M4, figures quoted from docs/where-we-stand.md), a tiered
install (quick scan without Docker; standard/deep with the openultrasast:dev
image; the plane on ax for maintainers; k3s planned), then one section per
docs page, each linked by relative path so it works on GitHub and on the site.
Stale claims removed: `ousast index` (deleted), `OPENROUTER_EMBEDDING_MODEL`
(no longer read), the duplicated scan-mode, rule and gate tables (now on
docs/scanning.md and docs/evaluation.md), the HarnessX-era wording. Adds a
Licence section stating that no licence file exists yet.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Legacy inventory 2, the six STATUS NOTE items (the archive README row landed with the R2 commit).
ai-service-plane: 11 of 11 tasks checked, spec.json phase set to implementation-complete as 5b96199
did for four other specs. contributor-scan: dated note in requirements.md -- paused since 2026-09-23,
27 of 51 tasks, v1.2.0-alpha.1 NO-GO, remaining tasks wait on pre-push M2/M3. finding-feedback-loop:
dated note -- dormant, design unapproved since 2026-09-09, depends on the paused spec. The steering
overview's spec table carries the same three states, its leftover 2026-09-09 module-count line is
removed and the collected-test count is the measured 2,023. docs/memory-and-detection.md names the
archive path of model_sinks, the one stale script name left in docs after 067adb9.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
New page docs/detection-techniques.md: the three representations (regex over the
file, tree-sitter IR, Joern CPG) and what each can see; Joern as a section of its
own (frontends, one graph per language partition, census as the instrument check,
batching against the measured cost curve, deadlines, degradations, where the engine
sits in the plane, cost and misses with records); taint from fact tables to query to
verdict with each refinement traced to its record or commit; obligations, dominance,
constant abstraction, census; regions, ranking ties and budgets; the residual
question and MAX_JUDGED_CANDIDATES; REGRESS verdicts and the unassigned
execution_confirmed rung; an honest table of what each technique misses.
Nav entry under "How it works" after Scanning; one link from the index; one
cross-link line each in scanning.md and architecture.md.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
openultrasast-scan: evidence ladder as verification.py defines it, with the levels nothing produces today; --mode prerequisites, --fail-on, recorded degradations, run artifacts, ousast mcp; retired [evidence] threshold language removed. openultrasast-triage: reasons verified against calibration.py; deterministic fusion; adjudications as labels, plane verdicts as features, rule changes through improve or a registered experiment; fix-audit merged as a fix-and-re-scan section. kiro: spec-quick and spec-batch removed; validate-gap and validate-design merged into spec-design (--gap, --review; the review gate stays); steering-custom merged into steering; core steering names overview.md, roadmap.md, safety-net.md; instrument rule in debug and verify-completion; repo gates named in validate-impl. AGENTS.md, README skills paragraph and tests/test_agent_skills.py (.agents/skills, two product skills, drift tests) updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Legacy inventory 2, item R3. The three keys loaded into ResolvedConfig and nothing read them: no
`.network`, `.workspace_readonly` or `.top_k` access exists in src outside config.py. The sandbox
runner always passes `--network none` and `--read-only` and rejects host networking by argv
inspection, so the two sandbox keys were knobs that turned nothing; no stage read top_k, and
`[complexity] max_hunter_hotspots` is the one complexity key the scan reads. They join RETIRED_KEYS
with the same 2026-10-02 mechanism as the other unread keys (RetiredConfigError naming the reason).
The fields and their loaders are removed; memory_mb, timeout_seconds, pids_limit (read by
regress/recipes.py and sandbox/runner.py) and max_hunter_hotspots stay. test_config.py covers the three
retirements and the override test now exercises max_hunter_hotspots. RELEASE_NOTES Unreleased carries
the line.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reworks the production topology section after the maintainer's three
decisions of 2026-10-02: the target cluster is k3s (containerd, no Docker
socket, runsc + RuntimeClass for gVisor workers, Traefik) with digest-pinned
images published to GHCR by GitHub Actions; kind stays a first-class local
execution mode (two profiles of one code base, every change proven on kind
first); and the reconciler stays the CLI, with a planned remote mode in which
the same `ousast plane run` submits to ax-server in k3s, tasks deliver their
artifacts to the S3 memory store through presigned URLs, and the CLI polls the
store for the same status and attribution table. The in-cluster reconciler
and the in-cluster receiver become a later option. The diagram is redrawn
without an inbound path; the pre-push section says the hook or CI calls the
CLI in remote mode and either waits with a deadline or reads a later `plane
status`. Every item that is not built stays labelled planned.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Answers the maintainer's questions on deployment.md: ax is a Kubernetes
application (ax-server Deployment + Redis in ax-system) that schedules every
Task as a gVisor actor on Agent Substrate (ate-system, WorkerPool CRD); there
is one runner image and concurrency equals warm workers, one actor per worker.
Names the three host-bound components (reconciler, receiver, Joern engine) with
their planned in-cluster replacements, the production shape of the pre-push
hook (local deterministic check in the deadline, delta Run on the plane
asynchronously), the estimated per-push cost from the two measured
per-candidate rates, what is not built (no `ousast plane scan`, no diff
candidates task, population-only Workspace generation, no in-cluster
reconciler), and a sizing table from plane/tasks/*.yaml. plane.md gets a
two-line pointer to the section.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Legacy inventory 2, item R2. tests/test_harnessx_removal_equality.py loaded
benchmarks/archive/harnessx_removal_equality.py by path and checked that its comparison sees a
difference. The spec is implementation-complete and the proof is the committed pair of records
(2026-09-30-harnessx-removal-{baseline,equality}); the archive README says these scripts are
reference only and a record is re-run at the commit it names, not with today's package. A unit test
that keeps the archived copy importable contradicted that and would break the next time cli or
learn.cli move. The archive README row now says the test was dropped. test_removed_plane_references.py
(the Req 2.4 name search) is a different test and stays.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer: the whole system moves to a Kubernetes cluster soon. Seven host-bound
assumptions become cluster resources or configuration; the engine becomes a
plane task on a larger pool; a delta Run command serves the pre-push use; every
step is proven on kind first.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Legacy inventory 2, item R1. The vector half of index.py (VectorIndex, VectorRecord, RetrievalHit,
RetrievalPackage, the EmbeddingClient protocol, index_reuse_key, build/write/read/query_vector_index,
build_retrieval_package and their helpers) had no importer in src; `ousast index` wrote index/chunks.json
that nothing read, and was the only reader of `[embeddings]`. index.py keeps CodeChunk and
chunk_text_namespace, which skills.py imports; chunk_file/build_code_chunks go with their only caller.
`[embeddings] model` and `[embeddings] store` join RETIRED_KEYS (RetiredConfigError naming the reason),
the same way the [variants] keys were retired; OPENROUTER_EMBEDDING_MODEL is no longer read, so the
README and .env.example drop it (`ousast learn memory embed --model` names the model). test_index.py
now tests the chunker only; test_config.py covers the two retired keys and no longer asserts the
removed field. The module audit record is regenerated (counts unchanged: 151 modules, 0 orphaned;
the index module's importer list shrinks to skills). RELEASE_NOTES Unreleased carries the line.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
One page, docs/where-we-stand.md, composed with the COBESY composition method (governing idea,
SCQA opener, MECE sections): the M4 gates and the state against them, the detection approach step
by step, ax on the server, token economics before and after, and the open items. Every figure is
quoted from a committed record whose path stands next to it.
decision-engine.md, evaluation.md and token-ergonomics.md link to the page where they repeated its
content (detail tables kept); decision-engine.md's Experiments section now records exp-002's
result instead of saying it has not run. Linked from the nav, the index and the README.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer decision 2026-10-02 (no symlink). AGENTS.md, README, the two spec
docs, the kiro-spec-batch skill and the provenance ignore list are repointed;
the OpenCode plugin package.json stays under .opencode.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reject B. 160 paired units (injection 119, access_control 41; 56 repositories),
both arms, two replicates each, no early stop. Canary agreement A 0.875 -> B
0.906 (CI [-0.03, +0.10], spans; B meets 0.90). Within-pair AUC A 0.815 -> B
0.741 (CI [-0.13, -0.02], against B). AUC -0.02 (against), ADVISORY recall
-0.06 and precision +0.01 (span), usd per candidate A $0.0021 -> B $0.0051
(against). Priced usage A $0.68 / B $1.63; balance $16.72 -> $16.04 at the
run's end (lags). Counts only; task 7.4 marked done.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
An experiment's register, run, result and arm_outcome rows share one object,
and RustFS's S3 Select infers one schema per object: exp-002's result row
carried `units` as an object beside the run row's integer (and `looks` as
objects beside the register row's fractions), and after `analyse --record` no
row of the experiment could be read (JSONParsingError). The result row now
writes `unit_counts` and `looks_taken`; a test asserts one JSON shape per
column across the four row kinds. The live object was repaired by dropping
the one row (drop_rows) and recording the analysis again.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The experiment row in s3://sast-memory now carries units_digest 0de66bfa... at
pin 9213e5f (manifest digest unchanged, f5004fb8...). Counts per family and the
two repairs the freeze needed are in the record; no model.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
160 units (injection 119 in 34 repositories, 76 paired as 38 candidate pairs;
access_control 41 in 24 repositories, 32 paired as 16 pairs), 56 repositories
in all, under the incumbent folds' seed on the 937-example memory (slice 2
evaluated 110 + 37 on the 904-example memory; the memory grew by the recovered
negatives on 2026-10-02). No model. sha256 0de66bfa3ff844825e06f4f4227a14210d1b77f1cc08c0441c6d23e23fa5d91f.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Static-profile feature records carry the nine plane-only features (agree.*,
ms.*, verify.*) as null in x; after cb85542 the 937 live example rows loaded,
but the renderer's validate_x refused every one of them ("features outside the
static allow-list"), so no program could prepare a prompt (found while probing
exp-002 model-free). profile_x drops a null feature outside the row's profile
and keeps a valued one, so a real leak still fails. Test added.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Static-profile feature records carry the plane-only instruments as null; every
one of the 937 live example rows was unreadable (TypeError), which blocked
ousast learn evaluate and freezing exp-002 units.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Labels regenerated (identical sha: the builder's fixed-side rule already
accepted these pairs), feature records built for the 9 corpus pairs the
matcher now resolves (36 rows; the store's ingest index had remembered them
from a build before the declares() gate and skipped them as a repeat after a
later build dropped them -- build_features now checks presence after an index
hit and restores the rows), static memory rebuilt on RustFS: 904 -> 937
examples (+16 fixed-side negatives, +17 positives: 7 advisory Ruby pairs, 9
corpus pairs, 1 population positive), 9 wrong excerpts corrected (Ruby blocks,
a TS default-value brace), no embedding call (33 new + 9 changed excerpts
without a vector, ~$0.0003 to embed). Positives-only groups 98 -> 99: all
data gaps (94 vibe-py same-snapshot fixed sides, 3 dev-php, 2 population
fixes that did not change the function). Leak audit: pairs 376 -> 385; static
no new flag; plane two new injection flags (agree.final, verify.votes at
0.2292 over 48 pairs), v1 profile unchanged, reported.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The brace matcher now finds a declaration head in more forms -- name(...) with
any return type, qualifiers, generics or a parameter list spread over many
lines (a paren-aware scan replaces the 8-line lookahead; a prototype's ';' is
not a body, a ')' closing an outer parenthesis marks a call), JS object-literal
methods and property/chained assignments of function expressions and arrows,
$ as an identifier character, Perl 'sub name {', C# expression bodies -- and a
file given as 'other'/'unknown' is sniffed for Perl and Ruby so their def/sub
forms are matched by the right pattern instead of a {} block inside them. The
memory build hands the matcher the label's language where the record's closed
vocabulary language ('other' for Go, Ruby, C#) resolves nothing. One test per
form. Fixed-side negatives still require the fixed side to declare the
function; the 94 vibe-py positives-only groups stay counted.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Registration rows at pin fbc3220 in s3://sast-memory; in-repo records under
benchmarks/experiments/<id>/registration.json. exp-002's units could not be
frozen: every example row in the store carries null plane-only instruments
and learn.examples.load_examples raises on them (unchanged code since
a90e570) -- a store-side change to diagnose before any paid run.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/experiments.py: YAML manifests under plane/experiments/<id>.yaml,
digest-checked at register, run and analyse time; `experiment`,
`arm_outcome` and `experiment_result` rows; paired arms in a seeded
repository order with a per-unit coin; arm A replay-only at $0; canary
replicates under a fresh cache salt; repository-cluster paired bootstrap,
exact McNemar, two O'Brien-Fleming looks, adopt/reject/equivalent/
inconclusive. `ousast learn experiment register|units|run|analyse`.
Program: `ProgramSpec.sampling = retrieval_ensemble` (k samples at
temperature 0 over k disjoint retrieved example sets; default unchanged);
`Caller.salt`; `retrieve(exclude=)`. verify: OUSAST_VERIFY_CALLERS=off.
Manifests exp-001-known-callers (plane arms, units pending) and
exp-002-retrieval-ensemble (slice-2 incumbents, ceiling $4) -- not run.
18 scripted tests; module audit regenerated (151 modules).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Every README and docs page checked against ousast --help, the code and the committed records. New docs/deployment.md (in the nav): what runs where today, this host's setup (up.sh, doctor, smoke Run, footprint), step-by-step guidance for a separate Kubernetes cluster with implemented vs planned stated per item (no cluster deployment has been made; kube context and doctor's registry check are hard-coded in doctor.py, egress.py, router.py), and what the plane/ manifests need changed. Decision-engine pages state slice 2 (six families, pooled and within-pair AUC, BLOCK offered nowhere, canary below 0.9 for three families, positives-only inflation) and that the engine is not adopted; exp-002 registered, not run, no result quoted. Fixed: model/judge.py reference (now model/pipeline.py), missing-key behaviour, .env reference (OPENROUTER_EMBEDDING_MODEL, store variables), docs extra, plane memory-normalise and ousast index listed, ax README's cluster table made precise. RELEASE_NOTES gets an Unreleased entry.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Counts-only record of every repository group of the static memory that holds
positives and no negatives (98), traced label -> feature record -> excerpt:
94 are vibe-py pairs whose fixed side is another function of the same
snapshot (data gap, counted), 3 dev-php recipes without a fixed pin, 1
population-v2 fix that did not change the function. Separately, 16 pairs are
absent from the memory on both sides for tooling reasons: 7 advisory Ruby
pairs matched under the feature vocabulary's 'other' language, 9 corpus pairs
whose declaration form the excerpt matcher misses (JS object-literal methods,
property-assignment arrows, a $ identifier, a C parameter list past the
8-line lookahead, Perl sub).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Decision-engine task 6.7 (partial). After the advisory-fix harvest (memory 724 -> 904 static examples),
one compile per evaluable family and an outer grouped 5-fold evaluation of that family's own candidates,
input profile v1, priors off, k = 5, cross-fitted Platt, nested operating points (repository-cluster
bootstrap 95%):
family n AUC(s) ADVISORY R / P 'vulnerable' P / R ECE canary
injection 110 0.77 [0.67, 0.88] 0.88 / 0.57 0.84 / 0.36 0.093 0.77
untrusted_dest. 62 0.85 [0.76, 0.94] 0.84 / 0.73 0.96 / 0.66 0.071 0.87
path 35 0.89 [0.79, 0.97] 0.86 / 0.69 0.89 / 0.76 0.119 0.90
output_encoding 44 0.91 [0.82, 0.97] 0.86 / 0.81 0.94 / 0.55 0.045 0.90
deserialization 46 0.90 [0.82, 0.97] 0.83 / 0.76 0.93 / 0.83 0.095 0.93
access_control 37 0.96 [0.90, 1.00] 0.91 / 0.91 1.00 / 0.61 0.076 0.83
BLOCK offered for none: calibration fails everywhere except output_encoding, where no fold reaches a
Wilson lower bound of 0.95. Within paired groups (a function and its own fix) AUC is lower, 0.72-0.93:
every single-label group in the new families holds positives only. Injection learning-curve point,
paired (same program, candidates, folds): memory 724 -> 904, AUC 0.76 -> 0.77 (no injection example
was added). $2.88 metered DeepSeek (ceiling $8; balance 17.73 -> 16.73), $0.0009 embeddings.
Instrument fix: compiled demonstrations now follow the near-duplicate rule (cos >= 0.98) that retrieval
and the pre-call assertion apply; the assertion had stopped the untrusted_destination evaluation.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
One page, two halves with a Mermaid diagram each. A: the plane's token economy (one Model per Task with a metered usd/calls budget, attribution table, credentials only in the start request, repo-facts and content-addressed reuse, replay at $0, triage-hunt-tie-break order, prefix-cache prompt order) with every cost quoted from plane-increment-1/2.json, the injection-slice and harvest records, and an honest not-yet list (no token budget in ax, triage applied not run, k and contrast fixed outside experiments). B: the development-side measurement (tool I/O 85.5%, previously 90%) and the benchmarks/dev remedies: read_guard hook, bulk_read portal, token_report, plus the practices. Linked from the nav, index, memory-and-detection and the README docs list.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The plane's memory store no longer names one vendor. `MinioStore`/`SdkClient`/
`minio_settings`/`MINIO_KEYS` become `S3Store`/`S3Client`/`s3_settings`/`S3_KEYS`;
the URL scheme is `s3://<bucket>[/<prefix>]` (`s3://` alone uses `S3_BUCKET`); the
settings are `S3_ENDPOINT` (a full URL), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`,
optional `AWS_SESSION_TOKEN` and `S3_REGION`. The client is boto3 (extra `s3`,
imported lazily): SigV4, path-style addressing, SDK checksum trailers off, paginated
listing, versions and delete markers listed and deleted by version (`purge`, used by
the contract tests' teardown so the bucket keeps nothing of a run).
Behaviour is unchanged: the store never configures its bucket and verifies
versioning, the `runs/` expiry rule, S3 Select and tag reads at startup; no local
fallback; every row carries every queryable field with "" placeholders. The admin
hints read `--endpoint-url "$S3_ENDPOINT"`. The fake client still reproduces RustFS's
two Select failures (unbound leading-row field, null-typed column).
Docs: rustfs.md stays the tested server's setup page (policy JSON unchanged, env
names updated, any S3-compatible store with versioning, lifecycle, tags and Select
works); memory.md, ops/ax/README.md, README.md, steering and the two current spec
designs use the S3 naming. Retired specs and committed measurements are untouched.
Real server (RustFS, OUSAST_MEMORY_TEST_S3=1): test_plane_memory + test_learn_compile
75 passed, 1 skipped. Store copy of the live local store to a temporary prefix: every
count matched (776 row objects, 2982 rows, 22 facts, 788 excerpts, 2601 responses,
2 programs, 1793 index entries), 4190 versions purged afterwards.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The 97 advisory-fix pairs (194 sides) harvested for the decision engine: verify a/b + agree on the 34
untrusted_destination sides and model roles on every side as plane Runs on ax ($0.38 metered, task
budgets $1.49, ceiling $5; DeepSeek balance 17.88 -> 17.73), engine features on the host (frozen
7b17eac, one container, --memory 3g: 54 ran, 58 asked no question, 82 no language model), feature
records for all 194 sides, static memory rebuilt with the merged harvest units: 724 -> 904 examples.
No unit was skipped for the 20 KB inline limit.
Instrument fix: SideReader read only the first pair-kind source, so the first rebuild left all 194
advisory sides `unread` (0 examples). It now reads every pair-kind source; test added.
Leak audit: the static profile flags only engine signals and facts.function_lines (withheld by v1);
the plane profile newly flags verify.flag_b and ms.operation (plane-only, not in any v1 prompt).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Read-only check at the pinned vulnerable commits via the GitHub API
(LICENSE, license.txt, plugin header License: line). The five license
fields now record the SPDX id with the evidence; nothing else in the
frozen population changes. protocol-v2.md notes the verification and
that the OpenCVE case in v1 is BSL-1.1 (not OSI; non-production use
permitted by the Additional Use Grant; change date 2030-08-14 to
Apache-2.0).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Comment-only changes outside learn/, plane/ and cli.py:
- model/pipeline.py: the `judge` its docstring contrasts with is the deleted
model/judge.py, now named as such.
- fusion.py: the panels are deterministic and only review and vote; the
"critique and revise" step belonged to the retired LLM panels.
- mcp.py: hunter tools stay inside the scan's stages, not "harness stages".
- fusion, mcp, gate, redaction, skills: their "Phase N" tags now say which
plan they cite (the retired openrouter-sast-harness).
Requirement citations to retired specs stay: they explain where the code
came from. No behaviour change; the byte-identical benchmark golden passes.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
- Root .dockerignore for the root Dockerfile's context: local state (.git,
.claude worktrees, .venv, .env, caches, reports, *.sarif) and the
benchmarks/measurements archive (~20 MB of evidence). Kept: everything the
Dockerfile COPYs, the two 2026-09-13 records ops/smoke_report.py reads
from /app inside the image, and the harnessx-removal baseline pairs.json
that learn/sources.toml names as a label source. plane/Dockerfile.runner
keeps its own ignore file.
- Remove 2026-09-30-harnessx-removal-baseline/scan/report.sarif: tracked
despite .gitignore's *.sarif, with no counterpart on the candidate side.
The directory's README says why.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A4: roadmap-2026-09-04.md moves to .kiro/archive/steering/ so it no longer
loads as steering (roadmap.md and safety-net.md replaced it); roadmap.md's
pointer follows it. overview.md's spec table is recounted from tasks.md and
spec.json (adds ai-service-plane, harnessx-removal, learned-decision-engine,
pre-push-safety-net; drops the stale "active one"), and the module line now
cites the regenerated audit: 150 modules, 145 load-bearing, 5 standalone,
0 orphaned; 1,964 tests collected.
A5: a dated status note at the top of the requirements (or brief) of 16
specs: retired, superseded, dormant or complete, and for
model-grounded-detection the removal of judge/calibrate/execution (closing
the Req 10 execution-tier option). spec.json phase corrected to
implementation-complete where every task is checked (model-grounded-
detection, propose-adjudicate-prove, tree-sitter-overlay-extra,
corpus-seeded-mechanisms, pair-corpus-honesty); openrouter-sast-harness gets
a spec.json with status retired. Nothing deleted, no spec moved.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Material for MkDocs site (mkdocs.yml, optional extra `docs`) with Mermaid via
the superfences custom fence. New pages: overview, scanning (modes, quick
rules, Joern engine, priors, pre-push delta), the plane on ax (one task as a
sequence, golden snapshot, Run DAGs), memory (row kinds, write/read paths,
FileStore vs S3, verify at startup), memory and detection, and RustFS setup
(admin steps, complete agent policy, .env, refusal messages, Select limit).
The ops READMEs are included at build time; evaluation gains the population
lifecycle and the fold layout. tests/test_docs_site.py runs the strict build
when mkdocs is importable and checks Mermaid blocks and the RustFS page.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A1-A3 of the legacy inventory, moved with git mv to benchmarks/archive/:
the stage-2 verifier chain (verify_batched, score_batched, verify_sinks,
model_sinks, sink_candidates), the v2 tool hunter (hunt.py) and the
harnessx-removal equality proof. The plane's verify/agree/roles tasks
replaced the first six; the harnessx-removal spec is closed.
- The six independent scripts keep HERE = benchmarks/independent (population,
evaluate.py, results) and add their own directory for archived siblings.
- harnessx_removal_equality's REPO root is one level deeper.
- tests/test_plane_scoring_parity.py loads score_batched from the archive
(ran here, not skipped); test_harnessx_removal_equality points at the new path.
- benchmarks/archive/README.md: what each script was, which committed
records cite it, the commit that last ran it. paths.md maps old -> new;
committed records are not rewritten.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
RustFS's S3 Select infers an object's schema from its first 1000 rows: a where
field absent there is unbound (EvaluatorBindingDoesNotExist), and a column null
in all of them but set later fails the whole object (JSONParsingError, measured:
999 leading nulls answer, 1000 fail). plane/memory.py now declares per kind the
fields a where may name (QUERY_FIELDS: facts candidates_digest, verdict
candidate/final, example profile, plus the ROW_FIELDS every row has) and writes
their union on every row at write time as strings, "" where one does not apply
(null measured to break). rows() refuses an unknown kind, an undeclared where
field and where = null before any read. `ousast plane memory-normalise
[--store URL] [--dry-run]` rewrites pre-rule rows once, idempotently, with
counts. The fake object client mimics both RustFS failures.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Nothing in src reads config.dynamic, config.evidence or config.variants, and
[models] patcher was only echoed into harness.json. They now go through the
existing RETIRED_KEYS mechanism and fail with RetiredConfigError naming why
("retired 2026-10-02 because nothing ever read it: ..."); RETIRED_KEYS now
carries the full reason per key so the HarnessX entries keep their wording.
- DynamicConfig, EvidenceConfig, VariantsConfig and their loaders removed;
ModelConfig.patcher removed. resolved_config.json loses those sections.
- harness.json keeps its shape: model_roles.patcher is always null.
- RELEASE_NOTES "Unreleased" lists the keys and the output shape change.
- Delete .opencode/skills/openultrasast-kiro-impl: it pointed agents at the
retired openrouter-sast-harness tasks.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer decisions 2026-10-02: "it has the featurs. fallback not desired" and
"Admin sets it once; agent verifies".
- MinioStore never configures its bucket: configure()/enable_versioning()/
expire() are gone; the client protocol only reads versioning and lifecycle.
- verify_bucket() runs when a MinioStore is opened (once per process per
store): versioning Enabled, an enabled lifecycle rule expiring <prefix>/runs/
(none expiring the whole store), S3 Select answering a probe, object tags
readable. Every missing piece is listed in one MemoryStoreError with the
aws-cli admin command or the permission to grant.
- No S3 Select fallback: a server without Select is refused at open, and a
failing Select on one object raises (naming RustFS's leading-row schema
inference when the field is unbound) instead of filtering locally.
- MINIO_REGION is passed through when set.
- Tests: fakes for versioning off/unreadable, lifecycle missing/disabled/
tag-narrowed/whole-store, Select missing, tags unreadable, and the pass case;
the real-server contract uses OUSAST_MEMORY_TEST_BUCKET (else MINIO_BUCKET)
and never writes bucket configuration.
- ops/ax/README.md: RustFS support, the admin one-time setup, the agent policy.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The reachability-first audit (previous commit) found these five with no
import path from any entry point:
- model/judge.py: replaced by model/pipeline.py, which never imported it.
- model/execution.py: the MGD Req 10 execution seam, never built or called.
- model/calibrate.py: no CLI, no importer; learn/calibrate.py is the live one.
- stage_processors.py + slot_contract.py: kept only so test_gate could pin
them; the scan runs findings.quick_scan_findings directly.
Their tests go with them, test_gate's zero-dependency list drops the two
entries, the audit lists drop their rows, and the manifest is regenerated:
150 modules, 145 load_bearing, 5 standalone, 0 orphaned.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
model/audit.py classified anything named in SPEC_OWNED or STANDALONE before
looking at the import graph, so five modules nothing imports were reported
load_bearing and the committed manifest said 0 orphans.
- Reachability from the gates, the CLI and every module with a __main__ block
(python -m, the plane runner) now decides the class; the lists only supply
reason text, and an unreachable listed module is orphaned with its listing.
- `from .pkg import sub` now records the submodule edge (plane.engine_alerts
was only reachable by its SPEC_OWNED entry).
- `python -m openultrasast.model.audit` regenerates the manifest.
- Manifest regenerated: 145 load_bearing, 5 standalone, 5 orphaned
(model.calibrate, model.execution, model.judge, slot_contract,
stage_processors). The zero-orphan test is strict-xfail until they go.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
6.2: the 724 static examples' 616 distinct excerpts embedded with openai/text-embedding-3-small via
OpenRouter, 198,819 tokens, $0.0040 metered (ceiling $1).
6.6, family injection only, memory = every family (at most 2 contrast examples per prompt), input
profile v1: smoke fixed k = 5 (repeat prefix-cache hit share 0.96); one compile (program c1237a49,
proposed instruction 5, balanced Brier 0.128 on 13 C_val candidates vs 0.199 baseline); outer grouped
5-fold, 110 out-of-repository candidates in 29 groups: ADVISORY recall 0.90 [0.82, 0.97], precision
0.54 [0.44, 0.65], AUC 0.74; BLOCK not offered (ECE 0.064 > 0.05; no fold reached a Wilson lower bound
of 0.95); canary re-ask agreement 0.83. $0.91 metered at list price; DeepSeek balance $18.20 -> $17.90.
Instrument fixes found on the way, each with a test:
- memory build: pair sides join the record stored under their harvest unit pin (--units), and only the
built profile's records are read (static and plane shared a key; 85 -> 724 examples);
- retrieval: stage 1 per side (family / contrast) and at most 2 contrast examples, never padded;
- program: the JSON answer contract is part of the signature, appended to proposed instructions
(DeepSeek JSON mode refused a prompt without "json");
- compile: candidate_families for a one-family compile; compile.toml k = 5 from the smoke rule.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The leak audit found facts.function_lines separating fixed from vulnerable pair
sides (a fix guard lengthens the function: 0.80 untrusted_destination, 0.75
config_secrets). Input profiles may now name single features; v1 withholds the
engine and the function length from prompts, retrieval and the proposer summary.
A withheld feature keeps its instrument block in the distance (state still counts).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer decision "Neutralise both for v1".
1. Phantom candidates. A pair side's candidates are now only the functions that
side declares (plane/harvest.py `declares`, the excerpt builder's
function_bounds); a labelled function absent from a side is listed under
`absent_on_side` and never asked, and a side declaring none of its verify
family's functions is no verify unit. The recorded harvest units held 58
phantom verify candidates (53 fixed, 5 vulnerable sides). build_features.py
skips label rows whose side does not declare the function and drops their
earlier rows (MemoryStore.drop_rows); rebuilt the pair records from the
recorded signals, no model call: 126 label rows absent_on_side (117 fixed,
9 vulnerable) -- access_control 13, config_secrets 4, deserialization 20,
injection 28, memory 4, output_encoding 16, path 15, untrusted_destination
26 -- so 252 feature rows dropped (1712 -> 1460); no other row changed.
2. Engine withheld. Input profiles in learn/schema.py (`v1` withholds the
engine instrument, `full` withholds nothing); ProgramSpec.inputs defaults to
v1 and compile.toml records it. Neither the engine's state nor its values
reach the candidate, examples, demonstrations or the instruction proposer's
error summary, and the Gower retrieval distance leaves the engine block out.
Stored records are unchanged. design.md Risks records the decision and that
the engine returns only through a pre-registered A/B on assumed-benign and
benign-control negatives.
3. `ousast learn audit-leaks` (learn/leaks.py): per feature and per instrument
state, separation = (pairs the signal alone puts on the right side - on the
wrong side) / pairs, over all pairs and per family; flagged at >= 0.20 over
>= 10 pairs. Counts under
benchmarks/measurements/2026-10-01-decision-engine-leak-audit/ (before and
after the rebuild). After: facts.function_lines (0.80 untrusted_destination,
0.75 config_secrets, 0.26 injection; higher on fixed), verify.turns_mean
0.67 config_secrets, eng.* 0.35-0.40 output_encoding, verify.flag_a/votes/
site_offset and agree.final 0.20-0.30 on 10-12 pair families. Before the
rebuild the phantom fixed sides drove state.source/state.roles/ms.flagged
to 1.0 and state.engine to 0.95 on deserialization.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The pair-corpus engine job finished: 682 of 686 sides (199 ran, 475 read their input
and asked no question, 8 no engine model; the 4 sides of the two family-unknown
pairs were not run). Feature records rebuilt (1712); counts-only record updated.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
`ousast plane harvest` (plane/harvest.py) writes two Runs over a label snapshot:
verify a/b + agree on the verify-family units (both sides of each labelled pair as
an opaque files-only Workspace, candidates outside the hunt's root, no declared
sites; v1 and dev-php pins with repo-facts first) and model roles on the candidate
files (pair excerpts grouped, one Task per population pin). Budgets sum under the
ceiling by construction. New roles task template; the generator re-pins it.
Spent by attribution: verify $2.54 (1961 calls, 150 units, 182 hunts per pass),
roles $1.40 (1034 calls, 799 files, 0 unclassified chunks); DeepSeek balance
19.63 -> 18.23. Engine on the pair corpus (benchmarks/learn/engine_pairs.py,
frozen e6ed785, one 3g container) is PARTIAL: 384 of 686 sides at commit, still
running; unfinished sides stay missing. benchmarks/learn/build_features.py built
1712 static+plane records into the memory store; counts-only record under
benchmarks/measurements/2026-10-01-decision-engine-harvest/.
source/roles parts are `none`, not `failed`, for a language without a declaration
pattern (C). ops/ax/README: Substrate golden-actor disk leak and the inline
Workspace limit (~20 KB).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
97 advisory-fix pairs, one repository each (309 advisories examined, 212
rejected with reasons in research-2026-10-01.md): a public fix commit
distinct from its parent, the advisory-named function declared on both
sides with a changed body. Pointer pairs: materialize.py reads one file
per side from a shallow blobless fetch into the pair cache and verifies
them with the label builder's matcher (verification.json, bytes read).
learn/sources.toml lists the catalog as the advisory-fixes source. Label
snapshot: negative repository groups untrusted_destination 8->25, path
3->21, output_encoding 1->22, deserialization 0->21, access_control 0->20;
all five reach the per-family floor. Five pairs on reserved repositories
were dropped after the reserved-population guard flagged them.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A pair now gives a fixed-side negative only when the fixed side's source declares the
labelled function (the excerpt builder's matcher; a name on a code line where the matcher
resolves neither side), or when a deterministic rename at a distinct fix commit replaces it
(one new function, same parameter list, changed body: provenance fixed_side_moved). Anything
else is counted under pair_fixed_side.absent_fixed_side, not labelled.
108 pair labels lose their negative: 80 seeded pointer pairs whose fixed side is a guard
helper of the same commit, and 28 Real-Vuln-Benchmark traps whose fixed side is a sibling
handler of the same snapshot (4 of those handlers are labelled positives themselves). The
memory build no longer substitutes the catalog's fix_function on the fixed side. The pair
cache is an allowed label-source tree so pointer sides are read through the guard.
Snapshot: benchmarks/measurements/2026-10-01-decision-engine-labels (counts only).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/compile.toml: the pre-registered spec -- balanced Brier score of the
raw score on C_val (cells by family x label, repositories capped at 5,
unsure = 0.5, ties within 0.005 to fewer prompt tokens), seed, the
compile split (25%, boot 60/40, compile-split-only memory), bootstrap
(150 candidates, confidence >= 0.7, 4 sets of 3), search (6 proposals +
baseline, 20 errors shown, 60 validation candidates), k = 1, the
retrieval settings, and the $3 / 1,200-call ceiling.
learn/compile.py: bootstrap few-shot keeps only right, confident
rationales that cite a line and name no repository, path or CVE/GHSA id;
seeded demo sets (positive, negative, any; families mixed) with
other-framework alternates; one proposal call; 7 instructions scored with
set 1, the best with sets 2-4. The artifact is canonical JSON stored as
programs/<program_id>.json, program_id = sha256 of the artifact without
its id; it records the spec sha, memory/compile-groups/folds digests and
the response-cache digest, with spend priced from cached usage, so a
rerun from the cache is byte-identical at $0 (tested on FileStore and
MinioStore through the fake client; real MinIO gated as today). A ceiling
ends the compile `unfinished` with no artifact. packageable() keeps only
permissively licensed demonstrations (MIT, BSD, Apache-2.0, ISC, zlib),
swapping in a permissive alternate or dropping, under a new program id.
learn/calibrate.py: Platt scaling on logit(s), pure Python Newton, per
family / pooled with family intercepts / insufficient data by the floors;
cross-fitted (fold j's map is fitted on the other folds only); the check
(5-bin reliability, slope and intercept with repository-bootstrap CIs,
ECE <= 0.05), ADVISORY only when it fails.
learn/evaluate.py: predictions under each fold's boundary, families below
the floor never asked; operating points read from the calibrated,
cross-fitted predictions of the other folds (BLOCK: Wilson lower bound
>= 0.95, never on a majority unsure; ADVISORY: recall >= 0.90);
repository-cluster bootstrap intervals; strata with "insufficient data";
>5% parse failures is an instrument failure; explanations keep only
signals the record holds; version mismatch names the fallback reason;
the memory-size learning curve.
`ousast learn compile|evaluate|curve`: paid runs need --budget-usd (exit
3 at the ceiling), --replay-only runs from the cache. Nothing was spent:
every test and the dry run used scripted clients.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/program.py: the declared signature (code, signals, roles, family in;
verdict, family, confidence, rationale with cited lines out), the
baseline instruction, rendering as [system: instruction + demonstrations]
-- byte-identical for every candidate, so the provider's prefix cache can
hit -- then [user: retrieved examples with their labels, the candidate
without one]. Signals pass validate_x first (no EXCLUDED_FIELDS, missing
instruments in words). Answers are one JSON object; one that does not
parse or cites no line inside the excerpt is asked once more at
temperature 0 and otherwise becomes unsure/parse_failed, never
not_vulnerable. Classify(k): sample 1 at 0.0, the rest at 0.7; score s =
mean p_j; majority verdict, ties unsure; a majority unsure cannot BLOCK.
Caller caches each response as responses/<sha>.json keyed by (model,
Model parameters digest, messages, temperature, sample, json_object) and
replays at $0; spend is priced from the recorded usage once per distinct
request, so a replay reports the same usd. Program.decide asserts the
evaluation boundary on the rendered ids before any call.
MeteredClient.complete passes temperature/logprobs through (only when
set); DeepSeekChatClient sends temperature beside thinking=disabled.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/retrieve.py: stage 1 is a per-instrument Gower distance over the
profile's allow-listed features with explicit missing (both ran: mean
feature distance; differing states 1; same non-ran state 0; null in one 1;
priors dropped with priors=off), n1 = 40 with seeded tie-breaks; stage 2
re-ranks by r = lam(1 - D) + (1 - lam)cos, signals_only without the
target's vector; the balance rule takes k_ret = 6 (<= k/2 per label, <= 2
per group, <= 2 other families; `short` recorded). eligible() is the only
way into a prompt: not the target's group, not an evaluated group, not the
held-out source or framework, and outside deployment not a near-duplicate
(cos >= 0.98). assert_boundary repeats the check on rendered ids;
demonstrations of a held-out framework swap to their alternates;
deployment_fold excludes a scanned repository that is a corpus group;
seed_memory never hands a case its own group.
learn/folds.py: compile split (25% of groups per primary family, 60/40
boot/val), outer grouped 5-fold stratified by family and label mix,
leave-one-source-out, leave-one-framework-out (insufficient data below 10
groups), assert_disjoint and a folds digest; all seeded.
Tests: test_learn_retrieve.py (hand-computed Gower, lam 0/0.5, balance,
near-duplicates, test_boundary over outer/LOSO/LOFO folds with
demonstrations), test_learn_folds.py.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
plane/models/openrouter-embedding.yaml: openai/text-embedding-3-small via
OpenRouter (embeddings only), input_per_m 0.02, cache/output 0, egress
openrouter.ai; `openrouter` joins the provider extensions.
plane/budget.py MeteredEmbeddingClient: prompt_tokens priced from the
Model manifest, usd/calls ceilings stop the next call, a priced call that
reports no tokens raises UnmeteredUsage (a silent zero is an instrument
failure). OpenRouterEmbeddingClient keeps one usage row per call
(parse_embedding_payload); its api key is out of repr.
learn/embeddings.py + `ousast learn memory embed`: excerpts embedded in
batches into embeddings/<model-slug>/<excerpt_sha>.json (float32 base64),
cached by content hash so a rerun makes no call; the declared dimension is
checked; --dry-run estimates tokens and cost without a call; a paid run
needs --budget-usd and exits 3 (unfinished) at its ceiling. Not run against
the provider: the store embedding waits for its budget go-ahead.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/excerpt.py: the labelled function at its pin as numbered lines, bounded
(80 lines / 4,000 chars for candidates, 40 / 2,000 for memory examples),
centred on focus lines; advisory ids anywhere and comments with security
wording (labels.security_reason, multilingual) become <redacted>; owner,
owner/name, path and commit ids are replaced; a delta unit appends the
base->head diff (<= 40 lines). C/C++ and other brace languages without a
repo-facts declaration pattern use a brace matcher (346 c_cpp pair rows
had no excerpt without it).
learn/examples.py + `ousast learn memory build`: label + feature record +
excerpt -> `example` rows (no rationale), excerpts as content-addressed
blobs; refuses origin: user and sources outside sources.toml (exit 2);
conditional labels skipped; exit 2 when >10% of a source's recorded
candidates have no excerpt. A fixed side that was read but no longer
declares the labelled function is counted apart (absent_fixed_side): a
label defect, not an unread clone.
plane/memory.py: `example` in KINDS; put_blob/get_blob/blob_names on the
MemoryStore base, so FileStore and MinioStore share them (tested on both,
MinIO through the fake object client).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
In-house DSPy-style program (retrieve -> classify(k)) on the metered client;
example store with signal-profile Gower neighbours re-ranked by code embeddings;
balanced Brier metric, bootstrap few-shot and instruction search on a compile
split; Platt calibration on other folds; cached replay; tasks 6.1-6.7 with paid
steps flagged for go-ahead.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer: data will not scale; the decision engine is an AI classifier, not ML,
with local memory, DSPy style. In-house on the metered client; retrieval by signal
profile re-ranked by code embeddings; compiled (bootstrap few-shot, instruction
search) per training fold; retrieval never crosses the evaluation boundary;
confidence calibrated on held-out repositories. Design and tasks 6+ to revise.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/sources.toml is the fail-closed source list: every read goes through one guard
that refuses an unlisted path or an [[excluded]] file name before opening it, and a
source whose evaluation record is missing cannot label. Population v3 is named only
as excluded, by file name, and is never opened (tested with a v3-named fixture and
open/read_text tracked).
learn/labels.py and `ousast learn labels` apply the design's rules: v2 declared
sites and v1 fix-range functions positive, the same function on the fixed pin
negative only when the fix changed it, introduce/repair delta units, benign control
pushes as conditional negatives; pairs vulnerable/fixed (unscorable and title tier
out, title as a sensitivity arm); dev-php recipe known functions; recorded
adjudications with their reasons. v2 rows are labels marked design_informed, as the
design says. Repository groups: normalised owner/name, merged by shared CVE/GHSA,
shared fix commits and [[alias]] rows.
assumed_benign (maintainer decision): ordinary commits of the population
repositories, >= 90 days from any fix, no later security commit touching their
files, weight 0.5, conditional, reported separately and never merged with verified
negatives. The security filter is multilingual (CJK, Korean, Russian, European and
other languages) plus CVE/GHSA/CWE references and dependency bumps; widened after
reading accepted pushes (validation, passwords, guards).
First snapshot, counts only (benchmarks/measurements/2026-09-30-decision-engine-
labels/labels.json): verified pin labels in positive/negative repository groups
injection 44/41, untrusted_destination 35/34, deserialization 20/20 (per-family),
output_encoding 15/15, path 15/14, access_control 12/12 (pooled), config_secrets 5/5,
memory 5/5 (insufficient). assumed_benign: 969 pushes in 27 groups from 36,000
commits scanned.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Wrapper inference (learn/roles.py, no model): a function whose parameter reaches a
call already in the role set becomes a derived sink (or a sanitizer when it returns
the cleaned value) of the same operation, at depth + 1; a function returning a
source's value becomes a derived source. The role set starts from language-level
entries only (priors off by default, optional), grows to depth 3, and reads
vendored dependency source when present, so a vendored database class reaching
mysqli makes `->query` a sink without a framework table. A name the repository
declares more than once becomes a role only when every definition derives it.
Model roles (plane/tasks/roles.py): the model_sinks.py classification moved into
the package and extended to sink/source/sanitizer/guard/dispatch, per file chunk,
metered and resumable, client injected; a failed chunk is unclassified, never
"no roles". vocabulary_overlay merges inferred and model roles over the
language-level facts in the shape taint_specs takes, origin in the entry id.
Features: the roles (static) and model_sinks (plane) instruments are filled.
On the recorded validation-46 facts (15 cases, 14,368 product files, 87 MB read):
1,999 wrapper roles (1,753 sinks, 157 sources, 89 sanitizers; depth 1/2/3 =
973/651/375), 14 of 15 cases with at least one; the WordPress gopay plugin has
none with priors off, which is what model roles are for.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ruleset/frameworks.toml lists 19 framework and library ids (wordpress, flask, django, express,
spring, jakarta-servlet, react, vue and 11 libraries), each with its languages, dependency
package names and a symbol lexicon used only by the lint and for stratification.
Every framework/library entry of section 2 is tagged `framework`/`library`, and mixed entries
are split so a tag covers a whole entry (split pieces keep the entry's id and flags, in place):
semantic facts 38 tagged entries (wordpress 6, django 6, flask 4, express 4, jakarta-servlet 3,
pyyaml 2, one each for spring, sqlalchemy, requests, httpx, lodash, node-http-clients,
escape-html, sanitize-html, dompurify, esapi, commons-text, react, vue); the dispatch fact
behind the hook-as-source path is wordpress; the PHP layout row is renamed php_layout and stays
untagged (a product filter). Quick rules: 9 tagged (wordpress 6, flask, pyyaml,
node-http-clients); php-unserialize is split into php-unserialize and php-maybe-unserialize;
the lint also caught WordPress escapers in exclusions and axios/yaml.load in mixed rules, so
those keep their ids as tagged rules and gain a `fallback_for` language-level rule (6) that is
active only when the prior is off.
load_facts and load_ruleset take priors ("all" by default -- today's scan --, "off", or a set
of ids). tests/test_semantic_priors.py: the lint (no untagged entry contains a lexicon symbol
of its language), "off" removes exactly the tagged entries and activates the fallbacks, split
entries keep their language-level calls. Rule and layout counts in three existing tests follow the split and the rename.
Unchanged defaults: test_benchmark_output_is_byte_identical_to_golden_baseline passes, and the
quick findings of all 15 benchmark manifests (full StaticFinding records) are byte-identical
before and after. priors=off on the same manifests: 61 -> 57 findings; the 4 lost are js-ssrf
(axios), php-wpdb-sql-composition, python-ssti and python yaml.load; 5 PHP and 1 Python finding
move to their fallback rule at the same location. Semantic calls with priors off: php 75/92,
python 32/77, javascript 42/70, java 24/41, c 27/27 (the engine-side effect needs Joern and was
not measured).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/schema.py is the closed, typed allow-list of candidate features: 66 features over
11 instruments (language one-hot, quick rules with 23 mechanism buckets and a prior count,
engine, repo-facts, source span, entry-point distance, roles, delta, verify, agree, model
sinks), two profiles (static 57; plane adds verify, agree and model sinks for 66), schema_version and a feature-set digest.
No repository, path, file, function, rule or commit identity is a feature; site_match,
in_fix_range, fixed-pin alerts, case.json fields and costs are refused. Missing is not zero:
each record carries per-instrument state (ran | none | failed | not_applicable), and every
feature of an instrument that did not run is null.
learn/features.py builds records on the host (build_for_scan, whole repository or base..head
delta); plane/tasks/features.py builds them per case in a Run from the delivered artifacts,
dropping site_match and in_fix_range on read and ignoring fixed-pin alerts. The generator
adds a model-free `features` entry per case before `remember`, which stores the records as
`features` rows (re-validated). memory KINDS gains features, label, decision, experiment,
arm_outcome and experiment_result. The committed validation-46 manifests are regenerated
(entries appended; the first Run's entries are unchanged). The runner image must be rebuilt
and re-pinned before a Run executes the new task.
Proof on real artifacts (scratch store, not committed): the recorded validation-46 run gives
75 feature records over 15 cases (43 verify candidates plus vulnerable-pin alerts), each
case's vulnerable pin exported and read (0.5k-9k files); remember stored 75 features rows
beside 43 verdict, 84 alert, 61 unit_cost, 15 facts and 8 coverage rows.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Three instruments scored separately at one analyzer commit (quick PHP rules
with a fixed rule-to-family table, the Joern engine, the plane pipeline
triage -> verify a/b -> tie-break -> 2-of-3), protocol v2's six pins, one
matching rule for all six families (declared site or fix range, same family;
evaluate.matches_v2 unchanged), per-family meaning of a match, the M4 gates
copied from protocol v2 and applied per instrument, a $60 total / $6 per case
model-spend ceiling, and the retune rule.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
freeze.py, extended for v3 and still reading only git history, file lists and
fix diffs:
- a file the advisory names without a function resolves to the function(s)
the fix's diff changes at the vulnerable pin (maintainer's decision 2),
recorded with its basis (advisory-named / resolved-from-fix-diff); a new
helper inserted at file scope adds no site;
- an inferred fix must change a declared site function, else the case drops;
- a single-language population gets a strict benign proposal (PHP change,
merges judged with all they bring in, pre-vulnerable history as fallback).
All 15 cases verified, none dropped (pagible and podlove fixes confirmed).
Nine sites in eight cases resolved from diffs; five benign proposals
replaced by hand.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Assumed-benign pushes mined from non-security commits as a separate label source
(maintainer decision). Ten tasks in the design's commit order; the harvest,
exp-001 and the v3 check wait for budget go-ahead.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer: hard-coded application specifications are problematic; WordPress is
not the only PHP application, same for Python and JS. Language-level knowledge
stays; framework roles are inferred per repository; existing framework entries
become tagged optional priors kept only by A/B evidence; leave-one-framework-out
evaluation.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer's selection 2026-09-30, "Recommended 16, minus v2 mentions": the
research's rule-based 16 with Winter, elFinder and Easy!Appointments replaced
by the next candidate in their family by the same rule (named fix first, then
newest first), v2-mentioned alternates (Grav, Backpack CRUD, Koel) skipped.
Winter -> mantisbt-tokenizer-eval-rce; elFinder -> pagible (inferred fix, to
be confirmed at freeze); Easy!Appointments -> empty slot (no remaining
untrusted_destination candidate), so 15 cases are reserved.
The reservation guard now covers v3: all six families, a declared-language
population, status "reserved".
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer: the tool will never run against the test cases it is trained with;
its purpose is a repo safety net. Repository-grouped folds, repository-agnostic
features, BLOCK/ADVISORY operating points for changed code, out-of-repo
calibration, adaptation kept separate.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer direction: A/B testing to extrapolate detection instead of hard-coding
it, and a trained decision engine. Instruments become signal producers; labels
from ground truth with provenance; a simple calibrated engine with population
folds; experiments as plane Runs; extrapolation with intervals, checked once on
frozen v3; the engine is what scan/pre-push/plane report.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Quick mode had no PHP rules, so a PHP quick scan read the files and reported a plausible zero.
ruleset/php/rules.toml adds 13 line/statement rules built on semantic/php.toml's sinks, sources and
sanitizers (sql/wpdb, command, code_eval, unserialize, echo, file_inclusion, filesystem,
outbound_request, redirect). 9 are enabled and 4 are shadow (sql-call-composition, code-eval,
unserialize, xss-echo-input). A rule is shadow when it fired on the corpus but never at a known site.
precision_estimate is the measured lower bound.
Measured in-sample on the development corpus only (pmpro introduce/repair/benign, population-v1 PHP
cases; population-v2 was not read). Known site hit on the vulnerable pin in 4 of 5 cases (LearnPress
missed). The hit disappears on the fixed pin for pmpro and YesWiki. Numbers, per-pin files/bytes read
and the engine comparison are in benchmarks/measurements/2026-09-30-php-quick-rules/.
- fixtures php-vulnerable / php-benign plus manifests: ousast benchmark covers PHP
(9/14 reported, 13/14 including shadow; the benign twin records 2 known false alerts)
- PHP `#` comments are comments; `#[...]` attributes are not
- the alerts task now reports PHP as quick coverage; alerts-engine keeps running the engine for PHP
next to the quick rules (ENGINE_ALONGSIDE_QUICK)
- other languages' benchmark findings and metrics are unchanged
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Quick mode has rules only for c/cpp, groovy, java, javascript/typescript and
python, so every PHP case's "0 alerts" read as clean. The `alerts` task now
records per language of the files read whether quick mode covers it
(derived from the bundled ruleset, not a list): coverage quick | none, with
file counts per pin, and `uncovered`. `remember` turns that into `coverage`
memory rows (a new kind; the Run passes the alerts summary.json as input),
loop-measure reports coverage per language, and improve/memory.py's M1/M2
skip a repository+pin whose coverage for the rule's language is only
`none` (counted under `coverage_none` in the proposal's excluded).
`ousast plane alerts-engine <Run.yaml> --source <frozen export>`
(plane/engine_alerts.py) does the alerts task on the host for cases whose
files include a language the engine covers but quick mode does not (PHP):
exports both pins from the case cache, runs the quick scan and the
openultrasast:dev engine container (finding_dump.py, --network none,
--memory 3g, one at a time, per-pin deadline), writes engine findings in
the alerts.jsonl row shape (rule_id engine:<family>, source engine,
in_fix_range from case.json) with a summary (files read, questions,
completed, seconds, degradations per pin), and marks the task done via
reconciler.mark_done. No result, no files read, or questions == 0 fails
loudly and leaves the run untouched. Refuses a git checkout as source.
Runner image rebuilt and re-pinned (sha256:ebb40a9a...); validation-46
regenerated (digest only). Live, recorded in
benchmarks/measurements/2026-09-30-php-engine-alerts/: froxlor 366 files,
130/130 questions per pin, 6 alerts per pin, 0 in the fix range; wp-gopay
19 files, 101/101 questions, 0 alerts. The loop: 32 run rows (coverage 8,
php=engine), 0 proposals, gate no_proposals, $0.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
README: "Configuring the HarnessX agentic plane" becomes "The agentic
plane (ax)" (ousast plane run|status|doctor|remember, per-task Model,
budget and egress, pointer to ops/ax/README.md, one tagged retirement
line); "The HarnessX self-improving cycle" becomes "The self-improving
cycle", and its MetaAgent paragraph is replaced by the memory proposer
(`ousast improve --memory`, provenance, train-on-test guard). The fusion
panel_model/decider_model lines and the [harnessx] spend caps are gone.
The README reference test now passes without its strict xfail.
docs/examples.md section 6 runs the plane; docs/threat-model.md describes
the per-task EgressPolicy, the usd/calls budgets and the start-request
credential path. RELEASE_NOTES.md gets an Unreleased entry: HarnessX
removed, the retired keys and their errors, the silent loss of the
second hunter for users who set [models] hunter, and the plane as the
agentic path.
Dated retirement notes (2026-09-30) head the requirements, design and
brief of harnessx-self-improving-rulesets, learning-harness,
three-stage-scan and openrouter-sast-harness, and follow the spec table
in steering/overview.md; history is unchanged. All tasks are checked, so
spec.json moves to implementation-complete.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reran benchmarks/harnessx_removal_equality.py record from a git archive
export of b9afb66 (tasks 1-6 landed; same venv freeze hash as the
baseline, harnessx not importable, no API keys, empty config) and
compared it against 2026-09-30-harnessx-removal-baseline:
equal=True differences=0
All 13 output files (pairs.json, benchmark, standard scan, improve
--dry-run) are raw-byte identical to the baseline. The suite map
(suite-map.json) accounts for 29 deleted ids, 2 renamed ids and 83
added ids; each deleted or renamed id cites the design's Tests table and
commit cbadc38 or b9afb66. Every surviving test id keeps its outcome
(1740 passed, 38 skipped incl. the README xfail).
Instrument note: a first record failed test_push_hook twice because the
recording shell's PATH lacked .venv/bin (the hook execs `ousast`); the
same two tests fail at b9afb66 without it and pass with it. The
committed record has .venv/bin on PATH.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Deletes harness_ext.py and hunter_harness.py with their HarnessX-only
tests (the judge path, verify_judge.py, went in task 5). The harnessx
extra, its mypy override and its uv.lock entries are removed (uv lock
drops only packages; no remaining version changes). The conftest
fixture becomes assert_cold_import; test_gate's two never-import checks
become one allow-list (stdlib, openultrasast, yaml) over the governance
planes' module-level imports, and the skipped live-extra tolerance test
is deleted. Audit rows for the two modules are removed, stage_processors
is reworded, and the module-audit manifest is regenerated (no orphan).
test_removed_plane_references implements Req 2.4 over src, tests and
pyproject.toml: a match must carry a `retired 2026-..` tag (the spec's
own slug is not counted). The README part is a strict xfail until task 8
rewrites the README's plane sections.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The standard scan now runs run_hunter_pool, verify_findings and the
deterministic fuse_findings directly; the HarnessX hunter orchestrator,
the LLM judge dispatcher (verify_judge.py, deleted with its test) and the
LLM fusion panels are gone, and so are their extra-absent degradations.
host_under_harnessx is dropped from stage_processors.
config: [models] verifier and [fusion] panel_model/decider_model raise
RetiredConfigError naming the plane replacement (the CLI prints it and
exits 2); a [harnessx] section loads with one warning and is otherwise
ignored. harness.json keeps "verifier": null. Tests that asserted the
fallback are renamed to the behaviour they assert; the module-audit
manifest drops verify_judge. Nothing outside the two modules deleted in
task 6 imports harness_ext or hunter_harness.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
`ousast plane workspaces --validation-set ... --loop` appends the loop to a Run:
per case `alerts` (plane/tasks/alerts.py: the quick-mode scan on the vulnerable
and the fixed pin, a fixed-pin Workspace per case, alerts attributed to their
enclosing function by the repo-facts patterns, `in_fix_range` from the fix's
old-side and new-side ranges, failing on an unread tree) feeding `remember`,
then `memory-snapshot` -> `loop-measure` -> `loop-propose` -> `loop-improve`
(plane/tasks/loop.py). Every new task binds no Model and has budget
{usd: 0, calls: 0}. The snapshot is the store's rows after the train-on-test
guard, written on the host by memory.seed (a sandboxed run of it fails loudly);
propose runs M1/M2 against this repository at the commit under test; improve
runs one unchanged evolve.run_round and delivers gate.json, journal.json, the
sidecar, rule_policy.json only when accepted, and proposal_outcome rows that
the host ingests. Adoption stays a maintainer commit.
`remember` is now a Run entry per case (the image contains it), with the facts
row keyed by the image the facts were computed on (repo-facts records it in
its summary; unknown -> the row is omitted, never guessed). Alert rows carry
their own pin; improve/memory.py counts a repeated alert once.
Runner image rebuilt (+0.39 MB: only the new modules; the quick scan needs no
extra and the rulesets were already in the package), pushed, plane/ re-pinned
and regenerated (validation-46: 105 entries). Fixture test runs the whole loop
graph through the fake ax, whose new `exec` mode runs the real task modules.
Live: one loop Run over three validation-46 cases on the local ax, recorded in
benchmarks/measurements/2026-09-30-harnessx-removal-loop/ (all done, $0 new
spend, 0 proposals, gate verdict no_proposals).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
improve/memory.py: rules M1 (enabled->shadow on repeated false alerts across
>=2 repositories, no agreed or site_match hit) and M2 (shadow->enabled on
>=2 agreed declared sites no enabled rule catches); M3 disputed families are
advisory (signals.json) and never reach the validator. The train-on-test
guard drops rows of --qualify-population, gated benchmark cases and holdout
pair repositories before any rule sees a row. run_round(proposals=...) adds
memory edits after the benchmark's through the unchanged validator and gate;
provenance (row ids, object keys and versions, counts, exclusions, index
digest) goes to memory_proposals.jsonl and the journal's evidence field;
proposal_outcome rows are written back after a non-dry run.
ousast improve --memory is opt-in; default improve output is byte-identical
to the task-1 baseline.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
plane/memory.py: the MemoryStore interface with FileStore (index.jsonl, facts by
content hash, repos/<host>__<owner>__<name>/<pin>.jsonl, idempotent ingest,
deterministic row ids, refusal below 1 GiB free naming the path) and MinioStore
(object metadata and tags, S3 Select pushdown probed once per process with a
logged local fallback, object version ids on every row read, versioning and
runs/ lifecycle helper, presigned URLs). OUSAST_MEMORY selects the backend;
MINIO_* settings come from .env; the minio SDK is the optional extra `minio`.
plane/tasks/remember.py: model-free rows (facts, verdict, unit_cost, alert)
from a case's artifacts; remember_run computes them on the host for a recorded
run without writing into it. reconciler.mark_done (477 lines total), seed
before and ingest after `ousast plane run`, `ousast plane remember <run>`.
The generator annotates each repo-facts Task with openultrasast.io/memory-key
(repo, pin, candidates digest, image digest) and the Run with its population
and split; plane/ regenerated.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Equality script (design section 3) records suite outcomes, `ousast pairs --json`,
the python-vulnerable quick benchmark, one standard scan and `improve --dry-run`
from a frozen export with the harnessx extra absent and no model reachable.
Two independent records are byte-identical (determinism.json).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer: use MinIO features including its JSON support; a server exists.
One MemoryStore interface with FileStore (laptop, tests) and MinioStore (the
store of record for the separate Kubernetes deployment): tags for indexing,
S3 Select pushed-down queries probed with a local fallback, versioning for
proposal provenance, lifecycle for raw outputs, presigned URLs and bucket
notifications kept for the follow-on that removes the host receiver.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer amendment: no HarnessX path stays, optional or not; the plane memory
layer replaces the HarnessX evolution proposer and carries loop engineering
(Requirement 6: persistent run memory, proposals from it through the existing
validator and gate, the loop as a plane Run, provenance per proposal).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The ai-service-plane increment met its gate on ax (plane-increment-2.json), the
condition for removing HarnessX. Requirements: an explicit fate per capability
(moved to a plane task or retired), code/packaging/tests removed, deterministic
outputs proven byte-identical against a recorded baseline, docs and specs
retold, one reviewable sequence. Brief inventory re-verified.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A third verify pass on the 10 disputed candidates, 2-of-3 agreement (Req 6.4),
run on ax by extending the same Run: 16/20 declared sites agreed (reference 16),
$0.0205 per candidate with recorded triage (reference $0.022). Caveats recorded:
a design change on the same set; 17 vs 19 sites flagged in any pass; three
non-declared pgadmin candidates promoted.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
agree's declared_sites_matched counts candidate coverage, not detection. Recounted
from agreed.json rows, plane passes flagged 17 of 20 declared sites against the
reference's 19 (budibase read and fuxa getDaqValue missed by both passes). The
earlier "detection unchanged" reading was wrong.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The design revisit after plane-increment-1.json (14/20 agreed, 19/20 in either
pass): a third verify pass on the candidates passes a and b dispute, and agree
decides 2-of-3 where pass c asked (the a/b rule elsewhere; c alone never decides).
- verify: optional OUSAST_INPUT_ONLY restricts the hunt to the named candidates
(agree's disputed.json); an empty list is done with 0 units and no model client.
- agree: optional OUSAST_INPUT_PASS_C; metrics keep every key and add tiebreak
{asked, flipped_to_agreed, flipped_to_rejected, rule}; usd_total, hunts and
declared-site metrics are on the final decision.
- generate: per case <case>-vc and <case>-final appended to validation-46; the
first 60 entries stay byte-identical, so a rerun keeps their done state.
- Offline replay over the recorded outputs with an empty pass c reproduces the
recorded agree summaries exactly (14/20 agreed, 23 agreed, 10 disputed).
- Spec: Requirement 6.4, design Measurement, task 8.1.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
validation-46 ran done on google/ax: 60 tasks, 357 DeepSeek calls, $0.83 with
recorded triage ($0.0194 per candidate, reference $0.022: cost met). Declared
sites agreed 14/20 against the reference's 16/20: recall criterion not met.
Both runs find 19/20 sites in at least one pass; the two-pass agreement
flicker moved three sites. Design revisit before task 9.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ax names a task's ActorTemplate <task>-tmpl-<8 hex> and Substrate caps names at
63 bytes; validation-46's first task (51 characters) fell back to a missing
default template on the live cluster. Task names are capped at 49 with a
stable hash suffix when longer.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A producer's artifact rendered as spec.files of a generated <task>-inputs
Workspace ends up in ax's AX_WORKSPACES_YAML env value; Agent Substrate
rejects values over 32768 characters, so a 182 KB facts.json broke ax apply
on the live cluster. The reconciler now passes OUSAST_INPUTS instead, the
receiver serves a running task its declared inputs over GET /inputs/..., and
the runner fetches them after the start (failed fetch -> failed summary).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
On the live cluster the case clone ran at boot: the golden boot had no egress
policy, and the real actor was still being restored, so both TLS handshakes to
GitHub were cut. After the start the actor is RUNNING under its policy (the
delivery the same minute went through). Adds the one-case check Run (a subset
of validation-46) and re-pins the task manifests to the current runner digest.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The end-to-end smoke Run finished done on the local ax: submit, egress policy,
resume, router start, repo-facts in gVisor, artifacts delivered through the
egress gateway, attribution table printed, task and policy cleaned up.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Agent Substrate's egress gateway denies an actor without an EgressPolicy, so
no task could deliver its artifacts or reach a model. egress.py grants each
task exactly: http/80 to ousast-receiver.ax-system.svc.cluster.local, and
TLS passthrough/443 to the bound Workspaces' https Git hosts and the bound
Model's openultrasast.io/egress-hosts (api.deepseek.com for deepseek-flash).
No IPs, no wildcards, no https (MITM) rules.
The reconciler writes the policy with kubectl-ate between ax apply and
resume, reads it back, waits OUSAST_EGRESS_SETTLE (11 s, the gateway's
cached deny), and after ax delete task confirms the policy went with its
actor (kubectl-ate has no delete verb; the store cascades).
The receiver is a selector-less Service (ops/ax/receiver-service.yaml.tmpl,
applied by up.sh together with the no-build agentgateway egress gateway)
on port 18090 of the host. Measured live: the gateway checks the policy on
Host but connects to the address the actor dialled, so the runner dials the
Service's ClusterIP (OUSAST_ARTIFACT_DIAL) with the receiver's name as Host.
The runner now labels its delivery with the Run's task name (OUSAST_TASK),
which is what the receiver accepts; the ax Task name was refused with 403.
The Receiver moved to egress.py to keep reconciler.py at 460 lines.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
On the live cluster ax created the Task before its Workspace in the same apply
and the runner saw an empty AX_WORKSPACES_YAML. Tests now find rendered
documents by kind and name, not position. ops/ax/smoke-run.sh renders a
model-free repo-facts Run with the current runner digest and executes it
through the reconciler.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
agree matched declared sites through benchmarks/independent/evaluate.py,
which the runner image does not carry, so site matching would have been
"not assessed". The generator now resolves the fix's old-side ranges
(evaluate.hunks, line for line) into case.json, with the declared sites in
the set and the reference's cost, and agree.matches_declared reproduces
matches_v2; a parity test runs both over the recorded outputs of the 15
validation cases (200 probes, identical verdicts) and agree on those outputs
returns the reference's 16 of 20 and $0.022.
agree reports cost_per_candidate (plane spend) and
cost_per_candidate_with_recorded_triage; Requirement 6.2 is judged on the
latter (design.md). --runner-image re-pins the templates from ops/ax/up.sh's
pin file; the committed manifests are regenerated at the current digest.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Agent Substrate's golden-snapshot boot runs the same image with the same task env,
so the runner no longer starts its command by itself: after the workspaces are
ready it waits for POST /ousast/v1/start (run/task checked, once per boot, refused
after a delivered run). The reconciler sends it through svc/atenet-router with
ate-target-actor, which never addresses the golden actor (plane/router.py), and the
bound Model's secretKey variable travels only in that body, into the child's env.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The live cluster ran the task command twice: in the actor and in Substrate's
golden-snapshot boot, which gets the same image and task env. Requirement 4.5
gates the command on a router-addressed start request; 4.6 carries provider
credentials only in that request.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
On the live cluster ax hands AX_TASK_YAML with metadata.creationTimestamp and a
status block; the strict authoring schema rejected it, readiness never turned
green and every resume timed out. Other unknown fields still fail. up.sh retries
the first resume while the golden snapshot builds.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The first live run on the local ax cluster and ax's source (docs/runner.md,
docs/sandbox.md, internal/controller/reconciler.go, internal/server/server.go)
showed a lifecycle the task-3/4 code did not model:
- `ax apply` creates a Task Suspended (Ready=False, TaskSuspended); it runs only
after `ax resume task`. The first resume of a new image can fail with
DeadlineExceeded while Agent Substrate builds the golden snapshot, and ax's
server then stores phase Failed; a later resume succeeds.
- ax has no Completed phase and never reads a command's exit back.
- The runner is PID 1 and must keep serving after the command exits; stop and
suspend deliver SIGTERM, to be forwarded to the command's process group.
Our runner imported the task in-process and exited, which stopped the
container ("Container guest is stopped") and made every resume time out.
- ax's controller probes /readyz?check=workspace; the query string made our
handler answer 404.
Runner (Req 4.1-4.4): spec.command runs as a child
`python -m openultrasast.plane.tasks.<name>` in its own process group, cwd the
first workspace, AX_METADATA_URL set; ax's metadata paths are served. After the
child exits the runner reads summary.json (a crash without one becomes failed
with the exit code and stderr tail), delivers the output directory, writes
/workspace/.ousast-state/<task>.done.json and serves until SIGTERM (forwarded
to the group, SIGKILL after OUSAST_TERM_GRACE, exit 0). A boot with the marker
never reruns the command; a failed delivery is retried alone. Workspaces are
prepared on the first boot only. AX_RUNNER_EXIT_AFTER_COMMAND=1 is the
unit-test escape.
Reconciler (Req 2.2, 2.3, 3.1-3.3): `ax resume task` after apply, retrying
DeadlineExceeded/Unavailable with backoff up to OUSAST_RESUME_TIMEOUT (900 s);
completion is the artifact delivery; Failed after a resume went through, a
vanished task, or no delivery within OUSAST_TASK_TIMEOUT (7200 s) fail the
task with ax's condition message; `ax delete task` follows. Ax gains resume()
and phase() returns the condition message. doctor moves to plane/doctor.py
(tasks.md task 0 allows the sibling) to keep reconciler.py under 500 lines.
Tests: the fake ax follows the real lifecycle (Suspended, a failing first
resume that leaves Failed, Running forever, delivery only after a resume went
through), plus Failed-with-message, vanish, resume-timeout and task-timeout
scenarios; runner tests cover the child's cwd/group/metadata URL, stay-up and
SIGTERM in a real subprocess, SIGKILL of a SIGTERM-ignoring group, and the
marker. The opt-in live test now checks resume, Running, suspend and a second
resume instead of a Completed phase. design.md and tasks.md state the
lifecycle; the module audit manifest counts plane.doctor.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Model deepseek-flash (declared provider extension, endpoint prices in
parameters, secretKey naming the DEEPSEEK_API_KEY secret), digest-pinned
Task templates repo-facts / verify / agree, and a generator
(plane/generate.py, `ousast plane workspaces --validation-set ...`) that
emits per case the vulnerable-pin Workspace, a files-only inputs Workspace
(recorded candidates after the recorded triage, declared sites, triage),
per-case Tasks and one Run validation-46: facts -> verify a, verify b ->
agree for the 15 cases, 46 candidates (43 kept), budgets 1.5x the recorded
per-pass share. A Run entry cannot add a binding or env, and Run.inputs
names only task artifacts, so the case is fixed by per-case Tasks.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Substrate installs no WorkerPool (resume failed: no free workers) and rejects
images not pinned by digest; ax's default runner image on gcr.io needs
credentials, so the smoke task runs our runner (repo-facts over a one-file
workspace). A release ax CLI skewed from the server (UpdateTask unimplemented).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Requirement 1.1 (Task/Workspace/Model use only fields ax documents) and
Requirement 1.4 (no ax field repurposed) were violated: manifests accepted
metadata.namespace and a Workspace git `commit`, and the reconciler dumped
every git field into what `ax apply` receives.
Facts from ax's source (pkg/apis/v1alpha1/ax.proto, internal/workspace/setup.go):
- ObjectMeta is `name` and `atespace` (plus server-set creation_timestamp);
no `namespace`, no `annotations`.
- GitRepo is `name`, `repo`, `branch`, `dir`, `depth`; no `commit`.
- The API server decodes with protojson.UnmarshalOptions{}: an unknown field
is an error at `ax apply` (confirmed on the live cluster: a rendered
Workspace+Model applied cleanly; the same Workspace with `commit` was
refused with `unknown field "commit"`).
- ValidateModel checks only metadata; any spec.provider string is accepted.
Changes:
- manifests: metadata takes `name`, `atespace`, `annotations` (our extension,
never sent to ax); git entries take exactly ax's five fields (depth int);
`commit` in spec.git is rejected with a pointer to the new Workspace
annotation `openultrasast.io/git-commits: "<git name>=<40-hex>,..."`,
validated for unknown git names and non-hex shas.
- reconciler: rendered docs carry only ax fields; each bound Workspace's pins
reach the Task as env OUSAST_GIT_PINS = {"<workspace>/<git name>": "<sha>"}.
A test walks every rendered doc against ax's field lists.
- runner: pins come from OUSAST_GIT_PINS; git is fetched as setup.go does
(init/remote/fetch with --depth when depth > 0/checkout FETCH_HEAD, default
branch main, cloneDestination's dir and placeholder-name rules), then the
pin is checked out. A pinned entry without a branch fetches the commit.
- workspaces: emits the annotation, `dir: repo` and `depth: 1` instead of
`commit`, so generated case workspaces keep <path>/repo at depth 1.
- design.md: ax field names verbatim; pins and model binding as stripped
openultrasast.io/* annotations carried to the task by env.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
kubectl prints the sentence as one whitespace-split row, which passed the
ready check and reported an absent ax controller as "1 pods ready". Rows must
carry a READY column; an empty namespace fails. Regression test added.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ai-service-plane task 3, Requirements 4.1-4.4 (with 2.2 and 2.3 honoured by the
exit-code contract):
- src/openultrasast/plane/runner.py: the ax-task-runner entrypoint. Reads the Task
from AX_TASK_YAML and the bound Workspaces from AX_WORKSPACES_YAML (task-1 loaders),
serves /healthz and /readyz on port 80 (AX_RUNNER_PORT; 0 picks a free port,
AX_RUNNER_HTTP=0 disables it for unit tests), materialises each workspace at its
bound path (spec.files verbatim; spec.git at <path>/<name>, exported from the
OUSAST_CASE_CACHE checkout whose origin matches, else a shallow clone at the pin;
a repo already present on a resumed volume is kept) and only then answers 200 on
/readyz. Runs spec.command by importing openultrasast.plane.tasks.<name> with the
remaining elements as argv; sets OUSAST_WORKSPACE_DIR from the first binding when
the Task's env leaves it unset. A crash writes summary.json with status failed and
the traceback. When OUSAST_ARTIFACT_URL is set, the whole OUSAST_OUTPUT_DIR is
POSTed as an uncompressed tar (Content-Type application/x-tar, X-Ousast-Task,
X-Ousast-Run) with three attempts and backoff; a failed delivery rewrites
summary.json to failed and exits 2. Exit codes: done 0, failed 2, unfinished 3.
- plane/Dockerfile.runner: python:3.12-slim + git, `pip install .` (no extras), a
shell wrapper at /usr/local/bin/ax-task-runner exec'ing the module; built and
loaded into kind by ops/ax/up.sh, not by the tests.
- tests/test_plane_runner.py (+ tests/plane_stub_task.py registered as the task
module): env contract on a files workspace, readyz 503 then 200, delivery to a
local receiver with the headers and tar checked, delivery failure -> failed and
exit 2 after three attempts, crash -> failed with traceback, exit codes per
status, case-cache export at the pin, shallow-clone fallback, resume keeps the
checkout.
- model/audit.py: SPEC_OWNED row for plane.runner; module-audit measurement
regenerated (122 modules, no orphans, note kept).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Requirements 3.1, 3.2, 3.3, 3.4, 3.5, 2.2, 7.1, 7.3 of ai-service-plane.
- plane/reconciler.py (492 lines, test-counted): a Run's tasks submitted to ax
in dependency order through the injectable `ax` CLI (apply / get task /
delete task), serialize labels never overlapping, --workers, state.json with
pending|running|suspended|done|failed|unfinished, a PID lock per run, done
tasks skipped on rerun, unfinished tasks holding their dependants, a failed
task halting the run. Budgets, inputs, output dir, artifact URL and the bound
Model (annotation openultrasast.io/model; identity and priced parameters) go
to the task as env in the rendered Task; consumed artifacts travel as files
of a generated Workspace bound under /workspace. A threaded HTTP receiver
takes each task's tar delivery, validating run/task and rejecting traversal.
Outcome comes from the delivered summary.json (done only when every unit
finished), never from ax's phase alone. status prints the token attribution
table per task and Model from the summaries' usage fields (n/a, never an
estimate) and writes attribution.json; doctor runs the four host checks.
- plane/workspaces.py: one Workspace manifest per case pin of a population.
- CLI: ousast plane run|status|doctor|workspaces.
- tests/test_plane_reconciler.py: a fake ax executable drives the DAG;
tests/test_plane_ax_live.py (marker ax) applies ops/ax/smoke-task.yaml on a
live cluster and skips unless doctor passes.
- Module audit rows and regenerated manifest.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Requirements 5.3, 2.1, 2.2, 2.3, 7.2 of ai-service-plane.
- plane/tasks/verify.py: the batched hunt of verify_batched.py moved into
the package (system prompt, one hunt per file, up to 6 candidates, 6
steps, candidate-anchored sites). New against the reference: each
candidate's known callers from facts.json in the prompt (capped at 8),
tool turns and usage per hunt in units.jsonl, per-unit resume (rows
with "error" are re-asked), pass label from env, MeteredClient wrapper:
BudgetExhausted -> unfinished (exit 3), AccountError -> failed with the
provider message (exit 2); summary.json is written last, names the bound
Model and the summed usage, and is done only when every unit finished.
- plane/tasks/agree.py: verify_batched.agreement over two pass outputs,
agreed.json / disputed.json, per-candidate cost and turns, declared-site
match via benchmarks/independent/evaluate.matches_v2 loaded by file;
metrics block in summary.json; no model calls.
- tests/test_plane_verify.py: a scripted usage-recording hunter client;
callers line, resume, 402, budget stop and resume, model and summed
usage, unpriced usd None, both main() env contracts, agreement metrics.
- model/audit.py + module-audit manifest: SPEC_OWNED rows for plane.tasks,
plane.tasks.verify, plane.tasks.agree; manifest regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
plane/tasks/repo_facts.py computes, from source text only and without a
model or git, the product files of a workspace (tests, vendored trees,
node_modules, dist/build, minified bundles and dot-directories excluded via
the layout facts plus the model_sinks.py filter), the functions each file
declares (Python, PHP, JavaScript/TypeScript, Java, Go, Ruby declaration
patterns extended from sink_candidates.py) and, per candidate function, the
call sites in other product files with path, 1-based line and enclosing
function, capped at 40 and sorted (Requirement 5.1). Output is byte-identical
across runs over the same tree (Requirement 5.2).
Under the task contract the unit is one file: a row goes to units.jsonl as
each file finishes and a restart reuses the rows computed for the same
candidate set (Requirement 2.2); facts.json is assembled from the rows and
summary.json is written last, refusing `done` unless every unit finished.
The entrypoint reads OUSAST_WORKSPACE_DIR, OUSAST_OUTPUT_DIR and optional
OUSAST_INPUT_CANDIDATES only (Requirement 2.1) and exits 0 on done, 2 on
failed with the traceback recorded in summary.json.
tests/test_plane_repo_facts.py: a three-file tree with a tests/ file, a
vendored file, a node_modules bundle and a dot-directory; cross-file callers
with line and enclosing function, same-file and declaration lines excluded,
the 40 cap, byte-identical second run, resume that recomputes only the
missing unit, a crash that leaves `failed` with the unit counts and resumes
to `done`, and the declaration patterns per language.
The two new modules are registered as spec-owned in the module audit (their
importer, the runner, lands in task 3) and the audit manifest is regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
kind cluster + registry, Agent Substrate, ax control plane via ko, one files-only
smoke Task; helpers run from their own checkout because they resolve paths from
the git toplevel. Footprint and README follow once the cluster is up.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Add openultrasast.plane.manifests: frozen dataclasses and loaders for ax's
Task, Workspace and Model kinds (ax.io/v1alpha1, ax's fields only) and the
Run kind (openultrasast.io/v1alpha1: tasks, dependsOn, inputs, outputs,
budget, serialize). Every violation raises ManifestError naming the kind,
metadata.name and field path: unknown fields at any depth, wrong apiVersion
or kind, a provider outside google/anthropic unless deepseek is declared via
the openultrasast.io/provider-extension annotation, a Run with a dependency
cycle, an input naming no task, and a Run task naming no loaded Task.
load_manifests reads multi-document YAML files and indexes by kind and name.
PyYAML becomes the one declared core dependency; the two zero-dependency
guards now pin exactly that, and the module audit registers the plane
package under the ai-service-plane spec.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer decisions 2026-09-29: ax (kind + Agent Substrate + ax controller on
this host) is the only executor; the Run layer carries dependencies, artifacts,
budgets and attribution until ax has them. Requirement 7 declares routing per
Task (one Model per Task, no runtime switching) and a token attribution table
from provider usage fields. Task 0 (bring-up, doctor), runner artifact delivery,
ax-backed reconciler with a fake-ax test seam. harnessx-removal brief records
the HarnessX inventory; removal is gated on the increment's measurement.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
MeteredClient wraps a ChatClient, sums the provider's usage rows per call,
prices them from a Model manifest's parameters (cache-hit, input, output per
million), and raises BudgetExhausted before a call once the usd or calls
ceiling is reached. A 401/402 or "Insufficient Balance" failure becomes
AccountError with the provider's message. An unpriced model reports usd None,
never zero, and refuses a usd budget; summary() gives the spend block of a
task's summary.json.
The two plane modules are registered as spec-owned in the module audit (their
consumers land in tasks 3-4) and the audit manifest is regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Nine tasks in dependency order: manifests, budget metering, runner contract,
reconciler and CLI, repo-facts, verify and agree, the increment's manifests, the
measurement against the recorded batched check, handover.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Package src/openultrasast/plane/: manifests (ax kinds + Run), a reconciler under
500 lines with persisted status and resume, an ax-contract runner, budget metering,
and task entrypoints. First increment: repo-facts (source-only callers, no model),
verify (the batched hunt with callers in the prompt, per-unit resume), agree.
Measurement on the 46-candidate set against the recorded batched check.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Six requirements from the brief: ax kinds plus a Run kind; isolated, idempotent,
resumable tasks with metered budgets; a local reconciler under 500 lines with no
pipeline logic; ax's runner contract; repo-facts as the memory layer; and a
measured first increment (repo-facts + verify on the 46-candidate set: at least 16
of 20 declared sites, under $0.022 per candidate for two passes).
Phase: requirements, not yet approved.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Measured on the main session: 90% of development tokens were tool I/O. Three
tools from the Portal pattern:
- read_guard.py: PreToolUse hook that denies a whole-file Read over 350 lines and
points to a ranged read or the bulk reader (pipe-tested: deny, range, small).
- bulk_read.py: answers a question about files with the cheap chat model, citing
path:line, under a spend ceiling.
- token_report.py: where a session's tokens go, from its transcript.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Adopt the design of google/ax (Task / Workspace / Model manifests, a reconciler,
isolated resumable execution, a runner contract), not yet its software: it needs
Kubernetes and Agent Substrate, lists only Google and Anthropic providers, and has
no pipelines, artifacts or budgets. We add a thin Run layer for those.
Each pipeline concern (classify, triage, repo-facts, verify, judge, recheck, score)
becomes a declared task with its own inputs, outputs and budget. That bounds the
tokens per task and lets each be developed alone. Decisions listed for the
maintainer.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Exploratory, 7 of 17 cases, stopped when the account emptied:
- source-only scope: a model reading every product file puts the vulnerable
function among the candidates in 15/17 (ranker 1/17, vocabulary 2/17);
- per-file triage + per-file 6-step hunts + two-pass agreement: 12/15 candidate
cases on the validation set; 4/7 on the full run, two more in one pass of two;
- precision: 5 pgAdmin agreed findings read against source, 3 true (privileged),
2 false with self-contradicting rationales; agreement cannot catch an error
both passes make;
- cost: $0.009-0.022 per candidate for two passes.
Four instrument defects fixed on the way (model id, provider routing, JSON-mode
final turn, unbounded tool output) void the earlier hunter number. No gate is
met; qualification needs population v3 and a protocol that states the judge
stage, the pass rule and a config-family question.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
- verify_batched.py --fixed asks the agreed candidates again at the fixed pin
(no triage); a function the fix removed is recorded as removed.
- score_batched.py applies protocol v2's matching to the batched output
(vulnerable pin, and the fixed pin where rechecked) and draws the precision
sample.
- A finding's site is anchored to the candidate operation asked about; the
model's own reported location is kept in `reported_at`. ReactPress's confirmed
findAll was reported at the controller file and failed to match its declared
site. The scorer anchors older records the same way.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
verify_batched.py replaces the per-candidate verifier's design:
- triage per file (no tools): where does the operation's data originate;
internal/constant candidates are not verified;
- one 6-step hunt per file for up to 6 candidates, a verdict per operation;
- two independent passes; only agreement is reported, the rest is "disputed".
On the 46-candidate validation set ($1.01):
- declared sites: 16 of 20 agreed, 12 of 15 cases (the per-candidate design
confirmed 11). Budibase and Froxlor are new; Unleash dropped to one pass of two.
winml is triaged out as "constant": a CORS literal, which the reachability
question does not fit.
- the 26 pgAdmin candidates that flickered between single runs: 2 triaged out,
9 agreed in both passes, 4 disputed, 13 quiet.
- $0.022 per candidate for two passes vs $0.033 under the old design; on a file
with several candidates, 2.3x cheaper per pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Declared vulnerable sites: 11 of the 15 that were candidates are confirmed (8 at 3
steps, 3 more at 6). Missed: winml (a CORS config; the reachability question does
not fit), FUXA, Budibase, Froxlor.
Restart-based escalation is strictly worse than a straight 6-step budget: 88% of
pgAdmin candidates used every step, so they paid 3 + 6 ($0.0201 per candidate vs
$0.0139 baseline). Defaults are now 6 steps, no escalation. The cache order change
did not lower the per-call cost ($0.002): the conversation prefix was already
cached; 56% of prompt tokens are served from cache.
The larger finding: on 180 pgAdmin candidates verified twice, 15 and 13 were
flagged with only 2 in common. Verdicts on borderline candidates flip run to run,
while the declared sites were found consistently. Precision needs agreement
across runs or a second judgement, not a single pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
--declared-sites-only, --candidates-from <jsonl> and --results-suffix, so the
efficiency change can be validated on the 180 already-verified pgAdmin candidates
and the 15 declared sites without touching the main results.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Token efficiency, measured on 180 verified pgAdmin candidates: stage 2 costs
$0.0139 per candidate, 90% of the spend, because the conversation is re-sent on
every turn and grows with each tool result.
- Context files now precede the question in the hunter's first message. DeepSeek
bills a cached prefix at 1/30 of a fresh one, by prefix; with the question first,
every candidate asked about the same file (about 5 per file) broke the prefix on
its first line and paid for the file again. A test pins the order.
- The verifier runs each candidate at 3 steps and repeats it once at 6 only when
the hunt spent every step on tool calls without a verdict. On the seven declared
v2 sites, 3 steps found 5 and 8 steps found 6; most candidates conclude in 3.
- Per-candidate token usage (prompt, cached, completion, calls) and escalation
are recorded, and each case reports its cache share, so the next efficiency
decision is made on numbers.
Validation on the 180 pgAdmin candidates plus the 22 known sites (about $1)
waits for account credit: DeepSeek returns 402 at the moment.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The rerun hit DeepSeek HTTP 402 ("Insufficient Balance") after 180 pgAdmin
candidates. Every later call failed at $0, and the runner wrote sixteen cases as
complete with nothing asked, because an unverified candidate counted toward
completion. That is a clean-looking zero from an instrument that could not call
the model.
Now:
- 401/402 or "Insufficient Balance" stops the run at once, exit 2, nothing
recorded;
- only verified candidates count as done, so unverified ones are asked again on
resume;
- a case is complete only when every candidate was verified.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
FUXA failed every hunt with HTTP 400, and neither run could say why. The client
discarded the response body. With the body shown, the reason was "This model's
maximum context length...": FUXA ships minified bundles whose single lines run to
megabytes, and grep_repo returned a matched line whole.
- grep_repo and find_refs quote a matched line (240 characters), never the whole
line.
- Every tool result is capped at 16 x max_chars, with a truncation marker.
- OpenRouterChatClient errors carry the provider's response body (first 500
characters). That would also have exposed the rejected model id in one call
instead of eighteen days.
After the fix, FUXA candidates verify normally at $0.004-0.013 each. A test covers
the megabyte-line case.
verify_sinks.py: skip futures cancelled at the spend ceiling instead of reading
them (the run crashed with CancelledError).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The final-answer turn requested the provider's JSON mode with the tools still
attached. DeepSeek (deepseek-flash) then answered with the format itself
(`{"type": "json_object"}`), wrapped the array (`{"type": "json_object",
"value": [...]}`), or emitted its tool-call markup as text. A hunt that had found
the vulnerability reported nothing.
Measured on the v2 sink verifier with the declared vulnerable functions: 0 of 7
confirmed in JSON mode, 5 of 7 asked plainly (6 of 7 at 8 steps). The model named
Medplum's actual redirect-URI prefix-match bug, LMDeploy's redirect bypass of
_is_safe_url, and ContextForge's unsandboxed Jinja2.
The final turn is now plain text without tools. The parser also reads a
`{"value": [...]}` wrapper. Two tests that pinned JSON mode now pin the plain
turn, and a new test reproduces the wrapped answer.
Invalidates as measurements: the hunter experiment on v2 (c4a0bd8) and the first
sink-verifier run. Both used this turn.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Each grep_repo and find_refs call re-enumerated the tree (walk, ignore rules,
vendored checks) and re-read every file. The v2 sink verifier spent almost three
hours on its first case (pgAdmin, about 1,800 files) at 108% CPU with one open
connection. It was CPU-bound in these tools, not waiting on the model.
Source lines are now memoised per root, keyed by path and modification time. A hit
is re-validated file by file, because an edit inside a subdirectory does not touch
the root's mtime; a test covers one enumeration and a detected edit.
verify_sinks.py now runs candidates in worker processes, not threads, and appends
each candidate's result to a per-case log, so a stopped run resumes where it left
off. GoPay: 12 candidates in 14 s, the real site found.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Per candidate function, one bounded tool-hunter pass (custom prompt, the file as
context, 3 steps) asks whether attacker-controlled data reaches the operation
without a sufficient guard. It reports only with a stated source-to-sink path.
Hard spend ceiling, parallel workers; one run per pin; the fixed pin is rechecked
only for detected sites. Exploratory on v2.
Pilot: on GoPay, the one real site was reported in 3 of 4 runs and nothing else
out of 12 candidates.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Source-only selection on the v2 vulnerable pins, compared on the same cases:
- ranker top-20 entry points: the vulnerable file was in scope in 1/17;
- shipped-vocabulary sink-first: the declared site was a candidate in 2/17;
- model-classified sinks (deepseek-flash, no vocabulary): the declared site was a
candidate in 15/17, and a fix or site file in 16/17. Every chunk was classified;
$14.42.
Misses: User Registration (file in scope, not the function) and Sulu (JCR-SQL2
query builder). The cost of this reach is volume: 3-1037 candidate functions per
repository, 4,762 in total. The next stage must decide which candidates are fed by
attacker data without a guard. v2 is spent for tuning; this qualifies nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This asks DeepSeek, per product source file (in chunks of 400 numbered lines),
which functions perform the family's dangerous operation on non-constant data. It
is scored like sink_candidates.py: is the declared vulnerable function among the
candidates? A failed chunk counts as unclassified, never as "no sinks". The spend
ceiling is enforced across worker threads.
Pilot: GoPay found its declared site for $0.03. Estimated full v2 run: ~$14-18.
The maintainer raised the ceiling to $20.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This is a source-only diagnostic: no engine and no model. At each v2 vulnerable
pin, every call matching the family's shipped sink vocabulary becomes a candidate,
attributed to its enclosing function.
- The declared vulnerable function is a candidate in 2/17 cases (Medplum, GoPay);
the ranker's top 20 managed 1/17.
- A fix-changed or site file is among the candidates in 8/17.
- Candidates per repository range from 2 to 904 functions.
The misses are vocabulary, not direction. Examples: wp_redirect, download_url,
spawn(shell:true), ky(), requests.Session().get, sympy parse_expr, Jinja
from_string().render, Mongo collection.find, JCR-SQL2 createQuery, TypeORM
andWhere template strings, and CORSMiddleware(allow_origins=["*"]) are not named.
v2 is spent for tuning, so this informs design and qualifies nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The run follows hunter-protocol-v2.md: deepseek-flash, the ranker's top 20
regions per pin, protocol v2 scoring, $8.88 spent.
- Recall: 1/17 (Wilson 0.01-0.27).
- Precision: 2/14 sampled (both true positives need a privileged role, one of
them GoPay's CVE).
- Fixed-side alerts: 1. Benign alerts: 16.
- Stability: 73% of findings differ between two identical runs.
The deciding observation: the ranker's top 20 contained the vulnerable file in 1
of 17 cases, and in that case (GoPay) the hunter detected it. With 4 tool steps
per hunt it cannot search its way to code it is not handed. The result therefore
says less about the model than about scope selection, which sits upstream of the
engine too.
Instrument failures, reported as failures: every FUXA pin and three Directorist
pins (DeepSeek HTTP 400 on every hunt; neither offered a region of its family).
Scorer fix: protocol v2 defines benign alerts and the precision pool over the
CASE FAMILY, but score_v2 pooled every family. The engine's dump emits only the
case family, so its recorded result (21adc40) is unchanged by this; rescoring
reproduces 3 benign alerts and 49 pooled findings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This is the deciding experiment of the strategy review, fixed before any v2 pin is
hunted.
- Hunter: the unchanged tool hunter (default prompt, tools and 4 steps) on
deepseek-flash.
- Scope: the ranker's top 20 regions for the case family, in four hunts. The model
never sees the fix diff or the declared sites.
- Families: a fixed keyword table maps the model's family to the protocol's three.
- Scoring: exactly protocol v2's scorer.
- Cost: a hard $10 ceiling.
hunt.py runs it and scores it (--score). A pilot on a v1 development case
(mongo-express) found the vulnerability at lib/bson.js:60, at $0.031 per pin,
alongside a lot of speculative noise.
Transport fix found by the pilot: the DeepSeek endpoint the hunter builds was
unpriced, so the cost meter read $0.00 for any number of calls. It is now priced.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Every model call has failed since 2026-09-10 (83ab533):
- The default detector and judge model was `deepseek-v4.1-flash`, the product name.
The API rejects it with HTTP 400 ("The supported API model names are
deepseek-flash, deepseek-v4-pro"). The judge and pipeline record an endpoint
failure as a degradation, never a verdict, so nothing failed loudly.
- `tool_hunter.resolve_hunter_client` resolved OpenRouter only. In this project
DeepSeek is the chat provider and the OpenRouter key is scoped to embeddings, so
the hunter could not reach a chat model at all; the call came back 401.
The defaults are now `deepseek-flash` (priced). The hunter resolves DeepSeek first
when its key is set, and OpenRouter stays as the explicit or fallback route.
Verified live: plain completion and a tool call through both the endpoint resolver
and the hunter client.
Not affected: the independent evaluations (finding_dump runs with
max_model_calls=0) and the 2026-09-06 hunter measurement, which predates the id.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
v1 (1/11) and v2 (0/17 recall, 0/17 sampled precision, 12% completion) show that
fixes learned on one population do not transfer to the next. The causes are
breadth: framework-shaped entry points, module boundaries the frontends do not
resolve, name-matched sinks colliding, and scale.
Options:
- A: continue engine-first;
- B: the model proposes, the engine proves;
- C: narrow the product to one ecosystem.
Recommendation: first run the unchanged tool hunter on the v2 pins as the deciding
experiment.
For maintainer decision; no approved spec changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The analyzer was frozen at a75b9f6. 17 cases, 102 scans (vulnerable and fixed pins
twice each), scored exactly as pre-registered.
- Recall: 0/17 (Wilson 95% 0-0.18). No unstable detections.
- Precision: 0/17 sampled (0-0.18). Every sampled finding is false: regex
match/exec read as sinks, int casts, ORM statements, client-SDK code, the
process's own argv, a generated database name, a registered-URI check.
- Completion: 13206/108374 questions (12%).
- Fixed-side alerts: 0, trivially, since nothing was detected.
- Benign alerts: 3.
- Stability: 2 of 100 findings differ between runs.
The zero is not an instrument artefact: FUXA, GoPay, cct-studio, ReactPress, User
Registration and Directorist completed questions in the sink file itself and still
found nothing.
Instrument failures, reported as failures: LMDeploy (every taint batch failed on all
six pins) and Froxlor's two benign scans.
This record is committed before any analyzer change informed by v2.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Committed before any v2 case is scanned. It has two changes from v1, both chosen by
the maintainer from lessons of the v1 run:
- declared sites: a finding at a `path::function` the advisory names is credited,
as well as one on the fix's changed lines;
- two runs of the vulnerable and fixed pins: detection needs the finding in both
runs, a fixed-side alert in either counts, and the difference between runs is
reported as stability.
evaluate.py takes --population. v2 scans six pins per case and scores with
matches_v2 / score_v2. v1's scoring is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The analyzer was not run. freeze.py read each case's source at both pins:
- both pins are commits in the repository and in the right order;
- the declared sink site's file is present at the vulnerable pin, and its declared
functions appear in it;
- the fix touches a declared site file (Budibase's sites reordered so the changed
file, queries.ts, comes first).
Benign controls:
- 12 are reviewed automatic proposals.
- 3 automatic proposals were rejected as not benign: qwed's merged a ReDoS fix,
ReactPress's corrected a SQL column whitelist, and Sulu's was a branch merge that
can carry the fix.
- Those three, plus cct-studio and Directorist (no automatic proposal), were chosen
by hand. Directorist's comes from before the vulnerable pin, because only 3
commits follow its fix.
freeze.py now accepts a recorded benign control instead of reporting that none was
proposed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
v1 was evaluated and then used to retune the analyzer. A retune can only be
qualified on a population that did not inform it.
The candidates came from web research alone (research-2026-09-26.md):
- advisories were taken newest-first per language under fixed criteria, never by
how easy they look;
- 261 advisories were examined and 224 rejected, each with a reason;
- every SHA pair was verified through the GitHub API;
- no clone and no scan.
The maintainer selected the rule-based "balanced" set: 5 Python, 6 JS/TS, and 6 PHP
of which 4 are WordPress plugins; 10 injection, 6 destination and 1 config.
Each case declares the advisory's sink sites before any scan, for protocol v2.
freeze.py takes --population and writes freeze-vN.json. The reservation tests now
cover every population-v*.toml.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This reverts commit cc885f0. It failed the acceptance bar set before the run (each
recall pin within +2 findings, completion unchanged):
- WP Statistics: 16 -> 31.
- PMPro: 16 -> 26, and completion 378/382 -> 103/382. Six prior findings were lost
to the budget; CVE line 936 survived.
- YesWiki: completion 2798 -> 1968, still not detected.
Two causes:
- Cost: wrapper confirmation runs a flow query per parameter of every sink-bearing
method, repeated at each engine start of a portioned scan.
- Precision: a helper whose parameter reaches a sink under its own escaping (for
example `getMemberOrderByPayPalToken(sanitize_text_field(...))`) is not a sink
for its caller.
Doing this properly needs summaries that carry the wrapper's internal
sanitisation, computed once per scan, not per request batch.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This is the second attempt at the YesWiki shape: SQL run through
`$this->dbService->query($query)`, whose body is the `mysqli_query` the facts name,
through a receiver the frontend cannot type. The first attempt was reverted in
c7e3a05.
A method is now a wrapper for a parameter only when the engine's own dataflow
inside the method carries that parameter to a fact sink's argument, on a plausible
path that no cleansing or guard discharges. The guard rails:
- one level only;
- nothing under a bundled library tree (lib/, vendor/, node_modules/, ...);
- a name counts only if every project method with that name is a wrapper;
- wrappers are asked in the flow pass only, never in the evidence pass, so no
region's selection or pruning changes;
- at a wrapper call, only the confirmed argument is asked.
Three Joern-gated tests cover the direct shape (reported), a literal, an
intval-casting helper and a level-2 feeder (not reported), a harmless same-named
method, and a wrapper inside lib/. Each fails when its rule is removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This reverts commit 54c9dfc. The recall-pin check (p16) showed a large precision
and completion regression, although every recall target survived:
- MW WP Form: 3 -> 60 findings. Form-field renderers became sinks.
- WP Statistics: 16 -> 40.
- PMPro: 16 -> 65, 20 of them inside bundled Braintree/Stripe code. Completion fell
to 390 of 1189 questions, because wrappers made many more regions look
sink-bearing.
- YesWiki: completion fell from 2798/2807 to 216/2807.
The structural summary was too loose. Any identifier in a sink argument counted as
fed by a parameter if any assignment anywhere in the method mentioned it, with no
order or dominance. It also ignored vendored library code and cascaded three
levels by name. A redesign needs dataflow-confirmed, level-limited summaries that
leave the question count alone.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Real code rarely calls the driver where it builds the query. YesWiki builds its
INSERT in `FormManager::create($data)` and runs it through
`$this->dbService->query($query)`, whose body is the `mysqli_query` the facts name.
The only sink the facts knew sat in a helper that no request reached directly.
A method whose parameter feeds a family sink is now a sink for that argument,
reported at the call site. The summary is structural, to stay cheap and
deterministic. It is not a flow query per parameter, which would repeat at every
engine start of a portioned scan. The parameter must reach the sink call's
argument directly or through at most two local assignments, with at least one
occurrence outside every cleansing call. It iterates up to three levels, so a
wrapper of a wrapper counts.
Calls resolve by name, because an injected receiver has no type the frontend can
see. A name therefore counts only if every project method with that name is a
wrapper. At a wrapper call, only the summarised arguments are asked.
Two Joern-gated tests cover:
- a request value into a second-level wrapper, reported at the controller's call;
- direct use, a literal, and an `intval`-casting helper, which are not wrappers;
- a harmless same-named method, which disqualifies the name.
Both fail with their part removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reading /proc/<pid>/stat for a process that is reaped during the read raises
ProcessLookupError (ESRCH), not FileNotFoundError. The previous race fix caught only
the latter, so the hanging-process-group test failed twice today under full-suite
load. It passed alone. The disappearance is exactly what the test asserts.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
p14 showed PMPro's functions.php:906 discharged by `elseif (in_array('E', $levels))`.
That checks whether a constant is in `$levels`; it says nothing about `$levels`. The
guard counted a carried variable in any argument. Only argument 1, the tested value
(`in_array`'s needle, the URL given to `url_has_allowed_host_and_scheme`), is
validated now. The allowlist test gains the haystack shape as a reported control.
The test fails when any argument counts.
The other p14 removals were right: WP Statistics export_data x2
(`if (!in_array($table, $allow_tbl)) $table = false;`) and PMPro memberorder.php:1633.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Ultimate Member's fixed pin kept one alert. The fix writes
`$this->sql_order = " ORDER BY u.{$sortby} "` inside
`elseif (in_array($sortby, $allowed, true))`, and after the if/elseif chain it
passes the field through `apply_filters(...)` into a second assignment. The guard
was judged only at the flow's final node, which is outside the branch.
A guard is now also judged at every point where the path stores the value (an
assignment target). A value stored while the check held carries the check forward,
because the rest of the path derives from that store. A mere read inside a branch
is not a candidate: the engine lists uses in branches the flow does not depend on.
The Joern-gated test gains the shape and its control: the same store in the
unguarded else branch, through the same filter, is still reported. The test fails
when only the end is judged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Three of the four fixed-side alerts left in the independent population had one
cause: the fix was a check, and checks were not modelled.
- wger's fix is `url_has_allowed_host_and_scheme`: `if not ...(next_url):
next_url = reverse(..)` and `if next_url and ...(next_url): return redirect`.
- Ultimate Member's fix is an allowlist: `in_array(strtoupper($order), [...],
true) ? $order : 'ASC'` and `elseif (in_array($sortby, $allowed, true))`.
(The fourth, Signal K, is `assertAllowedHost`, a project-local function. It is not
covered.)
Sanitizer facts can now say `guard = true`. A guard is a check, never a cleansing
call on a path. A flow is discharged only when a guard tests a variable the path
carries, and the flow is in one of these places:
- inside the branch whose condition requires the check. The guard may be combined
with `&&` only;
- past a failure test, a single `!` combined with `||` only, whose branch exits
(return, exit, die, wp_die, throw) or overwrites the value, and whose condition
dominates the flow's end;
- through a ternary whose condition requires the check, where the other arm cannot
carry the value. The engine can enter through the condition's own copy of the
value, which never becomes the result.
Facts: `in_array` (PHP) and Django's `url_has_allowed_host_and_scheme` /
`is_safe_url` (Python). A row now consults guards even when a request lists no
sanitizers.
Two Joern-gated tests cover the discharged shapes and, as controls, an unguarded
redirect, a guard on a different variable, a failure that is only logged, the
unguarded else branch, and the ternary arm the check does not select. Both tests
fail without the guard.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The enumeration dropped every `!` line and matched with fnmatch, whose `*` crosses
`/`. YesWiki's `tools/*` followed by `!tools/bazar` therefore removed all 583 PHP
files under tools/, the vulnerable FormManager.php among them. The scan read 243 of
803 files, completed 1128 of 1128 questions and looked clean: a zero from an unread
tree.
Rules now follow git:
- the last matching rule wins, and `!` re-includes;
- `*` and `?` stay inside one path component, and `**` crosses components;
- a rule with an inner `/` is anchored to the root;
- a trailing `/` matches directories only;
- nothing under an ignored directory can be re-included.
In a work tree, a tracked file is never ignored, because `.gitignore` describes what
the repository does not hold.
The test uses `git check-ignore` as the oracle on 16 paths covering negation,
anchoring, depth, directory-only and `**`. It also covers a git-archive export with
no .git, and a tracked file under an ignored directory. All three tests fail on the
old reader.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Two defects found by the cause-B rescan of the independent population (exploratory,
a retune after protocol-v1):
- `typescript_property_support_unvalidated` still reached the whole repository.
YesWiki is a PHP application that ships a few `.ts` files, and all 1,128 of its PHP
questions were unresolved. The limit is now owned by the TypeScript partition. A
TypeScript project (Mockoon, Signal K) is unchanged.
- `isSourceCall`, and the method-level "reads a source" heuristics, matched source
patterns as substrings. Flask's `request.get` therefore claimed Django's
`request.get_host()`. wger's fix passes that host to
`url_has_allowed_host_and_scheme`, and both fixed functions were reported as open
redirects from it. Patterns are now matched as bounded tokens, like sinks and
sanitizers. wger's vulnerable-pin detections rest on `request.GET`, which still
matches.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This extends dbcb230 to the remaining graph-integrity gaps. In the independent
evaluation these gaps were repository-wide:
- alerta: one symlink, `alerta/app.wsgi`, which is not even source, made 261 of 261
questions unresolved.
- wger and Signal K: a language no frontend reads did the same.
- YesWiki: unparsed files did the same.
Now:
- the partition build records the paths a symlink boundary skipped (a directory
ends in `/`) and the languages no frontend reads;
- `files_unparsed` carries its full paths;
- the scan demotes only questions about an owned path or under an owned directory,
and only questions in an owned language.
Every gap stays in the report as a boundary. A gap that names nothing keeps its
partition-wide or repository-wide reach. Tier-0 pruning stays off, as before.
The tests that pinned repository-wide reach now assert ownership. The one about a
vendor exclusion masking an integrity failure keeps its intent, using a census gap
that names no file.
Retune after protocol-v1; exploratory only.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
`esc_sql` and the mysql/pg `*_escape_string` functions only backslash quotes. A value
they escape is safe inside a quoted literal and nowhere else. Ultimate Member
CVE-2024-1071 is `$sortby = esc_sql(...)` built into `" ORDER BY u.{$sortby} "`, and
it read as sanitized. Sanitizer facts can now say `quoted_only`. The escape keeps its
credit unless the first string the value is built into (concat, interpolation,
addition, f-string) places it after literal text that opens no quote. With no string
built downstream, the credit stays as before. pg_escape_literal adds its own quotes
and stays a full cleanser.
Two further defects hid the same CVE:
- The engine is field-insensitive on the object. Its one path from `$this->sql_order`
went `$this` -> `$this->having` -> `esc_sql($this->having)` -> ... -> the query.
absorbedByObject cut it, and it credited an unrelated field's escape. When a path
leaves a field through the object for another field and reads the first field
again, the detour is spliced out. The later read may be on the path or inside the
sink argument where the path ends.
- The field join judged a field clean if ANY sanitizer name was on its path. It now
uses the same quote-aware check as a row. This matters because the engine's own
path to the query changed between runs: 2 of 4 identical runs chose a path via
`$wpdb->prepare`. The field join is the deterministic route.
Exploratory (retune after protocol-v1): the Ultimate Member vulnerable pin is detected
at class-member-directory-meta.php:936. The fixed pin still alerts at :945. Its fix is
an allowlist (`in_array(...) ? $v : 'ASC'`, and an `in_array` guard), which is not
modelled yet.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
python.toml's sources were Flask-only, so no flow could start anywhere on a Django
project. OpenCVE's SSRF reads its URL from `form.cleaned_data`, and the scan answered
the sink file and found nothing. Added: request.GET/POST/body/COOKIES/FILES/META,
DRF's query_params, and cleaned_data.
With those sources known, OpenCVE reported three redirects that cannot leave the
site: `reverse("cves") + "?" + request.GET.urlencode()` (twice) and
`reverse(route, kwargs=...)`. Destination sink facts can now set
`prefix_fixes_origin` and `origin_anchors`. A path is cut when it enters a
concatenation, f-string or `+=` through anything other than its leftmost operand,
and that operand fixes the origin. Origin-fixing operands:
- a literal `/x`, `?` or `#`;
- a literal of the form scheme://host/;
- an anchor call;
- a variable whose every assignment fixes the origin.
A path is also cut when it passes through an anchor. `"/" + x` (//evil) and
`"https://" + host` stay reported. A Joern-gated test covers all seven shapes.
Exploratory (retune after protocol-v1): OpenCVE vulnerable and fixed pins are both
0 findings, down from 2 and 3. The SSRF itself is still missed, because the flow
crosses a registry dispatch (`REGISTRY.get(t)(extras).run()`) that the engine does
not follow.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
mongo-express CVE-2019-10758 passes the request body to vm.runInNewContext. No
injection sink named that call, so the flow was answered and missed. runInNewContext,
runInContext, runInThisContext and compileFunction are now CWE-95 sinks.
Once the file's questions completed, the fixed pin showed a second defect. Every call
on `require('mongodb-query-parser')` has that package name as its full name. The
bounded-token match treats `-` as a boundary, so the name matched the SQL sink
`query`. In a full name, `-` belongs to a package name, so it no longer separates
tokens.
Exploratory (retune after protocol-v1): mongo-express vulnerable pin now detected at
lib/bson.js:60.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The first independent evaluation (results-v1.json) completed no question on eight of
eleven projects. On the cases traced, the cause was a census gap: `.eslintrc.js` or
`webpack.config.js` missing from a JavaScript graph, or `contrib/apache/api.wsgi` from a
Python graph, and every question in that language was then marked unresolved. Now only
questions about a named missing file are demoted. The gap is still reported as a
boundary. A gap that names no files still reaches its whole partition.
This is a retune after the evaluation. Per protocol-v1 it cannot be qualified on the
same population.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Scored exactly per the pre-registered protocol-v1.md, analyzer ebcc2cf,
44 scans on eleven untouched cases.
Recall 1/11 (95% CI 0.016-0.377): only Signal K's SSRF sink is detected.
Precision 1/12 on the full pooled sample: the other eleven findings are
guarded redirects (isSafeRelativeUrl, getSafeDestination), Host-header
redirects, constant SET NAMES queries, a config-file eval and an installer.
Four fixed-side false alerts (mongo-express bson.js, both wger sinks after
their guard was added) and one benign.
The dominant cause is coverage, not precision: at the vulnerable pin eight
of eleven cases completed ZERO questions and Saltcorn asked none, so most
targets were never decided. Recorded before any investigation or change.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Committed before any case is scanned: what is under test (the analyzer at the
runner's commit, per-case family only, four scans per case), how a finding
MATCHES a case (a fix-changed file, the case family, the named function or
within 10 lines of a fix hunk), detection / fixed-side / benign false alerts,
unanswered and instrument failures, the precision sampling rule (every third
pooled vulnerable-pin finding, or all if 15 or fewer), what is reported
against the M4 gates, and that no retune may be informed by the results.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Every reserved case verified with git and file reads only -- nothing
scanned: pins present and ordered, the sink present at the vulnerable pin,
the fix touching the sink (or LearnPress's declared source-side fix site),
and the sink file hashed at both pins (benchmarks/independent/freeze.py,
record freeze-v1.json).
Each case now carries a benign control from its own history, reviewed by
diff. The mechanical keyword filter was not enough: it proposed an access
restriction, a filter check, API-token scopes, a login change and a
TLS-verification option, all rejected; Ultimate Member and alerta got
hand-picked controls (a WPCS reformat; a dependency change with both sides
still vulnerable). The guard test requires a benign pair per case once
frozen.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Eleven cases chosen by the maintainer from fifteen researched by advisory
and fix diff only -- nothing cloned, nothing scanned, none among the 262
repositories the benchmarks reference -- plus the earlier Ghost
reservation. Four are PHP repositories that ship JavaScript (two WordPress
plugins), satisfying the multi-language requirement; families are limited
to the v1 scope (injection, untrusted destination, permissive config).
The development subjects shaped every rule of the last week and can no
longer measure whether those rules generalise; this population can, until
something scans it. A guard test fails the moment a reserved repository is
referenced by a recipe, catalog, manifest or measurement outside
benchmarks/independent/. Freezing (pins and sinks verified by reading
source, a benign change per case) is the next step.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The largest remaining class in the third PMPro sample (2 of 4 false
positives): every MemberOrder method builds its statement in
$this->sqlQuery and runs it at once. One method writes request data there,
the field join keyed the field file-wide, and deleteMe() -- which had just
set it from the internal $this->id -- and updateTimestamp() (no caller in
the plugin) were reported as SQL injections.
A field read is now a field-join source only when it reads state another
method left: not the target of a write, and not a read that the method's
own write to the same field dominates. If that own write carries request
data, the ordinary flow from the source still finds it. A real-Joern
control covers all three: the method whose write carries $_POST (found),
the method that overwrote it with an internal id (quiet), and a method
reading what another left (found). It fails on the old query.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Found on PMPro's re-added class-pmpro-levels.php:87: every REST handler
casts the level id, `isset($p["id"]) ? intval($p["id"]) : null`, and the
flow was reported unsanitized. Traced, the engine steps from intval's
argument straight to the conditional, so `intval` is never a path element
and the element-wise sanitizer check never saw it.
- sanitizedBetween: where a path element contains the previous one, the
syntax between them is what the value was wrapped in; a sanitizer there
sanitizes the flow. A sanitizer elsewhere in the expression (the other
branch of a conditional) does not count. Used by the sink query and the
hook join; the field join already matched sanitizer text.
- The test exposed the conditional itself as a source: isSourceCall matched
any operator by its full text. Of the operators only an access
(field/index) names its own value; a conditional or concatenation built
from `$_GET` is not a source.
A real-Joern control covers the cast inside a conditional (sanitized) and
a sanitizer in the other branch (not); 20 query tests pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
profile.php:424 survived the hook-argument fix through the field join: in
another function of the same file, `$user = new stdClass; $user->ID =
$_POST['user_id']`, and the join keys a field by its TEXT across the whole
file, so the WP_User parameter's `$user->ID` became a field source.
The join exists to carry object state between methods ($this->attachments
written in the constructor, read in _delete_files). That reading of a
field's text holds only when the base IS the object ($this, this, self).
Any other base is a variable, matched within its own method only, where
the engine's dataflow already sees it. A real-Joern control covers both
(a local variable's field in another function stays quiet; $this state
across methods is traced) and fails on the old query.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The mapper took the FIRST 'callback' in a 12-line window. One registration
commonly carries an array of endpoints; PMPro's /discount_code has a GET
and a POST handler, so the writer, pmpro_rest_api_set_discount_code, was an
ordinary `function` entry. Since da0ee57 such an entry's parameters are
not attacker input, so its $request stopped being a source and the REST
route that injects SQL through the discount-code save() had no source at
all.
The argument list of the call is now read to its balancing parenthesis,
every callback in it is a route, and each keeps the permission callback of
its own endpoint (the first after it before the next callback, else the
last before it). PMPro: 15 -> 18 REST routes.
This restores the source, not the finding: traced unfiltered, the engine
does not carry $discount_code->code = $code (written in the handler) into
$this->code inside save() once other methods have been called on the
object; the one path it reports detours through a prepared query's result.
Recorded as the next recall gap: a cross-file field join keyed by class.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The core-hook-argument class from the second PMPro adjudication:
pmpro_membership_history_profile_fields($user) on `edit_user_profile` was
a route entry, so its parameters were attacker input -- and $user is the
WP_User WordPress loaded; its ID became an SQL injection.
A callback registered in a string-keyed registry is handed its arguments by
the matching apply call, not by the request. The mapper now records the
registration call on the entry (`registered_by`), regions carry it, and the
dispatch fact says which registrations DO hand a callback request-borne
arguments (`request_arguments`; WordPress: add_shortcode, whose $atts are
post content). Every other registered callback keeps its call depth but
its parameters stop being sources; request data that reaches an apply
call's arguments is still followed by the hook half of the taint query.
REST routes are not registry entries and keep the parameter rule.
Controls: the fact field is validated against `register`; a hook callback
is not input, a shortcode and a plain route are.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The apply_filters class from the second PMPro adjudication (2 of 3
remaining false positives): `apply_filters('pmpro_random_code', $code,
$this)` returned whatever $this carried, and a report's `$type` reached SQL
only as an extra filter argument. Two causes, both fixed generally:
- The dispatch fact now says what an apply call RETURNS: `returns` names
the calls whose result is the dispatched value and `value_arg` its
position (WordPress: apply_filters/apply_filters_ref_array return
argument 2; do_action returns nothing). A path through such a call must
ENTER by that argument -- the first of the run of the call's arguments
before it, because the engine hops between sibling arguments
($_GET["c"] -> md5("x") -> apply_filters). Loaded and validated as data;
the query knows no framework.
- A call was a source when the pattern appeared anywhere in its text, so
`apply_filters(..., $_GET["c"])` and `sanitize_text_field($_POST[..])`
were sources themselves -- the "a token matches the thing it names"
invariant. A call is now a source when the pattern names its callee,
receiver or the access it is; the inner access stays a source.
Controls: dispatch facts load/validate; a real-Joern case with request
data as filter context (quiet) and as the filtered value (found). 1,453
host and 18 query tests pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
da0ee57 turned parameter sources off for the mapper's catch-all `function`
entries, and with them the field join's seeds, which read the same flag:
MW WP Form's CVE-2023-6559 ($this->Data, filled by a constructor) was lost
(10 -> 3). The field join now has its own request field,
`fieldParameterSources`, set by the old entry rule; a region's own
parameters follow the new one. Both known CVEs are found again (MW WP Form
10 -> 4, WP Statistics 39 -> 18, 91/91 and 774/774 completed). The dropped
findings sampled against their call sites all take internal values:
ExistTable('useronline'), wp_statistics_visit('today'), remove() on the
plugin's own temp directory, _render('text', ...).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Two classes the traced PMPro survivors named after 1f91c0b:
- The mapper records every named PHP function as an entry of kind
`function` (3,507 of PMPro's 4,412), and every entry's parameters were
sources, so internal helpers became injections: sendCancelAdminEmail's
$user, a gateway hook callback's $user_id/$morder fired by do_action.
Regions now carry the entry kind; parameters are input only for request
entries (route, parser, privileged surface). Call depth is unchanged, so
such a region is still followed into its callees. JavaScript and Python
emit no `function` entries.
- The object rule is general: a field read continues a flow through the
bare object only if the taint came through the SAME field, or was
assigned to the variable or passed in as a parameter. It now also cuts
$this->Gateway -> this -> $this->user_id (memberorder.php:705).
plausible() memoises the sink test per node and skips the object walk for
paths with no field access (the filter had cost one outlier 101 s -> 160 s).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Traced element by element (new opt-in `trace` on taint requests), every
false PMPro SQL injection in the adjudicated sample took one of two steps
no real injection needs, both over-approximations of Joern's dataflow:
A. Through a query: `$id = $wpdb->get_var("...'" . $token . "'")` is the
injection; the engine carried $token on through the query's RESULT
into later queries (getMemberOrderByID:705, profile.php:515).
B. Through the object: request data written to $order->ProfileStartDate,
or passed as an argument beside $wpdb, tainted the whole object and a
DIFFERENT field read from it came out tainted
(memberorder.php:1122/1633, pmproemail.php:262).
A path is now dropped when it passes through a call to a sink of its own
family, or when a field read is reached through the bare object after the
taint entered it by writing another field or without being assigned to
it. A member of a value that IS tainted (the result of a read, a return,
an assignment) and a field read back after being written are kept. Applied
to the sink query, the field join and the hook join. A real-Joern control
covers both cuts and both keeps, and fails on the old query; all 17 query
tests pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reverts 6ebbc89 and e0c260c (keeping finding_dump's completed-regions
record). Measured on PMPro, comparing only questions both runs answered
(157): 37 -> 31 findings, but the six removed included the adjudicated TRUE
positive discount-codes.php:154 (REST-reachable), and only one of the six
sampled object-field false positives went away; five are still reported.
It also cut completed questions 295 -> 158. MW WP Form and WP Statistics
were unchanged either way.
The diagnosis was wrong: the object-field false positives do not come from
setter parameters. The witnesses name one field (e.g.
$update_order->BillingFrequency) and the query interpolates another
($this->membership_id), which points at field-insensitive propagation
through the object instead. To be confirmed on a flow path before any fix.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Asked of every parameter up front, 6ebbc89's caller check ran a repository-
wide flow query per parameter of every method in a file and cut PMPro's
completed injection questions from 272 to 67 in the same budget -- so its
48 -> 26 findings could not be read as precision. The flow into the field
is asked once as before; only a parameter at the head of an unsanitized
flow is checked against its callers. finding_dump now records completed
regions so runs compare only where both answered.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The field join (task 5.11) seeded every parameter of every method in the
file, so every setter made its field a source. Adjudicated on PMPro, six of
eight false SQL injections in a 1-in-5 sample were object fields holding
database or internal values ($this->membership_id, $morder->user_id).
A parameter now seeds the join only where some call site passes it request
data without a sanitizer on the way (memoised per method and index); a
request source read in the file still seeds as before. A real-Joern control
covers both directions -- a setter only given a constant stays quiet, a
constructor handed $_GET (the CVE-2023-6559 shape) is still found -- and
fails on the old query with exactly the PMPro false positive.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
48 findings, every 5th reviewed against source: 2 true (both privileged),
8 false; 6 of the 8 share one cause, object fields treated as attacker
input when they are loaded from the database. benchmarks/push/finding_dump.py
writes every finding of chosen families for review.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The loop the maintainer called out: four scale defects in the PHP transfer
replay, found one per two-hour Joern run, all Python bookkeeping between
engine calls. benchmarks/push/python_overhead.py ends that: the real
replay() on the real PMPro pins, with a JoernBackend whose graphs are stubs
and whose queries answer instantly with rows recorded from the real engine.
Engine time is zero, so everything measured is ours, in about a minute,
under the sampling profiler. Its first pass named two more walls:
- Region discovery re-parsed the ruleset for every region: 89,127 TOML
parses, 275 s of a 339 s discovery, twice per push. The fact, obligation
and taxonomy loaders now keep the parsed (immutable) result keyed by each
file's name, size and mtime -- an edit in place or an added file reloads.
Discovery 339 s -> 7 s; output digest identical on PMPro.
- affected_context decoded every span's and every line anchor's path for
every context row -- a changed file has an anchor per unchanged line --
78% of a stubbed push. Decoded once per call, anchors indexed by line and
bisected, original order kept. A randomised control compares it against
the row-by-row matching over 200 contexts (>100 attachments).
Result: the whole stubbed PMPro push, 27,542 answered requests and 2,688
outcomes, runs in 58.5 s of Python. What is left is the engine.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
b4b9308 was committed with two failing tests: the module audit read
`from . import profiling` as an import of the package, called profiling
orphaned, and the committed audit manifest drifted. My check read tail's
exit status, not pytest's. Now a from-import; manifest regenerated (109
load-bearing, 0 orphaned); 1,444 pass with pytest's own exit checked.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The cache fsynced every payload and manifest -- two per entry, ~18 ms, and a
push publishes one entry per question (~22,000 on a WordPress plugin). A
read already rejects any entry whose manifest receipt, payload size or
payload SHA-256 does not check, so a torn write is a miss, never a wrong
answer; the fsyncs bought nothing the verification does not.
The broader problem: three quadratic walls in this replay were found one
per two-hour run, because all were Python bookkeeping between engine calls
and the only record was the engine log. `OUSAST_SAMPLE_PROFILE=<path>`
starts an in-process sampler that rewrites the hottest stacks every 30 s,
so a run killed at its deadline still names where its time went. The host
cannot ptrace the container, hence in-process. The replay harness turns it
on per case.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The third wall under the 13.3 replay, found by the streamed case log: the
evidence and context passes finished at 14:53, then nothing -- no engine
call, no log line -- for 82 minutes, until the case was killed 60 s past
its 7,200 s deadline. `affected_context` tested membership with
`relation not in relationships` on a list: 7,998,000 comparisons for 4,000
relationships (profiled, 83% of the time), and a push over a WordPress
plugin attaches tens of thousands. The deadline is checked once per
question, so a single late question could run for minutes, which is also
why the process overran its cancellation allowance.
Membership now goes through a set beside the ordered list, and each
distinct context path is resolved once instead of once per row. Measured:
4,000 relationships 7.4 s -> 0.44 s; 400,000 in 16 s. The control counts
comparisons rather than timing them, and fails on the old code with
7,998,000.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The 13.3 replay on 7cf256b crashed on case 1: `pre-push` ran past its
7,200 s deadline plus the harness's 60 s grace, `subprocess.run` raised
TimeoutExpired, the harness died without running cases 2 and 3, and the
case's captured output died with it. Each case now streams stdout and
stderr to files with INFO logging on; an overrun kills the case's process
group, is recorded as `deadline_overrun` with its size -- the product
failing its cancellation contract -- and the next case still runs.
`--only` reruns selected cases. CLI log lines now carry a timestamp, so a
log can say where the time went.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The real cause of the 13.3 replay wall. Every cache publication called
`_make_room`, which listed and stat'ed every entry to total the cache size,
and a push publishes one answer per question -- 22,125 on a WordPress
plugin. Measured on the live replay cache: one scan 0.68 s at 12,238
entries, ~70 minutes spent scanning by then, ~2.3 hours to finish the
evidence pass. The engine was idle while the Python process sat at 83% CPU;
the 13/31/51-minute portions were the quadratic growing.
The scan is now remembered, keyed by the cache root's modification stamp:
our own publications and evictions update the tally in place, and any
change another process makes moves the stamp and forces a rescan, so the
size limit stays exact under concurrent pushes. Lock stripes are created up
front so their first use never invalidates the tally. Eviction order and
lease handling are unchanged.
Controls: 61 publications (one a same-key replacement) take exactly one
scan and the tally equals a fresh scan; a second writer on the same root
forces a rescan and the limit holds.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
`test_hanging_process_group_killed_inside_shared_allowance` failed about one
run in thirty under load with FileNotFoundError: it checked that
/proc/<pid>/stat existed, the killed process was reaped, and the read then
failed. A reaped process is the outcome the test wants, so it now reads and
treats a missing entry as gone. The engine-stage test had the same race.
150 of 150 runs pass under engine load.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The PHP transfer replay (pre-push 13.3) spent its whole 7,200 s case in the
head side's evidence pass. A probe on one PMPro graph named the cause: not
compute -- context adds ~20% engine time -- but OUTPUT. Each evidence request
listed every method its scope reached, up to 1,845 rows; 200 requests into
the bundled Braintree SDK returned 52,831 rows, 9.4 MB, against 211 rows
without context, over 22,125 requests in a 3 GB container.
`affected_context` only ever attaches a location in a changed file, and
every other row was checked (real file under the root, usable extent) and
discarded. The scan now sends the changed files; the engine itemises those
and folds the rest into one `context_elsewhere` row carrying exactly what
that check reads. With no safe file list the engine itemises everything,
as before.
Equivalence: unit controls over valid, missing-file, no-extent and
outside-root cases, with and without a changed-file row; a real-Joern
control; and on PMPro itself, 2,440 relationships and zero gaps decided
identically from both forms, 52,831 rows down to 3,241 with the most
referenced file as the change. Probe: benchmarks/push/context_cost_probe.py.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Same 500-region PHP run on frozen source: completion 281/406 against 153
(hard cap) and 131 (7.8), no portion killed, per-family findings never
lower, declared target still entailed. Records the 7.9 task, its limits,
and corrects the 7.8 reuse note, which was stated backwards.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The measured-cost run left 99 taint requests unasked to keep time for
arbitration and reported them as `query_failed` -- an engine that could not
answer -- because they were simply missing from the answer. They are now
`query_budget_reserved`, both as a degradation and as each question's outcome
reason, and are no longer double counted when nothing answered.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The 7.8 trace showed the completion loss was a sizer death spiral, not
expensive families: after two kills the budget fell to 3 sink visits and
never recovered, because portions finishing in 25-30 s sat under an assumed
70 s fixed start and read as "no information", so the budget could only
shrink.
- taint.sc streams each answer's own cost (`ms`); the backend carries it as a
reserved `__timing__` list through shards, partitions and reuse.
- The sizer budgets predicted seconds from per-family seconds-per-visit rates
learned from those costs, and measures the fixed start as the wall time the
answers do not explain (median), instead of assuming it.
- A kill names its culprit: the first unanswered id in request order. The ids
behind it never started and are re-queued at once; the culprit is asked
alone after the rest. One that still fails alone after more than a
portion's target is reported as `query_too_expensive`, apart from
`query_failed`.
- Portions grow id by id from the queue head, and a single-portion scan now
goes through the queue too, so a kill there re-asks its remainder.
- Fix: the reuse wrapper wrapped only `run_batch`, so since 7.8 every
portioned query bypassed the graph-reuse cache and its source-root
normalization. `run_batch_once` is wrapped too.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Final end-to-end on the 500-region PHP subject: findings identical at 125 either way, completion
slightly lower under portioning (131 of 406 vs the hard-cap-plus-reserve 153), wall time higher
(2,982 vs 2,338 s). Told straight: portioning does not beat the cap on throughput here; it matches
it on what it finds, and what it buys is kill-safety -- a killed batch keeps its streamed answers
instead of losing them. Reaching the same 125 findings is the evidence that streaming loses nothing
the cap kept.
Completion did not rise because the requeue re-asks a killed portion remainder, and on this subject
those are the expensive output-encoding questions that keep hitting the ceiling in smaller pieces,
spending budget without completing. Telling a transient kill from a too-expensive family is the next
lever, not more portioning. Neither approach is release-ready anyway: 32-38% completion against a
95% gate on development data that qualifies nothing.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Four interacting defects surfaced in sequence when the portioner was measured end to end, each
hidden by the one before: run_batch timing polluted by its own retry, a single-attempt path missing
on the partitioned graph every plugin uses, and killed remainders dropped by a fixed chunk list. All
fixed and unit-tested; the end-to-end completion number is re-measuring on a frozen tree.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The portioner iterated a fixed pre-split list, so a killed portion left its unanswered ids behind:
on the plugin the list ran out at 1,263 s of a 3,600 s budget with those remainders lost, and
completion sat far below what the budget could buy. It is now a work queue. A portion that comes back
short pushes its unanswered ids to the front, to be re-asked at the budget the kill just halved;
`fit` splits an over-budget head, so heads shrink toward single requests, and a single request a
whole ceiling could not answer is dropped rather than looped on forever. A control asserts both the
re-ask and that termination.
This is the last piece of portioning-over-cap: streaming keeps what a kill finished, single-attempt
timing keeps the sizer honest, coverage tells a too-big portion from a slow one, and the queue keeps
working while budget and work remain.
1,425 tests pass, nineteen skipped.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
run_batch_once was wired on the disposable and sharded paths but not the partitioned one -- and a
multi-language repository, which every WordPress plugin is, builds through the partitioned graph. So
the portioner fell back to run_batch there, whose internal split-retry folded into the time it
measured: a portion read 956 s and collapsed the sizer, exactly as before the single-attempt fix.
The fix had no effect on the one subject it was written for.
PartitionGraph now has run_batch_once, dispatching to each sub-graph the same way run_batch does --
the routing is factored into one _dispatch method that takes the per-graph method name. Sub-graphs
are built through the disposable path and already carry run_batch_once.
The lesson, again: a capability added at one construction site is not added at the others. There are
three that build a queryable graph -- disposable, sharded, partitioned -- and the one that mattered
for the measurement was the one missed.
1,424 tests pass, nineteen skipped.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
End-to-end, the adaptive portioner regressed completion -- 64 of 406 against 7.7 hard-cap 153 --
while finding the same defects. The cause: the portioner timed each portion with run_batch, which
splits on failure internally, so one portion folded its own eight-way retry into its number and read
953 s. The sizer fitted a per-visit rate from that and collapsed its budget to 14 visits, spending
the back half of the deadline on tiny portions.
Two portioning layers were fighting. The portioner is now the sole retry layer: it asks
run_batch_once, a single invocation whose kill returns the answers it streamed, so its wall time is
the cost of one attempt. And it observes COVERAGE, not emptiness -- a portion that came back short
of the ids it asked halves the budget, because it was too big for the ceiling; only a whole answer
gives a clean per-visit sample to fit from.
run_batch_once is new on CpgResult, wired on the disposable and sharded paths; a backend without it
falls back to run_batch unchanged. Host controls cover the single-attempt path and the
halve-on-short-answer rule.
1,424 tests pass, nineteen skipped. The end-to-end plugin re-measure on a frozen tree is the open
item; the mechanism (a kill keeps what finished) and the sizer logic are unit-verified.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The helper took a destination from the caller and rm -rf it. Re-running it into the same path while
a container had that path mounted deleted taint.sc out from under a live scan, and every request
after printed no such cpg query -- the live-mount hazard the helper exists to prevent, in a new
form. It now mints base/frozen-<commit>-<pid>-<n> itself and prints it, so reuse is impossible by
construction and the caller captures the printed path.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Measured on the plugin, first run of the adaptive sizer. A 600-visit portion answered in 184 s and
the budget was correctly cut to 262. The next portion, 199 visits, answered in 73 s -- three seconds
of work above the 70 s fixed start -- and 3 s over 199 visits read as 0.015 s each, so the budget
grew thirteen-fold to 3,404 visits. The ceiling killed that portion. The harvest kept the 13
answers it had streamed, which is the mechanism working; the sizing was not.
Two rules. A sample with less than ten seconds of work above the fixed start is not extrapolated
from, because it carries no information about cost per visit. And a budget may double per portion,
never more: the sizing may still be wrong, but by at most one factor of two, which a kill undoes.
1,423 tests pass, nineteen skipped; the sizer control asserts both rules.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The warning once read 22126 of 22125 requests answered in parts, because the census rides the
answer under its own key and was counted as a request. An instrument that miscounts by one casts
doubt on every number beside it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Every long run mounts src into a container, and mounting the working tree is a live mount: edits
made during a multi-hour run are picked up mid-run, and a measurement whose code changed underneath
it is not a measurement of anything. A five-hour PHP transfer replay was lost that way today.
The export has to be checked too. git archive piped into tar once produced a tree missing the whole
ontology -- every tracked file after groovy alphabetically -- and reported success, because tar
status in a pipeline was never read. Every scan on that tree died at its first fact load and looked
like a slow run for fourteen minutes.
So: archive to a file, extract, count against git, and refuse an export without the ontology.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The joern-gated check passes on the real engine: the taint query streams one line per request, and
the comment-sink and literal-destination controls hold under the streamed form. Two defects the work
exposed are recorded as fixed -- the unportioned evidence pass, and a harvest that read a timed-out
pipe before killing its writer -- and one bypass is recorded as a limit rather than fixed: the reuse
layer re-query of missing ids calls the backend batch directly around the sink-weighted portioning.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The evidence pass bypasses the sink-weighted portioning by design: it is the pass that PRODUCES the
weights, and per request it is cheap, 17,700 answered in 44 s. It was still one all-or-nothing batch
protected only by the ceiling. On a plugin push transaction that batch is 22,125 requests, and under
the CPU and memory contention of concurrent test runs it exceeded the ceiling and cost the whole case.
It now runs in portions of 4,000 requests, roughly ten seconds of work behind one JVM start, with
answers merged and the census kept from the first portion that carried it. Streaming keeps what a
killed portion finished; portioning keeps that kill from being the entire pass.
Task 13.3 records the attempt that exposed this honestly: the case deadline was exhausted with no
artifact, and the failure was mine twice over -- the container mounts src live and the backend was
edited mid-run, and the evidence pass was unportioned. Neither is the algorithm. The rerun uses a
frozen export of the committed source with per-invocation logging.
1,423 tests pass, nineteen skipped; mypy clean.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A hard per-invocation ceiling is all-or-nothing. A batch killed at 301 s lost every answer it had
already computed, and the retry paid a fresh JVM start per part to ask them all again: measured, 116
requests killed at 300 s, then eight restarts carrying the same everywhere-sinks in every part.
Three coordinated changes make portioning, not the cap, the thing that governs cost.
The taint query STREAMS. The census first, then one line per request as it is answered, flushed as
it goes, inside the unchanged fence. The other queries do not stream and need not; the driver
reads both forms, and a legacy single document parses exactly as before.
The driver HARVESTS a kill. The process group is terminated first -- a first cut read the pipe
before killing and a pipe whose writer is alive never reaches end-of-file; the suite own
hanging-engine control sat on that read for twenty-eight minutes -- then the pipe is drained and
every complete line becomes an answer, marked partial. The retry asks only the remainder.
Portion sizing ADAPTS. After each portion the observed seconds per sink visit, less the fixed
start, resizes the rest toward a 120 s target; a portion that did not answer halves the budget
rather than being fitted, because its true cost is unknown and above the ceiling. A portion cut
down to fit puts its remainder back at the front of the queue.
A census LINE carries nothing but the census. A first cut misread a legacy document that carried the
census beside its answers as a census line and dropped every answer it sat beside; a control caught
it. Task 7.8 in contributor-scan.
Verified: 1,423 host tests pass, nineteen skipped, including new controls for the assembler in both
forms and under a kill, for a retry that asks only the remainder, and for the sizer. Not yet
verified: the joern-gated streaming check and the plugin measurement, which wait for the transfer
replay to free the machine and are stated as open in the task.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Detection is not transfer, and the status says so: the comparison run is what decides M3. But the
target that task 13.2 could not reach for months is now found, and the reason is worth keeping where
the next reader will see it -- nothing about detection changed, the population became affordable.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
classes/class.memberorder.php:936:getMemberOrderByCode, injection. A value concatenated into SQL with
no prepare, which is the declared M3 target for PHP, found for the first time.
Nothing about detection changed. The population became affordable: tier-zero pruning scoped to its
partition, batches budgeted by the sink work they imply, a per-query timeout no longer cancelling the
scan, and a reserve kept for arbitration. Before those this population completed zero questions at
every budget tried.
At 500 regions with a 3,600 s deadline: 271 of 406 questions completed, 213 findings, 2,950 s, and
eight findings in the target file including the target itself.
The earlier bounded attempt failed for a reason no budget increase could fix -- 25 regions of 4,461
is 0.6 percent of the population and the target was never asked about -- so this sizes the replay
before spending hours on it. One scan at this budget is 2,950 s, a push transaction is two scans plus
discovery and preparation, so the three declared comparisons are roughly five and a half hours.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A census that reports what it found cannot say whether a defect it missed was examined and cleared or
never looked at, and that is the difference between a negative and a silence. It is also how a
bounded run budget gets sized without guessing: task 13.3 needs to know at what population the
declared PHP target enters scope, and asking the record is cheaper than binary-searching with
forty-minute scans.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
PMPro at 25 regions, which completed before this work and still does: 27 questions completed and 19
findings before and after, injection 1 and output encoding 18 in both. The batching changes what a
large population can afford and nothing about what a small one answers.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
On the 500-region population where every earlier run produced nothing: 153 of 406 questions completed
and 125 findings, inside the deadline. 380 requests carrying 5,807 sink visits split into 11 batches,
each answering in 80 to 197 s, none near the 300 s ceiling. Every family that can speak on this
subject now does, injection included, which is the one holding a true positive here.
Four defects, each hidden by the one before it, and three of them one rule.
Batches sized by request count, when cost is per matched sink.
Packing heaviest-first, which starved injection of all 114 of its questions.
Any per-query timeout cancelling the whole scan, abandoning batches with 33 minutes left.
No budget kept for arbitration, which removing that cancellation exposed: rows are not findings.
The reserve had to be checked per BATCH. Checked between query kinds it changed nothing at all,
because one kind is a single loop iteration that runs every one of its batches inside it.
What it does not claim: 253 of 406 questions still went unanswered and one batch still records a
timeout; the raw finding count is dominated by output encoding, the family already measured to be
imprecise on PHP; and the reserve share of a fifth is a first value, not a tuned one.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The reserve was computed once and checked between kinds, and one kind is a single iteration that runs
every one of its batches inside it. So taint spent the whole budget and the reserve only stopped the
cheap kinds queued behind it: 0 of 406 questions completed, 2,435 s of a 2,700 s deadline in queries,
identical to having no reserve at all.
The granularity of a budget check has to match the granularity of the work it guards. It is now
checked before each batch, and the batches left unasked are logged rather than silently dropped.
1,420 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Querying fills rows. Arbitration turns them into findings and the outcomes are assembled from that,
and neither costs the engine anything -- so a scan that spends its whole budget on queries reports
nothing at all, however many of them answered.
Found by accident, which is the honest way to tell it. A per-query timeout used to cancel the whole
scan. Removing that cancellation was right on its own terms and made the measured result worse: the
query phase then ran the full 2,700 s deadline to exhaustion and completed 0 of 406 questions, where
the cascade had stopped it at 958 s and left time to arbitrate 78 findings. The cascade had been an
accidental budget guard. This is the deliberate one.
The query loop now stops issuing batches with a reserve in hand: a fifth of what was left when
querying started, a sixty-second floor, and never more than half. The cap is not decoration -- held
back without it the floor swallowed a whole small budget and the query loop never ran at all, which a
control caught. A scan with a fifth of a second to spend must still spend it.
What the loop skipped is recorded as query_budget_reserved with the kind and the request count, not
left to look like a query that failed.
1,420 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Two different timeouts reached the same handler and got the same consequence. If the SCAN deadline is
gone, cancellation is right: stop everything promptly, which is what the push two-second gate
measures. If a single query merely exceeded its own per-query ceiling, the scan may have most of its
budget left and every other batch is independent and cheaper -- and cancelling took them all.
Measured on Paid Memberships Pro: one batch of 116 requests hit a 300 s ceiling 734 s into a 2,700 s
scan, and the four batches behind it returned in 0.0 s unanswered with thirty-three minutes to
spare. The handler already distinguished the two cases in the reason it recorded, deadline_exhausted
against process_timeout, and then set the cancel deadline in both.
Fourth instance this week of one rule, now written in the design: a failure is owned by the narrowest
scope that caused it.
1,420 tests pass, eighteen skipped, including the cancellation gate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A taint batch costs what its matched sink NODES cost, because reachableByFlows is asked per sink.
Measured at a fixed 100 requests on one plugin graph: SQL injection 147 s with six declared sinks,
path 194 s with sixteen, output encoding 704 s with seven. The declared count does not predict cost
and the request count does not predict cost.
So batches are budgeted in sink visits rather than requests, using the count the evidence pass
already produced: a family whose sinks are rare rides in one batch, and a family whose sinks are
everywhere is split until it fits. A request over the budget on its own gets its own batch rather
than being dropped, because the scan job is to report that it is expensive, not to decide it is
unaskable.
Measured on Paid Memberships Pro at 500 regions, where every earlier run produced nothing at all:
380 arbitration requests carrying 5,807 sink visits split into 12 batches, every one answered,
heaviest 103 s and none near the 300 s ceiling. 144 findings where there were none.
The order is the ranker and must stay the ranker. A first attempt packed heaviest-first, which packs
better and analyses worse: the expensive output-encoding batches ran first, spent the deadline, and
left injection with 114 questions asked and none answered -- the one family holding a true positive
on that subject. A deadline always cuts a tail, and the tail it should cut is the one the ranker put
last.
1,420 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Open since September and re-measured nine times. The cost of a taint batch is the number of CALL
NODES a family sink names match in the repository. Not the request count, not the declared sink
count, not any request parameter. reachableByFlows is asked per matched sink, and PHP
soutput-encoding sinks are echo and print, which a WordPress plugin writes thousands of times.
Three controlled sweeps on one graph, built once.
Batch size exonerated: 400 requests answer in 170 s, and cost per request FALLS with size because
roughly 70 s is fixed JVM start and graph load.
Request parameters exonerated: at 200 requests the full production shape, call depth 3 with
parameter sources, costs 221 s against a 150 s baseline.
The family is the variable: at the same 100 requests, SQL injection 147 s with six declared sinks,
path 194 s with sixteen, output encoding 704 s with seven.
And the driver named its own expensive step once it was asked to: 2,435 evidence requests in 45.5 s,
then 315 arbitration requests unanswered at 1,200 s, then a split abandoned for want of time.
It survived since September because every reproduction outside the driver carried the SQL sinks --
the family the declared target belongs to, and the one family that is cheap on this subject. Nine
hypotheses were eliminated by re-measuring the cheap case.
Splitting never rescued it because a failed batch is split by REQUEST COUNT and each part carries the
same everywhere-sinks. contributor-scan 7.7 carries the fix: budget a batch by the sink nodes it
implies, using the count the evidence pass already computes, and report a family whose sinks are
everywhere as expensive rather than as failed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Nine hypotheses about the PHP query cost have now been eliminated by rebuilding the request outside
the driver: batch size, call depth, parameter sources, hook table, argument length, sharding,
dataflow cost, the malformed-CPG bug and the engine version. Every one of them answered in under
four minutes while the driver spent whatever ceiling it was given.
The reason the guessing continued is that a scan issues several invocations and the record could not
say which of them burned the budget. It now logs the query kind, the request count, the elapsed time
and whether it answered, for the whole batch and for each part of a split retry, including the part
where a split is abandoned for want of time.
This is task 16 applied to the backend rather than to a harness: an instrument that cannot name its
own expensive step turns a measurement into a guess, and this one has been a guess since September.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Four census runs on this subject spent whatever query ceiling they were given: 363 s of 300, 1,255 s
of 1,200, 1,874 s of 1,800, at question counts of 2,526 and 406. Cost that tracks the budget rather
than the work is not per-question cost, and tier-zero pruning cutting the count six-fold changed
nothing. A census runs one batch, so no census could answer whether the batch itself scales.
One graph, built once, five batch sizes, every one answered:
25 requests 71.4 s 2.86 s each
50 requests 76.0 s 1.52 s each
100 requests 155.4 s 1.55 s each
200 requests 166.7 s 0.83 s each
400 requests 169.7 s 0.42 s each
Sublinear, and roughly 70 s of it is fixed JVM start and graph load. Four hundred dataflow requests
answer in 170 seconds on the graph production cannot finish 406 on.
So the difference is in the request PARAMETERS. The probe sent callDepth=0 and no parameter sources;
production sends callDepth=3 with parameter sources on every entry point, plus a hook table, a
dispatch list and context evidence. Task 13.2 recorded in September that every reproduction outside
the push path answered in under a minute and the difference was not known. Size is now removed as a
candidate and a four-variant probe is running to isolate the rest.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The mechanism already existed. A question whose evidence vector says no sink of its family is in
reach cannot produce a flow, and tier-zero pruning defers it. That was switched off for the whole
scan whenever any of twelve degradation reasons appeared, and a multi-language repository always
carries two of them, so the measured plugin arbitrated every question it could ask. Third instance
this week of one blast radius, and the invariant written into the design on Monday named it.
A prune blocker that names a partition now withholds pruning from that partition questions and from
nobody else. cross_partition_semantics_unresolved leaves the blocker set entirely, for the reason it
left the integrity gaps: a PHP family sink is PHP, so a second language cannot be why its vector saw
none.
Measured on Paid Memberships Pro. At 25 regions arbitration falls from 127 questions to 34 with 93
recorded deferred as tier_zero, and 19 findings before and after, identical family by family. At 500
it falls from 2,526 to 406 with 2,120 deferred.
It is not sufficient, and that is the useful half. The 500-region run still fails in the same way at
the same cost, because 406 questions at roughly one and a half seconds each still exceed the
300-second per-batch ceiling. This lever moved the volume; the ceiling is a separate limit.
The census now records what a scan skipped and why, per family, so a cheaper scan and a narrower one
stop looking identical: a family whose every question was pruned reports no sink of this family in
reach rather than region offered, no question asked.
1,420 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
NodeGoat models 18 of 175 distinct call names, covering 12 percent of 1,185 calls. That share is not
a score: most of what a program calls is not security-relevant, and a high number would mean the
tables were too loose. It is a baseline to move deliberately.
The top product candidate is res.render, 23 calls with five in a method that reads untrusted input.
Task 14.3 reasoned its way to that same gap by hand and declined to model it as a flow sink; the
instrument found it in one run, which is the validation that matters. It also surfaced Math.random in
the user data-access module, and weak randomness is a weakness class this ontology has no family for
at all -- so no family census could have found it, because the family list is itself the blind spot.
That is the kind of gap only a coverage instrument finds.
It had to be taught what is not an API. The first run ranked _tmp_3.push and __ecma.Array.factory
first, which is how the frontend lowers a destructuring and an array literal, and a Cypress suite and
a Gruntfile held four of the next five. Both are separated now, because noise at the top of a
worklist is how a worklist stops being read.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Five of seven families established nothing in the family census, and every one failed for an ontology
reason with the flow engine ready and nothing to aim at. Each was found by investigating one family
at a time over days. vocabulary.sc asks it once: census the calls, subtract everything any fact table
names, rank the remainder by how often it is called and whether it sits in a method that reads
untrusted input. It needs no dataflow, because the structural pass already walks every call.
The harness reads its token set from the SPECS rather than from the TOML, so it measures what the
analyser actually consults: a fact the loader drops is not part of the ontology however neatly it is
written. Ontology sizes today are 152 tokens for JavaScript, 167 for Python, 84 for PHP.
Both carry the constraint the maintainer set: the output is a candidate list, never an ontology. A
name becomes a fact only when someone resolves it to a library API, states the weakness class and
records provenance, because adding a name because a subject called it is how a general-purpose tool
becomes a fixture of its own benchmark.
Also records what the bounded PHP transfer attempt found. At 25 regions the first case now runs to
completion and writes a full artifact in 1,706 s, where at 100 it had exhausted a 2,400 s deadline
with nothing. But 25 of 4,461 regions is 0.6 percent of the population and the declared target was
not among them, so the question was never asked. Task 13.3 therefore depends on 7.3, skipping
arbitration where the evidence says there is nothing to find: raising the budget is already measured
to fail, and making most of the budget unnecessary has not been tried.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Named by a week in which three separate causes looked identical as one count of unresolved. The
census reported 19 findings and 0 completed questions side by side without complaint, a stale image
measured the engine a bug had been diagnosed against, and a partition fix looked applied while an
unscoped copy of the same gap kept demoting everything.
Every harness record should carry the counts that disambiguate its failure modes, the degradations
whole rather than by reason, and the engine and ruleset identities actually used; and each harness
should assert its own consistency, because findings above zero with completion at zero is a
contradiction an instrument can notice faster than a person.
This was missed from the previous commit by an escaping error in the edit, not by intent.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The source, sink, sanitizer, operation, discharger and layout rows are not a pattern list. They are an
ontology of program elements and their security relations, and nearly all of this tool accuracy lives
there: the family census found five of seven families establishing nothing, every one for an ontology
reason with the flow engine ready and nothing to aim at.
The hook is general-purpose, not application-specific. The ontology describes libraries, frameworks
and language constructs; it does not describe repositories. An entry names an API and is justified by
what that API does, carries provenance and a weakness class, and may not encode a path or identifier
peculiar to one project -- a plugin bundled library tree was measured and declined on exactly that
ground. A measured gap is a candidate, never an entry: accretion by subject is how a general tool
becomes a fixture of its benchmark.
contributor-scan owns the ontology, so its design carries that statement and a new group 7 makes
growth measurable rather than anecdotal: report what a repository calls that nothing models, key
sanitizers to the weakness they prevent, skip arbitration where the evidence says there is nothing to
find, share one matcher across the queries, require a degradation to declare its scope, and commit
golden findings so a silent change cannot pass.
pre-push-safety-net carries the general-purpose test for its own reviews, and a task for harnesses
that report the cause of their own failure -- named by a week in which three separate causes looked
identical as one count.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Task 13.4 found that a second language in a repository cost every question its completion, and so
cost every finding its admission, on the declared v0.1 target: a WordPress plugin is PHP plus its
admin JavaScript. Neither subject that passes today could reveal it, because both hold exactly one
language. A population of single-language repositories would qualify a capability that fails on the
product own target.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
_blocking_degradations already exempted declared exclusions. It now also ignores a gap another
language partition raised, because a JavaScript file missing from the JavaScript graph cannot make a
PHP answer wrong. Treating it as though it could left a WordPress plugin -- PHP plus its admin
JavaScript, which is every plugin -- unable to admit a finding it had already established.
A gap that names no partition still reaches everything, and so does every gap when the claim own
language is unknown, because unknown scope is not evidence of independence. The control asserts all
four directions.
With this and the scan fixes, task 13.4 is verified on the subject that motivated it: PMPro completes
120 of 127 questions where it completed none, its 19 findings unchanged, and the seven still
unresolved are the JavaScript partition own questions, which is the correct answer.
1,420 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The scoping fix looked applied and was not. The same census gap is raised in TWO passes, arbitration
and evidence, and only the first carried the partition it came from, so an unscoped copy kept
demoting every question in the repository. One key in one dict.
What found it was the record, not the code: the census now prints each outcome reason rather than
only its status, and three separate causes had been looking identical as a bare count of unresolved.
Measured on Paid Memberships Pro at 25 regions: questions completed goes from 0 to 120 of 127, with
its 19 findings unchanged. The seven still unresolved are the JavaScript partition own questions,
which is correct, because the four missing files are its files and its answers are the ones that
could be wrong.
Still open, and recorded rather than implied: the push policy _blocking_degradations exempts declared
exclusions but does not yet respect a partition, and a post-hoc diagnostic copy carries no language,
so a JavaScript gap would still block admission of a PHP finding.
1,419 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A missing JavaScript file cannot make a PHP answer wrong. A census gap now carries the partition it
was raised for, and reaches only that partition questions. On the measured plugin four JavaScript
files -- a webpack build output, a minified select2, a build config and one source file -- had taken
the admissibility of 4,425 PHP regions with them.
Two controls exercise the real boundaries rather than injecting a reason, and both were confirmed to
FAIL with the old classification restored, so they guard what they claim to.
And the rule is now in the design, because this was the fifth instance found and fixed one at a time.
Two invariants, eleven instances between them:
A degradation is owned by the narrowest scope that caused it. Vendor exclusion as a global veto,
line ambiguity inherited by every question, an empty scope blocking a sibling comparison, a language
boundary demoting every answer, a census gap demoting every answer. The fourth and fifth cost the
declared v0.1 target its entire admissibility, and neither passing subject could reveal it because
both hold one language.
A token matches the thing it names, not text that contains it. gets inside fgets, resolve inside
resolveUrl, user inside users, a sink word inside a comment, a dotted setting on its last segment,
a flow query reading the receiver.
Each carries a review test: a new degradation must state the scope it reaches and justify demoting
that far, and a text match must be bounded AND confined to the part of the node that names the callee.
1,419 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Membership of GRAPH_INTEGRITY_GAPS should turn on whether a gap can make a reported answer WRONG,
not on whether it cost coverage. An unparsed file may hold the sanitizer that would have cleared a
flow this scan reports, so it belongs there. A flow this tool cannot follow from one language into
another can hide a finding and never invent one, because a PHP sink is sanitised in PHP.
cross_partition_semantics_unresolved therefore moves to DECLARED_EXCLUSION_GAPS. It touches exactly
two places, both of which already honour the distinction: the outcome demotion in scan.py, and
_blocking_degradations in the push policy, whose docstring already says a declared exclusion is not
one of them. The reason stays published in the scope unresolved boundaries either way, so the limit
is still reported; the class decides only whether every answer in the repository is demoted.
This corrects the task note written earlier today, which said not to reclassify on the grounds that
it would claim cross-language flows are out of scope rather than unresolved. That was a misreading
of the class.
PMPro still completes none of its questions, and the measurement now names why: the JavaScript
partition expected 36 files and holds 32, so partition_file_census_incomplete fires and is still an
integrity gap. The four absent files are js/blocks/blocks.build.js, js/jsapi.js, js/select2.min.js
and webpack.config.js -- a build output, a source file, a minified third-party library and a build
config. A bundled select2 costs 4,425 PHP regions their admissibility. Same over-broad blast
radius, third instance of one principle in this thread, and the fix direction is recorded.
The census harness now records each degradation whole rather than only its reason, because a
partition census that reports files missing carries WHICH files and without them the gap is a label
rather than a lead.
1,417 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
No PHP question is ever marked completed, and PHP was never the variable. partitions.py raises
cross_partition_semantics_unresolved whenever more than one language partition exists, that reason
is a member of GRAPH_INTEGRITY_GAPS, and a graph-integrity gap demotes every completed outcome in
the scan. Admission requires a completed outcome.
The evidence is one column wide. NodeGoat is JavaScript alone and completes 299 of 299. VAmPI is
Python alone and completes 35 of 35. PMPro is PHP with 36 regions of bundled JavaScript and
completes 0 of 127 and 0 of 520 while establishing 19 and 43 findings. NodeGoat own degradation,
vendor_semantics_unresolved, is a DECLARED exclusion rather than an integrity gap, which is exactly
why it does not demote.
This lands on the declared v0.1 target. A WordPress plugin is PHP plus its admin JavaScript, so
every one of them is multi-language, so on the target this tool was built for it can detect defects
and never admit one. Neither subject that currently passes could reveal it, which is why a census
across a third subject was worth running.
Recording the boundary is right: a flow from PHP into JavaScript is semantics the partitioned graph
cannot follow. Demoting every question is not, including one wholly inside a single PHP file. Task
12.3 already fixed this bug class one level down by making a gap question-owned so it could not
block another question comparison; the same principle applies to a partition boundary. Reclassifying
the reason as a declared exclusion is explicitly the wrong fix, because that would claim
cross-language flows are out of scope when they are unresolved.
Task 13.4 rewritten with the diagnosis, and the milestone amendment now ranks it with M2 rather
than behind it: a narrow JavaScript-only release can proceed without it, the WordPress target
cannot proceed at all.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The replay hardcoded 500 regions, which on a 673-file PHP plugin asks 2,526 questions and fits no
ceiling this project sets. The budget is now an argument, defaulting to 500 so every earlier
measurement keeps its shape.
It bounds the RUN, never the repository: the artifact census still reports what was left
unexamined, and a bounded population establishes transfer rather than coverage. Task 13.3.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Four decisions, recorded after the family detection census and the PHP correction. They change
sequence and declared scope; they change no acceptance threshold and no owner boundary.
Close M3 at a scale whose questions complete, before anything else. The not-established PHP record
came from two instrument defects rather than from the analyser, and bounded to 25 and 100 regions
PMPro completes with no timeout and establishes a true-positive SQL injection. Tasks 13.3 and 13.4
carry the replay and the question-completion gap, the second because admission reads outcomes and
M4 measures completion at 95 percent.
Reserve the untouched M4 population now, in parallel. M4 already permits preparation before the
frozen candidate exists and it is the longest lead-time item on the path. Task 15.1.
M2 is the only engineering item left and cannot be scoped away: a changed-code transaction spends
its whole budget in the build, and meeting the latency gate by analysing less is explicitly not
the same result.
Declare the v1 advisory scope as the families whose claims can be ADMITTED, which requires the
entailed rung: injection, untrusted destination and configuration. Access control is excluded
because on an application that splits route from data access the obligation is carried and a
carried obligation corroborates by design, so the family would ship as silence. Task 15.2.
The risk the amendment does not remove is stated with it: every number so far comes from subjects
used to build the analyser, one week of examining two of them surfaced seven false positives, and
the first independent evaluation should be expected to fail.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
PHP is not silent. Paid Memberships Pro establishes findings, including a true positive, and
this project's own instrument is why it looked otherwise.
The census harness ordered its verdict on completion before findings, so a family with eighteen
findings and no completed question reported "asked but no question completed". Findings are now
checked first: a question-outcome count is bookkeeping and cannot hide the thing being censused.
NodeGoat and VAmPI were never affected, because they complete every question and their verdicts
reached the findings check. The scan summary now also carries the total findings and the outcome
status counts, so the two numbers can be seen disagreeing instead of one silently winning.
The second mistake was mine rather than the harness's: only the 500-region runs had been looked
at, which are the ones that time out.
What PHP actually detects on PMPro. At 25 regions, 19 findings in seven minutes with no timeout;
at 100 regions, 43 findings in twelve minutes with no timeout. Among them adminpages/
discountcodes.php:749, where sanitize_text_field($_REQUEST["s"]) reaches
$wpdb->get_results($sqlQuery, OBJECT) through `$sqlQuery .= "WHERE code LIKE '%$s%' "` with no
prepare. That is the class of CVE-2023-23488, which php.toml's own comment names as the reason
those WordPress sanitizers are kept out of the sanitizer list. The prediction and the measurement
agree.
The precision cost beside it is measured too: output encoding reports eighteen findings at 25
regions and the three sampled are all properly escaped, two with esc_attr and one coerced with
intval. Those escapers are absent from a sanitizer list that is flat across families, because
none of them stops SQL injection. Keeping the injection finding costs this family its precision.
The scale limit is roughly one to two seconds per question. 25 and 100 regions complete cleanly;
500 regions asks 2,526 questions, needs more than the 1,800-second ceiling and is killed.
Four things are measured NOT to be the problem: the malformed-CPG bug (fixed in 4.0.625, verified
on the real trigger file), the dataflow overlay (42 s on the full 637-file graph), the engine
version (4.0.625 and 4.0.631 indistinguishable), and the bundled Stripe, Braintree and Twocheckout
SDKs under includes/lib that the vendored list does not cover -- that was my hypothesis for the
cost and the overlay measures 42 s with them and 37 s without, so it is refuted.
What remains is bookkeeping, not detection: questions_completed is zero in every PHP run including
the ones that find things, with cross_partition_semantics_unresolved and
partition_file_census_incomplete present throughout. The admission policy reads those outcomes.
1,417 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The graph bug is fixed and was never what these runs were hitting. joernio/joern#6281, this
project's own report, is a PHP `global` inside a closure giving a node two AST parents, which made
the dataflow overlay throw and cost the whole graph. Filed against 4.0.623, fixed by #6270 in
v4.0.625, which the image has pinned since before any measurement recorded under task 13.2.
Verified today on the real trigger rather than a transcription of the repro, which does not
reproduce: PMPro's includes/fields.php builds and applies the default overlays cleanly on 4.0.625
and on 4.0.631, files=2 methods=158, no ReachingDefPass assertion.
What remains is runtime. A taint batch of 2,526 questions over the 673-file checkout does not
complete in thirty minutes. Raising the query ceiling six-fold, from 300 to 1,800 seconds, changed
nothing but how long the failure took: query time went from 363 s to 1,874 s, and zero questions
completed in both. The eight-way split retry did not rescue it either, finishing its parts in
about three minutes against a deadline with 3,300 seconds still on it, which is worth its own
look.
The engine pin stays at 4.0.625. An image on 4.0.631, today's release, was built and measured and
is indistinguishable on this subject: 627.7 s against 619.1 s, 363.4 s against 367.3 s of query
time, zero completed in both. There is no demonstrated benefit to weigh against re-verifying every
JavaScript and Python measurement on a new engine. Should that change, the bump is three values
rather than a port: the upstream AstGenRunner.scala is byte-identical in 4.0.631 and its patch
anchor still appears exactly once, so only the jar path, the jar checksum and the source URL tag
move. Both are recorded.
Also closes task 14.5. Neither prototype pollution nor deserialization has a subject that can
answer, each for its own reason. Deserialization's mechanism exists on exactly one available
subject and that subject cannot be queried at all. Prototype pollution has no available subject
containing the mechanism, so its ability to answer is established on a control instead.
1,417 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Across four WordPress plugins and 1,164 PHP files there is exactly one deserialization site, and
it is spelled maybe_unserialize. The bare token never reached it, and correctly so: the sink
matcher is word-bounded and the character before unserialize inside maybe_unserialize is a word
character. A token a repository never spells is a family that can never answer, which is what the
2026-09-19 census recorded for this one. maybe_unserialize is a real entry point rather than a
safe wrapper: it calls unserialize whenever the string looks serialized.
Also adds the prototype control. That family had never established anything on any subject, and
its only census report was the word `set` inside a comment. No repository available to this
project contains the mechanism: the only first-party matches on the Node subject are
app.set("view engine", "html") and its sibling, Express settings whose arguments are literals. So
the control demonstrates what the family can do, on a merge of the request body into an object,
and asserts the Express setting is not reported alongside it.
Corrects the PHP record under task 13.2. joernio/joern#6281, this project's own report, is a PHP
`global` inside a closure producing a node with two AST parents, which made the dataflow overlay
throw and cost the whole graph. It was filed against 4.0.623 and fixed by #6270 in v4.0.625,
which the image has pinned since before any measurement recorded here. Verified today on the real
trigger rather than on a transcription of the repro: PMPro's includes/fields.php, the file the
original bisection found, builds and applies the default overlays cleanly on 4.0.625 and on
4.0.631 alike. What remains under that task is a runtime failure and must not be re-diagnosed as
the graph bug.
1,417 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
NodeGoat's server.js configures a permissive session at line 78 and disables template
auto-escaping at line 135, both inside the same function. The arbiter returned one verdict per
question, so the escaping defect that task 14.3 added was detected on every run and reported on
none: which of two real defects a contributor saw depended on where in the file it sat.
Configuration now returns one verdict per setting whose value the model could read. The pipeline
already emitted one finding per entailed answer, so this arbiter was the only place still folding
them to one.
Three boundaries are drawn rather than assumed.
- A setting whose value is COMPUTED stays one verdict per region. That band is a question for the
judge rather than a finding, and returning every undecided setting would multiply model calls
for claims the graph did not establish.
- A call carrying two permissive literals is one finding. cors({origin: true, credentials: true})
is one misconfiguration, not two.
- Config verdicts now carry a path:line:function location. Without one they shared the region's
identity and the deduplicator kept a single finding, so the plural answer collapsed straight
back to one. That is the same gap dominance carried until its findings stopped arriving with no
line at all.
Dominance is left answering once per region, and that is settled rather than pending: its claim is
that a region contradicts itself, so the region is the unit and a second verdict would restate the
same inconsistency. The pipeline now says so where it used to call this a known gap.
Measured: NodeGoat's config_secrets goes from one finding to two, server.js:78 for the session and
server.js:135 for the disabled escaping, with the applied marked.setOptions fix between them
staying silent. VAmPI is unchanged and no other family moved.
1,417 tests pass, seventeen skipped; thirteen Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Two of NodeGoat's three corroborated claims were correct code. `const { userId } = req.session`
lowers to a temporary and then a field read, so nothing in the graph spells session.userId by the
time the value reaches dao.getByUserId(userId, ...), and the constraint the handler really applies
was invisible.
The arbiter now walks back three hops over the method's own assignments from the names the
operation is GIVEN, and req.session joins the identity sources. The token is only safe because of
that requirement: what discharges an object-level obligation is that the value passed to the
operation comes from the authenticated context, not that a session is mentioned nearby. A handler
that reads the session for logging and keys its query on a path parameter is the bug this family
exists for, and it stays unguarded.
Two exclusions the measurement forced, each caught by a page whose whole purpose is the leak it
demonstrates.
- A callback is not a constraint. Taking identifiers from the whole argument subtree credited
NodeGoat's memo listing with a discharge, because memosDAO.getAllMemos((err, docs) => ... userId
...) reads every memo in the collection and only RENDERS the session's user. Method references
are excluded from the argument names.
- An assignment is not a dominating guard. The session read dominates the query, so that clause
credited it instead and the memo listing went quiet again. An assignment is a read of the
authenticated context, not a check applied to it, so operator calls are excluded there -- the
same exclusion the sink matcher needed. The leak stays reported.
On NodeGoat displayContributions and displayProfile go silent, the allocations insecure direct
object reference, both benefits pages and the memo listing stay, and the login and signup handlers
become honestly unguarded rather than accidentally cleared by a callback's text. Six corroborated
where there were seven, and a better six. VAmPI is unchanged at two entailed findings.
The sibling scope is decided by measurement and kept at the file. On NodeGoat the region's own
file entails two handlers, both public routes and both false, and no true defect at all. Every
handler in the application entails six: four true, and the same two public routes. Both scopes
carry the same false positives, because neither can tell a public route from a protected one --
a semantic question that is already the corroborated band's residual. With the carried rung capped
the scope changes no verdict, only the witness, so widening it would buy a repository-wide walk
per question and no precision. A public-route discriminator would change that; the file scope is
not what holds this family back.
1,415 tests pass, sixteen skipped; thirteen Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
NodeGoat's documented insecure direct object reference reads with
allocationsCol.find(searchCriteria()), and `find` was absent from an operation table holding
findOne, findById, findAll and findByPk. The obligation was missing at the commonest read there
is, so that defect raised nothing at any point in the pipeline.
It was absent for a good reason, and the reason is the argument, not the name. Measured on
wp-statistics, a WordPress plugin that ships jQuery: its assets hold 88 calls named find, of
which 70 pass a jQuery selector, 13 pass a predicate to Array.prototype.find and 5 pass a
selector built at run time by a template literal or a concatenation. None is a collection read.
So the obligation is claimed only where the first argument has the shape of a query document.
The operation fact states which argument with query_document_arg, and the arbiter that can see
argument shapes decides it: no arguments at all, an object literal, or a builder call are
claimed; a method reference, a literal, an operator call and a bare identifier are not. Each
exclusion is one of the measured classes above rather than a guess.
Zero of those 88 calls became an obligated operation, and the zero is measured rather than
empty: the same run's census reports 60 files and 4,552 methods, and the files holding those
calls are in it.
On NodeGoat all three find calls are collection reads and all three are claimed, which raises
the obligation at four more handlers -- the allocations handler that takes userId from req.params
on an authenticated route, the benefits pages whose isAdmin guard is commented out in the route
table, and the memo listing. Seven of fifteen handlers now reach a verdict, all corroborated,
none reported without a judge, and the census is unchanged on both subjects.
A bare identifier is deliberately not claimed: col.find(query) and arr.find(pred) are the same
shape and neither repository measured holds one, so this is a stated false negative rather than
an unmeasured false-positive risk.
1,415 tests pass, sixteen skipped; twelve Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The census measured this family completing 35 questions on NodeGoat with no obligation raised
anywhere: every modelled data operation lives in app/data/*-dao.js, none in a route handler
file, and operations were scoped to the region's own file. The family had nothing to be right or
wrong about.
The obligation now follows the one named call the handler writes, out of the region's file and
no further. How it is followed is the point. jssrc2cpg does not resolve
allocationsDAO.getByUserIdAndThreshold(...) to the DAO body: the callee is an external stub with
no code. That stub's fullName records where the member was written, so the implementation is
reached by the graph's own provenance rather than by a repository-wide name guess, which is the
false positive the file scope was closed against. A control asserts it: a third module defining
a member of the same name that nothing imports is not attributed.
A row is attributed to the region's DECLARED function, which is what lets a question about
displayAllocations match a graph that holds a <lambda>. A carried row is also checked for a guard
at the CALL, because that is where a handler establishes who is asking. The method holding a
carried operation joins the question's context scope, so the module holding the query cannot
change under a claim that depends on it without the change being attributed. The witness names
both sites and the finding is located at the call, which is the line a contributor changed.
A carried obligation corroborates, it does not entail, and that was measured rather than
assumed. Entailing it reported handleSignup, which looks a user up by name to check the name is
free during registration and is public by design, because two authenticated siblings in the same
file pass req.session.userId into the same module. The entailed rung's justification is that one
FILE contradicts itself, same file and same trust context; carried, the compared operations live
in a data-access module that handlers of different trust contexts share. The corroborated band
already asks the right residual: does this operation require a check, or is the route
legitimately public?
Measured on NodeGoat: 15 handlers asked, obligations raised at zero before and six now, three
reaching a verdict, all corroborated, none reported without a judge. VAmPI unchanged at two
entailed findings, because it keeps handler and query in one file and carries nothing.
What this does not reach is recorded, not implied. NodeGoat's documented insecure direct object
reference still raises nothing, because its module reads with allocationsCol.find(...) and find
is not a modelled operation: Array.prototype.find and jQuery's .find are everywhere, so it needs
an argument-shape discriminator and its own measurement (task 14.8). Two of the three corroborated
claims are correct code whose identity read a destructuring spelled away, and the cross-file
contradiction that would entail the real defect needs a sibling-scope decision (task 14.7).
1,415 tests pass, fifteen skipped; eleven Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The census recorded this family completing 44 questions on a Node application and finding
nothing, which is what a client-only vocabulary must do there: all six declared sinks were DOM
sinks. NodeGoat's cross-site scripting is server-side.
Decision: the family asks about a value reaching a rendering call, and states in its own
declaration that it cannot decide template-mediated escaping. Whether res.render is safe depends
on the engine's autoescape setting and on the template's markup, neither of which is at the
call, so res.render is not a sink and the family's limits field says so. A new CWE-79 sink
covers res.write, res.end, res.send and the response.* spellings; res.json stays out because an
application/json body is not a document.
Template auto-escaping is claimed where its evidence is, as a configuration defect:
swig.setDefaults, nunjucks.configure and marked.setOptions are security settings now.
It establishes a finding: app/routes/research.js:25, model_entailed,
`req.query.url + req.query.symbol -> res.write(body)`. The handler fetches a URL the caller
chooses and writes the response body into a text/html response, so the document served from
this origin is the attacker's. The record states how coarse the step is: the path from url to
body runs through the client's callback parameter, which the graph treats as reachable from the
call's arguments, so the conclusion rests on the call's semantics rather than on what the graph
established. Five of seven families now establish findings.
Four defects were found on the way, three of them false positives this change would otherwise
have shipped.
- The permissive-value set was shared across every setting, while a boolean's polarity belongs
to the setting: true leaves the door open in cors({origin: true}) and closes it in
swig.setDefaults({autoescape: true}). Adding the template engines would have reported the fix
as the defect on the line that offers it. ConfigSpec now carries per-setting values, and
Jinja2's Environment moved out of Python's shared list for the same latent reason.
- config.sc matched a dotted setting on its trailing segment alone, so marked.setOptions would
have claimed an editor's setOptions. dominance.sc already required the qualifier.
- taint.sc asked its flow query of every argument including the receiver at index 0. On a
response sink the receiver is the response object, so res.end() -- which takes no data -- was
reported exactly like the res.write(body) two lines above it. Flows now use the real
arguments, as arity already did.
- The arbiter sorted a file's settings by line number as TEXT, so line 126 preceded line 78 and
which setting a file reported depended on the spelling of its line numbers.
Template escaping disabled at server.js:135 is detected, measured model_entailed against the
real query and the real arbiter, and still not reported: the arbiter returns one verdict per
question and that file has an earlier permissive setting. Task 14.6 carries it rather than
hiding it.
1,415 tests pass, thirteen skipped; nine Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The family census recorded this family completing 44 questions on NodeGoat, finding the open
redirect, and having no way to ask about the repository's documented SSRF: needle.get on a
query-string URL at app/routes/research.js:16. A thin vocabulary is a silent false negative.
One data change, no query change. The declared set now covers axios, got, needle, request,
superagent, unirest, undici and fetch as bare module tokens, plus http.get, http.request,
https.get and https.request dotted. Bare is the documented call form of each module and covers
its verb methods too, since a token matches word-bounded against the callee and axios is
bounded inside axios.get; axios.get and axios.post are dropped as subsumed rather than
unwanted. http and https stay dotted because their namespace also holds createServer, and a
bare token would have claimed every server a repository starts.
The request is now reported at app/routes/research.js:16, model_entailed, witness
`req.query.url + req.query.symbol -> needle.get(url, ...)`. This family goes from one finding
to two on NodeGoat, every other family is unchanged, and 299 of 299 questions still complete.
Measured on that graph, needle matches exactly one call and the eleven other clients match
none in first-party or vendored code, so NodeGoat shows the gap closing and cannot show those
clients' noise.
A control asserts both halves. A handler calling axios.get with a literal and needle.get with
req.query.url reports the tainted client alone, and it reads the shipped vocabulary rather than
a copy, so removing the client fails it. The CWE-601 redirect entry is untouched: a
Location-header redirect is value-shaped and stays outside what this family asks.
1,411 tests pass, eleven skipped; seven Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Seven vulnerability families are declared and two had been proven. The census asks each of
them, on whole checkouts of NodeGoat and VAmPI, whether it establishes anything at all, and
reports a family that answered and found nothing separately from one that was never asked.
Four establish findings: injection, access_control, config_secrets and untrusted_destination.
Output encoding, path, prototype and deserialization establish nothing on either subject. All
299 NodeGoat and 35 VAmPI questions completed, so the silences are answers.
Three of those silences are diagnosed gaps, not clean code. NodeGoat's documented SSRF calls
needle.get, which the sink vocabulary does not hold. Its XSS is server-rendered, and output
encoding declares only DOM sinks, so no question it can ask could have found it. Its two
access-control defects raise no obligation at all: measured with the real dominance query,
every modelled data operation lives in app/data/*-dao.js and none in a route handler file,
while the family scopes operations to the region's own file. Tasks 14.2 to 14.5 carry each
gap; none is fixed here.
The census also found a false positive and it is fixed. NodeGoat's only prototype report was
the word `set` inside `// set these up in case we have an error case`: the matched node is the
<operator>.assignment holding the entire 1,000-character signup handler, and sinkMatches
searched a node's whole text. It now confines the text clause to the callee and excludes
operator nodes. Fourth instance of this bug class after fgets/gets, resolve/resolveUrl and
user/users. Measured on NodeGoat's non-vendor calls, injection sinks fall 28 to 3, prototype
13 to 2 and untrusted destination 26 to 7, with no true positive lost; req.query had been an
injection sink because `query` is word-bounded inside it. Witnesses now name the sink call
rather than the assignment that contained it.
1,411 tests pass, ten skipped; the six Joern-gated query tests pass in the image, including a
new one that fails on the shape this fixed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Paused at the maintainer's direction to return to vulnerability work, with the
state recorded rather than left implied.
Measured and eliminated: batch size, since 17,700 requests answer in 44 s with a
2.6 MB response; dataflow cost, 50 execution requests in 24 s; call depth 3 with
parameter sources, 24.7 s for 20 requests; argument length, a 16 KB hook table
against a 128 KB single-argument cap; hook callback resolution, one second of
difference with 233 hooks; and build or sharding, 79 s with no shards. Preparation
measures 273.5 s, which is real but a small share of a 3,600-second case.
Every reproduction of the failing 22,125-request batch outside the push path answers
in under a minute. The difference from the production invocation is not identified,
and the exception itself has still never been read: the backend logged a stack
trace's tail, which is frames, and the corrected extractor reports its head, which
on this engine is JVM startup noise.
Four instrument defects were found and fixed along the way, all of the same shape:
a failure detail that discarded the cause, stages that reported no cost, logging
that was never configured so the new stage lines could not emit, and a harness that
buffers stderr until a case ends.
Node and Python transfer stand. PHP stays unestablished.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Every PHP failure logged `cpg batch taint failed: dure1.java:10)`, which is the
middle of a stack frame and says nothing. A trace puts its exception on the FIRST
line and frames after it, and the backend logged the last 400 characters, so the
cause was discarded and the noise kept. Four investigations were spent on that.
Failures now report the leading non-frame lines, which is where the exception and
its message are, with the frames dropped. A control asserts an OutOfMemoryError
survives while a frame does not, and that a timeout and an empty exit stay
distinguishable.
1,411 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The stage costs added for PHP diagnosis never appeared: nothing in this project
configures logging, so the default root level of WARNING dropped every
informational line and a PMPro case ran for an hour producing none.
That is the same failure shape the project keeps relearning. A run that reports
nothing is indistinguishable from a run with nothing to report, and it is how four
PHP investigations were spent reading a later stage's timeout as the cause.
Logging is now configured from OUSAST_LOG_LEVEL, quiet by default and sent to
stderr when asked, so a hook stays silent while a diagnostic run says where its
budget went. Verified end to end: six stage lines with durations on a real replay.
1,410 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Three PHP investigations were spent reading `cpg batch taint failed: timeout` as a
slow query. The query is not slow: 4,425 evidence requests answer in 30 s, 50
dataflow requests in 24 s, and 20 entry-point requests at call depth 3 with
parameter sources in 24.7 s. The production PHP build takes 79 s and does not
shard. None of that explains a case running 26 minutes without finishing.
The reason it could not be diagnosed is that nothing reported where the budget
went. A run that is killed, or that blames a later stage's timeout, left no account
of what had already consumed the transaction, and a subprocess bounded by the
remaining time reports its own timeout whatever spent it.
Each stage now records and logs what it cost as it finishes, so a killed run still
says where it was. A control asserts the stages report and that each line carries
its duration.
1,410 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Retried the PMPro transfer with batch splitting, a 4500-second deadline and a
900-second query ceiling. Stopped after 26 minutes with no case completed and no
verdict, so PHP transfer remains unestablished.
A probe narrowed the cause and corrected my earlier reading of it. The evidence
pass answers all 4,425 PMPro PHP regions in ONE batch in 29.7 seconds with a 681 KB
response, exit 0, and its cost is fixed rather than per request: one request took
40.2 s and 4,425 took 29.7 s. Neither request count nor response size is anywhere
near a limit, so batch size and serialization are not the cause as I first supposed
from a single extrapolated data point.
What fails is the execution pass, which runs reachableByFlows per candidate.
The probe's own scope is recorded with it: it sent evidenceOnly=true and therefore
measured the structural pass, not the failing one. Those numbers would otherwise
read as evidence that PMPro's taint work is cheap, which they are not.
Splitting stays, because it is correct on its own terms and recovers a batch that
fails whole. It is worth noting it also makes an unrecoverable batch slower to
abandon, since each of the eight parts burns its own share of the query ceiling.
M3 remains two of three: Node established in M1b, Python established, PHP blocked
on dataflow cost for a 1,274-file repository.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
PMPro's 1,274 PHP files produced a response the engine could not serialize: it
failed inside ujson while building its own answer array, and every question in that
batch was lost with it. Three budgets from 900 to 3000 seconds made no difference,
because the size of the answer was the problem rather than the time available.
One invocation stays the first thing tried, so a working scan costs exactly what it
did. Each request in a batch is independent, so a batch that fails as a whole can
still answer in parts: on failure the requests are split and retried, and a part
that fails leaves its own requests unanswered rather than discarding every answer
beside them. Only a batch where no part answered is reported as unavailable, which
keeps "could not ask" distinct from "no rows".
Controls cover all three shapes: a whole batch that fails is answered in parts with
the split recorded, a failing part leaves only its own requests unanswered while the
rest are kept, and a batch no part can answer is still unavailable rather than
empty.
Whether this is enough for PMPro is a separate question the measurement will
answer: a single taint request on that tree already takes 40 seconds and returns
36 KB, so the batch may be slow as well as large.
1,410 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
All three PMPro comparisons exhausted a 3000-second deadline with a 1200-second
query ceiling and produced no artifact, so there is no target evidence at all. The
engine reports `cpg batch taint failed` twice per case on the 1,274-file tree:
first an exception whose trace runs through ujson array transformation, then a
timeout on the retry.
Budgets tried: 900 s, 1200 s and 3000 s deadlines against 300 s and 1200 s query
ceilings. The pins, source receipts and declared target are all correct, so the
question was asked properly and the engine could not answer it. This is a runtime
blocker on a large repository, not evidence about whether the design transfers to
PHP.
Two harness defects were found on the way, and both had made an earlier run look
like a transfer failure rather than an instrument problem. The manifest labels a
case by what it is, vulnerable or fixed or benign, while the id says what the
comparison does, introduce or repair or benign; filtering on one while keying
expectations by the other ran a single case and reported an all-false verdict. And
the per-query ceiling is taken as the minimum with the remaining transaction time,
so raising only the deadline could never have helped, and twice did not.
M3 is not met. Its exit asks for Node, PHP and Python together: Python is
established, Node was established in M1b, PHP is blocked.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Generalized the replay instrument so the case family is an argument rather than a
hard-coded target, since a harness that only knows one language cannot say whether
the design transfers. Targets, pins and expectations come from the committed inputs,
so Python is asked exactly the way Node was, through the same core, ranker,
comparison and admission.
VAmPI-SQLI models/user_model.py:73:get_user detected, answered
VAmPI-BOLA api_views/users.py:187:update_password detected, answered
VAmPI-BOLA-BOOKS api_views/books.py:51:get_by_title detected, answered
35 of 35 questions complete on every case, nothing reported new, zero admitted
defects.
Two results stand out. The SQL injection is the evidence the milestone spec names
as previously unresolved. And two of the three targets are access control, the
family that could not complete a question on any repository before group 12, so
this is the first time broken object-level authorization has been asked on Python
at all, and it answered on both endpoints.
VAmPI has the same revision on both sides, so the regression mode establishes
detection and completion, never novelty. It asserts that directly: with nothing
changed, a target reported new would be a comparison defect, and none was. That
tests the novelty logic from the opposite side to the NodeGoat case.
Identical revisions cannot show novelty, and VAmPI is inspected teaching data that
cannot qualify production Python support. No capability enabled, no latency claim.
1,407 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Measured after scoping a registration-only region to its file.
M1a, before group 12 0 of 436 head questions completed
after per-family context 298 of 436
after region attribution 299 of 299, and 299 of 299 on the base
The count falls from 436 to 299 because the mis-attributed function-scoped regions
in a route-registration module are gone, replaced by file-scoped questions the graph
can answer. Nothing is unresolved on either revision.
Detections are unchanged: the vulnerable revision is still new with
operation_absent_from_comparable_base, the fixed twin analyzed and quiet, the
unchanged revision not a regression.
What remains is recorded limits rather than failures: 210 bounded-reachability
boundaries, each owned by the question that reported it, and one vendor exclusion.
Neither blocks a comparison.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A route-registration module registers handlers that other modules define.
app/routes/index.js requires seven handler modules and defines none, yet regions
there were scoped to handleContributionsUpdate and its siblings. The graph rightly
held no such method, so every question in those regions was unresolvable: 138
across six families on that one repository.
The entry record still names the handler. The authorization question is which
middleware guards which handler, and that is cross-file by nature. It now also
records whether the file defines what it names, and a region built from an entry
that does not is scoped to the file instead of to a method the file lacks. The
handler's own module still gets its function-scoped region, so nothing is lost.
My first attempt changed the mapper to stop naming cross-file handlers at all. An
existing control caught it by asserting that a multi-line registration names the
handler its middleware guards, which is exactly the attribution the authorization
family needs. The evidence was against me and the fix belonged one layer up, in
region construction.
1,407 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Final verification of the three security capability fixes on the pinned revisions.
head questions completed 205 -> 298 of 436
base questions completed 436 of 436
access control completed 0 -> 52
context_projection_unavailable 116 -> 0
context_scope_empty 138 -> 0
Every remaining unresolved question now reports context_function_unresolved and
names the function its file does not define, so nothing is unresolved for a reason
the tool could not explain. That is a region-attribution question for the ranker,
not a limit of what the engine can establish.
Detections are unchanged: the vulnerable revision is still new with
operation_absent_from_comparable_base and every verdict entry holds.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
After per-family context, every remaining unresolved question reported
context_scope_empty, all in three files. The cause is not a missing producer:
app/routes/index.js requires seven handler modules and defines none of them, so a
question naming handleContributionsUpdate there names a function that file
genuinely does not hold, and the empty scope is the correct answer.
Reporting that as "scope empty" hid which of two different things had happened. A
file the frontend produced no methods for is one problem; a question naming a
function defined in another module is another, and it is the common one. Taint and
configuration now separate them, reporting context_function_unresolved with the
name in the first case and context_scope_empty only when the file has no methods at
all. The driver no longer stacks its generic gap on top of a query that already
explained itself, and the new gap is question-owned so it cannot block another
question's comparison.
Completion is deliberately unchanged. The tempting fix is to widen such a question
to its whole file, which would raise the number by answering a question nobody
asked; a question whose function the graph does not hold stays unresolved, with the
name now recoverable from the question that asked it.
Group 12 is complete: admission no longer rejects genuine findings on a
mis-extracted operation symbol, both families that could never complete their change
context now can, and the remaining unresolved questions say precisely why.
1,406 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Rerunning the pinned revisions with each family producing its own change context.
head questions completed 205 -> 298 of 436
context_projection_unavailable 116 -> 0
access_control completed 0 -> 52
Access control is the headline: the family could not complete a question on any
repository before, and now completes every question it is asked. That is a
vulnerability class the tool could not reason about at all.
Detections are unchanged. The vulnerable revision is still new with
operation_absent_from_comparable_base, the fixed twin is analyzed and quiet, the
unchanged revision is not a regression.
The remaining 138 unresolved questions are all context_scope_empty and sit in three
files, none of them access control. They are function-scoped questions whose named
function the query cannot locate in the graph, so the scope it describes is honestly
empty. That is the query-to-operation identity problem M1a recorded at the witness
end, showing up at the scope end, and it is task 12.3 rather than something to patch
by widening a question to its whole file.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Access control and configuration could not complete their change context on any
repository, which left 231 of 436 questions unresolved on the measured Node case
and made two of the three shipped families unable to answer.
The gap was larger than a missing query block. The evidence pass dispatches taint
only, because only taint has an evidence vector to rank by, so these families were
never asked for context at all. A new context pass asks each family with the flag
set.
Each query describes its own scope, which the owner research required and which
matters: copying taint's projection would describe a reachability walk these
queries never perform. Dominance exports the file's methods labelled operation,
guard or sibling, because its claim is that a file contradicts itself. Configuration
exports the file's methods labelled setting or sibling, because it asks what a file
declares. Both report an empty scope explicitly rather than implying one.
A family whose context query answers nothing keeps no context and stays unresolved,
so none can complete on another family's context. A control asserts that directly.
Confirmed on a real NodeGoat graph through the production backend: both families
produce a context summary and seven scoped methods where they previously produced
nothing. All seven are siblings for that file, which is the honest answer since it
declares no access-control operation and no matched setting.
1,405 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Admission extracted everything before the first parenthesis, so the operation for
`const preTax = eval(req.body.preTax)` was `const preTax = eval`. No real
declaration can enumerate every assignment a codebase might write, so admission
would have rejected genuine findings on operation_semantics_mismatch. The M1b
demonstration only passed because its declaration listed the assignment spellings.
The symbol is now the trailing qualified callee. Receiver spelling stays part of
it, so `$wpdb->prepare` and `prepare` remain different operations and a declaration
cannot widen from one to the other. A spelling that is not a qualified name, such
as a dynamic `handlers[name](x)`, yields the statement text, matches no declaration
and stays ineligible exactly as before.
Controls cover the assigned call, the receiver distinction, the dynamic spelling,
and that a declaration for a different operation still does not cover it. The
recorded-veto report reads the same helper instead of repeating the extraction.
1,403 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Both attacks on engine startup are measured and neither reaches the gate. The
session is equivalent for every shipped query kind and helps a long transaction,
326 s to 243 s, but its own 9 to 13 second startup is a third of a 30-second budget
so it usually declines to start. Removing JavaScript's second engine start is real,
11.0 s against 21.0 s on identical graphs, and was reverted for an asymmetric
census gate that made 12 transactions pay two starts.
What the profile names is not startup. A changed-code transaction reaches its
deadline inside the build: head build 25.2 s of 30 s, query time 0.0 s. The cost is
building a fresh whole-repository graph for every pushed revision.
Direction accepted: reuse the graph across revisions rather than optimise the
rebuild. M2 stays unmet, the gates stay unchanged, and the work is not scheduled
here. Group 12 returns to security capability, which is where the remaining release
risk sits.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reverts f04a839 on its own measurement. The seven-class profile rerun recorded
joern_parse_census_incomplete on 12 transactions: the faster build was rejected by
its own census gate and the frontend plus overlay pass ran as well, so those
transactions paid two engine starts instead of one. Every workload kept the same
p50 and the verdict stayed NO-GO.
The cause is an asymmetric gate. The frontend path is not census-gated at build
time, so whatever it produces is accepted, while the joern-parse path was gated and
the same shortfall rejected it. Holding the new path to a standard the old one
never has to meet guarantees the fallback fires. The probe tree and the
materialized push snapshot also differ, so equivalence on the checkout did not
transfer to the real path.
What still stands is the underlying timing fact, measured under the production
retention environment on identical graphs, same 44 files, 579 methods, file names,
overlays, Gruntfile and eighteen test files: 11.0 s against 21.0 s. JavaScript does
pay two engine starts where every other language pays one. A future attempt needs a
gate both paths are held to, or a direct comparison against what the frontend would
produce, and must be measured in the push path rather than on a checkout.
The profile also names the remaining cost plainly: a changed-code transaction
spends its budget before any query runs, head build 25.2 s of a 30 s deadline with
query time 0.0 s. Building one new graph per push is the cost, not querying it.
1,401 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
JavaScript was the only language paying two engine starts: the declared frontend
and then a separate overlay pass. joern-parse applies the overlays itself. On the
pinned Node tree that is 11.0 s against 21.0 s.
The declared frontend stays as the fallback, because it is what the versioned
retention adaptation and the explicit exclusion policy run through, and the census
decides per build: a graph that does not hold every readable declared file is
rebuilt through the frontend rather than trusted. Exclusions keep the frontend path
outright.
Measured under the production retention environment, both paths produce the same 44
files, the same 579 methods, the same file names, the same five overlays, the
Gruntfile and all eighteen first-party test files, with no file present in one
graph and missing from the other.
My first measurement did not establish that. It ran without the retention
environment, both paths dropped to 25 files, and the apparent equivalence was two
runs discarding the same things. The existing control named for that decision
failed against the change built on it, which is exactly what it is for; the
evidence was wrong, not the control. That control now asserts the intent directly:
the faster path is tried first, and a short census rebuilds through the frontend.
1,401 tests passed, nine skipped. A build saving is not the latency gate; the
profile rerun decides that. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The seven-class profile showed a 25.3 s head "build" of which the trace attributes
only 4.5 s to the frontend. The remaining twenty seconds were the session's startup
poll, which was bounded by `min(startup_timeout, deadline - now)` and therefore ran
until the transaction deadline was exhausted. The fallback then had nothing left,
so the transport's failure mode consumed exactly the budget it existed to save.
A session now refuses to start unless enough budget survives for the work it exists
to serve, and the wait is bounded by its own allowance rather than by the whole
deadline. Under the 30-second hook deadline it declines immediately instead of
burning what is left.
Controls cover both: refusing is immediate and starts no process, and a startup
wait ends at its allowance nowhere near a long deadline.
This does not make the gate reachable; it removes a self-inflicted cost that was
hiding the real one. 1,401 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The unchanged seven-class profile, three samples each, with the engine session
enabled. Every changed-code workload still times out at the 30-second deadline, and
identical-tip moved the wrong way: p50 rose from 18.5 s to 24.2 s.
The cause is in the artifacts rather than inferred. Thirteen transactions report
session_unavailable:session_startup_timeout, so under the real deadline the session
usually never starts and everything falls back to the disposable path. Startup is 9
to 13 seconds, about a third of the whole budget, and it is spent before any
analysis happens. Where a session did start, on cached-graph transactions whose
remaining work is small, it cost more than it saved.
That is the honest shape of the result: a transaction-owned session amortizes
engine startup across the invocations inside one transaction, and a warm
changed-code push does not have enough of them. It helps a long development-budget
transaction, 326 s to 243 s measured earlier, and cannot help a 30-second one.
Correctness is unchanged throughout: the same three pinned revisions through the
production rules with the session enabled reproduce the M1 verdict exactly.
Group 11 as designed cannot reach the gate, so the dominant remaining cost is named
and the decision is raised rather than worked around. Task 11.6 records the
candidates, since two of them change the product's shape: a session shared across
transactions is a resident local service, which the accepted milestone spec does
not imply, and analysing less is a scope change rather than a speed change. The
30-second and 2-second gates stay unchanged.
Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The leak was the whole cause. With the session closed by the transaction that owns
it, a session-enabled replay of the same pinned revisions under the same budget
keeps its session for the whole run: no session_query_failed appears on either
side, where the previous run reported one per side and fell back to the disposable
path for everything.
Measured on the same pins, instrument and budget, changing only the transport:
vulnerable revision 326.4 s -> 243.3 s
head query time 63.6 s -> 32.0 s
base query time 85.1 s -> 51.2 s
dominance and config ~14 s each -> ~4 s each
Detections are unchanged: witness present, fixed twin analyzed and quiet, unchanged
revision not a regression, kill criterion not triggered. Speed bought by losing a
detection would not be speed.
This is a cold transaction on one teaching repository under a 900-second
development budget. It is not the hook latency gate, which is warm changed-code p95
against the frozen seven-class profile, and that is task 11.5. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The first session-enabled replay produced correct detections and no speedup,
reporting session_query_failed:census:session_request_failed on both sides. A
targeted probe ruled out the sequence I suspected: an overlay followed by a census
in one session both answer correctly, with the census confirming the overlays were
saved.
What the probe exposed instead is a leak. A build runs on a copy of the backend, so
the session started during the build belonged to that copy and was never closed,
leaving a JVM holding the configured 2 GB heap for the life of the process. Two
builds per transaction meant two of them, which is enough to exhaust a 3 GB
container and make every later request fail. It also matches the host running out
of memory during these runs.
The build now closes any session its copy started, and the driver ends the session
with the transaction, since that is what transaction-owned means. Controls cover
both: a build leaves nothing alive on the copy that started it, and a completed
scan closes the backend's session. Closing never fails a scan that found things.
This is a genuine defect found by measurement rather than by the suite, which
passed throughout. Whether it was the whole cause of the census failure is not yet
established; the next session-enabled run will say.
1,399 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The session-enabled NodeGoat run reported session_query_failed:census:
session_request_failed and fell back, which is correct behaviour but says nothing
about why. A bare transport failure cannot distinguish a timeout from a crash from
an out-of-memory kill, and the only account of which it was is in the server's own
log.
A transport failure now carries the exception and the tail of that log, and the
backend logs it with the process exit code alongside its own label. A control
asserts an OutOfMemoryError in the log reaches the caller.
This is diagnosis, not a repair: the run's timings were unchanged because the
session died early and everything fell back to the disposable path.
1,397 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A measured run needs to enable the transport without changing any default. The
switch reads as enabled only for the exact value "1"; unset, empty, "0", "true"
and anything else leave the disposable path in place, and an explicit constructor
argument still wins in both directions.
1,396 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The overlay was the last fixed cost bypassing the session: a whole JVM start,
15.8 s in the measured Node runs, while the census and query batches already route
through it.
`save` writes under the process working directory, which for a session is its own
private scratch rather than the build scratch, so the saved graph is now looked for
in both. Finding neither keeps the raw graph and records it rather than reporting
success, because a graph without its dataflow layer has that layer recomputed by
every later batch, 54 s each on a 637-file plugin, with nothing otherwise saying so.
The disposable invocation remains the fallback and runs unchanged whenever the
session is off or cannot answer.
Controls cover both new paths: an engine that reports success but saves no graph
keeps the raw one and notes it, and a graph saved under the session's workspace is
found and applied.
1,395 tests passed, nine skipped. The transport stays off by default; task 11.5
measures against the frozen profile. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A session is only usable if it returns exactly what the disposable invocation
returns. On a real NodeGoat graph, taint, dominance and configuration all produce
byte-identical payloads through both transports, with parameters built by each
arbiter's own request function through the driver's family lookup rather than a
hand-written shape.
The failure shapes that would make a session dangerous rather than merely slow are
correct too. An empty answer stays present and distinguishable from an unavailable
one, which is the distinction that once turned a failed scan into a clean bill of
health. An unknown query and an absent graph are both unavailable. Two graphs in
one session each answer about the graph the request names.
Amortized over five taint requests the session took 22.5 s against 72.0 s
disposable, all five answered and identical to each other.
Two instrument defects are recorded rather than hidden. The first run created a
fresh backend per comparison, so every session served one request and paid full
startup, measuring the session's worst case and making it look slower; that is why
the amortized comparison exists. The switched-graph check crashed on a scratch
directory it never created, so it produced no verdict at all.
These timings are orientation, not the latency gate. Task 11.5 measures against the
frozen seven-class profile, and the transport stays off by default until then.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
JVM startup, not analysis, is what a push scan spends its time on: each disposable
script invocation costs 13 to 23 seconds while the work takes under a second.
The session owns one Joern server per transaction on loopback with an OS-chosen
port and no published port. Three properties are load-bearing, and the probe that
settled the design demonstrated why each is needed.
An HTTP status is not an answer. The server returns success even for a compile
error, with the diagnostic in its body, so every request redirects the evaluated
code's console output into a fresh private file behind a nonce, and the nonce is
verified before the answer is accepted. The engine's body is retained for exactly
the case where the receipt is missing, without which that failure explains nothing.
A poisoned session answers nothing afterwards, and cancellation kills the process
group rather than the launcher, which this project has already run a machine out of
memory by neglecting.
What the probe established about the engine, after three iterations that each
failed loudly: runScript is absent from the server REPL; a shipped @main script
cannot be nested in a block expression, so definitions go to the top level
unwrapped; and the REPL shadows Console with io.joern.console.Console, which has no
withOut, so capture must qualify scala.Console. With that fixed, both mechanisms
answer identically to the disposable invocation, 0.8 s against 9.8 s.
The backend routes batch and single queries through the session only when
session_transport is enabled, which it is not by default. Any failure falls back to
the disposable invocation rather than reporting a zero, and each script is defined
under its own name because every script declares exec.
Equivalence for taint, dominance and configuration is task 11.2 and gates any
measured run. 1,391 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The M1b exit run established the comparison but could not show a developer-readable
consequence or repair direction, because both belong to an evaluated capability
declaration and the installed registry is deliberately empty.
The experimental evaluation output now renders, for each change-attributed finding,
its family, exact location, witness, change attribution, provenance, engine method
and operation, together with a consequence and repair direction produced from a
supplied declaration through the same grounded-template path the admitted case
uses. No prose is invented: when no declaration matches, or a template is not
grounded in the operation and context, or the declaration does not cover the
operation symbol, the explanation is absent and the reason is stated.
Declarations reach this rendering only. The runner never passes them to admission,
so nothing in that file can enable a capability or emit an alert however it is
written, and a control asserts that even a declaration claiming PASS admits
nothing. The flag requires explicit base/head replay and the recorded-veto flag.
The evaluation instrument generates its own unreviewed declaration bound to the
run's analysis semantics, asserting that it is neither enabled nor PASS, so the
demonstration cannot drift into a qualified claim.
1,377 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
All three pinned NodeGoat revisions replayed through the production rules with
exact pins and reopened source receipts.
The vulnerable revision reports all three sites new with
operation_absent_from_comparable_base, and admission now rejects on
capability_unavailable alone: dependency_unresolved is gone, which verifies 10.7
on real data since those gaps belonged to unrelated questions. The fixed twin is
analyzed and quiet with three base-only findings. The unchanged revision reports
all three sites unchanged with base locations resolved to the same lines.
Every finding carries its question, the engine-reported method and its witness.
The head census is the complete 44 files and all 436 base questions complete.
Zero admitted defects on every revision.
All five vetoes M1a recorded are repaired, plus two the repairs exposed. What
remains is recorded rather than repaired: 231 head questions still report an
incomplete change context from families whose queries export none, which did not
block this case and whose families stay experimental, and the vendor exclusion
and correspondence ambiguity remain reported as the limits they are.
M1b is not yet verified. The milestone also asks for a developer-readable finding
carrying a supported security consequence and a concrete repair direction, and
both are owned by an evaluated capability declaration that is deliberately empty.
Writing that text without a declaration is what admission exists to prevent, so
task 10.10 renders it through an explicitly experimental declaration with
eligibility still disabled.
Runtime, independent qualification and rollout all remain NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
M1a recorded that the NodeGoat target is selected through a file-scoped question
naming no function while the engine reports method <lambda>2, and that this did
not block the comparison once the question completed. It was still only visible by
digging the method out of a serialized detail blob.
Evidence now carries the engine-reported method beside the question that produced
it, and the relationship between them is recorded: exact when the question named
that method, file_scope when the question named no function and the engine
supplied one, unresolved when the two disagree, and method_unreported when the
engine named none.
Recorded, not applied. No region is renamed, no benchmark function name is
injected into a region, and no coincident site is counted. The recorded-veto
report now reads the field instead of reparsing the detail blob.
1,371 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The runner handed every candidate the whole transaction's coverage reason list as
its unresolved dependencies, so admission always answered dependency_unresolved.
That list even included other candidates' comparison outcomes, so one unresolved
comparison anywhere marked every other claim as having an unresolved dependency.
Gaps are now attributed. A boundary carrying a question id counts only when that
question is in this candidate's required comparison scope, the same scope 10.2
established. A gap nothing attributed to any question always counts, because
unknown ownership is not evidence of independence. Integrity degradations count.
A declared exclusion, a correspondence ambiguity and a bounded reachability do
not, because each was already weighed against this specific claim: the first is a
scope choice, the second is decided per operation, and the third is decided per
claim by the comparison.
Found by the M1b production rerun rather than by the suite: the comparison reached
a correct new verdict and admission still rejected it for a dependency belonging
to unrelated questions. It is not an M1b exit blocker, since eligibility stays
disabled, but it would have blocked admission forever once capabilities qualify.
1,371 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Rerunning the three pinned revisions after tasks 10.1-10.6 shows the production
comparison classifying all of them with no lifting, and agreeing exactly with the
recorded view. That agreement is the point: the vetoes M1a recorded were repaired
rather than bypassed.
The vulnerable revision is new with operation_absent_from_comparable_base, and the
three eval witnesses at contributions.js lines 32 to 34 are attributed to the
change that introduced them. The fixed twin is analyzed and quiet with three
base-only findings. The unchanged revision is unchanged. In M1a every one of the
436 questions was demoted on both sides and production reported
witness_identity_unresolved on every finding.
All three artifacts still report finding_status none with zero admitted defects.
Admission rejects on capability_unavailable, the deliberate eligibility gate, and
on dependency_unresolved, which the rerun exposed as coarse plumbing: the runner
passes every coverage reason as every candidate's dependency gaps. That is now
task 10.7, ahead of M4, since it would block admission forever once capabilities
qualify. It is not a comparison failure and not an M1b exit blocker.
Three comparisons on one inspected teaching repository under a development budget.
No capability enabled, no alert emitted, no latency claim. Runtime, independent
qualification and rollout all remain NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The NodeGoat rerun after 10.4-10.6 showed both target questions completing, which
is what those tasks were for, but the comparison still reported
head_context_incomplete. Two checks inside comparison had not been updated with
the same distinction: any degradation blocked, and any scope boundary blocked.
A declared exclusion and a correspondence ambiguity both reached those checks. The
first is a scope choice rather than a failure to read the graph, and the second is
decided per operation since 10.3, so neither should block a claim. Both remain in
aggregate coverage. Integrity degradations, unknown ownership and every other
boundary still block, and bounded reachability is still filtered separately
because it blocks only claims resting on a flow not having been traced.
1,368 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
vendor_semantics_unresolved was grouped with the graph integrity failures, so any
declared exclusion demoted every otherwise complete answer in the repository to
graph_incomplete. Every real repository excludes its dependencies, so this fired
on all of them, including on a complete first-party census with successful query
receipts.
Graph boundaries are now split by what they mean. An integrity failure says the
graph could not be read or is incomplete, so an answer drawn from it is demoted:
unparsed files, an empty or sharded graph, an incomplete or unavailable census,
an unresolved symlink, an unsupported frontend, unreadable source, an ambiguous
frontend path and unvalidated TypeScript property support all keep that behavior.
A declared exclusion is a deliberate scope choice. It stays physical, stays out of
targets and graphs, and stays reported in coverage, but it is not evidence that a
first-party answer is wrong.
Whether a particular answer needed the excluded semantics is carried by that
question's own unresolved call destinations, recorded per question in 10.5, rather
than assumed for every answer in the repository.
Controls exercise the real partition logic rather than an injected reason: a
vendored directory leaves the answer completed with the exclusion still reported,
an unresolved symlink still demotes, and the two together still demote.
1,365 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
dynamic_external_or_depth_context_unresolved was applied as though it invalidated
everything about a question. It does not. Bounded reachability can hide further
flows; it cannot invalidate a flow that was traced, and it cannot remove a call
from a structurally scoped enumeration.
At scan level the boundary is now recorded against its owning question without
demoting that question's answer. An unavailable context projection, an empty
context scope and an unreadable method extent are unknown ownership and still
demote.
In comparison it no longer blocks the head's own claim, since the finding rests
on a path that was actually traced, and no longer blocks an absence read off the
enumerated inventory. It still blocks every claim that rests on a flow not having
been traced, including the anchored source-connection branch, which now reports
base_reachability_bounded rather than silently passing.
The query declares the basis of its enumeration. A file- or nesting-scoped
question walks calls structurally, so an unresolved destination cannot remove one
from scope. Following the call graph makes the reachable set the scope, so the
same unresolved destination can shrink what was enumerated; that inventory cannot
establish absence while reachability is bounded, and says so.
One prior control asserted a question's own reachability gap blocks its own
claim. It is updated with its reason and split, so an own projection gap is still
proven to block while an own reachability gap no longer does.
1,362 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A taint answer reports traced flows, so an absent row means either that the
operation is not present or that no traced path reached it. Change comparison
needs that distinction to say a change introduced an operation, and reachability
is deliberately bounded, so the two can never be separated from flow rows alone.
That is why the veto guarding the confusion is permanently true on any real
Express application, and why it could be neither discharged nor safely removed.
The taint query now exports one row per operation it enumerated in scope, plus a
completion marker. The marker is what makes an empty inventory meaningful:
without it a scope that was never enumerated and a scope holding no such
operation are indistinguishable.
Comparison separates three outcomes on a changed line: the operation is present
in the base without a traced flow, which is a reachability statement and not
novelty; the operation is absent from an enumerated base, which establishes that
the change introduced it; or the base scope was never enumerated, which stays
explicitly unresolved. A malformed inventory row is not an empty inventory.
The arbiter now skips any row carrying a kind, so an inventory row cannot be
normalized into a flow with an empty source that names no attacker input.
Verified against the real engine on an authored two-revision JavaScript control:
the operation is enumerated on both sides, the base carries it with zero traced
flows, the zero-operation marker appears for an empty scope, and answers and
findings are unchanged. No new sink inference, alias resolver or second source
IR. 1,354 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Rerunning the three NodeGoat revisions after tasks 10.1-10.3 shows the recorded
view now classifies the case: the vulnerable revision reaches new with
operation_absent_from_comparable_base, and the unchanged revision reaches
unchanged with its base location resolved to the same lines. Both previously
stopped at operation_correspondence_unresolved and witness_identity_unresolved.
The rerun also caught a defect in the repair itself: the new novelty reason was
missing from the admission allowlist, so admission rejected the candidate as
change_unsupported. Fixed with a control.
The production path is still blocked by the last two vetoes, and the measurement
shows neither can be scoped or relaxed as the plan described. A taint answer
reports traced flows, so an absent row means either that the operation does not
exist in the base or that no traced flow reached it.
dynamic_external_or_depth_context_unresolved exists to stop the second being read
as the first, and removing it would turn a wrong absence into a wrong new
finding. It is also computed over every in-scope method, so on any real Express
application it is always true and can never be discharged, which is why the tool
reports nothing on real projects.
The resolution is new evidence rather than a relaxed rule: export the operations
the query actually enumerated, separately from the flows it traced, so absence is
established from the inventory while a traced-flow claim still requires complete
reachability. That is now task 10.4, ahead of the two vetoes it unblocks, and it
adds no detector, alias resolver or second source IR.
1,348 tests passed, nine skipped. No capability enabled; rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Two vetoes M1a recorded were global where the evidence is local.
Comparison required every selected base question to complete, so one unsupported
family anywhere prevented an otherwise complete target comparison. Completeness
is now demanded over the required scope: the counterpart question plus the
dependencies the change context actually recorded for it. Every other gap stays
in aggregate coverage and is still reported, never discharged. An incomplete
counterpart, an incomplete recorded dependency, an unknown-ownership boundary
and a truncated base scan all still yield unknown.
Correspondence ambiguity was recorded file-wide and inherited by every question,
so repeated unchanged lines in one file invalidated every question in the
repository and blocked the whole transaction comparison. It is now decided per
operation by the correspondence classifier: an operation needing an ambiguous
anchor stays uncomparable, while a uniquely mapped or newly introduced operation
in the same file is unaffected. The mapping is still withheld and still reported.
On the NodeGoat case the ambiguity is on the target's own file, so file-level
scoping would not have been enough; only per-operation resolution is.
1,347 tests passed, nine skipped. No capability enabled; rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
M1a measured that a newly introduced operation sits on a changed line, has no
base line anchor, and therefore stayed unclassified even with every completeness
veto lifted. Establish absence over the complete counterpart answer instead of
over a line anchor.
Correspondence now reports why a head operation does not map: `changed_line`
inside a recorded head edit, `ambiguous` where several base lines could
correspond, or `unmapped` where no anchor exists. Only the first supports
reasoning about an introduced operation, and only when the base counterpart
answer is complete and contains no operation with that spelling.
Movement is matched on the operation spelling alone, so a benign identifier
rename that changes only the source text reads as movement rather than a new
flow. The existing rename control caught that; the key was too brittle before.
One prior control is superseded and updated with its reason: a new operation on
a changed line in a comparable existing file is now new rather than unknown.
Movement, ambiguity, unmapped operations, discharged or unentailed claims and an
incomplete or failed base all remain unknown.
1,337 tests passed, nine skipped. No capability enabled; rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Add group 9 to the approved task plan as the bounded owner amendment release
milestone M1a requires, implement the task-local recorded-veto evaluation flag,
and run the three pinned NodeGoat revisions under it.
The flag re-runs the production comparison and admission over a lifted copy of
the same evidence and records each veto instead of applying it. Recognized
completeness vetoes are lifted and logged; failures never are, because silence
from a failure is not a quiet fixed twin. It is off by default, accepted only on
explicit base/head replay, refused in Git stdin mode, labels every payload it
produces, and admits no defect even when a lifted candidate otherwise would.
M1a outcome, inside its one-week box and with expectations frozen on disk first:
the three entailed <lambda>2 witnesses are present at contributions.js 32-34; the
fixed twin was genuinely analyzed and quiet, answering its target question with
zero rows and no deadline reason while the same findings appear as base-only; the
unchanged revision was analyzed and recorded unchanged. The kill criterion did
not trigger.
Five vetoes fired and were recorded. M1a also corrected the diagnosis: with all
five lifted the detection still stops at operation_correspondence_unresolved,
because a newly introduced operation sits on a changed line and has no base
anchor. The unchanged revision resolves its base location on the same lines and
classifies as unchanged, which isolates correspondence as the mechanism. That
fifth veto was not in the earlier four-veto diagnosis and is now M1b's central
repair.
Diagnosis only. No defect admitted, no capability enabled, no gate changed, no
latency claim; the 900-second development budget cannot measure the hook gate.
Rollout and task 8.3 remain NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Add release-milestones.md as the executable outcome contract: prove one useful
Node detection, make it fast, verify PHP/Python transfer, qualify on untouched
cases, then enable a narrow advisory rollout. Link it from the roadmap, safety-net
steering, requirements, tasks, status and spec.json.
Split M1 after review. The raw NodeGoat <lambda>2 witness already exists in the
2026-09-14 post-census bundle; four admission vetoes discard it. M1a runs the
vulnerable/fixed/unchanged revisions under a task-local experimental flag that
records each veto instead of applying it, with a one-week box and a kill
criterion if the fixed twin is not quiet. M1b repairs only the vetoes that fired
and reruns through production rules. Existing approvals and thresholds unchanged.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
- Whether the failure is deterministic or intermittent
### 1b. Verify the Instrument Before You Believe the Measurement
Every wrong diagnosis recorded in this project had the same shape: the input was unreadable and the tool reported a plausible zero (`file_exists()` false, the parser wrote nothing, the frontend exited 0), and "no errors, 0 findings" read exactly like a clean result. Before trusting any measurement in the failure report:
- **Prove the tool read its input.** Make it open one file and print the size. A zero from an unread tree and a zero from clean code are indistinguishable afterwards.
- **Distrust a number that is too good.** A Joern script "finishing" in 0.5 s when a JVM takes 15 s to start is a failed launch. A harness must fail loudly with the exit code.
- **Prefer the shipped environment** (the Docker image has a native `php`); local wrappers that proxy stdio are not valid instruments for `php2cpg`.
- **When a result contradicts a working baseline, suspect the instrument first.**
A plausible zero is a failed instrument until proven otherwise; classify it as `CONFIG_GAP` or `RUNTIME_MISMATCH`, not as `LOGIC_ERROR` in the code under test.
@@ -134,6 +142,7 @@ If the issue is fixable by repo changes inside the current task plan, do not esc
| “Let’s try a few fixes” | Multi-fix guessing hides root cause. |
| “The spec is probably wrong, I’ll adapt it” | Spec conflicts must be surfaced explicitly. |
| “The docs search is optional” | For runtime/dependency issues, docs and version issues often contain the shortest path to root cause. |
| “It ran clean: no errors, zero findings” | A tool that never read its input reports the same. Prove the read (one file, its size) before believing the zero. |
@@ -20,7 +20,7 @@ description: Entry point for new work. Determines the best action path or work d
Gather **only metadata** to determine the action path. Do NOT read full file contents yet.
- **Specs inventory**: Scan `.kiro/specs/*/spec.json` for `name`, `phase` fields and `approvals` status. Note feature names and their current status.
- **Steering existence**: Check which files exist in `.kiro/steering/` (product.md, tech.md, structure.md, roadmap.md). Do NOT read their contents yet.
- **Steering existence**: Check which files exist in `.kiro/steering/` (core: overview.md, roadmap.md, safety-net.md; custom files may exist too). Do NOT read their contents yet.
- **Roadmap check**: If `.kiro/steering/roadmap.md` exists, read it. This contains project-level context (approach, scope, constraints, spec list) from a previous discovery session. Use it to restore project context.
- **Top-level structure**: List the project root directory to note key directories and files. Do NOT recurse into subdirectories.
@@ -60,7 +60,7 @@ For Path A/B, recommend the next action and stop.
**Only for Path C, D, and E.** Now load the context needed for discovery.
**In main context** (essential for dialogue with user):
- **Steering documents**: Read product.md and tech.md (if they exist) for project goals, constraints, and tech stack
- **Steering documents**: Read overview.md and safety-net.md (if they exist) for project goals, constraints, and the current state of the stack
- **Relevant specs**: If the request is adjacent to an existing spec, read that spec's requirements.md to understand boundaries and avoid overlap
**Delegate to sub-agent** (keeps exploration out of main context):
Then write `.kiro/specs/<feature>/brief.md` for **every** feature listed under `## Specs (dependency order)` using the Path C brief format. This enables parallel spec creation via `/kiro-spec-batch`.
Then write `.kiro/specs/<feature>/brief.md` for **every** feature listed under `## Specs (dependency order)` using the Path C brief format. Each spec is then created one at a time with `/kiro-spec-init`, in dependency order.
**For Path E (mixed decomposition)**:
@@ -219,7 +219,7 @@ Use the same roadmap structure as Path D, plus these additional sections:
```
Path E rules:
- Keep `## Specs (dependency order)` reserved for **new specs only** so `/kiro-spec-batch` can still parse it unchanged
- Keep `## Specs (dependency order)` reserved for **new specs only** so the list stays a plain, parseable creation order
- Record existing-spec extensions under `## Existing Spec Updates`
- Record true no-spec work under `## Direct Implementation Candidates`
- Write `brief.md` only for the **new specs** listed under `## Specs (dependency order)`
@@ -235,15 +235,12 @@ Suggest the next command and stop. Do NOT automatically run downstream spec gene
- Path A: `/kiro-spec-requirements {feature}` to update the existing spec
- Path B: Recommend direct implementation without creating a spec
- Optional fast path: `/kiro-spec-quick <feature-name>`when the user explicitly wants to continue immediately
- Path D: Default to `/kiro-spec-batch` (creates all specs in parallel based on roadmap.md dependency order)
- Optional cautious path: `/kiro-spec-init <first-feature-name>` when the user wants to validate the first slice before batching the rest
- Path C: `/kiro-spec-init <feature-name>`, then the step-by-step phases (requirements, design, tasks), each human-reviewed
- Path D: `/kiro-spec-init <first-feature-name>`for the first spec in roadmap.md dependency order; validate that slice before the next one
- Path E: Choose the next command based on the new-spec portion of the decomposition
- If there is exactly one new spec:`/kiro-spec-init <new-feature-name>`
- If there are multiple new specs: `/kiro-spec-batch`
- `/kiro-spec-init <new-feature-name>` for the first (or only) new spec, the rest in dependency order afterwards
- Also note which existing specs should be revisited with `/kiro-spec-requirements <feature>`
- Re-entry: `/kiro-spec-init <next-feature-name>` or `/kiro-spec-batch` if multiple specs remain
- Re-entry: `/kiro-spec-init <next-feature-name>`for the next spec that remains
If the decomposition contains only existing-spec updates plus direct implementation candidates, do NOT use Path E. Prefer Path A when one existing spec is the clear home, or recommend the existing-spec update plus direct implementation work without creating roadmap entries.
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to the selected task's boundary, runtime prerequisites, integrations, domain rules, security/performance constraints, or team conventions that affect implementation or validation
- Relevant local agent skills or playbooks only when they clearly match the task's host environment or use case; read the specific artifact(s) you need, not entire directories
description: Create comprehensive technical design for a specification
description: Create comprehensive technical design for a specification; optionally run the implementation gap analysis first (--gap) or an interactive design review of an existing design (--review)
- `.kiro/specs/$1/research.md` (if exists, contains the gap analysis from Step 1b or an earlier run)
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (this repository's core steering; custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to requirement coverage, architecture boundaries, integrations, runtime prerequisites, security/performance constraints, or team conventions that affect implementation readiness
- `.kiro/settings/templates/specs/design.md` for document structure
- Read `rules/design-principles.md` from this skill's directory for design principles
@@ -35,6 +40,16 @@ metadata:
- If `-y` flag provided ($2 == "-y"): Auto-approve requirements in spec.json
- Otherwise: Verify approval status (stop if unapproved, see Safety & Fallback)
### Step 1b (optional, `--gap`): Implementation Gap Analysis
Recommended for brownfield features; skip for greenfield. Analyze the gap between the approved requirements and the existing codebase to inform the design, following `rules/gap-analysis.md` from this skill's directory.
1. Read `rules/gap-analysis.md` for the analysis framework.
2. Research in parallel where possible (sub-agents if available, otherwise sequentially): existing implementations, architecture patterns, integration points and extension possibilities; external dependency compatibility and version constraints when needed; the requirements, core steering, task-relevant extra steering and relevant local agent skills or playbooks.
3. Execute the gap analysis: Grep and Read the codebase, use WebSearch/WebFetch for external dependencies if needed, evaluate multiple implementation approaches (extend / new / hybrid) with trade-offs, flag areas needing further research. Provide information and options, not final implementation choices.
4. Write the analysis to `.kiro/specs/$1/research.md` (append after a `---` rule if the file exists; never overwrite earlier research) and read it back to verify the write. Use the language specified in spec.json.
5. Unapproved requirements are a warning here, not a stop: gap analysis can inform requirement revisions. Then continue with Step 2, or stop and report if the user asked for the gap analysis alone.
### Step 2: Discovery & Analysis
**Critical: This phase ensures design is based on complete, accurate information.**
@@ -137,6 +152,15 @@ After all findings return, synthesize in main context before proceeding.
A human-paced quality review of an existing `.kiro/specs/$1/design.md`, in addition to the automatic gate of Step 5. Follow `rules/design-review.md` from this skill's directory.
1. Gather context: spec.json, requirements.md, design.md, core steering, review-relevant extra steering and local playbooks; survey existing codebase patterns, naming conventions and component structure for reference (parallel sub-agents if available).
2. Review: Analysis -> Critical Issues -> Strengths -> GO/NO-GO. Limit to the 3 most important concerns; recognize 1-2 strengths; engage interactively (clarifying questions, alternatives); leverage conversation history for requirements context and user intent. Quality assurance, not perfection seeking; accept acceptable risk; every suggestion must be implementable.
3. Decide: a clear GO/NO-GO with rationale. GO: `/kiro-spec-tasks $1` (or `-y`). NO-GO: address the critical issues, re-run `/kiro-spec-design $1`, then `/kiro-spec-design $1 --review` again.
4. If design.md does not exist, stop: "Run `/kiro-spec-design $1` first to generate design document". If the design phase is not marked generated in spec.json, warn but proceed.
## Critical Constraints
- **Type Safety**:
- Enforce strong typing aligned with the project's technology stack.
@@ -199,7 +223,7 @@ Provide brief summary in the language specified in spec.json:
**If Design Approved**:
- Review generated design at `.kiro/specs/$1/design.md`
- **Optional**: Run `/kiro-validate-design $1` for interactive quality review
- **Optional**: Run `/kiro-spec-design $1 --review` for the interactive quality review (Step 7)
- Then `/kiro-spec-tasks $1 -y` to generate implementation tasks
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to feature scope, user personas, business/domain rules, compliance/security constraints, operational constraints, or existing product boundaries
- Relevant local agent skills or playbooks only when they clearly match the feature's host environment or use case and contain domain terminology or workflow rules that shape user-observable requirements
@@ -40,7 +40,7 @@ The following research areas are independent. Decide the optimal decomposition b
**In main context** (essential for requirements generation):
- EARS format rules, requirements review gate, requirements template
- Core steering: product.md, tech.md (directly inform scope and constraints)
- Core steering: overview.md, safety-net.md (directly inform scope and constraints)
**Delegate to sub-agent** (keeps exploration out of main context):
- **Codebase hints** (brownfield projects): Spawn a sub-agent to explore existing implementations that inform requirement scope. Ask it to summarize: (1) what already exists, (2) relevant interfaces/APIs, (3) patterns that new requirements should align with. Return a summary under 150 lines.
@@ -133,7 +133,7 @@ Provide output in the language specified in spec.json with:
**If Requirements Approved**:
- Review generated requirements at `.kiro/specs/$1/requirements.md`
- **Optional Gap Analysis** (for existing codebases):
- Run `/kiro-validate-gap $1` to analyze implementation gap with current code
- Run `/kiro-spec-design $1 --gap` to analyze the implementation gap with the current code before designing
- Identifies existing components, integration points, and implementation strategy
- Recommended for brownfield projects; skip for greenfield
- Then `/kiro-spec-design $1 -y` to proceed to design phase
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to requirements coverage, design boundaries, runtime prerequisites, or team conventions that affect task executability
description: Manage .kiro/steering/ as persistent project knowledge
description: Manage .kiro/steering/ as persistent project knowledge, including custom steering documents for specialized contexts
metadata:
shared-rules: "steering-principles.md"
---
@@ -11,9 +11,12 @@ metadata:
<background_information>
**Role**: Maintain `.kiro/steering/` as persistent project memory.
**Core steering in this repository**: `overview.md` (what the project is and what is actually true of it, dated current state), `roadmap.md` (milestones and status), `safety-net.md` (product direction). Custom files may exist beside them (for example a domain or standards file); all `.kiro/steering/*.md` are loaded as project memory. The generic bootstrap templates under `.kiro/settings/templates/steering/` are named `product.md`, `tech.md`, `structure.md`; they are starting points for a fresh project, not this repository's file names.
**Mission**:
- Bootstrap: Generate core steering from codebase (first-time)
- Sync: Keep steering and codebase aligned (maintenance)
- Custom: Create specialized steering documents beyond the core files (optional, see below)
- Preserve: User customizations are sacred, updates are additive
**Success Criteria**:
@@ -27,8 +30,9 @@ metadata:
Check `.kiro/steering/` status:
**Bootstrap Mode**: Empty OR missing core files (product.md, tech.md, structure.md)
**Sync Mode**: All core files exist
**Bootstrap Mode**: Empty OR missing core files (here `overview.md`, `roadmap.md`, `safety-net.md`; in a fresh project the template names `product.md`, `tech.md`, `structure.md`)
**Sync Mode**: All core files exist
**Custom Mode**: The user asks for a specialized steering document (see Custom Steering below)
---
@@ -75,6 +79,20 @@ After all parallel research completes, synthesize patterns for steering files.
---
## Custom Steering (optional)
Create a specialized steering document for one domain beyond the core files.
1. **Ask the user** for the domain/topic (e.g. "API standards", "testing approach") and the specific patterns to document
2. **Check for a template** in `.kiro/settings/templates/steering-custom/`: `api-standards.md`, `testing.md`, `security.md`, `database.md`, `error-handling.md`, `authentication.md`, `deployment.md`. Load the matching one as a starting point; without a template, generate from domain knowledge
3. **Analyze the codebase** (JIT) for the domain's patterns with Glob/Grep/Read (template and principles in parallel with the domain analysis when multi-agent is available)
4. **Generate** the document: follow the template structure if any, apply `rules/steering-principles.md`, patterns not exhaustive lists, one domain per file, 100-200 lines (a 2-3 minute read), never secrets
5. **Create**`.kiro/steering/{name}.md`; ensure it does not duplicate core steering content
Custom files are loaded as project memory like the core files and carry equal weight.
---
## Granularity Principle
From `rules/steering-principles.md` (in this skill's directory):
@@ -106,9 +124,9 @@ Chat summary only (files updated directly).
✅ Steering Created
## Generated:
- product.md: [Brief description]
- tech.md: [Key stack]
- structure.md: [Organization]
- overview.md: [What it is, current state]
- roadmap.md: [Milestones, status]
- safety-net.md: [Product direction]
Review and approve as Source of Truth.
```
@@ -118,8 +136,8 @@ Review and approve as Source of Truth.
✅ Steering Updated
## Changes:
- tech.md: React 18 → 19
- structure.md: Added API pattern
- overview.md: Current state dated, retired component removed
- roadmap.md: Milestone status updated
## Code Drift:
- Components not following import conventions
@@ -128,6 +146,20 @@ Review and approve as Source of Truth.
- Consider api-standards.md
```
### Custom:
```
✅ Custom Steering Created
## Created:
- .kiro/steering/api-standards.md
## Based On:
- Template: api-standards.md
- Analyzed: src/api/ directory patterns
Review and customize as needed.
```
## Examples
### Bootstrap
@@ -136,7 +168,12 @@ Review and approve as Source of Truth.
### Sync
**Input**: Existing steering, new `/api` directory
- **overview.md**: What the project is and what is actually true of it; a dated current-state section wins over older history
- **roadmap.md**: Milestones, their measured status, and the accepted order of work (not a task list)
- **safety-net.md**: Product direction and acceptance criteria (not feature catalogs)
- **Custom files**: Specialized patterns (API, testing, security, etc.); they may exist beside the core files and carry equal weight
(The generic bootstrap templates `product.md`, `tech.md`, `structure.md` under `.kiro/settings/templates/steering/` cover purpose, key frameworks and organization patterns for a fresh project.)
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to the validated boundaries, runtime prerequisites, integrations, domain rules, security/performance constraints, or team conventions that affect the GO/NO-GO call
**Discover canonical validation commands**:
**This repository's gates** (`.github/workflows/ci.yml`; run them as written, judge each by its own exit code):
- `TEST_COMMANDS`: `pytest -q -p no:cacheprovider < /dev/null; echo "exit=$?"` -- judge ONLY by the printed exit code, never by a summary line; `ruff check src tests`; `ruff format --check src tests`; `mypy src/openultrasast`
- Detection gates: `python -m openultrasast.gate` (>=90% recall, <10% FP per language; hard, score-independent), `python -m openultrasast.map_gate`, and the pair gate `python -m openultrasast.pair_gate` (local vuln vs fix). A ruleset change that fails a gate is NO-GO.
- Docs: `mkdocs build --strict` when `docs/` or `mkdocs.yml` changed
- `SMOKE_COMMANDS`: `ousast --help`, and for a scan change `ousast scan <fixture> --mode quick` whose `manifest.json` names the stages that ran and the files preprocess kept. A plausible zero (0 findings, 0 degradations) from a tool that never read its input is a failed instrument, not a pass: prove the tool read its input first.
**Discover canonical validation commands** (for other repositories, or anything the list above does not cover):
- Inspect repository-local sources of truth in this order: project scripts/manifests (`package.json`, `pyproject.toml`, `go.mod`, `Cargo.toml`, app manifests), task runners (`Makefile`, `justfile`), CI/workflow files, existing e2e/integration configs, then `README*`
- Derive a feature-level validation set for this repo: `TEST_COMMANDS`, `BUILD_COMMANDS`, and `SMOKE_COMMANDS`
- Prefer commands already used by repo automation over ad hoc shell pipelines
5. Reject claims that are broader than the evidence.
6. If mandatory validation cannot be completed, return `MANUAL_VERIFY_REQUIRED`.
7. Only then allow the claim.
7. Verify the instrument before you believe the measurement: prove the tool read its input (it opened one file and printed the size; the stage list names what ran). A plausible zero -- "0 findings, 0 errors, exit 0" from a tree the tool never read, or a Joern script that "finished" before a JVM could start -- is a failed instrument, not evidence. For pytest, judge only by the printed exit code (`pytest -q -p no:cacheprovider < /dev/null; echo "exit=$?"`), never by a summary line.
8. Only then allow the claim.
## Claim-Specific Rules
@@ -116,6 +117,7 @@ Return `NOT_VERIFIED` when:
| “Build should be fine because lint passed” | Lint does not prove build success. |
| “Tests passed and build succeeded, so it must run” | Type erasure, module loading, native ABI, and boot-time config issues can still fail at runtime. |
| “The feature is done because all tasks are checked off” | `FEATURE_GO` also requires coverage, integration, and design alignment. |
| “The scan found nothing, so the code is clean” | An unread input and clean code both report zero. Prove the tool read its input first. |
description:OpenUltraSAST scan workflows - `ousast scan` in quick, standard or deep mode, the evidence ladder, `--fail-on` exit codes, recorded degradations, the run directory and `ousast mcp`. Use when running or planning a scan, reading a run directory, or wiring OpenUltraSAST into an agent.
---
# OpenUltraSAST Scan
Use this skill to run `ousast scan PATH --mode quick|standard|deep` from an agent and to read what
it wrote.
## Workflow
1. Confirm the target is a local checkout (a path, not a URL).
2. Run through the project entry point: `uv run ousast ...` inside the repository, or `ousast` from an
environment where it is installed. Never activate a virtualenv by hand.
3. Start with `--mode quick` unless the task needs the Joern model layer (`standard`) or sandboxed
regression snippets (`deep`). Check the prerequisites below first; a mode whose engine is missing
still exits 0 and records a degradation.
4. Read the printed `scan_id=` and `run_dir=` lines; the run directory is `.openultrasast/runs/<scan-id>/`.
5. Report the finding count, how many the verifier accepted, and **every** recorded degradation
(`manifest.json`, key `degradations`). No findings plus a degradation is not a clean scan: the scan
answered a smaller question than it was asked.
6. Never call a finding verified from its title or severity. `verification.json` holds the status.
## Commands
```bash
uv run ousast scan <repo> --mode quick
uv run ousast scan <repo> --mode standard --fail-on verified
uv run ousast scan <repo> --mode deep --fail-on worth-fixing
uv run ousast scan <repo> --mode quick --config openultrasast.toml
uv run ousast mcp # the MCP server over stdio, for agent hosts
| `standard` | the MAP stage: complexity map, semantic overlay, authorization obligations, the Joern model layer (one code property graph per repository; taint, guard-dominance and configuration questions), deterministic fusion | Joern (shipped in the Docker image, with `php-cli` for PHP); a provider key for the residual LLM question | `cpg_unavailable`; without a key the LLM question is not asked and only the graph's entailed findings are reported |
| `deep` | REGRESS: promoted candidates run as snippets in a Docker sandbox (no network, read-only source, non-root) and get a `triggerable` / `not_triggerable` / ... verdict in `verdicts.json` | a working `docker` | `sandbox_unavailable` |
Model stages read `DEEPSEEK_API_KEY` (every LLM call); with only `OPENROUTER_API_KEY` set, chat
calls fall back to OpenRouter. An exported shell variable wins over `.env` silently. `[models]
hunter` in the config enables the tool hunter in `standard`; it is off by default.
## Exit code: `--fail-on`
| Policy | Exit 1 when |
| --- | --- |
| `never` (default) | never |
| `findings` | any finding was reported |
| `verified` | any verification result is `verified` (accepted at `static_corroboration` or above) |
| `worth-fixing` | a deep-mode sandbox verdict is marked `worth_fixing` |
A degradation never changes the exit code. Read `manifest.json` for it.
## Recorded degradations
Each entry in `manifest.json``degradations` is `{"stage": ..., "reason": ...}`. The ones to expect:
-`cpg_unavailable` (stage `model`): no Joern on the host; the model layer ran nothing.
-`sandbox_unavailable` (stage `regress`): no working `docker`; deep mode fell back to standard.
-`learning_endpoint_unavailable` (stage `model`): no provider key resolved. The graph's entailed
findings are still reported; the residual question is skipped. This is recorded in `manifest.json`
but has no line in `report.md`'s "What could not be analysed" section.
-`hunter_model_unavailable` (stage `map`): `[models] hunter` is unset, so the tool hunter did not run.
Recorded on every standard scan without that setting; it is expected, not an error.
- coverage facts from the graph, rendered in `report.md` under "What could not be analysed":
`region_failed`; plus `max_findings_exceeded` and `facts_unavailable` (overlay, obligations).
Prove the instrument read its input before believing a zero: `manifest.json` names the stages that
ran and the files the preprocess step kept (`preprocess/file_targets.json`).
## Evidence ladder
`src/openultrasast/verification.py` defines the levels in this order. A level moves one rung at a
time and never backward.
| Level | Meaning | Produced today by |
| --- | --- | --- |
| `suspicion` | a rule, model or weak signal raised a hypothesis | the Joern model layer (its rung, `model_entailed` or `suspicion`, travels in the finding's tags), the tool hunter, authorization-obligation checks |
| `static_corroboration` | deterministic code evidence at the line | quick rules, the semantic overlay |
| `crash_reproduced` | an artifact-backed reproducer | nothing: there is no crash reproducer. Deep-mode sandbox verdicts (`verdicts.json`, `worth_fixing`) do not raise the level |
| `root_cause_explained` | independent verifier reasoning | nothing: the LLM judge was retired in v2.0.0 |
| `exploit_demonstrated` | a demonstrated exploit | nothing |
| `patch_validated` | relevant checks re-run after a patch | nothing: there is no patch stage |
The verifier's status per finding (`verification.json`): below `static_corroboration` is
`needs_evidence`; `static_corroboration` with reachability `unknown` is `rejected` (the tie-breaker
asks for call-graph, route, CLI, parser or dynamic evidence); `static_corroboration` with
function-level reachability is `accepted`. `verified` means accepted at `static_corroboration` or
above, and that is what `--fail-on verified` counts. There is no configurable report threshold: the
`[evidence]` config section was retired in v2.0.0 and is ignored.
## Run artifacts
Under `.openultrasast/runs/<scan-id>/`:
-`report.md`, `report.sarif` (SARIF 2.1.0 for code hosts and IDEs), `manifest.json` (stages, degradations, model payload)
-`findings.json`, `verification.json`, `score.json`, `shadow_findings.json` (rules shipped as `shadow`)
-`harness.json`: the resolved harness configuration the run used
description:OpenUltraSAST finding triage - false-positive adjudication with the fixed reason taxonomy, the evidence ladder, what a verdict may feed (labels, never rule edits), and fixing then re-scanning. Use when reviewing findings, reducing false positives, deciding whether evidence is enough, or auditing a fix for a finding.
---
# OpenUltraSAST Triage
Use this skill when triaging OpenUltraSAST findings or auditing a fix for one.
## Triage Loop
1. Identify the finding: id, path, rule, CWE and policy severity, confidence, evidence level and
verification status (`findings.json`, `verification.json` in the run directory).
2. Separate deterministic evidence (rule match, graph entailment, attached reachability) from model
Order follows the dependency chain: ax bring-up (parallel with 1-2) -> manifests -> budget -> runner ->
ax-backed reconciler -> tasks -> increment run on ax -> measurement. Every task ships with its tests; a task is done when its tests pass in the host suite, ruff and
mypy are clean, and the commit is gated on pytest's own exit code. Script-level pipeline work stays paused.
- [x] 0. ax bring-up on this host
- Install `kind`, `kubectl`, the `ax` CLI and `ko` (user-local). `ops/ax/up.sh`: create the kind cluster and
install Agent Substrate with its `hack/create-kind-cluster.sh` and `hack/install-ate-kind.sh
--deploy-ate-system`, deploy the ax control plane with `ko` to the kind registry, apply one trivial Task
and wait for it to finish; `ops/ax/down.sh` tears it down. Record the measured idle footprint (RSS, CPU)
in `ops/ax/README.md` next to the 7 GB host constraint.
-`ousast plane doctor` (in `reconciler.py` or a sibling `doctor.py`): the four checks of Requirement 3.5.
starts without a request. `plane/Dockerfile.runner`: this package on Python 3.12, installed to
`/usr/local/bin/ax-task-runner`, loaded into kind by `ops/ax/up.sh`.
-`tests/test_plane_runner.py`: the env contract on a fixture workspace; readyz 503 then 200; delivery to a
local receiver; delivery failure yields `failed`; the child's cwd and `AX_METADATA_URL`; the runner stays
up after the command; SIGTERM kills a long-running command's group; the marker prevents a rerun; no command
before a start request, 409 on a wrong run/task and on a second start, the credential reaches the child
(its hash in the artifact) and no file or log record, a crash printing it is redacted, and of two runners
with identical env only the addressed one runs (the golden scenario).
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6_
- [x] 4. ax-backed reconciler and CLI
-`src/openultrasast/plane/reconciler.py` (under 500 lines, no pipeline logic, no local subprocess
executor): `state.json`, statuses, topological submission through the `ax` CLI (path injectable),
`ax resume task` after `ax apply` (a created Task is `Suspended`), retrying DeadlineExceeded/Unavailable
with backoff up to `OUSAST_RESUME_TIMEOUT`; completion is the artifact delivery, never an ax phase (ax has
none for a finished command); `Failed` or a vanished task before delivery, or no delivery within
`OUSAST_TASK_TIMEOUT`, fails the task with ax's condition message; the artifact HTTP receiver, `serialize` labels never overlapping, `--workers`, budgets and
artifact env passed in the rendered Task manifest, outcome read from the delivered `summary.json`, skip
`done` on rerun, PID lock per run, `ax delete` of finished Tasks. Once ax reports `Running` after a
resume, one start request per resume through Agent Substrate's router (`src/openultrasast/plane/router.py`:
`kubectl port-forward svc/atenet-router` for the run, or `OUSAST_ROUTER_URL`; header
`ate-target-actor: <atespace>/<task>`; 502/503/504 and connection errors retried up to
`OUSAST_START_TIMEOUT`, a 409 final), carrying the bound Model's `secretKey.key` variable from the
reconciler's environment; a Model-bound task whose variable is unset fails before `ax apply`.
Task-produced inputs travel as `OUSAST_INPUTS`, not as a rendered Workspace: the runner fetches them after
the start from the receiver's authorised `GET /inputs/<producer>/<artifact>` (a 182 KB `facts.json` in
`AX_WORKSPACES_YAML` exceeded Substrate's 32768-character env limit on the live cluster, 2026-09-29).
-`ousast plane run <Run.yaml>`, `ousast plane status <run>` (token attribution table per task and Model
from the tasks' `summary.json` usage; `--units` from `units.jsonl`; also written to `attribution.json`),
`ousast plane workspaces <population.toml>` (generates Workspace manifests per case pin).
-`tests/test_plane_reconciler.py`: a fake `ax` CLI with ax's real lifecycle (Suspended, a first resume
failing DeadlineExceeded, Running without a Completed phase) drives a DAG: order, non-overlap,
skip-on-rerun, `unfinished` and `failed` propagation, resume retry and timeout, `Failed` with a condition
message, the task timeout without delivery, the lock, the attribution table, and the
line-count assertion; with a fake router: one start per task after the accepted resume and before the
delivery, the credential in the verify start only and in no file under the run or tmp dirs, a missing
secret failing the task, a refused start failing it. `tests/test_plane_router.py`: port parsing from a
fake kubectl and its termination on exit, retry on 503/502 then 202, 409 raised with redacted text, the
start timeout. `tests/test_plane_ax_live.py` (marker `ax`, skipped without a live cluster): one
> **Status 2026-10-02:** dormant. Tasks were approved but none of the 24 was started; the absence-bug case it targets is handled in practice by the dominance arbiter (`model/dominance.py`) and `semantic/obligations/`. Reopen it only with a measured gap the arbiter misses.
## Introduction
AppSec engineers and maintainers of OpenUltraSAST scan web code, increasingly vibe-coded or agent-written, in which the dominant bugs are absences rather than flows: a handler queries a user-owned record without constraining it to the caller, a privileged route has no guard on its path, an identity is read from the request body instead of the authenticated context, a security control stays at its permissive default. Today the detector, the overlay, and the variant search model only source-to-sink flows, so these pairs score zero and cannot teach a mechanism (6 of 19 non-teaching vibe-py pairs; 15 of 29 agent-vfc rows carry absence labels). This spec adds obligations: an operation that must be discharged by a guard, an identity constraint, or a safe value before it is reached. The application's own policy, recovered from the consistency of its sibling handlers and optionally declared in a versioned file, says which operations carry which obligations. Violations are reported with the missing discharger and its evidence, measured leave-one-out on the absence pairs, and admitted into the detector through the existing improve-loop gate.
> **Status 2026-10-02:** superseded by `model-grounded-detection` (spec.json `superseded_by`). 4 of 20 tasks were done and the tasks were never approved; none will be. Kept as decision history; nothing in src, tests or CI points here.
## Introduction
The detector's instability is structural, not statistical. Measured on 2026-09-06 over twelve injection pairs at
| Evaluation and eligibility | Frozen identities and populations; unchanged 30s/2s and precision/recall/completion gates; no capability enabled by a microbenchmark |
Production changes require task-local review and fresh ordinary scan gates, runtime smoke,
and downstream acceptance evidence. This research record is not that verification.
> **Status 2026-10-02:** paused since 2026-09-23 (its last change). 27 of 51 tasks done: the MAP integration was delivered and `v1.2.0-alpha.1` rolled out NO-GO (`census-repair-verification.md`); the remaining tasks wait on `pre-push-safety-net` M2/M3 and nothing here moved after the ax plane landed. Not active.
## Introduction
`model-grounded-detection` built an arbiter and measured it: 41.3% of the corpus reaches `model_entailed`
first-party tests. The maintainer approved its dependency adaptation and acceptance checks;
task 2.18 now implements it before downstream partition completion.
## Group 7 — Grow the ontology by measurement, not by subject (2026-09-21)
Named by the family detection census and the week of repairs after it. Five of seven families
established nothing and every one failed for an ontology reason, so this group is where detection
efficiency actually lives. The design amendment above states the constraint these tasks work under:
the ontology describes libraries, frameworks and language constructs, never repositories.
- [ ] 7.1 Report what a repository calls that the ontology does not model
- Census the call names, and the receiver-qualified spellings, that appear in a repository's first-party graph; subtract everything any fact table names; rank the remainder by how often it is called and by how close it sits to a modelled source. The output is a measured worklist of what this tool is blind to here.
- It needs no dataflow, which is why it is cheap: the same structural pass that answers 17,700 requests in 44 s already walks every call.
- **It reports candidates, not entries.** A candidate becomes ontology only when someone resolves it to a library API, states the weakness class and records provenance. Nothing may be added because a subject called it.
- Three defects this would have named in one run rather than three investigations: `needle.get` absent while the family found the redirect two files away, `find` absent so the documented object-reference defect raised no obligation, and `maybe_unserialize` absent so the only deserialization site in 1,164 PHP files was invisible.
- Done when the instrument reports a ranked gap list for each measured subject, every entry carries how often it is called, and adding nothing to the ontology changes no result.
- _First run 2026-09-21 on NodeGoat:_ 18 of 175 distinct call names modelled, covering 12% of 1,185 calls. The top product candidate is `res.render`, 23 calls with five in a method reading untrusted input — the same gap task 14.3 reasoned its way to by hand, found here in one run, which is the validation that matters. It also surfaced `Math.random` in the user data-access module: weak randomness is a weakness class this ontology has no family for, so no census could have found it, because the family list is itself the blind spot.
- _It had to be taught what is not an API._ The first run ranked `_tmp_3.push` and `__ecma.Array.factory` first, which is how the frontend lowers a destructuring and an array literal, and a Cypress suite and a Gruntfile held four of the next five. Both are now separated, because noise at the top of a worklist is how a worklist stops being read. Evidence: `benchmarks/measurements/2026-09-21-ontology-coverage-nodegoat.json`.
- [ ] 7.2 Key sanitizers to the weakness they actually prevent
- The sanitizer list is flat across families, and that forces a false choice which is now measured: adding WordPress's `esc_attr` removes eighteen output-encoding false positives on one plugin and simultaneously hides a real SQL injection, because no HTML escaper stops SQL injection. `php.toml` has carried a comment predicting exactly this since before the measurement existed.
- Key each sanitizer to the weakness class it discharges, so a family consults the cleansers that apply to it. Do not solve it by choosing which defect to keep.
- Done when an HTML escaper clears an output-encoding flow and leaves an injection flow standing, on the measured plugin, with both outcomes recorded.
- _Boundary: shipped semantic facts_
- _Requirements: 6.2, 8.1_
- [ ] 7.3 Do not arbitrate a question whose evidence says there is nothing to find
- The structural pass answers 17,700 requests in 44 s; arbitration costs one to two seconds per question, and a 673-file plugin asks 2,526 of them. Gate arbitration on the evidence vector already computed: a question whose scope holds no modelled sink, or no reachable source, cannot produce a flow and does not need the expensive pass.
- This attacks the scale limit from the opposite side to graph reuse and can be sized today from data already on disk, by counting how many of those 2,526 questions had any sink in scope at all.
- Done when the arbitrated count falls without any finding being lost on the measured subjects, and the questions skipped are recorded as skipped rather than as answered.
- _Implemented and measured 2026-09-22, and the mechanism already existed._ Tier-zero pruning defers a question whose evidence vector says no sink of its family is in reach. It was switched off for the WHOLE SCAN whenever any of twelve degradation reasons appeared, and a multi-language repository always carries two of them, so the measured plugin arbitrated every question it could ask. Third instance this week of the same blast radius. A prune blocker that names a partition now withholds pruning from that partition's questions and nobody else's, and `cross_partition_semantics_unresolved` leaves the blocker set entirely for the reason it left the integrity gaps.
- _The arbitrated count falls and no finding is lost._ At 25 regions: 127 questions to 34, with 93 recorded deferred as `tier_zero`, and 19 findings before and after, identical family by family. At 500: 2,526 to 406, with 2,120 deferred. A family whose every question was pruned now reports "no sink of this family in reach" rather than "region offered, no question asked", because nothing to ask is a different statement from never offered.
- **It is not sufficient, and that is the useful half of the result.** The 500-region run still fails in the same way at the same cost, because 406 questions at roughly one and a half seconds each still exceed the 300-second per-batch ceiling. This lever moved the volume; the ceiling is a separate limit and pre-push 13.3 still cannot reach the population that holds its target without it. Evidence: `benchmarks/measurements/2026-09-22-tier-zero-pruning.json`.
- _Boundary: contributor-scan scan budget_
- _Requirements: 4.1, 6.3, 8.1_
- [ ] 7.4 One matcher, used by every query
- Four query files carry near-copies of the same token matcher and each acquired the same defect separately: the configuration matcher needed a receiver check the dominance matcher already had, and the sink matcher searched a whole node's text including its comments. Six instances of one bug class are listed in the pre-push design's invariants.
- Extract one matcher with callee confinement and word boundaries built in, and have taint, dominance and config use it. A rule in a document gets rediscovered; a rule in a constructor does not.
- Done when one implementation is shared, each caller's behaviour is unchanged on the measured subjects, and a new query cannot match text that merely contains a name.
- _Boundary: contributor-scan queries_
- _Requirements: 6.2, 8.1_
- [ ] 7.5 A degradation must declare the scope it reaches
- Five instances of one bug class are listed in the pre-push design's invariants, two of them found this week, and two of those cost the declared target its entire admissibility. A degradation is currently a bare dict, so nothing forces a new reason to say whether it reaches one question, one file, one partition or the whole read.
- Make it a record with a required scope, and let the demotion and the admission policy read that field rather than a hand-maintained membership set.
- Done when a degradation cannot be constructed without a scope, the existing reasons carry theirs, and the measured subjects' outcomes are unchanged.
- [ ] 7.6 Golden findings per subject, so a silent change cannot pass
- Nothing fails today when a change alters what NodeGoat or VAmPI report. Four changes in one week altered detection and each time the census had to be re-run from memory; one false positive was caught by reading a witness rather than by a test.
- Commit an expectation per subject at the finding level, site and family and rung, and diff against it. Keep it honest about what it is: a regression guard on development subjects, never evidence of qualification.
- Done when a deliberate change to a matcher or a fact table fails the diff, and the expectation records the ruleset and engine identities it was taken under.
- _Boundary: evaluation harness_
- _Requirements: 8.1, 8.2_
- [ ] 7.7 Budget a batch by the sinks it implies, not by the requests it holds
- **Named by the 2026-09-22 identification.** A taint batch costs what its matched sink NODES cost, because `reachableByFlows` is asked per sink. Measured at a fixed 100 requests on one graph: SQL injection 147 s, path 194 s, output encoding 704 s — and output encoding declares seven sink names against path's sixteen. What separates them is that `echo` and `print` match thousands of calls in a WordPress plugin.
- The driver currently splits a failed batch into eight parts by request count, which cannot help: each part carries the same everywhere-sinks.
- The count needed is already computed. `taint.sc` memoises its candidate sink list per sink set, and the evidence pass reports the in-scope sink list per request before any dataflow runs. A driver that reads it can size a batch by expected work, and can report a family whose sinks are everywhere as EXPENSIVE rather than as failed.
- Done when a batch is sized by the sink nodes it implies, the measured plugin's arbitration completes within its ceiling, and a family too expensive to arbitrate is reported as such rather than as a timeout.
- _Implemented and measured 2026-09-22._ On the 500-region population where every earlier run produced nothing: **153 of 406 questions completed and 125 findings, inside the deadline**, against 0 and 0. 380 requests carrying 5,807 sink visits split into 11 batches, each answering in 80 to 197 s, none near the 300 s ceiling. Every family that can speak on this subject now does: injection 49 completed and 24 findings, path 64 and 15, output encoding 32 and 70, untrusted destination 8 and 16. Eight batches were left unasked to keep time for arbitration, and that is recorded rather than silent.
- _Four defects, each hidden by the one before it, and three of them the same rule._ Batches sized by request count when cost is per matched sink. Packing heaviest-first, which starved injection — the one family with a true positive here — of all 114 of its questions. Any per-query timeout cancelling the whole scan, abandoning batches with 33 minutes of budget left. And no budget kept for arbitration, which removing that cancellation exposed: rows are not findings, so the query phase ran the full deadline and completed nothing, where the cascade had been an accidental guard.
- _The reserve had to be checked per BATCH._ Checked between query kinds it changed nothing at all, because one kind is a single loop iteration that runs every one of its batches inside it. The granularity of a budget check has to match the granularity of the work it guards. Evidence: `benchmarks/measurements/2026-09-22-sink-weighted-batching.json`.
- _What it does not claim:_ 253 of 406 questions still went unanswered and one batch still records a timeout; the raw finding count is dominated by output encoding, the family already measured to be imprecise on PHP; and the reserve share of a fifth is a first value, not a tuned one.
- [ ] 7.8 Portion the work instead of capping it: a kill keeps what finished
- **Asked for by the maintainer on 2026-09-22, after 7.7.** A hard per-invocation ceiling is all-or-nothing: a batch killed at 301 s loses every answer it had already computed, and the retry then pays a fresh JVM start per part to ask them all again. Measured on the plugin: 116 requests killed at 300 s, eight restarts, and the same everywhere-sinks in every part.
- Three coordinated changes replace the cap as the thing that governs cost. The taint query STREAMS: the census first, then one line per request as it is answered, flushed as it goes, inside the unchanged fence. The driver harvests a killed process's stdout and assembles every complete line, marking the result partial; the retry then asks only the remainder. And portion sizing ADAPTS: after each portion the observed seconds per sink visit, less the fixed start, resizes the rest toward a target duration, and a portion that did not answer halves the budget rather than being fitted. The ceiling stays as a safety net that the sizing is meant to make rare.
- Legacy single-document payloads must keep parsing unchanged; the other queries do not stream and need not. A census line is one whose only key is the census — a first cut misread a legacy document that carried the census beside its answers as a census line and dropped every answer it sat beside, which a control caught.
- Done when a killed taint batch hands back the answers it finished and the retry asks only the rest; portion sizes follow observed cost; the streamed and legacy forms both parse; and the plugin population completes more questions per unit of budget than under 7.7 with no finding lost.
- _Status 2026-09-22:_ implemented; host controls pass for the assembler in both forms and under a kill, for the remainder-only retry, and for the sizer; the joern-gated check passes on the real engine — the taint query streams one line per request, and the comment-sink and literal-destination controls hold under the streamed form. The plugin measurement is running on a frozen export of the committed source.
- _Two things the measurement exposed on the way, both fixed:_ the evidence pass was one unportioned batch protected only by the ceiling — 22,125 requests on a plugin push transaction — and now runs in portions of 4,000; and a first cut of the harvest read a timed-out process's pipe BEFORE killing it, which blocks forever because a pipe whose writer is alive never reaches end-of-file. The suite's own hanging-engine control caught that after twenty minutes of what I mistook for CPU starvation.
- _A stated limit:_ the reuse layer's re-query of `missing` ids calls the backend batch directly and so bypasses sink-weighted portioning. Those re-queries are the small remainder of a cached scan, and streaming still keeps what a killed one finished, so it is recorded here rather than fixed. _Corrected 2026-09-23:_ the limit was stated backwards and was worse. The reuse layer wrapped only `run_batch`, and the portioner asks `run_batch_once`, so every portioned query bypassed the reuse cache — and its source-root normalization — entirely. Fixed under 7.9: both paths go through the cache.
- _End-to-end tuning, 2026-09-23, the honest version._ The first end-to-end run REGRESSED completion (64 of 406 against 7.7's 153) while finding as many defects, and each fix exposed the next: the adaptive sizer read one portion at 953 s because it timed `run_batch`, whose own split-retry folded in, and collapsed; giving the portioner a single-attempt path (`run_batch_once`) fixed the timing but only after it was wired on the PARTITIONED graph too, which every plugin uses and which the first wiring missed; and even then completion sat at 56 of 406 because the portioner iterated a fixed list and dropped the remainders of killed portions, stopping at 1,263 s of a 3,600 s budget. The queue re-asks those remainders. The sizer now adapts both ways in the trace — 268 up to 535 after a cheap portion, halved to 267 to 133 on kills — and re-asks what a kill left. Each step is unit-tested: the assembler in both forms and under a kill, the single-attempt path, coverage-halving, and queue termination. _Final end-to-end 2026-09-23, mixed and told straight._ Against the hard-cap-plus-reserve baseline on the same 500-region PHP run: findings IDENTICAL at 125 either way, completion slightly LOWER under portioning (131 of 406 against 153), wall time higher (2,982 s against 2,338). Portioning does not beat the cap on throughput here; it matches it on what it finds. What it buys is kill-safety -- the maintainer's actual ask -- and reaching the same 125 findings is the evidence that streaming loses nothing the cap kept while no longer discarding a near-complete batch when the ceiling falls mid-answer. Completion did not rise because the requeue re-asks a killed portion's remainder, and on this subject those are the expensive output-encoding questions that keep hitting the ceiling in smaller pieces; telling a transient kill from a genuinely-too-expensive family is the next lever, not more portioning. Neither approach is release-ready regardless: 32-38% completion against a 95% gate, on development data that qualifies nothing. Evidence: `benchmarks/measurements/2026-09-23-portioning-vs-hard-cap.json`.
- [x] 7.9 Size portions from what each answer cost, and tell a transient kill from an expensive question
- **The next lever named by 7.8, and the trace said it was a different lever.** Re-read on 2026-09-23, the 7.8 run's sizer did not lose completion to expensive families so much as to a death spiral: after two kills the budget fell to 3 sink visits per portion and never recovered. Portions of 3–6 visits finished in 25–30 s, which under the assumed 70 s fixed start read as "no information", so the budget could shrink and never grow — and the rest of the run paid a JVM start per handful of requests.
- The instrument first: the taint query streams each answer's own cost (`ms`), carried as a reserved `__timing__` list through shards, partitions and reuse. The sizer then MEASURES what it used to assume — the fixed start as the wall time the answers do not explain (median over whole portions), and a seconds-per-sink-visit rate per family from that family's own answers. Portions are cut in predicted seconds and grow request by request from the head of the queue.
- A kill names its culprit. Answers stream in request order, so the first unanswered id was running at the kill and every id after it never started: those go straight back to the queue, the culprit is asked alone after everything else, and its family is costed at the lower bound it proved. A request that fails alone after more than a portion's target is `query_too_expensive`, reported apart from `query_failed`.
- Two defects fixed on the way: a scan small enough for one portion took a single attempt and lost whatever a kill cut off (it now goes through the queue); and requests the arbitration reserve held back were reported as `query_failed` although they were never asked (now `query_budget_reserved`).
- _Measured 2026-09-23, same 500-region PHP run, 3,600 s, frozen at `1b28465`:_ **281 of 406 questions completed, against 153 (hard cap) and 131 (7.8)**, in 2,987 s. Findings per family never fell: injection 24→33, output encoding 70→125, path 15→35, untrusted destination 16→31; 224 in total against 125. The declared target `classes/class.memberorder.php:936:getMemberOrderByCode` is found at `model_entailed`. The measured fixed start was 22–24 s throughout. **No portion was killed in the whole run** — every one answered whole — so the gain is the sizing, and the culprit path is covered by host controls but was NOT exercised end to end here.
- _Limits, stated:_ one subject, one budget. The census keeps ten examples per family, so it shows per-family counts did not fall, not that the earlier findings are a strict subset. The extra findings are unadjudicated: output encoding on PHP `echo` is the family most likely to carry false positives, and 224 is a count, not a precision. 99 taint questions were still held for arbitration, and access control, config and prototype were never reached (0 of 39) — the ranker puts them last, and the reserve cuts the tail. Completion is 69%, still short of the 95% M4 gate; this is development data and qualifies nothing. Evidence: `benchmarks/measurements/2026-09-23-measured-portioning.json`.
> **Status 2026-10-02:** complete, 17 of 17 tasks (spec.json phase corrected to `implementation-complete`). Its `[variants]` config keys were retired on 2026-10-02: the variant search they configured (`semantic/variants.py`) had already been deleted.
## Introduction
OpenUltraSAST's labeled pair corpus measures the detector but does not improve it, and its mechanism memory is empty because only sandbox-proven findings may write to it. This spec turns every trusted pair into a mechanism record, adds a structural variant search that proposes matches on any scanned tree, measures the corpus's own detection rate leave-one-out, and gives the improve loop a lever that can raise recall under the existing false-positive ceiling and per-profile holdout gate.
> **Status 2026-10-02:** dormant. Requirements approved and design generated on 2026-09-09; the design was never approved and no tasks exist. It depends on `contributor-scan`, which is paused; reopen it with that spec.
## Introduction
Nine engine defects were found in one session against two repositories. **Three were the same bug in three
# Brief: flow-aware-ranking — spend the expensive query where the cheap evidence is strongest
> **Status 2026-10-02:** dormant. Brief only and unapproved; its 2026-09-12 amendment placed ranking inside the pre-push safety net, where the ranker now lives.
`src/openultrasast/improve/memory.py` (deterministic proposal rules). None of it is in the reconciler. The
reconciler gains only `mark_done` (below). It is 465 lines today, and `tests/test_plane_reconciler.py:759-761`
keeps it under 500.
## Components and Interfaces
### 1. Fate of each HarnessX capability (Requirement 1)
| Capability | Where today | Fate | What the user loses |
| --- | --- | --- | --- |
| LLM hunter pool (`HxScanOrchestrator`) | `hunter_harness.py:49-185`; gated in `cli.py:400-456` (`hx_hunter` at :405, orchestrator at :438-448, degradation at :452-455) | **Retired.**`run_hunter_pool` becomes the only standard-mode hunter pool. LLM hunting of a repository stays available in two places: the MAP-stage tool hunter in `ousast scan --mode standard/deep` with `[models] hunter` (`cli.py:631-640`, not HarnessX), and plane `verify` via `ousast plane run plane/runs/validation-46.yaml` | `hx-hunter:<path>:<line>` findings from an anthropic/openai/litellm agent (`hunter_harness.py:155`). No run of this path is recorded under `benchmarks/`, so no measured TP is lost. |
| LLM judge | `verify_judge.py:22-63`; gated in `cli.py:406, 499-511` | **Retired.**`verification.verify_findings` is called directly and `verify_judge.py` is deleted (it is only the dispatcher and the judge). The plane's LLM verification is `verify` + `agree` (+ `vc`/`final` tie-break) | Per-finding `plausible/unsupported` verdicts that set `ACCEPTED/REJECTED` (`verify_judge.py:83-88`). Unmeasured. `[models] judge` (model-grounded-detection, `model/endpoint.py:195`) is a different capability and stays. |
| LLM fusion panels | `fusion.py:194-326` (`_fuse_with_llm_panels`, `_run_llm_panels`); gated in `cli.py:519-541` | **Retired.** The deterministic panels (`_panel_a`, `_panel_b`, `_reconcile`, `fusion.py:101-167`) become the only path. `fuse_findings_dispatch` is renamed `fuse_findings(findings, verifications, *, high_assurance, mitigated_ids)`. The plane counterpart of independent cross-examination is `agree` across independent passes with the 2-of-3 tie-break | Steel-man LLM panels (`decision_source: "llm-panels"`). Unmeasured. `fusion.json` for deterministic decisions is unchanged. |
| Deterministic stages hosted under HarnessX | `stage_processors.py:129-156` (`host_under_harnessx`); only caller is the skipped test `test_slot_contract.py:119-133` | **Retired, no replacement.** ax runs whole tasks, so a model-disabled in-loop host has nothing to host. `stage_processors`/`slot_contract` stay as the standalone deterministic pipeline that `test_gate.py:147-148` pins | Nothing. The function has no production caller. |
| LLM evolution proposer (`MetaAgent.evolve`) | docstring only: `improve/evolve.py:8-9`, `harness_ext.py:7-8` | **Carried by the memory layer** (Requirement 6): `improve/memory.py` rules over the store, fed into `run_round` as extra proposals | Nothing implemented is lost. |
No CLI flag is HarnessX-specific, so Requirement 1.3 applies to config keys. A key that asked for a retired
capability makes `load_config` raise `RetiredConfigError(ValueError)`, and the CLI prints the message and exits 2.
A silent downgrade is not allowed:
| Key | Message (abridged) |
| --- | --- |
| `[models] verifier` | "`[models] verifier` was retired with HarnessX on <date>: LLM verification runs on the plane (`verify` + `agree`, `ousast plane run`, see ops/ax/README.md). Remove the key." |
| `[fusion] panel_model`, `[fusion] decider_model` | "... LLM fusion panels were retired; fusion is deterministic; independent LLM agreement is the plane's `agree` task ..." |
`[harnessx]` is different (Requirement 3.3). It only tuned budgets and never requested a capability, so it loads
with **one**`logging.warning` on logger `openultrasast.config` that names the plane ("`[harnessx]` is ignored:
HarnessX was removed; agentic work runs on the ax plane with per-task budgets, see ops/ax/README.md"). The section
is otherwise ignored. `[models] hunter` keeps its meaning (tool hunter, obligations intent adjudication at
`cli.py:1237`), so it gets no message. With the extra installed it no longer starts a second, HarnessX hunter. The
release notes say so.
Requirement 1.2 (moved capabilities have a CLI path, a Model per task and a fixture test) is met by the existing
`verify`/`agree` tasks (`ousast plane run`; Model `deepseek-flash` bound per Task; `tests/test_plane_verify.py`,
`tests/test_plane_scoring_parity.py`, `tests/test_plane_tiebreak.py`), and by the new memory and loop tasks below, each with its own fixture test. No
capability is moved into a new Model-bound task.
### 2. Removal mechanics (Requirement 2)
**Source.**
- Delete `harness_ext.py` (97 lines), `hunter_harness.py` (197) and `verify_judge.py` (130).
- `cli.py`: drop the imports at :44, :46 and :94. Drop `harnessx_present`, `hx_hunter` and `hx_verify` (:400-406),
the orchestrator branch and its degradation (:438-456), and the verify degradation (:499-502). Call
`verify_findings(findings)` at :503-511. Drop `hx_fusion` and its degradation (:521-527). Call
`fuse_findings(findings, verifications, high_assurance=config.fusion.high_assurance)`. The `verifier_model`
local goes away.
- `fusion.py`: drop the import at :24, the `panel_model`/`decider_model`/`provider`/`use_harnessx` parameters, and
`_with_degradation`, `_fuse_with_llm_panels`, `_run_llm_panels` and `_text` (:229-338). Rewrite the module
docstring (:10-14) and update `__all__`.
- `stage_processors.py`: drop `host_under_harnessx` (:129-156) and its `__all__` entry, and rewrite the module
docstring (:10-11). `slot_contract.py:7-14` becomes "the zero-dependency substrate for the deterministic
pipeline".
- `config.py`: drop `HarnessxConfig` (:99-107), `ResolvedConfig.harnessx` (:201), `_load_harnessx` (:345-351) and
its call (:230). Drop `ModelConfig.verifier` (:37), `FusionConfig.panel_model/decider_model` (:123-124) and
their loaders (:266, :366-367). Add `RETIRED_KEYS` and the one-warning rule above. `harness.py:183` keeps the
`"verifier": None` literal in `model_roles`, so `harness.json` keeps its schema byte for byte (the key has always
been `null` in the baseline config).
- Comment-only mentions: `gate.py:174` ("a model-backed stage"), and `improve/evolve.py:8-9`, which now names the
memory proposer.
- `plane/Dockerfile.runner:23` loses "no HarnessX".
**Packaging.**
- Delete the `harnessx` extra and its comment (`pyproject.toml:14-22`) and the `harnessx.*` mypy override and its
comment (:67-71).
- `uv lock` regenerates `uv.lock`. The `harnessx` package (:1062), the extra (:2145-2146), the requirement
(:2169) and `provides-extras` (:2178) go.
- Extras that remain: `semantic`. There is no `all` extra. `pip install .` and `pip install '.[semantic]'` in a
fresh venv are the Requirement 2.2 check.
- No source or test imports a HarnessX transitive dependency. The only third-party imports are `yaml`,
`tree_sitter` and `pytest` (checked with an import scan), so dropping the lock entry cannot break an import.
**Tests** (14 files plus `conftest.py`):
| File | Action |
| --- | --- |
| `test_harness_ext.py`, `test_hunter_harness.py`, `test_verify_judge.py`, `test_cli_hx_dispatch.py` | **Delete.** Each exercises only HarnessX or its dispatcher. |
| `conftest.py` | Rename fixture `assert_cold_of_harnessx` to `assert_cold_import(code)`: a fresh interpreter runs `code`, with no module-name guard. `test_semantic_extra.py:28-35` is its remaining user (the tree-sitter check brings its own assertion). |
| `test_gate.py` | Delete `test_harnessx_stage_stays_within_tolerance_of_baseline` (:124-150) and the `has_harnessx` import (:23). Replace the two "never import" tests (:155-175) with `test_governance_planes_import_only_stdlib_and_declared_dependencies`, an allow-list (stdlib, `openultrasast`, `yaml`) over module-level imports of `GOVERNANCE_SCORING_BENCHMARK_PLANES`. It covers any extra without naming one. Update the docstring at :5-6. |
| `test_scan_pipeline_integrity.py` | Rename `test_standard_scan_extra_absent_matches_deterministic_baseline` to `test_standard_scan_with_a_hunter_model_keeps_deterministic_findings` (config `hunter` only). Rename `test_quick_scan_is_unaffected_by_the_extra` to `test_quick_scan_records_no_degradation`. Drop the `has_harnessx` monkeypatch and the `harnessx` parameter. Recompute the expected degradation set from the post-removal run, since `hunter_pool`/`verify` entries no longer exist. |
| `test_slot_contract.py` | Delete `test_host_under_harnessx_disables_the_model` and the imports at :16 and :22. Keep the rest. |
| `test_config.py` | Replace `test_harnessx_defaults_when_section_absent` and `test_load_config_reads_harnessx_section` with `test_retired_agentic_section_loads_with_one_warning` (caplog: exactly one record naming the plane) and `test_retired_llm_keys_fail_naming_the_replacement` (verifier, panel_model, decider_model). |
| `test_fusion.py` | Delete `test_dispatch_records_degradation_when_llm_requested_but_extra_absent` (:142-146) and the import at :16. Rename the dispatch calls to `fuse_findings`. |
| `test_tool_hunter.py` (:229, :266), `test_scan_deep.py` (:144) | Drop the `has_harnessx` monkeypatch line. |
| `test_semantic_extra.py` | Use `assert_cold_import`. |
| `test_module_audit.py` | The subsystem tuple at :31 becomes `("fusion", "mcp", "skills")`. A new assertion checks that the three deleted modules are absent from the audit. |
**Module audit.** `model/audit.py:76-77` (`harness_ext`, `hunter_harness`) are deleted, and `:78`
(`stage_processors`) is reworded to "deterministic slot-contracted scan stages; pinned by the zero-dependency
guard in test_gate". Regenerate `benchmarks/measurements/2026-09-08-module-audit.json` in place with `to_dict(audit(SRC))`.
`test_the_manifest_matches_the_tree` pins that path. The diff shows three rows removed and no orphan.
**Search (Requirement 2.4).** `grep -rni harnessx src tests pyproject.toml README.md` may match only lines tagged
`retired 2026-..`:
- the `RETIRED_KEYS` entry and warning text in `config.py`
- the two retirement tests in `test_config.py`
- the one README line in the plane section
Requirement 3.3 cannot be implemented or tested without naming the section, so these lines count as retirement
notes. The check is a test (`test_no_harnessx_outside_retirement_notes`) that fails on any untagged match. It
builds its pattern at runtime, so it does not match itself.
### 3. Equality proof (Requirement 3)
The spec's "`ousast regress`" is not a subcommand (`cli.py:133-251`). Regress exists only as the deep-mode
sandbox stage. On this project the regression command over the development pins is `ousast pairs`
(README.md:38): the vuln-vs-fixed replay over local fixtures and vendored pinned VFC pairs, offline by default.
The benchmark replay is README.md:35's manifest. The baseline also covers the two paths the removal touches
directly: one standard-mode scan (verify and fusion) and one `improve --dry-run` (evolve). Everything is
quick or standard mode with no Joern on the host, so each command takes minutes, not hours, on this 7 GB host.
**Environment (instrument first).**
- Run from a `git archive HEAD` export in the scratchpad (the source is frozen, per the long-runs rule).
- `uv sync --frozen --extra semantic`, which drops the installed `harnessx`. It is installed in `.venv` today.
| `loop-improve` (once) | none | `{0, 0}`, `serialize: engine` | proposals; Workspace = this repository at the commit under test -> `gate.json` (the `RoundOutcome`), `journal.json`, `rule_policy.json` (only when accepted) |
Before the Run starts, `memory-snapshot` is seeded into the run directory like reused facts. It holds the store
rows after the guard, since a sandboxed task cannot read the host store. A `{usd: 0, calls: 0}` budget makes
`budget.py` refuse any call, so a model-free task that tries one fails loudly rather than spending. Attribution
per task is the existing `ousast plane status` table.
**The gate's verdict** lands in `~/ousast-results/plane/<run>/loop-improve/gate.json` and is ingested as
`proposal_outcome` rows. An accepted `rule_policy.json` is adopted into the repository only by a maintainer
commit, as any ledger change is today (README "Rule-level loop").
### 7. Documentation and specs (Requirement 4)
- **README.** Replace "Configuring the HarnessX agentic plane" (README.md:293-329) with "The agentic plane (ax)":
what runs there (repo-facts, verify, agree, memory), `ousast plane run|status|remember`, a pointer to
`ops/ax/README.md`, and one tagged line saying HarnessX was retired. Rename "The HarnessX self-improving cycle"
(:330) to "The self-improving cycle". Its calibration ledger and `ousast improve` content is not HarnessX and
stays. The MetaAgent paragraph (:393-400) is replaced by the memory proposer (`--memory`, provenance, guard).
Drop the `panel_model`/`decider_model` lines at :227-228.
- **`docs/examples.md` section 6** becomes "Running the agentic plane", with the plane doctor, run and status
commands.
- **`docs/threat-model.md:17-20, 32-34`**: egress is the per-task EgressPolicy (receiver, Git hosts, Model
`egress-hosts`, passthrough only). Credentials travel only in the start request. Spend is the per-task
`usd`/`calls` budget in the Run.
- **Retirement notes.** A dated note ("Retired 2026-..: HarnessX was removed by `harnessx-removal`; the agentic
plane is ax (`ai-service-plane`), proposals come from the plane memory. History below is unchanged.") goes at
the top of the requirements (and design, where present) of `harnessx-self-improving-rulesets`,
`learning-harness`, `three-stage-scan` and `openrouter-sast-harness`, and in `.kiro/steering/overview.md` near
:175. The other specs that mention HarnessX in passing keep their history untouched.
- `RELEASE_NOTES.md` gets one entry listing the retired keys. Kiro tooling stays out of user docs.
## Data Models
- `memory.jsonl` (remember output) and `repos/<repo>/<pin>.jsonl`: the rows above, one JSON object per line,
- `RetiredConfigError`: exit 2 with the key and its replacement, before any stage runs.
- A `[harnessx]` section produces exactly one warning, and loading continues.
- `memory.ingest` on a run without `remember` outputs is a no-op that says so. A malformed row fails the ingest
and names the file and line, and nothing partial is written: rows are staged in a temporary file and renamed.
- A low-disk refusal names the store and the free space. The plane run's result is unaffected, and ingest can be
repeated with `ousast plane remember`.
- `seed` never overwrites an existing task directory or a non-pending state. A facts file whose sha256 no longer
matches is dropped from the store with a warning, and that task recomputes.
- `propose_from_memory` with an empty store after the guard returns no proposals, and `run_round` reports
`no_proposals` as today.
- The equality script exits non-zero on the first failed instrument check. It also exits non-zero on any
difference outside the exclusion list and prints the first differing path and key.
## Testing Strategy
- `tests/test_config.py`: the one-warning section and the retired keys (above).
- `tests/test_no_harnessx_references.py`: the tagged-only search of Requirement 2.4.
- `tests/test_plane_memory.py`: ingest is idempotent and content-addresses identical facts. `seed` marks a facts
task done and serves its file, and does not for a changed pin, candidate digest or image. Low-disk refusal
(monkeypatched `disk_usage`).
- `tests/test_plane_remember.py`, `test_plane_alerts.py`, `test_plane_loop.py`: fixture run directories built from
the recorded validation-46 shapes. `remember` rows match `agree`'s per-candidate rows. `alerts` attributes
functions. `measure` reproduces the increment's recorded metrics on the recorded artifacts. `propose` and
`improve` work end to end on a fixture manifest.
- `tests/test_improve_memory.py`:
- M1 and M2 fire at their thresholds and not below.
- A disputed candidate counts neither way.
- The guard removes rows by population, manifest case and holdout pair, so a proposal that exists only because
of a holdout row disappears.
- Provenance ids are stable across runs.
- A reverted memory edit is not re-proposed on the same evidence, and is re-proposed on new evidence.
- `run_round` without proposals writes a journal byte-identical to today's.
- `tests/test_plane_reconciler.py`: `mark_done` under the lock. The existing under-500-lines test stays.
- The existing gate and pair tests, `test_benchmark_output_is_byte_identical_to_golden_baseline`
(`test_gate.py:104`) and the equality script (section 3) are the regression net.
## Requirement Traceability
| Requirement | Design section | Verified by |
| --- | --- | --- |
| 1.1 fate of every capability | 1 (table) | this document; commit 5 |
| 1.2 moved capabilities reachable, Model per task, fixture test | 1 (existing `verify`/`agree`), 4-6 (new tasks are Model-free) | `test_plane_verify.py`, `test_plane_tiebreak.py`, new plane task tests |
`stage_processors.py` (`host_under_harnessx`), `config.py` (`HarnessxConfig`); the `harnessx` extra, its mypy
override and lock entry; the tests that exercise HarnessX; README and docs sections; dated retirement notes in
the specs and steering that name HarnessX.
- **Also in scope**: the plane memory layer as the source of evolutionary proposals and the loop (Requirement 6),
replacing the HarnessX proposer.
- **Out of scope**: changing the deterministic engine, rulesets or `evolve`'s validator and gate (proposals change
source, not how they are validated and gated); new detection logic; qualification on an independent population; the plane's own precision work (judge design) --
where a capability moves to the plane, this spec wires it, it does not improve it.
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Overview
**Purpose.** This feature splits OpenUltraSAST's single overloaded regex layer — which today is detector, severity authority, and volume knob at once — into three governed owners: **rules detect**, a **central CWE policy decides severity and scope**, and a **single project score** becomes the optimization target. On top of that governance plane it adopts the HarnessX runtime exactly where it pays off — the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds.
> **Status 2026-10-02:** retired with HarnessX (see the note below). Its task 6.3 modules `stage_processors.py` and `slot_contract.py` were deleted on 2026-10-02: nothing imported them. Kept as decision history; nothing in src, tests or CI points here.
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Introduction
OpenUltraSAST today fuses three responsibilities into one regex layer: detection patterns also carry their own severity strings and act as the sole volume control, so a noisy rule can dominate scan output, set its own criticality, and fire forever with no central governance. This feature splits those responsibilities into three governed owners — rules detect, a central CWE policy decides severity and scope, and a single project score becomes the optimization target — and adopts the HarnessX runtime exactly where it pays off: the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds.
# Brief: ir-consolidation — stop maintaining an IR we did not need, and adopt one we do
> **Status 2026-10-02:** dormant. Brief only and unapproved. The subtraction it asks for still needs a measured recall comparison before the flat IR goes.
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Problem
The tool is a hand-authored static analyzer with a learning loop bolted on the side, and the loop cannot learn. Three deliberate choices cap it: closed vocabularies (sinks, sources, obligation kinds) that grow only by writing a new spec; a corpus that is the only teacher and a shape that is the only lesson; and a ban on retaining anything the model discovers. Results have ceilings by construction, the loop trains on its own test set, and there is no failure analysis feeding the next round. Detection logic is ~16% of the source and the other 84% cannot say whether the 16% works.
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Overview
**Purpose**: Turn OpenUltraSAST from a hand-authored analyzer with a leaking improvement loop into a learning harness: an auto-classifier routes every pair, finding and code region to one of ten verifier-aligned families; one detector configuration per family is evolved one change per round from structured failure facts; a class-aware pair-wise scorer with unscorable denominators and a measured per-family noise floor decides acceptance; only verifier-confirmed claims are reported above suspicion; the split is enforced by construction.
> **Status 2026-10-02:** retired, superseded by `model-grounded-detection` (spec.json `superseded_by`). 30 of 33 tasks were done; the open three will not be. Kept as decision history; nothing in src, tests or CI points here.
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Introduction
OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The static machinery is treated as the detector while the model with tools already outperforms it (12/14 versus 5/14 pair-correct on the vibe-py holdout once scoring is keyed on class and function), the pair scorer keys detection on text tokens rather than on a vulnerability class, the improve loop learns from its own holdout pairs, and no change is attributed to the class it targets, so an improvement in one class silently degrades another. This feature turns the tool into a learning harness: an auto-classifier assigns every labeled pair, finding and code region to one of ten verifier-aligned families; one detector configuration per family is evolved round by round from structured failure facts; a class-aware, pair-wise scorer with unscorable denominators and a measured per-family noise floor decides acceptance; only verifier-confirmed claims are reported above suspicion; and the train/holdout split is enforced by construction. Context, measurements and sources: `brief.md`, `research.md`, `benchmarks/measurements/2026-09-06-*`.
"summary":"MCP agent + skills so developers and coding agents get the safety net's feedback early (shift-left, in-editor) and so a coding agent can be required to pass a security check before it ships. Delivery surface after G0-G5 qualification; builds on the existing ousast mcp server."
> **Status 2026-10-02:** complete, 16 of 16 tasks (spec.json phase corrected to `implementation-complete`). On 2026-10-02 three of its modules were deleted because nothing imported them: `model/judge.py` (Req 8; `model/pipeline.py` does the judging), `model/calibrate.py` (Req 9; the decision engine's `learn/calibrate.py` is the live calibration) and `model/execution.py`. Deleting `execution.py` closes the Req 10 execution-tier option: a future execution tier is a new design, not this seam.
## Introduction
This feature supersedes `learning-harness` and `constrained-detector`. It replaces the assumption that the LLM
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Architecture Summary
OpenUltraSAST is a harness-oriented SAST system, not a monolithic scanner. The core runtime composes processors, model roles, retrieval, tools, sandbox policy, traces, verification gates, and reporting into a reproducible scan harness.
> **Status 2026-10-02:** retired (see the note below). Its OpenCode skill `openultrasast-kiro-impl`, which still sent agents to this tasks.md, was deleted. Kept as decision history; nothing in src, tests or CI points here.
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Overview
OpenUltraSAST shall be an independent OpenCode-driven security harness for finding, validating, reporting, and optionally fixing source-code vulnerabilities at scale. It shall use OpenRouter-selected chat models for reasoning, OpenRouter embeddings for semantic retrieval, Docker for isolated analysis, and an explicit evidence ladder to prevent AI suspicion from being reported as verified security impact.
> **Status 2026-10-02:** complete, 32 of 32 tasks (spec.json phase corrected to `implementation-complete`).
## Introduction
Engineers evolving OpenUltraSAST's overlay cannot trust the pair scoreboard: coverage detections are dropped, vfc labels mostly say `sink = "unknown"`, a quarter of the vfc corpus fails to parse because excerpts start mid-comment, two sast pairs are unlabelable as vendored, and there is no Node/TypeScript/Python web corpus and no agent-authored corpus. This spec makes the scoreboard honest per slice, per mechanism, and per provenance, adds three web and agent-authored slices, and lets the improve loop accept a change only when no provenance profile regresses. It supersedes `real-world-vfc-slice` for the corpus and scorer plane.
**Why a file plus env, not env only.** Eleven values change together per cluster; env-only has no reviewable
record of what "k3s" is (the `ops/ax/README.md` table at lines 193-204 already struggles to list them), and a
CI job would carry eleven exports. The env override keeps the existing tests (`OUSAST_ROUTER_URL`,
`OUSAST_ARTIFACT_HOST`) and one-off experiments working. **Rejected:** a `[plane]` section in the scan config
TOML (the scan config is a user's tool configuration; the plane is a maintainer surface and must not leak into
`ousast scan`); a Kubernetes ConfigMap (there is no in-cluster reconciler to read it).
`doctor()` takes the profile: it checks `kube_context` reachability, the two namespaces, that the `images` file's
runner reference resolves (`docker manifest inspect` for GHCR, the `/v2/` catalog for a local registry), that
`memory` opens (`open_store` verifies the bucket), and prints each configured address (Req 1.2). The literal scan
(section 9) covers `src/`, `ops/`, `plane/` and `docs/` except the profile files themselves and
`ops/ax/up.sh`, which may name kind because it creates it.
### 2. Execution mode: the CLI with `--exec local|remote` (Req 2)
**Decision.** `ousast plane run RUN.yaml [--exec local|remote]` (default from the profile). Both modes run the
same `reconciler.run`: render, `ax apply`, egress policy, resume, start, await, delete. What differs is
addressing and principal, and none of it is new client code:
| Call | Through | Principal in `remote` |
| --- | --- | --- |
| `ax apply/resume/get/delete` (`reconciler.Ax`) | ax's own tunnel: `kubectl port-forward -n ax-system svc/ax-server` (`tunnel.go:203-238`), context from the profile | kubeconfig for SA `ousast-ci` |
| `kubectl ate create/get/update egress-policy` (`egress.Egress`) | kubectl-ate's own port-forward to `ate-api-server`, bearer = the SA token (`root.go:59-60`) | same SA; Substrate's authorization of that token for `egress-policy` is an open question (below) |
| `POST /ousast/v1/start` (`router.open_router`) | `kubectl port-forward svc/atenet-router` in `ate-system`, as today | same SA |
| artifacts, inputs, state | the S3 store (section 3) | the store's agent key from the CI secrets |
**Auth: kubeconfig with a ServiceAccount token, not an ingress.** ax-server has no authentication
(`deploy/ax-server.yaml:71-79`: `--addr=:8080`, nothing else) and Substrate's router routes on a header. Exposing
either through Traefik would publish an unauthenticated control plane behind whatever middleware we bolt on;
tunnelling through the Kubernetes API keeps one authenticated, audited door and uses the clients as they ship.
**Rejected:** Traefik IngressRoute for ax-server and the router (unauthenticated upstreams, a second auth layer to
design, and ax's CLI would still tunnel); a bespoke ax HTTP client (Req boundary: no replacement of the `ax` CLI
beyond what addressing needs, and the tunnel already handles addressing).
**RBAC (`ops/k8s/base/rbac-ci.yaml`).** ServiceAccount `ousast-ci` in `ax-system`; Role in `ax-system` and in
`ate-system`: `pods` get/list, `services` get, `pods/portforward` create; nothing cluster-wide, no verbs on
Substrate's or ax's CRDs (both are reached through their servers, not through the Kubernetes API). The token is a
bound TokenRequest (`kubectl create token ousast-ci --duration=...`) stored as the CI secret `KUBECONFIG`. Secrets
for the provider and the store stay where the reconciler runs: `.env` on a laptop, repository secrets in CI,
Kubernetes Secrets only on the day an in-cluster reconciler exists (2.4). The start-request path is unchanged
(Req 2.3).
**Run state of record moves to the store.** `state.json` is written locally under `$OUSAST_RESULTS/plane/<run>/`
as today and mirrored to `runs/<run>/state.json` on the store after each change; a rerun from another machine
seeds its local copy from the store. This is what lets CI job N+1 resume CI job N's Run (Runs are long,
resumable and artifact-driven: `reconciler.py:272-312`).
**Kind keeps `local` (Req 2.4).** Same command, same manifests, local registry, laptop as reconciler; the only
differences are the profile's values. The kind proof of each increment is run in `local`, and increment 3 runs
`remote` against kind from a container that holds only the kubeconfig and the secrets (section 7).
#### 2.4 Deferred: an in-cluster reconciler
Not needed for the move. It would add: survival of a laptop sleeping or a CI job timing out mid-Run, scheduled
Runs, and Secrets injected by Kubernetes. It would cost: a workload (Job per Run is the natural shape, keeping
the CLI path identical), a PVC or store-only state, a submit path, and RBAC for ax's and Substrate's Services from
inside the cluster. Revisit when unattended Runs are wanted; the store-backed state above is its prerequisite.
### 3. Delivery: presigned URLs in the start request, both profiles (Req 3)
**Decision.** The runner PUTs `output.tar` to a presigned S3 URL and GETs each declared input from a presigned
URL; the CLI mints them and polls the store. The URLs travel in the start body, never in env:
prints the Run name in the hook's report; `--wait SECONDS` blocks for CI. `ousast plane status <run>` reports
later. The wrapper `ops/pre-push` gains an opt-in `OUSAST_PLANE_SCAN=1`; it never blocks on the plane.
### 7. Increment order, kind proof and measurement
| # | Increment | Kind proof (`local` unless stated) | Recorded under `benchmarks/measurements/` |
| --- | --- | --- | --- |
| 1 | `PlaneProfile`, literals replaced, `doctor` by profile | `ousast plane doctor` passes with `kind.toml`; literal scan 0 | `<date>-k8s-01-profile/record.json`: doctor lines per profile, scan count |
| 2 | Store delivery + receiver-less egress; receiver removed after | `ops/ax/smoke-run.sh` delivers `facts.json` via the store; policy has one TLS rule | `-02-store-delivery/`: bytes, PUT-to-poll seconds, rendered policies |
| 3 | `--exec remote` from a CI-shaped container (kubeconfig + secrets only) against kind | smoke Run completes; audit log lists only the Role's verbs | `-03-remote-exec/`: kube audit verbs vs Role, run status |
| 4 | Images on GHCR, digests, repin (section 8) | kind pulls `ghcr.io/norandom/ousast-runner@sha256` (pull secret if private); smoke Run | `-04-ghcr-images/`: sizes, pull seconds per image |
| 5 | `engine` task + `engine-pool` | froxlor case as `engine`: rows equal the 2026-09-30 host record (6/6), placement on engine-pool shown | `-05-engine-task/`: rows diff, seconds, peak memory, placement |
| 7 | `plane scan` on a real pair (a PHP CVE pair from the dev corpus) | Run completes; rows under `repos/<repo>/<head>` | `-07-plane-scan/`: candidates, cost, agreement, rows |
| 8 | Runbook (`docs/deployment.md`), checklist mapped to R1-R7 | n/a (documentation; every "planned" label removed only with a recorded run) | the checklist links the records above |
### 8. GitHub Actions and re-pinning
`.github/workflows/images.yml` on tags `v*` (separate from `release.yml`, which builds the wheel): build
`plane/Dockerfile.runner` as `ghcr.io/norandom/ousast-runner` and the root `Dockerfile` as
`ghcr.io/norandom/ousast-engine` with `docker/build-push-action`, tag by version, and write the two digests to a
release asset `images.json` (`{"runner": "ghcr.io/...@sha256:...", "engine": "..."}`) and to job outputs.
`generate.repin_templates` (`generate.py:484-497`, one image for all templates) becomes `repin(plane, images)`:
`engine.yaml` gets `images["engine"]`, every other template `images["runner"]`; `ousast plane repin images.json`
exposes it and `--runner-image FILE` keeps working for a single-image file. Private packages: the pools' pod
template names `imagePullSecret` from the profile.
### 9. Tests
- `tests/test_plane_literals.py`: scans `src/`, `plane/`, `ops/` (minus `ops/ax/up.sh`, `ops/k8s/profiles/`) and
`docs/` for `kind-ousast`, `localhost:5001`, `172.19.`, `127.0.0.1:` and `port-forward` outside `router.py`
and the ax tunnel note; built at runtime like `test_removed_plane_references.py`.
- `tests/test_plane_profile.py`: defaults, file + env precedence, unknown key rejected, no secret field, both
committed profiles load.
- `tests/test_plane_k8s.py`: render both profiles; kind has no HPA and no pull secret; k3s has both; engine pool
limits >= the engine task's limits; every image reference is digest-pinned.
- `tests/test_plane_rbac.py`: the rendered Role's verbs equal the set of Kubernetes API calls the CLI makes
(fake `kubectl`/`ax`/`kubectl-ate` record their argv; the test maps them to `pods/portforward`, `pods`,
`services`) and nothing more.
- `tests/test_plane_egress.py`: for every committed Task template, `policy_for` yields one TLS rule whose hosts are
exactly the store host, the bound Git hosts and the Model hosts; no `http` rule; an IP endpoint is refused.
- `tests/test_plane_delivery.py`: start body with `delivery` accepted; URLs never logged; expiry; PUT and GET
against a local server; `delivered()` polling; declared-only inputs.
- `tests/test_plane_engine_task.py`: fixture record -> rows and summary equal `engine_alerts.produce`'s shape;
the instrument failures.
- `tests/test_plane_scan.py`: candidates from a two-commit fixture repository; Run composition; budgets.
- **Only live on kind:** placement on `engine-pool`; Substrate accepting the SA token for `egress-policy`;
the gateway passing TLS to the store host; GHCR pulls; the HPA (k3s only). These are the kind proofs above.
## Data Models
Store layout additions: `runs/<run>/state.json`, `runs/<run>/<task>/output.tar`, `runs/<run>/<task>/<artifact>`,
`runs/<run>/<task>/done.json`, `runs/<run>/inputs/<name>`; all under the existing 30-day `runs/` rule. Start body:
`credentials` as today plus optional `delivery {put, inputs{name: url}}`. `PlaneProfile` as in section 1.
`images.json` as in section 8.
## Error Handling
A profile naming an unreadable context, a bucket that fails `verify_bucket`, or an images file without a digest
fails before `ax apply` with the field named. A presigned PUT that fails three times ends the task `failed` with
the HTTP status (`runner.deliver` semantics kept); an expired URL after a re-resume is re-minted by the next
start. A store HEAD error during polling is retried with the resume backoff and reported after ten consecutive
failures, like `ax get` (`reconciler.py`, `_await`). A `delivery` URL that is not `https` to a hostname is a 400
from `StartGate.offer`. The engine task fails loudly on an unread input, as `read_record` does.
The plane moves from this laptop's kind cluster to a Kubernetes cluster of the maintainer's choosing without
changing what runs: the same ax manifests, runner image, memory store and credential path. Every host-bound
piece becomes a cluster resource or a configuration value, each change is proven on kind first, and the
production entry point (a delta Run for a push) is added. Context: `brief.md`, `docs/deployment.md`.
## Boundary Context
- **In scope**: two profiles of one code base, `kind` (local, kept) and `k3s` (production); a remote execution mode of the CLI; artifacts delivered to the store; configurable
cluster context, registry and image pins; our own snapshot and memory buckets; the Joern engine as a plane task
on a larger worker pool; worker pool sizing and autoscaling; a delta Run command and Workspaces from URL +
commit; a deployment runbook with a proof on kind and a checklist for the target cluster.
- **Out of scope**: provisioning or operating the target cluster; modifying Agent Substrate or ax; any change to
detection, scoring, populations or the decision engine; replacing the `ax` CLI with a bespoke client beyond what
remote submission needs.
- **Adjacent expectations**: `ai-service-plane` requirements (one Model + budget per task, attribution,
credentials only in the start request, egress policies, verify-at-startup memory store) keep holding;
`harnessx-removal` Req 6 (memory on the S3 store) keeps holding; nothing is deleted before its replacement
passes its test (the `harnessx-removal` sequencing rule).
## Target Environment (maintainer, 2026-10-02)
- **Cluster: k3s.** Consequences: containerd (no Docker socket; the engine's host-Docker path does not exist there),
gVisor needs `runsc` and a `RuntimeClass` installed on the nodes Substrate's workers run on, k3s ships Traefik
and local-path storage, and single-node k3s is close enough to kind that the kind proof transfers.
- **Images and source: GitHub.** Images are published to GitHub Container Registry (`ghcr.io/norandom/...`),
built by GitHub Actions from tagged commits, digest-pinned in the manifests; private packages need an
`imagePullSecret` on the worker ServiceAccounts and the CI principal. Substrate's and ax's own images are built with
`ko` from their checkouts and pushed to the same registry.
## Requirements
### Requirement 1: Nothing assumes this host
**User Story:** As the maintainer, I want every host-bound assumption replaced by a cluster resource or a
configuration value, so the plane runs the same on kind and on the target cluster.
#### Acceptance Criteria
1. The kube context, the registry (GHCR in production), the ax-server address, the router address, the receiver address and the
snapshot and memory buckets are configuration (environment or one config file), with the kind values as the
documented defaults for development; no module names `kind-ousast`, `localhost:5001` or `172.19.0.1`.
2. `ousast plane doctor` checks the configured cluster, not kind by name, and reports each configured address.
3. A test enumerates the source tree for host-only literals and fails on any.
### Requirement 2: The reconciler stays the CLI, with a remote execution mode
**User Story:** As the maintainer, I want to trigger the checks in the cluster from the CLI and get the results
fed back, so that remote execution is a mode of the same tool, not a separate service.
#### Acceptance Criteria
1. `ousast plane run` (and the later `ousast plane scan`) takes an execution mode, `local` or `remote`, from the
profile or a flag; `remote` submits the Run to ax-server in the target cluster (over the cluster's ingress or a
kubeconfig), `local` is today's kind path. The manifests, the memory-key seeding, the budgets and the
attribution table are identical in both modes.
2. In remote mode the CLI needs no inbound path: it polls the memory store for task completion and artifacts and
prints the same status, attribution table and verdicts; `ousast plane status <run>` reports a Run started
earlier from any machine with the store credentials.
3. Provider and store credentials come from the caller's environment (a laptop `.env`, a CI secret), travel only
in the start request, and never appear in a manifest, an ActorTemplate or a log.
4. The principal the CLI uses against the cluster has the least privilege the ax API needs and nothing against
Agent Substrate directly; a test compares the rendered role with the calls the CLI makes.
5. kind stays a first-class local execution mode, not a stepping stone: `ousast plane run` from a laptop
against the kind cluster keeps working with the development profile (same code, same manifests, local
registry), and every release is proven on it before anything is applied to k3s.
6. An unattended in-cluster reconciler (a Deployment or Job that runs without a caller) is a later option, not
part of this spec.
### Requirement 3: Artifacts land in the store, not on a host
**User Story:** As the maintainer, I want task artifacts delivered to the memory store so that no host has to
be reachable from the cluster.
#### Acceptance Criteria
1. Tasks deliver their artifacts to the S3 memory store (presigned URLs minted by the CLI and carried in the
start request, or a scoped delivery credential; the design chooses one and states why) and the CLI reads
completion from the store. The receiver thread and the kind gateway EndpointSlice remain only if the design
shows the local profile needs them; one delivery path for both profiles is preferred.
2. Egress policies grant each task exactly the hosts it needs (the store's endpoint, git hosts, the bound
Model's host) and nothing else; a test renders the policy for every committed Task.
### Requirement 4: The engine is a plane task
**User Story:** As the maintainer, I want Joern to run where the other tasks run, on a worker sized for it.
#### Acceptance Criteria
1. A second WorkerPool sized for the engine (memory and CPU from a measured run) exists alongside the default
pool; an `engine` task module wraps today's host engine run and emits the same `alerts.jsonl` rows and
coverage as `alerts-engine`.
2. The engine image is digest-pinned in the registry like the runner image; its size and pull time are recorded.
3. `alerts-engine` on the host remains available for development and is marked as such.
### Requirement 5: Sizing and autoscaling
**User Story:** As the maintainer, I want the pool to follow the load of pushes without manual resizing.
#### Acceptance Criteria
1. Worker pool replicas, limits and the autoscaling rule (Substrate's assigned-worker signal) are manifests under
`ops/k8s/` with the kind values as a small profile and a documented production profile.
2. The runbook records the measured footprint per task type and the queueing consequence of one actor per
worker.
### Requirement 6: A delta Run for a push
**User Story:** As a user with a repository the project has never seen, I want a push checked by the plane.
#### Acceptance Criteria
1. `ousast plane scan <repo-url> --base <commit> --head <commit>` generates Workspaces from the URL and commits,
derives candidates from the changed functions (quick rules, inferred roles, repository facts), and submits a
Run of facts, verify a/b, tie-break and agree with per-task budgets; a whole-repository mode is secondary.
2. Results are written to the memory store under the repository and head commit and reported with the
attribution table; the decision engine's verdict is reported only when a model is adopted, else the plane's
agreement (ai-service-plane Req 6 and learned-decision-engine Req 6 unchanged).
3. The latency split is explicit: a blocking pre-push carries quick rules and the delta engine within its
deadline; plane verdicts arrive asynchronously.
### Requirement 7: Proven on kind, then documented for the target
**User Story:** As the maintainer, I want each step shown working on kind before I touch the real cluster.
#### Acceptance Criteria
1. Every requirement above is exercised end to end on the kind cluster (remote mode included, with the kind cluster as the remote target), with
a recorded run under `benchmarks/measurements/`.
2. `docs/deployment.md` becomes the runbook: prerequisites for the target cluster (Substrate, ax, registry,
buckets, Secrets, node requirements for gVisor workers), the profile switch, and a checklist whose every item
maps to a requirement here; planned items are labelled planned until their run is recorded.
Status: approved for task generation on 2026-09-12 following the maintainer's "ok, next kiro phase". The acceptance profile defines evaluation targets, not measured achievements.
"action":"Accepted the generated task plan and authorized starting group 1 (1.1\u20131.4). Implementation started; tasks require independent review and fresh verification before completion."
"action":"Accepted the generated task plan and authorized starting group 1 (1.1–1.4). Implementation started; tasks require independent review and fresh verification before completion."
},
{
"at":"2026-09-12T18:32:19.033933Z",
"source":"ok, next kiro phase, approved; correct. we can continue with the spec",
"action":"Authorized group 2 (2.1\u20132.3), resumed after local Claude Mem repair. Independent review and fresh verification remain required."
"action":"Authorized group 2 (2.1–2.3), resumed after local Claude Mem repair. Independent review and fresh verification remain required."
},
{
"at":"2026-09-13T07:48:02.155687Z",
"source":"continue with the task",
"action":"Authorized continuing group 3 (3.1\u20133.4) after the group 3 next-step summary. Independent task review and fresh verification remain required."
"action":"Authorized continuing group 3 (3.1–3.4) after the group 3 next-step summary. Independent task review and fresh verification remain required."
},
{
"at":"2026-09-13T09:17:01.788332+00:00",
@@ -59,7 +59,7 @@
{
"at":"2026-09-13T10:26:47.307154+00:00",
"source":"move on to group 4 as a whole",
"action":"Authorized all of group 4 (4.1\u20134.3), sequential implementation, independent task review and fresh verification. Groups 5\u20138 remain pending."
"action":"Authorized all of group 4 (4.1–4.3), sequential implementation, independent task review and fresh verification. Groups 5–8 remain pending."
},
{
"at":"2026-09-13T12:40:57.961577+00:00",
@@ -69,17 +69,22 @@
{
"at":"2026-09-13T13:58:52.336506+00:00",
"source":"continue with group 6 as a whole. do we need to amend the spec to get a go for the rollout",
"action":"Authorized group 6 (6.1\u20136.4). Clarified complete preparation/discovery reuse after measured group 5 bottleneck; rollout gates remain unchanged. Manual Kiro review fallback remains necessary after a fresh agent dispatch hit the thread limit."
"action":"Authorized group 6 (6.1–6.4). Clarified complete preparation/discovery reuse after measured group 5 bottleneck; rollout gates remain unchanged. Manual Kiro review fallback remains necessary after a fresh agent dispatch hit the thread limit."
},
{
"at":"2026-09-13T15:45:15.902603+00:00",
"source":"ok, continue with group 7 as a whole",
"action":"Authorized tasks 7.1\u20137.3; manual Kiro fallback after fresh dispatch hit thread limit. Group 8 and rollout remain pending."
"action":"Authorized tasks 7.1–7.3; manual Kiro fallback after fresh dispatch hit thread limit. Group 8 and rollout remain pending."
},
{
"at":"2026-09-13T16:41:21.889949+00:00",
"source":"move on to group 8",
"action":"Authorized tasks 8.1\u20138.4 and their qualification work; preserve joint gates and record NO-GO wherever prerequisites or measurements do not pass. Fresh dispatch hit thread limit; manual Kiro fallback."
"action":"Authorized tasks 8.1–8.4 and their qualification work; preserve joint gates and record NO-GO wherever prerequisites or measurements do not pass. Fresh dispatch hit thread limit; manual Kiro fallback."
},
{
"at":"2026-09-16T13:30:00+00:00",
"source":"add the M1a task to tasks.md and start it; continue with the plan",
"action":"Added group 9 (M1a) to the approved task plan as the bounded owner amendment the milestone requires, implemented the task-local recorded-veto flag (9.1) and ran the three NodeGoat revisions (9.2). Diagnosis only: no capability enabled, no gate changed, rollout remains NO-GO."
"feasibility":"NO-GO: all 21 frozen NodeGoat transactions timed out; 0/24 declared target checks completed. Cancellation/reporting remained below 2 seconds. Priming did not qualify graph reuse; named census and engine lifecycle work routed to contributor-scan.",
"feasibility":"NO-GO: post-census profile measured all 21 cases; 18 timed out, three identical-tip replays completed in 18.521–18.874s with valid reuse; 0/24 target checks completed. Maximum cancellation/report overrun 0.874s. See 2026-09-14-nodegoat-post-census-feasibility.json.",
"qualification":"NO-GO: eligibility implementation and evidence delivered; required independent populations and verified VAmPI transitive witness remain missing. Zero capabilities enabled.",
"result":"Raw <lambda>2 witnesses present at contributions.js 32-34; fixed twin analyzed and quiet (target question answered with zero rows, no deadline reason); unchanged revision recorded 'unchanged'. Five vetoes fired and were recorded. With all five lifted the detection stops at operation_correspondence_unresolved, a fifth veto not previously named. No defect admitted, no capability enabled."
"result":"Exit run recorded. All three pinned NodeGoat revisions classify through the production rules: vulnerable three sites new with operation_absent_from_comparable_base and admission rejecting on capability_unavailable alone; fixed twin analyzed and quiet with three base-only findings; unchanged recorded unchanged with base locations resolved. Every veto M1a listed is repaired, plus two the repairs exposed. Zero admitted defects.",
"remaining_gap":"The supported security consequence and concrete repair direction are not demonstrated. Both are owned by an evaluated capability declaration and the registry is deliberately empty. Task 10.10 renders them through an explicitly experimental declaration with eligibility still disabled; inventing the text is what admission exists to prevent."
}
},
"release":{
@@ -157,5 +202,26 @@
"channel":"alpha",
"rollout_decision":"NO-GO",
"status_record":"status.md"
},
"release_milestones":{
"record":"release-milestones.md",
"direction_approved":true,
"approval_source":"I agree. please note this into the raodmap as a spec",
"current":"M1b",
"verified":[
"M1a"
],
"sequence":[
"M1a: NodeGoat detection with admission vetoes recorded, not applied (MET 2026-09-16)",
"M1b: repair only the vetoes M1a showed to matter; rerun through production rules",
"M2: equivalent bounded runtime",
"M3: shared-language transfer",
"M4: independent qualification",
"M5: narrow advisory rollout"
],
"implementation_design_approval":"Existing approvals preserved; this records milestone order and exit criteria, not blanket approval of future design changes.",
"m1_split_accepted":"2026-09-14: maintainer asked to split M1 into M1a/M1b after review",
"m1a_outcome":"Met inside the one-week box. Corrected the owner diagnosis: all four named vetoes fire, and operation correspondence for an operation on a changed line is the blocker that survives every other repair.",
"m1b_position":"Comparison half verified and recorded on NodeGoat; the consequence and repair rendering (10.10) remains before M1b is verified."
- [x] 9. Diagnose the first useful detection with vetoes recorded (release milestone M1a)
- [x] 9.1 Add the task-local recorded-veto evaluation flag
- Add an explicit experimental evaluation path that **records** each comparison and admission veto per raw finding instead of applying it. Keep it task-local, off by default, and available only on explicit `--base`/`--head` replay; refuse it on the Git stdin path so it can never become a hook capability.
- Lift only recognized completeness vetoes over a copy of the evidence: answered-then-demoted question outcomes, the vendor exclusion completion veto, and the diagnosed context/correspondence boundary kinds. Never lift failures: deadline exhaustion, unanswered questions, unreadable input, empty graphs and unparsed files stay applied, because silence from failure is not a quiet fixed twin.
- Retain the production novelty, reason and admission reasons beside the recorded ones for every finding, plus the query-to-witness identity difference, base-only findings and readable census receipts. Every payload carries an explicit experimental label; the recorded path admits no defect and enables no capability.
- Done when the flag is off by default, produces a labeled artifact section and terminal line under explicit replay, is refused in Git stdin mode, and unit controls prove recognized vetoes are recorded while failures and unrecognized boundaries remain applied.
- _Boundary: Delta and Actionability Policy, Push runner, Result and Hook Adapter, CLI integration_
- _Requirements: 3.1, 3.4, 5.5, 6.3, 7.3_
- _Verified 2026-09-16:_ 1,330 tests passed, nine skipped; global Ruff/format and touched mypy passed. Flag off by default, refused in Git stdin mode, labeled artifact section and terminal line under explicit replay. Controls prove recognized vetoes are recorded while deadline exhaustion, unanswered questions, empty graphs and unrecognized boundaries stay applied, and that the recorded path emits no defect even when a lifted candidate would otherwise admit. Regenerated the committed module audit for the added module.
- [x] 9.2 Run the three NodeGoat revisions and produce the per-finding veto list
- Record exact base/head revisions, expected operation and expected outcome before running. Exercise the shipped replay path, existing ranker and physically vendor-free graph; do not substitute a hand-selected slice or hard-coded target function.
- Replay the pinned vulnerable, fixed and unchanged revisions under the flag with an explicitly named development budget. That budget cannot satisfy or measure the hook latency gate.
- Deliverable: the raw finding on the vulnerable revision with location and witness, the fixed twin genuinely analyzed and quiet, the unchanged revision not reported as a new regression, and for every finding the exact list of vetoes that fired and why, with source and census receipts.
- Done when one replayable three-revision record states each expectation against its observed outcome and names the vetoes that actually fired. A fixed twin that timed out or was not analyzed fails M1a and blocks M1b until diagnosed.
- _Verified 2026-09-16:_ all three pinned revisions replayed through the production CLI under a named 900-second development budget; expectations frozen on disk first. Raw `<lambda>2` witnesses present at lines 32–34; fixed twin analyzed and quiet (target question answered with zero rows, no deadline reason, three base-only findings); unchanged revision analyzed and recorded `unchanged`. Five vetoes fired and were recorded; the remaining blocker is `operation_correspondence_unresolved` for an operation on a changed line. Zero admitted defects. Evidence: [M1a record](m1a-recorded-vetoes.md) and `benchmarks/measurements/2026-09-16-nodegoat-m1a-recorded-vetoes.json`. Diagnosis only; no capability enabled and no latency claim.
- [x] 10. Repair only the vetoes M1a recorded (release milestone M1b)
- [x] 10.1 Establish operation correspondence for an operation on a changed line
- M1a measured this as the blocker that survives every other repair: a newly introduced operation sits on a changed line, has no base line anchor, and therefore cannot be classified even when every completeness veto is lifted.
- Bind the required base/head correspondence to the specific operation identity rather than to a line anchor. A complete base answer for the counterpart question that contains no operation with the same operation and source establishes absence; a matching operation elsewhere in that answer is movement, not novelty.
- Keep the existing conservative rules: an incomplete or failed base stays unknown, benign movement and renames stay unchanged, and lexical alignment is never semantic proof.
- Add absent, renamed, removed-guard, moved-operation and repeated-identical-operation controls. Done when a newly introduced operation on a changed line is classified new, while movement, renames and an incomplete base remain unchanged or unknown.
- _Boundary: Delta and Actionability Policy_
- _Requirements: 2.4, 3.1, 3.4_
- _Verified 2026-09-16:_ 1,337 tests passed, nine skipped; Ruff/format/mypy passed. Correspondence now reports `anchored`, `ambiguous`, `changed_line`, `unmapped` or `unchanged_path`, and only `changed_line` supports an introduced operation. Movement keys on the operation spelling, which the existing benign-rename control proved necessary. One superseded control was updated with its reason; movement, ambiguity, unmapped operations, discharged or unentailed claims and an incomplete base all remain unknown.
- [x] 10.2 Establish the required comparison scope explicitly
- Comparison currently requires every selected base question to complete with no base degradation and no non-tier-zero deferred question, so one unsupported family anywhere prevents an otherwise complete target comparison. The owner research names this and requires the amendment to state the required scope rather than relax the bar.
- Define the comparison's required scope as the counterpart question plus the questions its evidence actually depends on, and require completeness over exactly that scope. Retain every other gap in aggregate coverage; a question outside the required scope stays reported and unresolved, never silently discharged.
- A missing raw finding in an incomplete base still cannot establish novelty. Unknown dependency remains inside the required scope.
- Done when an unrelated unsupported family no longer prevents a complete target comparison, while an incomplete counterpart, an incomplete dependency of it, or unknown dependency ownership still yields unknown.
- _Boundary: Delta and Actionability Policy_
- _Depends: 10.1_
- _Requirements: 2.2, 2.4, 3.4, 6.4_
- _Verified 2026-09-16:_ 1,342 tests passed, nine skipped; Ruff/format/mypy passed. Completeness is now demanded over the counterpart plus its recorded dependencies. Controls prove an unrelated unsupported family no longer blocks a complete target comparison, while an incomplete counterpart, an incomplete recorded dependency, an unknown-ownership boundary and a truncated base scan all still yield unknown. Every other gap stays in aggregate coverage.
- [x] 10.3 Scope correspondence ambiguity to the operations it actually affects
- `ambiguous_line_correspondence` is recorded file-wide and inherited by every question, so repeated unchanged lines invalidate distinct, uniquely mapped operations elsewhere.
- Retain ambiguous mappings as unavailable and record which regions are ambiguous, so only an operation whose required correspondence needs an ambiguous line is vetoed. Do not delete the ambiguity check.
- Done when a uniquely mapped operation is comparable in a file that also contains repeated unchanged lines, while an operation needing an ambiguous line stays unresolved.
- _Boundary: Snapshot Adapter, Delta and Actionability Policy_
- _Depends: 10.2_
- _Requirements: 2.4, 3.4_
- _Verified 2026-09-16:_ 1,347 tests passed, nine skipped; Ruff/format/mypy passed. Ambiguity is no longer inherited as a context gap by every question, and no longer blocks the transaction comparison; it is decided per operation by the correspondence classifier. Controls prove a uniquely mapped or newly introduced operation stays comparable in a file with repeated unchanged lines, an operation needing an ambiguous anchor stays unresolved, and any other transaction gap still blocks. The mapping is still withheld and still reported in aggregate coverage.
- [x] 10.4 Export the enumerated operation inventory so absence is provable
- **Prerequisite discovered 2026-09-16, not in the original plan.** The 10.1-10.3 rerun showed the remaining two vetoes cannot be scoped or relaxed soundly as written. A taint answer reports flows, so an absent row means either that the operation does not exist in the base or that no traced flow reached it. `dynamic_external_or_depth_context_unresolved` exists precisely to stop the second being read as the first, and on a real Express application it is always true, because ordinary framework and dependency calls have unresolved destinations. It therefore carries no discrimination and can never be discharged, while removing it would make an unreached operation look absent.
- Export the operations the query actually enumerated in the question's scope, separately from the flows it traced, so absence of an operation is established by the inventory rather than by the absence of a flow. Sanitizer and source facts are unchanged; this adds no detector, alias resolver or second IR.
- Done when a base answer distinguishes "this operation does not exist here" from "no traced flow reached it", with an unenumerated scope remaining explicitly unresolved.
- _Boundary: contributor-scan query/evidence contract, Delta and Actionability Policy_
- _Depends: 10.1_
- _Requirements: 2.1, 2.2, 3.4, 8.1, 8.2_
- _Verified 2026-09-16:_ 1,354 tests passed, nine skipped; Ruff/format/mypy passed. The taint query exports one row per enumerated operation plus a completion marker, which is what makes an empty inventory meaningful. Comparison now separates three outcomes: operation present in the base without a traced flow, operation absent from an enumerated base, and a base scope never enumerated, which stays explicitly unresolved along with any malformed inventory. The arbiter skips every row carrying a kind, so an inventory row cannot become a flow with an empty source. Real-engine smoke on an authored two-revision JavaScript control confirmed emission, the zero-operation marker and unchanged answer integrity: `benchmarks/measurements/2026-09-16-operation-inventory-smoke.json`. Pre-inventory fixtures were given an enumerated base, since an unenumerated one can no longer establish absence.
- [x] 10.5 Make the external and dynamic context boundary question-owned
- M1a recorded `dynamic_external_or_depth_context_unresolved` on the target's own question. The flag is computed over every in-scope method, so one unresolved call anywhere vetoes every question.
- With 10.4 supplying inventory evidence, bind the boundary to what it actually limits: reachability, not enumeration. A question establishing absence from the inventory no longer needs complete reachability; a question asserting a traced flow still does. Unknown ownership remains a blocker; absent evidence of independence is not proof of independence.
- Done when an absence established from the inventory survives an unrelated unresolved destination, while a reachability claim under a bounded scope stays unresolved.
- _Verified 2026-09-16:_ 1,362 tests passed, nine skipped; Ruff/format/mypy passed. The boundary is now bound to what it limits. At scan level it is recorded against its owning question without demoting that question's answer, while an unavailable projection, an empty scope and an unreadable extent still demote. In comparison it no longer blocks the head's own traced flow, and no longer blocks an absence read off a structural enumeration, but still blocks every claim resting on a flow not having been traced. The query declares its enumeration basis, because a scope followed through the call graph can be shrunk by the same unresolved destination; such an inventory cannot establish absence while reachability is bounded. Controls cover both directions and both bases; every other gap kind still blocks.
- [x] 10.6 Stop vendor exclusion acting as a global completion veto
- `vendor_semantics_unresolved` degrades every otherwise complete outcome to `graph_incomplete`, including on a complete 44-file first-party census with successful query receipts.
- Keep the exclusion physical and keep the coverage note. Let the inventory and reachability evidence from 10.4 and 10.5 carry whether a specific answer needed excluded semantics; a recorded dependency on an excluded path still blocks that question.
- Done when an unrelated vendor exclusion no longer invalidates a first-party answer, while a question with evidence of dependency on excluded code remains unresolved and the exclusion stays out of targets and graphs.
- _Boundary: contributor-scan graph/scope evidence, Cache and Engine Boundary_
- _Depends: 10.5_
- _Requirements: 2.2, 2.3, 8.2, 8.4_
- _Verified 2026-09-16:_ 1,365 tests passed, nine skipped; Ruff/format/mypy passed. Graph boundaries are split into integrity failures, which still demote a completed answer, and declared exclusions, which do not. Whether a particular answer needed the excluded semantics is carried by that question's own unresolved call destinations from 10.5, not assumed for every answer. Controls exercise the real partition logic rather than an injected reason: a vendored directory leaves the answer completed with the exclusion still reported, an unresolved symlink still demotes, and the two together still demote. The exclusion stays physical and out of targets and graphs. The NodeGoat rerun showed the same distinction was still missing inside comparison, where any degradation and any boundary blocked: comparison now filters declared exclusions and correspondence ambiguity on both sides, keeping integrity degradations and every other boundary blocking.
- [x] 10.7 Bind candidate dependency gaps to the candidate
- **Discovered 2026-09-16 in the M1b production rerun.** The runner passes every coverage reason as every candidate's `dependency_gaps`, so admission always answers `dependency_unresolved`. That is not "an unresolved dependency that undermines the claim"; it is every gap anywhere in the transaction.
- Supply the gaps that pertain to this candidate's required comparison scope, the same scope 10.2 established. Unknown ownership still counts against the candidate.
- Not an M1b exit blocker, since eligibility stays disabled and no alert is emitted either way. It would prevent admission from ever passing after qualification, so it belongs before M4.
- Done when an unrelated transaction gap no longer appears as this candidate's unresolved dependency, while a gap in its required scope still does.
- _Boundary: Push runner, Delta and Actionability Policy_
- _Depends: 10.2_
- _Requirements: 3.1, 3.3, 3.4_
- _Verified 2026-09-16:_ 1,371 tests passed, nine skipped; Ruff/format/mypy passed. Gaps are now attributed: a boundary carrying a question id counts only when that question is in this candidate's required scope, an unattributed gap always counts because unknown ownership is not evidence of independence, and integrity degradations count. A declared exclusion, correspondence ambiguity and bounded reachability do not, because each was already weighed against this specific claim. The runner no longer hands every candidate the whole transaction's reason list, which had included other candidates' comparison outcomes.
- [x] 10.8 Carry query-to-operation provenance into the reported evidence
- The target is selected through a file-fallback question with `function=None` while rows report method `<lambda>2`. M1a showed this did not block the comparison once the question completed, so it is provenance quality rather than the blocker.
- Record verified query-to-operation provenance on the evidence; do not inject benchmark function names into regions or count coincident sites.
- Done when a reported operation names the question that produced it and the method the engine reported, with unresolved provenance explicit.
- _Boundary: Delta and Actionability Policy, contributor-scan normalization_
- _Requirements: 3.1, 3.4, 6.3_
- _Verified 2026-09-16:_ 1,371 tests passed, nine skipped; Ruff/format/mypy passed. Evidence now carries the method the engine reported beside the question that produced it, so the link is inspectable rather than inferred or dug out of a detail blob. Provenance is recorded as `exact`, `file_scope` when the question named no function and the engine supplied one, `unresolved` when the two disagree, and `method_unreported` when the engine named none. Controls cover all four. No region is renamed, no benchmark function name is injected and no coincident site is counted; the state is recorded, not applied.
- [x] 10.9 Rerun the three revisions through the production rules
- Replay the same pinned vulnerable, fixed and unchanged revisions **without** the recorded-veto flag under the named development budget. Demonstrate a developer-readable evaluation finding with a supported consequence, exact location, witness, change attribution and repair direction.
- Verify the fixed twin is analyzed and produces no actionable defect and the unchanged revision is not reported as a new regression. Keep qualification eligibility disabled; experimental evaluation output cannot advertise an unqualified hook alert.
- Done when one replayable record states which vetoes were repaired and which remain, with completed required checks and readable source and census receipts.
- _Verified 2026-09-16 with one named exception:_ all three revisions replayed through the production rules with exact pins and reopened source receipts. Vulnerable: three sites new with `operation_absent_from_comparable_base`, admission rejecting on `capability_unavailable` alone. Fixed twin: analyzed and quiet with three base-only findings. Unchanged: three sites `unchanged` with base locations resolved to the same lines. Every finding carries its question, the engine-reported method and its witness; the head census is the complete 44 files; all 436 base questions complete. Zero admitted defects. The repaired and remaining veto lists are in [the M1b record](m1a-recorded-vetoes.md). **Exception:** the supported security consequence and concrete repair direction are not demonstrated, because both are owned by an evaluated capability declaration and the registry is deliberately empty. M1b's comparison half is verified; its explanation half waits on M4.
- [x] 10.10 Render the experimental evaluation finding with its consequence and repair
- **The remaining half of M1b's exit, named by the 10.9 run.** The comparison is verified, but the milestone also asks for a developer-readable finding carrying a supported security consequence and a concrete repair direction. Both come from an evaluated capability declaration and the registry is deliberately empty, so nothing renders them.
- Generate an explicitly unreviewed, experimental declaration for this capability with the existing registry tool, and render the finding through the same grounded-template path the admitted case uses. Keep eligibility disabled: an experimental declaration is not enabled and not PASS, so admission must still reject it and no normal hook alert may be advertised.
- Do not write consequence or repair text that no declaration supports. Refusing generic advice without a supported context and operation is the rule admission exists to enforce, and the experimental path must not become a way around it.
- Done when the experimental output shows family, exact location, witness, change attribution, consequence and repair direction for the vulnerable revision, is labeled experimental throughout, and the production result still reports no actionable defect with zero admitted defects.
- _Boundary: Result and Hook Adapter, Delta and Actionability Policy, Evaluation harness_
- _Depends: 10.9_
- _Requirements: 3.1, 3.2, 3.3, 5.1, 5.5, 6.3_
- _Verified 2026-09-16:_ 1,386 tests passed, nine skipped; Ruff/format/mypy passed. Three evaluation findings rendered on the vulnerable revision with exact location, witness, change attribution, provenance, consequence and repair, all from the declaration's templates through the grounded-template path. The declaration reports verdict experimental and enabled false, the production result still reports no actionable defect, and all three artifacts carry zero admitted defects. Controls prove an ungrounded template is refused, an uncovered operation symbol is stated, no matching declaration leaves both sentences absent with the reason, and a declaration claiming PASS still admits nothing because the rendering path is outside admission. Evidence: `benchmarks/measurements/2026-09-16-nodegoat-m1b-explained.json`.
- [x] 11. Fit the same useful analysis into the push budget (release milestone M2)
- [x] 11.1 Give the backend a transaction-owned engine session
- The measured cost is JVM startup, not analysis. In the M1b runs each disposable `joern --script` invocation cost 13 to 23 seconds while the work itself took under a second, and the 2026-09-14 lifecycle probe measured one session at 12.8 s to start, 3.6 s to load a graph and 0.22 to 0.47 s per warm request.
- Own the session inside the existing backend for the length of one transaction: loopback only on an ephemeral port with no published port, a private working directory, one shared deadline, serialized requests, explicit graph identity and epoch, and kill-and-reap of the whole process group on cancellation, failure or exit.
- An HTTP status is not an answer contract. Require a per-request receipt written to a fresh private file and verified before the answer is accepted, the way the probe's repaired instrument did after its first attempt read an empty stdout as a census.
- Poison the session on any crash or malformed response so a failed session can never answer a later request, and fall back to the disposable path rather than reporting a zero.
- Done when a controlled hanging in-session request is cancelled inside the allowance with no surviving process group, a crashed session answers nothing afterwards, and the disposable path still works unchanged.
- _Boundary: contributor-scan runtime, Cache and Engine Boundary_
- _Requirements: 4.1, 4.2, 7.3_
- _Verified 2026-09-16:_ 1,391 tests passed, nine skipped; Ruff/format/mypy passed. Sixteen session tests drive a real fake-server process: receipt verified before an answer is accepted, a mismatched receipt or crash poisons the session so nothing answers afterwards, an empty body with its receipt is a valid empty answer, a hanging request is cancelled and the process group is gone inside the allowance, and a launch failure is recorded rather than raised. The probe took four iterations, each failing loudly: `runScript` is absent from the server REPL; the server answers success even for a compile error, so the receipt is the only proof; a shipped `@main` script cannot be nested in a block, so definitions go to the top level unwrapped; and the REPL shadows `Console`, so capture must qualify `scala.Console.withOut`. Evidence: `benchmarks/measurements/2026-09-16-session-script-execution.json`.
- [x] 11.2 Prove every shipped query kind answers identically through both transports
- Run taint, dominance, configuration and census over the same graph through the session and through the disposable path and require identical payloads, including empty answers and the distinction between an empty answer and an unavailable one.
- Cover a malformed or absent response, a switched graph, a second graph loaded in the same session, and a request arriving after cancellation. Equivalence on a census alone does not establish equivalence for security queries.
- Done when every shipped query kind matches byte for byte on a real graph, and every failure shape is reported as unavailable rather than empty.
- _Verified 2026-09-16:_ taint, dominance and configuration all return byte-identical payloads through both transports on a real NodeGoat graph, with parameters built by each arbiter's own request function through the driver's family lookup. Failure shapes are correct: an empty answer stays present and distinguishable from an unavailable one, an unknown query and an absent graph are both unavailable, and two graphs in one session each answer about the graph the request names. Amortized over five taint requests the session took 22.48 s against 71.96 s disposable, all answered and identical; a single request per session is its worst case and is recorded as such. Evidence: `benchmarks/measurements/2026-09-16-session-transport-equivalence.json`.
- [ ] 11.3 Route the batch path through the session behind an explicit setting
- Consume the session from the existing batch and census paths, keeping the disposable path as the default and as the fallback. No capability, ranking decision or evidence semantics changes; only the transport does.
- Account for session startup, load, requests and teardown inside the one shared deadline, and record them as separate stage costs so the next decision is made on measurements rather than estimates.
- Done when a replay produces the same evidence and admission through either transport under a sufficient budget, with per-stage costs recorded for both.
- _Boundary: Cache and Engine Boundary, Push runner_
- _Depends: 11.2_
- _Requirements: 4.1, 4.3, 6.4, 7.3_
- [x] 11.4 Remove the startup cost from the critical path where the measurement says it pays
- With 11.3's stage costs in hand, decide what else to change. The candidates the M1b runs point at are the overlay pass, 15.8 s in a separate invocation, and session startup, 12.8 s, which may overlap the frontend build rather than follow it.
- Change only what the measurement implicates, and keep each change's correctness evidence separate from its timing evidence. Do not substitute a faster census for a completed security check.
- Done when the recorded stage costs after the change are compared against the same profile before it, with any regression in completed checks reported rather than traded away.
- _Boundary: contributor-scan runtime_
- _Depends: 11.3_
- _Requirements: 4.1, 4.2, 6.2, 6.5_
- _Verified 2026-09-17:_ 1,399 tests passed, nine skipped; Ruff/format/mypy passed. The overlay, the last fixed cost bypassing the session, now runs in-session and the saved graph is verified rather than assumed. The first session-enabled replay then showed correct detections and no speedup, and a targeted probe disproved the sequence hypothesis before finding the real defect: a build runs on a copy of the backend, so the session it started was never closed, leaking a JVM holding the configured heap, twice per transaction, enough to exhaust the container. With that fixed the session stays alive for the whole transaction and no `session_query_failed` appears. Measured on the same pins and budget, the vulnerable revision went from 326.4 s to 243.3 s, head query time from 63.6 s to 32.0 s and base from 85.1 s to 51.2 s, with dominance and configuration each falling from about 14 s to about 4 s. Detections unchanged. Evidence: `benchmarks/measurements/2026-09-17-nodegoat-session-transport.json`.
- [x] 11.5 Rerun the frozen seven-class runtime profile and decide
- Rerun the unchanged seven-class profile: identical-tip, function edit, dependency edit, configuration edit, cold, growth and multi-ref, three samples each, retaining every timeout and unresolved check. Identical-tip reuse cannot substitute for warm changed-code work.
- Verify the M1 outcome still holds on the same three revisions through the production rules, so speed is not bought with a lost detection.
- Apply the existing gates without alteration: representative warm changed-code p95 at most 30 s, cancellation and reporting at most 2 s, supported-check completion at least 95%. A failed gate stays NO-GO and names the dominant remaining cost.
- Done when one versioned scorecard reports p50 and p95 with sample counts beside the M1 correctness outcome, and states GO or NO-GO against the unchanged gates.
- _Measured 2026-09-17, verdict NO-GO:_ the unchanged seven-class profile, three samples each, session enabled. Every changed-code workload still times out at the 30-second deadline, and identical-tip got worse, p50 rising from 18.5 s to 24.2 s. The cause is recorded in the artifacts rather than inferred: 13 transactions report `session_unavailable:session_startup_timeout`, so under the real deadline the session usually never starts and everything falls back. Startup is 9 to 13 seconds, about a third of the whole budget, spent before any analysis. Where a session did start, on cached-graph transactions with little remaining work, it cost more than it saved. Correctness is unchanged: the same three revisions through the production rules with the session on reproduce the M1 verdict. Evidence: `benchmarks/measurements/2026-09-17-session-runtime-profile.json`. **Dominant remaining cost: engine startup itself, which a transaction-owned session relocates but cannot remove.**
- [x] 12. Security capability work found during M1a and M1b
- Group 11 established that M2 needs graph reuse rather than more runtime tuning, and that decision is recorded there. This group returns to what the tool can actually establish about a codebase, which is where the remaining release risk sits: two of the three shipped families cannot complete their change context at all, and admission rejects genuine findings on a mis-extracted operation symbol.
- [x] 12.1 Extract the operation symbol from the operation, not from the statement
- **Found 2026-09-16 in the M1b explanation run.** Admission takes the text before the first parenthesis as the operation symbol, so `const preTax = eval(req.body.preTax)` yields `const preTax = eval`. The demonstration declaration had to enumerate each assignment spelling for its coverage check to pass.
- A qualified declaration cannot enumerate every assignment a codebase might write, so admission would reject genuine findings on `operation_semantics_mismatch`. Take the symbol from the called operation the engine reported, preserving receiver spelling, and leave an unresolved or dynamic spelling ineligible as it is today.
- Done when the symbol for an assigned call is the call's own name, a receiver-qualified call keeps its receiver, a dynamic spelling stays ineligible, and a declaration naming the plain symbol covers the assigned form.
- _Boundary: Delta and Actionability Policy_
- _Requirements: 3.1, 3.2, 3.4_
- _Verified 2026-09-17:_ 1,403 tests passed, nine skipped; Ruff/format/mypy passed. The symbol is now the trailing qualified callee rather than everything before the first parenthesis, so `const preTax = eval(...)` yields `eval` and a declaration naming the call covers the assigned form. Receiver spelling is preserved, so `$wpdb->prepare` and `prepare` remain different operations. A spelling that is not a qualified name, such as a dynamic `handlers[name](x)`, yields the statement text, matches no declaration and stays ineligible exactly as before. The recorded-veto report reads the same helper rather than repeating the extraction.
- [x] 12.2 Supply per-family context producers for dominance and configuration
- The M1b exit run left 231 of 436 head questions reporting `change_context_incomplete` from `context_projection_unavailable` and `context_scope_empty`, because those families' queries export no context. They did not block the Node injection case, so M1b correctly left them alone, but their families stay experimental until they can answer.
- Give dominance and configuration their own context producers describing the actual scope of the query they accompany. Do not copy taint context into them: dominance evaluates sibling operations across a file and configuration has different scope and obligations.
- Done when a dominance and a configuration question complete their change context on a real repository, with missing extents and unsupported reasoning still explicit.
- _Verified 2026-09-17:_ 1,405 tests passed, nine skipped; Ruff/format/mypy passed. The gap was larger than a missing query block: the evidence pass dispatches taint only, because only taint has an evidence vector, so these families were never asked for context at all. Each query now describes its own scope, dominance the file's methods labelled operation, guard or sibling because its claim is about a file contradicting itself, and configuration the file's methods labelled setting or sibling. Neither copies taint's reachability projection, which would describe work they never do, and both report an empty scope explicitly. A new context pass asks each family with the flag set; a family whose query answers nothing keeps no context and stays unresolved, which a control asserts so no family can complete on another's context. Confirmed on a real NodeGoat graph: both produce a context summary and seven scoped methods. Evidence: `benchmarks/measurements/2026-09-17-per-family-context.json`.
- [x] 11.6 Decide what to do about engine startup, which the session cannot remove
- **Named by the 11.5 measurement.** A transaction-owned session amortizes engine startup across the invocations inside one transaction, and a warm changed-code push does not have enough of them: startup is 9 to 13 seconds of a 30-second budget, so the session either fails to start or costs more than it saves. Group 11 as designed cannot reach the gate.
- The options are a decision for the maintainer, not an implementation detail, because two of them change the product's shape. A session shared across transactions is a resident local service, which the accepted milestone spec explicitly does not imply. A smaller budget for what a push analyses is a scope change, not a speed change. A faster cold path would mean attacking the frontend and overlay costs directly, which the session does not touch.
- Record the measured cost of each candidate before choosing, and keep the 30-second and 2-second gates unchanged; a gate met by analysing less is not the same result.
- Done when one record states the candidates, their measured or estimated costs, their effect on the product's shape, and the maintainer's choice.
- _Decided 2026-09-17 by the maintainer: do not rebuild a whole graph per push._ Both startup attacks were measured and neither reaches the gate. The session helps a long transaction, 326 s to 243 s, and cannot help a 30-second one: startup is 9 to 13 seconds of the budget, so it usually declines. Removing JavaScript's second engine start is real, 11.0 s against 21.0 s on identical graphs, but its census gate was asymmetric and it was reverted after costing two engine starts on 12 profile transactions. What the profile names is not startup at all: a changed-code transaction reaches its deadline during the build, head build 25.2 s of 30 s with query time 0.0 s. The remaining cost is building a fresh whole-repository graph for every pushed revision. **Direction: reuse the graph across revisions instead of rebuilding it, rather than further optimising the rebuild.** Not scheduled here; M2 stays unmet and the 30-second and 2-second gates stay unchanged.
- [x] 12.3 Resolve function-scoped questions whose function the query cannot locate
- **Named by the 12.2 measurement.** With per-family context in place, head completion rose from 205 to 298 of 436 and `context_projection_unavailable` fell to zero. Every remaining unresolved question reports `context_scope_empty`, and they sit in three files: `app/routes/index.js` (114), `app/routes/tutorial.js` (12) and `server.js` (12). None is access control.
- These are function-scoped questions whose named function the query cannot locate in the graph, so the scope it describes is honestly empty. It is the same query-to-operation identity problem M1a recorded at the witness end, appearing at the scope end.
- Carry verified provenance from the region's declared function to the method the engine actually holds, and leave an unresolvable name explicit. Do not widen a question to the whole file to make it complete, and do not inject a declared name into a region.
- Done when a question whose function the graph holds completes, a question whose function it does not hold stays unresolved with the name recorded, and no question completes by widening its scope.
- _Verified 2026-09-17:_ 1,406 tests passed, nine skipped; Ruff/format/mypy passed. The cause is recorded rather than guessed: `app/routes/index.js` requires seven handler modules and defines none of them, so a question naming `handleContributionsUpdate` there names a function that file genuinely does not hold and the empty scope is correct. Taint and configuration now separate the two causes, reporting `context_function_unresolved` with the name when a question names a function the file does not define, and `context_scope_empty` only when the file has no methods at all. The driver no longer adds its generic empty-scope gap on top of a query that already explained itself, and the new gap is question-owned so it cannot block another question's comparison. Completion is unchanged by design: a question whose function the graph does not hold stays unresolved, and nothing completes by widening its scope to the whole file.
- [x] 12.4 Attribute a region to the file that defines its handler
- **The region-attribution half of 12.3.** A route-registration module registers handlers other modules define. `app/routes/index.js` requires seven handler modules and defines none, yet regions there were scoped to `handleContributionsUpdate` and its siblings, so the graph rightly held no such method and every question in those regions was unresolvable: 138 across six families.
- The entry record still names the handler, because the authorization question is which middleware guards which handler and that is cross-file by nature. It now also records whether the file defines what it names, and a region built from an entry that does not is scoped to the file rather than to a method the file lacks.
- _Verified 2026-09-17:_ 1,407 tests passed, nine skipped; Ruff/format/mypy passed. A first attempt changed the mapper to stop naming cross-file handlers and was reverted: an existing control proved the name carries the authorization attribution, so the fix belongs in region construction, not in the mapper. Controls assert the registering file asks about itself with no function-scoped region, while the defining module keeps its function-scoped region.
- [ ] 13. Show the shared design transfers across languages (release milestone M3)
- Status 2026-09-17: Python established, PHP blocked by runtime on a large repository. M3 is not met. Its exit asks for Node, PHP and Python together, and two of the three are in hand.
- [x] 13.1 Replay the declared Python family through the shared core
- Generalize the replay instrument so a case family is an argument rather than a hard-coded target: a harness that only knows one language cannot say whether the design transfers. Targets, pins and expectations come from the committed inputs.
- VAmPI is a regression snapshot with the same revision on both sides, so it can establish detection and completion but never novelty. The mode asserts that too: with nothing changed, a target reported new would be a comparison defect rather than a finding.
- _Verified 2026-09-17:_ all three declared targets detected at their declared lines with their questions answered, 35 of 35 questions complete on every case, nothing reported new, zero admitted defects. The SQL injection at `models/user_model.py:73` is the evidence the milestone names as previously unresolved. Two of the three are access control, the family that could not complete a question anywhere before group 12, so this is the first time broken object-level authorization has been asked on Python. Evidence: `benchmarks/measurements/2026-09-17-m3-python-transfer.json`.
- _Limits:_ identical revisions establish detection, not novelty; VAmPI is inspected teaching data and cannot qualify production Python support.
- [ ] 13.2 Replay the declared PHP pair through the shared core
- PMPro 2.9.7 against 2.9.8 with its upstream prepare fix, plus a comment-only benign variant. A different language, a different mechanism and a fix this project did not author.
- The first attempt selected one case of three, because the manifest labels a case by what it is while the case id says what the comparison does; the harness now maps label to role explicitly. That run also exhausted a 900-second budget on the single case it ran.
- Done when all three comparisons complete with their declared target evidence recorded, or the budget they need is stated as a finding rather than reported as a pass.
- _Measured 2026-09-17, PHP transfer NOT established._ All three comparisons exhausted a 3000-second deadline with a 1200-second query ceiling and produced no artifact, so there is no target evidence at all. The engine reports `cpg batch taint failed` twice per case on the 1,274-file PMPro tree: first an exception whose trace runs through ujson array transformation, then a timeout on the retry. Budgets tried: 900 s, 1200 s and 3000 s deadlines against 300 s and 1200 s query ceilings. Pins, source receipts and the declared target are all correct, so the question was asked properly and the engine could not answer it. This is a runtime blocker on a large repository, not evidence about whether the design transfers to PHP. Evidence: `benchmarks/measurements/2026-09-17-m3-php-transfer.json`.
- _Two harness defects found and fixed on the way, each of which had made an earlier run look like a transfer failure:_ the manifest labels a case by what it is while the id says what the comparison does, and filtering on one while keying expectations by the other ran a single case; and the per-query ceiling is taken as the minimum with the remaining transaction time, so raising only the deadline could never have helped.
- _Retried 2026-09-18 with batch splitting, still not established._ A batch that fails whole is now retried in eight parts, which a probe made look promising: the evidence pass answers all 4,425 PMPro regions in one batch in 29.7 s with a 681 KB response, and cost is fixed rather than per request, so neither count nor size is near a limit. That probe sent `evidenceOnly=true` and therefore measured the structural pass, not the failing one; recorded because those numbers would otherwise read as evidence that PMPro's taint work is cheap. With a 4500-second deadline and a 900-second query ceiling the run was stopped after 26 minutes with no case completed. Splitting also makes an unrecoverable batch slower to abandon, since each part burns its own share of the ceiling.
- _Correction 2026-09-20: the malformed-CPG bug is fixed and was never what these runs were hitting._ joernio/joern#6281, this project's own report, is a PHP `global` inside a closure producing a node with two AST parents, which made the dataflow overlay throw and cost the WHOLE graph. It was filed against 4.0.623 and fixed by #6270 in **v4.0.625**, which is the version the image has pinned since before any measurement here. Verified today on the real trigger: PMPro's `includes/fields.php`, the file the original bisection found, builds and applies the default overlays cleanly on 4.0.625 and on 4.0.631. So what remains under this task is a RUNTIME failure and must not be re-diagnosed as the graph bug.
- _Quantified 2026-09-20, and it is a runtime failure._ A taint batch of 2,526 questions over the 673-file checkout does not complete in thirty minutes. Raising the query ceiling six-fold, from 300 to 1,800 seconds, changed nothing but how long the failure took: query time went from 363 s to 1,874 s and zero questions completed in both. The eight-way split retry did not rescue it, finishing its parts in about three minutes against a deadline with 3,300 seconds still on it, which is worth its own look. Joern 4.0.625 and 4.0.631 are indistinguishable here, 627.7 s against 619.1 s. The engine pin stays at 4.0.625: the newer release was built and measured and is equivalent on this subject, so there is no benefit to weigh against re-verifying every JavaScript and Python measurement. Evidence: `benchmarks/measurements/2026-09-20-php-runtime-after-graph-fix.json`.
- _Corrected 2026-09-20: PHP is not silent, and this project's own instrument is why it looked silent._ The census harness ordered its verdict completion-first, so a family with eighteen findings and no completed question reported "asked but no question completed" — and only the 500-region runs, the ones that time out, had been looked at. Fixed: findings are checked first, because a question-outcome count is bookkeeping and cannot hide the thing being censused. NodeGoat and VAmPI were never affected, since they complete every question.
- _What PHP actually detects on PMPro._ At 25 regions, 19 findings in seven minutes with no timeout; at 100 regions, 43 findings in twelve minutes with no timeout. Among them a true positive at `adminpages/discountcodes.php:749`: `sanitize_text_field($_REQUEST["s"])` reaching `$wpdb->get_results($sqlQuery, OBJECT)` through `$sqlQuery .= "WHERE code LIKE '%$s%' "` with no `prepare`. That is the class of CVE-2023-23488, which `php.toml`'s own comment names as the reason those WordPress sanitizers are excluded — the prediction and the measurement agree.
- _And a measured precision cost beside it._ Output encoding reports eighteen findings at 25 regions and the three sampled are all properly escaped, two with `esc_attr` and one coerced with `intval`. Those escapers are deliberately absent from a sanitizer list that is flat across families, because none of them stops SQL injection; keeping the injection finding above costs this family its precision, and that is the documented tension's price.
- _The scale limit, quantified._ Roughly one to two seconds per question. 25 and 100 regions complete cleanly; 500 regions asks 2,526 questions, needs more than the 1,800-second ceiling and is killed. Nothing pathological, and the retry's fast failure is still unexplained.
- _What is NOT the problem, each measured rather than assumed:_ the malformed-CPG bug (fixed in 4.0.625, verified on the real trigger file); the dataflow overlay (42 s on the full 637-file graph); the engine version (4.0.625 and 4.0.631 indistinguishable); and the bundled Stripe, Braintree and Twocheckout SDKs under `includes/lib`, which the vendored list does not cover — that was the hypothesis and the overlay costs 42 s with them and 37 s without, so it is refuted.
- **The remaining real gap is bookkeeping, not detection:**`questions_completed` is zero in every PHP run including the ones that find things, with `cross_partition_semantics_unresolved` and `partition_file_census_incomplete` present throughout. Findings and question outcomes disagree in public, and the admission policy reads the outcomes. Group 12 raised completion for JavaScript and Python and PHP was never measured. Evidence: `benchmarks/measurements/2026-09-20-php-detection-corrected.json`.
- **IDENTIFIED 2026-09-22.** The cost is the number of CALL NODES a family's sink names match in the repository. Not the request count, not the declared sink count, not any request parameter. `reachableByFlows` is asked per matched sink, and PHP's output-encoding sinks are `echo` and `print`, which a WordPress plugin writes thousands of times.
- _Three controlled sweeps on one graph, built once._ Batch size is exonerated: 400 requests answer in 170 s and cost per request FALLS with size, because roughly 70 s is fixed JVM start and graph load. Request parameters are exonerated: at 200 requests the full production shape, call depth 3 with parameter sources, costs 221 s against a 150 s baseline. The family is the variable: at the same 100 requests, SQL injection costs 147 s with six declared sinks, path costs 194 s with sixteen, and output encoding costs 704 s with seven.
- _And the driver named its own expensive step, once it was asked to._`taint: 2435 request(s) in 45.5s, answered` for the evidence pass, then `taint: 315 request(s) in 1200.1s, unanswered` for arbitration across all families, then a split abandoned for want of time.
- _Why it survived since September:_ every reproduction outside the driver carried the SQL sinks, because that is the family the declared target belongs to. The one family that is cheap on this subject was the one being re-measured, and nine hypotheses were eliminated that way.
- _Why splitting never rescued it:_ a failed batch is split into eight parts BY REQUEST COUNT, and each part carries the same everywhere-sinks, so the whole is no cheaper. Splitting by request count cannot help when the cost is per matched sink. Evidence: `benchmarks/measurements/2026-09-22-php-query-cost-identified.json`.
- **Blocker identified; the fix is contributor-scan 7.7.** Six hypotheses were measured and eliminated: batch size (17,700 requests answer in 44 s with a 2.6 MB response), dataflow cost (50 execution requests in 24 s), call depth 3 with parameter sources (24.7 s for 20), argument length (a 16 KB hook table against a 128 KB cap), hook callback resolution (1 s difference with 233 hooks), and build or sharding (79 s, no shards). Preparation measures 273.5 s. Every reproduction of the failing 22,125-request batch outside the push path answers in under a minute, and the difference from the production invocation is not known. The exception has never been read, because the backend logged a stack trace's tail and the corrected extractor reports its head, which on this engine is JVM startup noise. Evidence: `benchmarks/measurements/2026-09-19-php-blocker-open.json`.
- [x] 14. Ask every declared family whether it establishes anything (precondition for M4)
- Seven families are declared and two were proven end to end. A declared family that has never been asked whether it finds anything is advertised coverage, and M4 cannot qualify a capability that establishes nothing.
- [x] 14.1 Census the seven declared families on NodeGoat and VAmPI
- Run production discovery and scan over each whole checkout and report, per family, the regions that offered it, the questions asked, the questions completed, and what was found with the witness and rung. No comparison, no novelty, no admission: detection only.
- A family that answered and found nothing must be reported differently from one that was never asked, and neither may read as clean.
- _Verified 2026-09-19:_ 1,411 tests passed, ten skipped; the six Joern-gated query tests pass in the image. Four families establish findings across the two subjects: injection (three `eval` flows on NodeGoat, one SQL flow on VAmPI), access_control (two sibling contradictions on VAmPI), config_secrets (one on each) and untrusted_destination (one open redirect on NodeGoat). All 299 NodeGoat and 35 VAmPI questions completed. Evidence: `benchmarks/measurements/2026-09-19-family-detection-census.json`.
- _The census found one false positive and it is fixed._ NodeGoat's only prototype report was the word `set` inside the comment `// set these up in case we have an error case`: the matched node is the `<operator>.assignment` holding the whole 1,000-character signup handler, and its text was searched entire. `sinkMatches` now confines its text clause to the callee and excludes operator nodes, which is the fourth instance of this bug class after `fgets`/`gets`, `resolve`/`resolveUrl` and `user`/`users`. Measured on NodeGoat's non-vendor calls: injection sinks 28 to 3, prototype 13 to 2, untrusted_destination 26 to 7, with no true positive lost. `req.query` had been an injection sink, because `query` is word-bounded inside it.
- [x] 14.2 Give untrusted destination the sinks a Node server actually calls
- **Named by the 14.1 census.** NodeGoat's documented SSRF is `needle.get(req.query.url + req.query.symbol)` at `app/routes/research.js:16`, and `needle.get` is absent from a vocabulary holding axios, fetch, got, `http.get` and request. The family works and found the open redirect in the same repository; it could not ask about the request.
- This is a vocabulary gap, so fix it as data rather than as a query change, and state which client libraries the declared set now covers.
- Done when the research handler's request is asked about, with a witness naming the call, and no benign client call is reported.
- _Verified 2026-09-19:_ 1,411 tests passed, eleven skipped; seven Joern-gated query tests pass in the image; Ruff/format/mypy passed. One data change, no query change. The declared set now covers axios, got, needle, request, superagent, unirest, undici and fetch as bare module tokens, plus `http.get`, `http.request`, `https.get` and `https.request` dotted. Bare is the documented call form of each module and also covers its verb methods, because a token matches word-bounded against the callee and `axios` is bounded inside `axios.get`; `axios.get` and `axios.post` were therefore dropped as subsumed. `http` and `https` stay dotted because their namespace also holds `createServer`, and a bare token would have claimed every server a repository starts.
- The research handler's request is now reported at `app/routes/research.js:16`, model_entailed, witness `req.query.url + req.query.symbol -> needle.get(url, ...)`. This family goes from one finding to two on NodeGoat and every other family is unchanged, with 299 of 299 questions still completing. Measured on that graph, `needle` matches exactly one call and the eleven other clients match none, in first-party or vendored code, so NodeGoat shows the gap closing and cannot show those clients' noise.
- A control asserts both halves rather than only the recall half: a handler calling `axios.get` with a literal and `needle.get` with `req.query.url` reports the tainted client alone, and the control reads the shipped vocabulary rather than a copy so removing the client fails it. The CWE-601 redirect entry is untouched; a Location-header redirect is value-shaped and stays outside what this family asks. Evidence: `benchmarks/measurements/2026-09-19-untrusted-destination-vocabulary.json`.
- _Boundary: shipped semantic facts_
- _Depends: 14.1_
- _Requirements: 6.2, 8.1_
- [x] 14.3 Decide what output encoding claims for server-rendered JavaScript
- **Named by the 14.1 census.** All six declared JavaScript sinks are DOM sinks. NodeGoat's XSS is server-side, rendered into Swig templates and written with `res.write`, so the 44 questions this family completed could not have found it whatever the code did.
- Either the family gains server-side sinks and template-rendering sinks, or it declares itself client-side only. Both are honest; silence while declaring XSS coverage is not.
- Done when one record states the decision, and the family either asks about `res.write` and template rendering or says in its declaration what it does not cover.
- _Verified 2026-09-19:_ 1,415 tests passed, thirteen skipped; nine Joern-gated query tests pass in the image; Ruff/format/mypy passed. **Decision: the family asks about a value reaching a rendering call and states in its own declaration that it cannot decide template-mediated escaping.** Whether `res.render` is safe depends on the engine's autoescape setting and on the template's markup, neither of which is at the call, so `res.render` is not a sink and the family's `limits` now says so. The setting is claimed where its evidence is instead, as a configuration defect.
- It now establishes a finding: `app/routes/research.js:25`, model_entailed, `req.query.url + req.query.symbol -> res.write(body)`. The handler fetches a URL the caller chooses and writes the response body into a text/html response, so the document served from this origin is the attacker's. The record states how coarse the step is: the path from `url` to `body` runs through the client's callback parameter, which the graph treats as reachable from the call's arguments, so the conclusion rests on the call's semantics rather than on what the graph established. Five of seven families now establish findings. Evidence: `benchmarks/measurements/2026-09-19-output-encoding-family.json`.
- _Four defects found on the way, three of them false positives this change would otherwise have shipped._ The permissive-value set was shared across every setting while a boolean's polarity belongs to the setting, so `swig.setDefaults({autoescape: true})` would have been reported as the defect on the line that repairs it; `ConfigSpec` now carries per-setting values and Jinja2's `Environment` was moved out of Python's shared list for the same latent reason. `config.sc` matched a dotted setting on its trailing segment alone, so `marked.setOptions` would have claimed an editor's `setOptions`. `taint.sc` asked its flow query of every argument including the receiver, so `res.end()` — which takes no data — was reported exactly like the `res.write(body)` two lines above it. And the arbiter sorted a file's settings by line number as text, so which setting a file reported depended on the spelling of its line numbers.
- _Boundary: shipped semantic facts, family taxonomy_
- _Depends: 14.1_
- _Requirements: 6.2, 8.1_
- [x] 14.6 Report every permissive setting in a file, not the first one
- **Named by the 14.3 measurement.** The configuration arbiter returns one verdict per question. NodeGoat's `server.js` holds a permissive session at line 78 and template auto-escaping disabled at line 135 in the same function, so the escaping defect is detected — measured model_entailed against the real query and the real arbiter — and never reported. A contributor is shown one of two defects and told nothing about the other.
- This is the arbiter's contract, not a data gap: `verdict()` returns `Verdict | None` and the scan collects one per question. Changing it touches every family that returns a verdict, so it is scheduled rather than folded into a vocabulary change.
- Done when a file with two permissive settings reports both, with their own sites and witnesses, and no family reports a defect twice.
- _Verified 2026-09-20:_ 1,417 tests passed, seventeen skipped; thirteen Joern-gated query tests pass in the image; Ruff/format/mypy passed. Configuration now returns one verdict per setting whose value the model could read. The pipeline already emitted one finding per entailed answer, so only this arbiter was still folding them to one. NodeGoat's `server.js` reports both defects with their own sites, `server.js:78` for the permissive session and `server.js:135` for the disabled template auto-escaping, which is the defect task 14.3 could detect and never report. Its applied fix at line 126 stays silent. VAmPI is unchanged and no other family moved. Evidence: `benchmarks/measurements/2026-09-20-every-permissive-setting.json`.
- _Three boundaries drawn rather than assumed._ A setting whose value is COMPUTED stays one verdict per region: that band is a question for the judge, and returning every undecided setting would multiply model calls for claims the graph did not establish. A call carrying two permissive literals is one finding, because `cors({origin: true, credentials: true})` is one misconfiguration. And config verdicts now carry a `path:line:function` location, without which the plural answer collapsed straight back to one in the deduplicator, which is the same gap dominance carried until its findings stopped arriving with no line.
- _Dominance is left answering once per region, and that is settled rather than pending._ Its claim is that a region contradicts itself, so the region is the unit and a second verdict would restate the same inconsistency. The pipeline says so where it used to call this a known gap.
- [x] 14.4 Raise the access-control obligation where the application splits route from query
- **Named by the 14.1 census and measured with the real dominance query.** Every modelled data operation in NodeGoat lives in `app/data/*-dao.js`; none lives in a route handler file. Operations are scoped to the region's own file and access_control regions are the handler files, so no obligation is raised where the question is asked, and inside the DAO files every operation is equally unguarded so the sibling rung has nothing to contradict. The family answers correctly on VAmPI, which keeps handler and query in one file.
- The file scope was chosen deliberately, because unscoped rows attributed a model's unguarded query to a handler of the same name. Widening it is not the fix; carrying the obligation along the call the handler actually makes might be.
- NodeGoat's two documented defects are also outside what the model can claim: an authenticated handler reading `userId` from `req.params` (`app/routes/allocations.js:17`), and a commented-out `isAdmin` in the route table (`app/routes/index.js:58`). Record which of the two this task is for, and do not silently widen the claim to the other.
- Done when a route-and-DAO split raises the obligation at the handler with a witness naming both sites, and a same-name model function is still not attributed to a handler.
- _Verified 2026-09-20:_ 1,415 tests passed, fifteen skipped; eleven Joern-gated query tests pass in the image; Ruff/format/mypy passed. **This task is for the first defect, the route-and-DAO split itself.** The obligation now follows the one named call the handler writes, out of the region's file and no further. Attribution is by the graph's own provenance, not by name: jssrc2cpg leaves the callee an external stub, and that stub's fullName records the file and member where the implementation was written, so a module the handler never imports is never consulted. A row is attributed to the region's DECLARED function, which is what lets a question about `displayAllocations` match a graph that holds a `<lambda>`. A carried row is also checked for a guard at the CALL, because that is where a handler establishes who is asking, and the method holding the carried operation joins the question's context scope so the query's own module cannot change unattributed.
- Measured on NodeGoat: 15 handlers asked, obligations raised at zero before and at six now, three of which reach a verdict. All three are corroborated and none is reported without a judge. VAmPI is unchanged at two entailed findings, because it keeps handler and query in one file and carries nothing. Evidence: `benchmarks/measurements/2026-09-20-access-control-module-split.json`.
- _A carried obligation corroborates, it does not entail, and that was measured rather than assumed._ Entailing it reported `handleSignup`, which looks a user up by name to check the name is free during registration and is public by design, because two authenticated siblings in the same file pass `req.session.userId` into the same module. The entailed rung's justification is that one FILE contradicts itself, same file and same trust context; carried, the compared operations live in a data-access module that handlers of different trust contexts share. The corroborated band already asks the right residual: does this operation require a check, or is the route legitimately public?
- _What this does not reach, recorded rather than implied:_ NodeGoat's documented IDOR at `app/routes/allocations.js:17` still raises nothing, because its module reads with `allocationsCol.find(...)` and `find` is not a modelled operation (task 14.8). The commented-out `isAdmin` at `app/routes/index.js:58` is not claimable by this abstraction at all, since the evidence that `/benefits` needs a stronger guard than its siblings exists only in a comment.
- [x] 14.7 Decide the sibling scope, and see identity that a destructuring spelled away
- **Named by the 14.4 measurement.** Two of NodeGoat's three corroborated claims are correct code: `const { userId } = req.session` followed by `dao.getByUserId(userId, ...)` is identity-constrained, and the destructuring erases the `session.userId` spelling the token list holds, so the guard is invisible at the call. A judge would be asked about handlers that are not defects.
- The contradiction that would make NodeGoat's IDOR entailed is cross-file: `allocations` passes a URL parameter while `contributions` and `profile` pass session identity. The sibling set is one file, and widening it is a decision, not an implementation detail: measured, a route file mixes public with authenticated handlers, so application-wide siblings would contradict every login and signup handler there is.
- Record the measured cost of each candidate before choosing, and do not widen the scope to make a finding appear.
- Done when one record states the candidates and the choice, a destructured identity read discharges the obligation it actually discharges, and no public route is reported.
- _Verified 2026-09-20:_ 1,415 tests passed, sixteen skipped; thirteen Joern-gated query tests pass in the image; Ruff/format/mypy passed. A destructured identity read now discharges what it reaches, through a bounded three-hop walk back over the method's own assignments from the names the operation is GIVEN. `req.session` joins the identity sources and is only safe because of that requirement: what discharges an object-level obligation is that the value PASSED to the operation comes from the authenticated context, not that a session is mentioned nearby. On NodeGoat `displayContributions` and `displayProfile` go silent, which is correct code no longer put to a judge, and VAmPI is unchanged.
- _Two exclusions the measurement forced, each caught by a page whose whole purpose is the leak it demonstrates._ Taking identifiers from the whole argument subtree credited NodeGoat's memo listing with a discharge, because `memosDAO.getAllMemos((err, docs) => ... userId ...)` reads every memo and only RENDERS the session's user; a callback is not a constraint, so method references are excluded. Then the session read still dominated the query and the dominating-guard clause credited that instead; an assignment is a read of the authenticated context rather than a check applied to it, so operator calls are excluded from that clause, which is the same exclusion the sink matcher needed. The leak stays reported.
- _The sibling scope is decided by measurement, and the choice is to keep the file._ On NodeGoat the region's own file entails two handlers, both public routes and both false, and no true defect at all. Every handler in the application entails six: four true, and the same two public routes. Both scopes carry the same false positives, because neither can tell a public route from a protected one — which is a semantic question and already the corroborated band's residual. With the carried rung capped at corroborated the scope changes no verdict, only the witness, so widening it would buy a repository-wide walk per question and no precision. What would change the decision is a discriminator for a public route; with one, the wider scope is worth four true entailments on this subject. Evidence: `benchmarks/measurements/2026-09-20-sibling-scope-and-identity.json`.
- [x] 14.8 Model the collection read an application actually writes
- **Named by the 14.4 measurement.** NodeGoat's documented insecure direct object reference reads with `allocationsCol.find(searchCriteria())`, and `find` is absent from an operation vocabulary holding `findOne`, `findAll` and `findById`. It is the commonest read there is, so the family's obligation is missing at exactly the shape it exists for.
- It is absent for a reason: `Array.prototype.find` is everywhere in JavaScript and jQuery's `.find` is everywhere in a vendored bundle. A collection read takes a query document where those take a closure or a selector, so the discriminator is the argument's shape rather than the name.
- Done when a collection read raises the obligation, an array search and a selector do not, and the false-positive count is measured on a repository with vendored bundles rather than argued from the shape alone.
- _Verified 2026-09-20:_ 1,415 tests passed, sixteen skipped; twelve Joern-gated query tests pass in the image; Ruff/format/mypy passed. The obligation is claimed where the first argument has the shape of a query document, which the operation fact states as `query_document_arg` and the arbiter decides: no arguments at all, an object literal, or a builder call. Not claimed: a method reference, a literal, an operator call, and a bare identifier.
- _The false-positive count is measured, on wp-statistics, a WordPress plugin that ships jQuery._ Its assets hold 88 calls named `find` — 70 jQuery selectors, 13 array predicates and 5 selectors built at run time by a template literal or a concatenation, and no collection read at all. Zero became an obligated operation. The zero is a measured one and not an empty graph: the same run's census reports 60 files and 4,552 methods, and the files holding those 88 calls are in it.
- On NodeGoat all three `find` calls are collection reads and all three are claimed, which raises the obligation at four more handlers including `app/routes/allocations.js:displayAllocations` — the documented insecure direct object reference, which had raised nothing at any point in the pipeline before. Seven of fifteen handlers now reach a verdict, all corroborated, none reported without a judge, and the census is unchanged on both subjects. Evidence: `benchmarks/measurements/2026-09-20-collection-read-obligation.json`.
- _A bare identifier is deliberately not claimed._`col.find(query)` and `arr.find(pred)` are the same shape and neither repository measured holds one, so this is a stated false negative rather than an unmeasured false-positive risk.
- [x] 14.5 Ask the prototype and deserialization families somewhere they can answer
- **Named by the 14.1 census.** Prototype pollution has been asked on one repository and its only report was a false positive; deserialization was never asked at all, because VAmPI holds no such sink and NodeGoat is not a language where the family is declared. Neither has established anything anywhere.
- Find or record a subject where the family has a sink, and report what it does. A negative on a subject that contains the mechanism is a result; silence on a subject that does not contain it is not.
- _Verified 2026-09-20:_ 1,417 tests passed, eighteen skipped; Ruff/format/mypy passed. **Neither family has a subject that can answer, and each for its own reason, which is the result rather than a failure to produce one.**
- _Deserialization: the mechanism exists on exactly one subject and that subject cannot be queried._ Across four WordPress plugins and 1,164 PHP files there is one deserialization site, `maybe_unserialize` at `includes/fields.php:1111` in Paid Memberships Pro. The vocabulary held only the bare `unserialize`, which never reached it and correctly so: the matcher is word-bounded and the character before the token inside `maybe_unserialize` is a word character. That spelling is now declared, so the family has a sink to ask about — and the census asked 487 deserialization questions on that plugin and completed none, because no query over that repository completes at all. That is task 13.2's runtime failure, now quantified there, not a statement about this family.
- _Prototype pollution: no available subject contains the mechanism._ The only first-party matches on the Node subject are `app.set("view engine", "html")` and its sibling, Express settings whose arguments are literals, and the other JavaScript available is minified browser bundles with no request source. So the family's ability to answer is established on a control instead: a merge of the request body into an object is reported, and the Express setting beside it is not. Declared and never demonstrated has become demonstrated on a control and never on a real subject, which is a different and honest claim.
- _Boundary: Evaluation harness_
- _Depends: 14.1_
- _Requirements: 6.2, 8.1_
- [ ] 13.3 Re-run the declared PHP transfer at a scale whose questions complete
- **Named by the 2026-09-20 correction.** The not-established record came from two instrument defects, not from the analyser: the census verdict checked question completion before findings, and a 500-region budget asks 2,526 questions, which no ceiling this project sets can hold. Bounded to 25 and 100 regions, PMPro completes with no timeout and establishes a true-positive SQL injection at `adminpages/discountcodes.php:749`.
- Replay the three declared comparisons with the corrected harness at a region budget that completes, and record the declared target's evidence or its absence. State the budget as a limit of the run, not as a property of the repository: a bounded population establishes transfer, never coverage.
- Done when all three comparisons complete with their declared target evidence recorded, or the budget they need is stated as a finding.
- _Attempted 2026-09-21, and the attempt named its own blocker._ At 25 regions the first case now runs to completion and writes a full artifact, in 1,706 s, where at 100 it had exhausted a 2,400 s deadline with nothing. But 25 regions of 4,461 is 0.6% of the population, and the declared target at `classes/class.memberorder.php:936` was not among them: zero findings at the target, because the question was never asked. A bounded population establishes transfer only if it CONTAINS the target, and this one cannot.
- **So this task depends on contributor-scan 7.3, not on a larger budget.** The population that holds the target is affordable only if arbitration stops being paid for questions whose evidence says there is nothing to find. Raising the budget is the thing already measured to fail; making most of the budget unnecessary is the thing that has not been tried.
- _Two further observations from that run, recorded because they cost nothing to keep._ The head scan cost 1,178 s against the base's 219 s, which is worth explaining before M2 designs around either number. And every coverage reason was `ambiguous_line_correspondence`, one per file — so in the PUSH path, unlike the census, correspondence is what stands between this subject and a completed question. The partition work removed one blocker and revealed the next.
- _Sized and attempted 2026-09-22._ One whole-repository scan at 500 regions with a 3,600 s deadline puts the declared target in scope and finds it: 271 of 406 questions completed, 213 findings, 2,950 s. The three-case replay at that budget then exhausted a 7,200 s case deadline with no artifact, and the failure is mine to own twice over: the container mounts `src` live and I edited the backend heavily during the run, including a version that could hang on a timed-out pipe; and the push's EVIDENCE pass is one unportioned 22,125-request batch protected only by the ceiling, which it exceeded under the CPU and memory contention my concurrent test runs created. Neither is the algorithm. The evidence pass is now portioned, streaming keeps what a killed portion finished, and the rerun uses a frozen export of the committed source with per-invocation logging so the record names its own expensive step.
- _Attempted again 2026-09-23 on frozen `cda679c`, after contributor-scan 7.9, and it failed at a new wall._ Case 1 (introduce) spent its whole 7,200 s in the HEAD side's evidence pass and saved no artifact; cases 2 and 3 were stopped on the same path. The evidence portions took 13, 31 and 51 minutes each. A push asks evidence of the WHOLE population with change context, 22,125 requests, where the census asks about 2,400 without it.
- _Diagnosed by probe, not by timestamp._ On one PMPro graph from the production request builder, context costs ~20% more engine time per request -- not the wall. The wall is OUTPUT: each request itemised every method its scope reached, up to 1,845 rows; 200 requests into the bundled Braintree SDK returned 52,831 rows, 9.4 MB, against 211 without context. `affected_context` only ever attaches a location in a CHANGED file and discards every other row after checking it, so nearly all of it was shipped to be thrown away. Fixed in `3c195c9`: the engine itemises the changed files and summarises the rest into one row carrying what the check reads. On PMPro the same 2,440 relationships and zero gaps are decided from both forms, 52,831 rows down to 3,241 with the most-referenced file as the change. Probe: `benchmarks/push/context_cost_probe.py`. The replay is re-running on frozen `3c195c9`.
- _Corrected the same day: that was real but it was not the wall._ The re-run on `3c195c9` showed the same 13 → 31 → 51-minute portions. Sampled in flight, no engine was running for minutes at a time while the `pre-push` Python process sat at 83% CPU. The cause was the reuse cache: every publication listed and stat'ed the WHOLE cache to total its size, and a push publishes one answer per question, so the pass was quadratic. Measured on the live cache, one scan took 0.68 s at 12,238 entries, ~70 minutes of scanning by then and ~2.3 hours to finish the pass. The growing portion times were the quadratic, not the engine. Fixed in `7cf256b`: the scan is remembered, keyed by the cache root's modification stamp, so another writer still forces a rescan and the size limit stays exact. The probe did not catch it because it called the engine directly, without the cache -- the instrument lesson again: a probe that skips a layer cannot clear that layer. The output fix stays: it is still 16× less data per push. The replay is re-running on frozen `7cf256b`.
- _The loop, named and broken, 2026-09-23._ Walls kept appearing one per two-hour run -- the cache rescan, then the harness losing a killed case's output (`982cb14`), then a linear-list relationship check that also overran the deadline by >60 s (`e7e2941`) -- because every one was Python bookkeeping between engine calls, every one grew with the repository, and the only full-scale instrument was a Joern replay. Two instruments replace that: an in-process sampling profiler (`OUSAST_SAMPLE_PROFILE`, on for every replay case) and `benchmarks/push/python_overhead.py`, which runs the real `replay()` on the real PMPro pins with an instant stub engine answering real recorded rows. Its first pass found the remaining two walls at once: discovery re-parsed the ruleset per region (89,127 TOML parses, 339 s → 7 s, output digest identical) and change attribution decoded every line anchor per row (78% of a stubbed push; now indexed). The whole stubbed PMPro push now costs 58.5 s of Python; per-entry cache fsync is gone too (`b4b9308`, `6b85a60`).
- _The budget, stated correctly._ The push's head scan can select 2,500 questions (500 regions × 5 families) where change context leaves them incomplete, because tier-zero pruning never prunes an incomplete question -- ~6 hours of engine at ~9 s per PHP question. That does NOT make a 7,200 s case infeasible: contributor-scan 7.7–7.9 portion the arbitration phase in ranked order and hold time back for arbitration, so a case answers what fits and reports the rest as `query_budget_reserved`. What defeated every earlier case was the phases BEFORE portioning -- discovery, the cache, change attribution -- each spending the deadline before arbitration began. With those fixed (preparation 290 s → 19.5 s in the real run), the three cases re-run at 7,200 s on frozen `6b85a60`; completion is expected to be partial and stated, and the declared target's file ranks 19th of 86.
- [x] 13.4 Let a question inside one partition complete, whatever else the repository holds
- **Diagnosed 2026-09-21, and PHP was never the variable.** A repository holding TWO languages can never complete a question, so it can never admit a finding. `partitions.py` raises `cross_partition_semantics_unresolved` whenever more than one language partition exists; that reason is a member of `GRAPH_INTEGRITY_GAPS`; and a graph-integrity gap demotes every completed outcome in the scan.
- The evidence is one column wide: NodeGoat is JavaScript alone and completes 299 of 299, VAmPI is Python alone and completes 35 of 35, PMPro is PHP with 36 regions of bundled JavaScript and completes 0 of 127 and 0 of 520 while establishing 19 and 43 findings. NodeGoat's own `vendor_semantics_unresolved` is a DECLARED exclusion rather than an integrity gap, which is why it does not demote.
- **This is larger than the PHP runtime limit and it lands on the declared v0.1 target.** A WordPress plugin is PHP plus its admin JavaScript, so every one of them is multi-language, so on the target this tool was built for it can detect defects and never admit one. Neither subject that currently passes could reveal it.
- Recording the boundary is right: a flow from PHP into JavaScript is semantics the partitioned graph cannot follow. Demoting every question is not, including one wholly inside a single PHP file. Task 12.3 already fixed this bug class one level down, making an empty-scope gap question-owned "so it cannot block another question's comparison"; the same principle applies to a partition boundary.
- Do not reclassify the reason as a declared exclusion. That would claim cross-language flows are out of scope when they are unresolved, which is the opposite of what this project's degradation vocabulary exists to say.
- Done when a question whose scope lies inside one partition completes, a question whose scope could cross a partition boundary stays unresolved with the boundary named, and no question completes by widening its scope. Evidence: `benchmarks/measurements/2026-09-21-two-languages-defeat-completion.json`.
- _Both halves implemented 2026-09-21._ A language boundary is now a declared limit: it can hide a finding and never invent one, so it stays published and stops demoting. And an integrity gap raised by a partition census now carries the partition it came from and reaches only that partition's questions, because a missing JavaScript file cannot make a PHP answer wrong. Two controls exercise the real boundaries rather than injecting a reason, and both were confirmed to FAIL with the classification restored. The rule behind both is now written in the design as an invariant, with all five instances listed, because this is the fifth time it has been found and fixed one instance at a time.
- _Verified on the subject 2026-09-21:_ PMPro completes **120 of 127** questions where it completed none, with its 19 findings unchanged. The seven still unresolved are the JavaScript partition's own questions, which is correct: the four missing files are its files, so its answers are the ones that could be wrong. 1,419 tests pass, eighteen skipped; Ruff/format/mypy pass.
- _Three causes, not one, and the third was hidden by the record rather than by the code._ The same census gap is raised in TWO passes, arbitration and evidence, and carrying the partition in one but not the other left an unscoped copy that kept demoting everything, so the fix looked applied and was not. It was found by making the census print each outcome's reason instead of only its status, after which three separate causes stopped looking identical.
- _Admission is scoped too._`_blocking_degradations` already exempted declared exclusions; it now also ignores a gap another language partition raised, so a JavaScript build artefact cannot block a PHP finding. A gap naming no partition still reaches everything, and so does every gap when the claim's own language is unknown, because unknown scope is not evidence of independence. Evidence: `benchmarks/measurements/2026-09-21-partition-gap-blast-radius.json`.
- _Checked 2026-10-02 per the operational-readiness audit:_ implemented and verified on PMPro (120 of 127, above). 13.3 stays open on its own blocker; this box was left unchecked only because it waits on it.
- _Depends: 13.3_
- _Requirements: 2.1, 2.2, 6.3, 8.1_
- [ ] 15. Prepare the independent population and declare the shippable scope (M4 preparation)
- Runs in parallel with M2 and M3. M4 already permits population preparation before the frozen candidate exists, and it is the longest lead-time item on the path: nothing can be qualified without it however good the analyser gets.
- [ ] 15.1 Reserve and review an untouched evaluation population
- Freeze reviewed vulnerable, fixed and benign changes from projects not used to develop or tune this analyser. NodeGoat, VAmPI, PMPro, WP Statistics, MW WP Form and libpng are all development subjects and none of them may qualify anything.
- Reserve the cases untouched: do not examine their scan results before the population is frozen, and record the reservation so a later run cannot be mistaken for a blind one.
- **The population MUST hold a multi-language repository.** Task 13.4 found that a second language in a repository cost every question its completion, and so cost every finding its admission — on the declared v0.1 target, which is PHP plus admin JavaScript. Neither subject that passes today could reveal it, because both hold exactly one language. A population of single-language repositories would qualify a capability that fails on the product's own target.
- Done when a frozen population exists with its provenance and review recorded, at least one member holds more than one language, and nothing in it has been scanned.
- _Reserved 2026-09-24, not yet frozen._ Eleven cases selected by the maintainer from fifteen researched by advisory and fix diff alone (no clone, no scan), none among the 262 repositories this project's benchmarks reference, plus the earlier Ghost reservation: 4 PHP with JavaScript (Ultimate Member and LearnPress WordPress plugins, phpMyFAQ, YesWiki; all SQL injection), 4 Node/TypeScript (Signal K SSRF, Saltcorn SQLi, mongo-express code injection, Mockoon permissive CORS), 3 Python (alerta SQLi, wger open redirect, OpenCVE SSRF). Record: `benchmarks/independent/population-v1.toml`; research and rejected alternates: `research-2026-09-24.md`. A guard test fails if any reserved repository is referenced elsewhere in the tree. Remaining to FREEZE: per case, verify the pins and the sink by reading source, and choose a benign change from the same history without scanning; the OpenCVE licence must be checked before any result is published.
- _Frozen 2026-09-24._`benchmarks/independent/freeze.py` verified every case with git and file reads only -- no analyzer, query or tooling: both pins present and ordered, the sink text present at the vulnerable pin (LearnPress's fix is source-side and recorded as `fix_site`; Mockoon's sink is recorded as `sink_evidence`), the fix touching the sink or its declared fix site, and SHA-256 of the sink file at both pins. Benign controls were proposed mechanically and REVIEWED by diff: the keyword filter alone let through an access-restriction, a filter check, API-token scopes, a login change and a TLS-verification option, all rejected; Ultimate Member's and alerta's controls were replaced by hand (a coding-standards reformat; a dependency-plumbing change with both sides still vulnerable). Record: `freeze-v1.json`. The guard test now requires a benign pair for every case of a frozen population. Nothing in it has been scanned; the first scan is the evaluation.
- _Boundary: Evaluation harness, corpus provenance_
- _Requirements: 8.1, 8.2_
- [ ] 15.2 Declare the v1 advisory scope as the families whose claims can be admitted
- **Admission requires the entailed rung.** Injection, untrusted destination and configuration entail with witnesses on the measured subjects. Access control does not: on an application that splits its route from its data access the obligation is carried, and a carried obligation corroborates by design, so shipping the family would ship silence. Output encoding, path, prototype and deserialization have not earned a place.
- Record the scope, the reason each excluded family is excluded, and what a contributor is therefore NOT told. M4 does not require every language or family to become eligible, and a published limitation is the honest form of a narrow release.
- Done when one record states the declared scope and every exclusion's reason, and the packaged advisory output names the limits it ships with.
- [ ] 16. Make a harness report the cause of its own failure (robustness)
- **Named by a week in which three separate causes looked identical.** The census reported 19 findings and 0 completed questions side by side for days without complaint; a stale image measured the engine a bug had been diagnosed against; and a partition fix looked applied while an unscoped copy of the same gap kept demoting everything. Each was found by reading code, and each would have been visible in a record that carried its own causes.
- Every harness result should carry the counts that disambiguate its failure modes, the degradations whole rather than by reason, and the engine and ruleset identities actually used. Two of these were added reactively during that week and should be the standard shape rather than a reaction.
- Add a self-consistency assertion to each harness. Findings above zero with completion at zero is a contradiction, and an instrument that says so loudly is the difference between an hour and a week.
- Done when each harness record carries its provenance and per-cause counts, and a deliberately contradictory result is reported as a contradiction rather than as a number.
- _Boundary: Evaluation harness_
- _Requirements: 8.1, 8.2_
- [x] 17. Operational readiness (2026-10-02)
- **Named by the 2026-10-02 operational-readiness audit**, maintainer-approved as the first increment. The maintainer's definition: a user installs the hook on ANY repository and gets honest, advisory-only results; nothing ever blocks a push. The audit found the plumbing sound (snapshots, resolution, deadline accounting, advisory exit 0, the JSON artifact) and five things in the way: the hook can block a push, quick rules are not part of pre-push, the engine cannot finish a 26-file repository inside 30 s, the terminal hides every skip reason, and there is no supported install path without the source checkout. Before numbers for every item are the audit's live runs on records (Python), serve-static (JavaScript) and unrolled/secure (Go), each with one planted commit.
- This group is plumbing and honesty. It enables no capability, edits no rule for a planted sample, and leaves graph reuse (the engine speed work decided in 11.6) as a later task.
- Record: `benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json` (see 17.7).
- _Done 2026-10-02:_ all seven subtasks done. The engine still does not fit the 30 s hook deadline (graph reuse, the 11.6 direction, stays a later task), and plane verdicts after the push were not in this increment.
- _Requirements: 9.1–9.6_
- [x] 17.1 Never block a push the user did not choose to block
- The installed hook exits 0 with one line when `ousast` is missing (the check did not run, and how to install). In hook (stdin) mode, an argument or usage error (an invalid `OUSAST_PUSH_DEADLINE`, a cache directory inside the repository) is advisory: one line and exit 0. `--mode blocking` stays the explicit opt-in that can reject a push, and keeps its usage errors. The prior-hook input-retention failure keeps exit 1, because it is the chained hook's verdict that cannot be obtained, not ours.
- Files: `ops/pre-push`, `src/openultrasast/cli.py`. Tests: `tests/test_push_hook.py` (a push with `ousast` absent from PATH succeeds and prints the line; a bad deadline in hook mode exits 0; blocking mode keeps exit 2).
- Evidence: the audit's `hook.sh` push 1 (rejected before) and `fail.sh` (rc=2 before) rerun with exit 0.
- _Done 2026-10-02:_ the hook prints one line and exits 0 (chaining a prior hook if present) when `ousast` (or `OUSAST_COMMAND`) is missing; in hook mode without `--mode blocking`, argparse/usage errors and unexpected exceptions print one line and exit 0. Tests: `test_missing_ousast_never_blocks_the_push`, `test_hook_mode_usage_errors_are_advisory_unless_blocking` (blocking keeps exit 2), and `test_cli_refuses_the_flag_outside_explicit_replay` updated (the experimental flag is still refused in hook mode, now advisory). Gates: 1987 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.1, 5.3, 5.4_
- [x] 17.2 Name every skipped check in plain words
- One terminal line per distinct coverage reason: hex-encoded paths decoded, sha256-suffixed reasons grouped with a count, JSON unchanged. "language not covered: <lang> (<n> files)" for a language with no engine frontend and no quick rules, in both `pre-push` and `scan`. A first push of a new branch says there is no base and what was checked instead. `reports.py` gains lines for the eight scan degradations `cli.py` records and the report did not render (`learning_endpoint_unavailable`, `cpg_unavailable`, `facts_unavailable`, `policy_invalid`, `model_withheld_execution_budget`, `max_findings_exceeded`, `hunter_model_unavailable`, `sandbox_unavailable`).
- Evidence: the three repositories' terminal output names engine-missing, Go not covered and the new branch.
- _Done 2026-10-02:_ new `push/explain.py` turns every raw coverage reason into one plain `Skipped:` line per distinct kind (hex paths decoded, sha256 tokens dropped and counted, unknown reasons still printed); the existing one-line `Notice:` keeps its next action and gains an engine-missing variant. `language_not_covered:<lang>:<changed>:<total>` is recorded for a changed language with no engine frontend and no quick rules; `ousast scan` prints `notice: language not covered: go (N files)` and its report says so. `reports.py` renders the eight previously dropped degradations and prints any reason it has no sentence for. The hook passes `OUSAST_COMPARISON_BASE`. The new-branch line names the base it lacks; what was checked instead lands with the quick tier (17.3). Tests: `tests/test_push_explain.py` (6), two in `tests/test_reports.py`; module-audit manifest regenerated for the new module. Gates: 1995 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.2, 5.2, 8.2_
- [x] 17.3 Run the quick rules as the fast first tier
- Inside the deadline and before the engine, the shipped quick rules run over the head side of the changed files and keep the findings on changed lines or in changed functions. They print as advisory pattern matches, labeled apart from admitted alerts, recorded in the artifact's `quick_tier`, and never block.
- Evidence: serve-static's planted `eval` in the new `debug.js` is shown; whether records' planted `%`-formatted SQL is shown is recorded as is, with no rule edits for it.
- _Done 2026-10-02:_ new `push/quick.py` runs the shipped quick rules (no shadow rules, no model) over the changed head files inside `_prepare` under the shared deadline, before the engine, keeping matches on a changed line or inside a changed function (entry-point ranges); it records `files`/`bytes_read` as proof of input. The terminal shows them under "Quick rules: N pattern match(es) on changed code ... Advisory: not verified by the engine."; the artifact keeps `quick_tier`; admission and push disposition are untouched. A new branch runs the quick tier over its last commit's changes and says so. `_analyze` gained `engine="off"`. On the audit's serve-static planted commit the `eval` in the new `debug.js` is now shown (js-eval, js-xss-response at debug.js:2, ~1.1 s on the host); the planted `%`-formatted SQL in records is NOT matched by any quick rule (coverage gap, no rule edited). Tests: `tests/test_push_quick.py` (3). Gates: 1998 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.3_
- [x] 17.4 Classify a file new in head as new, not unknown
- `ChangeContext` records the paths Git reports as added. An operation in an added file, when no add/delete pairing is unresolved and its head evidence is complete, is `new` with reason `file_added_in_head`, and admission accepts that reason as change evidence. An engine finding with established novelty whose only vetoes are capability vetoes renders as an advisory engine finding, not an alert. Every other unknown comparison keeps its veto.
- Evidence: unit tests on the comparison and admission; a live engine run if the host allows one (free memory >= 3 GB).
- _Done 2026-10-02:_`ChangeContext.added_paths` records Git status-A paths (rename targets excluded). In `_compare_evidence`, after every existing context/head/deadline check, an operation in an added file is `new`/`file_added_in_head` when the claim is entailed, undischarged and has change evidence, and no base answer holds the same operation (else `operation_moved_within_change`); an unpaired add/delete still blocks as `change_context_unresolved`. Admission accepts `(new, file_added_in_head)`; with the empty registry such a candidate's only veto is `capability_unavailable`, and the terminal now shows it under "Engine: N finding(s) this push introduced. Advisory ... not an alert." Any comparison, evidence or location veto keeps a candidate out of that line. Tests: `tests/test_push_new_file.py` (9). Live engine evidence depends on host memory (17.7). Gates: 2007 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.4, 3.4_
- [x] 17.5 Say when the engine cannot finish, and offer it in the background
- One line when the engine did not finish (deadline, engine not installed). `--engine background` starts the engine detached after the quick tier, writes its result under the artifact directory and, on the next run, prints "engine result for <head> from the previous push". One background engine at a time; `--engine off` skips it with a stated line. No speed work.
- Evidence: a hook push with `OUSAST_PUSH_ENGINE=background` followed by a second push showing the previous result.
- _Done 2026-10-02:_`--engine inline|background|off` (hook: `OUSAST_PUSH_ENGINE`) and `--background-deadline` (default 900 s). `off` and `background` run the quick tier and record `engine_off`/`engine_background` with a plain skip line; the deadline line names the deadline and the background option. New `push/background.py` starts one detached explicit replay (`start_new_session`, stdin closed, output to a log) per artifact directory and repository (lock file with a live-pid check: a second start reports busy), writes its artifact under `<artifact dir>/engine/<repo key>/<head>.json`, and the next run prints "Engine result for <head> from the previous push: ..." once, then marks it shown. Not usable through the Docker wrapper (the container exits with the hook); documented in 17.6. Tests: `tests/test_push_background.py` (4, one launching a real detached run). Gates: 2011 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.5, 4.1, 4.2_
- [x] 17.6 Install, Docker and documentation for a first-time user
- `ousast pre-push install|uninstall [--force] [--docker] [REPO]` ships the hook as package data, honours `core.hooksPath`, refuses an existing hook unless `--force` (which chains it as `pre-push.before-ousast`), and works from a pip install. `ops/ousast-docker` is a hook-callable script (not a shell function) that passes stdin and the artifact path through. The `openultrasast:dev` image is rebuilt only if `df -h /` shows at least 4 GB free; otherwise the rebuild command is documented and the record says it was not rebuilt. `docs/scanning.md` gains a first-run guide, a glossary of skip reasons, a GitHub Actions recipe and full `--help` text; README tiers 1 and 2 updated, pulling `ghcr.io/norandom/openultrasast:2.0.1` where the wrapper allows.
- Evidence: install from the package into a scratch clone, push through it, uninstall restores the prior hook.
- _Done 2026-10-02:_`ousast pre-push install|uninstall [REPO] [--force] [--docker]` (`push/install.py`) installs the hook shipped as package data (`openultrasast/push/hook/pre-push`, byte-identical to `ops/pre-push`, verified in the built wheel), honours `core.hooksPath` without editing it, refuses an existing hook unless `--force` (ours is replaced; a foreign hook moves to `pre-push.before-ousast` and is chained), and uninstall removes only a hook it wrote and restores the chained one. `ops/ousast-docker` (also package data; `--docker` installs it beside the hook, which then uses it) is a script that mounts the repository read-only and the artifact directory at their own absolute paths, passes stdin with `-i`, runs as the user with no network and `--memory 3g`, defaults to `ghcr.io/norandom/openultrasast:2.0.1`, never pulls inside a hook, and exits 0 unless `OUSAST_PUSH_MODE=blocking`. The hook passes `--engine`/`--comparison-base` only when set, so an older ousast or image still runs. Every pre-push flag has help text; `docs/scanning.md` has the first-run guide, settings table, new-branch and background notes, the artifact keys, a skip-reason glossary (a test keeps it in sync with `push/explain.py`), a GitHub Actions recipe and the full `--help` text; README tiers 1 and 2 and `ops/README.md` updated (pull and tag the GHCR image instead of building). **Image not rebuilt:**`df -h /` showed 1.1 GB free (< 4 GB); rebuild with `docker build -t openultrasast:dev .`. Tests: `tests/test_push_install.py` (11) and the glossary test. Gates: 2022 passed, 42 skipped; ruff, format, mypy clean; `mkdocs build --strict` exit 0.
- _Requirements: 9.6, 7.4_
- [x] 17.7 Prove it on the audit's three repositories
- Rerun records, serve-static and unrolled/secure (same planted commits) with the new build in quick mode and with the hook installed, plus one Docker engine run on serve-static if free memory >= 3 GB and free disk >= 1 GB. Prove each run read its input (files, bytes).
- Files: `benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json`. Evidence: before (the audit) and after: exit codes, wall time against the deadline, findings shown, every skip line printed. A planted sample still missed is a coverage gap, recorded, never a rule edit.
- _Done 2026-10-02:_`benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json` (before = the audit, after = this build installed from its wheel into a scratch venv). 10 hook pushes and 6 replays, all exit 0, 0.69-1.21 s against 30 s on the host; the quick tier read exactly the changed file and every head snapshot read the full tree (records 43,577 B, serve-static 61,525 B, secure 103,023 B). The planted JS `eval` in the new `debug.js` surfaces: quick tier on the host, and as two advisory engine findings classified `new`/`file_added_in_head` in one Docker engine run (190.9 s against a 300 s deadline; MemAvailable 3.28 GB, 1.16 GB disk). **The planted Python `%`-formatted SQL is still missed**: no quick rule matches it and the host has no engine. Recorded as a coverage gap, no rule edited. Go is named as not covered. The proof caught a wrong install hint (`pip install openultrasast`, not on PyPI), corrected in the hook in this commit.
- _Requirements: 9.1–9.6, 6.3_
## Implementation Notes
- Task5.2 measured PMPro timing out after both snapshots were materialized but before change context/graph analysis. Group6 must account for compatible snapshot/discovery reuse as well as graph reuse; identical-tip hits cannot substitute for the later representative changed-code latency gate. No new detector, scope heuristic or incremental CPG mutation is implied.
- Cancellation must cover semantic planning and reporting, not only engine subprocesses: `_collect` and `_scope_work` now consume the shared deadline, and replay instantiates the budget-aware backend once. Preserve the 400-question failure-census and real Node artifact-retention regressions.
- Group 9 (M1a) corrected the owner diagnosis: all four named vetoes fire on the NodeGoat case, and a fifth, `operation_correspondence_unresolved`, is the actual remaining blocker because a newly introduced operation sits on a changed line and has no base anchor. The unchanged revision resolves its base location on the same lines and is classified `unchanged`, which isolates correspondence as the mechanism. M1b repairs context producers, question-owned vendor relevance, operation correspondence, file-wide ambiguity scope and query-to-operation provenance; vetoes that did not fire stay unchanged.
- Group 6: complete discovery/graph/query reuse is explicit via `--cache-dir`; source/declaration edits invalidate affected units. Installed frontend/adaptation bytes now bind graph and admission provenance. A valid cached census cannot qualify a fresh answer that omitted its own census. Identical-revision timing is not the changed-code rollout gate.
"summary":"OpenCode-MCP proof-authoring arm for search-with-proof: abliterated/GLM proof model, DSPy-optimized from OpenObserve traces; trust-domain decision (A scoped-key-in-cluster vs B brain-on-VM) must be resolved first. Creds in .env by name."
}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
