x.method() with a variable receiver binds complete iff exactly one same-name
method declaration exists repository-wide; collisions stay a finite candidate
set (receiver_type_unknown) and module-level functions never bind. Authorized
by maintainer strategy-review 2026-10-31; gated by dev-pool FA non-regression.
Maintainer's finding of 2026-10-14 recorded in hunter-learning-loop (a dated section that overrides the shape
annotations as the model's input), its design (amendment pointing to the prover's trace) and the steering
overview's current state.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Opus agent delivery under spec access-control-prover, requirement 10. `semantic/cfg.py` builds a control-flow
graph per function from the syntax tree (if/else, switch arms, loops, try/catch/finally, return, throw, break,
continue, exit, assert) with dominators and a per-line control context, and leaves a function without a graph
rather than a wrong one. `obligations/cfg_dominance.py::CfgDominance` discharges a statement guard only when it
lies on every path to the operation and the operation is unreachable from its failing arm; a predicate whose
result nothing consumes is `value_only` and never discharges; router, decorator and earlier-hop witnesses behave
as before; without a graph the order rule applies and `cfg_unavailable` is recorded. The trace's guards carry
`relation` and `effect`, the trace carries `operation_context`; Java parameter lists carry names only; a
handler-named operation's arguments are the handler's parameters. The checker's default stays order-based.
Smoke on the 23 fix changes: 15 caught, 55 of 93 locations, precision 15 of 15, no failure, no change over
120 s (unchanged from delivery B). Gate: 3575 tests pass; the reserved-repository guard passes on the export.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Raised by the implementing agent: the design named only exits on the failing arm; if (can(x)) { op(); } is the
common shape and the operation is unreachable from the failing arm there too.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Codex delivery under spec access-control-prover, requirement 9. `semantic/obligations/trace.py` serialises each
(entry, operation) pair the prover examined, discharged or unguarded: the entry with its boundary, access and
parameters; each hop in order with its call line, the guards before the call and the values forwarded; the
operation with the provenance of each argument traced through the function's binds; the guards after it; the
verdict. One canonical text form and one JSON form, no language, framework, repository, size or count fields.
The prover result, the arm record and each unguarded finding carry the traces. On the ec2-plugin fix: 32
traces, 4 unguarded. Gate: 3534 tests pass; the reserved-repository guard passes on the tracked export.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Requirement 10, the structural control-flow graph per function and CfgDominance behind the existing protocol,
tasks 4.4 and 4.5. Textual order stays the checker's default; the prover selects control flow.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The maintainer's correction: counts over a function, language or framework carry nothing about safety; the
object a decision model may see is the path from an attacker-reachable entry to a protected operation, with the
checks on it in control-flow order and the provenance of the operation's arguments. Requirement 9, the trace
contract and tasks 4.2 and 4.3.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Codex delivery under spec access-control-prover plus one fix of mine. Operation rows may now name a handler by
name pattern and framework (Jenkins form validation and list filling, Tornado request methods, Remix actions);
the mapper recognises Symfony, Livewire and Laravel controllers, Remix exports and Tornado handlers;
`obligations/enforcement.py` turns Laravel route middleware, Symfony attributes and security.yaml rules and
Tornado's authenticated decorator into router and decorator witnesses. The path walk looks each node's callers
up once per change and does not walk a tree with no recognised entry: one seatunnel change fell from 670 s to
35 s. One name Codex could not source (Tornado get_current_user) is left out and said so in the facts header.
Smoke on the 23 fix changes: 15 pushes caught (65 percent, Wilson 45 to 81), 55 of 93 locations, precision
15 of 15 flagged, no instrument failure, no change over 120 s. Delivery A caught 7.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The previous commit was gated through a pipe whose exit status was tail's, not ruff's; CI caught the
unformatted file.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
CI failed test_unchanged_file_empty_read_refuses: the previous fix skipped failed records in the zero-byte rule.
Now the arm records a file's size only after reading it (a git-empty file is counted under empty_files and
not listed), a real zero-byte read keeps its zero and the summary refuses it on every record as the test says.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Codex delivery under spec access-control-prover. `semantic/obligations/paths.py` builds entry-to-function path
records from the cached caller index; `mapping.py` names Stapler web methods and Jetty handlers as entries;
`check_obligations` gains the opt-in premise `assume_entries_protected` with the label `unguarded_path`;
`hunt/prover.py` proves a change with no model call and states its coverage; `benchmarks/valid/prover_arm.py`
scores it with the valid-benchmark metrics. Two fixes after the first smoke run: a product file git holds empty
is read, not unread (four changes failed on `__init__.py` and empty assets), and absent-input reasons are
counted by name.
Smoke on the 23 fix changes of the training pool: 7 pushes caught (30 percent, Wilson 16 to 51), 28 of 93
locations, precision 7 of 8 flagged, no instrument failure; three changes over the 120 second bound. Twelve of
the 17 Jenkins fixes reach the fixed method but check no operation because the method is the operation;
the spec now carries handler-named operations for delivery B.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Codex stopped on a real conflict: with path records and no clause or sibling norm the checker returns no finding
by design. The prover's premise (a recognised entry is protected unless declared public) is now an opt-in keyword
with its own label and evidence, default off, existing behaviour and tests unchanged.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A deterministic check that a protected operation is reachable from an entry point without a check dominating
the path, per framework, built on the existing obligations checker, caller index and entry mapper; scored as a
rule arm on the access-control training pool. Follows the maintainer's finding that the hunter's features
describe shape and cannot decide unsafety.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The DSPy arm over the second pool failed with LMInvalidRequestError on every change with more than two rows, at no
cost: `source.facts_for` attached every entry point of the product tree to every row, 8190 records and 1.5 MB per
row on apache/seatunnel (the first pool's arm lost 11 changes the same way on trigger.dev and shopper), and the
rendered prompt was refused. The screen table of the second pool weighed 814 MB for the same reason.
A row now carries the entry points declared in its own file or in a file that calls the function, which is what the
input has described all along ("in-repository entry points that reach the file"). `source.entry_point_count`
follows that definition.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The second access-control screen spun for 2h20m on one ordinary mage-ai change: a Next.js build commit whose 3 MB
one-line bundles hold 12,355 functions each. `source.ranges_from_ir` ran two regex scans of the whole line once per
function, and the screen kept its 422 finished changes in memory only, so stopping it lost them.
- `source.ranges_from_ir`: a line over MINIFIED_LINE_CHARS (2000) is not checked per function; JavaScript names on it
stay anonymous, other languages keep the parser's name.
- `source.generated_reason`: why a changed file is build output (a directory the product walk leaves out, a minified
name, over MAX_FILE_BYTES, a line over the cap). `hunt.producers._candidates` leaves such files out and records the
reason on the context; the fix site of a vulnerability is always kept. The DSPy arm shares the enumeration.
- The screen writes rows.jsonl and log.jsonl per change (rows first, entry last) and resumes by skipping logged ids.
Tests: the minified line never reaches the per-function scan; generated_reason on the four shapes and two source files;
build output in an ordinary change yields no rows while a vulnerability path in a one-line file is kept.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
call_sites rescanned every line of every product file for declarations on each call; with one name per
candidate row that was the whole cost of the source producer. code_index prepares a file once (the names each
line declares), call_sites takes it, and callers_of keeps a cache of prepared files the producer context owns.
Same calls, tested against the unprepared scan. One Laravel change of 33 rows: 249 seconds before this pair of
commits, 20 now.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
facts_for walked the whole product tree and re-read every product file for each candidate row; on a Laravel
application one change of 33 rows took four minutes, 96 percent of it there. The producer context already
holds the product file list and the texts it read for entry points, so facts_for and callers_of now accept
them. Same facts, tested by a call whose tree walk raises and whose file on disk is emptied after reading.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Java parsed already, but the producer failed every Java row because no obligation facts existed for the
language and that failure took roles and def_use down with it. PHP had no grammar. Now: tree-sitter-php in the
semantic extra with PHP node kinds in the CST reader; obligation facts for Jenkins/Stapler, Spring Security and
Servlet code (java.toml) and for Laravel, WordPress and Symfony (php.toml), each header naming the training
clones and references the names were checked against; a language with an IR but no obligation facts keeps
roles and def_use and records obligations as unchecked (None), which the engineered features treat as missing,
never as zero. Proven on the committed pool: 6 of 6 Java rows and 33 of 33 PHP rows of one fix change each now
carry semantic features. Implemented by Codex; gated here with ruff, mypy and the full suite.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Batch 1 and the spot-checked batch 2 combined. The walk and freeze record are in the training work directory.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The combined access-control freeze stopped at its 31st repository: one ordinary commit of a PHP tree touched a
Python file that does not parse, functions() raised Rejected, and ordinary_history let it end the walk. The
file is now marked parse_failed on its row and such a commit is rejected as an ordinary push (reason
function_parse), which keeps the draw conservative without losing the repository.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The workflow runs pytest twice: in the dagger container (now one failure) and on the runner host, whose job
synced the dev group only, so dspy and tree-sitter were missing there (25 failures). The one failure shared by
both runs was the partial-clone extraction test: engine_trace.main sets GIT_NO_LAZY_FETCH in the process
environment, the variable outlived its tests, and git 2.44 and newer refuse the lazy fetch with "lazy fetching
disabled". The local git ignores the variable, so the leak was invisible here; the test module now restores
the variable after each test (leak checked with a session-end probe: present before, absent after).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
CI's first full pytest run since 2836b78 showed 65 failures in three classes. uv run re-synced the environment
to the lockfile's default extras before pytest, removing dspy again (UV_NO_SYNC=1 now). The learn slice, pair
corpus and hunt producer tests need tree-sitter (semantic extra installed). Thirty-two tests need bubblewrap
isolation, which CI's container cannot provide: they carry a sandbox marker and skip where the isolation
preflight fails, with the cause's type in the reason; proven locally by hiding bwrap (7 skipped, 23 passed).
The partial-clone extraction test now probes one lazy fetch through git itself first, so git's own stderr
reaches the next CI log instead of git_exit_128.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
CI's pytest stage had not run since 2836b78 (ruff and mypy failed first). Once it ran, the hunt tests reached
dspy at run time and CI never installs the optional extra. Installing it mirrors the local environment the
suite and mypy are gated in.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
66 errors had accumulated in src since the last green run (the hunt modules of groups 4-6, stats, candidates,
runner). Codex typed them without behaviour change; the 17 inline import-untyped ignores it added are replaced by
one override for dspy, pandas, statsmodels and scikit-learn, and the hunt package's lazy imports of benchmarks/
(never type-checked, not in files) are followed silently so src stays strict against their signatures.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
CI runs ruff format --check and ruff check over the whole repository; 29 files had drifted from the formatter and
four measurement scripts carried lint findings, so every push since 25ae4b4 failed before pytest ran.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Three builds: the first two read 150 rows as empty (a producer bug, fcb43fd), the third is clean. The diff
producer was blind to removed lines (537b61e). Screen: deletion_count 0.89 is the reversed-fix construction;
on rows whose file lost lines entry_point_count 0.83 and caller_count 0.73 are the class signal; semantic covers
262 rows (TypeScript only). Repository folds assigned, leak audit recorded, manifest ac-ablation-1 registered
for decision (five arms, about 35 dollars).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
DSPy baseline: 2/5 pushes, 4/48 locations, 14 of 90 ordinary pushes flagged, 25 suspicions (3 true), 11 invalid-request
failures in shopper and trigger.dev, $6.13. Pilot hunter: 2/5 pushes, 4/48 locations, 99 of 100 ordinary pushes
flagged, 594 findings (3 true). Different pushes caught; nothing in trigger.dev.
The runner booked usd 0.0 per change so --budget-usd never stopped a run; it now prices every LM call of the change
at the registered prices and writes usd into the record.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The 150 unread rows were not a checkout failure: the entry-point scan read every product file and one empty
file raised empty_source, which escaped the cached property and failed the source producer on every row of
the change (and re-read the whole tree per row). Empty product files are skipped; the row's own file still
has to be non-empty.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The training clones are sparse-checkout and a worktree inherits it; without this the hunter arm raises
unreadable_hotspot on every change of the access-control pool. Same helper and checkout proof as the hunt runner.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Two instrument findings from the first access-control feature table (952 rows, 74 positive rows):
1. A fix that adds a guard is, replayed base-to-head, a push that removes it. The diff producer only kept
hunks with added lines at head, so 38 of 74 positive rows had no hunk in their file at all and 69 of 74
were "not on a changed line", while 60% of ordinary rows were. The diff annotation separated the labels
by construction and was blind to exactly the access-control shape. The producer (version 2) now records
per file the head lines after which the push removed code; distance and on_changed_line count them, the
model's hunks input renders them as removed_after_line, and diff.deletion_count is an engineered feature.
2. 150 rows (16%, every positive of one repository) read empty files from the worktree although git holds
the blobs; the build ran while another process checked out from the same clone. _full_checkout now
proves the checkout: git status must be clean, a dirty tree is restored once, then InstrumentFailure
checkout_incomplete. The table script cross-checks every unread row against the blob size and refuses
to screen a partly unread tree, and records the diff-scoped recall ceiling (positive rows a label-blind
enumeration would reach).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Java 63 of 69 sites. verify: 148 records, 701 deepseek-flash calls, no provider errors. The 21
uncloned sources are in sources_access_control_dropped.json for a later clone pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
sources --family <family> keeps only advisories whose CWE the taxonomy maps to that family, writes
sources_<family>.json beside the training set, and excludes the pilot, every repository the
benchmarks already name, the strict used set and every earlier batch; --prefilter-only runs the
dump pass without GitHub. Implemented by Codex (gpt-6-astra), gated here: names normalize rejects
exclude nothing instead of aborting, and the strict used set is in the exclusions.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
label_rows honoured only a change's primary site, so 43 of the 48 accepted access-control
locations were unlabelled; every entry of targets is now a positive with its own label_ref and
span. The training clones were emptied with sparse-checkout to save disk and a new worktree
inherits it: every file read failed and the source and semantic producers reported an unread tree
on all 599 rows. The runner now disables sparse-checkout in the worktree it adds.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
mcp-documentation-server (8 ordinary commits) and line-desktop-mcp (13) are left out of this freeze;
their locations stay accepted for a later one.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
ordinary_history(limit=) walks only the newest commits; assemble.freeze(history_limit=) and
--history-limit pass it through and the record carries it. The held-out benchmark passes none.
Reason: trigger.dev's 7,109-commit walk runs for hours at the per-commit cost of the function
and generated checks; the training freeze now walks its newest 1,500.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
rounds.py runs a candidate over compile or held-out folds through runner.run, so a tool-using
program has its tools bound to each change's checkout, and turns paired records into the
FoldResults accept.decide reads (closes the gap noted in bacd7a1). report.py runs an accepted
program once over the held-out pilot: a ledger line reserved before the run refuses a second run
per program id, contaminated or unchecked training runs are refused, the instrument is proved per
change and more than 5% failures makes the run invalid with no numbers quoted, the pilot's own
scorer and metrics give the after numbers, the recorded hunter numbers give the before, baseline
repeats pair with the single run, the proof arm's workload per 100 pushes is counted, and the gate
distance is read off the confidence curve. 124 hunt and stats tests pass. Implemented by Codex
(gpt-6-astra), gated here.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
propose.py: one change per round (instruction, demonstrations or one mask), examples built from
compile folds only with a provenance record of the change ids used, a registry for GEPA, MIPROv2
and BootstrapFewShot (MIPROv2 without the data-aware proposer so labels are never serialised),
compile-fold scoring with Wilson and the repository-paired bootstrap, and a compile-only history
the next round may read. accept.py: the rule as independent clauses (both folds non-negative with
one strictly positive, per-family regression budget on held-out recall, tie tolerance, no
acceptance without an interval, instruction cap, one change only), the comparison recorded in the
candidate artifact's configurations, rejections logged without held-out numbers. 48 fixture tests.
Known gap for live rounds: the optimisers and propose._score call the program directly, so a
tool-using program must be scored through the runner with its tools bound per checkout (6.1).
Implemented by Codex (gpt-6-astra), gated here.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
One append-only JSONL per clone (named by the sha256 of its top-level path), one line per commit
with the row as the walk built it and HISTORY_CACHE_VERSION, flushed after every commit so a
killed run keeps its progress. A cached row is reused only when its parents match the current log;
other versions and torn lines are recomputed. With no cache directory nothing changes. The
7,400-commit trigger.dev walk that took over an hour per freeze becomes a one-time cost.
Implemented by Codex (gpt-6-astra), gated here.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
contamination.py reads the repositories named by the held-out pilot manifest and by every
benchmark report (accepted_repos.json and NOTES.md links under benchmarks/measurements), marks
each run directory clean or contaminated with the report digest and time, and refuses to count
an unchecked run toward a gain. ablation.analyse carries excluded_from_gain when its run is
contaminated. Implemented by Codex (gpt-6-astra), gated here; the benchmark import is lazy so
the package imports without the benchmarks tree.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
ablation.py registers the manifest (table digest, arms as masks, folds, metric, sample, budget,
rule, seed, program id) before any run and refuses a run or an analysis that does not match it.
Arms run over the same changes through runner.run, paired by change; effects on recall and on
suspicions per ordinary push are reported pooled and per family with the repository-clustered
bootstrap and exact McNemar, plus the recall-against-suspicions curve at every reported
confidence (never a tuned cut). The importance table and decisions read held-out effects only;
a neutral annotation survives only when a kept one needs it. 30 fixture tests with a scripted
model. Implemented by Codex (gpt-6-astra), gated here.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The held-out benchmark keeps the 40-commit floor and the union of twelve-commit stratified draws.
A training freeze passes floor=20 (never below the ratio) and the record carries ordinary_floor.
When the union of twelve-commit draws cannot reach the count (small cells stay unallocated on
every draw), one stratified draw of the full count is used; with count equal to the eligible
commits that is every one of them.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Sourced apart from the 12 pilot repositories: 212 candidates (pool-fresh, pool-retry, OSV for PyPI,
npm, Maven and Packagist), 49 dropped because the benchmarks already name their repository, 413
catalog locations, 99 passing all three verifier checks in 28 repositories. Short of the 40-positive
target. catalog.py now reads .java and .php. Freeze waits on the maintainer's spot-check
(CANDIDATES.md, SOURCING.md).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The full suite after the replay found regressions from 062a7b7 and 94e96c6:
- catalog_gen and candidates.labeled_spans still unpacked 3-tuples and expected None on a parser
rejection; source.ranges returns FunctionRange (4 fields, module entry first) and raises ValueError.
- ranges_from_ir passed the language to declarator_name, so the repo-facts declaration table named a
callback call (exec(..., function () {) as a function; the pre-P1 call had no language.
- the catalog_gen fail-loud test blocked the old module name.
- docs: a renumbered examples anchor, and two .kiro paths in the public threat model.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
DSPy 3.4.1 raises AdapterParseError for an unparseable ReAct step, which is not the ValueError its loop
catches, so one malformed step discarded the whole hunt (7 of 60 hunts in the repeats). The step now ends
the loop like an invalid tool choice. Records carry loop_parse_failed beside parse_failed.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Committed before the repeats finish. Unit is (change, repeat); paired bootstrap
over changes and exact McNemar from openultrasast.stats; failures stay in the
denominator. Dry run on the existing single runs reproduces 6/12 and 3/12.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
3 of 12 with the fixed answer check against the pilot hunter's 6 of 12, one
run each, not significant. The intended zero-cost replay was not a replay:
answers at temperature 0 differ between runs and the cache did not serve the
second run. Programs must be compared over repeated runs, paired by change.
About $1.10 for both runs.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The answer check counted only the rendered function text as shown, so 7 of
8 baseline suspicions citing lines the model had read with read_file were
marked unknown (baseline 1/12 against the pilot hunter's 6/12). Tools now
record the lines they return. A DSPy parse failure scores as zero findings
with a flag instead of a broken run. The per-run response cache now takes
effect through dspy.configure_cache.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
0 of 12 positives; 12.6 false alarms per 100 ordinary pushes (2.1 on changed
lines only). The misses come from what the rules and facts express (access
control lives in obligations, which do not convert; sources cover request
objects only), not from the engine. Comparison arm only: a Semgrep miss is
never evidence against a hunter finding.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A Semgrep, quick-rule or engine match may enter the feature table as
positive evidence; its absence is never a feature, filter, demotion or
gate on a hunter suspicion, and no acceptance rule counts it. The Semgrep
arm is a comparison only and may at most replace the regex quick tier.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
openultrasast.stats: Wilson intervals (statsmodels), exact McNemar, a
repository-clustered percentile bootstrap and its paired form with one
quantile convention, grouped folds (GroupKFold, leave-one-group-out), and
per-column separation (ROC AUC) for the leak audit and the signal screen.
Checked against the benchmark scorer's Wilson and the learn harness's
McNemar. hunt/features.py turns the feature table into the engineered
frame (distances, counts, flags) and keeps the map from every engineered
feature to the annotation the model is shown, so an ablation mask drops
both, and screen() says which annotation carries signal before a model
call is spent. The [hunt] extra now carries the libraries; the core
install is unchanged. The learn harness's own copies stay until that line
is revived.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
hunter-learning-loop groups 1-3 (first increment). The feature table is the
contract: one row per touched function with diff, source, semantic and
taxonomy columns, provenance per producer, labels by the benchmark's rule.
Producers read the repository at head only (a planted advisory id and
commit text never reach a feature). The hunt program is a DSPy ReAct loop
over a signature whose inputs are the unmasked columns in a fixed order;
answers outside the closed family list or citing unseen lines score as
unknown. The runner writes records the benchmark scorer reads, resumable
by change, program id and mask, and refuses the held-out pilot unless the
run is a report run. Research extra [hunt] = dspy, pandas; the core
install is unchanged. 43 tests on a scripted model, no network.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Engine LGPL-2.1, shelling out carries no obligation; the registry rules are
under the Semgrep Rules License since 2024-12-13 and cannot ship, only our
own converted rules can. Measured 4.7 s startup and 5 to 10 ms per file;
converted regex and taint rules fire for Python, PHP and JavaScript; C
propagation and Groovy do not; one sanitizer semantic we refuse on purpose
is accepted by Semgrep. Recommendation: a Semgrep arm on the valid pilot
first. Recorded in the hunter-learning-loop research note and breadth.toml.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
610 pair files, 570 labelled functions. lizard matches the parser engine on
C, Python and Java but loses JavaScript names and TypeScript end lines:
keep ours, take its cyclomatic complexity as a feature candidate.
tree-sitter-language-pack gives identical spans, is faster, replaces six
grammar wheels with one and adds Go, Ruby, PHP and Perl: adopt as grammar
packaging. Also found: .c files should try the c grammar before cpp.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Six groups: table and producers; the hunter as a program with the search
metric and the run artifact; runner integration and a baseline on the 12
pilot positives; training set, folds, paired ablation and the importance
table; proposals and the acceptance rule; the pilot gain report.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Feature engineering is the centre: a feature table with switchable columns
is the contract between producers (source.facts_for, semantic facts, entry
points, diff hunks; lizard and Semgrep only after their evaluations), the
ablation runner and the hunter program. DSPy runs the program (ReAct over a
HuntSignature; GEPA and MIPROv2 propose), our rule accepts on held-out
folds. The pilot stays held out; the hook, its client and the core install
are untouched. Requirements approved by the maintainer ('4 first').
Co-Authored-By: Claude Fable 5.1 <[email protected]>
src/openultrasast/source.py replaces the copies in semantic/functions.py,
plane/tasks/repo_facts.py, learn/features.py, learn/excerpt.py and
plane/tasks/verify.py. Two engines share one sentinel (<module>), one
DECLARATION/KEYWORDS/RESERVED set and one alias map: the parser engine
(ranges, enclosing, bounds, call_sites, callers_of, facts_for) for the
scorer, the overlay and the hunter's annotations; the line-pattern engine
(declared_functions, enclosing_at, function_span, function_bounds, the brace
matcher) that labels, features, roles, excerpts and plane facts were built
on. learn no longer imports plane.tasks. Frozen records that store the old
sentinel are mapped on read (module_function).
Gate: labels rebuilt before and after are identical, 7,921 rows, after
sentinel normalisation (benchmarks/measurements/2026-10-13-source-module-
label-diff/record.json). 490 targeted tests pass; 3,312 collect. Net: 657
lines deleted, 212 added. benchmarks/unseen keeps <global> for its own
frozen manifests.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts).
Deleted: the four hunter tools no caller enables (read_definition,
entry_points, flows, obligations) with their schemas and dispatch, the dead
mapping branch of find_refs; 20 functions whose only caller was one test
(mapping task records, compute_ranking_metrics, build_quick_hunter_prompt,
validate_evidence_transition, should_fuse, harness processors,
rank_targets_with_model, model/report slice rendering, taxonomy
hierarchical/related, config_value.verdict, taint.verdict, roles.load_roles,
features.language_of and build_for_scan, folds.leave_one_*,
features.recorded_case, search BatchExecutor and the three __main__ shims,
delivery.put_input); the receiver delivery mode of the plane. push_inputs.py
moved to benchmarks/push. memory.purge stays: the live-S3 test cleanup uses
it. Net: 2,730 lines deleted, 207 added. 473 targeted tests passed; 3,282
tests collect. breadth.toml records the removals.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts).
Deleted: src/openultrasast/regress/ and the deep-mode REGRESS branch (deep
now records a regress_retired degradation and equals standard; proof is the
search-with-proof arm's job), the sandbox docker runner and probe that only
regress used, --fail-on worth-fixing, RegressConfig, the two ledger helpers
only regress called; skills.py and index.py with the hunter pool's skill
routing; model/audit.py; cpg/queries/vocabulary.sc with
benchmarks/push/ontology_gap.py. Docs and the scan skill describe the
retirement. Net: 3,895 lines deleted, 86 added. Targeted tests: 180 passed,
28 skipped; 3,331 tests collect.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts).
Deleted: src/openultrasast/improve/ (1,175 lines), the ousast improve
subcommand, plane/tasks/loop.py and loop.yaml with the --loop generation,
the memory-snapshot step in memory.seed, pairs.build_pair_signals, and the
four improve tests plus the loop cases in other tests. Docs no longer
describe the lever. Net: 3,073 lines deleted, 83 added. Targeted tests:
154 passed; 3,436 tests collect.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
PATHFINDER-2026-10-13/: feature inventory, ten per-unit flowcharts with
file:line evidence, cross-feature and within-unit duplication reports, a
proposal that keeps every package on one of the four active lines, and
copy-ready plan prompts in order. .kiro/breadth.toml: every spec and
capability with a status and the deletion that closes it. No code changed.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
fossil-mcp dead-code over src found 72 candidates; 69 are referenced by tests,
__all__ or other modules, 3 are not and are removed (cli._first_source,
pairs._scorable, provider.openrouter._extract_message_content). Eight spec
directories that steering already called retired or superseded leave the tree
(history keeps them, commits named in overview.md): harnessx-removal,
harnessx-self-improving-rulesets, openrouter-sast-harness, learning-harness,
constrained-detector, three-stage-scan, real-world-vfc-slice,
unseen-repo-evaluation. overview.md gets a current-state section: the valid
benchmark numbers, what they say, and the four active lines in order.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The hunter gets an annotation model (named, versioned facts about the push)
and statistical ablation decides which annotations stay. Labels from fix
sites and ordinary pushes, training set apart from the held-out pilot,
compiled replayable program, train-and-holdout acceptance rule, two stages
reported separately, no thresholds set in this spec.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Sweep over the pilot's hunter records: most 'vulnerable' suspicions on ordinary
pushes sit far from the diff. Scope to the change, then confidence, then the
config_secrets family; 5/12 recall kept at 8.8 false alarms per 100 (in-sample).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Opt-in filter on the classifier verdict for hunter-sourced defects; the default
keeps everything. Each combined record notes the kept verdicts and the counts.
Live check on the pilot: (a)+(b) filtered to 'vulnerable' gives recall 5/12 and
54.4 false alarms per 100 ordinary pushes, matching the scratch numbers.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Arm (b) hunter over all 252 changes: recall 6/12 raw, 5/12 with the classifier
filter; false alarms 92.5 and 54.4 per 100 ordinary pushes. Arm (a) push hook at
163/252: recall 0/12, 0.7 false alarms per 100. Per-positive table, classifier
stats, the one hook false alarm, and the disk fix for the replay output.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Live run: every classification failed with ValueError because Caller.ask
reads client.usage as a dict (the learn CLI always hands it a MeteredClient)
while a bare DeepSeekChatClient keeps usage as a list of rows. build_caller
now wraps a client that carries a usage list in MeteredClient; scripted and
already-metered clients pass through unchanged.
Regression test runs classify_family through the real DeepSeekChatClient
wrapper with a stub transport (the module fixture that stubs classify is
restored for it). The error field stays the exception type only, since a
message can carry a model key.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The tool hunter's family tag is the model's free text (a live run returned
family:sql-parser and family:readonly-enforcement), so every arm (b) defect
scored family "unknown" and score.catches() could never credit the hunter
even when it landed on the labelled file.
hunter_arm now classifies each finding into the closed taxonomy vocabulary
with the decision-engine classifier (learn.program.classify with
families=FAMILY_IDS minus unknown): a numbered, clamped excerpt around the
finding sets valid_lines, the canonical ANSWER_FORMAT is reused, an off-list
family parses to "none" and lands as "unknown". The benchmark label is never
a fallback. The defect keeps the raw hunter family, title and rationale plus
the classifier's verdict, confidence, rationale, cited lines and parse
failures for audit; the record counts classifier calls and families.
--no-classify keeps the old heuristic path.
14 new offline tests, 67 green with the benchmark tests.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
benchmarks/valid/hunter_arm.py runs the shipped deep-mode tool hunter
(tool_hunter.run_tool_hunter via resolve_hunter_client) over each frozen
change off-cluster: detached worktree at head with git hooks disabled,
hotspots scoped to the changed files, instrument rule on every hotspot
(bytes read printed, empty or unreadable fails the change loudly), model id
from the config resolver, family from the taxonomy, never from the label.
Records land in the exact shape score.py reads (push.admission.defects with
locations), resumable per change.
`combine` joins arm (a) replay records with arm (b) hunter records per change
id, dedupes defects by (path, line, family), keeps (a)'s instrument, records
which arms were present, and writes a file score.py loads directly.
9 offline tests (stub hunter, real scorer), 53 green with the benchmark
tests. No network, no key handling, no live run.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The pre-push hook has no LLM detection tier today: push/assistance.py only
selects a witness for already-admitted defects and cannot add a finding. The
hunter lives in scan --mode deep. The maintainer's call is that the LLM tier is
required on pre-push, so R6 now measures two arms over the same changes:
(a) the hook as shipped (quick rules + engine), (b) the hook plus the hunter
arm run off-cluster. (b) is the number the tool is held to; the gap is product
work in pre-push-safety-net, not a benchmark artefact.
Adds group 6 (hunter arm runner + combine + offline test) to tasks, aligns the
brief and design, and labels the overnight replay as arm (a).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
First live run of the valid detection benchmark. Labels come from the fix
site (fix^1), never SZZ.
- 32 held-out Python/JS advisories sourced from the public pool-p1 set,
CWE + text from OSV. 133 candidate locations -> 125 after dropping
tests/minified.
- 125 DeepSeek verification calls: 31 locations passed all three checks
(vulnerable pre-fix, fix addresses it, reachable in-repo) across 12 repos.
- Maintainer spot-check accepted all 12.
- 20 ordinary negatives per positive change from the same repos, 240 total.
- Held-out guard: 0 dropped against the training-only used set.
- Frozen manifest at benchmarks/valid/pilot/ (252 changes), provenance and
NOTES under benchmarks/measurements/2026-10-12-valid-benchmark-pilot/.
Three fixes found by the live run:
- catalog.py: skip a site the fix renamed or removed instead of aborting.
- assemble.py: negative ratio counts positive changes, not accepted
functions (malla would have demanded 260 negatives). Test updated.
- held-out guard: run against training sources only; build_used() also
scans measurements/populations, which flagged the 12 repos via my own
SZZ-pool analysis outputs (circular).
Replay and metrics not yet run; see NOTES.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Builds the benchmark from .kiro/specs/valid-detection-benchmark/brief.md:
sound labels from the fix (no SZZ), app-shaped verified positives,
ordinary negatives from the same repos, whole tool on unseen repos,
standard metrics.
Spec: requirements.md, design.md, tasks.md. Reuses the existing sourcing,
fix-site labelling, negative drawing, replay, and Wilson scoring; drops
the SZZ layer.
New offline modules under benchmarks/valid/:
- catalog.py positives labelled at the fix site (never introducing/last_touch)
- verify.py one LLM call per case (vulnerable? fix addresses it? reachable
in-repo?) + human spot-check; this is the app-shaped filter.
Honors the instrument rule: bytes read per function, reject on
an empty read.
- assemble.py >=20x ordinary negatives + a frozen, deterministic manifest with
the held-out guard
- metrics.py precision (PPV), false alarms per 100 pushes, exploit-verified
rate, and the plain headline, layered on score.py
36 new offline tests, 495 regression tests green. No network, no harvest,
no model calls in tests. Group 5 (the live pilot) is not started.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Lead with the names a security engineer already knows instead of the
coined ones:
- catch rate -> recall (true-positive rate)
- false-alarm rate -> false-positive rate (still reported as false
alarms per 100 pushes)
- proved-catch / proof rate -> exploit-verified findings (a working
proof of concept)
- precision stays precision (positive predictive value)
The plain-English question for each metric stays as the explanation.
Updated the valid-benchmark brief and the roadmap. No numbers changed.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Ran the humanizer rules over 79 prose files in the active specs (brief,
design, requirements, research and narrative docs). Removed em dashes,
significance inflation, forced rule-of-three, bold-label list items and
filler. Skipped tasks.md checklists and the retired specs (harnessx-*,
openrouter-sast-harness, unseen-repo-evaluation).
Every number, path, command, hash and identifier was preserved: checked
each file against HEAD for drift in numbers, inline code and code fences,
zero drift across all 79 files.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The frozen SZZ-labeled pool is not a valid benchmark. SZZ puts the vulnerability in
the wrong place about half the time, and ~76% of entries are library-internal (not
reachable in-repo), so its catch/false-alarm numbers (record 07's "3/92") are noise.
Retired as a metric and a gate across the roadmap and briefs; no pool-scored number
is quoted anywhere.
Added .kiro/specs/valid-detection-benchmark/brief.md: positives labelled from the
fix (not SZZ), kept only when app-shaped and reachable in-repo, each verified;
ordinary negatives from the same repos; unseen repos; the whole tool measured; plain
metrics (catch rate, false-alarms per 100 pushes, precision, proved-catch).
Ran a plain-English pass (the humanizer rules) on the roadmap, README, and the two
active briefs via Codex, then reviewed the diff: only these files changed, every
number/path/hash/command is unchanged, and the retired-metric decision is intact.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Follow-up to a36f99e (the prior git-add aborted on the already-removed pathspec and
left this edit unstaged). Removes the change-detector image assertions and renames
the function to test_rustfs_engine_queue_policy; the least-privilege rustfs policy
assertions are what remains.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
They pinned exact Dockerfile text / an engine-stage SHA, so every legitimate image
edit broke them and the "fix" was to paste the new value back — a change-detector,
not a contract. Deleted:
- tests/test_unseen_image.py (test_image_stages) entirely.
- the Dockerfile-text assertions in test_engine_queue's image test.
Kept as a genuine contract: the rustfs least-privilege queue policy assertions
(renamed test_rustfs_engine_queue_policy). The partial-clone extraction test and
its new hermetic-git-env fixture stay — that one verifies real behaviour.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- test_engine_image_and_rustfs_policy: root Dockerfile was refactored to the
digest-pinned ousast-task-base (carries semantic+s3 deps) + a joern-build stage,
installing the package --no-deps. Updated the stale `COPY benchmarks` (now
--chown=1000) and `.[semantic,s3]` (now --no-deps + joern-cli copy + pinned base)
assertions to the new contract.
- test_image_stages: plane/Dockerfile.engine-task refactored the same way; the
split marker is now `FROM ${ENGINE_REF} AS replay`, base is ousast-task-base
(not ax-task-runner:v0.3.1), and the replay stage no longer installs git (base
provides it). Re-pinned the engine-stage SHA and updated the markers/assertions.
- test_partial_clone_lazy_fetch: passed alone but failed mid-suite — a leaked
GIT_DIR/GIT_OBJECT_DIRECTORY/alternates from an earlier test changed the
missing-blob count. Added an autouse fixture that neutralises inherited git
control vars so every test in the file is self-contained (reproduced + verified).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Roadmap "Where we are" table said Baseline (G0.2) = "Not yet, run on kube" -- stale.
Record 07 (G0 exit) was completed right after on the docker lane (e901640, 1300/1300,
catch 3.3% / FP 3.1%, regex/quick-rule floor, engine_covered=0). G0 is exited; the kube
`ax` lane was never needed for the baseline. Reframed the readiness plan: steps 1-4 done,
the `ax` lane moves to G1 (engine arm at scale, gated by spec tasks 4.4/5.4).
Also fixed the ax binary in the coordinator commands: use $HOME/go/bin/ax, NOT
$HOME/.local/ax-v0.3.1/ax (the latter hits the AX-lane dispatch blocker recorded in the
unseen spec tasks.md). README.md + benchmarks/unseen/README.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
test_no_pair_repeats_a_repository_the_benchmarks_already_name failed: advisory-fix
catalog repos also appeared elsewhere under benchmarks/. Root cause split two ways:
1. Genuine leak: 16 catalog repos are also subjects in benchmarks/search/private/
pilot.json (a real private pilot population). Removed those 16 pairs from the
three catalogs and their verification.json rows (totals recomputed, catalog ==
verification preserved).
2. Spurious: the overlap scan greped ~49k vendored files under
benchmarks/search/private/runs/*/checkout/node_modules/, so any catalog repo a
vendored dependency's README names collided. Added a narrow vendored-tree skip
(node_modules/vendor) to the scan rather than deleting labelled pairs over a
dependency mention.
Implemented from Codex's first pass (gpt-6.1-sol), then gated and corrected: Codex
removed 20 pairs; 4 of those (backstage/backstage, koajs/koa, beaugunderson/
ip-address, cakephp/cakephp) were driven only by vendored/non-population mentions
and are kept. Also excluded advisory-fixes-spread.json (catalog-derived report)
and research-2026-09-26.md (selection write-up) as non-population.
tests/test_advisory_fix_pairs.py + tests/test_learn_labels.py: 53 passed.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
`pre-push-for-developers.md` and `use-case-fix-and-reverify.md` (authored
2026-10-07) were in docs/ but absent from nav, so `mkdocs build --strict`
errored and test_docs_site::test_site_builds_strictly failed. File both under
"Using it", restoring the strict build.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Call-graph trace: no shipped product path or in-repo benchmark/test reached the
real single-query JoernBackend.query("taint"). The push/scan/replay path is
double-shielded -- batch fetch (run_batch) in Phase 2, and a _prefetched replay
CpgResult in Phase 3 whose run ignores the query -- so scan_region's cpg.run
("taint") never reaches JoernBackend.query. Benchmarks use the batch path; tests
use fake canned-row results. Only out-of-product ad-hoc harnesses hit it.
Regression onset 562b670 (2026-09-25).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
taint.request_params emits guards/fixedOrigin/originAnchors/quotedSanitizers/
fieldParameterSources, which taint.sc reads from per-request JSON in its BATCH
branch but does NOT declare on `@main def exec` and does NOT consume in its
single-request branch (it defaults them). The single-query path passed them as
`--param guards=...` (disposable) or named args (session), so joern errored
`Unknown arguments` and every cpg.run("taint", ...) failed -- a silent zero of
the shape AGENTS.md warns about. scan_region->taint_verdict->_usable_flows and
the pair-path verdict()/verdicts() were dead; the push/scan batch route was
unaffected (it carries the extras as JSON the script parses itself).
Fix: parse the script's `@main def exec(...)` signature once (_declared_exec_params
+ memoised _declared_params) and, in query(), filter params to the declared set
before building either route. Faithful, not lossy -- the single-request branch
could never use those 5 fields. No-op for config/dominance (their request_params
emit only declared params; verified). Only taint sheds the 5 extras.
Verified end-to-end on banks (patched backend in the replay image): single-query
path now runs; baseline 0 findings, annotated search 2 path/model_entailed
findings (prompt->write_text). Regression test asserts no undeclared param
reaches the command line and the query parses, with a fake runner (no Joern).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Same CPG, only the search changes. Baseline (shipped path spec) resolves 0
prompt->write_text flows on banks; injecting the two annotations an LLM steer
would supply (parameter_sources=True + write_text/read_text path sinks) makes the
engine resolve prompt -> path-join -> prompt_file.write_text(prompt.raw). First
measured instance of "#2 guided by #1" closing on an unseen target; fix needed
exactly the param-source + file-write-sink annotations.
Files the instrument bug the first MVP hit: single-query cpg.run("taint",
request_params(...)) is dead -- request_params emits 5 keys (guards/fixedOrigin/
originAnchors/quotedSanitizers/fieldParameterSources) absent from taint.sc's exec
signature; both single-query sub-routes reject them (silent None). Batch path
(run_batch->requestsFile JSON) unaffected; push/scan route uses batch.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
All 3 hits have a runtime oracle (search FAMILY_ORACLES: injection->sql/command, ssrf for
untrusted_destination -- corrects the earlier families.toml-based 'dcmcp unprovable'). But
search/verify.verify() needs an AUTHORED runnable demo + both head/base building OFFLINE + a
sandbox differential (3/3 affected vs 0/3 safe). The scan/push path reaches only suspicion/
static_corroboration; no code path from a scan suspicion to exploit_demonstrated, no Coordinator
caller from the push flow. So proof-arm lift AT PUSH TIME = 0/3 -- proving is a per-target,
offline-build, runtime, differential operation that can't run inside a pre-push hook (prior
per-pair demos were mostly inconclusive on this wiring).
Strategic consequence: proof/BLOCK is an ASYNC lane (CI/plane/triage with authored demos), not
pre-push. Honest pre-push product = advisory-with-conditions (bet #2); proof arm = async BLOCK
lane (bet #3). Local sandbox exists (userns).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Ran ousast scan --mode deep on the 5 app-shaped + sound pool targets (the fair change-scoped
eval set from the shape classification). Detection 3/5: graphiti (hunter, Cypher injection),
mcp-remote (hunter, url-scheme injection), dcmcp (ENGINE overlay -- 'outbound.request reached
from parameter' on readFileFromUrl; parameter-as-source fired because an entry point is present).
2/5 missed: malla (DOM/template XSS) and dms (Jinja SSTI), both in families families.toml
documents as limited.
This validates the strategic pivot: the near-zero catch was substantially a TARGET MISMATCH --
on app-shaped repos the engine itself detects (0 -> 3/5). But all hits stop at suspicion/
static_corroboration; release-grade BLOCK still needs proof-arm elevation + family coverage +
the DSPy hint loop. Cost ~$0.007/repo. So bet #1 (fix eval target) is necessary and material
but not sufficient.
Next: recompute a floor on an app-shaped sound subset / real app corpus (VAmPI, NodeGoat,
DjanGoat, vibe-py); measure proof-arm suspicion->proven lift.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Named the core problem with the maintainer: the tool spots dangerous ops but cannot prove
attacker-reachability -- the trust boundary lives outside the changed library code. Chosen
strategic bet (of three): FIX THE EVALUATION TARGET FIRST.
LLM shape classification of the 49 public pool vulns ($0.033): reachability_in_repo only
24% (12/49); 76% are pure libraries. A legitimate change-scoped catch needs sound label AND
reachability in-repo -> just 5/49 (~10%): graphiti, desktopcommandermcp, malla,
document-merge-service, mcp-remote. Shape predicted the hunter outcome (graphiti app-shaped ->
caught; bedrock library -> missed). So record-07's 3/92 is scored against a ~90%-illegitimate
denominator -- the near-zero catch is largely a target mismatch.
Next: deep scan on the 5 fair targets for a per-tier catch rate; then an app-shaped eval set
from known-vulnerable apps. Artifacts: 2026-10-12-pool-p1-shape-classification/ (+ earlier
label-audit, hunter-sound-targets). Memory: evaluation-target-mismatch.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
ousast scan --mode deep (DeepSeek deepseek-flash, model layer + tool_hunter, 60-call bound) on
bedrock + graphiti, the two sound pilot targets the static engine missed:
- graphiti: HIT -- falkordb_driver.py:207, tag family:cypher-injection; rationale traces
build_fulltext_query concatenating user input into a Cypher string at line 278 (inside the
labelled function, location-matches). The static engine produced zero findings here.
- bedrock: miss -- 12 suspicions on adjacent concerns (network-PUBLIC, download_file), never
named the install_packages -> pip command injection.
Cost is a non-issue: ~$0.007/repo, 64 model calls for a whole-repo deep scan. BUT everything
stopped at suspicion rung -- the proof arm did not elevate to proven; release-grade BLOCK needs
proven findings.
Direction (maintainer): prompt evolution is a core lever -- a DSPy-based loop (GEPA/MIPRO,
pluggable; evolving field) whose objective is to learn what hints/annotations steer the model
to the real sink, sourced from the framework knowledge already shipped (no new user
annotation), with LLM-proposed boundaries graduating into the facts.
Next: measure proof-arm lift on graphiti; stand up the DSPy hint loop + re-run bedrock; widen
the hunter to the 23 sound targets for a catch rate; then a change-scoped + proven run.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Widened the banks label-mislocation finding across all 49 public pool vulns via DeepSeek
(advisory + head code + pool label -> sound/unsound), $0.038 total. Result: 23/49 (47%)
sound; 53% unsound (25 pre-exist in base, 16 sink outside the changed lines) -- a change-scoped
hook correctly stays quiet there. Systematic by family: injection 10/13 sound, but path 1/7,
output_encoding 1/8, config_secrets 1/4.
Implications: record-07's floor (catch 3/92) is scored against a corpus ~half unfair to a delta
hook; the honest catchable denominator is the sound subset (caveat added to record 07 NOTES).
Fair pilot re-score: only bedrock + graphiti (injection) of the 5 are sound and the engine
missed both = the real detection gaps for the LLM-guided hunter.
Also: GEPA-style reflective-retry (future spec) + GLM cyber model (future spec) noted; DeepSeek
is the working default. Artifact: benchmarks/measurements/2026-10-12-pool-p1-label-audit/.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
DeepSeek deepseek-flash reachable: ~$0.0002/call, ~1.1s (sub-cent/push). The LLM gives the
correct verdict WHEN aimed at the real sink + source context: asked about the pool's
SZZ-labelled site (_load) it correctly says not-vulnerable; asked about the real sink
(from_prompt_path line 44, prompt.name request-derived) it correctly says vulnerable. So '2
guided by 1' works as a mechanism.
Two blockers, both concrete: (1) it needs the agentic search-with-proof hunter to supply
caller/source context -- single-shot on the ranker's region is insufficient; (2) the pool's
banks label is mislocated (head_lines 30/104/138-141, function _load; real sink is line 44),
so even a correct finding wouldn't score as a catch and the sink pre-exists in base -- a
change-scoped hook correctly stays quiet. Part of the pilot engine 0/5 is SZZ mislocation, not
a detection failure. GLM/abliteration fallback errors HTTP422 on the thinking param (client
fix needed; DeepSeek works).
Next: validate the 5 pilot labels (change-located & introduced?) before detection runs; then
size the agentic hunter on the sound targets; feed confirmed source shapes back as guarded facts.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Verified root cause: the pilot repos are libraries whose untrusted input is public-API
parameters, not HTTP requests. Python taint sources (ruleset/semantic/python.toml) are web/CLI
only; the parameter-as-source mode (_parameters_are_input) fires only for recognised entry
points, and the pilot vuln regions are entry:false -- so their params are deliberately not
sources. Widening that statically would reintroduce the WordPress/PMPro false-SQL-injection
flood the docstring documents. So the static engine cannot safely source library APIs on its
own -- which is exactly the seam for the maintainer's '2 guided by 1' directive: the LLM
adjudicates which library parameter is a trust boundary, the taint engine traces that
source->sink as proof (not LLM say-so), and repeated source shapes graduate to guarded static
facts. First increment (pending budget): LLM-guided-source mode on the 5 pilot regions,
measuring source naming, traced finding, catch lift vs regex 1/5, and false-alarm on ordinary
changes; staged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Inspecting head.findings (not ranker evidence) overturns the previous commit's read:
the dataflow/taint tier produced ZERO findings in the vuln families (path/injection/deser)
on all 5 repos -- only config_secrets (banks 1, socketio 2; launch-editor/graphiti/bedrock 0).
The tier1/score2.0 'sink' I cited was the RANKER's static candidate, not a confirmed taint
finding. taint_evidence_not_applicable = no source->sink flow was traced (banks: sink found,
no source). So ambiguous_line_correspondence is downstream and moot -- there are no
vuln-family findings to anchor, and the region-level witness fix (Option A) is refuted before
implementation. The LLM adjudication tier was disabled (model_calls=0).
Reprioritised: #1 LLM-tier pilot L1/L2 (now primary, staged, costs money), #2 taint
source/sink coverage + region ranking, #3 change-anchoring (only after tiers yield findings).
Instrument rule: caught the ranker-vs-finding misread before building the wrong fix.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Five public pool repos, vuln change each, clean host, 300s. The shipped engine added NO
catches over the regex floor. 3/5 repos finished their region queue (deferred=0) and still
emitted no finding, so change-scoping is not the path to catch. The engine ranked the vuln
region in all 5 and found the right sink on 2 (score 2.0), but withholds any finding it
cannot anchor 1:1 to changed lines -- all 5 hit ambiguous_line_correspondence ->
witness_unresolved -> coverage incomplete, suppressing even the scored sinks. Confirmed not a
shallow-clone artifact (correspondence is computed from both blobs).
Reprioritised engine levers: #1 change-anchoring robustness (region-level witness fallback),
#2 family ranking/taint coverage, #3 the LLM adjudication tier (the pilot is the case for it).
Release read: the pre-push net is NOT releasable on detection yet.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
On a clean host the shipped engine runs dataflow (query 0->35s, selected 0->356 regions).
The evidence/taint batch (~225s) is invariant to warm session and graph preload; importCpg
is cheap (~35s). The real barrier is taint.sc's ~160s per-batch flow work, so build+evidence
~255s of a 300s budget defers 1009/1365 regions. Lever = change-scope the family/file set the
taint batch walks, not session/preload. Corrects the prior ~200s graph-load wall diagnosis.
Next decisive experiment: engine-on unseen pilot vs record-07 regex floor (catch 3/92).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Verified on the records: the push replay arms hardcode model="disabled",
max_model_calls=0, cost_usd=0 (NO LLM) and engine_covered=0 (engine never ran,
~200s wall). So the measured catch~2.5% is the quick pattern-rule tier ALONE --
grep-with-structure. Pattern rules alone will never be enough.
Reprioritize G2 after record 07 to an efficiency barrier analysis over the two
tiers that carry detection and both sat out:
- engine (Joern dataflow): E1 split the ~150s evidence-fixed (JVM/graph-load/
census); E2 does a warm session + graph preload at build kill it (decisive);
E3 analysis per-pair; E4 build vs repo size.
- LLM adjudication: L1 model latency + calls per push; L2 recall lift over
regex+dataflow on the pool.
Deliverable: a barrier table (budget x tier combo -> fits? measured recall/FP)
for a realistic pre-push window (<=120s target, 300s hard) -> minimal releasable
config. Change-scoping branch is a late add-on (trims only the per-pair tail),
not the lever. Also fixed a stray </content> tag in the brief.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Records the full G2 engine-detection finding on main (code attempt parked on
branch engine-change-scoped-evidence-pass, unmerged). The engine arm has ~200s
intrinsic fixed overhead -- ~48s CPG build + ~150s first-query graph-load+census
(measured uncontended: build 48s, evidence 243.79s over 1015 pairs, query 0) --
so a 300s budget cannot reach the flow queries. Pair-count change-scoping trims
only the ~86s per-pair tail; the warm-session env times out at startup contended
and spares only subsequent batches. Real fix is clean-host engine work (reliable
session startup + graph preload, or census-cost reduction). Parked to finish G0.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Saved-artifact run: total=291s build=52s query=0s arbitrate=238.75s, and
query_seconds_by_kind[evidence]=238.54s. The evidence-tier pass is the whole
middle; affected_context + _scope_work are 0.21s. It tiered 1015 pairs at
~235ms/pair (contended) and exhausted the budget -> selected=0, deferred=1365
all deadline_exhausted. Secondary anomaly: ~50-100x the design's assumed
2.5ms/pair -- the evidence query is pathologically slow on a cold large CPG.
Increment 2's single target is the evidence pass (scan.py:338-359). Fork
recorded: (i) scope+bound the pass change-first, (ii) two-phase neighbourhood
pass, (iii) per-pair speedup. Must preserve reach-into-unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A first scoping attempt (restrict analyzed regions to changed files) was reverted:
it breaks a tested design invariant. The engine deliberately selects an unchanged
region's question when a changed region reaches it
(test_changed_dependency_reaches_unchanged_ranked_question;
test_change_does_not_replace_ranker_selection). _changed_files feeds _evidence_pass
only as contextFilter/contextPaths (itemise vs summarise), not to shrink the tier
population. Any scope must keep changed regions PLUS the regions they reach.
Also: increment 1 over-claimed. Increment 0 measured build=67s/query=0s but NOT
the split of the ~236s middle (evidence-tier pass vs affected_context/context
resolution vs _scope_work enumeration). Increment 2 must capture per_kind /
other_seconds / affected_context timing and cut the dominant phase while
preserving reach-into-unchanged -- instrument before fixing.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The 4095 candidates are not an in-scan artifact: push/runner.py:345 discovers
head_regions across the whole head tree, then runner.py:397 passes them wholesale
to scan_repository with change_context threaded in but unused for scoping. The
scan enumerates all regions x families (scan.py:167/364) and, with
order_by_evidence=True, evidence-tiers every pair before analysis -> the ~236s
from increment 0 -> deadline at query_seconds=0.
Fix locus (increment 2): scope the analyzed region set to context.changed_paths +
dataflow neighbourhood (regions.py:279 primitive exists; compare_targeted_base is
precedent). Keep the CPG whole-repo so taint can trace into the change from
unchanged code; only the question set is change-scoped.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Timed live bedrock replay: build_seconds=66.69, query_seconds=0.0, total 303s,
4095 candidate questions all analysis_not_started=deadline_exhausted, selected=[],
regions_scanned=0. ~67s builds the CPG; ~236s enumerates/ranks/prepares 4095
repo-wide candidates for a 3-file change, then the deadline hits before any query
runs. The CPG build is not the bottleneck; the over-broad candidate set is ->
change-scoping is the confirmed lever (and compounds with the correspondence fix,
since per-candidate prep includes correspondence work).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Track the engine-detection lever found while driving G0. Live diagnosis on the
frozen pool: the engine's "no finding" on unseen changes is usually
analysis-never-runs -- it ranks repo-wide candidates and defers them all
(deadline_exhausted) before analyzing the pushed change (bedrock injection,
303s, selected=[], ranked sites unrelated to the change). Lever: seed analysis
from the change's declarations + dataflow neighbourhood (rank.py/mapping.py) so
it runs within budget.
- .kiro/specs/engine-change-scoped-analysis/brief.md: problem (measured),
hypothesis, anchors, and checkable increments (0 confirm build-vs-analysis
cost split -> 1 locate selection/budget -> 2 seed from change -> 3 soundness
+ tests -> 4 live re-test -> 5 measure delta on the engine arm).
- roadmap.md G2: active workstreams (change-scoped analysis; the done-but-
necessary-not-sufficient line-correspondence fix), for progress tracking.
Honest scope: G2, gated on the pool metric; not part of the G0 quick-rule
record-07. Coverage also needs the capability registry (Task 8.3); arm-scale
measurement needs the kube lane.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The correspondence mapped an equal-region base<->head line pair only when the
line's content was GLOBALLY unique in the file, so common lines (`}`, blanks,
imports) tripped `ambiguous_line_correspondence` and the finding's anchor was
discarded -> the operation came back `unmapped` -> novelty `unknown` ->
`comparison_unknown` -> the engine finding was suppressed (a direct cause of
"near-blind to fixes").
Replace the global-uniqueness gate with run-aware positional anchoring: segment
the equal-region pairs into contiguous runs (strictly +1/+1, as _equal_line_pairs
emits), and map a pair when its content occupies exactly one old run and one new
run. Duplicates WITHIN a contiguous run are positionally forced and now map;
duplicates ACROSS runs split by an edit are aligned arbitrarily by git and stay
withheld + flagged (soundness preserved -- the moved-repeated-operation guard
still passes, so no false `new`/identity claims).
Tests: update test_change_context_unknown_base_binary_and_ambiguous_lines_are_
unresolved (its two in-run sinks now anchor); keep the moved-operation soundness
test; add intra-run-maps, cross-run-withheld, and distinctive-line-anchors cases.
163 push tests pass, ruff clean.
Honest scope (design.md amendment): this removes one real cause but is
necessary-not-sufficient -- live-testing a messy 14-file Java change showed
coverage stays `incomplete` due to other, dominant blockers (capability_unavailable
until Task 8.3; the engine produced no actionable defect; eligibility/projection
gaps). Engine coverage on unseen diffs needs those too, not just correspondence.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Fold the night's learnings into the roadmap, steering and the
unseen-repo-evaluation spec so the process is reproducible and the
project is "ready now":
- roadmap.md: a 2026-10-11 state table (pool p1 frozen; harvest hardened
+ bucket-first; replay-image provenance pinned; baseline-on-kube next)
and a 5-step readiness plan to record 07.
- storage-streaming.md: RustFS-over-local as the default (local disk is
the scarce resource); the implemented ousast-corpus layout; replay
outputs belong on RustFS; the AX store endpoint is operator-owned.
- design.md: an "Amendments (2026-10-11)" section (bucket-first
persistence, ~2.2% rate-bound yield, machine-label record 07, pinned
replay image provenance).
- tasks.md: a folded-in amendment block (pool p1 frozen + freeze recipe;
bucket-first; CI replay image; machine-label baseline; kube lane +
ax/store as operator-owned infra).
Facts stated are verified (measured yield, freeze digest, GHCR anon pull,
batch.py's endpoint check); infra capability/policy is left to the
operator, not assumed. The frozen pool artifacts themselves are not
committed here (a separate samples/licensing decision).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The real bottleneck was not enumeration or extraction: canonicalize_used
resolves the entire exclusion set (measured: 4,499 names) via serial
GitHub get_repo on every launch, before any harvesting -- ~30 min of
network-latency-bound work that also burns ~4,499 of GitHub's 5,000/hr
budget before real work starts (hence the low CPU and the "never
finishes"). The earlier resolution-cache only banked via
BankedGitHub.resolve, but canonicalize_used calls eligibility.resolve ->
get_repo directly, so it was never cached and re-ran every time.
Cache get_repo at the real choke point. CorpusBank gains cache_repo /
get_cached_repo storing one object per normalized name at
<prefix>/repos/<sha256(name)>.json -- NOT seed-scoped, since repo
metadata is seed-independent, so every seed/run shares it. 404s are
cached as None too (a found-but-None hit) so known-missing repos are not
re-fetched. BankedGitHub.get_repo now checks the local journal, then the
bucket, and only then GitHub (banking the result). canonicalize_used,
advisory_index and candidate resolution all funnel through get_repo, so
all of them are covered.
Effect: the first run pays the lookups once and banks them; every later
launch reads them from the bucket (no GitHub, no rate-limit spend), so
the rate budget goes to actual candidate harvesting and restarts are
cheap. Proven live against ousast-corpus (repos/ fills 1:1 with
resolutions). 206 unseen tests + new repo-cache test pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The decision bank made extraction durable, but every restart still
re-paid the ~26-minute advisory enumeration + per-advisory resolution,
since those lived only in the local journal. Mirror them into the bucket
so a restart reads them back and goes essentially straight to any
remaining extraction.
bank.py: bank_resolution/resolved at <prefix>/<seed>/resolved/<sha256(ghsa)>.json
(streamed one-by-one as each advisory is decided), cache_advisories/
cached_advisories at <prefix>/<seed>/advisories/list.json, and
bank_repository/repositories (caches the advisory_index repo-resolution
pass, preserving renames and 404s). All gated by RESOLUTION_CONTRACT="v1"
(extract.py) -- bumped only when enumeration/eligibility/source.candidate
output changes; recent fetch/parse/pool changes do not bump it.
draft.py run(): with --bank, advisories load from the cache (else
enumerate then cache); eligibility decisions load from the bucket
(contract-matched) to skip re-resolution; new resolutions stream to the
bucket before the local checkpoint write. Repository lookups reuse the
repositories/ cache. No --bank => local-journal behavior unchanged.
Net: advisories + resolutions + extraction all resume from ousast-corpus;
restarts no longer re-enumerate or re-resolve. Fresh and resumed runs
produce byte-identical manifests at 1 and 4 workers. Bank methods proven
live against ousast-corpus. 356 tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The draw lost all progress on any restart because the local journal
invalidates on a code change and is the only cross-run state. Make the
bucket the source of truth: stream every per-repo accept/reject to
ousast-corpus as it is decided, keyed by sha256(repository+"\n"+fix) so
writes are idempotent (content dedup at rest), and resume by listing the
bucket and skipping already-decided repos. No supervisor/watchdog needed
-- a crash or a code change just re-runs and resumes.
New benchmarks/unseen/bank.py: corpus_settings (CORPUS_S3_* from .env,
missing key named, value never printed), CorpusObjects (lazy boto3,
path-style + SigV4 verified live against ousast-corpus @ 10.0.0.54:9001),
CorpusBank.bank/decided with one-retry put/get, split_decided. Decisions
live at <prefix>/<seed>/decisions/<key>.json with repository, fix,
accept/reject, reason, entry, exclusions, counts, contract, seed, ts.
draft.py: --bank [prefix] opens the bank on the coordinator (never shipped
to fork workers); run() resumes from bank.decided() before extracting,
reusing accepted entries across contracts (deterministic from repo+commit)
and rejections only when contract == EXTRACTION_CONTRACT ("v1", in
extract.py); collect() streams each decision one-by-one. Seed-order
acceptance, fork/used-set dedup, and duplicate-advisory reuse preserved.
Without --bank, local-journal behavior is unchanged.
CorpusBank proven live end-to-end (bank accept+reject, idempotent re-bank,
decided/split roundtrip). 325 tests pass offline, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The harvester cloned --filter=blob:none --no-checkout, then blame/log/diff
triggered the promisor to fetch each needed blob ON DEMAND -- hundreds of
round-trips per repo, the dominant cost that made the draw unable to
converge. The blobless clone was a premature optimization that backfired.
Drop --filter=blob:none from the clone and the two commit fetches. A
full-objects --no-checkout clone brings the whole commit graph AND every
blob in one pack, so blame/log/diff read locally; --no-checkout still
skips the working tree so disk stays bounded (plus the existing 500 MiB
clone_size_cap and the disk guard). With no promisor remote, a genuinely
missing object now errors loudly instead of silently round-tripping --
the instrument guarantee that the bulk fetch was complete.
Extraction semantics are unchanged (same SZZ introducing-commit positive,
same stratified ordinary negatives, byte-identical entries); only the
object-transfer strategy changes. Test inverted to assert no blob:none
filter on the clone; README updated. 145 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Git subprocess calls were deadline-bounded (Git.command_timeout uses the
remaining per-repo budget), but a pure-Python parse between git calls
(repository_changes -> history/blame/diff on a giant file) had no
deadline: when it hung, no further git call fired the git deadline, so
the worker stalled indefinitely (observed >720s with zero git
processes). An external watchdog was papering over it at ~20 min/stall.
Add extract.process_deadline(seconds): a SIGALRM/setitimer wall-clock
alarm installed on the worker's main thread that raises
ExtractionTimeout("wallclock_timeout"), with no-op guards (None/<=0, off
the main thread, or no SIGALRM support) and strict timer-cancel +
handler-restore in finally. Wrap the whole extract_repository body in it
so the timeout propagates to the existing `except source.Rejected` and
journals as a clean rejection, not a worker failure.
Budget = --extract-deadline (git, 240s) + --parse-deadline (new, 120s)
= 360s total ceiling, so git-bound repos keep their full budget (yield
unchanged) while a hung parse is rejected well before any external
watchdog. Record parse_deadline_seconds in the report.
295 unseen tests pass (incl. timeout + handler-restore + no-op cases),
ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The draw hardcoded 300 repos / 3 slices. At the measured lazy-fetch rate
the full 300 projects ~2.5 days; a 100-repo single slice (slice 1, all
the G0 baseline needs) is ~20h. Add --pool-size (default 300) and
--slices (default 3), validated (>=1, slices<=pool_size, evenly
divisible). Generalize assign_slices' stratified allocation from a
literal 3 to `slices`, the run() break / insufficient check / report
(changes = pool_size*13) to pool_size, and record both in the journal
inputs. Remove bisect's three-slice assumption; freeze reads the size
from the draft. Defaults (300/3) reproduce prior output exactly.
With --pool-size 100 --slices 1 the pool is the first 100 accepted repos
in seed order, all in slice 1 (deterministic, byte-identical across
worker counts). 280 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Git.materialize enumerated every object reachable from HEAD and tried to
`git fetch --no-filter` them by SHA. Git rejects that for real repos
("did not send all necessary objects" / "bad revision"), exiting 1 ->
InstrumentFailure(git_exit_1) that failed the whole repository. Measured
in a live draw: 326/670 attempts, ~49% of all failures, which froze the
accepted pool at 44/300. Reproduced on pallets/flask (26221 reachable,
9287 missing, fetch exit 1).
Git cannot batch-fetch arbitrary tree/blob objects by SHA, so remove
materialize and restore the original lazy path: the blob:none clone keeps
the commit graph and the introducing/blame/ordinary-history reads fetch
their blobs on demand through Git's promisor. The per-fix tolerant fetch
loop, the process-pool parallelism, the 25-with-zero-entries guard, the
per-repo deadline and resume are unchanged. Extraction content is
byte-identical (only the prefetch is gone).
217 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two defects stalled the draw at 40/300:
1. The systemic guard used an absolute `instrument_failures > 200` cap.
At ~10% extraction yield, reaching 300 entries legitimately produces
far more than 200 per-candidate failures, and the counter is rebuilt
from the journal on resume -- so once past 200 the guard tripped
immediately on every restart (permanent stall; the supervisor's
no-progress guard then stopped). Drop the absolute cap; keep only
`>=25 failures with zero entries`, which still catches a genuinely
broken instrument (it yields no entries) without aborting a working
low-yield draw. Bounded anyway by the population / max-hours.
2. repository_changes fetched all fixes in one `git fetch origin *fixes`,
so a single unfetchable secondary fix ref failed the whole repo
(extract_git_exit_1 was 166 of ~200 extract_ failures -- the dominant
loss). Fetch the extra advisory fixes per-fix, best-effort, skipping a
git_exit_ failure; candidate.fix stays strict (already fetched in
extract_repository). Only resolved fixes are materialized and
fix-site-indexed.
218 unseen tests pass (incl. 1000 failures after a success not tripping
the guard, and fetch-error regressions), ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The ProcessPool spawned its workers but ordered_extractions blocked the
main thread on the OLDEST future, so while one slow repo finished the
other workers sat idle: live profiling showed 1 of 4 cores busy, ~67%
idle, ~1 attempt/min.
Separate the seed-order acceptance cursor (`pending`: row+decision+key,
no futures) from the worker set (`in_flight`: future->key). Keep up to
`workers` jobs running; journal each result as it COMPLETES via
wait(FIRST_COMPLETED) in any order and refill freed slots, never waiting
on the oldest. A candidate is accepted/rejected only when its result is
in the journal, advanced strictly in seed order -- so the pool runs
out of order while the accepted pool stays byte-identical to a 1-worker
run. Fork-network dedup and the systemic guard still apply at the cursor.
New real-process test proves later extractions are journaled before an
earlier slow one completes (completion order != seed order) while
accepted entries stay in seed order; the old loop fails it. 210 unseen
tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Local disk is the scarce resource (host near full; the draw stops at
<2G free). Record the direction to stream durable run artifacts to RustFS
instead of hoarding on local disk, split across purpose-specific buckets
(sast-memory durable, sast-pools durable archive, sast-runs transient
with aggressive lifecycle), each with a lifecycle tuned to its artifact
-- the lesson from the engine-queue versioning bloat. Extends the
existing S3Store pattern; new buckets are an operator admin action.
Planned as a follow-up spec, not retrofitted into an in-flight run.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Group 3 was unimplemented, so there was no way to turn the draft into a
frozen pool. Add:
- bisect_guard.py: stages the cache draft and bisects it against the
reserved-repository guard, run as a black box
(pytest -k referenced, stdout/stderr discarded, exit code only; an exit
other than 0/1 aborts as an instrument failure). Drops guard-tripping
candidates and reports the dropped COUNT only -- never a name or URL.
Guard runner is injectable for offline tests.
- freeze.py: writes canonical pool-p1.toml (+ private counterpart,
reusing draft's public/private split and private_digest) and
freeze-p1.json (used-set digest, per-source counts, guard result,
dropped count, label-check counts, freeze digest = sha256 of the
canonical public manifest). A matching retry is a no-op; any changed
frozen artifact requires a new pool id (never an overwrite).
- protocol-p1.md: the frozen scoring/statistics protocol from design
sections 3 and 4.
Offline only; the guard subprocess is the sole subprocess and is mocked
in tests. 42 new tests pass (incl the group-1.3 guard digest check on a
fixture pool); neighbouring unseen suites green; ruff clean. New files
only -- does not touch the journal-pinned draft/extract modules, so a
running draw is unaffected.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Extraction is CPU-bound Python (git-output parsing, HUNK regex,
ast.parse of hundreds of source files per repo). A ThreadPoolExecutor
cannot parallelize it: live profiling showed one thread pegging a core
at 99.7% while three cores sat idle and the others waited on the GIL, so
six "workers" barely beat serial (~0.25 entries/min).
Switch ordered_extractions to a ProcessPoolExecutor with a fork context
(copy-on-write memory; lets the existing in-memory test fakes cross into
children without pickling). Effective workers are clamped to the core
count. The parent still owns the journal, REST (finish_license), decide,
counters, the fork-network dedup, strict seed-order acceptance, the
systemic guard, and the graceful drain. Children run extract_repository
with github=None on picklable args and return their own count deltas
(threads had shared a Counter; processes return counts in the result).
A broken pool is replaced; a collateral worker_crash is a skip outside
the guard, a task-specific extract_worker_failure counts toward it.
Determinism is unchanged and tested with REAL processes: workers=6 ==
workers=1 byte-identical (fork collisions, out-of-order completion,
resume). 208 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Serial extraction at ~147s/repo made a 300-repo pool take days, and
~626 lazy per-file blob faults per clone pushed slow repos past the
240s deadline (a draw reached 21 entries from 347 attempts, 160 of them
extraction_timeout). Both fixed.
Parallel: a bounded ThreadPoolExecutor (--extract-workers, default 6)
runs clones concurrently, each into its own clone with a private counter.
The main thread alone owns the journal, REST (license via finish_license),
counters, and acceptance. Entries are accepted in STRICT SEED ORDER with
the fork-network dedup applied at acceptance, so the pool is byte-
identical to the serial draw (tested: workers=6 == workers=1, including
fork collisions, out-of-order completion, and resume). The systemic guard
and resilient skips are unchanged; a graceful stop drains started work to
the journal for resume.
Batched blobs: repository_changes fetches the fixes together and
Git.materialize fetches all missing blobs for the read revisions in one
`fetch --no-filter --stdin` (rev-list --missing=print), instead of per-
file promisor faults. Content is byte-identical (tested; >=12 fetches
collapsed to 1).
202 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
ObjectStoreExecutor.submit re-PUT the identical command.json on every
0.2s result poll, so a task running minutes wrote ~1,500 versions of one
engine-queue/search-exec/<id>/command.json. On the versioned memory
bucket that was the dominant growth: 109,715 versions, 10.3 GB (91% of
the bucket). retry_mailbox already retries a lost PUT acknowledgement
internally, so one upload per task is the correct delivery guarantee.
Guard the PUT with an `uploaded` flag: upload once (still retried on
transient failure), then poll only the result. One command.json version
per task instead of ~1,500. All submit semantics preserved.
59 executor tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Record the policy: OpenUltraSAST notifies the system/AI operator on
Discord about its own processes and results, and especially when a
process is stuck or impaired (stall, insufficient result, abort,
instrument_failure, timeout, budget/resource limit). Operator-facing,
not developer debugging, and explicitly not k3s/ax infrastructure.
Credentials by name in gitignored .env (DISCORD_WEBHOOK_URL,
DISCORD_OPERATOR_ID), listed blank in .env.example; off when unset.
Messages carry status/counts/digests only, under the hardening and
measurement-record redaction (no payloads, secrets, or repo content).
Notifier implementation to be wired/specced; this states the behavior.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The resilient draw only covered extraction. A per-candidate transport
failure in the decide/eligibility phase -- get_commit on an unresolvable
fix ref raising ValueError("http_failure") (observed: ljharb/shell-quote
commit 1518179) -- still aborted the whole draft, because decide only
caught source.Rejected and eligibility.RepositoryNotFound.
decide now catches that http_failure and returns a "decide_http_failure"
skip, and the run loop feeds decide + extract per-candidate instrument
failures into one systemic guard (25 with zero successes, or >200), so a
single bad commit lookup skips while a broken instrument (e.g. a network
outage failing every lookup) still aborts loudly.
Full unseen draft suite passes (48), ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Individual repositories in the advisory population raise per-candidate
extract.InstrumentFailure during extraction (git_exit_128 on an
unfetchable fix commit, clone_size_cap, and others), and each one
aborted the whole 300-repo draft. The draw is healthy; one bad repo
should not reset it.
extract_repository now also catches extract.InstrumentFailure and
OSError and records them as "extract_<reason>" rejections, so the draw
skips that repo and continues (the reason, e.g. extract_git_exit_128,
shows in the rejections counter). ExtractionTimeout keeps its existing
extraction_timeout skip; any other exception still propagates as a real
abort, so genuine code bugs are not masked.
A systemic guard preserves the instrument rule: run() raises
InstrumentFailure("systemic_extraction_failure") if 25 extraction
attempts fail with zero successes (e.g. git missing, auth broken) or if
extraction failures exceed 200 overall. A handful of bad repos with many
successes never trips it.
190 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A pathologically slow repository (observed: tensorzero, 183s to clone,
then `git blame -C` past the 180s command limit) raised
subprocess.TimeoutExpired, which Git.run mislabeled as a fatal
InstrumentFailure("git_launch_failed") and aborted the whole 300-repo
draft. The same slowness stalled throughput (a 54-minute no-progress
gap).
Give each clone a per-repo wall-clock budget (--extract-deadline,
default 240s, plumbed Clones -> Git, counting the initial clone). A
timeout -- a command exceeding the remaining budget, or the budget
elapsing -- raises ExtractionTimeout(Rejected), which the existing
rejection handler turns into an "extraction_timeout" skip so the draw
moves to the next candidate. Genuine launch failures and non-zero git
exits stay fatal; timeouts and launch failures are now distinct. The
deadline is recorded on the report only, not the journal inputs, so
changing it does not refuse resume.
181 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Update the front door and Status to the current state: static
abstract-interpretation blocking is NO-GO by hand, so the work is a
dynamic proof arm that builds and runs the app and demonstrates a
finding by a differential (triggers on the vulnerable revision, not the
fixed one), earning a block by execution rather than calibration. It is
pilot, not shipped. The release gate is a measured catch/false-alarm
number on unseen repositories, now being produced. Bump v2.0.0 -> v2.0.1,
link the developer guides.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
disk_bytes() summed the clone tree with rglob + stat() as a size cap
before every git command. On a blob:none clone, `git show` triggers an
on-demand blob fetch that writes and deletes transient
.git/objects/pack/*.rev files, so a path rglob had just listed could be
gone by the stat() call -> FileNotFoundError aborted the whole draft as
an instrument_failure. The race can hit any candidate, so it was a
non-deterministic draft-killer (observed extracting a real npm repo).
Skip a file that raises OSError mid-walk; the size cap is unaffected.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The GitHub /advisories endpoint ignores the `page` query parameter: it
paginates with the Link header's rel="next" cursor. The page-number
pager re-fetched the same first 100 rows forever and never terminated,
so enumeration never completed and no draft converged (a run fetched
5137 "pip pages" holding only 100 unique advisories).
Surface the successful response headers from JSONTransport, and walk the
rel="next" cursor per ecosystem until it is absent. The ecosystem /
published>=2019 / bare-number cwes filters are preserved on the first
request. Journal keys stay ecosystem:n (now cursor-walk positions) with
the saved next URL for resume; journal_version 4 refuses older journals.
Live smoke via the production pager: 300 unique advisories pulled in 3
API calls, advancing past the first page. Offline: 170 unseen tests
pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
enumerate_advisories paged the entire global GitHub reviewed-advisory
database (every ecosystem, every year, every CWE) before extraction, so
a 300-repo draft never converged -- a live run reached 21,962 advisories
in 4.5h with extraction never started.
Push the existing candidate() rejections into the query: page per
ecosystem, published>=2019-01-01, and cwes restricted to the seven
families. The cwes filter uses bare CWE numbers (89), which is what the
GitHub advisories API matches -- "CWE-89" silently returns zero. The
whole cwes filter is omitted if the taxonomy ever holds a non-exact
token, so a selectable advisory is never dropped. Journal version 3
(ecosystem:page keys) refuses incompatible older journals.
Live page-1 smoke (pip, npm): 100/100 rows carry a family CWE, none
published before 2019. Offline: 166 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A live G0 sourcing run failed on both: a transient connection/decode error aborted
the whole enumeration (the bare except raised http_failure on the first blip), and
the advisory journal stored full raw advisory JSON and reached 1.77 GB on a near-full
host. Fix: the JSON transport now retries URLError/timeout/ConnectionError/IncompleteRead/
JSONDecodeError through the same attempt-4 backoff as HTTP 5xx/429 (fatal only after the
cap or on a genuinely non-retryable error); enumerate_advisories journals only the fields
resolution/extraction consume via source.reduce_advisory (ghsa id, published/withdrawn,
type, CWE ids, package ecosystem/name for OSV, fix-commit links), with journal_version
bumped to 2 so an old full-format journal is rejected and resume-skip stays intact.
Offline: test_unseen_source/draft/extract 133 passed; ruff + format clean; mypy unchanged
(pre-existing verify.py lambda only).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Maintainer direction: the same agentic endpoint authors the PoC, writes a candidate
fix against the repair direction + failing PoC, and hands it to the differential
verifier, which re-runs the PoC and confirms absence. Closes the no-auto-patch gap:
the proof is the acceptance test for the fix (a wrong fix is caught), the fix stays a
developer-reviewed proposal, and the green proof is the objective signal. Specced
alongside proof authoring, after qualification.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The loop: catch a regression in the diff, prove it with a reproducible check,
the dev/agent fixes it (tool gives a repair direction + the failing proof, not an
auto-patch), then the same proof re-runs and no longer triggers = absence
demonstrated by execution; the proof becomes a committed regression test. Example
GitHub Actions workflow uses the real 'ousast pre-push . --base --head --deadline
--mode --artifact' entry; PR-comment formatter marked as a placeholder. Honest
status separates what exists (re-verify by construction, --base/--head) from
delivery work (PR formatter, auto-landing the regression test) and the open
large-repo latency.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Revise the developer guide for engineers with full backlogs and no spare schedule:
a vague warning dies in the backlog, so the value of the demonstrated proof is the
prioritization conversation -- reproducible evidence (input, behavior, the one-line
fix difference) a dev takes to a product owner or the business to win capacity,
dropped into the ticket system as a location, repair direction, and reproduction.
mcp-shift-left-agent brief: the agent's return should be ticket-ready evidence.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
docs/pre-push-for-developers.md: COBESY-structured guide for engineers who are
strong at dev but not security. Leads with the conclusion (zero-setup, no
annotations, diff-scoped, proof-backed), frames the agent-era case (a pre-push
check is how you oversee coding agents), states control (advisory vs blocking,
budget, lane, models), and is honest about limits with an explicit disproof
condition (coverage, large-repo latency, generalization being measured by G0).
.kiro/specs/mcp-shift-left-agent/: follow-up brief for an MCP agent + skills that
give devs and coding agents early shift-left feedback and a required pre-push
security step for agent oversight; builds on the existing ousast mcp server;
sequenced after the safety net is qualified.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
verification-architecture.md documents the existing two-stage design: the Cairn-inspired
search (brain + model calls on the VM) and the DAST proof (gVisor AX tasks in kube-ax),
the trust boundary (secrets VM-only, cluster tasks secret-free with restricted egress),
and the three verify lanes (ax=cluster default, vm=local Bubblewrap for $0 proofs,
docker=replay). Clarifies the proof is dynamic, not abstract interpretation: AI triage
stays the static backbone, proof earns the block by execution.
proof-arm-opencode/ is a follow-up brief: drive proof-authoring with OpenCode MCP, an
abliterated/GLM proof model (DeepSeek stays the search model), DSPy-optimized from
OpenObserve traces, to beat the safety-classifier refusals and nondeterministic wiring.
Flags the trust-domain decision that must come first (A: scoped key + egress inside the
cluster; B: brain on VM, only DAST in gVisor — recommended to start). Credentials live in
.env by name only (ABLITERATION_*, OPENOBSERVE_*); no secret value is in any tracked file.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Lockless old npm projects resolve era-appropriate deps (fixes glance filed->mime@4 crash);
npm ci untouched; git-failure falls back to plain install.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A lockless npm project (e.g. glance, 2015) resolved an unpinned transitive range to the
latest major ([email protected] -> mime@4, which dropped .lookup()), so the built app crashed
serving files -- a build-reproducibility defect, not an app or agent fault. The executor
now, for the npm-install (no-lock) case only and only when .git is present, resolves
dependencies as of the checkout's HEAD commit date via `npm install --before <date>`,
giving era-appropriate, compatible versions. npm ci (lockfile present) is untouched, and
any git/date failure falls back to plain install. demo.py also recognizes
npm-shrinkwrap.json as a lock for the ci-vs-install choice.
Tests: executor + smoke 75 passed, demo 107 passed; ruff clean; mypy clean on executor.py.
Executor-side change -> image rebuild required.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Feed the milestone back into the spec and steering. HANDOFF: the open "first demonstrated"
question is RESOLVED -- xiaomusic (path, prefix-check-bypass) demonstrated on the vulnerable
side (3/3 affected, 0/3 safe) and the blind fixed-side search produced 0 false proofs; pair
total $1.22. Records the six-fix chain that enabled it (parity, verify-time-only, fair-share
budget, served-root guidance, oracle prefix-bypass coverage, control-signal feedback) plus the
keystone. tasks.md: implementation note; pilot (6.2/6.3) in progress. spec.json: pilot status.
safety-net steering: proof-based BLOCK mechanism reached first execution-backed demonstration
(feasibility-pilot scope, not release readiness; pilot exit + 73-negative qualification open;
classifier stays triage). search README: status note. Payload-free; demos/boards stay gitignored.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
search-with-proof produced its first demonstrated finding on a real CVE fix-pair end to
end on kube-ax: vuln observes, fixed does not, 0 false proofs, $0.37, 55 calls. Enabled
by the full session fix chain (parity, verify-time-only, fair-share budget, served-root
guidance, oracle prefix-bypass coverage, control-signal feedback). Payload-free record.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Non-blocking advisory hint when the control shows the app didn't serve ${served_root},
so the agent self-corrects its served-directory wiring. Host-side; verdicts unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
When a path demo does not serve ${served_root}, the keystone control fails
(SideRecord.control_observed False) but the search only reported a generic
inconclusive, so the agent kept making the same wiring mistake. The pilot now
surfaces the affected side's control_observed in the verify result, and the
coordinator appends one advisory, deduplicated "served-root" hint (visible in the
board snapshot the agent reads) telling it to set the served-directory config to the
${served_root} placeholder. Advisory only: verdicts, _end, the oracle, the differential
and the demo validator are unchanged; the hint fires only when control_observed is False.
Tests: coordinator + pilot_run suites 78 passed; ruff clean; mypy unchanged (pre-existing
benchmarks errors only). Host-side change, no image rebuild.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Oracle broadening works in the agent: demo now targets the prefix-sibling vector. Still
inconclusive (0 false proofs, $0.29): the agent set the served-dir env var to its own
value, not ${served_root}, so the sibling canary was outside the served tree. Served-root
wiring is nondeterministic (a prior run wired it). Last-mile agent-reliability gap.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Plant a sibling canary so prefix-bypass path traversals (startswith without separator)
are demonstrable; parent canary + differential unchanged; glance still demonstrates.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The path oracle only modelled classic parent traversal (a `canary` file in the parent of
${served_root}). A common real path-traversal class is the prefix-check bypass:
startswith(base) without a trailing separator, which the fix closes with base + os.sep;
it reaches a SIBLING whose path string-extends the served directory, not the parent.
PathOracle now also plants a sibling canary `public_leak` beside the served `public`
(same nonce, 0600), so a prefix-bypass demo leaks it on the vulnerable side and the fix
blocks it. ORACLE_GUIDE names the second target and tells the agent to pick the vector
its code shows. One shared nonce stays correct: a canary leaks only when the demo's own
request reaches it, and each demo targets one vector whose fix blocks it on the fixed side.
observe()/capture()/the differential/SideRecord are UNCHANGED. Local glance ceiling proof
still demonstrates (parent canary, 3/3 vuln, 0/3 fixed, control on both sides). Gate:
worker+control+verify suites 106 passed / 4 socket-skips; ruff clean; mypy clean on oracles.py.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Served-root guidance validated: authored demo now references ${served_root}. Still
inconclusive (0 false proofs); agent spread budget across vectors and hit run_ceiling
(164 calls, $1.47/$1.50) without converging, and the committed traversal did not
differential. Remaining = per-app exploit-authoring + convergence (the layer the $0 local
glance proof did by hand).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Agent guidance: point the app's served directory at the ${served_root} placeholder via
arg/env/config so the planted observable is reachable. Prompt strings only; 37 worker tests pass.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Path searches authored and verified many demos but all came back with no differential:
the demo never pointed the app's served directory at ${served_root}, so the observable
(which lives relative to ${served_root}) was outside the served tree and unreachable. The
agent had set an app env var to a value of its own; the guide only showed the --dir form.
ORACLE_GUIDE now states the agent MUST set the app's served/root to the ${served_root}
placeholder via whichever mechanism the app exposes (command-line argument, environment
variable, or config written to start.files), not only --dir, and that serving elsewhere
makes the observable unreachable. Verify-time-only clarification, observable names, and all
prohibitions are kept. Prompt strings only.
Tests: tests/test_search_worker.py 37 passed; ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Fair-share budget validated: the Python agent completed the full loop and verified 18
demos within $0.74 of $1.50 (0 verify tasks before). Still inconclusive, 0 false proofs.
Root cause of no-differential: the authored demo never references ${served_root}; the
agent pointed the app's served dir at its own value, so the observable was outside the
served tree. Next: guidance to wire the served dir to ${served_root} for the path oracle.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Each pilot search gets a fair share of --ceiling-usd with settled rollover, so one
expensive search can no longer starve the others; total stays within the ceiling.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
--ceiling-usd was one shared SpendBudget passed to every search, so an expensive
search drained the pool and starved later ones: in the last 2-pair/$2 run the first
search spent $1.60 and the second hit the ceiling at $0.34 after finishing its analysis.
The driver now allocates a dynamic fair share per search (remaining_ceiling /
searches_not_yet_started), rolls unused settled budget forward, caps each search at its
share, and keeps total settled <= ceiling. A search now gets room to author and verify
after it finds the path.
Driver-only change (benchmarks/search/pilot_run.py) plus tests. tests/test_search_pilot_run.py
39 passed; ruff clean; mypy unchanged (pre-existing benchmarks errors only).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2 path searches (ts/py), vulnerable, cluster, $1.95 of $2. The trap fix is validated
behaviourally: ts pair 9 now authors a demo (verify_tasks 0->2; fails could_not_build on
entrypoint), py pair 13 drops 118->36 calls and its board shows a complete route-to-sink
+ entrypoint trace but hits the per-run budget ceiling before authoring/verifying. Still
0 demonstrations, 0 false proofs. Payload-free aggregate; raw stays gitignored. Next
levers: per-run budget/step allocation for demo-write+verify after analysis; demo
entrypoint/build (start.path) for compiled/served apps.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Agent guidance so path searches stop trying to confirm the oracle observable during
exploration (where it is absent) and instead author a demo for the differential verifier.
Prompt strings only; validators untouched; 37 worker tests pass.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The parity fix named each oracle's observable, but path searches then spent their
whole budget without authoring a demo: the agent tried to confirm reachability during
exploration, where the oracle fixture does not exist (it is injected only in the
verifier's sandbox at verify time), concluded "unreachable", and never committed a demo.
ORACLE_GUIDE now states the fixture/observable is injected at VERIFY time only and is
absent during exploration; its explore-time absence never means unreachable or justifies
ending the search. The agent confirms the mechanism (sink reachability, access to data the
app should not expose) from app code and its own sandbox test data, targets the observable's
known verify-time location in the demo, and relies on differential verification as the only
proof. All prior prohibitions (no out-of-band/off-disk access, no input capture, no revision
sniffing) and the parity observable names are kept. Adjacent wording shortened to fit.
Tests: tests/test_search_worker.py 37 passed; ruff clean; mypy unchanged (pre-existing
verify.py lambda note only). Prompt strings only; validators untouched.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
3 path searches (php/ts/py), vulnerable side, cluster verify lane, $2.95 of $3.
Payload-free aggregate; raw boards/demos stay gitignored. Finding: the parity fix
produced directed, target-aware search (the agent reasons about reaching the named
observable and inspects the app's serving guard), but no demonstration landed -- all
three spent their per-pair allowance before a differential, and two wrote zero verify
demos (budget consumed in reason/explore). Next levers: allowance/budget split toward
demo-writing, entrypoint discovery for php, and the explore-vs-verify path layout the
py agent flagged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The path family's agent guidance named no target, while command (${marker_dir}/marker)
and ssrf (${callback_url}) disclose theirs, so path searches could not author a demo
that surfaces the observable through the app. ORACLE_GUIDE now names it (canary in the
parent of ${served_root}, matching PathOracle.prepare). The reason/explore constraints
are refined to forbid only out-of-band access (direct disk reads, non-app channels),
input capture, and revision sniffing, while allowing the app to disclose it through the
vulnerability under test. Prompt strings only; the demo validator and its _PRIVATE guard
are untouched, so direct references to verifier paths remain blocked.
Tests: tests/test_search_worker.py 37 passed; ruff clean; mypy unchanged (pre-existing
verify.py lambda note only).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Outcome-only record of the first end-to-end demonstrated verdict and the two
masking defects (regenerated npm lock pinning an incompatible mime; agent demo
targeting the wrong file). Payloads stay in the gitignored private area.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Honest serve signal for the HTTP verifier: SideRecord.control_observed records
whether a benign verifier-planted in-root marker came back through the app's
interface, as its own guarded app start that never changes the verdict. A
2xx-based block was rejected (any HTTP status counts as alive). Gate green;
ceiling re-proof on glance still demonstrated with control_observed on both
sides. The search task image needs a rebuild before the next cluster pilot.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The HTTP verifier could not distinguish a mis-wired or non-serving app from a
genuine no-effect: both yielded `inconclusive`. This adds a positive control for
the path oracle.
- PathOracle plants a benign in-root marker file (random token from os.urandom,
distinct from the secret canary) and exposes it via a new Oracle.control()
hook (default None; only the path oracle implements it today).
- On the first run of an HTTP side, _side runs a benign GET for that marker as
its own fully-guarded app start. The marker returning proves the interface
serves the fixture; its absence — or any control failure — is recorded in the
new SideRecord.control_observed field but never changes the verdict, since a
valid app may answer only its own routes. A crash is still caught by the
existing could_not_run path.
A 2xx-based "nothing served" block was rejected: the verifier deliberately
treats any HTTP status as the app being alive (a 503-only app is a valid
observed side), so status-based blocking would false-fail valid apps. Soundly
converting a mis-wired-but-responding app into could_not_run requires a
demo-declared control, which is a separate follow-up.
Tests: new tests/test_search_control.py covers marker observed, absent marker
not blocking, control failure not blocking a working attack, and crash still
could_not_run. Full search-verify/probe/worker suites pass; ruff clean; mypy
unchanged (only the pre-existing file_bindings lambda note).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Replays a stored demo through the executor task and the verifier lane for one
side or both, with per-phase timings, so verifier changes are checked on the
exact image before any paid search.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The pip recipe ran under the system Python 3.13 while apps started under the
venv 3.12, so compiled extensions never loaded. Both now use sys.executable.
Package-style starts see the checkout root; NODE_PATH lists the project
node_modules under the checkout before the products mount.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Demos may reference ${served_root}, ${database_path}, ${marker_dir} or
${callback_url}; the verifier substitutes them per side, so apps that take a
directory or database argument can be wired to the oracle. Python apps start
under the verifier interpreter (the one pip built the products for): the image
carries a second system Python whose ABI tag broke compiled extensions.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Archives are gzipped and uploaded in parts of at most 90 MB with per-part
digests (edge request limit); pip installs ignore the task venv so transitive
dependencies land in products; the verifier sets NODE_PATH and the Java
classpath from products when starting the app.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
HTTP start waits up to 90 s (CLI 30 s) with TCP then HTTP probing; app stdout
and stderr tails are recorded; tool-call turns get 4096 output tokens with one
retry on truncation; tasks get 40 steps with reminders.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The kind default "pin" had produced ousast-engine-pin-search-verify-* names,
which the operator per-host egress rules do not match.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Built projects spool to /workspace without package caches (cap 1 GiB, both
sides); a plain-text model reply is fed back once; mailbox PUT/GET retry for up
to 90 s with idempotent re-send by seq; executor tasks are deleted on every exit
path.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Demos may set a bounded environment (loader variables denied) and up to 8 small
data files under .demo/; the reason step sees conclusions and may not re-propose
them; a task spend limit concludes its intent, not the search; finish wins over
stray tool calls.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
All five oracles demonstrated, gaming demos inconclusive, 38 s, ~500 MB RSS.
The probe dispatcher accepts a skipped isolation check in task-boundary mode.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The agent passes a manifest file or project directory; pip uses -r for
requirements files and the directory for pyproject/setup, npm ci/install runs in
the directory, composer/maven/gradle likewise. Build and verify failures carry
phase, exit code and a sanitised stderr tail.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
On kube-ax the probe detected task-boundary and then failed the bwrap preflight it
no longer needs; now it skips the preflight, runs every oracle pair through the
verifier path a task uses, and reports ok only if every real demo is
demonstrated and no gaming demo is. Fixes a double fixture-path translation.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Demos declare their oracle within the family allowed set (injection: sql or
command; output_encoding: xss; untrusted_destination: ssrf; path: path);
builds and app starts use the search memory budget instead of the 256 MB tool
limit (npm died with SIGTRAP); malformed tool-call JSON is fed back once.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
budget_spent carries end_detail (spend, task_spend, reason_rounds, ...); a task
whose allowance cannot cover the next call finishes with its facts; per-task
metrics in pilot records; input reservation bytes/2 + 1024 per message (measured
3-4x over-reservation before).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
One base (Temurin 21 JDK, node/npm/tsc, php/composer, maven, sqlite, bwrap)
built only when its Dockerfile or dependencies change, plus a browser variant;
engine/search/VM images FROM base@sha256 with the code in the last layer; the
image workflow runs on dispatch, tags, Dockerfile or pin changes, with a build
cache. Executor NPROC bound relative to the current process count.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Settled ~$0.003 per model call; worst-case reservations exhausted the $0.05 task
allowance after 2-4 calls. Now $0.40 per task, $2 per search (maintainer decision).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The operator keys per-host egress rules on the task-name prefix; engine passes and
replays had none. Names stay within 49 bytes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Zero running workers behind "no free workers available" now raises
ClusterUnavailable instead of waiting out the capacity deadline; CapacityDeadline
carries its message; cleanup failures no longer mask the original exception; the
pilot driver aborts the run on infrastructure failure before any model call.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
First live run: the reason step got ~190 tokens and returned {} -> silent
no_evidence. Now: a bootstrap intent reads the candidate and its entry point; the
reason prompt states the goal, family oracles, demo schema and must propose
intents when none are open; empty replies retry once then end as reason_refused;
failure records keep messages.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
One side per run over selected pilot pairs: board, brain on the VM with the
project DeepSeek client, executor tasks on kube-ax, verification on the VM lane
(AX verify lane switchable). Hard run ceiling with reservation before every call,
per-search and step caps; fixed side never sees the vulnerable revision; records
carry digests only. Dry run with the stub model reaches demonstrated.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Operator 2026-10-05: Substrate boots a ~1 s golden-snapshot actor per new task on the
other worker; retry apply after 2-5 s for the first minute, then 15-30 s.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Reasons carry type plus a sanitised message tail; command failures carry phase,
exit code and bounded stderr. Scratch walk tolerates directories pip removes
mid-walk; writable /workspace/home and runtime paths. Capacity retries delete a
possibly created task before resubmitting. Docker smoke checks the image first.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The official Python base sets GPG_KEY (public CPython signing fingerprint); the
executor rightly refused it as a credential-like name on kube-ax.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
20 development pairs, one per repository, 4 per language, 5 per family; identities
in a gitignored manifest, counts and digest committed. Executor smoke drives one
executor task end to end without a model and fails loudly on an unread clone.
kube-ax operator constraints as steering.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Demos are data the verifier executes (fixed build recipes, runtime+path start,
HTTP/CLI steps with captures; the canary is never referable). The agent loop with
the key never runs repository code: executor tasks without a key exchange commands
and results through presigned objects. AX capacity errors are back-pressure with at
most 2 tasks in flight; caches and builds under /workspace with a 2 GiB guard;
verifiers take one presigned archive per side. Launcher fix: private /dev in bwrap.
190 tests on the host; probe 5/5 demonstrated.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Replaces the accepted key-exposure exception: the key never shares a sandbox with
repository code; executor AX tasks exchange commands and results via presigned
objects. Firecracker executors only if the pilot build-failure rate requires it.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
kube-ax probe: AX tasks have no capabilities (no namespaces, no setuid), so no
in-task isolation. Demos become data the verifier executes; the explore key risk is
accepted and bounded by cap and rotation.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The kube-ax probe showed bwrap cannot map uids in gVisor as an unprivileged user.
As root (gVisor is the outer boundary) the sandbox uses pid/ipc/uts/net/mount
namespaces without a user namespace and drops the artefact to its own uid via
setpriv; the mode is recorded. The probe reports each namespace kind separately.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Probe runs each oracle toy pair with a real demo and two gaming demos, records
image facts, RSS and disk, fails loudly if isolation is unavailable; dispatchable
as an AX task. Worker: reason step without tools, explore tool loop whose run tool
goes through the sandbox with a scrubbed env (model key absent), spend reserved
before each call. Host probe: 5/5 real demos demonstrated, 10/10 gaming demos
inconclusive. 89 tests.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Board as immutable versioned blobs (coordinator sole writer); budget reserve/settle
before model calls; per-task and run caps; coordinator loop over an injected
executor; batch.py accepts a fixed allow-list of search variables; XSS browser
oracle; ousast-search-task image (Python, Node/TypeScript, PHP, Java 17, sqlite,
bwrap, chromium) built by CI. 103 tests on the host.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The verifier, not the artefact, observes the effect: SQL, path, command and SSRF
oracles owned by the verifier (XSS needs a browser executor, else no_oracle);
artefact exit code and output ignored; immutable checkout and dependencies;
3 runs per side; one-sided build or readiness failure is inconclusive. An
isolation failure raises IsolationUnavailable instead of reading as
could_not_build (the NPROC limit now applies inside the user namespace).
35 tests, including gaming attempts and 300 host processes.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Req 6.5 (maintainer decision 2026-10-05, with search-with-proof): BLOCK comes from
verified demonstrations; the engine is judged by top-K recall per push and advisory
false alarms; the pooled block gate governs only families without an oracle.
Task 17 adds the metric and re-scores existing arms at no cost.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
False alarms count every visible finding; the verifier owns per-family oracles;
inconclusive on one-sided build failure; feasibility pilot separated from a
73-negative BLOCK qualification; coverage ceilings; spend reserved before calls;
key isolated from executed code; cluster model calls supersede the pool VM-only
rule for this arm only. Requirements approval reopened.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
After reviewing Cairn: detection so far classified excerpts and hit a precision
wall; this adds directed search with an objective goal (a re-verified
demonstration), a fact/intent board, and reason/explore steps as AX Tasks on
kube-ax. Design only learns from Cairn (AGPL); no code copied.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
gVisor replays exhausted the hook's 30 s deadline in the snapshot step on a
1,725-file repository; cluster replays now use 30 s x a measured slowdown
factor, verified on a native sample. The engine runner deletes each pin's fetched
source (1.8 GB had accumulated), caps the cache at 500 MB and refuses below
1.5 GB free. Proof 4.4: two engine pins via the generic dispatcher match the VM
baseline exactly. Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Implemented with Codex; task 1.5 run by the coordinator against the real
sources (S3 memory read-only).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Decisions: long-deadline arm (300 s), adoption needs no unseen regression
(Req 5.4), permissive repositories by name only, two AX lanes. Also adds the
trained-classifier and joint-optimisation arms to learned-decision-engine.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Maintainer 2026-10-04: the safety net is for unseen code and the A/B tests exist
to prove the approach generalises; infrastructure work had drifted from that.
Runs are designed for AX on kube-ax, kind is the fallback.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Two engine actors died at gVisor level during a 12-pin run (AX phase Failed, no
result). Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Through Cloudflare, signed requests for keys ending .tar/.gz/.zip/.bin reached
RustFS altered and failed with 403 (cf-cache-status BYPASS); the same signed
request straight to RustFS on the LAN returned 200. A Cloudflare cache-bypass
rule on files.because-security.com fixes it (all extensions 200, DYNAMIC); the
engine executors keep .dat as a safeguard, the plane keeps output.tar.
Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The Joern leak-audit record named spent population v1/v2 repositories, which
the reserved-repository guard forbids outside benchmarks/independent/ (no v3
repository was involved: 0 matches). Records now carry counts and SHA-256
digests. With Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Coverage 72 path, 231 asked-nothing, 9 failed, 189 unsupported. The engine
almost never distinguishes a vulnerable function from its fix, so a Joern-slice
prompt experiment is not funded; the lever is the engine's knowledge of fix
mechanisms.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A lean overlay omits otel-collector, prometheus and jaeger and disables the
exporters; on an existing cluster up.sh scales them to 0. Verified live: render
has no otel-system resources, up.sh idempotent (54 s second run), smoke Run
done, kind node 2.66 -> 1.82 GiB. Docs say server VM. Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Batch 3 of real (vulnerable function, fixed function) pointer pairs, mined at
zero model cost from reviewed GHSA advisories (pip, npm, maven, composer,
published since 2023, newest first). Same rules as batch 2, widened three
ways: a merged PR's single commit when the advisory links no commit, commits
up to 6 files / 240 lines, and two-family advisories with the family read
from the diff. Every candidate's function diff was read: 412 reviewed, 255
kept, 3 one-per-repository, 2 withheld by the reserved-population guard
(bisected with output discarded). Permissive licences only; no repository
named elsewhere under benchmarks/; no WordPress plugin.
Kept 251: injection 56, untrusted_destination 26, path 71, output_encoding
69, access_control 20, deserialization 9; python 83, typescript 48, php 46,
java 37, javascript 37. Only 12 post-cutoff: batch 2 had used up the
post-2026-06-01 pool. Memory 1133 -> 1635 static examples; complete pairs in
the whole memory 459 -> 710. Embeddings: 15 calls, $0.0030. No chat-model
call, no Docker, no Joern. Record:
benchmarks/measurements/2026-10-04-decision-engine-harvest-3.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The query fix stops a class-level call from aborting the whole taint batch on
real PHP code; gate, map_gate and pair_gate outputs are unchanged. Trace parsing
lives with the runner (benchmarks/learn/engine_trace_parse.py).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
On real PHP source (CPS-IT/mailqueue) a call outside any method has a null
.method; inScope/nestedInLabeled dereferenced it (NullPointerException) and the
script exited 1, losing every taint answer for the repository. 16 sites in
taint/dominance/config/vocabulary.sc now treat a null method as out of scope;
non-null behaviour is unchanged (same outputs on the repro graph for the other
scripts). Static regression test added. Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The worker sent tuple params as JSON arrays; taint.sc expects comma-joined
strings (ujson InvalidData, exit 1). Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Root causes of the 50 failed pins in trace run 1: advisory pairs were fed as
excerpts (invalid Java/JS fragments), the JVM defaulted to ~768 MB heap in a
3 GB container, and mixed-language pins were refused. Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The 2026-09-14 engine image has no PyYAML or boto3; the worker no longer imports
learn.examples. Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
slice.py renders source -> step -> sink lines with fixed wording for flow,
no-flow and not-analysable; the prompt option is default-off (byte-identical
default prompt). semantic IR: JS/C declarations count as assignments, augmented
assignment keeps its target (gate, map_gate, pair_gate outputs unchanged).
Audit over exp-004's 501 units: 59 flow, 161 no-flow, 281 not analysable;
has-flow within-pair accuracy 0.52, pooled AUC 0.55 -- no leak. Fixes a
numbering bug that rejected every excerpt with a blank line. With Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Population v2 scans are named <case>--<side>_a/_b.json; the script looked only
for <case>--<side>.json and recorded those examples as engine not-run. The two
repeats combine symmetrically for both sides. Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The estimator priced all input at the full rate (\$5.58 estimated vs \$1.24
spent); it now uses the family mean billed cost of cached responses when usage
is recorded. Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A pending-then-frozen registration left two rows and rows[0] could be the stale
pending one, so fill and run refused a properly frozen experiment. Implemented
with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Implemented with Codex; review fix: the bounded client keeps the configured
thinking setting so a fill answers as the run would.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Supplementary analysis, not a new decision: exp-003's verdict stands.
The 142 units excluded for an uncached response were filled with the same
six incumbent programs, parameters and request keys, on exp-003's
937-example memory (the harvest's 196 rows and its 2026-10-03 embeddings
left out; a zero-cost replay first reproduced exp-003 exactly: 211
decided, 1056 replayed, $0.582876). Fill: MeteredClient, ceiling $0.99
under the approved $1.00, 697 client calls, $0.331759, no provider error.
Replay on all 353 units (244 paired, 146 groups): 0 client calls, $0.0,
1768 responses replayed; no arm flags any unit, the registered rule reads
inconclusive. Calibration holds only for untrusted destination (ECE
0.0454) and arm A still finds no BLOCK point there.
benchmarks/experiments/exp-003-precision-bound-blocking/result-complete-cache.json
exp-004 registered in s3://sast-memory (manifest 87fceb65..., pin
2ad9c1d, units pending): benchmarks/experiments/exp-004-pooled-block-gate/
registration.json. docs/decision-engine.md: both, briefly.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer decision 2026-10-03: BLOCK precision is shown pooled across
families. Req 6.4 (amended; 7.4 annotated): one t_block on a per-family
Platt-normalised score (calibrate.fit_map, training folds, inner
cross-fit), Wilson 95% LB >= 0.95 over all families' training flags; a
family blocks only with >= F = 10 held-out flags and zero errors, else
advisory; t_quiet per family keeps >= 90% of training paired positives.
evaluate: normalised, pooled_threshold, quiet_threshold,
pooled_block_gate, family_floor, apply_floor. experiments: rule
pooled_block_gate (normalise: platt, floor, recall), decide_pooled on the
pre-floor pooled held-out LB, floor table, quiet_recall and after-floor
totals in analyse_rule; exp-003's analysis is unchanged.
plane/experiments/exp-004-pooled-block-gate.yaml: procedure and decision
rule registered now; units frozen at run time from the post-harvest
memory (units_status pending until then).
Tests (tests/test_learn_rules.py): pooled threshold where no family alone
reaches the bound, no held-out label moves a threshold or the map, a
family with one held-out error cannot block, t_quiet and no blind
silencing, the pooled verdict, the exp-004 manifest.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Real (vulnerable function, fixed function) pointer pairs for the six compiled
families, mined at zero model cost from reviewed GHSA advisories (pip, npm,
maven, composer), newest first: 79 of 98 published after 2026-06-01
(post_cutoff = true). Each pair: one linked one-parent fix commit, <= 4 source
files / 120 lines, the function holding the changed code lines declared once on
both sides with a changed body; every function diff read and kept only when the
named function holds the fix (145 reviewed, 100 kept); 2 dropped by the
reserved-population test; permissive licences only, no repository named
elsewhere under benchmarks/.
New label source advisory-fixes-2; the advisory-fix tests run over both
batches. Memory 937 -> 1133 static examples (98 paired groups added);
positives-only groups stay 99 (vibe-py/dev-php/v2 data gaps), their share of
positive groups in the six families 0.40 -> 0.29. Embeddings: 7 calls,
$0.0014. No chat-model call. Record:
benchmarks/measurements/2026-10-03-decision-engine-negatives-harvest.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Zero-cost replay of the six slice-2 programs: meter 0 client calls, $0.0;
211 of 353 units scored, 142 excluded for an uncached response (path lost
all 39). Arm A (calibrated) fails calibration (ECE 0.077 access control,
0.1067 injection); arm B (Wilson LB >= 0.95, nested) chooses no threshold on
any fold, as the registration's power note predicted (73 clean flags needed,
38 paired injection positives); secondary arm C flags nothing either. The
instrument check shows real scores; the zero is the rule's.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Registered in s3://sast-memory at dd7ebc1: manifest 5917f9a1, units 338de998
(353 units, 146 groups). Primary metric, decision rule and the threshold
selection are fixed in the manifest; the power note says arm B cannot reach
a Wilson LB of 0.95 on this data (73 clean flags needed, 38 paired
positives at most).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
kind: rule experiments score once from the response cache behind a
zero-ceiling meter (a miss is excluded and counted, never asked) and apply
each arm's BLOCK rule to the same held-out scores. Arm B chooses a per-family
raw-score threshold on the other outer folds only (lowest t with Wilson 95%
LB >= 0.95); a test fails if a fold's own labels move its threshold.
Manifest and frozen units (353 units, 146 groups, 244 paired) for exp-003.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Kind proofs: records 01 and 02. Task 2.6 (receiver removal) is paused while the
safety net and the decision engine come first. deployment.md conflicts resolved
in the plain-language style; corrects the RuntimeClass claim (Substrate runs
runsc inside its worker pod).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Live on kind with the runner image that carries 2.2
(ousast-runner@sha256:d2a55293...; templates and validation-46 Tasks repinned,
128 lines; kind-images.json updated): the smoke Run on store delivery finished
in 26 s and 24 s; the tar was PUT to runs/ax-e2e-smoke/facts/output.tar
(10240 B) and collected 2 s later; read back from the store its members
(facts.json 451 B, summary.json 238 B, units.jsonl 291 B) equal the extracted
files; facts.json, summary.json and done.json (149 B) re-put, state.json
(350 B) mirrored; a rerun from an empty results root seeded state.json from the
store and finished in 2 s without resubmitting. The actor's egress policy read
back live: one tlsPassthrough rule on 443 naming the store host, nothing else.
`ousast plane status ax-e2e-smoke` printed the table each time. Record:
benchmarks/measurements/2026-10-02-k8s-02-store-delivery/record.json with the
smoke outputs, store read-back and policy beside it. Task 2.6 (receiver
removal) is paused by the maintainer's reprioritisation.
Gates: pytest 2031 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0. (The first run, at 0.97 GiB free disk, failed 11 tests and errored 5
on the FileStore's 1 GiB free-space refusal; rerun after the disk was freed.)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Rerun records, serve-static and unrolled/secure (same planted commits) with
this build installed from its wheel: 6 replays and 10 hook pushes, all exit 0
in 0.69-1.21 s against the 30 s deadline, input read proven per run. The
planted JS eval in a new file surfaces in the quick tier on the host and, in
one Docker engine run (190.9 s, deadline 300), as two advisory engine findings
classified new/file_added_in_head. The planted Python %-formatted SQL is still
missed (no quick rule, no host engine): recorded as a coverage gap, no rule
edited. Go is named as not covered. The run caught a wrong install hint in
the hook (the package is not on PyPI); corrected here.
Record: benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
`ousast pre-push install|uninstall [REPO] [--force] [--docker]` ships in the
wheel with the hook as package data; it honours core.hooksPath, refuses an
existing hook unless --force (which chains it as pre-push.before-ousast), and
uninstall restores the chained hook. ops/ousast-docker is a hook-callable
script: the repository read-only and the artifact directory at their own
paths, stdin passed through, no network, --memory 3g, the GHCR 2.0.1 image by
default, advisory exit unless blocking. Help text for every pre-push flag; a
first-run guide, settings, skip-reason glossary, CI recipe and full --help in
docs/scanning.md; README tiers 1 and 2 pull the published image. The image
was not rebuilt: 1.1 GB free, below the 4 GB threshold. Requirements 7.4, 9.6.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
--engine inline|background|off (hook: OUSAST_PUSH_ENGINE). background returns
after the quick tier and starts one detached explicit replay of the same base
and head (push/background.py), one at a time per artifact directory; its
artifact lands under <artifact dir>/engine/<repo key>/<head>.json and the next
run prints "Engine result for <head> from the previous push" once. A deadline
miss says so in one line naming the deadline and the background option. No
engine speed work. Requirement 9.5.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ChangeContext records the paths Git reports as added. An entailed,
undischarged operation in an added file, with complete head evidence and no
identical operation in any base answer, is classified new
(file_added_in_head) instead of base_incomplete; admission accepts that reason
as change evidence. Unpaired add/delete pushes and every other unknown
comparison keep their vetoes. An engine finding whose novelty is established
and whose only vetoes are capability vetoes is shown as an advisory engine
finding, not an alert. Requirement 9.4.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Before the engine and inside the same deadline, the shipped quick rules run
over the head side of the changed files (push/quick.py) and keep matches on
changed lines or in changed functions. They print as advisory pattern matches,
labeled apart from admitted alerts, are kept in the artifact's quick_tier with
the files and bytes read, and never admit or block. A new branch with no
remote base runs the quick tier over its last commit and says so. The planted
eval in serve-static's new debug.js now surfaces; the planted Python
%-formatted SQL in records does not (a quick-rule coverage gap, no rule
edited). Requirement 9.3.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
One terminal line per distinct coverage reason (push/explain.py): hex-encoded
paths decoded, per-question sha256 tokens dropped and counted, unknown reasons
still printed; the JSON artifact keeps the raw values. A changed language with
no engine frontend and no quick rules is recorded and printed as "language not
covered: <lang> (...)", in pre-push and in `ousast scan`. The scan report
renders the eight degradations cli.py recorded but never showed, and any
reason without a sentence. The hook forwards OUSAST_COMPARISON_BASE.
Requirement 9.2.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The installed hook exits 0 with one line when ousast (or OUSAST_COMMAND) is
not on PATH, still chaining a prior hook. In Git hook mode without an explicit
--mode blocking, argument/usage errors (an invalid OUSAST_PUSH_DEADLINE, a cache
directory inside the repository, a refused experimental flag) and unexpected
exceptions print one "check did not run" line and exit 0. Blocking mode keeps
its usage errors. Requirement 9.1.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
_execute builds the task's `delivery {put, inputs}` from Delivery, passes it to
router.starter, grants the store's host in the egress policy, awaits
Delivery.delivered (the tar's exact key in the store), then collect()s:
download, extract with the member checks, re-put the declared artifacts, write
done.json; the outcome is the collected summary.json. Three findings from the
tests: a rerun of an unfinished task completed at once on the previous
attempt's tar, so Delivery.reset(task) deletes output.tar and done.json before
every attempt; a worker thread that raised left the run blocked on its queue,
so the worker now turns any exception into a failed task; a consumer may
declare a producer's summary.json without the producer listing it (the loop
Run does), so declared(run) publishes outputs[] plus every artifact a consumer
names. The receiver thread survives behind OUSAST_DELIVERY=receiver for this
group only (no presigned URLs, the http rule, OUSAST_ARTIFACT_URL in the env);
the default path renders no receiver address into any Task. The fake ax PUTs
to the start body's URL and GETs its inputs from theirs; the reconciler tests
run through the S3-shaped fake store and assert the objects, done.json, the
seeded rerun and the receiver mode. reconciler.py: 549 lines (bound 550).
Gates: pytest 2031 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Append task group 17 "Operational readiness (2026-10-02)" with one task per
maintainer-approved item (never block, name every skip, quick tier, new-file
classification, engine honesty and background mode, install/Docker/docs, proof),
add Requirement 9 (9.1-9.6, marked added 2026-10-02, maintainer-approved), and
check 13.4 and the parent of 14 as the audit found them done.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
policy_for(task, workspaces, model, store_host, receiver_host=None) emits one
tlsPassthrough rule on 443 over the sorted union of the memory store's host,
the bound Workspaces' https Git hosts and the bound Model's egress hosts. There
is no http rule: an http Git repo is refused by name (it could not be reached),
a store given as an address is refused naming S3_ENDPOINT (the gateway allows
hostnames only). receiver_host adds the old receiver's http rule for the path
group 2 keeps until 2.6; the reconciler passes it by keyword (passing the
receiver's address positionally would have raised outside the worker's try and
hung the run on the queue -- 2.4 hardens the worker). Tests: the store-only
policy, the Model-bound policy, the two refusals, the receiver exception, and
every committed template under plane/tasks/ rendering exactly the store plus,
for verify and roles, the provider host.
Gates: pytest 2027 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
StartGate.offer validates an optional `delivery` beside the credentials: https
URLs to a hostname (never an address, no userinfo; AX_RUNNER_INSECURE_DELIVERY=1
admits http in unit tests only), names ^[A-Z][A-Z0-9_]*$, a 400 that names the
field and echoes no URL. The runner PUTs its output tar to delivery.put and
GETs each declared input from delivery.inputs[NAME] (deliver() gains a method;
log lines and DeliveryError carry the URL without its query, so no presigned
signature reaches a log, a marker or a summary). OUSAST_ARTIFACT_URL/DIAL stay
honoured until task 2.6 so both paths run in the same build; the completion
marker and the retry-only boot are unchanged, and a re-start brings a fresh
URL for the retry. Router.start_task carries `delivery` in the body and
redacts its URLs like credential values. Tests: the gate's acceptance and
refusals, an end-to-end PUT/GET against the S3-shaped fake with no URL in any
log or file, the failed-PUT-then-retry-only boot, deliver() with PUT.
Gates: pytest 2024 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
src/openultrasast/plane/delivery.py: Delivery(store, run, base) mints a presigned
PUT for runs/<run>/<task>/output.tar and a GET per declared input
(runs/<run>/<producer>/<artifact>), expiry OUSAST_TASK_TIMEOUT + 600 s;
delivered() asks the store for the exact key; collect() downloads, extracts
under the run directory with the receiver's member checks (no symlink, hard
link or escape; nothing written on a rejection), re-puts the declared outputs
as single objects and writes done.json; put_input() for inputs no task
produces; save_state()/load_state() mirror state.json. The reconciler opens the
store first (no fallback; the CLI reports a store that cannot open and submits
nothing), mirrors state.json after every change and seeds a rerun with no local
state from the store (the 2.1 bullet's state mirroring is what gives the new
module its importer). The reconciler crossed 500 lines here (517) rather than at
2.4 as 1.6 projected: the bound moves to 550 with the reason in the assertion,
as 1.6 decided. tests/plane_fake_store.py is the S3-shaped fake (objects in a
dict, presigned URLs on a local server); tests/test_plane_delivery.py covers the
body, expiry, polling, rejection, declared-only re-put, state seeding and that
no URL reaches a log. Module-audit manifest regenerated (148/5, no orphan).
Gates: pytest 2020 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
benchmarks/measurements/2026-10-02-k8s-01-profile/record.json with the live
evidence of group 1: `ousast plane doctor --profile kind` passes every line
(exit 0, 1.4 s; the store check reaches the bucket: verify 0.62 s, probe object
24 B); `--profile k3s` fails on the unreachable context and on the images file
the release has not published yet (two expected lines; the namespaces are not
checked behind an unreachable context); the literal scan has 0 offenders (23
before 1.3) and fails naming workspaces.py:37 when a literal is reinserted;
up.sh reran idempotently in 38 s and the smoke Run finished in 24 s on the
rebuilt image (facts.json 451 B, summary.json 238 B); the placement probe says
`any`; the reconciler bound numbers of 1.6. Doctor, literal, smoke and up.sh
outputs beside it.
Gates: pytest full suite on this tree's src/ and tests/ content: 2010 passed,
42 skipped, exit 0 (the 1.5 run; the 1.8 run reported two module-audit
failures caused by the uncommitted 2.1 module present in the working tree);
ruff check 0; ruff format 0; mypy 0; mkdocs build --strict 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
docs/deployment.md B.1: Substrate's ateom-gvisor worker runs runsc from the
release tarball named in manifests/ate-install/sandboxconfig-gvisor.yaml
(gs://gvisor/releases/nightly/2026-09-02/<arch>/gvisor.tar.zstd, sha256-pinned
per architecture), fetched by the atelet on the node; the node must reach it or
a copied SandboxConfig must name a mirror with the same sha256. The check is the
atelet's fetch line in the worker pod's logs, run by hand on the day k3s exists;
labelled planned until then (design open question 3 stays open).
Gates: docs only; mkdocs build --strict passed with this paragraph present in
the 1.4 gate; no file a test or linter reads changed since the 1.5 gate.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
reconciler.py: 477 lines at 6bd44ca, 489 after 1.2 (the profile parameter of
run() and its docstring). Task 2.4 adds the Delivery wiring (about +25) while
the receiver path is kept for group 2, and 2.6 removes the receiver wiring
(about -20): projected 514 during group 2, 494 after 2.6. The estimate crosses
500 while both paths coexist, so 2.4 moves tests/test_plane_reconciler.py's
bound to 550 with the reason in the assertion message; 2.6 records the final
count. Numbers in benchmarks/measurements/2026-10-02-k8s-01-profile/record.json
(record 01, drafted here, completed by 1.8).
Gates: no file a test or linter reads changed since the 1.5 gate (pytest 2010
passed, 42 skipped, exit 0; ruff 0; format 0; mypy 0).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ops/ax/probe-placement.sh scales ousast-pool to one worker, applies probe-pool
(1 x 2 CPU / 3Gi, the already-built ateom-gvisor image, no ko build), runs one
model-free repo-facts Task requesting 2560Mi (limit 3Gi) and one requesting
256Mi, records `kubectl ate get workers -o json` after each resume, and removes
everything it created. Live on kind: both actors landed on the 1536Mi
ousast-pool worker (status.allocated.actors=1 there, 0 on probe-pool) ->
placement: any. design.md section 4 records the outcome and the fallback group
5 must take (an engine atespace with engine-pool as its only pool if Substrate
scopes pools by atespace; else a plain Kubernetes Job on k3s). Evidence under
benchmarks/measurements/2026-10-02-k8s-01-profile/probe/.
Gates (tree identical to 1.4 in src/ and tests/; ops/ gains the probe script,
which the literal scan covers): pytest 2010 passed, 42 skipped, exit 0; ruff
check 0; ruff format 0; mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ops/ax/up.sh reads kube_context, registry, pools.default and the images path
with `python -m openultrasast.plane.profile --print`, writes the runner digest to
ops/k8s/profiles/kind-images.json instead of ~/.cache/.../runner-image, renders
ops/ax/workerpool.yaml.tmpl from the profile's pool (POOL_NAME/REPLICAS/CPU/
MEMORY, WORKER_IMAGE), and keeps an existing pool and an ax CLI newer than its
checkout instead of rebuilding them (ko and go install would refill several GB
of Go cache on this disk). smoke-run.sh takes the runner from `--print
images.runner` and runs the checkout's code with OUSAST_PYTHON. repin_templates
reads the JSON images file (runner key; the one-line pin still reads). `--print
images` returns the file path (the first live rerun fell over this). Templates
and the generated validation-46 Tasks re-pinned to the rebuilt image (128
lines, one digest).
Live: up.sh reran idempotently on kind in 38 s (every step kept except the
runner build); smoke-run.sh done in 24 s on the new image, facts.json 451 B,
summary.json 238 B delivered (record: benchmarks/measurements/2026-10-02-k8s-01-profile/).
Gates: pytest 2010 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0; mkdocs build --strict 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
tests/test_plane_literals.py scans src/, plane/, ops/ (minus ops/ax/up.sh and
ops/k8s/profiles/) and docs/ for the kind context, the kind registry, the kind
network's gateway, a loopback address with a port and port-forward; router.py
keeps the last two, a line tagged ax-tunnel or loopback is allowed, and a
committed template's digest-pinned image is accepted only while it names the
kind profile's registry (checked by a third test). A self-check reinserts each
literal into a fixture and sees it caught. The runner's health URL and the Joern
session's query URL are tagged loopback; docs/deployment.md and ops/ax/README.md
stop naming this host and describe the kube context and the registry as the
profile's (what 1.2 made configurable).
Gates: pytest 2009 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0; mkdocs build --strict 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
doctor(profile) checks the profile's kube context, the two namespaces behind a
reachable context only, the images file's runner reference in the profile's
registry (a local registry over its v2 API, a remote one via docker manifest
inspect) and opens the memory store; its first line reports every configured
address (Req 1.2). Egress(executable, context), open_router(context, url=...),
receiver_cluster_ip(context) and Receiver(..., context) take the profile's
context and build none; generate.increment/render and harvest take the profile's
atespace. `ousast plane doctor|run|workspaces|harvest --profile`. The test fakes
(kubectl-ate, kubectl, ax) record the context they received and the tests assert
it is the profile's. grep "kind-" src/openultrasast/plane: no match.
Gates: pytest 2006 passed, 42 skipped, exit 0; ruff check 0; ruff format 0; mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ops/k8s/profiles/kind.toml and k3s.toml hold the values that differ per cluster
(exec, kube_context, registry, images, router_url, ax_server, memory, atespace,
snapshots_bucket, image_pull_secret, pools); kind-images.json carries the digest
pin ops/ax/up.sh wrote. load_profile() reads OUSAST_PLANE_PROFILE (name or path,
default kind), applies OUSAST_<FIELD> and OUSAST_POOL_<NAME>_<FIELD>, and refuses
an unknown key, a credential-shaped key or value, an exec outside local|remote
and an images file without digest pins. `python -m openultrasast.plane.profile
--print FIELD` serves shell scripts. The module-audit manifest is regenerated
for the new module (counts 147/5, no orphan).
Gates: pytest full suite 2002 passed, 1 failed (the module-audit manifest, fixed
by regenerating it; tests/test_module_audit.py re-run exit 0), ruff check 0,
ruff format 0, mypy 0.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
CI and the release job check `ruff format --check .` and `ruff check .` over the
whole repository, while the local gate checked only src and tests, so eight
benchmark scripts drifted and the v2.0.0 release job stopped before building.
The fixes are formatting, two noqa comments and one renamed loop variable; no
script computes anything differently. The local gate now checks the whole tree.
The release workflow gains an images job: Dockerfile -> ghcr.io/<owner>/openultrasast,
plane/Dockerfile.runner -> ghcr.io/<owner>/openultrasast-runner, tagged with the
version, and the digests written to the job summary.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Rewrite README.md as the front door of the docs site: a short "What it is", the
Status (NO-GO against M4, figures quoted from docs/where-we-stand.md), a tiered
install (quick scan without Docker; standard/deep with the openultrasast:dev
image; the plane on ax for maintainers; k3s planned), then one section per
docs page, each linked by relative path so it works on GitHub and on the site.
Stale claims removed: `ousast index` (deleted), `OPENROUTER_EMBEDDING_MODEL`
(no longer read), the duplicated scan-mode, rule and gate tables (now on
docs/scanning.md and docs/evaluation.md), the HarnessX-era wording. Adds a
Licence section stating that no licence file exists yet.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Legacy inventory 2, the six STATUS NOTE items (the archive README row landed with the R2 commit).
ai-service-plane: 11 of 11 tasks checked, spec.json phase set to implementation-complete as 5b96199
did for four other specs. contributor-scan: dated note in requirements.md -- paused since 2026-09-23,
27 of 51 tasks, v1.2.0-alpha.1 NO-GO, remaining tasks wait on pre-push M2/M3. finding-feedback-loop:
dated note -- dormant, design unapproved since 2026-09-09, depends on the paused spec. The steering
overview's spec table carries the same three states, its leftover 2026-09-09 module-count line is
removed and the collected-test count is the measured 2,023. docs/memory-and-detection.md names the
archive path of model_sinks, the one stale script name left in docs after 067adb9.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
New page docs/detection-techniques.md: the three representations (regex over the
file, tree-sitter IR, Joern CPG) and what each can see; Joern as a section of its
own (frontends, one graph per language partition, census as the instrument check,
batching against the measured cost curve, deadlines, degradations, where the engine
sits in the plane, cost and misses with records); taint from fact tables to query to
verdict with each refinement traced to its record or commit; obligations, dominance,
constant abstraction, census; regions, ranking ties and budgets; the residual
question and MAX_JUDGED_CANDIDATES; REGRESS verdicts and the unassigned
execution_confirmed rung; an honest table of what each technique misses.
Nav entry under "How it works" after Scanning; one link from the index; one
cross-link line each in scanning.md and architecture.md.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
openultrasast-scan: evidence ladder as verification.py defines it, with the levels nothing produces today; --mode prerequisites, --fail-on, recorded degradations, run artifacts, ousast mcp; retired [evidence] threshold language removed. openultrasast-triage: reasons verified against calibration.py; deterministic fusion; adjudications as labels, plane verdicts as features, rule changes through improve or a registered experiment; fix-audit merged as a fix-and-re-scan section. kiro: spec-quick and spec-batch removed; validate-gap and validate-design merged into spec-design (--gap, --review; the review gate stays); steering-custom merged into steering; core steering names overview.md, roadmap.md, safety-net.md; instrument rule in debug and verify-completion; repo gates named in validate-impl. AGENTS.md, README skills paragraph and tests/test_agent_skills.py (.agents/skills, two product skills, drift tests) updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Legacy inventory 2, item R3. The three keys loaded into ResolvedConfig and nothing read them: no
`.network`, `.workspace_readonly` or `.top_k` access exists in src outside config.py. The sandbox
runner always passes `--network none` and `--read-only` and rejects host networking by argv
inspection, so the two sandbox keys were knobs that turned nothing; no stage read top_k, and
`[complexity] max_hunter_hotspots` is the one complexity key the scan reads. They join RETIRED_KEYS
with the same 2026-10-02 mechanism as the other unread keys (RetiredConfigError naming the reason).
The fields and their loaders are removed; memory_mb, timeout_seconds, pids_limit (read by
regress/recipes.py and sandbox/runner.py) and max_hunter_hotspots stay. test_config.py covers the three
retirements and the override test now exercises max_hunter_hotspots. RELEASE_NOTES Unreleased carries
the line.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reworks the production topology section after the maintainer's three
decisions of 2026-10-02: the target cluster is k3s (containerd, no Docker
socket, runsc + RuntimeClass for gVisor workers, Traefik) with digest-pinned
images published to GHCR by GitHub Actions; kind stays a first-class local
execution mode (two profiles of one code base, every change proven on kind
first); and the reconciler stays the CLI, with a planned remote mode in which
the same `ousast plane run` submits to ax-server in k3s, tasks deliver their
artifacts to the S3 memory store through presigned URLs, and the CLI polls the
store for the same status and attribution table. The in-cluster reconciler
and the in-cluster receiver become a later option. The diagram is redrawn
without an inbound path; the pre-push section says the hook or CI calls the
CLI in remote mode and either waits with a deadline or reads a later `plane
status`. Every item that is not built stays labelled planned.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Answers the maintainer's questions on deployment.md: ax is a Kubernetes
application (ax-server Deployment + Redis in ax-system) that schedules every
Task as a gVisor actor on Agent Substrate (ate-system, WorkerPool CRD); there
is one runner image and concurrency equals warm workers, one actor per worker.
Names the three host-bound components (reconciler, receiver, Joern engine) with
their planned in-cluster replacements, the production shape of the pre-push
hook (local deterministic check in the deadline, delta Run on the plane
asynchronously), the estimated per-push cost from the two measured
per-candidate rates, what is not built (no `ousast plane scan`, no diff
candidates task, population-only Workspace generation, no in-cluster
reconciler), and a sizing table from plane/tasks/*.yaml. plane.md gets a
two-line pointer to the section.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Legacy inventory 2, item R2. tests/test_harnessx_removal_equality.py loaded
benchmarks/archive/harnessx_removal_equality.py by path and checked that its comparison sees a
difference. The spec is implementation-complete and the proof is the committed pair of records
(2026-09-30-harnessx-removal-{baseline,equality}); the archive README says these scripts are
reference only and a record is re-run at the commit it names, not with today's package. A unit test
that keeps the archived copy importable contradicted that and would break the next time cli or
learn.cli move. The archive README row now says the test was dropped. test_removed_plane_references.py
(the Req 2.4 name search) is a different test and stays.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer: the whole system moves to a Kubernetes cluster soon. Seven host-bound
assumptions become cluster resources or configuration; the engine becomes a
plane task on a larger pool; a delta Run command serves the pre-push use; every
step is proven on kind first.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Legacy inventory 2, item R1. The vector half of index.py (VectorIndex, VectorRecord, RetrievalHit,
RetrievalPackage, the EmbeddingClient protocol, index_reuse_key, build/write/read/query_vector_index,
build_retrieval_package and their helpers) had no importer in src; `ousast index` wrote index/chunks.json
that nothing read, and was the only reader of `[embeddings]`. index.py keeps CodeChunk and
chunk_text_namespace, which skills.py imports; chunk_file/build_code_chunks go with their only caller.
`[embeddings] model` and `[embeddings] store` join RETIRED_KEYS (RetiredConfigError naming the reason),
the same way the [variants] keys were retired; OPENROUTER_EMBEDDING_MODEL is no longer read, so the
README and .env.example drop it (`ousast learn memory embed --model` names the model). test_index.py
now tests the chunker only; test_config.py covers the two retired keys and no longer asserts the
removed field. The module audit record is regenerated (counts unchanged: 151 modules, 0 orphaned;
the index module's importer list shrinks to skills). RELEASE_NOTES Unreleased carries the line.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
One page, docs/where-we-stand.md, composed with the COBESY composition method (governing idea,
SCQA opener, MECE sections): the M4 gates and the state against them, the detection approach step
by step, ax on the server, token economics before and after, and the open items. Every figure is
quoted from a committed record whose path stands next to it.
decision-engine.md, evaluation.md and token-ergonomics.md link to the page where they repeated its
content (detail tables kept); decision-engine.md's Experiments section now records exp-002's
result instead of saying it has not run. Linked from the nav, the index and the README.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer decision 2026-10-02 (no symlink). AGENTS.md, README, the two spec
docs, the kiro-spec-batch skill and the provenance ignore list are repointed;
the OpenCode plugin package.json stays under .opencode.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reject B. 160 paired units (injection 119, access_control 41; 56 repositories),
both arms, two replicates each, no early stop. Canary agreement A 0.875 -> B
0.906 (CI [-0.03, +0.10], spans; B meets 0.90). Within-pair AUC A 0.815 -> B
0.741 (CI [-0.13, -0.02], against B). AUC -0.02 (against), ADVISORY recall
-0.06 and precision +0.01 (span), usd per candidate A $0.0021 -> B $0.0051
(against). Priced usage A $0.68 / B $1.63; balance $16.72 -> $16.04 at the
run's end (lags). Counts only; task 7.4 marked done.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
An experiment's register, run, result and arm_outcome rows share one object,
and RustFS's S3 Select infers one schema per object: exp-002's result row
carried `units` as an object beside the run row's integer (and `looks` as
objects beside the register row's fractions), and after `analyse --record` no
row of the experiment could be read (JSONParsingError). The result row now
writes `unit_counts` and `looks_taken`; a test asserts one JSON shape per
column across the four row kinds. The live object was repaired by dropping
the one row (drop_rows) and recording the analysis again.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The experiment row in s3://sast-memory now carries units_digest 0de66bfa... at
pin 9213e5f (manifest digest unchanged, f5004fb8...). Counts per family and the
two repairs the freeze needed are in the record; no model.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
160 units (injection 119 in 34 repositories, 76 paired as 38 candidate pairs;
access_control 41 in 24 repositories, 32 paired as 16 pairs), 56 repositories
in all, under the incumbent folds' seed on the 937-example memory (slice 2
evaluated 110 + 37 on the 904-example memory; the memory grew by the recovered
negatives on 2026-10-02). No model. sha256 0de66bfa3ff844825e06f4f4227a14210d1b77f1cc08c0441c6d23e23fa5d91f.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Static-profile feature records carry the nine plane-only features (agree.*,
ms.*, verify.*) as null in x; after cb85542 the 937 live example rows loaded,
but the renderer's validate_x refused every one of them ("features outside the
static allow-list"), so no program could prepare a prompt (found while probing
exp-002 model-free). profile_x drops a null feature outside the row's profile
and keeps a valued one, so a real leak still fails. Test added.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Static-profile feature records carry the plane-only instruments as null; every
one of the 937 live example rows was unreadable (TypeError), which blocked
ousast learn evaluate and freezing exp-002 units.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Labels regenerated (identical sha: the builder's fixed-side rule already
accepted these pairs), feature records built for the 9 corpus pairs the
matcher now resolves (36 rows; the store's ingest index had remembered them
from a build before the declares() gate and skipped them as a repeat after a
later build dropped them -- build_features now checks presence after an index
hit and restores the rows), static memory rebuilt on RustFS: 904 -> 937
examples (+16 fixed-side negatives, +17 positives: 7 advisory Ruby pairs, 9
corpus pairs, 1 population positive), 9 wrong excerpts corrected (Ruby blocks,
a TS default-value brace), no embedding call (33 new + 9 changed excerpts
without a vector, ~$0.0003 to embed). Positives-only groups 98 -> 99: all
data gaps (94 vibe-py same-snapshot fixed sides, 3 dev-php, 2 population
fixes that did not change the function). Leak audit: pairs 376 -> 385; static
no new flag; plane two new injection flags (agree.final, verify.votes at
0.2292 over 48 pairs), v1 profile unchanged, reported.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The brace matcher now finds a declaration head in more forms -- name(...) with
any return type, qualifiers, generics or a parameter list spread over many
lines (a paren-aware scan replaces the 8-line lookahead; a prototype's ';' is
not a body, a ')' closing an outer parenthesis marks a call), JS object-literal
methods and property/chained assignments of function expressions and arrows,
$ as an identifier character, Perl 'sub name {', C# expression bodies -- and a
file given as 'other'/'unknown' is sniffed for Perl and Ruby so their def/sub
forms are matched by the right pattern instead of a {} block inside them. The
memory build hands the matcher the label's language where the record's closed
vocabulary language ('other' for Go, Ruby, C#) resolves nothing. One test per
form. Fixed-side negatives still require the fixed side to declare the
function; the 94 vibe-py positives-only groups stay counted.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Registration rows at pin fbc3220 in s3://sast-memory; in-repo records under
benchmarks/experiments/<id>/registration.json. exp-002's units could not be
frozen: every example row in the store carries null plane-only instruments
and learn.examples.load_examples raises on them (unchanged code since
a90e570) -- a store-side change to diagnose before any paid run.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/experiments.py: YAML manifests under plane/experiments/<id>.yaml,
digest-checked at register, run and analyse time; `experiment`,
`arm_outcome` and `experiment_result` rows; paired arms in a seeded
repository order with a per-unit coin; arm A replay-only at $0; canary
replicates under a fresh cache salt; repository-cluster paired bootstrap,
exact McNemar, two O'Brien-Fleming looks, adopt/reject/equivalent/
inconclusive. `ousast learn experiment register|units|run|analyse`.
Program: `ProgramSpec.sampling = retrieval_ensemble` (k samples at
temperature 0 over k disjoint retrieved example sets; default unchanged);
`Caller.salt`; `retrieve(exclude=)`. verify: OUSAST_VERIFY_CALLERS=off.
Manifests exp-001-known-callers (plane arms, units pending) and
exp-002-retrieval-ensemble (slice-2 incumbents, ceiling $4) -- not run.
18 scripted tests; module audit regenerated (151 modules).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Every README and docs page checked against ousast --help, the code and the committed records. New docs/deployment.md (in the nav): what runs where today, this host's setup (up.sh, doctor, smoke Run, footprint), step-by-step guidance for a separate Kubernetes cluster with implemented vs planned stated per item (no cluster deployment has been made; kube context and doctor's registry check are hard-coded in doctor.py, egress.py, router.py), and what the plane/ manifests need changed. Decision-engine pages state slice 2 (six families, pooled and within-pair AUC, BLOCK offered nowhere, canary below 0.9 for three families, positives-only inflation) and that the engine is not adopted; exp-002 registered, not run, no result quoted. Fixed: model/judge.py reference (now model/pipeline.py), missing-key behaviour, .env reference (OPENROUTER_EMBEDDING_MODEL, store variables), docs extra, plane memory-normalise and ousast index listed, ax README's cluster table made precise. RELEASE_NOTES gets an Unreleased entry.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Counts-only record of every repository group of the static memory that holds
positives and no negatives (98), traced label -> feature record -> excerpt:
94 are vibe-py pairs whose fixed side is another function of the same
snapshot (data gap, counted), 3 dev-php recipes without a fixed pin, 1
population-v2 fix that did not change the function. Separately, 16 pairs are
absent from the memory on both sides for tooling reasons: 7 advisory Ruby
pairs matched under the feature vocabulary's 'other' language, 9 corpus pairs
whose declaration form the excerpt matcher misses (JS object-literal methods,
property-assignment arrows, a $ identifier, a C parameter list past the
8-line lookahead, Perl sub).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Decision-engine task 6.7 (partial). After the advisory-fix harvest (memory 724 -> 904 static examples),
one compile per evaluable family and an outer grouped 5-fold evaluation of that family's own candidates,
input profile v1, priors off, k = 5, cross-fitted Platt, nested operating points (repository-cluster
bootstrap 95%):
family n AUC(s) ADVISORY R / P 'vulnerable' P / R ECE canary
injection 110 0.77 [0.67, 0.88] 0.88 / 0.57 0.84 / 0.36 0.093 0.77
untrusted_dest. 62 0.85 [0.76, 0.94] 0.84 / 0.73 0.96 / 0.66 0.071 0.87
path 35 0.89 [0.79, 0.97] 0.86 / 0.69 0.89 / 0.76 0.119 0.90
output_encoding 44 0.91 [0.82, 0.97] 0.86 / 0.81 0.94 / 0.55 0.045 0.90
deserialization 46 0.90 [0.82, 0.97] 0.83 / 0.76 0.93 / 0.83 0.095 0.93
access_control 37 0.96 [0.90, 1.00] 0.91 / 0.91 1.00 / 0.61 0.076 0.83
BLOCK offered for none: calibration fails everywhere except output_encoding, where no fold reaches a
Wilson lower bound of 0.95. Within paired groups (a function and its own fix) AUC is lower, 0.72-0.93:
every single-label group in the new families holds positives only. Injection learning-curve point,
paired (same program, candidates, folds): memory 724 -> 904, AUC 0.76 -> 0.77 (no injection example
was added). $2.88 metered DeepSeek (ceiling $8; balance 17.73 -> 16.73), $0.0009 embeddings.
Instrument fix: compiled demonstrations now follow the near-duplicate rule (cos >= 0.98) that retrieval
and the pre-call assertion apply; the assertion had stopped the untrusted_destination evaluation.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
One page, two halves with a Mermaid diagram each. A: the plane's token economy (one Model per Task with a metered usd/calls budget, attribution table, credentials only in the start request, repo-facts and content-addressed reuse, replay at $0, triage-hunt-tie-break order, prefix-cache prompt order) with every cost quoted from plane-increment-1/2.json, the injection-slice and harvest records, and an honest not-yet list (no token budget in ax, triage applied not run, k and contrast fixed outside experiments). B: the development-side measurement (tool I/O 85.5%, previously 90%) and the benchmarks/dev remedies: read_guard hook, bulk_read portal, token_report, plus the practices. Linked from the nav, index, memory-and-detection and the README docs list.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The plane's memory store no longer names one vendor. `MinioStore`/`SdkClient`/
`minio_settings`/`MINIO_KEYS` become `S3Store`/`S3Client`/`s3_settings`/`S3_KEYS`;
the URL scheme is `s3://<bucket>[/<prefix>]` (`s3://` alone uses `S3_BUCKET`); the
settings are `S3_ENDPOINT` (a full URL), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`,
optional `AWS_SESSION_TOKEN` and `S3_REGION`. The client is boto3 (extra `s3`,
imported lazily): SigV4, path-style addressing, SDK checksum trailers off, paginated
listing, versions and delete markers listed and deleted by version (`purge`, used by
the contract tests' teardown so the bucket keeps nothing of a run).
Behaviour is unchanged: the store never configures its bucket and verifies
versioning, the `runs/` expiry rule, S3 Select and tag reads at startup; no local
fallback; every row carries every queryable field with "" placeholders. The admin
hints read `--endpoint-url "$S3_ENDPOINT"`. The fake client still reproduces RustFS's
two Select failures (unbound leading-row field, null-typed column).
Docs: rustfs.md stays the tested server's setup page (policy JSON unchanged, env
names updated, any S3-compatible store with versioning, lifecycle, tags and Select
works); memory.md, ops/ax/README.md, README.md, steering and the two current spec
designs use the S3 naming. Retired specs and committed measurements are untouched.
Real server (RustFS, OUSAST_MEMORY_TEST_S3=1): test_plane_memory + test_learn_compile
75 passed, 1 skipped. Store copy of the live local store to a temporary prefix: every
count matched (776 row objects, 2982 rows, 22 facts, 788 excerpts, 2601 responses,
2 programs, 1793 index entries), 4190 versions purged afterwards.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The 97 advisory-fix pairs (194 sides) harvested for the decision engine: verify a/b + agree on the 34
untrusted_destination sides and model roles on every side as plane Runs on ax ($0.38 metered, task
budgets $1.49, ceiling $5; DeepSeek balance 17.88 -> 17.73), engine features on the host (frozen
7b17eac, one container, --memory 3g: 54 ran, 58 asked no question, 82 no language model), feature
records for all 194 sides, static memory rebuilt with the merged harvest units: 724 -> 904 examples.
No unit was skipped for the 20 KB inline limit.
Instrument fix: SideReader read only the first pair-kind source, so the first rebuild left all 194
advisory sides `unread` (0 examples). It now reads every pair-kind source; test added.
Leak audit: the static profile flags only engine signals and facts.function_lines (withheld by v1);
the plane profile newly flags verify.flag_b and ms.operation (plane-only, not in any v1 prompt).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Read-only check at the pinned vulnerable commits via the GitHub API
(LICENSE, license.txt, plugin header License: line). The five license
fields now record the SPDX id with the evidence; nothing else in the
frozen population changes. protocol-v2.md notes the verification and
that the OpenCVE case in v1 is BSL-1.1 (not OSI; non-production use
permitted by the Additional Use Grant; change date 2030-08-14 to
Apache-2.0).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Comment-only changes outside learn/, plane/ and cli.py:
- model/pipeline.py: the `judge` its docstring contrasts with is the deleted
model/judge.py, now named as such.
- fusion.py: the panels are deterministic and only review and vote; the
"critique and revise" step belonged to the retired LLM panels.
- mcp.py: hunter tools stay inside the scan's stages, not "harness stages".
- fusion, mcp, gate, redaction, skills: their "Phase N" tags now say which
plan they cite (the retired openrouter-sast-harness).
Requirement citations to retired specs stay: they explain where the code
came from. No behaviour change; the byte-identical benchmark golden passes.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
- Root .dockerignore for the root Dockerfile's context: local state (.git,
.claude worktrees, .venv, .env, caches, reports, *.sarif) and the
benchmarks/measurements archive (~20 MB of evidence). Kept: everything the
Dockerfile COPYs, the two 2026-09-13 records ops/smoke_report.py reads
from /app inside the image, and the harnessx-removal baseline pairs.json
that learn/sources.toml names as a label source. plane/Dockerfile.runner
keeps its own ignore file.
- Remove 2026-09-30-harnessx-removal-baseline/scan/report.sarif: tracked
despite .gitignore's *.sarif, with no counterpart on the candidate side.
The directory's README says why.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A4: roadmap-2026-09-04.md moves to .kiro/archive/steering/ so it no longer
loads as steering (roadmap.md and safety-net.md replaced it); roadmap.md's
pointer follows it. overview.md's spec table is recounted from tasks.md and
spec.json (adds ai-service-plane, harnessx-removal, learned-decision-engine,
pre-push-safety-net; drops the stale "active one"), and the module line now
cites the regenerated audit: 150 modules, 145 load-bearing, 5 standalone,
0 orphaned; 1,964 tests collected.
A5: a dated status note at the top of the requirements (or brief) of 16
specs: retired, superseded, dormant or complete, and for
model-grounded-detection the removal of judge/calibrate/execution (closing
the Req 10 execution-tier option). spec.json phase corrected to
implementation-complete where every task is checked (model-grounded-
detection, propose-adjudicate-prove, tree-sitter-overlay-extra,
corpus-seeded-mechanisms, pair-corpus-honesty); openrouter-sast-harness gets
a spec.json with status retired. Nothing deleted, no spec moved.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Material for MkDocs site (mkdocs.yml, optional extra `docs`) with Mermaid via
the superfences custom fence. New pages: overview, scanning (modes, quick
rules, Joern engine, priors, pre-push delta), the plane on ax (one task as a
sequence, golden snapshot, Run DAGs), memory (row kinds, write/read paths,
FileStore vs S3, verify at startup), memory and detection, and RustFS setup
(admin steps, complete agent policy, .env, refusal messages, Select limit).
The ops READMEs are included at build time; evaluation gains the population
lifecycle and the fold layout. tests/test_docs_site.py runs the strict build
when mkdocs is importable and checks Mermaid blocks and the RustFS page.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A1-A3 of the legacy inventory, moved with git mv to benchmarks/archive/:
the stage-2 verifier chain (verify_batched, score_batched, verify_sinks,
model_sinks, sink_candidates), the v2 tool hunter (hunt.py) and the
harnessx-removal equality proof. The plane's verify/agree/roles tasks
replaced the first six; the harnessx-removal spec is closed.
- The six independent scripts keep HERE = benchmarks/independent (population,
evaluate.py, results) and add their own directory for archived siblings.
- harnessx_removal_equality's REPO root is one level deeper.
- tests/test_plane_scoring_parity.py loads score_batched from the archive
(ran here, not skipped); test_harnessx_removal_equality points at the new path.
- benchmarks/archive/README.md: what each script was, which committed
records cite it, the commit that last ran it. paths.md maps old -> new;
committed records are not rewritten.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
RustFS's S3 Select infers an object's schema from its first 1000 rows: a where
field absent there is unbound (EvaluatorBindingDoesNotExist), and a column null
in all of them but set later fails the whole object (JSONParsingError, measured:
999 leading nulls answer, 1000 fail). plane/memory.py now declares per kind the
fields a where may name (QUERY_FIELDS: facts candidates_digest, verdict
candidate/final, example profile, plus the ROW_FIELDS every row has) and writes
their union on every row at write time as strings, "" where one does not apply
(null measured to break). rows() refuses an unknown kind, an undeclared where
field and where = null before any read. `ousast plane memory-normalise
[--store URL] [--dry-run]` rewrites pre-rule rows once, idempotently, with
counts. The fake object client mimics both RustFS failures.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Nothing in src reads config.dynamic, config.evidence or config.variants, and
[models] patcher was only echoed into harness.json. They now go through the
existing RETIRED_KEYS mechanism and fail with RetiredConfigError naming why
("retired 2026-10-02 because nothing ever read it: ..."); RETIRED_KEYS now
carries the full reason per key so the HarnessX entries keep their wording.
- DynamicConfig, EvidenceConfig, VariantsConfig and their loaders removed;
ModelConfig.patcher removed. resolved_config.json loses those sections.
- harness.json keeps its shape: model_roles.patcher is always null.
- RELEASE_NOTES "Unreleased" lists the keys and the output shape change.
- Delete .opencode/skills/openultrasast-kiro-impl: it pointed agents at the
retired openrouter-sast-harness tasks.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer decisions 2026-10-02: "it has the featurs. fallback not desired" and
"Admin sets it once; agent verifies".
- MinioStore never configures its bucket: configure()/enable_versioning()/
expire() are gone; the client protocol only reads versioning and lifecycle.
- verify_bucket() runs when a MinioStore is opened (once per process per
store): versioning Enabled, an enabled lifecycle rule expiring <prefix>/runs/
(none expiring the whole store), S3 Select answering a probe, object tags
readable. Every missing piece is listed in one MemoryStoreError with the
aws-cli admin command or the permission to grant.
- No S3 Select fallback: a server without Select is refused at open, and a
failing Select on one object raises (naming RustFS's leading-row schema
inference when the field is unbound) instead of filtering locally.
- MINIO_REGION is passed through when set.
- Tests: fakes for versioning off/unreadable, lifecycle missing/disabled/
tag-narrowed/whole-store, Select missing, tags unreadable, and the pass case;
the real-server contract uses OUSAST_MEMORY_TEST_BUCKET (else MINIO_BUCKET)
and never writes bucket configuration.
- ops/ax/README.md: RustFS support, the admin one-time setup, the agent policy.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The reachability-first audit (previous commit) found these five with no
import path from any entry point:
- model/judge.py: replaced by model/pipeline.py, which never imported it.
- model/execution.py: the MGD Req 10 execution seam, never built or called.
- model/calibrate.py: no CLI, no importer; learn/calibrate.py is the live one.
- stage_processors.py + slot_contract.py: kept only so test_gate could pin
them; the scan runs findings.quick_scan_findings directly.
Their tests go with them, test_gate's zero-dependency list drops the two
entries, the audit lists drop their rows, and the manifest is regenerated:
150 modules, 145 load_bearing, 5 standalone, 0 orphaned.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
model/audit.py classified anything named in SPEC_OWNED or STANDALONE before
looking at the import graph, so five modules nothing imports were reported
load_bearing and the committed manifest said 0 orphans.
- Reachability from the gates, the CLI and every module with a __main__ block
(python -m, the plane runner) now decides the class; the lists only supply
reason text, and an unreachable listed module is orphaned with its listing.
- `from .pkg import sub` now records the submodule edge (plane.engine_alerts
was only reachable by its SPEC_OWNED entry).
- `python -m openultrasast.model.audit` regenerates the manifest.
- Manifest regenerated: 145 load_bearing, 5 standalone, 5 orphaned
(model.calibrate, model.execution, model.judge, slot_contract,
stage_processors). The zero-orphan test is strict-xfail until they go.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
6.2: the 724 static examples' 616 distinct excerpts embedded with openai/text-embedding-3-small via
OpenRouter, 198,819 tokens, $0.0040 metered (ceiling $1).
6.6, family injection only, memory = every family (at most 2 contrast examples per prompt), input
profile v1: smoke fixed k = 5 (repeat prefix-cache hit share 0.96); one compile (program c1237a49,
proposed instruction 5, balanced Brier 0.128 on 13 C_val candidates vs 0.199 baseline); outer grouped
5-fold, 110 out-of-repository candidates in 29 groups: ADVISORY recall 0.90 [0.82, 0.97], precision
0.54 [0.44, 0.65], AUC 0.74; BLOCK not offered (ECE 0.064 > 0.05; no fold reached a Wilson lower bound
of 0.95); canary re-ask agreement 0.83. $0.91 metered at list price; DeepSeek balance $18.20 -> $17.90.
Instrument fixes found on the way, each with a test:
- memory build: pair sides join the record stored under their harvest unit pin (--units), and only the
built profile's records are read (static and plane shared a key; 85 -> 724 examples);
- retrieval: stage 1 per side (family / contrast) and at most 2 contrast examples, never padded;
- program: the JSON answer contract is part of the signature, appended to proposed instructions
(DeepSeek JSON mode refused a prompt without "json");
- compile: candidate_families for a one-family compile; compile.toml k = 5 from the smoke rule.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The leak audit found facts.function_lines separating fixed from vulnerable pair
sides (a fix guard lengthens the function: 0.80 untrusted_destination, 0.75
config_secrets). Input profiles may now name single features; v1 withholds the
engine and the function length from prompts, retrieval and the proposer summary.
A withheld feature keeps its instrument block in the distance (state still counts).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer decision "Neutralise both for v1".
1. Phantom candidates. A pair side's candidates are now only the functions that
side declares (plane/harvest.py `declares`, the excerpt builder's
function_bounds); a labelled function absent from a side is listed under
`absent_on_side` and never asked, and a side declaring none of its verify
family's functions is no verify unit. The recorded harvest units held 58
phantom verify candidates (53 fixed, 5 vulnerable sides). build_features.py
skips label rows whose side does not declare the function and drops their
earlier rows (MemoryStore.drop_rows); rebuilt the pair records from the
recorded signals, no model call: 126 label rows absent_on_side (117 fixed,
9 vulnerable) -- access_control 13, config_secrets 4, deserialization 20,
injection 28, memory 4, output_encoding 16, path 15, untrusted_destination
26 -- so 252 feature rows dropped (1712 -> 1460); no other row changed.
2. Engine withheld. Input profiles in learn/schema.py (`v1` withholds the
engine instrument, `full` withholds nothing); ProgramSpec.inputs defaults to
v1 and compile.toml records it. Neither the engine's state nor its values
reach the candidate, examples, demonstrations or the instruction proposer's
error summary, and the Gower retrieval distance leaves the engine block out.
Stored records are unchanged. design.md Risks records the decision and that
the engine returns only through a pre-registered A/B on assumed-benign and
benign-control negatives.
3. `ousast learn audit-leaks` (learn/leaks.py): per feature and per instrument
state, separation = (pairs the signal alone puts on the right side - on the
wrong side) / pairs, over all pairs and per family; flagged at >= 0.20 over
>= 10 pairs. Counts under
benchmarks/measurements/2026-10-01-decision-engine-leak-audit/ (before and
after the rebuild). After: facts.function_lines (0.80 untrusted_destination,
0.75 config_secrets, 0.26 injection; higher on fixed), verify.turns_mean
0.67 config_secrets, eng.* 0.35-0.40 output_encoding, verify.flag_a/votes/
site_offset and agree.final 0.20-0.30 on 10-12 pair families. Before the
rebuild the phantom fixed sides drove state.source/state.roles/ms.flagged
to 1.0 and state.engine to 0.95 on deserialization.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The pair-corpus engine job finished: 682 of 686 sides (199 ran, 475 read their input
and asked no question, 8 no engine model; the 4 sides of the two family-unknown
pairs were not run). Feature records rebuilt (1712); counts-only record updated.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
`ousast plane harvest` (plane/harvest.py) writes two Runs over a label snapshot:
verify a/b + agree on the verify-family units (both sides of each labelled pair as
an opaque files-only Workspace, candidates outside the hunt's root, no declared
sites; v1 and dev-php pins with repo-facts first) and model roles on the candidate
files (pair excerpts grouped, one Task per population pin). Budgets sum under the
ceiling by construction. New roles task template; the generator re-pins it.
Spent by attribution: verify $2.54 (1961 calls, 150 units, 182 hunts per pass),
roles $1.40 (1034 calls, 799 files, 0 unclassified chunks); DeepSeek balance
19.63 -> 18.23. Engine on the pair corpus (benchmarks/learn/engine_pairs.py,
frozen e6ed785, one 3g container) is PARTIAL: 384 of 686 sides at commit, still
running; unfinished sides stay missing. benchmarks/learn/build_features.py built
1712 static+plane records into the memory store; counts-only record under
benchmarks/measurements/2026-10-01-decision-engine-harvest/.
source/roles parts are `none`, not `failed`, for a language without a declaration
pattern (C). ops/ax/README: Substrate golden-actor disk leak and the inline
Workspace limit (~20 KB).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
97 advisory-fix pairs, one repository each (309 advisories examined, 212
rejected with reasons in research-2026-10-01.md): a public fix commit
distinct from its parent, the advisory-named function declared on both
sides with a changed body. Pointer pairs: materialize.py reads one file
per side from a shallow blobless fetch into the pair cache and verifies
them with the label builder's matcher (verification.json, bytes read).
learn/sources.toml lists the catalog as the advisory-fixes source. Label
snapshot: negative repository groups untrusted_destination 8->25, path
3->21, output_encoding 1->22, deserialization 0->21, access_control 0->20;
all five reach the per-family floor. Five pairs on reserved repositories
were dropped after the reserved-population guard flagged them.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A pair now gives a fixed-side negative only when the fixed side's source declares the
labelled function (the excerpt builder's matcher; a name on a code line where the matcher
resolves neither side), or when a deterministic rename at a distinct fix commit replaces it
(one new function, same parameter list, changed body: provenance fixed_side_moved). Anything
else is counted under pair_fixed_side.absent_fixed_side, not labelled.
108 pair labels lose their negative: 80 seeded pointer pairs whose fixed side is a guard
helper of the same commit, and 28 Real-Vuln-Benchmark traps whose fixed side is a sibling
handler of the same snapshot (4 of those handlers are labelled positives themselves). The
memory build no longer substitutes the catalog's fix_function on the fixed side. The pair
cache is an allowed label-source tree so pointer sides are read through the guard.
Snapshot: benchmarks/measurements/2026-10-01-decision-engine-labels (counts only).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/compile.toml: the pre-registered spec -- balanced Brier score of the
raw score on C_val (cells by family x label, repositories capped at 5,
unsure = 0.5, ties within 0.005 to fewer prompt tokens), seed, the
compile split (25%, boot 60/40, compile-split-only memory), bootstrap
(150 candidates, confidence >= 0.7, 4 sets of 3), search (6 proposals +
baseline, 20 errors shown, 60 validation candidates), k = 1, the
retrieval settings, and the $3 / 1,200-call ceiling.
learn/compile.py: bootstrap few-shot keeps only right, confident
rationales that cite a line and name no repository, path or CVE/GHSA id;
seeded demo sets (positive, negative, any; families mixed) with
other-framework alternates; one proposal call; 7 instructions scored with
set 1, the best with sets 2-4. The artifact is canonical JSON stored as
programs/<program_id>.json, program_id = sha256 of the artifact without
its id; it records the spec sha, memory/compile-groups/folds digests and
the response-cache digest, with spend priced from cached usage, so a
rerun from the cache is byte-identical at $0 (tested on FileStore and
MinioStore through the fake client; real MinIO gated as today). A ceiling
ends the compile `unfinished` with no artifact. packageable() keeps only
permissively licensed demonstrations (MIT, BSD, Apache-2.0, ISC, zlib),
swapping in a permissive alternate or dropping, under a new program id.
learn/calibrate.py: Platt scaling on logit(s), pure Python Newton, per
family / pooled with family intercepts / insufficient data by the floors;
cross-fitted (fold j's map is fitted on the other folds only); the check
(5-bin reliability, slope and intercept with repository-bootstrap CIs,
ECE <= 0.05), ADVISORY only when it fails.
learn/evaluate.py: predictions under each fold's boundary, families below
the floor never asked; operating points read from the calibrated,
cross-fitted predictions of the other folds (BLOCK: Wilson lower bound
>= 0.95, never on a majority unsure; ADVISORY: recall >= 0.90);
repository-cluster bootstrap intervals; strata with "insufficient data";
>5% parse failures is an instrument failure; explanations keep only
signals the record holds; version mismatch names the fallback reason;
the memory-size learning curve.
`ousast learn compile|evaluate|curve`: paid runs need --budget-usd (exit
3 at the ceiling), --replay-only runs from the cache. Nothing was spent:
every test and the dry run used scripted clients.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/program.py: the declared signature (code, signals, roles, family in;
verdict, family, confidence, rationale with cited lines out), the
baseline instruction, rendering as [system: instruction + demonstrations]
-- byte-identical for every candidate, so the provider's prefix cache can
hit -- then [user: retrieved examples with their labels, the candidate
without one]. Signals pass validate_x first (no EXCLUDED_FIELDS, missing
instruments in words). Answers are one JSON object; one that does not
parse or cites no line inside the excerpt is asked once more at
temperature 0 and otherwise becomes unsure/parse_failed, never
not_vulnerable. Classify(k): sample 1 at 0.0, the rest at 0.7; score s =
mean p_j; majority verdict, ties unsure; a majority unsure cannot BLOCK.
Caller caches each response as responses/<sha>.json keyed by (model,
Model parameters digest, messages, temperature, sample, json_object) and
replays at $0; spend is priced from the recorded usage once per distinct
request, so a replay reports the same usd. Program.decide asserts the
evaluation boundary on the rendered ids before any call.
MeteredClient.complete passes temperature/logprobs through (only when
set); DeepSeekChatClient sends temperature beside thinking=disabled.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/retrieve.py: stage 1 is a per-instrument Gower distance over the
profile's allow-listed features with explicit missing (both ran: mean
feature distance; differing states 1; same non-ran state 0; null in one 1;
priors dropped with priors=off), n1 = 40 with seeded tie-breaks; stage 2
re-ranks by r = lam(1 - D) + (1 - lam)cos, signals_only without the
target's vector; the balance rule takes k_ret = 6 (<= k/2 per label, <= 2
per group, <= 2 other families; `short` recorded). eligible() is the only
way into a prompt: not the target's group, not an evaluated group, not the
held-out source or framework, and outside deployment not a near-duplicate
(cos >= 0.98). assert_boundary repeats the check on rendered ids;
demonstrations of a held-out framework swap to their alternates;
deployment_fold excludes a scanned repository that is a corpus group;
seed_memory never hands a case its own group.
learn/folds.py: compile split (25% of groups per primary family, 60/40
boot/val), outer grouped 5-fold stratified by family and label mix,
leave-one-source-out, leave-one-framework-out (insufficient data below 10
groups), assert_disjoint and a folds digest; all seeded.
Tests: test_learn_retrieve.py (hand-computed Gower, lam 0/0.5, balance,
near-duplicates, test_boundary over outer/LOSO/LOFO folds with
demonstrations), test_learn_folds.py.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
plane/models/openrouter-embedding.yaml: openai/text-embedding-3-small via
OpenRouter (embeddings only), input_per_m 0.02, cache/output 0, egress
openrouter.ai; `openrouter` joins the provider extensions.
plane/budget.py MeteredEmbeddingClient: prompt_tokens priced from the
Model manifest, usd/calls ceilings stop the next call, a priced call that
reports no tokens raises UnmeteredUsage (a silent zero is an instrument
failure). OpenRouterEmbeddingClient keeps one usage row per call
(parse_embedding_payload); its api key is out of repr.
learn/embeddings.py + `ousast learn memory embed`: excerpts embedded in
batches into embeddings/<model-slug>/<excerpt_sha>.json (float32 base64),
cached by content hash so a rerun makes no call; the declared dimension is
checked; --dry-run estimates tokens and cost without a call; a paid run
needs --budget-usd and exits 3 (unfinished) at its ceiling. Not run against
the provider: the store embedding waits for its budget go-ahead.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/excerpt.py: the labelled function at its pin as numbered lines, bounded
(80 lines / 4,000 chars for candidates, 40 / 2,000 for memory examples),
centred on focus lines; advisory ids anywhere and comments with security
wording (labels.security_reason, multilingual) become <redacted>; owner,
owner/name, path and commit ids are replaced; a delta unit appends the
base->head diff (<= 40 lines). C/C++ and other brace languages without a
repo-facts declaration pattern use a brace matcher (346 c_cpp pair rows
had no excerpt without it).
learn/examples.py + `ousast learn memory build`: label + feature record +
excerpt -> `example` rows (no rationale), excerpts as content-addressed
blobs; refuses origin: user and sources outside sources.toml (exit 2);
conditional labels skipped; exit 2 when >10% of a source's recorded
candidates have no excerpt. A fixed side that was read but no longer
declares the labelled function is counted apart (absent_fixed_side): a
label defect, not an unread clone.
plane/memory.py: `example` in KINDS; put_blob/get_blob/blob_names on the
MemoryStore base, so FileStore and MinioStore share them (tested on both,
MinIO through the fake object client).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
In-house DSPy-style program (retrieve -> classify(k)) on the metered client;
example store with signal-profile Gower neighbours re-ranked by code embeddings;
balanced Brier metric, bootstrap few-shot and instruction search on a compile
split; Platt calibration on other folds; cached replay; tasks 6.1-6.7 with paid
steps flagged for go-ahead.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer: data will not scale; the decision engine is an AI classifier, not ML,
with local memory, DSPy style. In-house on the metered client; retrieval by signal
profile re-ranked by code embeddings; compiled (bootstrap few-shot, instruction
search) per training fold; retrieval never crosses the evaluation boundary;
confidence calibrated on held-out repositories. Design and tasks 6+ to revise.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/sources.toml is the fail-closed source list: every read goes through one guard
that refuses an unlisted path or an [[excluded]] file name before opening it, and a
source whose evaluation record is missing cannot label. Population v3 is named only
as excluded, by file name, and is never opened (tested with a v3-named fixture and
open/read_text tracked).
learn/labels.py and `ousast learn labels` apply the design's rules: v2 declared
sites and v1 fix-range functions positive, the same function on the fixed pin
negative only when the fix changed it, introduce/repair delta units, benign control
pushes as conditional negatives; pairs vulnerable/fixed (unscorable and title tier
out, title as a sensitivity arm); dev-php recipe known functions; recorded
adjudications with their reasons. v2 rows are labels marked design_informed, as the
design says. Repository groups: normalised owner/name, merged by shared CVE/GHSA,
shared fix commits and [[alias]] rows.
assumed_benign (maintainer decision): ordinary commits of the population
repositories, >= 90 days from any fix, no later security commit touching their
files, weight 0.5, conditional, reported separately and never merged with verified
negatives. The security filter is multilingual (CJK, Korean, Russian, European and
other languages) plus CVE/GHSA/CWE references and dependency bumps; widened after
reading accepted pushes (validation, passwords, guards).
First snapshot, counts only (benchmarks/measurements/2026-09-30-decision-engine-
labels/labels.json): verified pin labels in positive/negative repository groups
injection 44/41, untrusted_destination 35/34, deserialization 20/20 (per-family),
output_encoding 15/15, path 15/14, access_control 12/12 (pooled), config_secrets 5/5,
memory 5/5 (insufficient). assumed_benign: 969 pushes in 27 groups from 36,000
commits scanned.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Wrapper inference (learn/roles.py, no model): a function whose parameter reaches a
call already in the role set becomes a derived sink (or a sanitizer when it returns
the cleaned value) of the same operation, at depth + 1; a function returning a
source's value becomes a derived source. The role set starts from language-level
entries only (priors off by default, optional), grows to depth 3, and reads
vendored dependency source when present, so a vendored database class reaching
mysqli makes `->query` a sink without a framework table. A name the repository
declares more than once becomes a role only when every definition derives it.
Model roles (plane/tasks/roles.py): the model_sinks.py classification moved into
the package and extended to sink/source/sanitizer/guard/dispatch, per file chunk,
metered and resumable, client injected; a failed chunk is unclassified, never
"no roles". vocabulary_overlay merges inferred and model roles over the
language-level facts in the shape taint_specs takes, origin in the entry id.
Features: the roles (static) and model_sinks (plane) instruments are filled.
On the recorded validation-46 facts (15 cases, 14,368 product files, 87 MB read):
1,999 wrapper roles (1,753 sinks, 157 sources, 89 sanitizers; depth 1/2/3 =
973/651/375), 14 of 15 cases with at least one; the WordPress gopay plugin has
none with priors off, which is what model roles are for.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ruleset/frameworks.toml lists 19 framework and library ids (wordpress, flask, django, express,
spring, jakarta-servlet, react, vue and 11 libraries), each with its languages, dependency
package names and a symbol lexicon used only by the lint and for stratification.
Every framework/library entry of section 2 is tagged `framework`/`library`, and mixed entries
are split so a tag covers a whole entry (split pieces keep the entry's id and flags, in place):
semantic facts 38 tagged entries (wordpress 6, django 6, flask 4, express 4, jakarta-servlet 3,
pyyaml 2, one each for spring, sqlalchemy, requests, httpx, lodash, node-http-clients,
escape-html, sanitize-html, dompurify, esapi, commons-text, react, vue); the dispatch fact
behind the hook-as-source path is wordpress; the PHP layout row is renamed php_layout and stays
untagged (a product filter). Quick rules: 9 tagged (wordpress 6, flask, pyyaml,
node-http-clients); php-unserialize is split into php-unserialize and php-maybe-unserialize;
the lint also caught WordPress escapers in exclusions and axios/yaml.load in mixed rules, so
those keep their ids as tagged rules and gain a `fallback_for` language-level rule (6) that is
active only when the prior is off.
load_facts and load_ruleset take priors ("all" by default -- today's scan --, "off", or a set
of ids). tests/test_semantic_priors.py: the lint (no untagged entry contains a lexicon symbol
of its language), "off" removes exactly the tagged entries and activates the fallbacks, split
entries keep their language-level calls. Rule and layout counts in three existing tests follow the split and the rename.
Unchanged defaults: test_benchmark_output_is_byte_identical_to_golden_baseline passes, and the
quick findings of all 15 benchmark manifests (full StaticFinding records) are byte-identical
before and after. priors=off on the same manifests: 61 -> 57 findings; the 4 lost are js-ssrf
(axios), php-wpdb-sql-composition, python-ssti and python yaml.load; 5 PHP and 1 Python finding
move to their fallback rule at the same location. Semantic calls with priors off: php 75/92,
python 32/77, javascript 42/70, java 24/41, c 27/27 (the engine-side effect needs Joern and was
not measured).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/schema.py is the closed, typed allow-list of candidate features: 66 features over
11 instruments (language one-hot, quick rules with 23 mechanism buckets and a prior count,
engine, repo-facts, source span, entry-point distance, roles, delta, verify, agree, model
sinks), two profiles (static 57; plane adds verify, agree and model sinks for 66), schema_version and a feature-set digest.
No repository, path, file, function, rule or commit identity is a feature; site_match,
in_fix_range, fixed-pin alerts, case.json fields and costs are refused. Missing is not zero:
each record carries per-instrument state (ran | none | failed | not_applicable), and every
feature of an instrument that did not run is null.
learn/features.py builds records on the host (build_for_scan, whole repository or base..head
delta); plane/tasks/features.py builds them per case in a Run from the delivered artifacts,
dropping site_match and in_fix_range on read and ignoring fixed-pin alerts. The generator
adds a model-free `features` entry per case before `remember`, which stores the records as
`features` rows (re-validated). memory KINDS gains features, label, decision, experiment,
arm_outcome and experiment_result. The committed validation-46 manifests are regenerated
(entries appended; the first Run's entries are unchanged). The runner image must be rebuilt
and re-pinned before a Run executes the new task.
Proof on real artifacts (scratch store, not committed): the recorded validation-46 run gives
75 feature records over 15 cases (43 verify candidates plus vulnerable-pin alerts), each
case's vulnerable pin exported and read (0.5k-9k files); remember stored 75 features rows
beside 43 verdict, 84 alert, 61 unit_cost, 15 facts and 8 coverage rows.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Three instruments scored separately at one analyzer commit (quick PHP rules
with a fixed rule-to-family table, the Joern engine, the plane pipeline
triage -> verify a/b -> tie-break -> 2-of-3), protocol v2's six pins, one
matching rule for all six families (declared site or fix range, same family;
evaluate.matches_v2 unchanged), per-family meaning of a match, the M4 gates
copied from protocol v2 and applied per instrument, a $60 total / $6 per case
model-spend ceiling, and the retune rule.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
freeze.py, extended for v3 and still reading only git history, file lists and
fix diffs:
- a file the advisory names without a function resolves to the function(s)
the fix's diff changes at the vulnerable pin (maintainer's decision 2),
recorded with its basis (advisory-named / resolved-from-fix-diff); a new
helper inserted at file scope adds no site;
- an inferred fix must change a declared site function, else the case drops;
- a single-language population gets a strict benign proposal (PHP change,
merges judged with all they bring in, pre-vulnerable history as fallback).
All 15 cases verified, none dropped (pagible and podlove fixes confirmed).
Nine sites in eight cases resolved from diffs; five benign proposals
replaced by hand.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Assumed-benign pushes mined from non-security commits as a separate label source
(maintainer decision). Ten tasks in the design's commit order; the harvest,
exp-001 and the v3 check wait for budget go-ahead.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer: hard-coded application specifications are problematic; WordPress is
not the only PHP application, same for Python and JS. Language-level knowledge
stays; framework roles are inferred per repository; existing framework entries
become tagged optional priors kept only by A/B evidence; leave-one-framework-out
evaluation.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer's selection 2026-09-30, "Recommended 16, minus v2 mentions": the
research's rule-based 16 with Winter, elFinder and Easy!Appointments replaced
by the next candidate in their family by the same rule (named fix first, then
newest first), v2-mentioned alternates (Grav, Backpack CRUD, Koel) skipped.
Winter -> mantisbt-tokenizer-eval-rce; elFinder -> pagible (inferred fix, to
be confirmed at freeze); Easy!Appointments -> empty slot (no remaining
untrusted_destination candidate), so 15 cases are reserved.
The reservation guard now covers v3: all six families, a declared-language
population, status "reserved".
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer: the tool will never run against the test cases it is trained with;
its purpose is a repo safety net. Repository-grouped folds, repository-agnostic
features, BLOCK/ADVISORY operating points for changed code, out-of-repo
calibration, adaptation kept separate.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer direction: A/B testing to extrapolate detection instead of hard-coding
it, and a trained decision engine. Instruments become signal producers; labels
from ground truth with provenance; a simple calibrated engine with population
folds; experiments as plane Runs; extrapolation with intervals, checked once on
frozen v3; the engine is what scan/pre-push/plane report.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Quick mode had no PHP rules, so a PHP quick scan read the files and reported a plausible zero.
ruleset/php/rules.toml adds 13 line/statement rules built on semantic/php.toml's sinks, sources and
sanitizers (sql/wpdb, command, code_eval, unserialize, echo, file_inclusion, filesystem,
outbound_request, redirect). 9 are enabled and 4 are shadow (sql-call-composition, code-eval,
unserialize, xss-echo-input). A rule is shadow when it fired on the corpus but never at a known site.
precision_estimate is the measured lower bound.
Measured in-sample on the development corpus only (pmpro introduce/repair/benign, population-v1 PHP
cases; population-v2 was not read). Known site hit on the vulnerable pin in 4 of 5 cases (LearnPress
missed). The hit disappears on the fixed pin for pmpro and YesWiki. Numbers, per-pin files/bytes read
and the engine comparison are in benchmarks/measurements/2026-09-30-php-quick-rules/.
- fixtures php-vulnerable / php-benign plus manifests: ousast benchmark covers PHP
(9/14 reported, 13/14 including shadow; the benign twin records 2 known false alerts)
- PHP `#` comments are comments; `#[...]` attributes are not
- the alerts task now reports PHP as quick coverage; alerts-engine keeps running the engine for PHP
next to the quick rules (ENGINE_ALONGSIDE_QUICK)
- other languages' benchmark findings and metrics are unchanged
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Quick mode has rules only for c/cpp, groovy, java, javascript/typescript and
python, so every PHP case's "0 alerts" read as clean. The `alerts` task now
records per language of the files read whether quick mode covers it
(derived from the bundled ruleset, not a list): coverage quick | none, with
file counts per pin, and `uncovered`. `remember` turns that into `coverage`
memory rows (a new kind; the Run passes the alerts summary.json as input),
loop-measure reports coverage per language, and improve/memory.py's M1/M2
skip a repository+pin whose coverage for the rule's language is only
`none` (counted under `coverage_none` in the proposal's excluded).
`ousast plane alerts-engine <Run.yaml> --source <frozen export>`
(plane/engine_alerts.py) does the alerts task on the host for cases whose
files include a language the engine covers but quick mode does not (PHP):
exports both pins from the case cache, runs the quick scan and the
openultrasast:dev engine container (finding_dump.py, --network none,
--memory 3g, one at a time, per-pin deadline), writes engine findings in
the alerts.jsonl row shape (rule_id engine:<family>, source engine,
in_fix_range from case.json) with a summary (files read, questions,
completed, seconds, degradations per pin), and marks the task done via
reconciler.mark_done. No result, no files read, or questions == 0 fails
loudly and leaves the run untouched. Refuses a git checkout as source.
Runner image rebuilt and re-pinned (sha256:ebb40a9a...); validation-46
regenerated (digest only). Live, recorded in
benchmarks/measurements/2026-09-30-php-engine-alerts/: froxlor 366 files,
130/130 questions per pin, 6 alerts per pin, 0 in the fix range; wp-gopay
19 files, 101/101 questions, 0 alerts. The loop: 32 run rows (coverage 8,
php=engine), 0 proposals, gate no_proposals, $0.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
README: "Configuring the HarnessX agentic plane" becomes "The agentic
plane (ax)" (ousast plane run|status|doctor|remember, per-task Model,
budget and egress, pointer to ops/ax/README.md, one tagged retirement
line); "The HarnessX self-improving cycle" becomes "The self-improving
cycle", and its MetaAgent paragraph is replaced by the memory proposer
(`ousast improve --memory`, provenance, train-on-test guard). The fusion
panel_model/decider_model lines and the [harnessx] spend caps are gone.
The README reference test now passes without its strict xfail.
docs/examples.md section 6 runs the plane; docs/threat-model.md describes
the per-task EgressPolicy, the usd/calls budgets and the start-request
credential path. RELEASE_NOTES.md gets an Unreleased entry: HarnessX
removed, the retired keys and their errors, the silent loss of the
second hunter for users who set [models] hunter, and the plane as the
agentic path.
Dated retirement notes (2026-09-30) head the requirements, design and
brief of harnessx-self-improving-rulesets, learning-harness,
three-stage-scan and openrouter-sast-harness, and follow the spec table
in steering/overview.md; history is unchanged. All tasks are checked, so
spec.json moves to implementation-complete.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reran benchmarks/harnessx_removal_equality.py record from a git archive
export of b9afb66 (tasks 1-6 landed; same venv freeze hash as the
baseline, harnessx not importable, no API keys, empty config) and
compared it against 2026-09-30-harnessx-removal-baseline:
equal=True differences=0
All 13 output files (pairs.json, benchmark, standard scan, improve
--dry-run) are raw-byte identical to the baseline. The suite map
(suite-map.json) accounts for 29 deleted ids, 2 renamed ids and 83
added ids; each deleted or renamed id cites the design's Tests table and
commit cbadc38 or b9afb66. Every surviving test id keeps its outcome
(1740 passed, 38 skipped incl. the README xfail).
Instrument note: a first record failed test_push_hook twice because the
recording shell's PATH lacked .venv/bin (the hook execs `ousast`); the
same two tests fail at b9afb66 without it and pass with it. The
committed record has .venv/bin on PATH.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Deletes harness_ext.py and hunter_harness.py with their HarnessX-only
tests (the judge path, verify_judge.py, went in task 5). The harnessx
extra, its mypy override and its uv.lock entries are removed (uv lock
drops only packages; no remaining version changes). The conftest
fixture becomes assert_cold_import; test_gate's two never-import checks
become one allow-list (stdlib, openultrasast, yaml) over the governance
planes' module-level imports, and the skipped live-extra tolerance test
is deleted. Audit rows for the two modules are removed, stage_processors
is reworded, and the module-audit manifest is regenerated (no orphan).
test_removed_plane_references implements Req 2.4 over src, tests and
pyproject.toml: a match must carry a `retired 2026-..` tag (the spec's
own slug is not counted). The README part is a strict xfail until task 8
rewrites the README's plane sections.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The standard scan now runs run_hunter_pool, verify_findings and the
deterministic fuse_findings directly; the HarnessX hunter orchestrator,
the LLM judge dispatcher (verify_judge.py, deleted with its test) and the
LLM fusion panels are gone, and so are their extra-absent degradations.
host_under_harnessx is dropped from stage_processors.
config: [models] verifier and [fusion] panel_model/decider_model raise
RetiredConfigError naming the plane replacement (the CLI prints it and
exits 2); a [harnessx] section loads with one warning and is otherwise
ignored. harness.json keeps "verifier": null. Tests that asserted the
fallback are renamed to the behaviour they assert; the module-audit
manifest drops verify_judge. Nothing outside the two modules deleted in
task 6 imports harness_ext or hunter_harness.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
`ousast plane workspaces --validation-set ... --loop` appends the loop to a Run:
per case `alerts` (plane/tasks/alerts.py: the quick-mode scan on the vulnerable
and the fixed pin, a fixed-pin Workspace per case, alerts attributed to their
enclosing function by the repo-facts patterns, `in_fix_range` from the fix's
old-side and new-side ranges, failing on an unread tree) feeding `remember`,
then `memory-snapshot` -> `loop-measure` -> `loop-propose` -> `loop-improve`
(plane/tasks/loop.py). Every new task binds no Model and has budget
{usd: 0, calls: 0}. The snapshot is the store's rows after the train-on-test
guard, written on the host by memory.seed (a sandboxed run of it fails loudly);
propose runs M1/M2 against this repository at the commit under test; improve
runs one unchanged evolve.run_round and delivers gate.json, journal.json, the
sidecar, rule_policy.json only when accepted, and proposal_outcome rows that
the host ingests. Adoption stays a maintainer commit.
`remember` is now a Run entry per case (the image contains it), with the facts
row keyed by the image the facts were computed on (repo-facts records it in
its summary; unknown -> the row is omitted, never guessed). Alert rows carry
their own pin; improve/memory.py counts a repeated alert once.
Runner image rebuilt (+0.39 MB: only the new modules; the quick scan needs no
extra and the rulesets were already in the package), pushed, plane/ re-pinned
and regenerated (validation-46: 105 entries). Fixture test runs the whole loop
graph through the fake ax, whose new `exec` mode runs the real task modules.
Live: one loop Run over three validation-46 cases on the local ax, recorded in
benchmarks/measurements/2026-09-30-harnessx-removal-loop/ (all done, $0 new
spend, 0 proposals, gate verdict no_proposals).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
improve/memory.py: rules M1 (enabled->shadow on repeated false alerts across
>=2 repositories, no agreed or site_match hit) and M2 (shadow->enabled on
>=2 agreed declared sites no enabled rule catches); M3 disputed families are
advisory (signals.json) and never reach the validator. The train-on-test
guard drops rows of --qualify-population, gated benchmark cases and holdout
pair repositories before any rule sees a row. run_round(proposals=...) adds
memory edits after the benchmark's through the unchanged validator and gate;
provenance (row ids, object keys and versions, counts, exclusions, index
digest) goes to memory_proposals.jsonl and the journal's evidence field;
proposal_outcome rows are written back after a non-dry run.
ousast improve --memory is opt-in; default improve output is byte-identical
to the task-1 baseline.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
plane/memory.py: the MemoryStore interface with FileStore (index.jsonl, facts by
content hash, repos/<host>__<owner>__<name>/<pin>.jsonl, idempotent ingest,
deterministic row ids, refusal below 1 GiB free naming the path) and MinioStore
(object metadata and tags, S3 Select pushdown probed once per process with a
logged local fallback, object version ids on every row read, versioning and
runs/ lifecycle helper, presigned URLs). OUSAST_MEMORY selects the backend;
MINIO_* settings come from .env; the minio SDK is the optional extra `minio`.
plane/tasks/remember.py: model-free rows (facts, verdict, unit_cost, alert)
from a case's artifacts; remember_run computes them on the host for a recorded
run without writing into it. reconciler.mark_done (477 lines total), seed
before and ingest after `ousast plane run`, `ousast plane remember <run>`.
The generator annotates each repo-facts Task with openultrasast.io/memory-key
(repo, pin, candidates digest, image digest) and the Run with its population
and split; plane/ regenerated.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Equality script (design section 3) records suite outcomes, `ousast pairs --json`,
the python-vulnerable quick benchmark, one standard scan and `improve --dry-run`
from a frozen export with the harnessx extra absent and no model reachable.
Two independent records are byte-identical (determinism.json).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer: use MinIO features including its JSON support; a server exists.
One MemoryStore interface with FileStore (laptop, tests) and MinioStore (the
store of record for the separate Kubernetes deployment): tags for indexing,
S3 Select pushed-down queries probed with a local fallback, versioning for
proposal provenance, lifecycle for raw outputs, presigned URLs and bucket
notifications kept for the follow-on that removes the host receiver.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer amendment: no HarnessX path stays, optional or not; the plane memory
layer replaces the HarnessX evolution proposer and carries loop engineering
(Requirement 6: persistent run memory, proposals from it through the existing
validator and gate, the loop as a plane Run, provenance per proposal).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The ai-service-plane increment met its gate on ax (plane-increment-2.json), the
condition for removing HarnessX. Requirements: an explicit fate per capability
(moved to a plane task or retired), code/packaging/tests removed, deterministic
outputs proven byte-identical against a recorded baseline, docs and specs
retold, one reviewable sequence. Brief inventory re-verified.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A third verify pass on the 10 disputed candidates, 2-of-3 agreement (Req 6.4),
run on ax by extending the same Run: 16/20 declared sites agreed (reference 16),
$0.0205 per candidate with recorded triage (reference $0.022). Caveats recorded:
a design change on the same set; 17 vs 19 sites flagged in any pass; three
non-declared pgadmin candidates promoted.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
agree's declared_sites_matched counts candidate coverage, not detection. Recounted
from agreed.json rows, plane passes flagged 17 of 20 declared sites against the
reference's 19 (budibase read and fuxa getDaqValue missed by both passes). The
earlier "detection unchanged" reading was wrong.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The design revisit after plane-increment-1.json (14/20 agreed, 19/20 in either
pass): a third verify pass on the candidates passes a and b dispute, and agree
decides 2-of-3 where pass c asked (the a/b rule elsewhere; c alone never decides).
- verify: optional OUSAST_INPUT_ONLY restricts the hunt to the named candidates
(agree's disputed.json); an empty list is done with 0 units and no model client.
- agree: optional OUSAST_INPUT_PASS_C; metrics keep every key and add tiebreak
{asked, flipped_to_agreed, flipped_to_rejected, rule}; usd_total, hunts and
declared-site metrics are on the final decision.
- generate: per case <case>-vc and <case>-final appended to validation-46; the
first 60 entries stay byte-identical, so a rerun keeps their done state.
- Offline replay over the recorded outputs with an empty pass c reproduces the
recorded agree summaries exactly (14/20 agreed, 23 agreed, 10 disputed).
- Spec: Requirement 6.4, design Measurement, task 8.1.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
validation-46 ran done on google/ax: 60 tasks, 357 DeepSeek calls, $0.83 with
recorded triage ($0.0194 per candidate, reference $0.022: cost met). Declared
sites agreed 14/20 against the reference's 16/20: recall criterion not met.
Both runs find 19/20 sites in at least one pass; the two-pass agreement
flicker moved three sites. Design revisit before task 9.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ax names a task's ActorTemplate <task>-tmpl-<8 hex> and Substrate caps names at
63 bytes; validation-46's first task (51 characters) fell back to a missing
default template on the live cluster. Task names are capped at 49 with a
stable hash suffix when longer.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A producer's artifact rendered as spec.files of a generated <task>-inputs
Workspace ends up in ax's AX_WORKSPACES_YAML env value; Agent Substrate
rejects values over 32768 characters, so a 182 KB facts.json broke ax apply
on the live cluster. The reconciler now passes OUSAST_INPUTS instead, the
receiver serves a running task its declared inputs over GET /inputs/..., and
the runner fetches them after the start (failed fetch -> failed summary).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
On the live cluster the case clone ran at boot: the golden boot had no egress
policy, and the real actor was still being restored, so both TLS handshakes to
GitHub were cut. After the start the actor is RUNNING under its policy (the
delivery the same minute went through). Adds the one-case check Run (a subset
of validation-46) and re-pins the task manifests to the current runner digest.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The end-to-end smoke Run finished done on the local ax: submit, egress policy,
resume, router start, repo-facts in gVisor, artifacts delivered through the
egress gateway, attribution table printed, task and policy cleaned up.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Agent Substrate's egress gateway denies an actor without an EgressPolicy, so
no task could deliver its artifacts or reach a model. egress.py grants each
task exactly: http/80 to ousast-receiver.ax-system.svc.cluster.local, and
TLS passthrough/443 to the bound Workspaces' https Git hosts and the bound
Model's openultrasast.io/egress-hosts (api.deepseek.com for deepseek-flash).
No IPs, no wildcards, no https (MITM) rules.
The reconciler writes the policy with kubectl-ate between ax apply and
resume, reads it back, waits OUSAST_EGRESS_SETTLE (11 s, the gateway's
cached deny), and after ax delete task confirms the policy went with its
actor (kubectl-ate has no delete verb; the store cascades).
The receiver is a selector-less Service (ops/ax/receiver-service.yaml.tmpl,
applied by up.sh together with the no-build agentgateway egress gateway)
on port 18090 of the host. Measured live: the gateway checks the policy on
Host but connects to the address the actor dialled, so the runner dials the
Service's ClusterIP (OUSAST_ARTIFACT_DIAL) with the receiver's name as Host.
The runner now labels its delivery with the Run's task name (OUSAST_TASK),
which is what the receiver accepts; the ax Task name was refused with 403.
The Receiver moved to egress.py to keep reconciler.py at 460 lines.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
On the live cluster ax created the Task before its Workspace in the same apply
and the runner saw an empty AX_WORKSPACES_YAML. Tests now find rendered
documents by kind and name, not position. ops/ax/smoke-run.sh renders a
model-free repo-facts Run with the current runner digest and executes it
through the reconciler.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
agree matched declared sites through benchmarks/independent/evaluate.py,
which the runner image does not carry, so site matching would have been
"not assessed". The generator now resolves the fix's old-side ranges
(evaluate.hunks, line for line) into case.json, with the declared sites in
the set and the reference's cost, and agree.matches_declared reproduces
matches_v2; a parity test runs both over the recorded outputs of the 15
validation cases (200 probes, identical verdicts) and agree on those outputs
returns the reference's 16 of 20 and $0.022.
agree reports cost_per_candidate (plane spend) and
cost_per_candidate_with_recorded_triage; Requirement 6.2 is judged on the
latter (design.md). --runner-image re-pins the templates from ops/ax/up.sh's
pin file; the committed manifests are regenerated at the current digest.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Agent Substrate's golden-snapshot boot runs the same image with the same task env,
so the runner no longer starts its command by itself: after the workspaces are
ready it waits for POST /ousast/v1/start (run/task checked, once per boot, refused
after a delivered run). The reconciler sends it through svc/atenet-router with
ate-target-actor, which never addresses the golden actor (plane/router.py), and the
bound Model's secretKey variable travels only in that body, into the child's env.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The live cluster ran the task command twice: in the actor and in Substrate's
golden-snapshot boot, which gets the same image and task env. Requirement 4.5
gates the command on a router-addressed start request; 4.6 carries provider
credentials only in that request.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
On the live cluster ax hands AX_TASK_YAML with metadata.creationTimestamp and a
status block; the strict authoring schema rejected it, readiness never turned
green and every resume timed out. Other unknown fields still fail. up.sh retries
the first resume while the golden snapshot builds.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The first live run on the local ax cluster and ax's source (docs/runner.md,
docs/sandbox.md, internal/controller/reconciler.go, internal/server/server.go)
showed a lifecycle the task-3/4 code did not model:
- `ax apply` creates a Task Suspended (Ready=False, TaskSuspended); it runs only
after `ax resume task`. The first resume of a new image can fail with
DeadlineExceeded while Agent Substrate builds the golden snapshot, and ax's
server then stores phase Failed; a later resume succeeds.
- ax has no Completed phase and never reads a command's exit back.
- The runner is PID 1 and must keep serving after the command exits; stop and
suspend deliver SIGTERM, to be forwarded to the command's process group.
Our runner imported the task in-process and exited, which stopped the
container ("Container guest is stopped") and made every resume time out.
- ax's controller probes /readyz?check=workspace; the query string made our
handler answer 404.
Runner (Req 4.1-4.4): spec.command runs as a child
`python -m openultrasast.plane.tasks.<name>` in its own process group, cwd the
first workspace, AX_METADATA_URL set; ax's metadata paths are served. After the
child exits the runner reads summary.json (a crash without one becomes failed
with the exit code and stderr tail), delivers the output directory, writes
/workspace/.ousast-state/<task>.done.json and serves until SIGTERM (forwarded
to the group, SIGKILL after OUSAST_TERM_GRACE, exit 0). A boot with the marker
never reruns the command; a failed delivery is retried alone. Workspaces are
prepared on the first boot only. AX_RUNNER_EXIT_AFTER_COMMAND=1 is the
unit-test escape.
Reconciler (Req 2.2, 2.3, 3.1-3.3): `ax resume task` after apply, retrying
DeadlineExceeded/Unavailable with backoff up to OUSAST_RESUME_TIMEOUT (900 s);
completion is the artifact delivery; Failed after a resume went through, a
vanished task, or no delivery within OUSAST_TASK_TIMEOUT (7200 s) fail the
task with ax's condition message; `ax delete task` follows. Ax gains resume()
and phase() returns the condition message. doctor moves to plane/doctor.py
(tasks.md task 0 allows the sibling) to keep reconciler.py under 500 lines.
Tests: the fake ax follows the real lifecycle (Suspended, a failing first
resume that leaves Failed, Running forever, delivery only after a resume went
through), plus Failed-with-message, vanish, resume-timeout and task-timeout
scenarios; runner tests cover the child's cwd/group/metadata URL, stay-up and
SIGTERM in a real subprocess, SIGKILL of a SIGTERM-ignoring group, and the
marker. The opt-in live test now checks resume, Running, suspend and a second
resume instead of a Completed phase. design.md and tasks.md state the
lifecycle; the module audit manifest counts plane.doctor.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Model deepseek-flash (declared provider extension, endpoint prices in
parameters, secretKey naming the DEEPSEEK_API_KEY secret), digest-pinned
Task templates repo-facts / verify / agree, and a generator
(plane/generate.py, `ousast plane workspaces --validation-set ...`) that
emits per case the vulnerable-pin Workspace, a files-only inputs Workspace
(recorded candidates after the recorded triage, declared sites, triage),
per-case Tasks and one Run validation-46: facts -> verify a, verify b ->
agree for the 15 cases, 46 candidates (43 kept), budgets 1.5x the recorded
per-pass share. A Run entry cannot add a binding or env, and Run.inputs
names only task artifacts, so the case is fixed by per-case Tasks.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Substrate installs no WorkerPool (resume failed: no free workers) and rejects
images not pinned by digest; ax's default runner image on gcr.io needs
credentials, so the smoke task runs our runner (repo-facts over a one-file
workspace). A release ax CLI skewed from the server (UpdateTask unimplemented).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Requirement 1.1 (Task/Workspace/Model use only fields ax documents) and
Requirement 1.4 (no ax field repurposed) were violated: manifests accepted
metadata.namespace and a Workspace git `commit`, and the reconciler dumped
every git field into what `ax apply` receives.
Facts from ax's source (pkg/apis/v1alpha1/ax.proto, internal/workspace/setup.go):
- ObjectMeta is `name` and `atespace` (plus server-set creation_timestamp);
no `namespace`, no `annotations`.
- GitRepo is `name`, `repo`, `branch`, `dir`, `depth`; no `commit`.
- The API server decodes with protojson.UnmarshalOptions{}: an unknown field
is an error at `ax apply` (confirmed on the live cluster: a rendered
Workspace+Model applied cleanly; the same Workspace with `commit` was
refused with `unknown field "commit"`).
- ValidateModel checks only metadata; any spec.provider string is accepted.
Changes:
- manifests: metadata takes `name`, `atespace`, `annotations` (our extension,
never sent to ax); git entries take exactly ax's five fields (depth int);
`commit` in spec.git is rejected with a pointer to the new Workspace
annotation `openultrasast.io/git-commits: "<git name>=<40-hex>,..."`,
validated for unknown git names and non-hex shas.
- reconciler: rendered docs carry only ax fields; each bound Workspace's pins
reach the Task as env OUSAST_GIT_PINS = {"<workspace>/<git name>": "<sha>"}.
A test walks every rendered doc against ax's field lists.
- runner: pins come from OUSAST_GIT_PINS; git is fetched as setup.go does
(init/remote/fetch with --depth when depth > 0/checkout FETCH_HEAD, default
branch main, cloneDestination's dir and placeholder-name rules), then the
pin is checked out. A pinned entry without a branch fetches the commit.
- workspaces: emits the annotation, `dir: repo` and `depth: 1` instead of
`commit`, so generated case workspaces keep <path>/repo at depth 1.
- design.md: ax field names verbatim; pins and model binding as stripped
openultrasast.io/* annotations carried to the task by env.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
kubectl prints the sentence as one whitespace-split row, which passed the
ready check and reported an absent ax controller as "1 pods ready". Rows must
carry a READY column; an empty namespace fails. Regression test added.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ai-service-plane task 3, Requirements 4.1-4.4 (with 2.2 and 2.3 honoured by the
exit-code contract):
- src/openultrasast/plane/runner.py: the ax-task-runner entrypoint. Reads the Task
from AX_TASK_YAML and the bound Workspaces from AX_WORKSPACES_YAML (task-1 loaders),
serves /healthz and /readyz on port 80 (AX_RUNNER_PORT; 0 picks a free port,
AX_RUNNER_HTTP=0 disables it for unit tests), materialises each workspace at its
bound path (spec.files verbatim; spec.git at <path>/<name>, exported from the
OUSAST_CASE_CACHE checkout whose origin matches, else a shallow clone at the pin;
a repo already present on a resumed volume is kept) and only then answers 200 on
/readyz. Runs spec.command by importing openultrasast.plane.tasks.<name> with the
remaining elements as argv; sets OUSAST_WORKSPACE_DIR from the first binding when
the Task's env leaves it unset. A crash writes summary.json with status failed and
the traceback. When OUSAST_ARTIFACT_URL is set, the whole OUSAST_OUTPUT_DIR is
POSTed as an uncompressed tar (Content-Type application/x-tar, X-Ousast-Task,
X-Ousast-Run) with three attempts and backoff; a failed delivery rewrites
summary.json to failed and exits 2. Exit codes: done 0, failed 2, unfinished 3.
- plane/Dockerfile.runner: python:3.12-slim + git, `pip install .` (no extras), a
shell wrapper at /usr/local/bin/ax-task-runner exec'ing the module; built and
loaded into kind by ops/ax/up.sh, not by the tests.
- tests/test_plane_runner.py (+ tests/plane_stub_task.py registered as the task
module): env contract on a files workspace, readyz 503 then 200, delivery to a
local receiver with the headers and tar checked, delivery failure -> failed and
exit 2 after three attempts, crash -> failed with traceback, exit codes per
status, case-cache export at the pin, shallow-clone fallback, resume keeps the
checkout.
- model/audit.py: SPEC_OWNED row for plane.runner; module-audit measurement
regenerated (122 modules, no orphans, note kept).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Requirements 3.1, 3.2, 3.3, 3.4, 3.5, 2.2, 7.1, 7.3 of ai-service-plane.
- plane/reconciler.py (492 lines, test-counted): a Run's tasks submitted to ax
in dependency order through the injectable `ax` CLI (apply / get task /
delete task), serialize labels never overlapping, --workers, state.json with
pending|running|suspended|done|failed|unfinished, a PID lock per run, done
tasks skipped on rerun, unfinished tasks holding their dependants, a failed
task halting the run. Budgets, inputs, output dir, artifact URL and the bound
Model (annotation openultrasast.io/model; identity and priced parameters) go
to the task as env in the rendered Task; consumed artifacts travel as files
of a generated Workspace bound under /workspace. A threaded HTTP receiver
takes each task's tar delivery, validating run/task and rejecting traversal.
Outcome comes from the delivered summary.json (done only when every unit
finished), never from ax's phase alone. status prints the token attribution
table per task and Model from the summaries' usage fields (n/a, never an
estimate) and writes attribution.json; doctor runs the four host checks.
- plane/workspaces.py: one Workspace manifest per case pin of a population.
- CLI: ousast plane run|status|doctor|workspaces.
- tests/test_plane_reconciler.py: a fake ax executable drives the DAG;
tests/test_plane_ax_live.py (marker ax) applies ops/ax/smoke-task.yaml on a
live cluster and skips unless doctor passes.
- Module audit rows and regenerated manifest.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Requirements 5.3, 2.1, 2.2, 2.3, 7.2 of ai-service-plane.
- plane/tasks/verify.py: the batched hunt of verify_batched.py moved into
the package (system prompt, one hunt per file, up to 6 candidates, 6
steps, candidate-anchored sites). New against the reference: each
candidate's known callers from facts.json in the prompt (capped at 8),
tool turns and usage per hunt in units.jsonl, per-unit resume (rows
with "error" are re-asked), pass label from env, MeteredClient wrapper:
BudgetExhausted -> unfinished (exit 3), AccountError -> failed with the
provider message (exit 2); summary.json is written last, names the bound
Model and the summed usage, and is done only when every unit finished.
- plane/tasks/agree.py: verify_batched.agreement over two pass outputs,
agreed.json / disputed.json, per-candidate cost and turns, declared-site
match via benchmarks/independent/evaluate.matches_v2 loaded by file;
metrics block in summary.json; no model calls.
- tests/test_plane_verify.py: a scripted usage-recording hunter client;
callers line, resume, 402, budget stop and resume, model and summed
usage, unpriced usd None, both main() env contracts, agreement metrics.
- model/audit.py + module-audit manifest: SPEC_OWNED rows for plane.tasks,
plane.tasks.verify, plane.tasks.agree; manifest regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
plane/tasks/repo_facts.py computes, from source text only and without a
model or git, the product files of a workspace (tests, vendored trees,
node_modules, dist/build, minified bundles and dot-directories excluded via
the layout facts plus the model_sinks.py filter), the functions each file
declares (Python, PHP, JavaScript/TypeScript, Java, Go, Ruby declaration
patterns extended from sink_candidates.py) and, per candidate function, the
call sites in other product files with path, 1-based line and enclosing
function, capped at 40 and sorted (Requirement 5.1). Output is byte-identical
across runs over the same tree (Requirement 5.2).
Under the task contract the unit is one file: a row goes to units.jsonl as
each file finishes and a restart reuses the rows computed for the same
candidate set (Requirement 2.2); facts.json is assembled from the rows and
summary.json is written last, refusing `done` unless every unit finished.
The entrypoint reads OUSAST_WORKSPACE_DIR, OUSAST_OUTPUT_DIR and optional
OUSAST_INPUT_CANDIDATES only (Requirement 2.1) and exits 0 on done, 2 on
failed with the traceback recorded in summary.json.
tests/test_plane_repo_facts.py: a three-file tree with a tests/ file, a
vendored file, a node_modules bundle and a dot-directory; cross-file callers
with line and enclosing function, same-file and declaration lines excluded,
the 40 cap, byte-identical second run, resume that recomputes only the
missing unit, a crash that leaves `failed` with the unit counts and resumes
to `done`, and the declaration patterns per language.
The two new modules are registered as spec-owned in the module audit (their
importer, the runner, lands in task 3) and the audit manifest is regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
kind cluster + registry, Agent Substrate, ax control plane via ko, one files-only
smoke Task; helpers run from their own checkout because they resolve paths from
the git toplevel. Footprint and README follow once the cluster is up.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Add openultrasast.plane.manifests: frozen dataclasses and loaders for ax's
Task, Workspace and Model kinds (ax.io/v1alpha1, ax's fields only) and the
Run kind (openultrasast.io/v1alpha1: tasks, dependsOn, inputs, outputs,
budget, serialize). Every violation raises ManifestError naming the kind,
metadata.name and field path: unknown fields at any depth, wrong apiVersion
or kind, a provider outside google/anthropic unless deepseek is declared via
the openultrasast.io/provider-extension annotation, a Run with a dependency
cycle, an input naming no task, and a Run task naming no loaded Task.
load_manifests reads multi-document YAML files and indexes by kind and name.
PyYAML becomes the one declared core dependency; the two zero-dependency
guards now pin exactly that, and the module audit registers the plane
package under the ai-service-plane spec.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer decisions 2026-09-29: ax (kind + Agent Substrate + ax controller on
this host) is the only executor; the Run layer carries dependencies, artifacts,
budgets and attribution until ax has them. Requirement 7 declares routing per
Task (one Model per Task, no runtime switching) and a token attribution table
from provider usage fields. Task 0 (bring-up, doctor), runner artifact delivery,
ax-backed reconciler with a fake-ax test seam. harnessx-removal brief records
the HarnessX inventory; removal is gated on the increment's measurement.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
MeteredClient wraps a ChatClient, sums the provider's usage rows per call,
prices them from a Model manifest's parameters (cache-hit, input, output per
million), and raises BudgetExhausted before a call once the usd or calls
ceiling is reached. A 401/402 or "Insufficient Balance" failure becomes
AccountError with the provider's message. An unpriced model reports usd None,
never zero, and refuses a usd budget; summary() gives the spend block of a
task's summary.json.
The two plane modules are registered as spec-owned in the module audit (their
consumers land in tasks 3-4) and the audit manifest is regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Nine tasks in dependency order: manifests, budget metering, runner contract,
reconciler and CLI, repo-facts, verify and agree, the increment's manifests, the
measurement against the recorded batched check, handover.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Package src/openultrasast/plane/: manifests (ax kinds + Run), a reconciler under
500 lines with persisted status and resume, an ax-contract runner, budget metering,
and task entrypoints. First increment: repo-facts (source-only callers, no model),
verify (the batched hunt with callers in the prompt, per-unit resume), agree.
Measurement on the 46-candidate set against the recorded batched check.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Six requirements from the brief: ax kinds plus a Run kind; isolated, idempotent,
resumable tasks with metered budgets; a local reconciler under 500 lines with no
pipeline logic; ax's runner contract; repo-facts as the memory layer; and a
measured first increment (repo-facts + verify on the 46-candidate set: at least 16
of 20 declared sites, under $0.022 per candidate for two passes).
Phase: requirements, not yet approved.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Measured on the main session: 90% of development tokens were tool I/O. Three
tools from the Portal pattern:
- read_guard.py: PreToolUse hook that denies a whole-file Read over 350 lines and
points to a ranged read or the bulk reader (pipe-tested: deny, range, small).
- bulk_read.py: answers a question about files with the cheap chat model, citing
path:line, under a spend ceiling.
- token_report.py: where a session's tokens go, from its transcript.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Adopt the design of google/ax (Task / Workspace / Model manifests, a reconciler,
isolated resumable execution, a runner contract), not yet its software: it needs
Kubernetes and Agent Substrate, lists only Google and Anthropic providers, and has
no pipelines, artifacts or budgets. We add a thin Run layer for those.
Each pipeline concern (classify, triage, repo-facts, verify, judge, recheck, score)
becomes a declared task with its own inputs, outputs and budget. That bounds the
tokens per task and lets each be developed alone. Decisions listed for the
maintainer.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Exploratory, 7 of 17 cases, stopped when the account emptied:
- source-only scope: a model reading every product file puts the vulnerable
function among the candidates in 15/17 (ranker 1/17, vocabulary 2/17);
- per-file triage + per-file 6-step hunts + two-pass agreement: 12/15 candidate
cases on the validation set; 4/7 on the full run, two more in one pass of two;
- precision: 5 pgAdmin agreed findings read against source, 3 true (privileged),
2 false with self-contradicting rationales; agreement cannot catch an error
both passes make;
- cost: $0.009-0.022 per candidate for two passes.
Four instrument defects fixed on the way (model id, provider routing, JSON-mode
final turn, unbounded tool output) void the earlier hunter number. No gate is
met; qualification needs population v3 and a protocol that states the judge
stage, the pass rule and a config-family question.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
- verify_batched.py --fixed asks the agreed candidates again at the fixed pin
(no triage); a function the fix removed is recorded as removed.
- score_batched.py applies protocol v2's matching to the batched output
(vulnerable pin, and the fixed pin where rechecked) and draws the precision
sample.
- A finding's site is anchored to the candidate operation asked about; the
model's own reported location is kept in `reported_at`. ReactPress's confirmed
findAll was reported at the controller file and failed to match its declared
site. The scorer anchors older records the same way.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
verify_batched.py replaces the per-candidate verifier's design:
- triage per file (no tools): where does the operation's data originate;
internal/constant candidates are not verified;
- one 6-step hunt per file for up to 6 candidates, a verdict per operation;
- two independent passes; only agreement is reported, the rest is "disputed".
On the 46-candidate validation set ($1.01):
- declared sites: 16 of 20 agreed, 12 of 15 cases (the per-candidate design
confirmed 11). Budibase and Froxlor are new; Unleash dropped to one pass of two.
winml is triaged out as "constant": a CORS literal, which the reachability
question does not fit.
- the 26 pgAdmin candidates that flickered between single runs: 2 triaged out,
9 agreed in both passes, 4 disputed, 13 quiet.
- $0.022 per candidate for two passes vs $0.033 under the old design; on a file
with several candidates, 2.3x cheaper per pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Declared vulnerable sites: 11 of the 15 that were candidates are confirmed (8 at 3
steps, 3 more at 6). Missed: winml (a CORS config; the reachability question does
not fit), FUXA, Budibase, Froxlor.
Restart-based escalation is strictly worse than a straight 6-step budget: 88% of
pgAdmin candidates used every step, so they paid 3 + 6 ($0.0201 per candidate vs
$0.0139 baseline). Defaults are now 6 steps, no escalation. The cache order change
did not lower the per-call cost ($0.002): the conversation prefix was already
cached; 56% of prompt tokens are served from cache.
The larger finding: on 180 pgAdmin candidates verified twice, 15 and 13 were
flagged with only 2 in common. Verdicts on borderline candidates flip run to run,
while the declared sites were found consistently. Precision needs agreement
across runs or a second judgement, not a single pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
--declared-sites-only, --candidates-from <jsonl> and --results-suffix, so the
efficiency change can be validated on the 180 already-verified pgAdmin candidates
and the 15 declared sites without touching the main results.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Token efficiency, measured on 180 verified pgAdmin candidates: stage 2 costs
$0.0139 per candidate, 90% of the spend, because the conversation is re-sent on
every turn and grows with each tool result.
- Context files now precede the question in the hunter's first message. DeepSeek
bills a cached prefix at 1/30 of a fresh one, by prefix; with the question first,
every candidate asked about the same file (about 5 per file) broke the prefix on
its first line and paid for the file again. A test pins the order.
- The verifier runs each candidate at 3 steps and repeats it once at 6 only when
the hunt spent every step on tool calls without a verdict. On the seven declared
v2 sites, 3 steps found 5 and 8 steps found 6; most candidates conclude in 3.
- Per-candidate token usage (prompt, cached, completion, calls) and escalation
are recorded, and each case reports its cache share, so the next efficiency
decision is made on numbers.
Validation on the 180 pgAdmin candidates plus the 22 known sites (about $1)
waits for account credit: DeepSeek returns 402 at the moment.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The rerun hit DeepSeek HTTP 402 ("Insufficient Balance") after 180 pgAdmin
candidates. Every later call failed at $0, and the runner wrote sixteen cases as
complete with nothing asked, because an unverified candidate counted toward
completion. That is a clean-looking zero from an instrument that could not call
the model.
Now:
- 401/402 or "Insufficient Balance" stops the run at once, exit 2, nothing
recorded;
- only verified candidates count as done, so unverified ones are asked again on
resume;
- a case is complete only when every candidate was verified.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
FUXA failed every hunt with HTTP 400, and neither run could say why. The client
discarded the response body. With the body shown, the reason was "This model's
maximum context length...": FUXA ships minified bundles whose single lines run to
megabytes, and grep_repo returned a matched line whole.
- grep_repo and find_refs quote a matched line (240 characters), never the whole
line.
- Every tool result is capped at 16 x max_chars, with a truncation marker.
- OpenRouterChatClient errors carry the provider's response body (first 500
characters). That would also have exposed the rejected model id in one call
instead of eighteen days.
After the fix, FUXA candidates verify normally at $0.004-0.013 each. A test covers
the megabyte-line case.
verify_sinks.py: skip futures cancelled at the spend ceiling instead of reading
them (the run crashed with CancelledError).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The final-answer turn requested the provider's JSON mode with the tools still
attached. DeepSeek (deepseek-flash) then answered with the format itself
(`{"type": "json_object"}`), wrapped the array (`{"type": "json_object",
"value": [...]}`), or emitted its tool-call markup as text. A hunt that had found
the vulnerability reported nothing.
Measured on the v2 sink verifier with the declared vulnerable functions: 0 of 7
confirmed in JSON mode, 5 of 7 asked plainly (6 of 7 at 8 steps). The model named
Medplum's actual redirect-URI prefix-match bug, LMDeploy's redirect bypass of
_is_safe_url, and ContextForge's unsandboxed Jinja2.
The final turn is now plain text without tools. The parser also reads a
`{"value": [...]}` wrapper. Two tests that pinned JSON mode now pin the plain
turn, and a new test reproduces the wrapped answer.
Invalidates as measurements: the hunter experiment on v2 (c4a0bd8) and the first
sink-verifier run. Both used this turn.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Each grep_repo and find_refs call re-enumerated the tree (walk, ignore rules,
vendored checks) and re-read every file. The v2 sink verifier spent almost three
hours on its first case (pgAdmin, about 1,800 files) at 108% CPU with one open
connection. It was CPU-bound in these tools, not waiting on the model.
Source lines are now memoised per root, keyed by path and modification time. A hit
is re-validated file by file, because an edit inside a subdirectory does not touch
the root's mtime; a test covers one enumeration and a detected edit.
verify_sinks.py now runs candidates in worker processes, not threads, and appends
each candidate's result to a per-case log, so a stopped run resumes where it left
off. GoPay: 12 candidates in 14 s, the real site found.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Per candidate function, one bounded tool-hunter pass (custom prompt, the file as
context, 3 steps) asks whether attacker-controlled data reaches the operation
without a sufficient guard. It reports only with a stated source-to-sink path.
Hard spend ceiling, parallel workers; one run per pin; the fixed pin is rechecked
only for detected sites. Exploratory on v2.
Pilot: on GoPay, the one real site was reported in 3 of 4 runs and nothing else
out of 12 candidates.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Source-only selection on the v2 vulnerable pins, compared on the same cases:
- ranker top-20 entry points: the vulnerable file was in scope in 1/17;
- shipped-vocabulary sink-first: the declared site was a candidate in 2/17;
- model-classified sinks (deepseek-flash, no vocabulary): the declared site was a
candidate in 15/17, and a fix or site file in 16/17. Every chunk was classified;
$14.42.
Misses: User Registration (file in scope, not the function) and Sulu (JCR-SQL2
query builder). The cost of this reach is volume: 3-1037 candidate functions per
repository, 4,762 in total. The next stage must decide which candidates are fed by
attacker data without a guard. v2 is spent for tuning; this qualifies nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This asks DeepSeek, per product source file (in chunks of 400 numbered lines),
which functions perform the family's dangerous operation on non-constant data. It
is scored like sink_candidates.py: is the declared vulnerable function among the
candidates? A failed chunk counts as unclassified, never as "no sinks". The spend
ceiling is enforced across worker threads.
Pilot: GoPay found its declared site for $0.03. Estimated full v2 run: ~$14-18.
The maintainer raised the ceiling to $20.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This is a source-only diagnostic: no engine and no model. At each v2 vulnerable
pin, every call matching the family's shipped sink vocabulary becomes a candidate,
attributed to its enclosing function.
- The declared vulnerable function is a candidate in 2/17 cases (Medplum, GoPay);
the ranker's top 20 managed 1/17.
- A fix-changed or site file is among the candidates in 8/17.
- Candidates per repository range from 2 to 904 functions.
The misses are vocabulary, not direction. Examples: wp_redirect, download_url,
spawn(shell:true), ky(), requests.Session().get, sympy parse_expr, Jinja
from_string().render, Mongo collection.find, JCR-SQL2 createQuery, TypeORM
andWhere template strings, and CORSMiddleware(allow_origins=["*"]) are not named.
v2 is spent for tuning, so this informs design and qualifies nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The run follows hunter-protocol-v2.md: deepseek-flash, the ranker's top 20
regions per pin, protocol v2 scoring, $8.88 spent.
- Recall: 1/17 (Wilson 0.01-0.27).
- Precision: 2/14 sampled (both true positives need a privileged role, one of
them GoPay's CVE).
- Fixed-side alerts: 1. Benign alerts: 16.
- Stability: 73% of findings differ between two identical runs.
The deciding observation: the ranker's top 20 contained the vulnerable file in 1
of 17 cases, and in that case (GoPay) the hunter detected it. With 4 tool steps
per hunt it cannot search its way to code it is not handed. The result therefore
says less about the model than about scope selection, which sits upstream of the
engine too.
Instrument failures, reported as failures: every FUXA pin and three Directorist
pins (DeepSeek HTTP 400 on every hunt; neither offered a region of its family).
Scorer fix: protocol v2 defines benign alerts and the precision pool over the
CASE FAMILY, but score_v2 pooled every family. The engine's dump emits only the
case family, so its recorded result (21adc40) is unchanged by this; rescoring
reproduces 3 benign alerts and 49 pooled findings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This is the deciding experiment of the strategy review, fixed before any v2 pin is
hunted.
- Hunter: the unchanged tool hunter (default prompt, tools and 4 steps) on
deepseek-flash.
- Scope: the ranker's top 20 regions for the case family, in four hunts. The model
never sees the fix diff or the declared sites.
- Families: a fixed keyword table maps the model's family to the protocol's three.
- Scoring: exactly protocol v2's scorer.
- Cost: a hard $10 ceiling.
hunt.py runs it and scores it (--score). A pilot on a v1 development case
(mongo-express) found the vulnerability at lib/bson.js:60, at $0.031 per pin,
alongside a lot of speculative noise.
Transport fix found by the pilot: the DeepSeek endpoint the hunter builds was
unpriced, so the cost meter read $0.00 for any number of calls. It is now priced.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Every model call has failed since 2026-09-10 (83ab533):
- The default detector and judge model was `deepseek-v4.1-flash`, the product name.
The API rejects it with HTTP 400 ("The supported API model names are
deepseek-flash, deepseek-v4-pro"). The judge and pipeline record an endpoint
failure as a degradation, never a verdict, so nothing failed loudly.
- `tool_hunter.resolve_hunter_client` resolved OpenRouter only. In this project
DeepSeek is the chat provider and the OpenRouter key is scoped to embeddings, so
the hunter could not reach a chat model at all; the call came back 401.
The defaults are now `deepseek-flash` (priced). The hunter resolves DeepSeek first
when its key is set, and OpenRouter stays as the explicit or fallback route.
Verified live: plain completion and a tool call through both the endpoint resolver
and the hunter client.
Not affected: the independent evaluations (finding_dump runs with
max_model_calls=0) and the 2026-09-06 hunter measurement, which predates the id.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
v1 (1/11) and v2 (0/17 recall, 0/17 sampled precision, 12% completion) show that
fixes learned on one population do not transfer to the next. The causes are
breadth: framework-shaped entry points, module boundaries the frontends do not
resolve, name-matched sinks colliding, and scale.
Options:
- A: continue engine-first;
- B: the model proposes, the engine proves;
- C: narrow the product to one ecosystem.
Recommendation: first run the unchanged tool hunter on the v2 pins as the deciding
experiment.
For maintainer decision; no approved spec changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The analyzer was frozen at a75b9f6. 17 cases, 102 scans (vulnerable and fixed pins
twice each), scored exactly as pre-registered.
- Recall: 0/17 (Wilson 95% 0-0.18). No unstable detections.
- Precision: 0/17 sampled (0-0.18). Every sampled finding is false: regex
match/exec read as sinks, int casts, ORM statements, client-SDK code, the
process's own argv, a generated database name, a registered-URI check.
- Completion: 13206/108374 questions (12%).
- Fixed-side alerts: 0, trivially, since nothing was detected.
- Benign alerts: 3.
- Stability: 2 of 100 findings differ between runs.
The zero is not an instrument artefact: FUXA, GoPay, cct-studio, ReactPress, User
Registration and Directorist completed questions in the sink file itself and still
found nothing.
Instrument failures, reported as failures: LMDeploy (every taint batch failed on all
six pins) and Froxlor's two benign scans.
This record is committed before any analyzer change informed by v2.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Committed before any v2 case is scanned. It has two changes from v1, both chosen by
the maintainer from lessons of the v1 run:
- declared sites: a finding at a `path::function` the advisory names is credited,
as well as one on the fix's changed lines;
- two runs of the vulnerable and fixed pins: detection needs the finding in both
runs, a fixed-side alert in either counts, and the difference between runs is
reported as stability.
evaluate.py takes --population. v2 scans six pins per case and scores with
matches_v2 / score_v2. v1's scoring is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The analyzer was not run. freeze.py read each case's source at both pins:
- both pins are commits in the repository and in the right order;
- the declared sink site's file is present at the vulnerable pin, and its declared
functions appear in it;
- the fix touches a declared site file (Budibase's sites reordered so the changed
file, queries.ts, comes first).
Benign controls:
- 12 are reviewed automatic proposals.
- 3 automatic proposals were rejected as not benign: qwed's merged a ReDoS fix,
ReactPress's corrected a SQL column whitelist, and Sulu's was a branch merge that
can carry the fix.
- Those three, plus cct-studio and Directorist (no automatic proposal), were chosen
by hand. Directorist's comes from before the vulnerable pin, because only 3
commits follow its fix.
freeze.py now accepts a recorded benign control instead of reporting that none was
proposed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
v1 was evaluated and then used to retune the analyzer. A retune can only be
qualified on a population that did not inform it.
The candidates came from web research alone (research-2026-09-26.md):
- advisories were taken newest-first per language under fixed criteria, never by
how easy they look;
- 261 advisories were examined and 224 rejected, each with a reason;
- every SHA pair was verified through the GitHub API;
- no clone and no scan.
The maintainer selected the rule-based "balanced" set: 5 Python, 6 JS/TS, and 6 PHP
of which 4 are WordPress plugins; 10 injection, 6 destination and 1 config.
Each case declares the advisory's sink sites before any scan, for protocol v2.
freeze.py takes --population and writes freeze-vN.json. The reservation tests now
cover every population-v*.toml.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This reverts commit cc885f0. It failed the acceptance bar set before the run (each
recall pin within +2 findings, completion unchanged):
- WP Statistics: 16 -> 31.
- PMPro: 16 -> 26, and completion 378/382 -> 103/382. Six prior findings were lost
to the budget; CVE line 936 survived.
- YesWiki: completion 2798 -> 1968, still not detected.
Two causes:
- Cost: wrapper confirmation runs a flow query per parameter of every sink-bearing
method, repeated at each engine start of a portioned scan.
- Precision: a helper whose parameter reaches a sink under its own escaping (for
example `getMemberOrderByPayPalToken(sanitize_text_field(...))`) is not a sink
for its caller.
Doing this properly needs summaries that carry the wrapper's internal
sanitisation, computed once per scan, not per request batch.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This is the second attempt at the YesWiki shape: SQL run through
`$this->dbService->query($query)`, whose body is the `mysqli_query` the facts name,
through a receiver the frontend cannot type. The first attempt was reverted in
c7e3a05.
A method is now a wrapper for a parameter only when the engine's own dataflow
inside the method carries that parameter to a fact sink's argument, on a plausible
path that no cleansing or guard discharges. The guard rails:
- one level only;
- nothing under a bundled library tree (lib/, vendor/, node_modules/, ...);
- a name counts only if every project method with that name is a wrapper;
- wrappers are asked in the flow pass only, never in the evidence pass, so no
region's selection or pruning changes;
- at a wrapper call, only the confirmed argument is asked.
Three Joern-gated tests cover the direct shape (reported), a literal, an
intval-casting helper and a level-2 feeder (not reported), a harmless same-named
method, and a wrapper inside lib/. Each fails when its rule is removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This reverts commit 54c9dfc. The recall-pin check (p16) showed a large precision
and completion regression, although every recall target survived:
- MW WP Form: 3 -> 60 findings. Form-field renderers became sinks.
- WP Statistics: 16 -> 40.
- PMPro: 16 -> 65, 20 of them inside bundled Braintree/Stripe code. Completion fell
to 390 of 1189 questions, because wrappers made many more regions look
sink-bearing.
- YesWiki: completion fell from 2798/2807 to 216/2807.
The structural summary was too loose. Any identifier in a sink argument counted as
fed by a parameter if any assignment anywhere in the method mentioned it, with no
order or dominance. It also ignored vendored library code and cascaded three
levels by name. A redesign needs dataflow-confirmed, level-limited summaries that
leave the question count alone.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Real code rarely calls the driver where it builds the query. YesWiki builds its
INSERT in `FormManager::create($data)` and runs it through
`$this->dbService->query($query)`, whose body is the `mysqli_query` the facts name.
The only sink the facts knew sat in a helper that no request reached directly.
A method whose parameter feeds a family sink is now a sink for that argument,
reported at the call site. The summary is structural, to stay cheap and
deterministic. It is not a flow query per parameter, which would repeat at every
engine start of a portioned scan. The parameter must reach the sink call's
argument directly or through at most two local assignments, with at least one
occurrence outside every cleansing call. It iterates up to three levels, so a
wrapper of a wrapper counts.
Calls resolve by name, because an injected receiver has no type the frontend can
see. A name therefore counts only if every project method with that name is a
wrapper. At a wrapper call, only the summarised arguments are asked.
Two Joern-gated tests cover:
- a request value into a second-level wrapper, reported at the controller's call;
- direct use, a literal, and an `intval`-casting helper, which are not wrappers;
- a harmless same-named method, which disqualifies the name.
Both fail with their part removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reading /proc/<pid>/stat for a process that is reaped during the read raises
ProcessLookupError (ESRCH), not FileNotFoundError. The previous race fix caught only
the latter, so the hanging-process-group test failed twice today under full-suite
load. It passed alone. The disappearance is exactly what the test asserts.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
p14 showed PMPro's functions.php:906 discharged by `elseif (in_array('E', $levels))`.
That checks whether a constant is in `$levels`; it says nothing about `$levels`. The
guard counted a carried variable in any argument. Only argument 1, the tested value
(`in_array`'s needle, the URL given to `url_has_allowed_host_and_scheme`), is
validated now. The allowlist test gains the haystack shape as a reported control.
The test fails when any argument counts.
The other p14 removals were right: WP Statistics export_data x2
(`if (!in_array($table, $allow_tbl)) $table = false;`) and PMPro memberorder.php:1633.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Ultimate Member's fixed pin kept one alert. The fix writes
`$this->sql_order = " ORDER BY u.{$sortby} "` inside
`elseif (in_array($sortby, $allowed, true))`, and after the if/elseif chain it
passes the field through `apply_filters(...)` into a second assignment. The guard
was judged only at the flow's final node, which is outside the branch.
A guard is now also judged at every point where the path stores the value (an
assignment target). A value stored while the check held carries the check forward,
because the rest of the path derives from that store. A mere read inside a branch
is not a candidate: the engine lists uses in branches the flow does not depend on.
The Joern-gated test gains the shape and its control: the same store in the
unguarded else branch, through the same filter, is still reported. The test fails
when only the end is judged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Three of the four fixed-side alerts left in the independent population had one
cause: the fix was a check, and checks were not modelled.
- wger's fix is `url_has_allowed_host_and_scheme`: `if not ...(next_url):
next_url = reverse(..)` and `if next_url and ...(next_url): return redirect`.
- Ultimate Member's fix is an allowlist: `in_array(strtoupper($order), [...],
true) ? $order : 'ASC'` and `elseif (in_array($sortby, $allowed, true))`.
(The fourth, Signal K, is `assertAllowedHost`, a project-local function. It is not
covered.)
Sanitizer facts can now say `guard = true`. A guard is a check, never a cleansing
call on a path. A flow is discharged only when a guard tests a variable the path
carries, and the flow is in one of these places:
- inside the branch whose condition requires the check. The guard may be combined
with `&&` only;
- past a failure test, a single `!` combined with `||` only, whose branch exits
(return, exit, die, wp_die, throw) or overwrites the value, and whose condition
dominates the flow's end;
- through a ternary whose condition requires the check, where the other arm cannot
carry the value. The engine can enter through the condition's own copy of the
value, which never becomes the result.
Facts: `in_array` (PHP) and Django's `url_has_allowed_host_and_scheme` /
`is_safe_url` (Python). A row now consults guards even when a request lists no
sanitizers.
Two Joern-gated tests cover the discharged shapes and, as controls, an unguarded
redirect, a guard on a different variable, a failure that is only logged, the
unguarded else branch, and the ternary arm the check does not select. Both tests
fail without the guard.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The enumeration dropped every `!` line and matched with fnmatch, whose `*` crosses
`/`. YesWiki's `tools/*` followed by `!tools/bazar` therefore removed all 583 PHP
files under tools/, the vulnerable FormManager.php among them. The scan read 243 of
803 files, completed 1128 of 1128 questions and looked clean: a zero from an unread
tree.
Rules now follow git:
- the last matching rule wins, and `!` re-includes;
- `*` and `?` stay inside one path component, and `**` crosses components;
- a rule with an inner `/` is anchored to the root;
- a trailing `/` matches directories only;
- nothing under an ignored directory can be re-included.
In a work tree, a tracked file is never ignored, because `.gitignore` describes what
the repository does not hold.
The test uses `git check-ignore` as the oracle on 16 paths covering negation,
anchoring, depth, directory-only and `**`. It also covers a git-archive export with
no .git, and a tracked file under an ignored directory. All three tests fail on the
old reader.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Two defects found by the cause-B rescan of the independent population (exploratory,
a retune after protocol-v1):
- `typescript_property_support_unvalidated` still reached the whole repository.
YesWiki is a PHP application that ships a few `.ts` files, and all 1,128 of its PHP
questions were unresolved. The limit is now owned by the TypeScript partition. A
TypeScript project (Mockoon, Signal K) is unchanged.
- `isSourceCall`, and the method-level "reads a source" heuristics, matched source
patterns as substrings. Flask's `request.get` therefore claimed Django's
`request.get_host()`. wger's fix passes that host to
`url_has_allowed_host_and_scheme`, and both fixed functions were reported as open
redirects from it. Patterns are now matched as bounded tokens, like sinks and
sanitizers. wger's vulnerable-pin detections rest on `request.GET`, which still
matches.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This extends dbcb230 to the remaining graph-integrity gaps. In the independent
evaluation these gaps were repository-wide:
- alerta: one symlink, `alerta/app.wsgi`, which is not even source, made 261 of 261
questions unresolved.
- wger and Signal K: a language no frontend reads did the same.
- YesWiki: unparsed files did the same.
Now:
- the partition build records the paths a symlink boundary skipped (a directory
ends in `/`) and the languages no frontend reads;
- `files_unparsed` carries its full paths;
- the scan demotes only questions about an owned path or under an owned directory,
and only questions in an owned language.
Every gap stays in the report as a boundary. A gap that names nothing keeps its
partition-wide or repository-wide reach. Tier-0 pruning stays off, as before.
The tests that pinned repository-wide reach now assert ownership. The one about a
vendor exclusion masking an integrity failure keeps its intent, using a census gap
that names no file.
Retune after protocol-v1; exploratory only.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
`esc_sql` and the mysql/pg `*_escape_string` functions only backslash quotes. A value
they escape is safe inside a quoted literal and nowhere else. Ultimate Member
CVE-2024-1071 is `$sortby = esc_sql(...)` built into `" ORDER BY u.{$sortby} "`, and
it read as sanitized. Sanitizer facts can now say `quoted_only`. The escape keeps its
credit unless the first string the value is built into (concat, interpolation,
addition, f-string) places it after literal text that opens no quote. With no string
built downstream, the credit stays as before. pg_escape_literal adds its own quotes
and stays a full cleanser.
Two further defects hid the same CVE:
- The engine is field-insensitive on the object. Its one path from `$this->sql_order`
went `$this` -> `$this->having` -> `esc_sql($this->having)` -> ... -> the query.
absorbedByObject cut it, and it credited an unrelated field's escape. When a path
leaves a field through the object for another field and reads the first field
again, the detour is spliced out. The later read may be on the path or inside the
sink argument where the path ends.
- The field join judged a field clean if ANY sanitizer name was on its path. It now
uses the same quote-aware check as a row. This matters because the engine's own
path to the query changed between runs: 2 of 4 identical runs chose a path via
`$wpdb->prepare`. The field join is the deterministic route.
Exploratory (retune after protocol-v1): the Ultimate Member vulnerable pin is detected
at class-member-directory-meta.php:936. The fixed pin still alerts at :945. Its fix is
an allowlist (`in_array(...) ? $v : 'ASC'`, and an `in_array` guard), which is not
modelled yet.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
python.toml's sources were Flask-only, so no flow could start anywhere on a Django
project. OpenCVE's SSRF reads its URL from `form.cleaned_data`, and the scan answered
the sink file and found nothing. Added: request.GET/POST/body/COOKIES/FILES/META,
DRF's query_params, and cleaned_data.
With those sources known, OpenCVE reported three redirects that cannot leave the
site: `reverse("cves") + "?" + request.GET.urlencode()` (twice) and
`reverse(route, kwargs=...)`. Destination sink facts can now set
`prefix_fixes_origin` and `origin_anchors`. A path is cut when it enters a
concatenation, f-string or `+=` through anything other than its leftmost operand,
and that operand fixes the origin. Origin-fixing operands:
- a literal `/x`, `?` or `#`;
- a literal of the form scheme://host/;
- an anchor call;
- a variable whose every assignment fixes the origin.
A path is also cut when it passes through an anchor. `"/" + x` (//evil) and
`"https://" + host` stay reported. A Joern-gated test covers all seven shapes.
Exploratory (retune after protocol-v1): OpenCVE vulnerable and fixed pins are both
0 findings, down from 2 and 3. The SSRF itself is still missed, because the flow
crosses a registry dispatch (`REGISTRY.get(t)(extras).run()`) that the engine does
not follow.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
mongo-express CVE-2019-10758 passes the request body to vm.runInNewContext. No
injection sink named that call, so the flow was answered and missed. runInNewContext,
runInContext, runInThisContext and compileFunction are now CWE-95 sinks.
Once the file's questions completed, the fixed pin showed a second defect. Every call
on `require('mongodb-query-parser')` has that package name as its full name. The
bounded-token match treats `-` as a boundary, so the name matched the SQL sink
`query`. In a full name, `-` belongs to a package name, so it no longer separates
tokens.
Exploratory (retune after protocol-v1): mongo-express vulnerable pin now detected at
lib/bson.js:60.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The first independent evaluation (results-v1.json) completed no question on eight of
eleven projects. On the cases traced, the cause was a census gap: `.eslintrc.js` or
`webpack.config.js` missing from a JavaScript graph, or `contrib/apache/api.wsgi` from a
Python graph, and every question in that language was then marked unresolved. Now only
questions about a named missing file are demoted. The gap is still reported as a
boundary. A gap that names no files still reaches its whole partition.
This is a retune after the evaluation. Per protocol-v1 it cannot be qualified on the
same population.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Scored exactly per the pre-registered protocol-v1.md, analyzer ebcc2cf,
44 scans on eleven untouched cases.
Recall 1/11 (95% CI 0.016-0.377): only Signal K's SSRF sink is detected.
Precision 1/12 on the full pooled sample: the other eleven findings are
guarded redirects (isSafeRelativeUrl, getSafeDestination), Host-header
redirects, constant SET NAMES queries, a config-file eval and an installer.
Four fixed-side false alerts (mongo-express bson.js, both wger sinks after
their guard was added) and one benign.
The dominant cause is coverage, not precision: at the vulnerable pin eight
of eleven cases completed ZERO questions and Saltcorn asked none, so most
targets were never decided. Recorded before any investigation or change.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Committed before any case is scanned: what is under test (the analyzer at the
runner's commit, per-case family only, four scans per case), how a finding
MATCHES a case (a fix-changed file, the case family, the named function or
within 10 lines of a fix hunk), detection / fixed-side / benign false alerts,
unanswered and instrument failures, the precision sampling rule (every third
pooled vulnerable-pin finding, or all if 15 or fewer), what is reported
against the M4 gates, and that no retune may be informed by the results.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Every reserved case verified with git and file reads only -- nothing
scanned: pins present and ordered, the sink present at the vulnerable pin,
the fix touching the sink (or LearnPress's declared source-side fix site),
and the sink file hashed at both pins (benchmarks/independent/freeze.py,
record freeze-v1.json).
Each case now carries a benign control from its own history, reviewed by
diff. The mechanical keyword filter was not enough: it proposed an access
restriction, a filter check, API-token scopes, a login change and a
TLS-verification option, all rejected; Ultimate Member and alerta got
hand-picked controls (a WPCS reformat; a dependency change with both sides
still vulnerable). The guard test requires a benign pair per case once
frozen.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Eleven cases chosen by the maintainer from fifteen researched by advisory
and fix diff only -- nothing cloned, nothing scanned, none among the 262
repositories the benchmarks reference -- plus the earlier Ghost
reservation. Four are PHP repositories that ship JavaScript (two WordPress
plugins), satisfying the multi-language requirement; families are limited
to the v1 scope (injection, untrusted destination, permissive config).
The development subjects shaped every rule of the last week and can no
longer measure whether those rules generalise; this population can, until
something scans it. A guard test fails the moment a reserved repository is
referenced by a recipe, catalog, manifest or measurement outside
benchmarks/independent/. Freezing (pins and sinks verified by reading
source, a benign change per case) is the next step.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The largest remaining class in the third PMPro sample (2 of 4 false
positives): every MemberOrder method builds its statement in
$this->sqlQuery and runs it at once. One method writes request data there,
the field join keyed the field file-wide, and deleteMe() -- which had just
set it from the internal $this->id -- and updateTimestamp() (no caller in
the plugin) were reported as SQL injections.
A field read is now a field-join source only when it reads state another
method left: not the target of a write, and not a read that the method's
own write to the same field dominates. If that own write carries request
data, the ordinary flow from the source still finds it. A real-Joern
control covers all three: the method whose write carries $_POST (found),
the method that overwrote it with an internal id (quiet), and a method
reading what another left (found). It fails on the old query.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Found on PMPro's re-added class-pmpro-levels.php:87: every REST handler
casts the level id, `isset($p["id"]) ? intval($p["id"]) : null`, and the
flow was reported unsanitized. Traced, the engine steps from intval's
argument straight to the conditional, so `intval` is never a path element
and the element-wise sanitizer check never saw it.
- sanitizedBetween: where a path element contains the previous one, the
syntax between them is what the value was wrapped in; a sanitizer there
sanitizes the flow. A sanitizer elsewhere in the expression (the other
branch of a conditional) does not count. Used by the sink query and the
hook join; the field join already matched sanitizer text.
- The test exposed the conditional itself as a source: isSourceCall matched
any operator by its full text. Of the operators only an access
(field/index) names its own value; a conditional or concatenation built
from `$_GET` is not a source.
A real-Joern control covers the cast inside a conditional (sanitized) and
a sanitizer in the other branch (not); 20 query tests pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
profile.php:424 survived the hook-argument fix through the field join: in
another function of the same file, `$user = new stdClass; $user->ID =
$_POST['user_id']`, and the join keys a field by its TEXT across the whole
file, so the WP_User parameter's `$user->ID` became a field source.
The join exists to carry object state between methods ($this->attachments
written in the constructor, read in _delete_files). That reading of a
field's text holds only when the base IS the object ($this, this, self).
Any other base is a variable, matched within its own method only, where
the engine's dataflow already sees it. A real-Joern control covers both
(a local variable's field in another function stays quiet; $this state
across methods is traced) and fails on the old query.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The mapper took the FIRST 'callback' in a 12-line window. One registration
commonly carries an array of endpoints; PMPro's /discount_code has a GET
and a POST handler, so the writer, pmpro_rest_api_set_discount_code, was an
ordinary `function` entry. Since da0ee57 such an entry's parameters are
not attacker input, so its $request stopped being a source and the REST
route that injects SQL through the discount-code save() had no source at
all.
The argument list of the call is now read to its balancing parenthesis,
every callback in it is a route, and each keeps the permission callback of
its own endpoint (the first after it before the next callback, else the
last before it). PMPro: 15 -> 18 REST routes.
This restores the source, not the finding: traced unfiltered, the engine
does not carry $discount_code->code = $code (written in the handler) into
$this->code inside save() once other methods have been called on the
object; the one path it reports detours through a prepared query's result.
Recorded as the next recall gap: a cross-file field join keyed by class.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The core-hook-argument class from the second PMPro adjudication:
pmpro_membership_history_profile_fields($user) on `edit_user_profile` was
a route entry, so its parameters were attacker input -- and $user is the
WP_User WordPress loaded; its ID became an SQL injection.
A callback registered in a string-keyed registry is handed its arguments by
the matching apply call, not by the request. The mapper now records the
registration call on the entry (`registered_by`), regions carry it, and the
dispatch fact says which registrations DO hand a callback request-borne
arguments (`request_arguments`; WordPress: add_shortcode, whose $atts are
post content). Every other registered callback keeps its call depth but
its parameters stop being sources; request data that reaches an apply
call's arguments is still followed by the hook half of the taint query.
REST routes are not registry entries and keep the parameter rule.
Controls: the fact field is validated against `register`; a hook callback
is not input, a shortcode and a plain route are.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The apply_filters class from the second PMPro adjudication (2 of 3
remaining false positives): `apply_filters('pmpro_random_code', $code,
$this)` returned whatever $this carried, and a report's `$type` reached SQL
only as an extra filter argument. Two causes, both fixed generally:
- The dispatch fact now says what an apply call RETURNS: `returns` names
the calls whose result is the dispatched value and `value_arg` its
position (WordPress: apply_filters/apply_filters_ref_array return
argument 2; do_action returns nothing). A path through such a call must
ENTER by that argument -- the first of the run of the call's arguments
before it, because the engine hops between sibling arguments
($_GET["c"] -> md5("x") -> apply_filters). Loaded and validated as data;
the query knows no framework.
- A call was a source when the pattern appeared anywhere in its text, so
`apply_filters(..., $_GET["c"])` and `sanitize_text_field($_POST[..])`
were sources themselves -- the "a token matches the thing it names"
invariant. A call is now a source when the pattern names its callee,
receiver or the access it is; the inner access stays a source.
Controls: dispatch facts load/validate; a real-Joern case with request
data as filter context (quiet) and as the filtered value (found). 1,453
host and 18 query tests pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
da0ee57 turned parameter sources off for the mapper's catch-all `function`
entries, and with them the field join's seeds, which read the same flag:
MW WP Form's CVE-2023-6559 ($this->Data, filled by a constructor) was lost
(10 -> 3). The field join now has its own request field,
`fieldParameterSources`, set by the old entry rule; a region's own
parameters follow the new one. Both known CVEs are found again (MW WP Form
10 -> 4, WP Statistics 39 -> 18, 91/91 and 774/774 completed). The dropped
findings sampled against their call sites all take internal values:
ExistTable('useronline'), wp_statistics_visit('today'), remove() on the
plugin's own temp directory, _render('text', ...).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Two classes the traced PMPro survivors named after 1f91c0b:
- The mapper records every named PHP function as an entry of kind
`function` (3,507 of PMPro's 4,412), and every entry's parameters were
sources, so internal helpers became injections: sendCancelAdminEmail's
$user, a gateway hook callback's $user_id/$morder fired by do_action.
Regions now carry the entry kind; parameters are input only for request
entries (route, parser, privileged surface). Call depth is unchanged, so
such a region is still followed into its callees. JavaScript and Python
emit no `function` entries.
- The object rule is general: a field read continues a flow through the
bare object only if the taint came through the SAME field, or was
assigned to the variable or passed in as a parameter. It now also cuts
$this->Gateway -> this -> $this->user_id (memberorder.php:705).
plausible() memoises the sink test per node and skips the object walk for
paths with no field access (the filter had cost one outlier 101 s -> 160 s).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Traced element by element (new opt-in `trace` on taint requests), every
false PMPro SQL injection in the adjudicated sample took one of two steps
no real injection needs, both over-approximations of Joern's dataflow:
A. Through a query: `$id = $wpdb->get_var("...'" . $token . "'")` is the
injection; the engine carried $token on through the query's RESULT
into later queries (getMemberOrderByID:705, profile.php:515).
B. Through the object: request data written to $order->ProfileStartDate,
or passed as an argument beside $wpdb, tainted the whole object and a
DIFFERENT field read from it came out tainted
(memberorder.php:1122/1633, pmproemail.php:262).
A path is now dropped when it passes through a call to a sink of its own
family, or when a field read is reached through the bare object after the
taint entered it by writing another field or without being assigned to
it. A member of a value that IS tainted (the result of a read, a return,
an assignment) and a field read back after being written are kept. Applied
to the sink query, the field join and the hook join. A real-Joern control
covers both cuts and both keeps, and fails on the old query; all 17 query
tests pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reverts 6ebbc89 and e0c260c (keeping finding_dump's completed-regions
record). Measured on PMPro, comparing only questions both runs answered
(157): 37 -> 31 findings, but the six removed included the adjudicated TRUE
positive discount-codes.php:154 (REST-reachable), and only one of the six
sampled object-field false positives went away; five are still reported.
It also cut completed questions 295 -> 158. MW WP Form and WP Statistics
were unchanged either way.
The diagnosis was wrong: the object-field false positives do not come from
setter parameters. The witnesses name one field (e.g.
$update_order->BillingFrequency) and the query interpolates another
($this->membership_id), which points at field-insensitive propagation
through the object instead. To be confirmed on a flow path before any fix.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Asked of every parameter up front, 6ebbc89's caller check ran a repository-
wide flow query per parameter of every method in a file and cut PMPro's
completed injection questions from 272 to 67 in the same budget -- so its
48 -> 26 findings could not be read as precision. The flow into the field
is asked once as before; only a parameter at the head of an unsanitized
flow is checked against its callers. finding_dump now records completed
regions so runs compare only where both answered.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The field join (task 5.11) seeded every parameter of every method in the
file, so every setter made its field a source. Adjudicated on PMPro, six of
eight false SQL injections in a 1-in-5 sample were object fields holding
database or internal values ($this->membership_id, $morder->user_id).
A parameter now seeds the join only where some call site passes it request
data without a sanitizer on the way (memoised per method and index); a
request source read in the file still seeds as before. A real-Joern control
covers both directions -- a setter only given a constant stays quiet, a
constructor handed $_GET (the CVE-2023-6559 shape) is still found -- and
fails on the old query with exactly the PMPro false positive.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
48 findings, every 5th reviewed against source: 2 true (both privileged),
8 false; 6 of the 8 share one cause, object fields treated as attacker
input when they are loaded from the database. benchmarks/push/finding_dump.py
writes every finding of chosen families for review.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The loop the maintainer called out: four scale defects in the PHP transfer
replay, found one per two-hour Joern run, all Python bookkeeping between
engine calls. benchmarks/push/python_overhead.py ends that: the real
replay() on the real PMPro pins, with a JoernBackend whose graphs are stubs
and whose queries answer instantly with rows recorded from the real engine.
Engine time is zero, so everything measured is ours, in about a minute,
under the sampling profiler. Its first pass named two more walls:
- Region discovery re-parsed the ruleset for every region: 89,127 TOML
parses, 275 s of a 339 s discovery, twice per push. The fact, obligation
and taxonomy loaders now keep the parsed (immutable) result keyed by each
file's name, size and mtime -- an edit in place or an added file reloads.
Discovery 339 s -> 7 s; output digest identical on PMPro.
- affected_context decoded every span's and every line anchor's path for
every context row -- a changed file has an anchor per unchanged line --
78% of a stubbed push. Decoded once per call, anchors indexed by line and
bisected, original order kept. A randomised control compares it against
the row-by-row matching over 200 contexts (>100 attachments).
Result: the whole stubbed PMPro push, 27,542 answered requests and 2,688
outcomes, runs in 58.5 s of Python. What is left is the engine.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
b4b9308 was committed with two failing tests: the module audit read
`from . import profiling` as an import of the package, called profiling
orphaned, and the committed audit manifest drifted. My check read tail's
exit status, not pytest's. Now a from-import; manifest regenerated (109
load-bearing, 0 orphaned); 1,444 pass with pytest's own exit checked.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The cache fsynced every payload and manifest -- two per entry, ~18 ms, and a
push publishes one entry per question (~22,000 on a WordPress plugin). A
read already rejects any entry whose manifest receipt, payload size or
payload SHA-256 does not check, so a torn write is a miss, never a wrong
answer; the fsyncs bought nothing the verification does not.
The broader problem: three quadratic walls in this replay were found one
per two-hour run, because all were Python bookkeeping between engine calls
and the only record was the engine log. `OUSAST_SAMPLE_PROFILE=<path>`
starts an in-process sampler that rewrites the hottest stacks every 30 s,
so a run killed at its deadline still names where its time went. The host
cannot ptrace the container, hence in-process. The replay harness turns it
on per case.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The third wall under the 13.3 replay, found by the streamed case log: the
evidence and context passes finished at 14:53, then nothing -- no engine
call, no log line -- for 82 minutes, until the case was killed 60 s past
its 7,200 s deadline. `affected_context` tested membership with
`relation not in relationships` on a list: 7,998,000 comparisons for 4,000
relationships (profiled, 83% of the time), and a push over a WordPress
plugin attaches tens of thousands. The deadline is checked once per
question, so a single late question could run for minutes, which is also
why the process overran its cancellation allowance.
Membership now goes through a set beside the ordered list, and each
distinct context path is resolved once instead of once per row. Measured:
4,000 relationships 7.4 s -> 0.44 s; 400,000 in 16 s. The control counts
comparisons rather than timing them, and fails on the old code with
7,998,000.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The 13.3 replay on 7cf256b crashed on case 1: `pre-push` ran past its
7,200 s deadline plus the harness's 60 s grace, `subprocess.run` raised
TimeoutExpired, the harness died without running cases 2 and 3, and the
case's captured output died with it. Each case now streams stdout and
stderr to files with INFO logging on; an overrun kills the case's process
group, is recorded as `deadline_overrun` with its size -- the product
failing its cancellation contract -- and the next case still runs.
`--only` reruns selected cases. CLI log lines now carry a timestamp, so a
log can say where the time went.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The real cause of the 13.3 replay wall. Every cache publication called
`_make_room`, which listed and stat'ed every entry to total the cache size,
and a push publishes one answer per question -- 22,125 on a WordPress
plugin. Measured on the live replay cache: one scan 0.68 s at 12,238
entries, ~70 minutes spent scanning by then, ~2.3 hours to finish the
evidence pass. The engine was idle while the Python process sat at 83% CPU;
the 13/31/51-minute portions were the quadratic growing.
The scan is now remembered, keyed by the cache root's modification stamp:
our own publications and evictions update the tally in place, and any
change another process makes moves the stamp and forces a rescan, so the
size limit stays exact under concurrent pushes. Lock stripes are created up
front so their first use never invalidates the tally. Eviction order and
lease handling are unchanged.
Controls: 61 publications (one a same-key replacement) take exactly one
scan and the tally equals a fresh scan; a second writer on the same root
forces a rescan and the limit holds.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
`test_hanging_process_group_killed_inside_shared_allowance` failed about one
run in thirty under load with FileNotFoundError: it checked that
/proc/<pid>/stat existed, the killed process was reaped, and the read then
failed. A reaped process is the outcome the test wants, so it now reads and
treats a missing entry as gone. The engine-stage test had the same race.
150 of 150 runs pass under engine load.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The PHP transfer replay (pre-push 13.3) spent its whole 7,200 s case in the
head side's evidence pass. A probe on one PMPro graph named the cause: not
compute -- context adds ~20% engine time -- but OUTPUT. Each evidence request
listed every method its scope reached, up to 1,845 rows; 200 requests into
the bundled Braintree SDK returned 52,831 rows, 9.4 MB, against 211 rows
without context, over 22,125 requests in a 3 GB container.
`affected_context` only ever attaches a location in a changed file, and
every other row was checked (real file under the root, usable extent) and
discarded. The scan now sends the changed files; the engine itemises those
and folds the rest into one `context_elsewhere` row carrying exactly what
that check reads. With no safe file list the engine itemises everything,
as before.
Equivalence: unit controls over valid, missing-file, no-extent and
outside-root cases, with and without a changed-file row; a real-Joern
control; and on PMPro itself, 2,440 relationships and zero gaps decided
identically from both forms, 52,831 rows down to 3,241 with the most
referenced file as the change. Probe: benchmarks/push/context_cost_probe.py.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Same 500-region PHP run on frozen source: completion 281/406 against 153
(hard cap) and 131 (7.8), no portion killed, per-family findings never
lower, declared target still entailed. Records the 7.9 task, its limits,
and corrects the 7.8 reuse note, which was stated backwards.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The measured-cost run left 99 taint requests unasked to keep time for
arbitration and reported them as `query_failed` -- an engine that could not
answer -- because they were simply missing from the answer. They are now
`query_budget_reserved`, both as a degradation and as each question's outcome
reason, and are no longer double counted when nothing answered.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The 7.8 trace showed the completion loss was a sizer death spiral, not
expensive families: after two kills the budget fell to 3 sink visits and
never recovered, because portions finishing in 25-30 s sat under an assumed
70 s fixed start and read as "no information", so the budget could only
shrink.
- taint.sc streams each answer's own cost (`ms`); the backend carries it as a
reserved `__timing__` list through shards, partitions and reuse.
- The sizer budgets predicted seconds from per-family seconds-per-visit rates
learned from those costs, and measures the fixed start as the wall time the
answers do not explain (median), instead of assuming it.
- A kill names its culprit: the first unanswered id in request order. The ids
behind it never started and are re-queued at once; the culprit is asked
alone after the rest. One that still fails alone after more than a
portion's target is reported as `query_too_expensive`, apart from
`query_failed`.
- Portions grow id by id from the queue head, and a single-portion scan now
goes through the queue too, so a kill there re-asks its remainder.
- Fix: the reuse wrapper wrapped only `run_batch`, so since 7.8 every
portioned query bypassed the graph-reuse cache and its source-root
normalization. `run_batch_once` is wrapped too.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Final end-to-end on the 500-region PHP subject: findings identical at 125 either way, completion
slightly lower under portioning (131 of 406 vs the hard-cap-plus-reserve 153), wall time higher
(2,982 vs 2,338 s). Told straight: portioning does not beat the cap on throughput here; it matches
it on what it finds, and what it buys is kill-safety -- a killed batch keeps its streamed answers
instead of losing them. Reaching the same 125 findings is the evidence that streaming loses nothing
the cap kept.
Completion did not rise because the requeue re-asks a killed portion remainder, and on this subject
those are the expensive output-encoding questions that keep hitting the ceiling in smaller pieces,
spending budget without completing. Telling a transient kill from a too-expensive family is the next
lever, not more portioning. Neither approach is release-ready anyway: 32-38% completion against a
95% gate on development data that qualifies nothing.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Four interacting defects surfaced in sequence when the portioner was measured end to end, each
hidden by the one before: run_batch timing polluted by its own retry, a single-attempt path missing
on the partitioned graph every plugin uses, and killed remainders dropped by a fixed chunk list. All
fixed and unit-tested; the end-to-end completion number is re-measuring on a frozen tree.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The portioner iterated a fixed pre-split list, so a killed portion left its unanswered ids behind:
on the plugin the list ran out at 1,263 s of a 3,600 s budget with those remainders lost, and
completion sat far below what the budget could buy. It is now a work queue. A portion that comes back
short pushes its unanswered ids to the front, to be re-asked at the budget the kill just halved;
`fit` splits an over-budget head, so heads shrink toward single requests, and a single request a
whole ceiling could not answer is dropped rather than looped on forever. A control asserts both the
re-ask and that termination.
This is the last piece of portioning-over-cap: streaming keeps what a kill finished, single-attempt
timing keeps the sizer honest, coverage tells a too-big portion from a slow one, and the queue keeps
working while budget and work remain.
1,425 tests pass, nineteen skipped.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
run_batch_once was wired on the disposable and sharded paths but not the partitioned one -- and a
multi-language repository, which every WordPress plugin is, builds through the partitioned graph. So
the portioner fell back to run_batch there, whose internal split-retry folded into the time it
measured: a portion read 956 s and collapsed the sizer, exactly as before the single-attempt fix.
The fix had no effect on the one subject it was written for.
PartitionGraph now has run_batch_once, dispatching to each sub-graph the same way run_batch does --
the routing is factored into one _dispatch method that takes the per-graph method name. Sub-graphs
are built through the disposable path and already carry run_batch_once.
The lesson, again: a capability added at one construction site is not added at the others. There are
three that build a queryable graph -- disposable, sharded, partitioned -- and the one that mattered
for the measurement was the one missed.
1,424 tests pass, nineteen skipped.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
End-to-end, the adaptive portioner regressed completion -- 64 of 406 against 7.7 hard-cap 153 --
while finding the same defects. The cause: the portioner timed each portion with run_batch, which
splits on failure internally, so one portion folded its own eight-way retry into its number and read
953 s. The sizer fitted a per-visit rate from that and collapsed its budget to 14 visits, spending
the back half of the deadline on tiny portions.
Two portioning layers were fighting. The portioner is now the sole retry layer: it asks
run_batch_once, a single invocation whose kill returns the answers it streamed, so its wall time is
the cost of one attempt. And it observes COVERAGE, not emptiness -- a portion that came back short
of the ids it asked halves the budget, because it was too big for the ceiling; only a whole answer
gives a clean per-visit sample to fit from.
run_batch_once is new on CpgResult, wired on the disposable and sharded paths; a backend without it
falls back to run_batch unchanged. Host controls cover the single-attempt path and the
halve-on-short-answer rule.
1,424 tests pass, nineteen skipped. The end-to-end plugin re-measure on a frozen tree is the open
item; the mechanism (a kill keeps what finished) and the sizer logic are unit-verified.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The helper took a destination from the caller and rm -rf it. Re-running it into the same path while
a container had that path mounted deleted taint.sc out from under a live scan, and every request
after printed no such cpg query -- the live-mount hazard the helper exists to prevent, in a new
form. It now mints base/frozen-<commit>-<pid>-<n> itself and prints it, so reuse is impossible by
construction and the caller captures the printed path.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Measured on the plugin, first run of the adaptive sizer. A 600-visit portion answered in 184 s and
the budget was correctly cut to 262. The next portion, 199 visits, answered in 73 s -- three seconds
of work above the 70 s fixed start -- and 3 s over 199 visits read as 0.015 s each, so the budget
grew thirteen-fold to 3,404 visits. The ceiling killed that portion. The harvest kept the 13
answers it had streamed, which is the mechanism working; the sizing was not.
Two rules. A sample with less than ten seconds of work above the fixed start is not extrapolated
from, because it carries no information about cost per visit. And a budget may double per portion,
never more: the sizing may still be wrong, but by at most one factor of two, which a kill undoes.
1,423 tests pass, nineteen skipped; the sizer control asserts both rules.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The warning once read 22126 of 22125 requests answered in parts, because the census rides the
answer under its own key and was counted as a request. An instrument that miscounts by one casts
doubt on every number beside it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Every long run mounts src into a container, and mounting the working tree is a live mount: edits
made during a multi-hour run are picked up mid-run, and a measurement whose code changed underneath
it is not a measurement of anything. A five-hour PHP transfer replay was lost that way today.
The export has to be checked too. git archive piped into tar once produced a tree missing the whole
ontology -- every tracked file after groovy alphabetically -- and reported success, because tar
status in a pipeline was never read. Every scan on that tree died at its first fact load and looked
like a slow run for fourteen minutes.
So: archive to a file, extract, count against git, and refuse an export without the ontology.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The joern-gated check passes on the real engine: the taint query streams one line per request, and
the comment-sink and literal-destination controls hold under the streamed form. Two defects the work
exposed are recorded as fixed -- the unportioned evidence pass, and a harvest that read a timed-out
pipe before killing its writer -- and one bypass is recorded as a limit rather than fixed: the reuse
layer re-query of missing ids calls the backend batch directly around the sink-weighted portioning.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The evidence pass bypasses the sink-weighted portioning by design: it is the pass that PRODUCES the
weights, and per request it is cheap, 17,700 answered in 44 s. It was still one all-or-nothing batch
protected only by the ceiling. On a plugin push transaction that batch is 22,125 requests, and under
the CPU and memory contention of concurrent test runs it exceeded the ceiling and cost the whole case.
It now runs in portions of 4,000 requests, roughly ten seconds of work behind one JVM start, with
answers merged and the census kept from the first portion that carried it. Streaming keeps what a
killed portion finished; portioning keeps that kill from being the entire pass.
Task 13.3 records the attempt that exposed this honestly: the case deadline was exhausted with no
artifact, and the failure was mine twice over -- the container mounts src live and the backend was
edited mid-run, and the evidence pass was unportioned. Neither is the algorithm. The rerun uses a
frozen export of the committed source with per-invocation logging.
1,423 tests pass, nineteen skipped; mypy clean.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A hard per-invocation ceiling is all-or-nothing. A batch killed at 301 s lost every answer it had
already computed, and the retry paid a fresh JVM start per part to ask them all again: measured, 116
requests killed at 300 s, then eight restarts carrying the same everywhere-sinks in every part.
Three coordinated changes make portioning, not the cap, the thing that governs cost.
The taint query STREAMS. The census first, then one line per request as it is answered, flushed as
it goes, inside the unchanged fence. The other queries do not stream and need not; the driver
reads both forms, and a legacy single document parses exactly as before.
The driver HARVESTS a kill. The process group is terminated first -- a first cut read the pipe
before killing and a pipe whose writer is alive never reaches end-of-file; the suite own
hanging-engine control sat on that read for twenty-eight minutes -- then the pipe is drained and
every complete line becomes an answer, marked partial. The retry asks only the remainder.
Portion sizing ADAPTS. After each portion the observed seconds per sink visit, less the fixed
start, resizes the rest toward a 120 s target; a portion that did not answer halves the budget
rather than being fitted, because its true cost is unknown and above the ceiling. A portion cut
down to fit puts its remainder back at the front of the queue.
A census LINE carries nothing but the census. A first cut misread a legacy document that carried the
census beside its answers as a census line and dropped every answer it sat beside; a control caught
it. Task 7.8 in contributor-scan.
Verified: 1,423 host tests pass, nineteen skipped, including new controls for the assembler in both
forms and under a kill, for a retry that asks only the remainder, and for the sizer. Not yet
verified: the joern-gated streaming check and the plugin measurement, which wait for the transfer
replay to free the machine and are stated as open in the task.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Detection is not transfer, and the status says so: the comparison run is what decides M3. But the
target that task 13.2 could not reach for months is now found, and the reason is worth keeping where
the next reader will see it -- nothing about detection changed, the population became affordable.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
classes/class.memberorder.php:936:getMemberOrderByCode, injection. A value concatenated into SQL with
no prepare, which is the declared M3 target for PHP, found for the first time.
Nothing about detection changed. The population became affordable: tier-zero pruning scoped to its
partition, batches budgeted by the sink work they imply, a per-query timeout no longer cancelling the
scan, and a reserve kept for arbitration. Before those this population completed zero questions at
every budget tried.
At 500 regions with a 3,600 s deadline: 271 of 406 questions completed, 213 findings, 2,950 s, and
eight findings in the target file including the target itself.
The earlier bounded attempt failed for a reason no budget increase could fix -- 25 regions of 4,461
is 0.6 percent of the population and the target was never asked about -- so this sizes the replay
before spending hours on it. One scan at this budget is 2,950 s, a push transaction is two scans plus
discovery and preparation, so the three declared comparisons are roughly five and a half hours.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A census that reports what it found cannot say whether a defect it missed was examined and cleared or
never looked at, and that is the difference between a negative and a silence. It is also how a
bounded run budget gets sized without guessing: task 13.3 needs to know at what population the
declared PHP target enters scope, and asking the record is cheaper than binary-searching with
forty-minute scans.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
PMPro at 25 regions, which completed before this work and still does: 27 questions completed and 19
findings before and after, injection 1 and output encoding 18 in both. The batching changes what a
large population can afford and nothing about what a small one answers.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
On the 500-region population where every earlier run produced nothing: 153 of 406 questions completed
and 125 findings, inside the deadline. 380 requests carrying 5,807 sink visits split into 11 batches,
each answering in 80 to 197 s, none near the 300 s ceiling. Every family that can speak on this
subject now does, injection included, which is the one holding a true positive here.
Four defects, each hidden by the one before it, and three of them one rule.
Batches sized by request count, when cost is per matched sink.
Packing heaviest-first, which starved injection of all 114 of its questions.
Any per-query timeout cancelling the whole scan, abandoning batches with 33 minutes left.
No budget kept for arbitration, which removing that cancellation exposed: rows are not findings.
The reserve had to be checked per BATCH. Checked between query kinds it changed nothing at all,
because one kind is a single loop iteration that runs every one of its batches inside it.
What it does not claim: 253 of 406 questions still went unanswered and one batch still records a
timeout; the raw finding count is dominated by output encoding, the family already measured to be
imprecise on PHP; and the reserve share of a fifth is a first value, not a tuned one.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The reserve was computed once and checked between kinds, and one kind is a single iteration that runs
every one of its batches inside it. So taint spent the whole budget and the reserve only stopped the
cheap kinds queued behind it: 0 of 406 questions completed, 2,435 s of a 2,700 s deadline in queries,
identical to having no reserve at all.
The granularity of a budget check has to match the granularity of the work it guards. It is now
checked before each batch, and the batches left unasked are logged rather than silently dropped.
1,420 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Querying fills rows. Arbitration turns them into findings and the outcomes are assembled from that,
and neither costs the engine anything -- so a scan that spends its whole budget on queries reports
nothing at all, however many of them answered.
Found by accident, which is the honest way to tell it. A per-query timeout used to cancel the whole
scan. Removing that cancellation was right on its own terms and made the measured result worse: the
query phase then ran the full 2,700 s deadline to exhaustion and completed 0 of 406 questions, where
the cascade had stopped it at 958 s and left time to arbitrate 78 findings. The cascade had been an
accidental budget guard. This is the deliberate one.
The query loop now stops issuing batches with a reserve in hand: a fifth of what was left when
querying started, a sixty-second floor, and never more than half. The cap is not decoration -- held
back without it the floor swallowed a whole small budget and the query loop never ran at all, which a
control caught. A scan with a fifth of a second to spend must still spend it.
What the loop skipped is recorded as query_budget_reserved with the kind and the request count, not
left to look like a query that failed.
1,420 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Two different timeouts reached the same handler and got the same consequence. If the SCAN deadline is
gone, cancellation is right: stop everything promptly, which is what the push two-second gate
measures. If a single query merely exceeded its own per-query ceiling, the scan may have most of its
budget left and every other batch is independent and cheaper -- and cancelling took them all.
Measured on Paid Memberships Pro: one batch of 116 requests hit a 300 s ceiling 734 s into a 2,700 s
scan, and the four batches behind it returned in 0.0 s unanswered with thirty-three minutes to
spare. The handler already distinguished the two cases in the reason it recorded, deadline_exhausted
against process_timeout, and then set the cancel deadline in both.
Fourth instance this week of one rule, now written in the design: a failure is owned by the narrowest
scope that caused it.
1,420 tests pass, eighteen skipped, including the cancellation gate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A taint batch costs what its matched sink NODES cost, because reachableByFlows is asked per sink.
Measured at a fixed 100 requests on one plugin graph: SQL injection 147 s with six declared sinks,
path 194 s with sixteen, output encoding 704 s with seven. The declared count does not predict cost
and the request count does not predict cost.
So batches are budgeted in sink visits rather than requests, using the count the evidence pass
already produced: a family whose sinks are rare rides in one batch, and a family whose sinks are
everywhere is split until it fits. A request over the budget on its own gets its own batch rather
than being dropped, because the scan job is to report that it is expensive, not to decide it is
unaskable.
Measured on Paid Memberships Pro at 500 regions, where every earlier run produced nothing at all:
380 arbitration requests carrying 5,807 sink visits split into 12 batches, every one answered,
heaviest 103 s and none near the 300 s ceiling. 144 findings where there were none.
The order is the ranker and must stay the ranker. A first attempt packed heaviest-first, which packs
better and analyses worse: the expensive output-encoding batches ran first, spent the deadline, and
left injection with 114 questions asked and none answered -- the one family holding a true positive
on that subject. A deadline always cuts a tail, and the tail it should cut is the one the ranker put
last.
1,420 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Open since September and re-measured nine times. The cost of a taint batch is the number of CALL
NODES a family sink names match in the repository. Not the request count, not the declared sink
count, not any request parameter. reachableByFlows is asked per matched sink, and PHP
soutput-encoding sinks are echo and print, which a WordPress plugin writes thousands of times.
Three controlled sweeps on one graph, built once.
Batch size exonerated: 400 requests answer in 170 s, and cost per request FALLS with size because
roughly 70 s is fixed JVM start and graph load.
Request parameters exonerated: at 200 requests the full production shape, call depth 3 with
parameter sources, costs 221 s against a 150 s baseline.
The family is the variable: at the same 100 requests, SQL injection 147 s with six declared sinks,
path 194 s with sixteen, output encoding 704 s with seven.
And the driver named its own expensive step once it was asked to: 2,435 evidence requests in 45.5 s,
then 315 arbitration requests unanswered at 1,200 s, then a split abandoned for want of time.
It survived since September because every reproduction outside the driver carried the SQL sinks --
the family the declared target belongs to, and the one family that is cheap on this subject. Nine
hypotheses were eliminated by re-measuring the cheap case.
Splitting never rescued it because a failed batch is split by REQUEST COUNT and each part carries the
same everywhere-sinks. contributor-scan 7.7 carries the fix: budget a batch by the sink nodes it
implies, using the count the evidence pass already computes, and report a family whose sinks are
everywhere as expensive rather than as failed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Nine hypotheses about the PHP query cost have now been eliminated by rebuilding the request outside
the driver: batch size, call depth, parameter sources, hook table, argument length, sharding,
dataflow cost, the malformed-CPG bug and the engine version. Every one of them answered in under
four minutes while the driver spent whatever ceiling it was given.
The reason the guessing continued is that a scan issues several invocations and the record could not
say which of them burned the budget. It now logs the query kind, the request count, the elapsed time
and whether it answered, for the whole batch and for each part of a split retry, including the part
where a split is abandoned for want of time.
This is task 16 applied to the backend rather than to a harness: an instrument that cannot name its
own expensive step turns a measurement into a guess, and this one has been a guess since September.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Four census runs on this subject spent whatever query ceiling they were given: 363 s of 300, 1,255 s
of 1,200, 1,874 s of 1,800, at question counts of 2,526 and 406. Cost that tracks the budget rather
than the work is not per-question cost, and tier-zero pruning cutting the count six-fold changed
nothing. A census runs one batch, so no census could answer whether the batch itself scales.
One graph, built once, five batch sizes, every one answered:
25 requests 71.4 s 2.86 s each
50 requests 76.0 s 1.52 s each
100 requests 155.4 s 1.55 s each
200 requests 166.7 s 0.83 s each
400 requests 169.7 s 0.42 s each
Sublinear, and roughly 70 s of it is fixed JVM start and graph load. Four hundred dataflow requests
answer in 170 seconds on the graph production cannot finish 406 on.
So the difference is in the request PARAMETERS. The probe sent callDepth=0 and no parameter sources;
production sends callDepth=3 with parameter sources on every entry point, plus a hook table, a
dispatch list and context evidence. Task 13.2 recorded in September that every reproduction outside
the push path answered in under a minute and the difference was not known. Size is now removed as a
candidate and a four-variant probe is running to isolate the rest.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The mechanism already existed. A question whose evidence vector says no sink of its family is in
reach cannot produce a flow, and tier-zero pruning defers it. That was switched off for the whole
scan whenever any of twelve degradation reasons appeared, and a multi-language repository always
carries two of them, so the measured plugin arbitrated every question it could ask. Third instance
this week of one blast radius, and the invariant written into the design on Monday named it.
A prune blocker that names a partition now withholds pruning from that partition questions and from
nobody else. cross_partition_semantics_unresolved leaves the blocker set entirely, for the reason it
left the integrity gaps: a PHP family sink is PHP, so a second language cannot be why its vector saw
none.
Measured on Paid Memberships Pro. At 25 regions arbitration falls from 127 questions to 34 with 93
recorded deferred as tier_zero, and 19 findings before and after, identical family by family. At 500
it falls from 2,526 to 406 with 2,120 deferred.
It is not sufficient, and that is the useful half. The 500-region run still fails in the same way at
the same cost, because 406 questions at roughly one and a half seconds each still exceed the
300-second per-batch ceiling. This lever moved the volume; the ceiling is a separate limit.
The census now records what a scan skipped and why, per family, so a cheaper scan and a narrower one
stop looking identical: a family whose every question was pruned reports no sink of this family in
reach rather than region offered, no question asked.
1,420 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
NodeGoat models 18 of 175 distinct call names, covering 12 percent of 1,185 calls. That share is not
a score: most of what a program calls is not security-relevant, and a high number would mean the
tables were too loose. It is a baseline to move deliberately.
The top product candidate is res.render, 23 calls with five in a method that reads untrusted input.
Task 14.3 reasoned its way to that same gap by hand and declined to model it as a flow sink; the
instrument found it in one run, which is the validation that matters. It also surfaced Math.random in
the user data-access module, and weak randomness is a weakness class this ontology has no family for
at all -- so no family census could have found it, because the family list is itself the blind spot.
That is the kind of gap only a coverage instrument finds.
It had to be taught what is not an API. The first run ranked _tmp_3.push and __ecma.Array.factory
first, which is how the frontend lowers a destructuring and an array literal, and a Cypress suite and
a Gruntfile held four of the next five. Both are separated now, because noise at the top of a
worklist is how a worklist stops being read.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Five of seven families established nothing in the family census, and every one failed for an ontology
reason with the flow engine ready and nothing to aim at. Each was found by investigating one family
at a time over days. vocabulary.sc asks it once: census the calls, subtract everything any fact table
names, rank the remainder by how often it is called and whether it sits in a method that reads
untrusted input. It needs no dataflow, because the structural pass already walks every call.
The harness reads its token set from the SPECS rather than from the TOML, so it measures what the
analyser actually consults: a fact the loader drops is not part of the ontology however neatly it is
written. Ontology sizes today are 152 tokens for JavaScript, 167 for Python, 84 for PHP.
Both carry the constraint the maintainer set: the output is a candidate list, never an ontology. A
name becomes a fact only when someone resolves it to a library API, states the weakness class and
records provenance, because adding a name because a subject called it is how a general-purpose tool
becomes a fixture of its own benchmark.
Also records what the bounded PHP transfer attempt found. At 25 regions the first case now runs to
completion and writes a full artifact in 1,706 s, where at 100 it had exhausted a 2,400 s deadline
with nothing. But 25 of 4,461 regions is 0.6 percent of the population and the declared target was
not among them, so the question was never asked. Task 13.3 therefore depends on 7.3, skipping
arbitration where the evidence says there is nothing to find: raising the budget is already measured
to fail, and making most of the budget unnecessary has not been tried.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Named by a week in which three separate causes looked identical as one count of unresolved. The
census reported 19 findings and 0 completed questions side by side without complaint, a stale image
measured the engine a bug had been diagnosed against, and a partition fix looked applied while an
unscoped copy of the same gap kept demoting everything.
Every harness record should carry the counts that disambiguate its failure modes, the degradations
whole rather than by reason, and the engine and ruleset identities actually used; and each harness
should assert its own consistency, because findings above zero with completion at zero is a
contradiction an instrument can notice faster than a person.
This was missed from the previous commit by an escaping error in the edit, not by intent.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The source, sink, sanitizer, operation, discharger and layout rows are not a pattern list. They are an
ontology of program elements and their security relations, and nearly all of this tool accuracy lives
there: the family census found five of seven families establishing nothing, every one for an ontology
reason with the flow engine ready and nothing to aim at.
The hook is general-purpose, not application-specific. The ontology describes libraries, frameworks
and language constructs; it does not describe repositories. An entry names an API and is justified by
what that API does, carries provenance and a weakness class, and may not encode a path or identifier
peculiar to one project -- a plugin bundled library tree was measured and declined on exactly that
ground. A measured gap is a candidate, never an entry: accretion by subject is how a general tool
becomes a fixture of its benchmark.
contributor-scan owns the ontology, so its design carries that statement and a new group 7 makes
growth measurable rather than anecdotal: report what a repository calls that nothing models, key
sanitizers to the weakness they prevent, skip arbitration where the evidence says there is nothing to
find, share one matcher across the queries, require a degradation to declare its scope, and commit
golden findings so a silent change cannot pass.
pre-push-safety-net carries the general-purpose test for its own reviews, and a task for harnesses
that report the cause of their own failure -- named by a week in which three separate causes looked
identical as one count.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Task 13.4 found that a second language in a repository cost every question its completion, and so
cost every finding its admission, on the declared v0.1 target: a WordPress plugin is PHP plus its
admin JavaScript. Neither subject that passes today could reveal it, because both hold exactly one
language. A population of single-language repositories would qualify a capability that fails on the
product own target.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
_blocking_degradations already exempted declared exclusions. It now also ignores a gap another
language partition raised, because a JavaScript file missing from the JavaScript graph cannot make a
PHP answer wrong. Treating it as though it could left a WordPress plugin -- PHP plus its admin
JavaScript, which is every plugin -- unable to admit a finding it had already established.
A gap that names no partition still reaches everything, and so does every gap when the claim own
language is unknown, because unknown scope is not evidence of independence. The control asserts all
four directions.
With this and the scan fixes, task 13.4 is verified on the subject that motivated it: PMPro completes
120 of 127 questions where it completed none, its 19 findings unchanged, and the seven still
unresolved are the JavaScript partition own questions, which is the correct answer.
1,420 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The scoping fix looked applied and was not. The same census gap is raised in TWO passes, arbitration
and evidence, and only the first carried the partition it came from, so an unscoped copy kept
demoting every question in the repository. One key in one dict.
What found it was the record, not the code: the census now prints each outcome reason rather than
only its status, and three separate causes had been looking identical as a bare count of unresolved.
Measured on Paid Memberships Pro at 25 regions: questions completed goes from 0 to 120 of 127, with
its 19 findings unchanged. The seven still unresolved are the JavaScript partition own questions,
which is correct, because the four missing files are its files and its answers are the ones that
could be wrong.
Still open, and recorded rather than implied: the push policy _blocking_degradations exempts declared
exclusions but does not yet respect a partition, and a post-hoc diagnostic copy carries no language,
so a JavaScript gap would still block admission of a PHP finding.
1,419 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A missing JavaScript file cannot make a PHP answer wrong. A census gap now carries the partition it
was raised for, and reaches only that partition questions. On the measured plugin four JavaScript
files -- a webpack build output, a minified select2, a build config and one source file -- had taken
the admissibility of 4,425 PHP regions with them.
Two controls exercise the real boundaries rather than injecting a reason, and both were confirmed to
FAIL with the old classification restored, so they guard what they claim to.
And the rule is now in the design, because this was the fifth instance found and fixed one at a time.
Two invariants, eleven instances between them:
A degradation is owned by the narrowest scope that caused it. Vendor exclusion as a global veto,
line ambiguity inherited by every question, an empty scope blocking a sibling comparison, a language
boundary demoting every answer, a census gap demoting every answer. The fourth and fifth cost the
declared v0.1 target its entire admissibility, and neither passing subject could reveal it because
both hold one language.
A token matches the thing it names, not text that contains it. gets inside fgets, resolve inside
resolveUrl, user inside users, a sink word inside a comment, a dotted setting on its last segment,
a flow query reading the receiver.
Each carries a review test: a new degradation must state the scope it reaches and justify demoting
that far, and a text match must be bounded AND confined to the part of the node that names the callee.
1,419 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Membership of GRAPH_INTEGRITY_GAPS should turn on whether a gap can make a reported answer WRONG,
not on whether it cost coverage. An unparsed file may hold the sanitizer that would have cleared a
flow this scan reports, so it belongs there. A flow this tool cannot follow from one language into
another can hide a finding and never invent one, because a PHP sink is sanitised in PHP.
cross_partition_semantics_unresolved therefore moves to DECLARED_EXCLUSION_GAPS. It touches exactly
two places, both of which already honour the distinction: the outcome demotion in scan.py, and
_blocking_degradations in the push policy, whose docstring already says a declared exclusion is not
one of them. The reason stays published in the scope unresolved boundaries either way, so the limit
is still reported; the class decides only whether every answer in the repository is demoted.
This corrects the task note written earlier today, which said not to reclassify on the grounds that
it would claim cross-language flows are out of scope rather than unresolved. That was a misreading
of the class.
PMPro still completes none of its questions, and the measurement now names why: the JavaScript
partition expected 36 files and holds 32, so partition_file_census_incomplete fires and is still an
integrity gap. The four absent files are js/blocks/blocks.build.js, js/jsapi.js, js/select2.min.js
and webpack.config.js -- a build output, a source file, a minified third-party library and a build
config. A bundled select2 costs 4,425 PHP regions their admissibility. Same over-broad blast
radius, third instance of one principle in this thread, and the fix direction is recorded.
The census harness now records each degradation whole rather than only its reason, because a
partition census that reports files missing carries WHICH files and without them the gap is a label
rather than a lead.
1,417 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
No PHP question is ever marked completed, and PHP was never the variable. partitions.py raises
cross_partition_semantics_unresolved whenever more than one language partition exists, that reason
is a member of GRAPH_INTEGRITY_GAPS, and a graph-integrity gap demotes every completed outcome in
the scan. Admission requires a completed outcome.
The evidence is one column wide. NodeGoat is JavaScript alone and completes 299 of 299. VAmPI is
Python alone and completes 35 of 35. PMPro is PHP with 36 regions of bundled JavaScript and
completes 0 of 127 and 0 of 520 while establishing 19 and 43 findings. NodeGoat own degradation,
vendor_semantics_unresolved, is a DECLARED exclusion rather than an integrity gap, which is exactly
why it does not demote.
This lands on the declared v0.1 target. A WordPress plugin is PHP plus its admin JavaScript, so
every one of them is multi-language, so on the target this tool was built for it can detect defects
and never admit one. Neither subject that currently passes could reveal it, which is why a census
across a third subject was worth running.
Recording the boundary is right: a flow from PHP into JavaScript is semantics the partitioned graph
cannot follow. Demoting every question is not, including one wholly inside a single PHP file. Task
12.3 already fixed this bug class one level down by making a gap question-owned so it could not
block another question comparison; the same principle applies to a partition boundary. Reclassifying
the reason as a declared exclusion is explicitly the wrong fix, because that would claim
cross-language flows are out of scope when they are unresolved.
Task 13.4 rewritten with the diagnosis, and the milestone amendment now ranks it with M2 rather
than behind it: a narrow JavaScript-only release can proceed without it, the WordPress target
cannot proceed at all.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The replay hardcoded 500 regions, which on a 673-file PHP plugin asks 2,526 questions and fits no
ceiling this project sets. The budget is now an argument, defaulting to 500 so every earlier
measurement keeps its shape.
It bounds the RUN, never the repository: the artifact census still reports what was left
unexamined, and a bounded population establishes transfer rather than coverage. Task 13.3.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Four decisions, recorded after the family detection census and the PHP correction. They change
sequence and declared scope; they change no acceptance threshold and no owner boundary.
Close M3 at a scale whose questions complete, before anything else. The not-established PHP record
came from two instrument defects rather than from the analyser, and bounded to 25 and 100 regions
PMPro completes with no timeout and establishes a true-positive SQL injection. Tasks 13.3 and 13.4
carry the replay and the question-completion gap, the second because admission reads outcomes and
M4 measures completion at 95 percent.
Reserve the untouched M4 population now, in parallel. M4 already permits preparation before the
frozen candidate exists and it is the longest lead-time item on the path. Task 15.1.
M2 is the only engineering item left and cannot be scoped away: a changed-code transaction spends
its whole budget in the build, and meeting the latency gate by analysing less is explicitly not
the same result.
Declare the v1 advisory scope as the families whose claims can be ADMITTED, which requires the
entailed rung: injection, untrusted destination and configuration. Access control is excluded
because on an application that splits route from data access the obligation is carried and a
carried obligation corroborates by design, so the family would ship as silence. Task 15.2.
The risk the amendment does not remove is stated with it: every number so far comes from subjects
used to build the analyser, one week of examining two of them surfaced seven false positives, and
the first independent evaluation should be expected to fail.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
PHP is not silent. Paid Memberships Pro establishes findings, including a true positive, and
this project's own instrument is why it looked otherwise.
The census harness ordered its verdict on completion before findings, so a family with eighteen
findings and no completed question reported "asked but no question completed". Findings are now
checked first: a question-outcome count is bookkeeping and cannot hide the thing being censused.
NodeGoat and VAmPI were never affected, because they complete every question and their verdicts
reached the findings check. The scan summary now also carries the total findings and the outcome
status counts, so the two numbers can be seen disagreeing instead of one silently winning.
The second mistake was mine rather than the harness's: only the 500-region runs had been looked
at, which are the ones that time out.
What PHP actually detects on PMPro. At 25 regions, 19 findings in seven minutes with no timeout;
at 100 regions, 43 findings in twelve minutes with no timeout. Among them adminpages/
discountcodes.php:749, where sanitize_text_field($_REQUEST["s"]) reaches
$wpdb->get_results($sqlQuery, OBJECT) through `$sqlQuery .= "WHERE code LIKE '%$s%' "` with no
prepare. That is the class of CVE-2023-23488, which php.toml's own comment names as the reason
those WordPress sanitizers are kept out of the sanitizer list. The prediction and the measurement
agree.
The precision cost beside it is measured too: output encoding reports eighteen findings at 25
regions and the three sampled are all properly escaped, two with esc_attr and one coerced with
intval. Those escapers are absent from a sanitizer list that is flat across families, because
none of them stops SQL injection. Keeping the injection finding costs this family its precision.
The scale limit is roughly one to two seconds per question. 25 and 100 regions complete cleanly;
500 regions asks 2,526 questions, needs more than the 1,800-second ceiling and is killed.
Four things are measured NOT to be the problem: the malformed-CPG bug (fixed in 4.0.625, verified
on the real trigger file), the dataflow overlay (42 s on the full 637-file graph), the engine
version (4.0.625 and 4.0.631 indistinguishable), and the bundled Stripe, Braintree and Twocheckout
SDKs under includes/lib that the vendored list does not cover -- that was my hypothesis for the
cost and the overlay measures 42 s with them and 37 s without, so it is refuted.
What remains is bookkeeping, not detection: questions_completed is zero in every PHP run including
the ones that find things, with cross_partition_semantics_unresolved and
partition_file_census_incomplete present throughout. The admission policy reads those outcomes.
1,417 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The graph bug is fixed and was never what these runs were hitting. joernio/joern#6281, this
project's own report, is a PHP `global` inside a closure giving a node two AST parents, which made
the dataflow overlay throw and cost the whole graph. Filed against 4.0.623, fixed by #6270 in
v4.0.625, which the image has pinned since before any measurement recorded under task 13.2.
Verified today on the real trigger rather than a transcription of the repro, which does not
reproduce: PMPro's includes/fields.php builds and applies the default overlays cleanly on 4.0.625
and on 4.0.631, files=2 methods=158, no ReachingDefPass assertion.
What remains is runtime. A taint batch of 2,526 questions over the 673-file checkout does not
complete in thirty minutes. Raising the query ceiling six-fold, from 300 to 1,800 seconds, changed
nothing but how long the failure took: query time went from 363 s to 1,874 s, and zero questions
completed in both. The eight-way split retry did not rescue it either, finishing its parts in
about three minutes against a deadline with 3,300 seconds still on it, which is worth its own
look.
The engine pin stays at 4.0.625. An image on 4.0.631, today's release, was built and measured and
is indistinguishable on this subject: 627.7 s against 619.1 s, 363.4 s against 367.3 s of query
time, zero completed in both. There is no demonstrated benefit to weigh against re-verifying every
JavaScript and Python measurement on a new engine. Should that change, the bump is three values
rather than a port: the upstream AstGenRunner.scala is byte-identical in 4.0.631 and its patch
anchor still appears exactly once, so only the jar path, the jar checksum and the source URL tag
move. Both are recorded.
Also closes task 14.5. Neither prototype pollution nor deserialization has a subject that can
answer, each for its own reason. Deserialization's mechanism exists on exactly one available
subject and that subject cannot be queried at all. Prototype pollution has no available subject
containing the mechanism, so its ability to answer is established on a control instead.
1,417 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Across four WordPress plugins and 1,164 PHP files there is exactly one deserialization site, and
it is spelled maybe_unserialize. The bare token never reached it, and correctly so: the sink
matcher is word-bounded and the character before unserialize inside maybe_unserialize is a word
character. A token a repository never spells is a family that can never answer, which is what the
2026-09-19 census recorded for this one. maybe_unserialize is a real entry point rather than a
safe wrapper: it calls unserialize whenever the string looks serialized.
Also adds the prototype control. That family had never established anything on any subject, and
its only census report was the word `set` inside a comment. No repository available to this
project contains the mechanism: the only first-party matches on the Node subject are
app.set("view engine", "html") and its sibling, Express settings whose arguments are literals. So
the control demonstrates what the family can do, on a merge of the request body into an object,
and asserts the Express setting is not reported alongside it.
Corrects the PHP record under task 13.2. joernio/joern#6281, this project's own report, is a PHP
`global` inside a closure producing a node with two AST parents, which made the dataflow overlay
throw and cost the whole graph. It was filed against 4.0.623 and fixed by #6270 in v4.0.625,
which the image has pinned since before any measurement recorded here. Verified today on the real
trigger rather than on a transcription of the repro: PMPro's includes/fields.php, the file the
original bisection found, builds and applies the default overlays cleanly on 4.0.625 and on
4.0.631 alike. What remains under that task is a runtime failure and must not be re-diagnosed as
the graph bug.
1,417 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
NodeGoat's server.js configures a permissive session at line 78 and disables template
auto-escaping at line 135, both inside the same function. The arbiter returned one verdict per
question, so the escaping defect that task 14.3 added was detected on every run and reported on
none: which of two real defects a contributor saw depended on where in the file it sat.
Configuration now returns one verdict per setting whose value the model could read. The pipeline
already emitted one finding per entailed answer, so this arbiter was the only place still folding
them to one.
Three boundaries are drawn rather than assumed.
- A setting whose value is COMPUTED stays one verdict per region. That band is a question for the
judge rather than a finding, and returning every undecided setting would multiply model calls
for claims the graph did not establish.
- A call carrying two permissive literals is one finding. cors({origin: true, credentials: true})
is one misconfiguration, not two.
- Config verdicts now carry a path:line:function location. Without one they shared the region's
identity and the deduplicator kept a single finding, so the plural answer collapsed straight
back to one. That is the same gap dominance carried until its findings stopped arriving with no
line at all.
Dominance is left answering once per region, and that is settled rather than pending: its claim is
that a region contradicts itself, so the region is the unit and a second verdict would restate the
same inconsistency. The pipeline now says so where it used to call this a known gap.
Measured: NodeGoat's config_secrets goes from one finding to two, server.js:78 for the session and
server.js:135 for the disabled escaping, with the applied marked.setOptions fix between them
staying silent. VAmPI is unchanged and no other family moved.
1,417 tests pass, seventeen skipped; thirteen Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Two of NodeGoat's three corroborated claims were correct code. `const { userId } = req.session`
lowers to a temporary and then a field read, so nothing in the graph spells session.userId by the
time the value reaches dao.getByUserId(userId, ...), and the constraint the handler really applies
was invisible.
The arbiter now walks back three hops over the method's own assignments from the names the
operation is GIVEN, and req.session joins the identity sources. The token is only safe because of
that requirement: what discharges an object-level obligation is that the value passed to the
operation comes from the authenticated context, not that a session is mentioned nearby. A handler
that reads the session for logging and keys its query on a path parameter is the bug this family
exists for, and it stays unguarded.
Two exclusions the measurement forced, each caught by a page whose whole purpose is the leak it
demonstrates.
- A callback is not a constraint. Taking identifiers from the whole argument subtree credited
NodeGoat's memo listing with a discharge, because memosDAO.getAllMemos((err, docs) => ... userId
...) reads every memo in the collection and only RENDERS the session's user. Method references
are excluded from the argument names.
- An assignment is not a dominating guard. The session read dominates the query, so that clause
credited it instead and the memo listing went quiet again. An assignment is a read of the
authenticated context, not a check applied to it, so operator calls are excluded there -- the
same exclusion the sink matcher needed. The leak stays reported.
On NodeGoat displayContributions and displayProfile go silent, the allocations insecure direct
object reference, both benefits pages and the memo listing stay, and the login and signup handlers
become honestly unguarded rather than accidentally cleared by a callback's text. Six corroborated
where there were seven, and a better six. VAmPI is unchanged at two entailed findings.
The sibling scope is decided by measurement and kept at the file. On NodeGoat the region's own
file entails two handlers, both public routes and both false, and no true defect at all. Every
handler in the application entails six: four true, and the same two public routes. Both scopes
carry the same false positives, because neither can tell a public route from a protected one --
a semantic question that is already the corroborated band's residual. With the carried rung capped
the scope changes no verdict, only the witness, so widening it would buy a repository-wide walk
per question and no precision. A public-route discriminator would change that; the file scope is
not what holds this family back.
1,415 tests pass, sixteen skipped; thirteen Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
NodeGoat's documented insecure direct object reference reads with
allocationsCol.find(searchCriteria()), and `find` was absent from an operation table holding
findOne, findById, findAll and findByPk. The obligation was missing at the commonest read there
is, so that defect raised nothing at any point in the pipeline.
It was absent for a good reason, and the reason is the argument, not the name. Measured on
wp-statistics, a WordPress plugin that ships jQuery: its assets hold 88 calls named find, of
which 70 pass a jQuery selector, 13 pass a predicate to Array.prototype.find and 5 pass a
selector built at run time by a template literal or a concatenation. None is a collection read.
So the obligation is claimed only where the first argument has the shape of a query document.
The operation fact states which argument with query_document_arg, and the arbiter that can see
argument shapes decides it: no arguments at all, an object literal, or a builder call are
claimed; a method reference, a literal, an operator call and a bare identifier are not. Each
exclusion is one of the measured classes above rather than a guess.
Zero of those 88 calls became an obligated operation, and the zero is measured rather than
empty: the same run's census reports 60 files and 4,552 methods, and the files holding those
calls are in it.
On NodeGoat all three find calls are collection reads and all three are claimed, which raises
the obligation at four more handlers -- the allocations handler that takes userId from req.params
on an authenticated route, the benefits pages whose isAdmin guard is commented out in the route
table, and the memo listing. Seven of fifteen handlers now reach a verdict, all corroborated,
none reported without a judge, and the census is unchanged on both subjects.
A bare identifier is deliberately not claimed: col.find(query) and arr.find(pred) are the same
shape and neither repository measured holds one, so this is a stated false negative rather than
an unmeasured false-positive risk.
1,415 tests pass, sixteen skipped; twelve Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The census measured this family completing 35 questions on NodeGoat with no obligation raised
anywhere: every modelled data operation lives in app/data/*-dao.js, none in a route handler
file, and operations were scoped to the region's own file. The family had nothing to be right or
wrong about.
The obligation now follows the one named call the handler writes, out of the region's file and
no further. How it is followed is the point. jssrc2cpg does not resolve
allocationsDAO.getByUserIdAndThreshold(...) to the DAO body: the callee is an external stub with
no code. That stub's fullName records where the member was written, so the implementation is
reached by the graph's own provenance rather than by a repository-wide name guess, which is the
false positive the file scope was closed against. A control asserts it: a third module defining
a member of the same name that nothing imports is not attributed.
A row is attributed to the region's DECLARED function, which is what lets a question about
displayAllocations match a graph that holds a <lambda>. A carried row is also checked for a guard
at the CALL, because that is where a handler establishes who is asking. The method holding a
carried operation joins the question's context scope, so the module holding the query cannot
change under a claim that depends on it without the change being attributed. The witness names
both sites and the finding is located at the call, which is the line a contributor changed.
A carried obligation corroborates, it does not entail, and that was measured rather than
assumed. Entailing it reported handleSignup, which looks a user up by name to check the name is
free during registration and is public by design, because two authenticated siblings in the same
file pass req.session.userId into the same module. The entailed rung's justification is that one
FILE contradicts itself, same file and same trust context; carried, the compared operations live
in a data-access module that handlers of different trust contexts share. The corroborated band
already asks the right residual: does this operation require a check, or is the route
legitimately public?
Measured on NodeGoat: 15 handlers asked, obligations raised at zero before and six now, three
reaching a verdict, all corroborated, none reported without a judge. VAmPI unchanged at two
entailed findings, because it keeps handler and query in one file and carries nothing.
What this does not reach is recorded, not implied. NodeGoat's documented insecure direct object
reference still raises nothing, because its module reads with allocationsCol.find(...) and find
is not a modelled operation: Array.prototype.find and jQuery's .find are everywhere, so it needs
an argument-shape discriminator and its own measurement (task 14.8). Two of the three corroborated
claims are correct code whose identity read a destructuring spelled away, and the cross-file
contradiction that would entail the real defect needs a sibling-scope decision (task 14.7).
1,415 tests pass, fifteen skipped; eleven Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The census recorded this family completing 44 questions on a Node application and finding
nothing, which is what a client-only vocabulary must do there: all six declared sinks were DOM
sinks. NodeGoat's cross-site scripting is server-side.
Decision: the family asks about a value reaching a rendering call, and states in its own
declaration that it cannot decide template-mediated escaping. Whether res.render is safe depends
on the engine's autoescape setting and on the template's markup, neither of which is at the
call, so res.render is not a sink and the family's limits field says so. A new CWE-79 sink
covers res.write, res.end, res.send and the response.* spellings; res.json stays out because an
application/json body is not a document.
Template auto-escaping is claimed where its evidence is, as a configuration defect:
swig.setDefaults, nunjucks.configure and marked.setOptions are security settings now.
It establishes a finding: app/routes/research.js:25, model_entailed,
`req.query.url + req.query.symbol -> res.write(body)`. The handler fetches a URL the caller
chooses and writes the response body into a text/html response, so the document served from
this origin is the attacker's. The record states how coarse the step is: the path from url to
body runs through the client's callback parameter, which the graph treats as reachable from the
call's arguments, so the conclusion rests on the call's semantics rather than on what the graph
established. Five of seven families now establish findings.
Four defects were found on the way, three of them false positives this change would otherwise
have shipped.
- The permissive-value set was shared across every setting, while a boolean's polarity belongs
to the setting: true leaves the door open in cors({origin: true}) and closes it in
swig.setDefaults({autoescape: true}). Adding the template engines would have reported the fix
as the defect on the line that offers it. ConfigSpec now carries per-setting values, and
Jinja2's Environment moved out of Python's shared list for the same latent reason.
- config.sc matched a dotted setting on its trailing segment alone, so marked.setOptions would
have claimed an editor's setOptions. dominance.sc already required the qualifier.
- taint.sc asked its flow query of every argument including the receiver at index 0. On a
response sink the receiver is the response object, so res.end() -- which takes no data -- was
reported exactly like the res.write(body) two lines above it. Flows now use the real
arguments, as arity already did.
- The arbiter sorted a file's settings by line number as TEXT, so line 126 preceded line 78 and
which setting a file reported depended on the spelling of its line numbers.
Template escaping disabled at server.js:135 is detected, measured model_entailed against the
real query and the real arbiter, and still not reported: the arbiter returns one verdict per
question and that file has an earlier permissive setting. Task 14.6 carries it rather than
hiding it.
1,415 tests pass, thirteen skipped; nine Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The family census recorded this family completing 44 questions on NodeGoat, finding the open
redirect, and having no way to ask about the repository's documented SSRF: needle.get on a
query-string URL at app/routes/research.js:16. A thin vocabulary is a silent false negative.
One data change, no query change. The declared set now covers axios, got, needle, request,
superagent, unirest, undici and fetch as bare module tokens, plus http.get, http.request,
https.get and https.request dotted. Bare is the documented call form of each module and covers
its verb methods too, since a token matches word-bounded against the callee and axios is
bounded inside axios.get; axios.get and axios.post are dropped as subsumed rather than
unwanted. http and https stay dotted because their namespace also holds createServer, and a
bare token would have claimed every server a repository starts.
The request is now reported at app/routes/research.js:16, model_entailed, witness
`req.query.url + req.query.symbol -> needle.get(url, ...)`. This family goes from one finding
to two on NodeGoat, every other family is unchanged, and 299 of 299 questions still complete.
Measured on that graph, needle matches exactly one call and the eleven other clients match
none in first-party or vendored code, so NodeGoat shows the gap closing and cannot show those
clients' noise.
A control asserts both halves. A handler calling axios.get with a literal and needle.get with
req.query.url reports the tainted client alone, and it reads the shipped vocabulary rather than
a copy, so removing the client fails it. The CWE-601 redirect entry is untouched: a
Location-header redirect is value-shaped and stays outside what this family asks.
1,411 tests pass, eleven skipped; seven Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Seven vulnerability families are declared and two had been proven. The census asks each of
them, on whole checkouts of NodeGoat and VAmPI, whether it establishes anything at all, and
reports a family that answered and found nothing separately from one that was never asked.
Four establish findings: injection, access_control, config_secrets and untrusted_destination.
Output encoding, path, prototype and deserialization establish nothing on either subject. All
299 NodeGoat and 35 VAmPI questions completed, so the silences are answers.
Three of those silences are diagnosed gaps, not clean code. NodeGoat's documented SSRF calls
needle.get, which the sink vocabulary does not hold. Its XSS is server-rendered, and output
encoding declares only DOM sinks, so no question it can ask could have found it. Its two
access-control defects raise no obligation at all: measured with the real dominance query,
every modelled data operation lives in app/data/*-dao.js and none in a route handler file,
while the family scopes operations to the region's own file. Tasks 14.2 to 14.5 carry each
gap; none is fixed here.
The census also found a false positive and it is fixed. NodeGoat's only prototype report was
the word `set` inside `// set these up in case we have an error case`: the matched node is the
<operator>.assignment holding the entire 1,000-character signup handler, and sinkMatches
searched a node's whole text. It now confines the text clause to the callee and excludes
operator nodes. Fourth instance of this bug class after fgets/gets, resolve/resolveUrl and
user/users. Measured on NodeGoat's non-vendor calls, injection sinks fall 28 to 3, prototype
13 to 2 and untrusted destination 26 to 7, with no true positive lost; req.query had been an
injection sink because `query` is word-bounded inside it. Witnesses now name the sink call
rather than the assignment that contained it.
1,411 tests pass, ten skipped; the six Joern-gated query tests pass in the image, including a
new one that fails on the shape this fixed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Paused at the maintainer's direction to return to vulnerability work, with the
state recorded rather than left implied.
Measured and eliminated: batch size, since 17,700 requests answer in 44 s with a
2.6 MB response; dataflow cost, 50 execution requests in 24 s; call depth 3 with
parameter sources, 24.7 s for 20 requests; argument length, a 16 KB hook table
against a 128 KB single-argument cap; hook callback resolution, one second of
difference with 233 hooks; and build or sharding, 79 s with no shards. Preparation
measures 273.5 s, which is real but a small share of a 3,600-second case.
Every reproduction of the failing 22,125-request batch outside the push path answers
in under a minute. The difference from the production invocation is not identified,
and the exception itself has still never been read: the backend logged a stack
trace's tail, which is frames, and the corrected extractor reports its head, which
on this engine is JVM startup noise.
Four instrument defects were found and fixed along the way, all of the same shape:
a failure detail that discarded the cause, stages that reported no cost, logging
that was never configured so the new stage lines could not emit, and a harness that
buffers stderr until a case ends.
Node and Python transfer stand. PHP stays unestablished.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Every PHP failure logged `cpg batch taint failed: dure1.java:10)`, which is the
middle of a stack frame and says nothing. A trace puts its exception on the FIRST
line and frames after it, and the backend logged the last 400 characters, so the
cause was discarded and the noise kept. Four investigations were spent on that.
Failures now report the leading non-frame lines, which is where the exception and
its message are, with the frames dropped. A control asserts an OutOfMemoryError
survives while a frame does not, and that a timeout and an empty exit stay
distinguishable.
1,411 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The stage costs added for PHP diagnosis never appeared: nothing in this project
configures logging, so the default root level of WARNING dropped every
informational line and a PMPro case ran for an hour producing none.
That is the same failure shape the project keeps relearning. A run that reports
nothing is indistinguishable from a run with nothing to report, and it is how four
PHP investigations were spent reading a later stage's timeout as the cause.
Logging is now configured from OUSAST_LOG_LEVEL, quiet by default and sent to
stderr when asked, so a hook stays silent while a diagnostic run says where its
budget went. Verified end to end: six stage lines with durations on a real replay.
1,410 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Three PHP investigations were spent reading `cpg batch taint failed: timeout` as a
slow query. The query is not slow: 4,425 evidence requests answer in 30 s, 50
dataflow requests in 24 s, and 20 entry-point requests at call depth 3 with
parameter sources in 24.7 s. The production PHP build takes 79 s and does not
shard. None of that explains a case running 26 minutes without finishing.
The reason it could not be diagnosed is that nothing reported where the budget
went. A run that is killed, or that blames a later stage's timeout, left no account
of what had already consumed the transaction, and a subprocess bounded by the
remaining time reports its own timeout whatever spent it.
Each stage now records and logs what it cost as it finishes, so a killed run still
says where it was. A control asserts the stages report and that each line carries
its duration.
1,410 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Retried the PMPro transfer with batch splitting, a 4500-second deadline and a
900-second query ceiling. Stopped after 26 minutes with no case completed and no
verdict, so PHP transfer remains unestablished.
A probe narrowed the cause and corrected my earlier reading of it. The evidence
pass answers all 4,425 PMPro PHP regions in ONE batch in 29.7 seconds with a 681 KB
response, exit 0, and its cost is fixed rather than per request: one request took
40.2 s and 4,425 took 29.7 s. Neither request count nor response size is anywhere
near a limit, so batch size and serialization are not the cause as I first supposed
from a single extrapolated data point.
What fails is the execution pass, which runs reachableByFlows per candidate.
The probe's own scope is recorded with it: it sent evidenceOnly=true and therefore
measured the structural pass, not the failing one. Those numbers would otherwise
read as evidence that PMPro's taint work is cheap, which they are not.
Splitting stays, because it is correct on its own terms and recovers a batch that
fails whole. It is worth noting it also makes an unrecoverable batch slower to
abandon, since each of the eight parts burns its own share of the query ceiling.
M3 remains two of three: Node established in M1b, Python established, PHP blocked
on dataflow cost for a 1,274-file repository.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
PMPro's 1,274 PHP files produced a response the engine could not serialize: it
failed inside ujson while building its own answer array, and every question in that
batch was lost with it. Three budgets from 900 to 3000 seconds made no difference,
because the size of the answer was the problem rather than the time available.
One invocation stays the first thing tried, so a working scan costs exactly what it
did. Each request in a batch is independent, so a batch that fails as a whole can
still answer in parts: on failure the requests are split and retried, and a part
that fails leaves its own requests unanswered rather than discarding every answer
beside them. Only a batch where no part answered is reported as unavailable, which
keeps "could not ask" distinct from "no rows".
Controls cover all three shapes: a whole batch that fails is answered in parts with
the split recorded, a failing part leaves only its own requests unanswered while the
rest are kept, and a batch no part can answer is still unavailable rather than
empty.
Whether this is enough for PMPro is a separate question the measurement will
answer: a single taint request on that tree already takes 40 seconds and returns
36 KB, so the batch may be slow as well as large.
1,410 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
All three PMPro comparisons exhausted a 3000-second deadline with a 1200-second
query ceiling and produced no artifact, so there is no target evidence at all. The
engine reports `cpg batch taint failed` twice per case on the 1,274-file tree:
first an exception whose trace runs through ujson array transformation, then a
timeout on the retry.
Budgets tried: 900 s, 1200 s and 3000 s deadlines against 300 s and 1200 s query
ceilings. The pins, source receipts and declared target are all correct, so the
question was asked properly and the engine could not answer it. This is a runtime
blocker on a large repository, not evidence about whether the design transfers to
PHP.
Two harness defects were found on the way, and both had made an earlier run look
like a transfer failure rather than an instrument problem. The manifest labels a
case by what it is, vulnerable or fixed or benign, while the id says what the
comparison does, introduce or repair or benign; filtering on one while keying
expectations by the other ran a single case and reported an all-false verdict. And
the per-query ceiling is taken as the minimum with the remaining transaction time,
so raising only the deadline could never have helped, and twice did not.
M3 is not met. Its exit asks for Node, PHP and Python together: Python is
established, Node was established in M1b, PHP is blocked.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Generalized the replay instrument so the case family is an argument rather than a
hard-coded target, since a harness that only knows one language cannot say whether
the design transfers. Targets, pins and expectations come from the committed inputs,
so Python is asked exactly the way Node was, through the same core, ranker,
comparison and admission.
VAmPI-SQLI models/user_model.py:73:get_user detected, answered
VAmPI-BOLA api_views/users.py:187:update_password detected, answered
VAmPI-BOLA-BOOKS api_views/books.py:51:get_by_title detected, answered
35 of 35 questions complete on every case, nothing reported new, zero admitted
defects.
Two results stand out. The SQL injection is the evidence the milestone spec names
as previously unresolved. And two of the three targets are access control, the
family that could not complete a question on any repository before group 12, so
this is the first time broken object-level authorization has been asked on Python
at all, and it answered on both endpoints.
VAmPI has the same revision on both sides, so the regression mode establishes
detection and completion, never novelty. It asserts that directly: with nothing
changed, a target reported new would be a comparison defect, and none was. That
tests the novelty logic from the opposite side to the NodeGoat case.
Identical revisions cannot show novelty, and VAmPI is inspected teaching data that
cannot qualify production Python support. No capability enabled, no latency claim.
1,407 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Measured after scoping a registration-only region to its file.
M1a, before group 12 0 of 436 head questions completed
after per-family context 298 of 436
after region attribution 299 of 299, and 299 of 299 on the base
The count falls from 436 to 299 because the mis-attributed function-scoped regions
in a route-registration module are gone, replaced by file-scoped questions the graph
can answer. Nothing is unresolved on either revision.
Detections are unchanged: the vulnerable revision is still new with
operation_absent_from_comparable_base, the fixed twin analyzed and quiet, the
unchanged revision not a regression.
What remains is recorded limits rather than failures: 210 bounded-reachability
boundaries, each owned by the question that reported it, and one vendor exclusion.
Neither blocks a comparison.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A route-registration module registers handlers that other modules define.
app/routes/index.js requires seven handler modules and defines none, yet regions
there were scoped to handleContributionsUpdate and its siblings. The graph rightly
held no such method, so every question in those regions was unresolvable: 138
across six families on that one repository.
The entry record still names the handler. The authorization question is which
middleware guards which handler, and that is cross-file by nature. It now also
records whether the file defines what it names, and a region built from an entry
that does not is scoped to the file instead of to a method the file lacks. The
handler's own module still gets its function-scoped region, so nothing is lost.
My first attempt changed the mapper to stop naming cross-file handlers at all. An
existing control caught it by asserting that a multi-line registration names the
handler its middleware guards, which is exactly the attribution the authorization
family needs. The evidence was against me and the fix belonged one layer up, in
region construction.
1,407 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Final verification of the three security capability fixes on the pinned revisions.
head questions completed 205 -> 298 of 436
base questions completed 436 of 436
access control completed 0 -> 52
context_projection_unavailable 116 -> 0
context_scope_empty 138 -> 0
Every remaining unresolved question now reports context_function_unresolved and
names the function its file does not define, so nothing is unresolved for a reason
the tool could not explain. That is a region-attribution question for the ranker,
not a limit of what the engine can establish.
Detections are unchanged: the vulnerable revision is still new with
operation_absent_from_comparable_base and every verdict entry holds.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
After per-family context, every remaining unresolved question reported
context_scope_empty, all in three files. The cause is not a missing producer:
app/routes/index.js requires seven handler modules and defines none of them, so a
question naming handleContributionsUpdate there names a function that file
genuinely does not hold, and the empty scope is the correct answer.
Reporting that as "scope empty" hid which of two different things had happened. A
file the frontend produced no methods for is one problem; a question naming a
function defined in another module is another, and it is the common one. Taint and
configuration now separate them, reporting context_function_unresolved with the
name in the first case and context_scope_empty only when the file has no methods at
all. The driver no longer stacks its generic gap on top of a query that already
explained itself, and the new gap is question-owned so it cannot block another
question's comparison.
Completion is deliberately unchanged. The tempting fix is to widen such a question
to its whole file, which would raise the number by answering a question nobody
asked; a question whose function the graph does not hold stays unresolved, with the
name now recoverable from the question that asked it.
Group 12 is complete: admission no longer rejects genuine findings on a
mis-extracted operation symbol, both families that could never complete their change
context now can, and the remaining unresolved questions say precisely why.
1,406 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Rerunning the pinned revisions with each family producing its own change context.
head questions completed 205 -> 298 of 436
context_projection_unavailable 116 -> 0
access_control completed 0 -> 52
Access control is the headline: the family could not complete a question on any
repository before, and now completes every question it is asked. That is a
vulnerability class the tool could not reason about at all.
Detections are unchanged. The vulnerable revision is still new with
operation_absent_from_comparable_base, the fixed twin is analyzed and quiet, the
unchanged revision is not a regression.
The remaining 138 unresolved questions are all context_scope_empty and sit in three
files, none of them access control. They are function-scoped questions whose named
function the query cannot locate in the graph, so the scope it describes is honestly
empty. That is the query-to-operation identity problem M1a recorded at the witness
end, showing up at the scope end, and it is task 12.3 rather than something to patch
by widening a question to its whole file.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Access control and configuration could not complete their change context on any
repository, which left 231 of 436 questions unresolved on the measured Node case
and made two of the three shipped families unable to answer.
The gap was larger than a missing query block. The evidence pass dispatches taint
only, because only taint has an evidence vector to rank by, so these families were
never asked for context at all. A new context pass asks each family with the flag
set.
Each query describes its own scope, which the owner research required and which
matters: copying taint's projection would describe a reachability walk these
queries never perform. Dominance exports the file's methods labelled operation,
guard or sibling, because its claim is that a file contradicts itself. Configuration
exports the file's methods labelled setting or sibling, because it asks what a file
declares. Both report an empty scope explicitly rather than implying one.
A family whose context query answers nothing keeps no context and stays unresolved,
so none can complete on another family's context. A control asserts that directly.
Confirmed on a real NodeGoat graph through the production backend: both families
produce a context summary and seven scoped methods where they previously produced
nothing. All seven are siblings for that file, which is the honest answer since it
declares no access-control operation and no matched setting.
1,405 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Admission extracted everything before the first parenthesis, so the operation for
`const preTax = eval(req.body.preTax)` was `const preTax = eval`. No real
declaration can enumerate every assignment a codebase might write, so admission
would have rejected genuine findings on operation_semantics_mismatch. The M1b
demonstration only passed because its declaration listed the assignment spellings.
The symbol is now the trailing qualified callee. Receiver spelling stays part of
it, so `$wpdb->prepare` and `prepare` remain different operations and a declaration
cannot widen from one to the other. A spelling that is not a qualified name, such
as a dynamic `handlers[name](x)`, yields the statement text, matches no declaration
and stays ineligible exactly as before.
Controls cover the assigned call, the receiver distinction, the dynamic spelling,
and that a declaration for a different operation still does not cover it. The
recorded-veto report reads the same helper instead of repeating the extraction.
1,403 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Both attacks on engine startup are measured and neither reaches the gate. The
session is equivalent for every shipped query kind and helps a long transaction,
326 s to 243 s, but its own 9 to 13 second startup is a third of a 30-second budget
so it usually declines to start. Removing JavaScript's second engine start is real,
11.0 s against 21.0 s on identical graphs, and was reverted for an asymmetric
census gate that made 12 transactions pay two starts.
What the profile names is not startup. A changed-code transaction reaches its
deadline inside the build: head build 25.2 s of 30 s, query time 0.0 s. The cost is
building a fresh whole-repository graph for every pushed revision.
Direction accepted: reuse the graph across revisions rather than optimise the
rebuild. M2 stays unmet, the gates stay unchanged, and the work is not scheduled
here. Group 12 returns to security capability, which is where the remaining release
risk sits.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reverts f04a839 on its own measurement. The seven-class profile rerun recorded
joern_parse_census_incomplete on 12 transactions: the faster build was rejected by
its own census gate and the frontend plus overlay pass ran as well, so those
transactions paid two engine starts instead of one. Every workload kept the same
p50 and the verdict stayed NO-GO.
The cause is an asymmetric gate. The frontend path is not census-gated at build
time, so whatever it produces is accepted, while the joern-parse path was gated and
the same shortfall rejected it. Holding the new path to a standard the old one
never has to meet guarantees the fallback fires. The probe tree and the
materialized push snapshot also differ, so equivalence on the checkout did not
transfer to the real path.
What still stands is the underlying timing fact, measured under the production
retention environment on identical graphs, same 44 files, 579 methods, file names,
overlays, Gruntfile and eighteen test files: 11.0 s against 21.0 s. JavaScript does
pay two engine starts where every other language pays one. A future attempt needs a
gate both paths are held to, or a direct comparison against what the frontend would
produce, and must be measured in the push path rather than on a checkout.
The profile also names the remaining cost plainly: a changed-code transaction
spends its budget before any query runs, head build 25.2 s of a 30 s deadline with
query time 0.0 s. Building one new graph per push is the cost, not querying it.
1,401 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
JavaScript was the only language paying two engine starts: the declared frontend
and then a separate overlay pass. joern-parse applies the overlays itself. On the
pinned Node tree that is 11.0 s against 21.0 s.
The declared frontend stays as the fallback, because it is what the versioned
retention adaptation and the explicit exclusion policy run through, and the census
decides per build: a graph that does not hold every readable declared file is
rebuilt through the frontend rather than trusted. Exclusions keep the frontend path
outright.
Measured under the production retention environment, both paths produce the same 44
files, the same 579 methods, the same file names, the same five overlays, the
Gruntfile and all eighteen first-party test files, with no file present in one
graph and missing from the other.
My first measurement did not establish that. It ran without the retention
environment, both paths dropped to 25 files, and the apparent equivalence was two
runs discarding the same things. The existing control named for that decision
failed against the change built on it, which is exactly what it is for; the
evidence was wrong, not the control. That control now asserts the intent directly:
the faster path is tried first, and a short census rebuilds through the frontend.
1,401 tests passed, nine skipped. A build saving is not the latency gate; the
profile rerun decides that. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The seven-class profile showed a 25.3 s head "build" of which the trace attributes
only 4.5 s to the frontend. The remaining twenty seconds were the session's startup
poll, which was bounded by `min(startup_timeout, deadline - now)` and therefore ran
until the transaction deadline was exhausted. The fallback then had nothing left,
so the transport's failure mode consumed exactly the budget it existed to save.
A session now refuses to start unless enough budget survives for the work it exists
to serve, and the wait is bounded by its own allowance rather than by the whole
deadline. Under the 30-second hook deadline it declines immediately instead of
burning what is left.
Controls cover both: refusing is immediate and starts no process, and a startup
wait ends at its allowance nowhere near a long deadline.
This does not make the gate reachable; it removes a self-inflicted cost that was
hiding the real one. 1,401 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The unchanged seven-class profile, three samples each, with the engine session
enabled. Every changed-code workload still times out at the 30-second deadline, and
identical-tip moved the wrong way: p50 rose from 18.5 s to 24.2 s.
The cause is in the artifacts rather than inferred. Thirteen transactions report
session_unavailable:session_startup_timeout, so under the real deadline the session
usually never starts and everything falls back to the disposable path. Startup is 9
to 13 seconds, about a third of the whole budget, and it is spent before any
analysis happens. Where a session did start, on cached-graph transactions whose
remaining work is small, it cost more than it saved.
That is the honest shape of the result: a transaction-owned session amortizes
engine startup across the invocations inside one transaction, and a warm
changed-code push does not have enough of them. It helps a long development-budget
transaction, 326 s to 243 s measured earlier, and cannot help a 30-second one.
Correctness is unchanged throughout: the same three pinned revisions through the
production rules with the session enabled reproduce the M1 verdict exactly.
Group 11 as designed cannot reach the gate, so the dominant remaining cost is named
and the decision is raised rather than worked around. Task 11.6 records the
candidates, since two of them change the product's shape: a session shared across
transactions is a resident local service, which the accepted milestone spec does
not imply, and analysing less is a scope change rather than a speed change. The
30-second and 2-second gates stay unchanged.
Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The leak was the whole cause. With the session closed by the transaction that owns
it, a session-enabled replay of the same pinned revisions under the same budget
keeps its session for the whole run: no session_query_failed appears on either
side, where the previous run reported one per side and fell back to the disposable
path for everything.
Measured on the same pins, instrument and budget, changing only the transport:
vulnerable revision 326.4 s -> 243.3 s
head query time 63.6 s -> 32.0 s
base query time 85.1 s -> 51.2 s
dominance and config ~14 s each -> ~4 s each
Detections are unchanged: witness present, fixed twin analyzed and quiet, unchanged
revision not a regression, kill criterion not triggered. Speed bought by losing a
detection would not be speed.
This is a cold transaction on one teaching repository under a 900-second
development budget. It is not the hook latency gate, which is warm changed-code p95
against the frozen seven-class profile, and that is task 11.5. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The first session-enabled replay produced correct detections and no speedup,
reporting session_query_failed:census:session_request_failed on both sides. A
targeted probe ruled out the sequence I suspected: an overlay followed by a census
in one session both answer correctly, with the census confirming the overlays were
saved.
What the probe exposed instead is a leak. A build runs on a copy of the backend, so
the session started during the build belonged to that copy and was never closed,
leaving a JVM holding the configured 2 GB heap for the life of the process. Two
builds per transaction meant two of them, which is enough to exhaust a 3 GB
container and make every later request fail. It also matches the host running out
of memory during these runs.
The build now closes any session its copy started, and the driver ends the session
with the transaction, since that is what transaction-owned means. Controls cover
both: a build leaves nothing alive on the copy that started it, and a completed
scan closes the backend's session. Closing never fails a scan that found things.
This is a genuine defect found by measurement rather than by the suite, which
passed throughout. Whether it was the whole cause of the census failure is not yet
established; the next session-enabled run will say.
1,399 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The session-enabled NodeGoat run reported session_query_failed:census:
session_request_failed and fell back, which is correct behaviour but says nothing
about why. A bare transport failure cannot distinguish a timeout from a crash from
an out-of-memory kill, and the only account of which it was is in the server's own
log.
A transport failure now carries the exception and the tail of that log, and the
backend logs it with the process exit code alongside its own label. A control
asserts an OutOfMemoryError in the log reaches the caller.
This is diagnosis, not a repair: the run's timings were unchanged because the
session died early and everything fell back to the disposable path.
1,397 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A measured run needs to enable the transport without changing any default. The
switch reads as enabled only for the exact value "1"; unset, empty, "0", "true"
and anything else leave the disposable path in place, and an explicit constructor
argument still wins in both directions.
1,396 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The overlay was the last fixed cost bypassing the session: a whole JVM start,
15.8 s in the measured Node runs, while the census and query batches already route
through it.
`save` writes under the process working directory, which for a session is its own
private scratch rather than the build scratch, so the saved graph is now looked for
in both. Finding neither keeps the raw graph and records it rather than reporting
success, because a graph without its dataflow layer has that layer recomputed by
every later batch, 54 s each on a 637-file plugin, with nothing otherwise saying so.
The disposable invocation remains the fallback and runs unchanged whenever the
session is off or cannot answer.
Controls cover both new paths: an engine that reports success but saves no graph
keeps the raw one and notes it, and a graph saved under the session's workspace is
found and applied.
1,395 tests passed, nine skipped. The transport stays off by default; task 11.5
measures against the frozen profile. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A session is only usable if it returns exactly what the disposable invocation
returns. On a real NodeGoat graph, taint, dominance and configuration all produce
byte-identical payloads through both transports, with parameters built by each
arbiter's own request function through the driver's family lookup rather than a
hand-written shape.
The failure shapes that would make a session dangerous rather than merely slow are
correct too. An empty answer stays present and distinguishable from an unavailable
one, which is the distinction that once turned a failed scan into a clean bill of
health. An unknown query and an absent graph are both unavailable. Two graphs in
one session each answer about the graph the request names.
Amortized over five taint requests the session took 22.5 s against 72.0 s
disposable, all five answered and identical to each other.
Two instrument defects are recorded rather than hidden. The first run created a
fresh backend per comparison, so every session served one request and paid full
startup, measuring the session's worst case and making it look slower; that is why
the amortized comparison exists. The switched-graph check crashed on a scratch
directory it never created, so it produced no verdict at all.
These timings are orientation, not the latency gate. Task 11.5 measures against the
frozen seven-class profile, and the transport stays off by default until then.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
JVM startup, not analysis, is what a push scan spends its time on: each disposable
script invocation costs 13 to 23 seconds while the work takes under a second.
The session owns one Joern server per transaction on loopback with an OS-chosen
port and no published port. Three properties are load-bearing, and the probe that
settled the design demonstrated why each is needed.
An HTTP status is not an answer. The server returns success even for a compile
error, with the diagnostic in its body, so every request redirects the evaluated
code's console output into a fresh private file behind a nonce, and the nonce is
verified before the answer is accepted. The engine's body is retained for exactly
the case where the receipt is missing, without which that failure explains nothing.
A poisoned session answers nothing afterwards, and cancellation kills the process
group rather than the launcher, which this project has already run a machine out of
memory by neglecting.
What the probe established about the engine, after three iterations that each
failed loudly: runScript is absent from the server REPL; a shipped @main script
cannot be nested in a block expression, so definitions go to the top level
unwrapped; and the REPL shadows Console with io.joern.console.Console, which has no
withOut, so capture must qualify scala.Console. With that fixed, both mechanisms
answer identically to the disposable invocation, 0.8 s against 9.8 s.
The backend routes batch and single queries through the session only when
session_transport is enabled, which it is not by default. Any failure falls back to
the disposable invocation rather than reporting a zero, and each script is defined
under its own name because every script declares exec.
Equivalence for taint, dominance and configuration is task 11.2 and gates any
measured run. 1,391 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The M1b exit run established the comparison but could not show a developer-readable
consequence or repair direction, because both belong to an evaluated capability
declaration and the installed registry is deliberately empty.
The experimental evaluation output now renders, for each change-attributed finding,
its family, exact location, witness, change attribution, provenance, engine method
and operation, together with a consequence and repair direction produced from a
supplied declaration through the same grounded-template path the admitted case
uses. No prose is invented: when no declaration matches, or a template is not
grounded in the operation and context, or the declaration does not cover the
operation symbol, the explanation is absent and the reason is stated.
Declarations reach this rendering only. The runner never passes them to admission,
so nothing in that file can enable a capability or emit an alert however it is
written, and a control asserts that even a declaration claiming PASS admits
nothing. The flag requires explicit base/head replay and the recorded-veto flag.
The evaluation instrument generates its own unreviewed declaration bound to the
run's analysis semantics, asserting that it is neither enabled nor PASS, so the
demonstration cannot drift into a qualified claim.
1,377 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
All three pinned NodeGoat revisions replayed through the production rules with
exact pins and reopened source receipts.
The vulnerable revision reports all three sites new with
operation_absent_from_comparable_base, and admission now rejects on
capability_unavailable alone: dependency_unresolved is gone, which verifies 10.7
on real data since those gaps belonged to unrelated questions. The fixed twin is
analyzed and quiet with three base-only findings. The unchanged revision reports
all three sites unchanged with base locations resolved to the same lines.
Every finding carries its question, the engine-reported method and its witness.
The head census is the complete 44 files and all 436 base questions complete.
Zero admitted defects on every revision.
All five vetoes M1a recorded are repaired, plus two the repairs exposed. What
remains is recorded rather than repaired: 231 head questions still report an
incomplete change context from families whose queries export none, which did not
block this case and whose families stay experimental, and the vendor exclusion
and correspondence ambiguity remain reported as the limits they are.
M1b is not yet verified. The milestone also asks for a developer-readable finding
carrying a supported security consequence and a concrete repair direction, and
both are owned by an evaluated capability declaration that is deliberately empty.
Writing that text without a declaration is what admission exists to prevent, so
task 10.10 renders it through an explicitly experimental declaration with
eligibility still disabled.
Runtime, independent qualification and rollout all remain NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
M1a recorded that the NodeGoat target is selected through a file-scoped question
naming no function while the engine reports method <lambda>2, and that this did
not block the comparison once the question completed. It was still only visible by
digging the method out of a serialized detail blob.
Evidence now carries the engine-reported method beside the question that produced
it, and the relationship between them is recorded: exact when the question named
that method, file_scope when the question named no function and the engine
supplied one, unresolved when the two disagree, and method_unreported when the
engine named none.
Recorded, not applied. No region is renamed, no benchmark function name is
injected into a region, and no coincident site is counted. The recorded-veto
report now reads the field instead of reparsing the detail blob.
1,371 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The runner handed every candidate the whole transaction's coverage reason list as
its unresolved dependencies, so admission always answered dependency_unresolved.
That list even included other candidates' comparison outcomes, so one unresolved
comparison anywhere marked every other claim as having an unresolved dependency.
Gaps are now attributed. A boundary carrying a question id counts only when that
question is in this candidate's required comparison scope, the same scope 10.2
established. A gap nothing attributed to any question always counts, because
unknown ownership is not evidence of independence. Integrity degradations count.
A declared exclusion, a correspondence ambiguity and a bounded reachability do
not, because each was already weighed against this specific claim: the first is a
scope choice, the second is decided per operation, and the third is decided per
claim by the comparison.
Found by the M1b production rerun rather than by the suite: the comparison reached
a correct new verdict and admission still rejected it for a dependency belonging
to unrelated questions. It is not an M1b exit blocker, since eligibility stays
disabled, but it would have blocked admission forever once capabilities qualify.
1,371 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Rerunning the three pinned revisions after tasks 10.1-10.6 shows the production
comparison classifying all of them with no lifting, and agreeing exactly with the
recorded view. That agreement is the point: the vetoes M1a recorded were repaired
rather than bypassed.
The vulnerable revision is new with operation_absent_from_comparable_base, and the
three eval witnesses at contributions.js lines 32 to 34 are attributed to the
change that introduced them. The fixed twin is analyzed and quiet with three
base-only findings. The unchanged revision is unchanged. In M1a every one of the
436 questions was demoted on both sides and production reported
witness_identity_unresolved on every finding.
All three artifacts still report finding_status none with zero admitted defects.
Admission rejects on capability_unavailable, the deliberate eligibility gate, and
on dependency_unresolved, which the rerun exposed as coarse plumbing: the runner
passes every coverage reason as every candidate's dependency gaps. That is now
task 10.7, ahead of M4, since it would block admission forever once capabilities
qualify. It is not a comparison failure and not an M1b exit blocker.
Three comparisons on one inspected teaching repository under a development budget.
No capability enabled, no alert emitted, no latency claim. Runtime, independent
qualification and rollout all remain NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The NodeGoat rerun after 10.4-10.6 showed both target questions completing, which
is what those tasks were for, but the comparison still reported
head_context_incomplete. Two checks inside comparison had not been updated with
the same distinction: any degradation blocked, and any scope boundary blocked.
A declared exclusion and a correspondence ambiguity both reached those checks. The
first is a scope choice rather than a failure to read the graph, and the second is
decided per operation since 10.3, so neither should block a claim. Both remain in
aggregate coverage. Integrity degradations, unknown ownership and every other
boundary still block, and bounded reachability is still filtered separately
because it blocks only claims resting on a flow not having been traced.
1,368 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
vendor_semantics_unresolved was grouped with the graph integrity failures, so any
declared exclusion demoted every otherwise complete answer in the repository to
graph_incomplete. Every real repository excludes its dependencies, so this fired
on all of them, including on a complete first-party census with successful query
receipts.
Graph boundaries are now split by what they mean. An integrity failure says the
graph could not be read or is incomplete, so an answer drawn from it is demoted:
unparsed files, an empty or sharded graph, an incomplete or unavailable census,
an unresolved symlink, an unsupported frontend, unreadable source, an ambiguous
frontend path and unvalidated TypeScript property support all keep that behavior.
A declared exclusion is a deliberate scope choice. It stays physical, stays out of
targets and graphs, and stays reported in coverage, but it is not evidence that a
first-party answer is wrong.
Whether a particular answer needed the excluded semantics is carried by that
question's own unresolved call destinations, recorded per question in 10.5, rather
than assumed for every answer in the repository.
Controls exercise the real partition logic rather than an injected reason: a
vendored directory leaves the answer completed with the exclusion still reported,
an unresolved symlink still demotes, and the two together still demote.
1,365 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
dynamic_external_or_depth_context_unresolved was applied as though it invalidated
everything about a question. It does not. Bounded reachability can hide further
flows; it cannot invalidate a flow that was traced, and it cannot remove a call
from a structurally scoped enumeration.
At scan level the boundary is now recorded against its owning question without
demoting that question's answer. An unavailable context projection, an empty
context scope and an unreadable method extent are unknown ownership and still
demote.
In comparison it no longer blocks the head's own claim, since the finding rests
on a path that was actually traced, and no longer blocks an absence read off the
enumerated inventory. It still blocks every claim that rests on a flow not having
been traced, including the anchored source-connection branch, which now reports
base_reachability_bounded rather than silently passing.
The query declares the basis of its enumeration. A file- or nesting-scoped
question walks calls structurally, so an unresolved destination cannot remove one
from scope. Following the call graph makes the reachable set the scope, so the
same unresolved destination can shrink what was enumerated; that inventory cannot
establish absence while reachability is bounded, and says so.
One prior control asserted a question's own reachability gap blocks its own
claim. It is updated with its reason and split, so an own projection gap is still
proven to block while an own reachability gap no longer does.
1,362 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A taint answer reports traced flows, so an absent row means either that the
operation is not present or that no traced path reached it. Change comparison
needs that distinction to say a change introduced an operation, and reachability
is deliberately bounded, so the two can never be separated from flow rows alone.
That is why the veto guarding the confusion is permanently true on any real
Express application, and why it could be neither discharged nor safely removed.
The taint query now exports one row per operation it enumerated in scope, plus a
completion marker. The marker is what makes an empty inventory meaningful:
without it a scope that was never enumerated and a scope holding no such
operation are indistinguishable.
Comparison separates three outcomes on a changed line: the operation is present
in the base without a traced flow, which is a reachability statement and not
novelty; the operation is absent from an enumerated base, which establishes that
the change introduced it; or the base scope was never enumerated, which stays
explicitly unresolved. A malformed inventory row is not an empty inventory.
The arbiter now skips any row carrying a kind, so an inventory row cannot be
normalized into a flow with an empty source that names no attacker input.
Verified against the real engine on an authored two-revision JavaScript control:
the operation is enumerated on both sides, the base carries it with zero traced
flows, the zero-operation marker appears for an empty scope, and answers and
findings are unchanged. No new sink inference, alias resolver or second source
IR. 1,354 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Rerunning the three NodeGoat revisions after tasks 10.1-10.3 shows the recorded
view now classifies the case: the vulnerable revision reaches new with
operation_absent_from_comparable_base, and the unchanged revision reaches
unchanged with its base location resolved to the same lines. Both previously
stopped at operation_correspondence_unresolved and witness_identity_unresolved.
The rerun also caught a defect in the repair itself: the new novelty reason was
missing from the admission allowlist, so admission rejected the candidate as
change_unsupported. Fixed with a control.
The production path is still blocked by the last two vetoes, and the measurement
shows neither can be scoped or relaxed as the plan described. A taint answer
reports traced flows, so an absent row means either that the operation does not
exist in the base or that no traced flow reached it.
dynamic_external_or_depth_context_unresolved exists to stop the second being read
as the first, and removing it would turn a wrong absence into a wrong new
finding. It is also computed over every in-scope method, so on any real Express
application it is always true and can never be discharged, which is why the tool
reports nothing on real projects.
The resolution is new evidence rather than a relaxed rule: export the operations
the query actually enumerated, separately from the flows it traced, so absence is
established from the inventory while a traced-flow claim still requires complete
reachability. That is now task 10.4, ahead of the two vetoes it unblocks, and it
adds no detector, alias resolver or second source IR.
1,348 tests passed, nine skipped. No capability enabled; rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Two vetoes M1a recorded were global where the evidence is local.
Comparison required every selected base question to complete, so one unsupported
family anywhere prevented an otherwise complete target comparison. Completeness
is now demanded over the required scope: the counterpart question plus the
dependencies the change context actually recorded for it. Every other gap stays
in aggregate coverage and is still reported, never discharged. An incomplete
counterpart, an incomplete recorded dependency, an unknown-ownership boundary
and a truncated base scan all still yield unknown.
Correspondence ambiguity was recorded file-wide and inherited by every question,
so repeated unchanged lines in one file invalidated every question in the
repository and blocked the whole transaction comparison. It is now decided per
operation by the correspondence classifier: an operation needing an ambiguous
anchor stays uncomparable, while a uniquely mapped or newly introduced operation
in the same file is unaffected. The mapping is still withheld and still reported.
On the NodeGoat case the ambiguity is on the target's own file, so file-level
scoping would not have been enough; only per-operation resolution is.
1,347 tests passed, nine skipped. No capability enabled; rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
M1a measured that a newly introduced operation sits on a changed line, has no
base line anchor, and therefore stayed unclassified even with every completeness
veto lifted. Establish absence over the complete counterpart answer instead of
over a line anchor.
Correspondence now reports why a head operation does not map: `changed_line`
inside a recorded head edit, `ambiguous` where several base lines could
correspond, or `unmapped` where no anchor exists. Only the first supports
reasoning about an introduced operation, and only when the base counterpart
answer is complete and contains no operation with that spelling.
Movement is matched on the operation spelling alone, so a benign identifier
rename that changes only the source text reads as movement rather than a new
flow. The existing rename control caught that; the key was too brittle before.
One prior control is superseded and updated with its reason: a new operation on
a changed line in a comparable existing file is now new rather than unknown.
Movement, ambiguity, unmapped operations, discharged or unentailed claims and an
incomplete or failed base all remain unknown.
1,337 tests passed, nine skipped. No capability enabled; rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Add group 9 to the approved task plan as the bounded owner amendment release
milestone M1a requires, implement the task-local recorded-veto evaluation flag,
and run the three pinned NodeGoat revisions under it.
The flag re-runs the production comparison and admission over a lifted copy of
the same evidence and records each veto instead of applying it. Recognized
completeness vetoes are lifted and logged; failures never are, because silence
from a failure is not a quiet fixed twin. It is off by default, accepted only on
explicit base/head replay, refused in Git stdin mode, labels every payload it
produces, and admits no defect even when a lifted candidate otherwise would.
M1a outcome, inside its one-week box and with expectations frozen on disk first:
the three entailed <lambda>2 witnesses are present at contributions.js 32-34; the
fixed twin was genuinely analyzed and quiet, answering its target question with
zero rows and no deadline reason while the same findings appear as base-only; the
unchanged revision was analyzed and recorded unchanged. The kill criterion did
not trigger.
Five vetoes fired and were recorded. M1a also corrected the diagnosis: with all
five lifted the detection still stops at operation_correspondence_unresolved,
because a newly introduced operation sits on a changed line and has no base
anchor. The unchanged revision resolves its base location on the same lines and
classifies as unchanged, which isolates correspondence as the mechanism. That
fifth veto was not in the earlier four-veto diagnosis and is now M1b's central
repair.
Diagnosis only. No defect admitted, no capability enabled, no gate changed, no
latency claim; the 900-second development budget cannot measure the hook gate.
Rollout and task 8.3 remain NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Add release-milestones.md as the executable outcome contract: prove one useful
Node detection, make it fast, verify PHP/Python transfer, qualify on untouched
cases, then enable a narrow advisory rollout. Link it from the roadmap, safety-net
steering, requirements, tasks, status and spec.json.
Split M1 after review. The raw NodeGoat <lambda>2 witness already exists in the
2026-09-14 post-census bundle; four admission vetoes discard it. M1a runs the
vulnerable/fixed/unchanged revisions under a task-local experimental flag that
records each veto instead of applying it, with a one-week box and a kill
criterion if the fixed twin is not quiet. M1b repairs only the vetoes that fired
and reruns through production rules. Existing approvals and thresholds unchanged.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
- Whether the failure is deterministic or intermittent
### 1b. Verify the Instrument Before You Believe the Measurement
Every wrong diagnosis recorded in this project had the same shape: the input was unreadable and the tool reported a plausible zero (`file_exists()` false, the parser wrote nothing, the frontend exited 0), and "no errors, 0 findings" read exactly like a clean result. Before trusting any measurement in the failure report:
- **Prove the tool read its input.** Make it open one file and print the size. A zero from an unread tree and a zero from clean code are indistinguishable afterwards.
- **Distrust a number that is too good.** A Joern script "finishing" in 0.5 s when a JVM takes 15 s to start is a failed launch. A harness must fail loudly with the exit code.
- **Prefer the shipped environment** (the Docker image has a native `php`); local wrappers that proxy stdio are not valid instruments for `php2cpg`.
- **When a result contradicts a working baseline, suspect the instrument first.**
A plausible zero is a failed instrument until proven otherwise; classify it as `CONFIG_GAP` or `RUNTIME_MISMATCH`, not as `LOGIC_ERROR` in the code under test.
@@ -134,6 +142,7 @@ If the issue is fixable by repo changes inside the current task plan, do not esc
| “Let’s try a few fixes” | Multi-fix guessing hides root cause. |
| “The spec is probably wrong, I’ll adapt it” | Spec conflicts must be surfaced explicitly. |
| “The docs search is optional” | For runtime/dependency issues, docs and version issues often contain the shortest path to root cause. |
| “It ran clean: no errors, zero findings” | A tool that never read its input reports the same. Prove the read (one file, its size) before believing the zero. |
@@ -20,7 +20,7 @@ description: Entry point for new work. Determines the best action path or work d
Gather **only metadata** to determine the action path. Do NOT read full file contents yet.
- **Specs inventory**: Scan `.kiro/specs/*/spec.json` for `name`, `phase` fields and `approvals` status. Note feature names and their current status.
- **Steering existence**: Check which files exist in `.kiro/steering/` (product.md, tech.md, structure.md, roadmap.md). Do NOT read their contents yet.
- **Steering existence**: Check which files exist in `.kiro/steering/` (core: overview.md, roadmap.md, safety-net.md; custom files may exist too). Do NOT read their contents yet.
- **Roadmap check**: If `.kiro/steering/roadmap.md` exists, read it. This contains project-level context (approach, scope, constraints, spec list) from a previous discovery session. Use it to restore project context.
- **Top-level structure**: List the project root directory to note key directories and files. Do NOT recurse into subdirectories.
@@ -60,7 +60,7 @@ For Path A/B, recommend the next action and stop.
**Only for Path C, D, and E.** Now load the context needed for discovery.
**In main context** (essential for dialogue with user):
- **Steering documents**: Read product.md and tech.md (if they exist) for project goals, constraints, and tech stack
- **Steering documents**: Read overview.md and safety-net.md (if they exist) for project goals, constraints, and the current state of the stack
- **Relevant specs**: If the request is adjacent to an existing spec, read that spec's requirements.md to understand boundaries and avoid overlap
**Delegate to sub-agent** (keeps exploration out of main context):
Then write `.kiro/specs/<feature>/brief.md` for **every** feature listed under `## Specs (dependency order)` using the Path C brief format. This enables parallel spec creation via `/kiro-spec-batch`.
Then write `.kiro/specs/<feature>/brief.md` for **every** feature listed under `## Specs (dependency order)` using the Path C brief format. Each spec is then created one at a time with `/kiro-spec-init`, in dependency order.
**For Path E (mixed decomposition)**:
@@ -219,7 +219,7 @@ Use the same roadmap structure as Path D, plus these additional sections:
```
Path E rules:
- Keep `## Specs (dependency order)` reserved for **new specs only** so `/kiro-spec-batch` can still parse it unchanged
- Keep `## Specs (dependency order)` reserved for **new specs only** so the list stays a plain, parseable creation order
- Record existing-spec extensions under `## Existing Spec Updates`
- Record true no-spec work under `## Direct Implementation Candidates`
- Write `brief.md` only for the **new specs** listed under `## Specs (dependency order)`
@@ -235,15 +235,12 @@ Suggest the next command and stop. Do NOT automatically run downstream spec gene
- Path A: `/kiro-spec-requirements {feature}` to update the existing spec
- Path B: Recommend direct implementation without creating a spec
- Optional fast path: `/kiro-spec-quick <feature-name>`when the user explicitly wants to continue immediately
- Path D: Default to `/kiro-spec-batch` (creates all specs in parallel based on roadmap.md dependency order)
- Optional cautious path: `/kiro-spec-init <first-feature-name>` when the user wants to validate the first slice before batching the rest
- Path C: `/kiro-spec-init <feature-name>`, then the step-by-step phases (requirements, design, tasks), each human-reviewed
- Path D: `/kiro-spec-init <first-feature-name>`for the first spec in roadmap.md dependency order; validate that slice before the next one
- Path E: Choose the next command based on the new-spec portion of the decomposition
- If there is exactly one new spec:`/kiro-spec-init <new-feature-name>`
- If there are multiple new specs: `/kiro-spec-batch`
- `/kiro-spec-init <new-feature-name>` for the first (or only) new spec, the rest in dependency order afterwards
- Also note which existing specs should be revisited with `/kiro-spec-requirements <feature>`
- Re-entry: `/kiro-spec-init <next-feature-name>` or `/kiro-spec-batch` if multiple specs remain
- Re-entry: `/kiro-spec-init <next-feature-name>`for the next spec that remains
If the decomposition contains only existing-spec updates plus direct implementation candidates, do NOT use Path E. Prefer Path A when one existing spec is the clear home, or recommend the existing-spec update plus direct implementation work without creating roadmap entries.
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to the selected task's boundary, runtime prerequisites, integrations, domain rules, security/performance constraints, or team conventions that affect implementation or validation
- Relevant local agent skills or playbooks only when they clearly match the task's host environment or use case; read the specific artifact(s) you need, not entire directories
description: Create comprehensive technical design for a specification
description: Create comprehensive technical design for a specification; optionally run the implementation gap analysis first (--gap) or an interactive design review of an existing design (--review)
- `.kiro/specs/$1/research.md` (if exists, contains the gap analysis from Step 1b or an earlier run)
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (this repository's core steering; custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to requirement coverage, architecture boundaries, integrations, runtime prerequisites, security/performance constraints, or team conventions that affect implementation readiness
- `.kiro/settings/templates/specs/design.md` for document structure
- Read `rules/design-principles.md` from this skill's directory for design principles
@@ -35,6 +40,16 @@ metadata:
- If `-y` flag provided ($2 == "-y"): Auto-approve requirements in spec.json
- Otherwise: Verify approval status (stop if unapproved, see Safety & Fallback)
### Step 1b (optional, `--gap`): Implementation Gap Analysis
Recommended for brownfield features; skip for greenfield. Analyze the gap between the approved requirements and the existing codebase to inform the design, following `rules/gap-analysis.md` from this skill's directory.
1. Read `rules/gap-analysis.md` for the analysis framework.
2. Research in parallel where possible (sub-agents if available, otherwise sequentially): existing implementations, architecture patterns, integration points and extension possibilities; external dependency compatibility and version constraints when needed; the requirements, core steering, task-relevant extra steering and relevant local agent skills or playbooks.
3. Execute the gap analysis: Grep and Read the codebase, use WebSearch/WebFetch for external dependencies if needed, evaluate multiple implementation approaches (extend / new / hybrid) with trade-offs, flag areas needing further research. Provide information and options, not final implementation choices.
4. Write the analysis to `.kiro/specs/$1/research.md` (append after a `---` rule if the file exists; never overwrite earlier research) and read it back to verify the write. Use the language specified in spec.json.
5. Unapproved requirements are a warning here, not a stop: gap analysis can inform requirement revisions. Then continue with Step 2, or stop and report if the user asked for the gap analysis alone.
### Step 2: Discovery & Analysis
**Critical: This phase ensures design is based on complete, accurate information.**
@@ -137,6 +152,15 @@ After all findings return, synthesize in main context before proceeding.
A human-paced quality review of an existing `.kiro/specs/$1/design.md`, in addition to the automatic gate of Step 5. Follow `rules/design-review.md` from this skill's directory.
1. Gather context: spec.json, requirements.md, design.md, core steering, review-relevant extra steering and local playbooks; survey existing codebase patterns, naming conventions and component structure for reference (parallel sub-agents if available).
2. Review: Analysis -> Critical Issues -> Strengths -> GO/NO-GO. Limit to the 3 most important concerns; recognize 1-2 strengths; engage interactively (clarifying questions, alternatives); leverage conversation history for requirements context and user intent. Quality assurance, not perfection seeking; accept acceptable risk; every suggestion must be implementable.
3. Decide: a clear GO/NO-GO with rationale. GO: `/kiro-spec-tasks $1` (or `-y`). NO-GO: address the critical issues, re-run `/kiro-spec-design $1`, then `/kiro-spec-design $1 --review` again.
4. If design.md does not exist, stop: "Run `/kiro-spec-design $1` first to generate design document". If the design phase is not marked generated in spec.json, warn but proceed.
## Critical Constraints
- **Type Safety**:
- Enforce strong typing aligned with the project's technology stack.
@@ -199,7 +223,7 @@ Provide brief summary in the language specified in spec.json:
**If Design Approved**:
- Review generated design at `.kiro/specs/$1/design.md`
- **Optional**: Run `/kiro-validate-design $1` for interactive quality review
- **Optional**: Run `/kiro-spec-design $1 --review` for the interactive quality review (Step 7)
- Then `/kiro-spec-tasks $1 -y` to generate implementation tasks
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to feature scope, user personas, business/domain rules, compliance/security constraints, operational constraints, or existing product boundaries
- Relevant local agent skills or playbooks only when they clearly match the feature's host environment or use case and contain domain terminology or workflow rules that shape user-observable requirements
@@ -40,7 +40,7 @@ The following research areas are independent. Decide the optimal decomposition b
**In main context** (essential for requirements generation):
- EARS format rules, requirements review gate, requirements template
- Core steering: product.md, tech.md (directly inform scope and constraints)
- Core steering: overview.md, safety-net.md (directly inform scope and constraints)
**Delegate to sub-agent** (keeps exploration out of main context):
- **Codebase hints** (brownfield projects): Spawn a sub-agent to explore existing implementations that inform requirement scope. Ask it to summarize: (1) what already exists, (2) relevant interfaces/APIs, (3) patterns that new requirements should align with. Return a summary under 150 lines.
@@ -133,7 +133,7 @@ Provide output in the language specified in spec.json with:
**If Requirements Approved**:
- Review generated requirements at `.kiro/specs/$1/requirements.md`
- **Optional Gap Analysis** (for existing codebases):
- Run `/kiro-validate-gap $1` to analyze implementation gap with current code
- Run `/kiro-spec-design $1 --gap` to analyze the implementation gap with the current code before designing
- Identifies existing components, integration points, and implementation strategy
- Recommended for brownfield projects; skip for greenfield
- Then `/kiro-spec-design $1 -y` to proceed to design phase
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to requirements coverage, design boundaries, runtime prerequisites, or team conventions that affect task executability
description: Manage .kiro/steering/ as persistent project knowledge
description: Manage .kiro/steering/ as persistent project knowledge, including custom steering documents for specialized contexts
metadata:
shared-rules: "steering-principles.md"
---
@@ -11,9 +11,12 @@ metadata:
<background_information>
**Role**: Maintain `.kiro/steering/` as persistent project memory.
**Core steering in this repository**: `overview.md` (what the project is and what is actually true of it, dated current state), `roadmap.md` (milestones and status), `safety-net.md` (product direction). Custom files may exist beside them (for example a domain or standards file); all `.kiro/steering/*.md` are loaded as project memory. The generic bootstrap templates under `.kiro/settings/templates/steering/` are named `product.md`, `tech.md`, `structure.md`; they are starting points for a fresh project, not this repository's file names.
**Mission**:
- Bootstrap: Generate core steering from codebase (first-time)
- Sync: Keep steering and codebase aligned (maintenance)
- Custom: Create specialized steering documents beyond the core files (optional, see below)
- Preserve: User customizations are sacred, updates are additive
**Success Criteria**:
@@ -27,8 +30,9 @@ metadata:
Check `.kiro/steering/` status:
**Bootstrap Mode**: Empty OR missing core files (product.md, tech.md, structure.md)
**Bootstrap Mode**: Empty OR missing core files (here `overview.md`, `roadmap.md`, `safety-net.md`; in a fresh project the template names `product.md`, `tech.md`, `structure.md`)
**Sync Mode**: All core files exist
**Custom Mode**: The user asks for a specialized steering document (see Custom Steering below)
---
@@ -75,6 +79,20 @@ After all parallel research completes, synthesize patterns for steering files.
---
## Custom Steering (optional)
Create a specialized steering document for one domain beyond the core files.
1. **Ask the user** for the domain/topic (e.g. "API standards", "testing approach") and the specific patterns to document
2. **Check for a template** in `.kiro/settings/templates/steering-custom/`: `api-standards.md`, `testing.md`, `security.md`, `database.md`, `error-handling.md`, `authentication.md`, `deployment.md`. Load the matching one as a starting point; without a template, generate from domain knowledge
3. **Analyze the codebase** (JIT) for the domain's patterns with Glob/Grep/Read (template and principles in parallel with the domain analysis when multi-agent is available)
4. **Generate** the document: follow the template structure if any, apply `rules/steering-principles.md`, patterns not exhaustive lists, one domain per file, 100-200 lines (a 2-3 minute read), never secrets
5. **Create**`.kiro/steering/{name}.md`; ensure it does not duplicate core steering content
Custom files are loaded as project memory like the core files and carry equal weight.
---
## Granularity Principle
From `rules/steering-principles.md` (in this skill's directory):
@@ -106,9 +124,9 @@ Chat summary only (files updated directly).
✅ Steering Created
## Generated:
- product.md: [Brief description]
- tech.md: [Key stack]
- structure.md: [Organization]
- overview.md: [What it is, current state]
- roadmap.md: [Milestones, status]
- safety-net.md: [Product direction]
Review and approve as Source of Truth.
```
@@ -118,8 +136,8 @@ Review and approve as Source of Truth.
✅ Steering Updated
## Changes:
- tech.md: React 18 → 19
- structure.md: Added API pattern
- overview.md: Current state dated, retired component removed
- roadmap.md: Milestone status updated
## Code Drift:
- Components not following import conventions
@@ -128,6 +146,20 @@ Review and approve as Source of Truth.
- Consider api-standards.md
```
### Custom:
```
✅ Custom Steering Created
## Created:
- .kiro/steering/api-standards.md
## Based On:
- Template: api-standards.md
- Analyzed: src/api/ directory patterns
Review and customize as needed.
```
## Examples
### Bootstrap
@@ -136,7 +168,12 @@ Review and approve as Source of Truth.
### Sync
**Input**: Existing steering, new `/api` directory
- **overview.md**: What the project is and what is actually true of it; a dated current-state section wins over older history
- **roadmap.md**: Milestones, their measured status, and the accepted order of work (not a task list)
- **safety-net.md**: Product direction and acceptance criteria (not feature catalogs)
- **Custom files**: Specialized patterns (API, testing, security, etc.); they may exist beside the core files and carry equal weight
(The generic bootstrap templates `product.md`, `tech.md`, `structure.md` under `.kiro/settings/templates/steering/` cover purpose, key frameworks and organization patterns for a fresh project.)
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to the validated boundaries, runtime prerequisites, integrations, domain rules, security/performance constraints, or team conventions that affect the GO/NO-GO call
**Discover canonical validation commands**:
**This repository's gates** (`.github/workflows/ci.yml`; run them as written, judge each by its own exit code):
- `TEST_COMMANDS`: `pytest -q -p no:cacheprovider < /dev/null; echo "exit=$?"` -- judge ONLY by the printed exit code, never by a summary line; `ruff check src tests`; `ruff format --check src tests`; `mypy src/openultrasast`
- Detection gates: `python -m openultrasast.gate` (>=90% recall, <10% FP per language; hard, score-independent), `python -m openultrasast.map_gate`, and the pair gate `python -m openultrasast.pair_gate` (local vuln vs fix). A ruleset change that fails a gate is NO-GO.
- Docs: `mkdocs build --strict` when `docs/` or `mkdocs.yml` changed
- `SMOKE_COMMANDS`: `ousast --help`, and for a scan change `ousast scan <fixture> --mode quick` whose `manifest.json` names the stages that ran and the files preprocess kept. A plausible zero (0 findings, 0 degradations) from a tool that never read its input is a failed instrument, not a pass: prove the tool read its input first.
**Discover canonical validation commands** (for other repositories, or anything the list above does not cover):
- Inspect repository-local sources of truth in this order: project scripts/manifests (`package.json`, `pyproject.toml`, `go.mod`, `Cargo.toml`, app manifests), task runners (`Makefile`, `justfile`), CI/workflow files, existing e2e/integration configs, then `README*`
- Derive a feature-level validation set for this repo: `TEST_COMMANDS`, `BUILD_COMMANDS`, and `SMOKE_COMMANDS`
- Prefer commands already used by repo automation over ad hoc shell pipelines
5. Reject claims that are broader than the evidence.
6. If mandatory validation cannot be completed, return `MANUAL_VERIFY_REQUIRED`.
7. Only then allow the claim.
7. Verify the instrument before you believe the measurement: prove the tool read its input (it opened one file and printed the size; the stage list names what ran). A plausible zero -- "0 findings, 0 errors, exit 0" from a tree the tool never read, or a Joern script that "finished" before a JVM could start -- is a failed instrument, not evidence. For pytest, judge only by the printed exit code (`pytest -q -p no:cacheprovider < /dev/null; echo "exit=$?"`), never by a summary line.
8. Only then allow the claim.
## Claim-Specific Rules
@@ -116,6 +117,7 @@ Return `NOT_VERIFIED` when:
| “Build should be fine because lint passed” | Lint does not prove build success. |
| “Tests passed and build succeeded, so it must run” | Type erasure, module loading, native ABI, and boot-time config issues can still fail at runtime. |
| “The feature is done because all tasks are checked off” | `FEATURE_GO` also requires coverage, integration, and design alignment. |
| “The scan found nothing, so the code is clean” | An unread input and clean code both report zero. Prove the tool read its input first. |
description:OpenUltraSAST scan workflows - `ousast scan` in quick, standard or deep mode, the evidence ladder, `--fail-on` exit codes, recorded degradations, the run directory and `ousast mcp`. Use when running or planning a scan, reading a run directory, or wiring OpenUltraSAST into an agent.
---
# OpenUltraSAST Scan
Use this skill to run `ousast scan PATH --mode quick|standard|deep` from an agent and to read what
it wrote.
## Workflow
1. Confirm the target is a local checkout (a path, not a URL).
2. Run through the project entry point: `uv run ousast ...` inside the repository, or `ousast` from an
environment where it is installed. Never activate a virtualenv by hand.
3. Start with `--mode quick` unless the task needs the Joern model layer (`standard`) or sandboxed
regression snippets (`deep`). Check the prerequisites below first; a mode whose engine is missing
still exits 0 and records a degradation.
4. Read the printed `scan_id=` and `run_dir=` lines; the run directory is `.openultrasast/runs/<scan-id>/`.
5. Report the finding count, how many the verifier accepted, and **every** recorded degradation
(`manifest.json`, key `degradations`). No findings plus a degradation is not a clean scan: the scan
answered a smaller question than it was asked.
6. Never call a finding verified from its title or severity. `verification.json` holds the status.
## Commands
```bash
uv run ousast scan <repo> --mode quick
uv run ousast scan <repo> --mode standard --fail-on verified
uv run ousast scan <repo> --mode deep --fail-on worth-fixing
uv run ousast scan <repo> --mode quick --config openultrasast.toml
uv run ousast mcp # the MCP server over stdio, for agent hosts
| `standard` | the MAP stage: complexity map, semantic overlay, authorization obligations, the Joern model layer (one code property graph per repository; taint, guard-dominance and configuration questions), deterministic fusion | Joern (shipped in the Docker image, with `php-cli` for PHP); a provider key for the residual LLM question | `cpg_unavailable`; without a key the LLM question is not asked and only the graph's entailed findings are reported |
| `deep` | nothing today: REGRESS was retired on 2026-10-13 (degradation `regress_retired`); execution proof belongs to the search-with-proof arm | - | - |
Model stages read `DEEPSEEK_API_KEY` (every LLM call); with only `OPENROUTER_API_KEY` set, chat
calls fall back to OpenRouter. An exported shell variable wins over `.env` silently. `[models]
hunter` in the config enables the tool hunter in `standard`; it is off by default.
## Exit code: `--fail-on`
| Policy | Exit 1 when |
| --- | --- |
| `never` (default) | never |
| `findings` | any finding was reported |
| `verified` | any verification result is `verified` (accepted at `static_corroboration` or above) |
| `worth-fixing` | a deep-mode sandbox verdict is marked `worth_fixing` |
A degradation never changes the exit code. Read `manifest.json` for it.
## Recorded degradations
Each entry in `manifest.json``degradations` is `{"stage": ..., "reason": ...}`. The ones to expect:
-`cpg_unavailable` (stage `model`): no Joern on the host; the model layer ran nothing.
-`regress_retired` (stage `regress`): deep mode requested the retired snippet stage; the scan equals standard.
-`learning_endpoint_unavailable` (stage `model`): no provider key resolved. The graph's entailed
findings are still reported; the residual question is skipped. This is recorded in `manifest.json`
but has no line in `report.md`'s "What could not be analysed" section.
-`hunter_model_unavailable` (stage `map`): `[models] hunter` is unset, so the tool hunter did not run.
Recorded on every standard scan without that setting; it is expected, not an error.
- coverage facts from the graph, rendered in `report.md` under "What could not be analysed":
`region_failed`; plus `max_findings_exceeded` and `facts_unavailable` (overlay, obligations).
Prove the instrument read its input before believing a zero: `manifest.json` names the stages that
ran and the files the preprocess step kept (`preprocess/file_targets.json`).
## Evidence ladder
`src/openultrasast/verification.py` defines the levels in this order. A level moves one rung at a
time and never backward.
| Level | Meaning | Produced today by |
| --- | --- | --- |
| `suspicion` | a rule, model or weak signal raised a hypothesis | the Joern model layer (its rung, `model_entailed` or `suspicion`, travels in the finding's tags), the tool hunter, authorization-obligation checks |
| `static_corroboration` | deterministic code evidence at the line | quick rules, the semantic overlay |
| `crash_reproduced` | an artifact-backed reproducer | nothing: there is no crash reproducer. Deep-mode sandbox verdicts (`verdicts.json`, `worth_fixing`) do not raise the level |
| `root_cause_explained` | independent verifier reasoning | nothing: the LLM judge was retired in v2.0.0 |
| `exploit_demonstrated` | a demonstrated exploit | nothing |
| `patch_validated` | relevant checks re-run after a patch | nothing: there is no patch stage |
The verifier's status per finding (`verification.json`): below `static_corroboration` is
`needs_evidence`; `static_corroboration` with reachability `unknown` is `rejected` (the tie-breaker
asks for call-graph, route, CLI, parser or dynamic evidence); `static_corroboration` with
function-level reachability is `accepted`. `verified` means accepted at `static_corroboration` or
above, and that is what `--fail-on verified` counts. There is no configurable report threshold: the
`[evidence]` config section was retired in v2.0.0 and is ignored.
## Run artifacts
Under `.openultrasast/runs/<scan-id>/`:
-`report.md`, `report.sarif` (SARIF 2.1.0 for code hosts and IDEs), `manifest.json` (stages, degradations, model payload)
-`findings.json`, `verification.json`, `score.json`, `shadow_findings.json` (rules shipped as `shadow`)
-`harness.json`: the resolved harness configuration the run used
description:OpenUltraSAST finding triage - false-positive adjudication with the fixed reason taxonomy, the evidence ladder, what a verdict may feed (labels, never rule edits), and fixing then re-scanning. Use when reviewing findings, reducing false positives, deciding whether evidence is enough, or auditing a fix for a finding.
---
# OpenUltraSAST Triage
Use this skill when triaging OpenUltraSAST findings or auditing a fix for one.
## Triage Loop
1. Identify the finding: id, path, rule, CWE and policy severity, confidence, evidence level and
verification status (`findings.json`, `verification.json` in the run directory).
2. Separate deterministic evidence (rule match, graph entailment, attached reachability) from model
closes="Pathfinder U8 lists which modules only the retired HarnessX loop used (engine_alerts, harvest, workspaces, profile are the suspects); delete those"
[capability.regress]
status="removed"# f5a904b, 2026-10-13: deep mode records regress_retired; proof is the search line
modules=["regress"]
lines=549
closes="Pathfinder U9: if search/ covers the triggerable verdict, delete regress/ and the deep-mode REGRESS branch in cli.py:839-880"
[capability.improve]
status="removed"# 94e96c6, 2026-10-13
modules=["improve"]
lines=1175
note="HarnessX-era improve lever; no active spec names it"
closes="U1 found fusion decisions never reach findings or score, verification runs before MAP, processors unreachable; decide keep-and-fix or delete per module"
[capability.mcp-and-skills]
status="parked"
spec="mcp-shift-left-agent"
modules=["mcp.py"]# skills.py and index.py removed f5a904b, 2026-10-13
lines=830
closes="see spec.mcp-shift-left-agent"
[capability.gates]
status="review"
modules=["gate.py","map_gate.py","pair_gate.py"]
lines=428
closes="U10: keep gate.py (CI + plane loop); delete map_gate.py and pair_gate.py if CI does not need them"
[capability.push-evaluation-instruments]
status="review"
modules=["push_scoring.py"]# push_inputs.py moved to benchmarks/push 2026-10-13; push_scoring stays (eligibility.py:18)
lines=462
closes="U3: move under benchmarks/push or delete if only tests use them"
note="maintainer 2026-10-13: 'semgrep oss is a good idea'; brief in benchmarks/measurements/2026-10-13-semgrep-brief/ says registry rules are not shippable, our converted rules are; 4.7 s startup floor"
closes="run the Semgrep arm on the valid pilot (benchmarks/valid/semgrep_arm.py) as a comparison; it may replace the regex quick tier only (same job: pattern matches), never act as a filter on other tiers. A Semgrep miss is never evidence (maintainer 2026-10-13). If it matches the quick tier at least as well, retire findings.py and ruleset/*.py and amend safety-net.md with the measurement"
| `entry:express:app-route` | for each `method` in `get`, `post`, `put`, `delete`, `patch`, `all`, `use`: ID `entry:express:app-route:<method>`, fact role `entry_registration` | `qualified_target`: `QUALIFIED_NAME`, field `qualified_target`, expected `app.<method>`; `call_shape`: `AST_SHAPE`, field `call_shape`, expected `member_call`; `receiver`: `RECEIVER_TYPE`, field `receiver`, expected `express_application`; `arguments`: `ARGUMENT_SHAPE`, field `arguments`, expected `route_path_and_handler` | kind `REGISTRATION`, role `entry_registration`, source `framework_instance`, target `entry_handler`, conditions `()` |
| `entry:express:router-route` | for the same methods: ID `entry:express:router-route:<method>`, fact role `entry_registration` | `qualified_target`: `QUALIFIED_NAME`, field `qualified_target`, expected `router.<method>`; `call_shape`: `AST_SHAPE`, field `call_shape`, expected `member_call`; `receiver`: `RECEIVER_TYPE`, field `receiver`, expected `express_router`; `arguments`: `ARGUMENT_SHAPE`, field `arguments`, expected `route_path_and_handler` | kind `REGISTRATION`, role `entry_registration`, source `framework_instance`, target `entry_handler`, conditions `()` |
| `entry:express:express-app` | ID `entry:express:express-app`, fact role `framework_instance` | `qualified_target`: `QUALIFIED_NAME`, field `qualified_target`, expected `express`; `call_shape`: `AST_SHAPE`, field `call_shape`, expected `factory_call`; `binding`: `ARGUMENT_SHAPE`, field `binding`, expected `application_instance_binding` | kind `REGISTRATION`, role `framework_instance`, source `framework_factory`, target `framework_instance`, conditions `()` |
The promoted route-method values are read from reviewed projection data rather than copied
into engine branches. A pattern hit alone is `INCOMPLETE`; it cannot produce
`FactMatchEvidence`, enter readiness, or support admission. The no-context legacy loader is
removed only after `mapping/legacy.py` uses the authority-bearing path, a repository census
finds no caller that treats the projection as v2 evidence or bypasses authority, and the
core conformance handoff passes. `analyze_entry_points` remains the stable public entry;
bootstrap and adapter construction occur at its target-aware mapping boundary rather than
| Extend the checker's inputs (chosen) | Supply paths, entries and witnesses to `check_obligations` from the hunter's context | One checker, one dominance rule, facts stay closed data | Order dominance is textual inside a function; nesting-aware dominance (`guard-dominance-regime`) never existed | Honest for "check present before operation"; the Jenkins shape |
| New prover module with its own graph | Build a call graph and CFG per change | Precise dominance | Weeks of work, duplicates the checker, no CFG for Java or PHP in the tree | Rejected for this increment |
| Joern engine | Dataflow and call graph from the engine | Precise | 225 s per batch, pilot produced no findings, no PHP in the budget | Rejected as the gate; usable later as a proof backend |
## Design Decisions
### Decision: paths from the caller index, not a call graph
- **Context**: Requirement 3.
- **Alternatives Considered**: a per-change call graph over all product files; the engine's call graph.
- **Selected Approach**: an upward walk from the function through `source.callers_of` on the cached index,
bounded by depth and fan-out, stopping at the first entry point of each chain.
- **Rationale**: the index exists, is cached per change, and the pool's shapes are shallow (the operation is in the
web method or one helper below it).
- **Trade-offs**: name-based caller resolution conflates same-named functions across files; chains are kept per
file but a caller of `save` in another class may be a false hop. Accepted and visible in the evidence.
- **Follow-up**: measure how many positives are reached at depth 1, 2 and 3 on the pool.
### Decision: framework enforcement as router and decorator witnesses
- **Context**: Requirement 2.
- **Selected Approach**: a small recogniser per framework reads route registrations and attributes and emits
`Discharge` witnesses with scope `router` or `decorator` for the handlers they cover; passed through
`extra_witnesses`.
- **Rationale**: the dominance rule already accepts those scopes; the facts files stay about calls.
- **Trade-offs**: coverage is as wide as the recognisers; an unrecognised middleware name means a finding on a
guarded route. Every recogniser lists its names in the facts header and the coverage statement counts routes
with unrecognised middleware.
### Decision: an opt-in premise in the checker, not a synthesized policy
- **Context**: Codex's first attempt stopped on a real conflict: with path records and neither a declared clause
nor a sibling norm the checker returns no finding by design (gating 3), so the prover's core verdict could not
be emitted without a change.
- **Alternatives Considered**: synthesize a `DeclaredPolicy` in memory marking every recognised entry
`authenticated` (dishonest: the finding would read "route declared authenticated" when nothing was declared);
a second checker in the prover (duplicates the rules); an opt-in keyword and a distinct label (chosen).
- **Selected Approach**: `assume_entries_protected=False` keyword, label `unguarded_path`, evidence naming the
premise. Default behaviour and tests unchanged.
- **Trade-offs**: the label makes the premise visible in every finding and in the static tags; consumers that
grouped by label need the new value (revalidation trigger for `authorization-obligations`).
### Decision: no model call inside the prover
- **Context**: Requirement 4.4; the maintainer's direction after the engine pilot: the model's job is the trust
boundary question the static side cannot answer, asked after a static result exists.
- **Selected Approach**: the prover is deterministic; the intent adjudication hook of the checker stays off in
the arm; a later increment may wire it as a separate, measured arm.
## Risks & Mitigations
- Jenkins web methods whose protected operation is an outbound connection test with stored secrets
(`doTestConnection`) call no listed operation name — mitigate by adding operation rows for credential
lookups and HTTP client calls used in those handlers, checked in the clones, and by counting the handlers the
facts do not cover.
- Textual order dominance inside a function misses a check in an `else` branch — accepted for this increment;
recorded as a degradation when a witness exists but does not dominate.
- Same-name conflation in callers — chains kept per file; evidence names the hop files.
- Over-wide entry recognition (every `doXxx`) raises false alarms on helper methods — the Stapler recogniser
requires a public method on an exposed type or a Stapler-typed parameter; the pool's ordinary pushes measure
"approval_note":"2026-10-14 maintainer directive in session: spec the prover and queue implementation with Codex (fast-track; the three gates were passed on that instruction, review the documents before the second delivery). 2026-10-29: requirements amended after the 2026-10-28 gates 1-5 measurements (trace resolution, programmatic entry facts, failure visibility). 2026-10-29 (later): maintainer instruction in session ('can you address the issues?') treated as requirements approval and as the directive to regenerate the design. 2026-10-29 (capability lift): per-target vocabulary became acquired data; engine/target split, pattern discipline and complexity caps were recorded in steering; all phases reopened. 2026-10-29 (abstract-interpretation amendment): the maintainer accepted the design-review recommendations by asking to make the amendments. Requirements and design were regenerated with a three-way judgment, explicit evidence algebra, stable symbols, resource/authority domains, label-blind acquisition and separate admission measurement; that directive is recorded as approval to regenerate tasks. The merged task plan passed an independent task-graph sanity review after two design repairs. Tasks await human approval before implementation. 2026-10-29 (implementation start): the maintainer invoked /kiro-impl access-control-prover in session; that directive is recorded as tasks approval. 2026-11-01: maintainer approved moving the production fact authority, matcher and artifact writer from detection-core into access-control-prover; Requirement 13, design ownership and the repaired tasks are approved for implementation. 2026-11-01 (repo-local review ceremony): maintainer approved recording the generator as producer, maintainer session approval as reviewer basis, and the checked-in digest-addressed directory as trusted store; Tasks 14.3-14.4 implement the wire/resolver and ceremony records before data migration in Task 14.5."
- [x] 14.1 Define the production access-control family contract
- Encode the versioned access-control security object, roles, three decision rules, residual question and closed incompleteness reasons through public detection-core types.
- Keep target names confined to the reviewed ecosystem declaration while pinning every digest-affecting field, status, origin policy and decision semantic declared by the design.
- Done when the canonical production contract validates, round-trips deterministically, and rejects every altered or undeclared role, decision, field and reason fixture.
- Define the immutable authority bundle and separate host bootstrap policy over reviewed pack/projection references, policies, lineages, matcher identity, reviewer/store admission, forbidden identity pairs and conflict disclosure.
- Re-derive canonical policy, attestation, disclosure and bundle identities before any target data is consumed; never let the bundle choose its own trust roots.
- Done when a complete independently reviewed authority fixture authenticates and every missing, duplicate, substituted, self-authored or non-canonical variant fails before pack loading.
- _Requirements: 13.2, 13.3_
- _Boundary: Fact Authority_
- _Depends: 14.1_
- [x] 14.3 Implement installed authority codecs and resolver
- Implement closed canonical JSON decoders for the bootstrap policy, installed authority, v2 fact packs, nested policies/lineages/evidence and immutable content references.
- Implement the checked-in digest-addressed resolver using only validated `content_digest` for filesystem addressing and re-derivable storage receipts under an injected bootstrap-admitted store identity.
- Done when canonical synthetic installed fixtures round-trip to Task 14.2 typed values and unknown/missing fields, noncanonical bytes, traversal-like artifact IDs, digest paths or byte substitutions fail before authentication.
- _Requirements: 13.2, 13.3_
- _Boundary: Installed Fact Authority Wire_
- _Depends: 14.2_
- [x] 14.4 Record the approved repo-local review ceremony
- Write the three closed canonical identity manifests with the exact schema, role, name and approval-basis values declared by the design.
- Bind bootstrap reviewer admission, custody policy, authority review and pack evidence identities to those manifest digests while keeping all trust/store admission outside `authority.json`.
- Done when the approved manifests and bootstrap load through Task 14.3, all identity bindings validate, and changing any field or approval basis invalidates authentication.
- _Requirements: 13.2, 13.3_
- _Boundary: Installed Authority Review Ceremony_
- _Depends: 14.3_
- [x] 14.5 Migrate the installed entry knowledge into reviewed v2 packs
- Produce one language-scoped v2 pack per promoted language and a separately authenticated canonical compatibility projection from the existing reviewed entry data.
- Map every migrated row to declared roles, typed structural requirements and relation effects without adding target vocabulary to engine modules.
- Preserve legacy pattern recognition byte-for-byte while preventing the projection from constructing detection-core evidence or readiness.
- Done when legacy recognition fixtures are unchanged, both reviewed language packs validate under task 14.2 authority, and projection-only input remains non-authoritative.
- [x] 14.6 Add canonical structural-observation and fact-match persistence
- Provide the downstream write boundary that persists canonical `StructuralObservation` bytes and returns immutable references through a read-only resolver.
- Persist completed `fact_match_bytes` only after its observation references exist, then return the match reference without inserting a circular self-reference into the match digest.
- Done when stored observation and match bytes resolve canonically and byte or reference substitution is refused deterministically.
- _Requirements: 13.5_
- _Boundary: Fact-Match Artifact Writer_
- [x] 14.7 Implement the authority-bearing structural matcher
- Build typed observations from repository structure for the requirements declared by the selected reviewed fact; do not infer missing structure from a regex or name hit.
- Use task 14.6 to persist every observation and completed `FactMatchEvidence`, then pass the immutable match reference with family contract, policies, lineages, custody and read-only resolver to detection-core.
- Bind matcher behavior and fact lineage to current custody and the evaluation population so same-population label influence is incomplete or refused.
- Done when complete fixtures apply the declared relations, while name-only, partial, ambiguous, substituted and contaminated fixtures cannot produce complete evidence.
- [x] 14.8 Produce the downstream conformance handoff
- Run detection-core fact-pack, fact-match, application and full-evidence validators over target-owned fixtures after proving nonzero fixture reads.
- Publish an immutable manifest naming every fixture, artifact digest, validator result and negative-case reason for detection-core task 9.4 to accept; do not add target fixtures to core.
- Done when the complete target-owned fixture passes, every incomplete/self-authored/substituted/contaminated/legacy-only variant is refused with a stable reason, and the immutable handoff manifest is reproducible.
- [x] 14.9 Migrate the production entry caller and prove zero bypasses
- Wire the externally bootstrapped authority-bearing adapter into the target-aware `mapping/legacy.py` entry path while keeping `analyze_entry_points` stable for its callers.
- Retain the legacy projection only for compatibility recognition; prohibit it from supplying v2 evidence, readiness or admission.
- Run import/caller and artifact-writer censuses proving no production path bypasses authority or treats legacy projection output as v2 evidence.
- Done when existing entry recognition remains byte-stable, explicit v2 matching uses the reviewed persisted path, and both production bypass censuses are zero.
- _Requirements: 1.1, 1.5, 1.7, 13.4, 13.7, 13.8_
- _Boundary: Production Mapping Integration_
- _Depends: 14.8_
- _Blocked: detection-core task 9.4 must accept the immutable handoff from task 14.8 before production caller migration_
## Task constraints
- Complexity: McCabe at most 10 per touched function, at most 50 logical lines per function and 400 per new module.
- Engine/target split: abstract-domain, resolution, summary, judgment and trace modules contain no target names.
- Target knowledge: names enter only through promoted facts or registered target-aware strategies.
- Compatibility: legacy checker defaults and completed baseline records remain unchanged.
- Instrument rule: validation must prove nonzero input reads before interpreting any clean or zero result.
## Implementation Notes
- Task 14.3 planning: the 963-byte legacy entry file was readable, but typed authority validation was not an installed-artifact format. The maintainer approved the repo-local review ceremony; Tasks 14.3–14.4 now separate wire/resolver implementation from the identity/bootstrap records before Task 14.5 migrates data.
## Baseline implementation record
- Deliveries A-C implemented the baseline paths, recognisers, witnesses, handler facts, prover, traces and CFG with
static checks and focused tests passing at delivery time.
- The full development arm then processed all 483 changes with zero instrument failures. Its maximum runtime exceeded
the amended hard budget, so the baseline is evidence, not an eligible arm.
- The 2026-10-28 ablation identified call resolution, resource/authority precision and failure visibility as the
amendment levers. Those observations inform tasks 7-13 but are not admission labels.
Order follows the dependency chain: ax bring-up (parallel with 1-2) -> manifests -> budget -> runner ->
ax-backed reconciler -> tasks -> increment run on ax -> measurement. Every task ships with its tests; a task is done when its tests pass in the host suite, ruff and
mypy are clean, and the commit is gated on pytest's own exit code. Script-level pipeline work stays paused.
- [x] 0. ax bring-up on this host
- Install `kind`, `kubectl`, the `ax` CLI and `ko` (user-local). `ops/ax/up.sh`: create the kind cluster and
install Agent Substrate with its `hack/create-kind-cluster.sh` and `hack/install-ate-kind.sh
--deploy-ate-system`, deploy the ax control plane with `ko` to the kind registry, apply one trivial Task
and wait for it to finish; `ops/ax/down.sh` tears it down. Record the measured idle footprint (RSS, CPU)
in `ops/ax/README.md` next to the 7 GB host constraint.
-`ousast plane doctor` (in `reconciler.py` or a sibling `doctor.py`): the four checks of Requirement 3.5.
starts without a request. `plane/Dockerfile.runner`: this package on Python 3.12, installed to
`/usr/local/bin/ax-task-runner`, loaded into kind by `ops/ax/up.sh`.
-`tests/test_plane_runner.py`: the env contract on a fixture workspace; readyz 503 then 200; delivery to a
local receiver; delivery failure yields `failed`; the child's cwd and `AX_METADATA_URL`; the runner stays
up after the command; SIGTERM kills a long-running command's group; the marker prevents a rerun; no command
before a start request, 409 on a wrong run/task and on a second start, the credential reaches the child
(its hash in the artifact) and no file or log record, a crash printing it is redacted, and of two runners
with identical env only the addressed one runs (the golden scenario).
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6_
- [x] 4. ax-backed reconciler and CLI
-`src/openultrasast/plane/reconciler.py` (under 500 lines, no pipeline logic, no local subprocess
executor): `state.json`, statuses, topological submission through the `ax` CLI (path injectable),
`ax resume task` after `ax apply` (a created Task is `Suspended`), retrying DeadlineExceeded/Unavailable
with backoff up to `OUSAST_RESUME_TIMEOUT`; completion is the artifact delivery, never an ax phase (ax has
none for a finished command); `Failed` or a vanished task before delivery, or no delivery within
`OUSAST_TASK_TIMEOUT`, fails the task with ax's condition message; the artifact HTTP receiver, `serialize` labels never overlapping, `--workers`, budgets and
artifact env passed in the rendered Task manifest, outcome read from the delivered `summary.json`, skip
`done` on rerun, PID lock per run, `ax delete` of finished Tasks. Once ax reports `Running` after a
resume, one start request per resume through Agent Substrate's router (`src/openultrasast/plane/router.py`:
`kubectl port-forward svc/atenet-router` for the run, or `OUSAST_ROUTER_URL`; header
`ate-target-actor: <atespace>/<task>`; 502/503/504 and connection errors retried up to
`OUSAST_START_TIMEOUT`, a 409 final), carrying the bound Model's `secretKey.key` variable from the
reconciler's environment; a Model-bound task whose variable is unset fails before `ax apply`.
Task-produced inputs travel as `OUSAST_INPUTS`, not as a rendered Workspace: the runner fetches them after
the start from the receiver's authorised `GET /inputs/<producer>/<artifact>` (a 182 KB `facts.json` in
`AX_WORKSPACES_YAML` exceeded Substrate's 32768-character env limit on the live cluster, 2026-09-29).
-`ousast plane run <Run.yaml>`, `ousast plane status <run>` (token attribution table per task and Model
from the tasks' `summary.json` usage; `--units` from `units.jsonl`; also written to `attribution.json`),
`ousast plane workspaces <population.toml>` (generates Workspace manifests per case pin).
-`tests/test_plane_reconciler.py`: a fake `ax` CLI with ax's real lifecycle (Suspended, a first resume
failing DeadlineExceeded, Running without a Completed phase) drives a DAG: order, non-overlap,
skip-on-rerun, `unfinished` and `failed` propagation, resume retry and timeout, `Failed` with a condition
message, the task timeout without delivery, the lock, the attribution table, and the
line-count assertion; with a fake router: one start per task after the accepted resume and before the
delivery, the credential in the verify start only and in no file under the run or tmp dirs, a missing
secret failing the task, a refused start failing it. `tests/test_plane_router.py`: port parsing from a
fake kubectl and its termination on exit, retry on 503/502 then 202, 409 raised with redacted text, the
start timeout. `tests/test_plane_ax_live.py` (marker `ax`, skipped without a live cluster): one
@@ -8,9 +8,9 @@ Nothing in the pipeline can represent "this operation should have been guarded"
## Desired Outcome
Absence bugs become first-class, measured, and honest. An obligated operation (protected data access, privileged action, security configuration) reached from an entry point without a discharger (a dominating auth guard, an identity constraint bound from the authenticated context, a non-permissive value) is reported as an obligation violation, with the missing discharger kind, the evidence for why the obligation exists (sibling handlers that discharge it, or a declared policy clause), and the known fix shape the corpus taught. Leave-one-out on the absence pairs is the objective, per obligation kind and per provenance profile, so the number rises as the corpus grows. `agent-vfc` and `vibe-py` absence pairs move from 0 detected to a measured baseline that the engine specs can raise.
Absence bugs become first-class and are measured without overstating results. An obligated operation (protected data access, privileged action, security configuration) reached from an entry point without a discharger (a dominating auth guard, an identity constraint bound from the authenticated context, a non-permissive value) is reported as an obligation violation, with the missing discharger kind, the evidence for why the obligation exists (sibling handlers that discharge it, or a declared policy clause), and the known fix shape the corpus taught. Leave-one-out on the absence pairs is the objective, per obligation kind and per provenance profile, so the number rises as the corpus grows. `agent-vfc` and `vibe-py` absence pairs move from 0 detected to a measured baseline that the engine specs can raise.
Principle written into the roadmap: **an operation carries obligations, not only data; the application's own consistency and its declared policy say which, and the tool reports the missing discharger rather than a phantom flow.**
Principle written into the roadmap: **an operation carries data and obligations; the application's own consistency and its declared policy say which, and the tool reports the missing discharger rather than a phantom flow.**
## Approach
@@ -18,8 +18,8 @@ Obligated operations and discharger kinds are closed facts data, like sinks and
## Scope
-**In**: obligated-operation and discharger facts; obligation shapes derived from pairs; consistency detection across sibling handlers; optional declared policy file with a closed schema; path-aware violation evaluation with a function-local degraded mode; new evidence rungs and report wording for obligations; worth-fixing weighting by resource sensitivity; corpus recipe shape for absence pairs (handler plus registration context); leave-one-out per obligation kind; `obligations` improve lever; offline tests.
-**Out**: taint or facts changes for flow bugs; the call graph and path records themselves (reachability spec); guard-dominance IR (guard-dominance-regime); LLM-authored policy or shapes; any merge gate; vendoring unlicensed code; sandbox proof of absence bugs (they do not crash).
-In: obligated-operation and discharger facts; obligation shapes derived from pairs; consistency detection across sibling handlers; optional declared policy file with a closed schema; path-aware violation evaluation with a function-local degraded mode; new evidence rungs and report wording for obligations; worth-fixing weighting by resource sensitivity; corpus recipe shape for absence pairs (handler plus registration context); leave-one-out per obligation kind; `obligations` improve lever; offline tests.
- Out: taint or facts changes for flow bugs; the call graph and path records themselves (reachability spec); guard-dominance IR (guard-dominance-regime); LLM-authored policy or shapes; any merge gate; vendoring unlicensed code; sandbox proof of absence bugs (they do not crash).
## Boundary Candidates
@@ -40,13 +40,13 @@ Obligated operations and discharger kinds are closed facts data, like sinks and
## Upstream / Downstream
-**Upstream**: `EntryPointRecord` (route, access level, decorators, conditions); `FileIR`; path records and dominance from the two engine specs; `PairCase` with `review_tier`; `MechanismStore` and the lever machinery; facts loader.
-**Downstream**: reports and SARIF; worth-fixing ranking; the roadmap's leave-one-out table gains obligation kinds; `guard-dominance-regime` consumes obligation discharger kinds as guard candidates for web code.
- Upstream: `EntryPointRecord` (route, access level, decorators, conditions); `FileIR`; path records and dominance from the two engine specs; `PairCase` with `review_tier`; `MechanismStore` and the lever machinery; facts loader.
- Downstream: reports and SARIF; worth-fixing ranking; the roadmap's leave-one-out table gains obligation kinds; `guard-dominance-regime` consumes obligation discharger kinds as guard candidates for web code.
## Existing Spec Touchpoints
-**Extends**: `corpus-seeded-mechanisms` (second shape family, same store, same lever contract), `pair-corpus-honesty` (registration-context excerpt mode, obligation label), `three-stage-scan` (one more proposer in MAP, new rungs below proof).
-**Depends**: `reachability-flow-model` (entry point to operation paths); `guard-dominance-regime` (a guard dominates the path). Reachability is now on the critical path for two consumers.
- Extends: `corpus-seeded-mechanisms` (second shape family, same store, same lever contract), `pair-corpus-honesty` (registration-context excerpt mode, obligation label), `three-stage-scan` (one more proposer in MAP, new rungs below proof).
- Depends: `reachability-flow-model` (entry point to operation paths); `guard-dominance-regime` (a guard dominates the path). Reachability is now on the critical path for two consumers.
-`src/openultrasast/cli.py` —`_check_obligations` after `_search_variants` in MAP (standard/deep); manifest `obligations` block; `mechanisms export` writes obligation rows too.
-`src/openultrasast/pairs.py` —`ExpectedFinding.obligation: str | None`; `_detections`-side additive rule: for a row with `obligation`, a finding with tag `obligation:<kind>` inside the labeled function counts.
-`src/openultrasast/semantic/mechanisms.py` — no schema change; `corpus_mechanisms()` unchanged; a helper `obligation_mechanisms(records)` selects `shape["family"] == "obligation"` (added in `obligations/shapes.py`, not in the store module).
-`src/openultrasast/cli.py`:`_check_obligations` after `_search_variants` in MAP (standard/deep); manifest `obligations` block; `mechanisms export` writes obligation rows too.
-`src/openultrasast/pairs.py`:`ExpectedFinding.obligation: str | None`; `_detections`-side additive rule: for a row with `obligation`, a finding with tag `obligation:<kind>` inside the labeled function counts.
-`src/openultrasast/semantic/mechanisms.py`: no schema change; `corpus_mechanisms()` unchanged; a helper `obligation_mechanisms(records)` selects `shape["family"] == "obligation"` (added in `obligations/shapes.py`, not in the store module).
I --> J[StaticFinding suspicion; tags obligation:*, discharger:*, obligation_evidence:*]
```
Gating: the checker emits a finding only when (1) the operation is reached — from an entry point along a `PathRecord`, or, without paths, when the operation sits in a handler that is itself an entry point — and (2) no discharger of the required kind dominates, and (3) the obligation is justified by a sibling set (`consistency_violation`), a policy clause (`declared_policy_violation`) or, in degraded mode without either, by the operation fact alone (`function_local`). A declared public route suppresses path-guard obligations for that route only.
Gating: the checker emits a finding only when (1) the operation is reached (from an entry point along a `PathRecord`, or, without paths, when the operation sits in a handler that is itself an entry point) and (2) no discharger of the required kind dominates, and (3) the obligation is justified by a sibling set (`consistency_violation`), a policy clause (`declared_policy_violation`) or, in degraded mode without either, by the operation fact alone (`function_local`). A declared public route suppresses path-guard obligations for that route only.
> **Status 2026-10-02:** dormant. Tasks were approved but none of the 24 was started; the absence-bug case it targets is handled in practice by the dominance arbiter (`model/dominance.py`) and `semantic/obligations/`. Reopen it only with a measured gap the arbiter misses.
## Introduction
AppSec engineers and maintainers of OpenUltraSAST scan web code, increasingly vibe-coded or agent-written, in which the dominant bugs are absences rather than flows: a handler queries a user-owned record without constraining it to the caller, a privileged route has no guard on its path, an identity is read from the request body instead of the authenticated context, a security control stays at its permissive default. Today the detector, the overlay, and the variant search model only source-to-sink flows, so these pairs score zero and cannot teach a mechanism (6 of 19 non-teaching vibe-py pairs; 15 of 29 agent-vfc rows carry absence labels). This spec adds obligations: an operation that must be discharged by a guard, an identity constraint, or a safe value before it is reached. The application's own policy, recovered from the consistency of its sibling handlers and optionally declared in a versioned file, says which operations carry which obligations. Violations are reported with the missing discharger and its evidence, measured leave-one-out on the absence pairs, and admitted into the detector through the existing improve-loop gate.
## Boundary Context
-**In scope**: obligated operations and discharger kinds as closed facts data; obligation shapes learned from trusted pairs; consistency detection across sibling handlers; an optional declared policy file with a closed schema; path-aware violation evaluation with a function-local degraded mode; obligation evidence rungs, report wording and worth-fixing weighting; a corpus recipe shape that keeps the handler's registration context; leave-one-out per obligation kind; an `obligations` improve lever; offline tests.
-**Out of scope**: flow taint, walker or sink facts; the call graph, path records and dominance computation (owned by `reachability-flow-model` and `guard-dominance-regime`); model-authored policy or shapes; sandbox proof of absence bugs; any merge gate; vendoring unlicensed code.
-**Adjacent expectations**: the entry-point mapper keeps providing routes with an access level and their decorators; the reachability spec provides entry-point-to-operation paths and the guard-dominance spec provides "a guard dominates the path"; the mechanism store, its deterministic ids and the lever contract from `corpus-seeded-mechanisms` are reused unchanged; the pair scorer accepts one additive label field; `local` remains the only slice that can fail CI.
- In scope: obligated operations and discharger kinds as closed facts data; obligation shapes learned from trusted pairs; consistency detection across sibling handlers; an optional declared policy file with a closed schema; path-aware violation evaluation with a function-local degraded mode; obligation evidence rungs, report wording and worth-fixing weighting; a corpus recipe shape that keeps the handler's registration context; leave-one-out per obligation kind; an `obligations` improve lever; offline tests.
- Out of scope: flow taint, walker or sink facts; the call graph, path records and dominance computation (owned by `reachability-flow-model` and `guard-dominance-regime`); model-authored policy or shapes; sandbox proof of absence bugs; any merge gate; vendoring unlicensed code.
- Adjacent expectations: the entry-point mapper keeps providing routes with an access level and their decorators; the reachability spec provides entry-point-to-operation paths and the guard-dominance spec provides "a guard dominates the path"; the mechanism store, its deterministic ids and the lever contract from `corpus-seeded-mechanisms` are reused unchanged; the pair scorer accepts one additive label field; `local` remains the only slice that can fail CI.
-**Discovery Scope**: Extension of the semantic layer (facts, shapes, store, MAP proposers), the entry-point mapper, reports, the pair corpus, leave-one-out and the improve lever; complex integration because it consumes two unimplemented upstream specs.
-**Key Findings**:
- Feature: `authorization-obligations`
- Discovery Scope: Extension of the semantic layer (facts, shapes, store, MAP proposers), the entry-point mapper, reports, the pair corpus, leave-one-out and the improve lever; complex integration because it consumes two unimplemented upstream specs.
- Key Findings:
- The absence pairs are real and structured. `vampi-books-get-by-title` has identical code on both sides except a constraint `user=user` bound from `token_validator(request.headers.get('Authorization'))['sub']`; the vulnerable branch queries `Book` by title alone. The difference is the provenance of a constraint, not a flow. 15 of 29 `agent-vfc` rows and 6 of the 19 non-teaching `vibe-py` pairs carry absence labels (`missing_auth_guard`, `permissive_default`, `identity_from_request_body`, `secret_in_client`, `validation_strength`).
- The entry-point mapper already classifies every route as `public`, `authenticated` or `role-restricted` from its decorators (`mapping._python_route_access`), records the decorators as `access_evidence`, and lists `conditions` (`if` tests that look like checks). That is the sibling detector's upstream for path guards and needs no change.
-`FileIR` binds carry `value_text`, `names` and `call_name`; `variants.source_kind_of_name` already traces a name through binds to a fact source, a parameter or a container read. Provenance of a constraint's value (authenticated context versus request input versus constant) is the same trace with one more source class.
@@ -17,29 +17,29 @@
### What the absence pairs actually differ by
-**Context**: Decide whether absence bugs can be a shape family at all.
-**Sources Consulted**: `benchmarks/pairs/vibe-py` excerpts for `missing_auth_guard` and `permissive_default` rows; `benchmarks/pairs/agent-vfc/recipes.toml` mechanisms; `mechanisms.toml` definitions.
-**Findings**: three recurring differences: (a) a constraint added to a data access whose value comes from the authenticated context; (b) a guard call or decorator added before the operation (early return on a failed token check, `login_required`); (c) a literal changed from permissive to restrictive (`CORS(app)` → origins list, `DEBUG=True` → `False`, RLS enabled). A fourth, rarer: identity read from the body replaced by identity from the token (same as (a) seen from the other side).
-**Implications**: discharger kinds `identity_constraint`, `path_guard`, `ownership_check`, `non_permissive_value`, `validated_input`; the value provenance is the discriminating feature for (a) and (d).
- Context: Decide whether absence bugs can be a shape family at all.
- Sources Consulted: `benchmarks/pairs/vibe-py` excerpts for `missing_auth_guard` and `permissive_default` rows; `benchmarks/pairs/agent-vfc/recipes.toml` mechanisms; `mechanisms.toml` definitions.
- Findings: three recurring differences: (a) a constraint added to a data access whose value comes from the authenticated context; (b) a guard call or decorator added before the operation (early return on a failed token check, `login_required`); (c) a literal changed from permissive to restrictive (`CORS(app)` → origins list, `DEBUG=True` → `False`, RLS enabled). A fourth, rarer: identity read from the body replaced by identity from the token (same as (a) seen from the other side).
- Implications: discharger kinds `identity_constraint`, `path_guard`, `ownership_check`, `non_permissive_value`, `validated_input`; the value provenance is the discriminating feature for (a) and (d).
### Where the policy comes from
-**Context**: Absence is relative to intent; the tool must not invent intent.
-**Findings**: in every Real-Vuln web app the handlers on one resource share a router and a model; the vulnerable handler is the odd one out. Route access classification already exists. A declared policy is the only way to say "this route is meant to be public"; the hunter can only guess it.
-**Implications**: consistency first (no declaration needed, measurable now), declared policy second (stronger rung), hunter last (records intent, never raises).
- Context: Absence is relative to intent; the tool must not invent intent.
- Findings: in every Real-Vuln web app the handlers on one resource share a router and a model; the vulnerable handler is the odd one out. Route access classification already exists. A declared policy is the only way to say "this route is meant to be public"; the hunter can only guess it.
- Implications: consistency first (no declaration needed, measurable now), declared policy second (stronger rung), hunter last (records intent, never raises).
### Dominance before guard-dominance-regime exists
-**Context**: Req 5 needs "a discharger dominates the path"; the spec that owns dominance is not written.
-**Findings**: within one handler, the existing `FunctionIR` order of binds and calls gives statement order; a guard call followed by an early return before the operation line is dominance in the common case; decorators dominate the whole handler; middleware attached to the router dominates every route under it. Across calls, `PathRecord.hops` gives the sequence of nodes; a discharger on any hop before the operation node dominates the path when the hop's function returns early on failure.
-**Implications**: define `Dominance` as a narrow protocol (`dominates(witness, operation) -> bool`) with a built-in `OrderDominance` implementation (statement order in the handler, decorators, router middleware, hop order) that `guard-dominance-regime` may replace with nesting-aware dominance. The degraded mode (`function_local`) is `OrderDominance` without path records.
- Context: Req 5 needs "a discharger dominates the path"; the spec that owns dominance is not written.
- Findings: within one handler, the existing `FunctionIR` order of binds and calls gives statement order; a guard call followed by an early return before the operation line is dominance in the common case; decorators dominate the whole handler; middleware attached to the router dominates every route under it. Across calls, `PathRecord.hops` gives the sequence of nodes; a discharger on any hop before the operation node dominates the path when the hop's function returns early on failure.
- Implications: define `Dominance` as a narrow protocol (`dominates(witness, operation) -> bool`) with a built-in `OrderDominance` implementation (statement order in the handler, decorators, router middleware, hop order) that `guard-dominance-regime` may replace with nesting-aware dominance. The degraded mode (`function_local`) is `OrderDominance` without path records.
### Store and lever reuse
-**Context**: Avoid a second store and a second lever machinery.
-**Findings**: `Mechanism.shape` is a dict validated where used; `corpus_mechanism_id` hashes `shape.key()`; `validate_mechanism` checks guard and source kinds against `variants` constants. A second family needs its own closed kinds check and a `family` field in the shape dict; the tombstone, snapshot and profile clause need nothing.
-**Implications**: `ObligationShape.to_dict()` carries `family = "obligation"`; the lever validator dispatches on `family`; `search_tree` ignores obligation rows (they are not sink shapes) and the obligation checker ignores sink rows.
- Context: Avoid a second store and a second lever machinery.
- Findings: `Mechanism.shape` is a dict validated where used; `corpus_mechanism_id` hashes `shape.key()`; `validate_mechanism` checks guard and source kinds against `variants` constants. A second family needs its own closed kinds check and a `family` field in the shape dict; the tombstone, snapshot and profile clause need nothing.
- Implications: `ObligationShape.to_dict()` carries `family = "obligation"`; the lever validator dispatches on `family`; `search_tree` ignores obligation rows (they are not sink shapes) and the obligation checker ignores sink rows.
## Architecture Pattern Evaluation
@@ -54,23 +54,23 @@
### Decision: Obligation findings are `suspicion` with a data label, not new ladder rungs
-**Context**: Req 6.1 names three labels; the ladder is closed and owned elsewhere.
-**Selected Approach**: `StaticFinding.evidence_level = "suspicion"`, tags `obligation:<kind>`, `discharger:<kind>`, `obligation_evidence:<label>`; reports render the label. Dispositions and `ALLOWED_EVIDENCE` unchanged.
-**Trade-offs**: SARIF consumers see `suspicion` plus properties; honest and non-invasive.
- Context: Req 6.1 names three labels; the ladder is closed and owned elsewhere.
- Selected Approach: `StaticFinding.evidence_level = "suspicion"`, tags `obligation:<kind>`, `discharger:<kind>`, `obligation_evidence:<label>`; reports render the label. Dispositions and `ALLOWED_EVIDENCE` unchanged.
- Trade-offs: SARIF consumers see `suspicion` plus properties; honest and non-invasive.
### Decision: Dominance is a protocol with an order-based default
-**Context**: `guard-dominance-regime` is unwritten; this spec cannot wait for it to be useful.
-**Selected Approach**: `Dominance` protocol; `OrderDominance` default (decorator, router middleware, statement order before the operation, hop order along a path). The guard spec replaces it without touching callers.
-**Trade-offs**: order-based dominance misses guards inside conditionals; recorded as the known coarseness and measured on the fixed twins.
- Context: `guard-dominance-regime` is unwritten; this spec cannot wait for it to be useful.
- Selected Approach: `Dominance` protocol; `OrderDominance` default (decorator, router middleware, statement order before the operation, hop order along a path). The guard spec replaces it without touching callers.
- Trade-offs: order-based dominance misses guards inside conditionals; recorded as the known coarseness and measured on the fixed twins.
### Decision: Policy is never written by the tool
-**Selected Approach**: `declared policy` is a hand-maintained file with a closed schema; the lever may only *propose* a clause, journaled as pending until a human accepts; scans never read pending clauses.
- Selected Approach: `declared policy` is a hand-maintained file with a closed schema; the lever may only *propose* a clause, journaled as pending until a human accepts; scans never read pending clauses.
### Decision: Resource identity for sibling grouping
-**Selected Approach**: a handler's resource is the trailing segment of its route path with parameters stripped, joined with the operation's target token (model, table, collection name from the operation fact match); router or module is the second key. Coarse but deterministic and text-free.
- Selected Approach: a handler's resource is the trailing segment of its route path with parameters stripped, joined with the operation's target token (model, table, collection name from the operation fact match); router or module is the second key. Coarse but deterministic and text-free.
E -->|"Candidate[] sorted, bounded"| J[judgments.judge_candidates]
J -->|"one typed call per batch, no tools"| M[[chat endpoint]]
M --> P[predicates.check]
P -->|violation| J
P -->|clean| F[StaticFinding: family, detector, verifier tags]
F --> S[learning.scoring.score_pair_family]
P -.->|same definition| S
E --> C[CandidateReport: ceiling per family per slice]
```
The dotted edge is the point of Requirement 4: one definition of the predicates, used to constrain and to score.
### Technology Stack
| Concern | Choice | Why |
|---|---|---|
| Candidate enumeration | `semantic/ir.py``FunctionIR.calls` and `.binds` | 96.6% ceiling measured; the ruleset's 12.1% is below the current baseline |
| Guards in scope | `hunter_tools.obligations`, `mapping.analyze_entry_points` | already deterministic, already used by the obligations checker |
| Typed answer | `json_object` on the DeepSeek adapter, one array per batch | the provider has no schema mode; the predicates are the schema |
| Retry on violation | one bounded retry with the violation injected | DSPy Assertions' backtracking, without the framework |
| Determinism | candidates sorted, batches fixed, no tools | Req 2.4; the model's text may vary, the call sequence may not |
## File Structure Plan
### Directory Structure
```
src/openultrasast/learning/
├── candidates.py # NEW Candidate, CandidateKind, enumerate_candidates, ceiling, CandidateReport
├── judgments.py # NEW Judgment, judge_candidates, batching, the typed prompt
├── predicates.py # NEW PREDICATES, check_claim, violation reasons
├── slices.py # NEW DECIDING_SLICES, DEVELOPMENT_SLICES, overfitting_gap, cross_slice_verdict
├── detectors.py # MOD run_constrained_detector beside run_family_detector
└── scoring.py # MOD imports predicates; outcome rules unchanged, asserted by the recorded baseline
"approval_note":"2026-09-06 brief drafted at the maintainer's direction ('I'd rather spend time to design the exact DSPy style interface and evolutionary / statistical prompt optimization than fixing piece by piece'). Not approved; requirements not generated.; 2026-09-06 renamed from detector-optimizer to constrained-detector after the trajectory measurement: the brief's own conclusion is that the optimiser is not the next thing to build, and the spec should be named for what it delivers; 2026-09-06 requirements generated at the maintainer's direction, with the maintainer's emphasis recorded as Requirement 3 ('don't forget the emphasis on the real world bugs from the larger projects to avoid overfitting. we otherwise benchmark for cases that are not relevant'); 2026-09-06 requirements approved by the maintainer in session ('approve the requirements and run the design phase'); 2026-09-06 design approved by the maintainer in session ('approve the design and generate the tasks'); 2026-09-08 SUPERSEDED by execution-grounded-detection: this session's measurements showed the noise architecture (K-runs, per-family floor, regression budget, sign test, evolve loop) is a statistical treatment for a structural problem, and that the field (Big Sleep, AIxCC finals, Clearwing) grounds detection in execution rather than in scoring against a labeled corpus. The taxonomy, the IR candidate enumerator (96.6% ceiling) and the obligation checker survive as the web/logic detection path to contribute to a Clearwing-based substrate.",
- [x] 1. The ceiling, offline: what a candidate detector could possibly find
- [x] 1.1 Candidate enumeration from the semantic IR, shaped by family
- `learning/candidates.py`: `Candidate` (path, line, kind, name, text, argument texts and names, the binding chain the IR resolved, the guards in scope, the enclosing function, and an `id` of `path:line:name`); `CandidateKind` of call, bind, operation, config; `FAMILY_SHAPE` mapping each family to the kinds that can carry its bug; `enumerate_candidates` returning a `CandidateSet` with the candidates, the count discarded past the bound, and a reason of empty, `unsupported_language` or `no_candidate`.
- Call and bind candidates come from `FunctionIR.calls` and `.binds`; operation candidates for access control come from the obligations facts with the decorators and in-body guards in scope; config candidates are bindings whose value is a literal in a configuration position. The pattern ruleset is never the generator.
- Ordering is `(path, line, name)` and the bound is a constant, so the batches a later task forms are determined by the input alone.
- Observable: on the shared obligations fixture every family shape yields the sites its docstring names; a file the IR cannot parse yields `unsupported_language` and no candidates; a labeled function with no calls or binds yields `no_candidate`; enumerating twice returns an identical sequence.
- _Requirements: 1.1, 1.2, 1.4, 1.5, 1.6_
- [x] 1.2 The ceiling report and the `learning candidates` command
- `candidates.ceiling(cases, taxonomy)` returns, per family and per slice, the number of labeled functions for which at least one candidate was enumerated, the sites-per-function distribution, and the rows that yielded nothing with their reason; `ousast learning candidates --catalog --slice --json` prints it and writes the artifact. No model is called at any point.
- Observable: the command runs offline with no endpoint configured; the artifact reproduces the recorded web-slice figures (ruleset 12.1%, IR 96.6%, median 10 sites) and adds the deciding slices; running it twice produces a byte-identical artifact.
- _Requirements: 1.3, 3.5_
- _Depends: 1.1_
- [x] 1.3 The stop-or-go reading of the ceiling
- The artifact states, per deciding slice, the ceiling against that slice's recorded detection rate from round zero, and names the comparison as the condition on which this design continues. The reading is written into the spec's Implementation Notes with the numbers, whichever way it goes.
- Observable: the artifact carries agent-vfc, vfc-js, vfc and github ceilings beside 44.4% and the other recorded rates, and the notes state whether any deciding slice fails the condition.
- _Requirements: 1.3, 3.5, 3.7_
- _Depends: 1.2_
- [ ] 2. One definition of the checks, used to constrain and to score
- [ ] 2.1 (P) Predicates extracted from the scorer without changing it
- `learning/predicates.py`: `Violation`, the named predicates `line_in_parsed_function`, `sink_at_line`, `family_in_taxonomy` and `source_in_function`, and `check_claim`; `scoring._hits`, `_others` and `_fabricated` delegate to them.
- Observable: the scorer's unit tests pass unchanged, and a test reproduces the recorded round-zero numbers for vibe-py and agent-vfc from the committed artifacts exactly, so the refactor's observable effect is none.
- _Requirements: 4.1, 4.4, 6.1_
- [ ] 2.2 The unscorable vocabulary gains `no_candidate`
- `no_candidate` joins the reasons `unscorable_reason` can return, distinct from `unresolved_label` (the corpus named a function nothing parses) and from `unsupported_language` (the tooling could not parse at all); it leaves every denominator and is listed by reason.
- Observable: a pair whose labeled function has no call or bind is reported unscorable with `no_candidate` and appears in the per-family unscorable tally, not as a miss.
- _Requirements: 1.5, 6.1_
- _Depends: 1.1, 2.1_
- [ ] 3. The detector
- [ ] 3.1 Typed judgment, one bounded question per candidate, no tools
- `learning/judgments.py`: `Judgment` (candidate id, verdict, source expression, guard or empty, family, rationale) and `judge_candidates(candidates, config, client, model, taxonomy, violation=None)` making one call per fixed-size batch with `tools=[]` and JSON-object mode, the family's prompt and checklist as the system message, and the batch rendered as data. An answer naming a candidate id absent from the batch is dropped before any predicate runs.
- Observable: a scripted client shows one call per batch and no tool schema offered; a reply with a fabricated candidate id yields no judgment and increments the `fabricated_candidate` tally; a reply that is prose yields no judgment rather than a parsed guess.
- _Requirements: 2.1, 2.2, 2.3_
- _Depends: 1.1_
- [ ] 3.2 `run_constrained_detector`, with the predicates enforced and one retry
- `detectors.run_constrained_detector` enumerates, batches, judges, checks each judgment with `check_claim`, retries a violating batch exactly once with the violation injected, discards what still violates, and returns `StaticFinding`s tagged `family:`, `detector:` and `candidate:<id>`. `run_family_detector` is untouched and stays selectable.
- Observable: a judgment naming a line outside the parsed function is retried once and then discarded, and the discard is counted by predicate name; every finding carries the candidate id it came from; the unconstrained hunter still runs from the same configuration.
- _Requirements: 2.5, 4.1, 4.2, 4.3, 6.5_
- _Depends: 2.1, 3.1_
- [ ] 3.3 Two runs make the same calls
- A recording client captures the full sequence of `(model, messages, tools)` for a run; two runs over the same region and configuration produce an identical sequence.
- Observable: the test fails if candidate ordering, batching or the retry condition becomes input-dependent in a way the input does not determine; it is run repeatedly in the suite to catch order instability.
- _Requirements: 2.4_
- _Depends: 3.2_
- [ ] 3.4 Selecting the detector, and its degradations
- `[learning] detector = "hunter" | "constrained"` defaulting to `hunter`; `--detector` on `learning score` and `learning baseline`; without the semantic extra the constrained detector reports `unsupported_language` and scores nothing; without an endpoint it records `learning_endpoint_unavailable`; its spend goes through the client meter and stops at the cap.
- Observable: both detectors run from the same command over the same catalog; the extra-free matrix reports the recorded reason rather than failing; a run with no endpoint still writes its artifact.
- _Requirements: 7.1, 7.2, 7.4, 7.5_
- _Depends: 3.2_
- [ ] 4. Real-world code decides
- [ ] 4.1 (P) Deciding and development slices, and the overfitting gap
- `learning/slices.py`: `DECIDING_SLICES` of agent-vfc, vfc-js, vfc and github; `DEVELOPMENT_SLICES` of vibe-py; `overfitting_gap` over per-slice metrics; `cross_slice_verdict(before, after)` rejecting by name any change that improves a development slice and regresses a deciding one; `teachable(slice, cases)` false while every row of a slice sits below a gating tier.
- Observable: a change that improves vibe-py and regresses agent-vfc is rejected with agent-vfc named; the gap over the recorded baseline reads 83.3% against 44.4%; `teachable("agent-vfc", …)` is false today and the reason is the tier.
- _Requirements: 3.2, 3.3, 3.4, 3.6_
- [ ] 4.2 Per-slice publication, and memory kept out of the web aggregate
- `publish` prints a per-slice table, the overfitting gap beside every headline number, the statement that the deciding slices teach nothing today, and the memory family separately from any figure a reader would take as web-taxonomy performance; no aggregate is printed without its per-slice rows.
- Observable: the published section carries a row per slice and refuses to print a cross-slice aggregate alone; the 176 vfc rows appear under their own heading; the regenerated roadmap section is byte-identical on a second run.
- _Requirements: 3.1, 3.4, 3.6, 3.7_
- _Depends: 4.1_
- [ ] 5. The comparison, and what it retires
- [ ] 5.1 Constrained against unconstrained, per slice
- Both detectors scored by the same scorer over the same catalog at the same K, on every slice; detection, leak and pair-correct rates published against the recorded 64.9%, 12.3% and 52.6%, with cost per correct pair against $0.030 and $0.136. If the constrained detector is behind on any deciding slice, that is the headline of the artifact.
- Observable: one artifact holds both detectors' numbers per slice with the deltas, and its summary line names the deciding-slice result rather than the aggregate.
- _Requirements: 6.2, 6.3, 6.4_
- _Depends: 3.4, 4.2_
- [ ] 5.2 The trajectory and the floor, re-measured
- The trajectory experiment re-run against the constrained detector, publishing distinct trajectories per pair; the per-family noise floor re-measured at the same K as the baseline and published beside it; every published K-run number labelled as repeated measurements or as an ensemble of distinct procedures.
- Observable: the artifact reports the distinct-trajectory count per pair for both detectors on the same twelve pairs, and the floors side by side.
- _Requirements: 5.1, 5.2, 5.4_
- _Depends: 5.1_
- [ ] 5.3 What the floor retires, named
- If a family's re-measured floor is zero, the artifact states that its K-run requirement and its regression budget are unnecessary and names the `learning-harness` requirements that would be amended — 3.5, 8.2 and 9.5 — without amending them here.
- Observable: the artifact carries the per-family verdict and the named requirements, and the spec's Implementation Notes record the escalation to the maintainer.
- _Requirements: 5.3_
- _Depends: 5.2_
- [ ] 5.4 Gates, matrices and the boundary
- Detection, map and local pair gate outputs byte-identical to the committed baseline; both test matrices green; the core install still resolving with no dependencies; every prompt and persisted trajectory redacted.
- Observable: the committed gate-baseline test passes in both matrices, and a planted key in a judgment prompt does not survive into the artifact.
- _Requirements: 7.3, 7.4, 7.5_
- _Depends: 5.1_
## Task-plan review notes
- Phase 1 is deliberately able to end the feature. Tasks 1.1 to 1.3 call no model, so the condition in 1.3 — a deciding slice whose ceiling is below its recorded detection rate — is reachable before any budget is spent. The ruleset variant of this design would have failed that condition at 12.1% against 64.9%.
- Task 2.1 is the only task that touches `scoring.py`, and its observable is that nothing changes. The recorded round-zero artifacts are the pin; if they cannot be reproduced exactly, the refactor is wrong and not the numbers.
- `(P)` marks the two tasks with no dependency on a model call: 2.1 and 4.1 can be built and reviewed while phase 1 runs.
- Task 3.3 is the one that would have caught the defect this whole spec exists for. It belongs in the suite, run repeatedly, not in a one-off measurement.
## Implementation Notes
- Task 1.1 (2026-09-06, RED first: 13 failing tests in `tests/test_candidate_enumeration.py`; then green). `learning/candidates.py`. Two things the fixtures taught: the IR emits a chained call twice at one line, once bare and once carrying the keyword arguments, so deduplication keeps the richest — a bare `filter_by` and `filter_by(user_id=owner)` are the same site and only one of them says what the constraint was. And `leaky` and `constrained` in the shared obligations fixture **both** discharge through `filter_by`; what separates them is that one binding chain ends at `request.args.get('owner')` and the other at `request.user.id`. The test asserts that contrast, which is the judgement the model is being narrowed to.
- Task 1.2 (2026-09-06, RED first: 3 failing tests; then green). `candidates.ceiling(cases, taxonomy, generator="ir"|"ruleset")` and `ousast learning candidates`. The `ruleset` arm exists so the comparison that rejected it stays a measurement rather than a remembered number. The command calls no model and its artifact is byte-identical on a second run.
- Task 1.3 (2026-09-06). **Verdict: go.** Measured offline across every slice, `benchmarks/measurements/2026-09-06-candidate-ceiling-all-slices.json`:
The margin is largest on agent-vfc, which is the slice that matters most and the one the tool is actually for. Per family: config_secrets, deserialization, output_encoding and untrusted_destination 100%; access_control 90%; path 87.5%; memory 86.4%; injection 84.6%; prototype 71.4%; `unknown` 0% of two rows, which is what abstention looks like.
- **A shape correction the measurement forced, before the detector existed.**`config_secrets` first read 63.6% with a bind-and-literal shape. A permissive default is as often an argument to a call — `cors({origin: '*'})`, `app.use(session({secure: false}))` — as it is a binding; adding call sites to that family's shape took it to 100% and recovered `new-erp-final-app-6e9644`, `zoonk-origin-bfbf67`, `melodix-playerprovider-089931` and `registry-proxy-worker-426847`.
- **Carry-forward, and it is not this feature's to fix.** vfc-js sits at 52.9% because eight of seventeen rows label a function the JavaScript CST names `<anon>`: `module.exports.publish = function () {}` and its relatives. The gap is in function *naming*, not in candidate generation, and it caps the existing hunter on that slice exactly as much. Same class as the `_declarator_start` defects found during learning-harness task 4.1.
- vfc's 21 `unsupported_language` rows are C and C++ files the IR declines; they are the non-gating memory slice and are already excluded from every web number.
| Evaluation and eligibility | Frozen identities and populations; unchanged 30s/2s and precision/recall/completion gates; no capability enabled by a microbenchmark |
Production changes require task-local review and fresh ordinary scan gates, runtime smoke,
and downstream acceptance evidence. This research record is not that verification.
> **Status 2026-10-02:** paused since 2026-09-23 (its last change). 27 of 51 tasks done: the MAP integration was delivered and `v1.2.0-alpha.1` rolled out NO-GO (`census-repair-verification.md`); the remaining tasks wait on `pre-push-safety-net` M2/M3 and nothing here moved after the ax plane landed. Not active.
## Introduction
`model-grounded-detection` built an arbiter and measured it: 41.3% of the corpus reaches `model_entailed`
@@ -10,7 +12,7 @@ hunter's 76.7% / 12.3%, at a fifth of the model calls. All 16 of its tasks are c
from a measurement harness, not from the tool. This feature closes that gap and answers the question the
maintainer actually posed: *a real world tool that helps developers when they want to contribute.*
That framing changes the work. A contributor has no labelled function and no declared weakness class — they
That framing changes the work. A contributor has no labelled function and no declared weakness class; they
have a repository they are about to touch. Everything measured so far is function-level pairs, which
exercise none of the questions a whole repository asks: which regions to look at, how to rank thousands of
candidates, what to spend, and whether a CPG build on a 100k-line tree finishes at all.
@@ -20,13 +22,13 @@ Context: `brief.md`, `.kiro/specs/model-grounded-detection/`, and the committed
## Boundary Context
- **In scope**: wiring `model/pipeline` into `ousast scan`; region selection from entry points; ranking and
- In scope: wiring `model/pipeline` into `ousast scan`; region selection from entry points; ranking and
budget at repository scale; the performance envelope; running against known-vulnerable checkouts;
contributor-facing output; the packaging already built (image, compose, shell wrappers, pyinfra); the
first-party tests. The maintainer approved its dependency adaptation and acceptance checks;
task 2.18 now implements it before downstream partition completion.
## Group 7 — Grow the ontology by measurement, not by subject (2026-09-21)
Named by the family detection census and the week of repairs after it. Five of seven families
established nothing and every one failed for an ontology reason, so this group is where detection
efficiency actually lives. The design amendment above states the constraint these tasks work under:
the ontology describes libraries, frameworks and language constructs, never repositories.
- [ ] 7.1 Report what a repository calls that the ontology does not model
- Census the call names, and the receiver-qualified spellings, that appear in a repository's first-party graph; subtract everything any fact table names; rank the remainder by how often it is called and by how close it sits to a modelled source. The output is a measured worklist of what this tool is blind to here.
- It needs no dataflow, which is why it is cheap: the same structural pass that answers 17,700 requests in 44 s already walks every call.
- **It reports candidates, not entries.** A candidate becomes ontology only when someone resolves it to a library API, states the weakness class and records provenance. Nothing may be added because a subject called it.
- Three defects this would have named in one run rather than three investigations: `needle.get` absent while the family found the redirect two files away, `find` absent so the documented object-reference defect raised no obligation, and `maybe_unserialize` absent so the only deserialization site in 1,164 PHP files was invisible.
- Done when the instrument reports a ranked gap list for each measured subject, every entry carries how often it is called, and adding nothing to the ontology changes no result.
- _First run 2026-09-21 on NodeGoat:_ 18 of 175 distinct call names modelled, covering 12% of 1,185 calls. The top product candidate is `res.render`, 23 calls with five in a method reading untrusted input — the same gap task 14.3 reasoned its way to by hand, found here in one run, which is the validation that matters. It also surfaced `Math.random` in the user data-access module: weak randomness is a weakness class this ontology has no family for, so no census could have found it, because the family list is itself the blind spot.
- _It had to be taught what is not an API._ The first run ranked `_tmp_3.push` and `__ecma.Array.factory` first, which is how the frontend lowers a destructuring and an array literal, and a Cypress suite and a Gruntfile held four of the next five. Both are now separated, because noise at the top of a worklist is how a worklist stops being read. Evidence: `benchmarks/measurements/2026-09-21-ontology-coverage-nodegoat.json`.
- [ ] 7.2 Key sanitizers to the weakness they actually prevent
- The sanitizer list is flat across families, and that forces a false choice which is now measured: adding WordPress's `esc_attr` removes eighteen output-encoding false positives on one plugin and simultaneously hides a real SQL injection, because no HTML escaper stops SQL injection. `php.toml` has carried a comment predicting exactly this since before the measurement existed.
- Key each sanitizer to the weakness class it discharges, so a family consults the cleansers that apply to it. Do not solve it by choosing which defect to keep.
- Done when an HTML escaper clears an output-encoding flow and leaves an injection flow standing, on the measured plugin, with both outcomes recorded.
- _Boundary: shipped semantic facts_
- _Requirements: 6.2, 8.1_
- [ ] 7.3 Do not arbitrate a question whose evidence says there is nothing to find
- The structural pass answers 17,700 requests in 44 s; arbitration costs one to two seconds per question, and a 673-file plugin asks 2,526 of them. Gate arbitration on the evidence vector already computed: a question whose scope holds no modelled sink, or no reachable source, cannot produce a flow and does not need the expensive pass.
- This attacks the scale limit from the opposite side to graph reuse and can be sized today from data already on disk, by counting how many of those 2,526 questions had any sink in scope at all.
- Done when the arbitrated count falls without any finding being lost on the measured subjects, and the questions skipped are recorded as skipped rather than as answered.
- _Implemented and measured 2026-09-22, and the mechanism already existed._ Tier-zero pruning defers a question whose evidence vector says no sink of its family is in reach. It was switched off for the WHOLE SCAN whenever any of twelve degradation reasons appeared, and a multi-language repository always carries two of them, so the measured plugin arbitrated every question it could ask. Third instance this week of the same blast radius. A prune blocker that names a partition now withholds pruning from that partition's questions and nobody else's, and `cross_partition_semantics_unresolved` leaves the blocker set entirely for the reason it left the integrity gaps.
- _The arbitrated count falls and no finding is lost._ At 25 regions: 127 questions to 34, with 93 recorded deferred as `tier_zero`, and 19 findings before and after, identical family by family. At 500: 2,526 to 406, with 2,120 deferred. A family whose every question was pruned now reports "no sink of this family in reach" rather than "region offered, no question asked", because nothing to ask is a different statement from never offered.
- **It is not sufficient, and that is the useful half of the result.** The 500-region run still fails in the same way at the same cost, because 406 questions at roughly one and a half seconds each still exceed the 300-second per-batch ceiling. This lever moved the volume; the ceiling is a separate limit and pre-push 13.3 still cannot reach the population that holds its target without it. Evidence: `benchmarks/measurements/2026-09-22-tier-zero-pruning.json`.
- _Boundary: contributor-scan scan budget_
- _Requirements: 4.1, 6.3, 8.1_
- [ ] 7.4 One matcher, used by every query
- Four query files carry near-copies of the same token matcher and each acquired the same defect separately: the configuration matcher needed a receiver check the dominance matcher already had, and the sink matcher searched a whole node's text including its comments. Six instances of one bug class are listed in the pre-push design's invariants.
- Extract one matcher with callee confinement and word boundaries built in, and have taint, dominance and config use it. A rule in a document gets rediscovered; a rule in a constructor does not.
- Done when one implementation is shared, each caller's behaviour is unchanged on the measured subjects, and a new query cannot match text that merely contains a name.
- _Boundary: contributor-scan queries_
- _Requirements: 6.2, 8.1_
- [ ] 7.5 A degradation must declare the scope it reaches
- Five instances of one bug class are listed in the pre-push design's invariants, two of them found this week, and two of those cost the declared target its entire admissibility. A degradation is currently a bare dict, so nothing forces a new reason to say whether it reaches one question, one file, one partition or the whole read.
- Make it a record with a required scope, and let the demotion and the admission policy read that field rather than a hand-maintained membership set.
- Done when a degradation cannot be constructed without a scope, the existing reasons carry theirs, and the measured subjects' outcomes are unchanged.
- [ ] 7.6 Golden findings per subject, so a silent change cannot pass
- Nothing fails today when a change alters what NodeGoat or VAmPI report. Four changes in one week altered detection and each time the census had to be re-run from memory; one false positive was caught by reading a witness rather than by a test.
- Commit an expectation per subject at the finding level, site and family and rung, and diff against it. Keep it honest about what it is: a regression guard on development subjects, never evidence of qualification.
- Done when a deliberate change to a matcher or a fact table fails the diff, and the expectation records the ruleset and engine identities it was taken under.
- _Boundary: evaluation harness_
- _Requirements: 8.1, 8.2_
- [ ] 7.7 Budget a batch by the sinks it implies, not by the requests it holds
- **Named by the 2026-09-22 identification.** A taint batch costs what its matched sink NODES cost, because `reachableByFlows` is asked per sink. Measured at a fixed 100 requests on one graph: SQL injection 147 s, path 194 s, output encoding 704 s — and output encoding declares seven sink names against path's sixteen. What separates them is that `echo` and `print` match thousands of calls in a WordPress plugin.
- The driver currently splits a failed batch into eight parts by request count, which cannot help: each part carries the same everywhere-sinks.
- The count needed is already computed. `taint.sc` memoises its candidate sink list per sink set, and the evidence pass reports the in-scope sink list per request before any dataflow runs. A driver that reads it can size a batch by expected work, and can report a family whose sinks are everywhere as EXPENSIVE rather than as failed.
- Done when a batch is sized by the sink nodes it implies, the measured plugin's arbitration completes within its ceiling, and a family too expensive to arbitrate is reported as such rather than as a timeout.
- _Implemented and measured 2026-09-22._ On the 500-region population where every earlier run produced nothing: **153 of 406 questions completed and 125 findings, inside the deadline**, against 0 and 0. 380 requests carrying 5,807 sink visits split into 11 batches, each answering in 80 to 197 s, none near the 300 s ceiling. Every family that can speak on this subject now does: injection 49 completed and 24 findings, path 64 and 15, output encoding 32 and 70, untrusted destination 8 and 16. Eight batches were left unasked to keep time for arbitration, and that is recorded rather than silent.
- _Four defects, each hidden by the one before it, and three of them the same rule._ Batches sized by request count when cost is per matched sink. Packing heaviest-first, which starved injection — the one family with a true positive here — of all 114 of its questions. Any per-query timeout cancelling the whole scan, abandoning batches with 33 minutes of budget left. And no budget kept for arbitration, which removing that cancellation exposed: rows are not findings, so the query phase ran the full deadline and completed nothing, where the cascade had been an accidental guard.
- _The reserve had to be checked per BATCH._ Checked between query kinds it changed nothing at all, because one kind is a single loop iteration that runs every one of its batches inside it. The granularity of a budget check has to match the granularity of the work it guards. Evidence: `benchmarks/measurements/2026-09-22-sink-weighted-batching.json`.
- _What it does not claim:_ 253 of 406 questions still went unanswered and one batch still records a timeout; the raw finding count is dominated by output encoding, the family already measured to be imprecise on PHP; and the reserve share of a fifth is a first value, not a tuned one.
- [ ] 7.8 Portion the work instead of capping it: a kill keeps what finished
- **Asked for by the maintainer on 2026-09-22, after 7.7.** A hard per-invocation ceiling is all-or-nothing: a batch killed at 301 s loses every answer it had already computed, and the retry then pays a fresh JVM start per part to ask them all again. Measured on the plugin: 116 requests killed at 300 s, eight restarts, and the same everywhere-sinks in every part.
- Three coordinated changes replace the cap as the thing that governs cost. The taint query STREAMS: the census first, then one line per request as it is answered, flushed as it goes, inside the unchanged fence. The driver harvests a killed process's stdout and assembles every complete line, marking the result partial; the retry then asks only the remainder. And portion sizing ADAPTS: after each portion the observed seconds per sink visit, less the fixed start, resizes the rest toward a target duration, and a portion that did not answer halves the budget rather than being fitted. The ceiling stays as a safety net that the sizing is meant to make rare.
- Legacy single-document payloads must keep parsing unchanged; the other queries do not stream and need not. A census line is one whose only key is the census — a first cut misread a legacy document that carried the census beside its answers as a census line and dropped every answer it sat beside, which a control caught.
- Done when a killed taint batch hands back the answers it finished and the retry asks only the rest; portion sizes follow observed cost; the streamed and legacy forms both parse; and the plugin population completes more questions per unit of budget than under 7.7 with no finding lost.
- _Status 2026-09-22:_ implemented; host controls pass for the assembler in both forms and under a kill, for the remainder-only retry, and for the sizer; the joern-gated check passes on the real engine — the taint query streams one line per request, and the comment-sink and literal-destination controls hold under the streamed form. The plugin measurement is running on a frozen export of the committed source.
- _Two things the measurement exposed on the way, both fixed:_ the evidence pass was one unportioned batch protected only by the ceiling — 22,125 requests on a plugin push transaction — and now runs in portions of 4,000; and a first cut of the harvest read a timed-out process's pipe BEFORE killing it, which blocks forever because a pipe whose writer is alive never reaches end-of-file. The suite's own hanging-engine control caught that after twenty minutes of what I mistook for CPU starvation.
- _A stated limit:_ the reuse layer's re-query of `missing` ids calls the backend batch directly and so bypasses sink-weighted portioning. Those re-queries are the small remainder of a cached scan, and streaming still keeps what a killed one finished, so it is recorded here rather than fixed. _Corrected 2026-09-23:_ the limit was stated backwards and was worse. The reuse layer wrapped only `run_batch`, and the portioner asks `run_batch_once`, so every portioned query bypassed the reuse cache — and its source-root normalization — entirely. Fixed under 7.9: both paths go through the cache.
- _End-to-end tuning, 2026-09-23, the honest version._ The first end-to-end run REGRESSED completion (64 of 406 against 7.7's 153) while finding as many defects, and each fix exposed the next: the adaptive sizer read one portion at 953 s because it timed `run_batch`, whose own split-retry folded in, and collapsed; giving the portioner a single-attempt path (`run_batch_once`) fixed the timing but only after it was wired on the PARTITIONED graph too, which every plugin uses and which the first wiring missed; and even then completion sat at 56 of 406 because the portioner iterated a fixed list and dropped the remainders of killed portions, stopping at 1,263 s of a 3,600 s budget. The queue re-asks those remainders. The sizer now adapts both ways in the trace — 268 up to 535 after a cheap portion, halved to 267 to 133 on kills — and re-asks what a kill left. Each step is unit-tested: the assembler in both forms and under a kill, the single-attempt path, coverage-halving, and queue termination. _Final end-to-end 2026-09-23, mixed and told straight._ Against the hard-cap-plus-reserve baseline on the same 500-region PHP run: findings IDENTICAL at 125 either way, completion slightly LOWER under portioning (131 of 406 against 153), wall time higher (2,982 s against 2,338). Portioning does not beat the cap on throughput here; it matches it on what it finds. What it buys is kill-safety -- the maintainer's actual ask -- and reaching the same 125 findings is the evidence that streaming loses nothing the cap kept while no longer discarding a near-complete batch when the ceiling falls mid-answer. Completion did not rise because the requeue re-asks a killed portion's remainder, and on this subject those are the expensive output-encoding questions that keep hitting the ceiling in smaller pieces; telling a transient kill from a genuinely-too-expensive family is the next lever, not more portioning. Neither approach is release-ready regardless: 32-38% completion against a 95% gate, on development data that qualifies nothing. Evidence: `benchmarks/measurements/2026-09-23-portioning-vs-hard-cap.json`.
- [x] 7.9 Size portions from what each answer cost, and tell a transient kill from an expensive question
- **The next lever named by 7.8, and the trace said it was a different lever.** Re-read on 2026-09-23, the 7.8 run's sizer did not lose completion to expensive families so much as to a death spiral: after two kills the budget fell to 3 sink visits per portion and never recovered. Portions of 3–6 visits finished in 25–30 s, which under the assumed 70 s fixed start read as "no information", so the budget could shrink and never grow — and the rest of the run paid a JVM start per handful of requests.
- The instrument first: the taint query streams each answer's own cost (`ms`), carried as a reserved `__timing__` list through shards, partitions and reuse. The sizer then MEASURES what it used to assume — the fixed start as the wall time the answers do not explain (median over whole portions), and a seconds-per-sink-visit rate per family from that family's own answers. Portions are cut in predicted seconds and grow request by request from the head of the queue.
- A kill names its culprit. Answers stream in request order, so the first unanswered id was running at the kill and every id after it never started: those go straight back to the queue, the culprit is asked alone after everything else, and its family is costed at the lower bound it proved. A request that fails alone after more than a portion's target is `query_too_expensive`, reported apart from `query_failed`.
- Two defects fixed on the way: a scan small enough for one portion took a single attempt and lost whatever a kill cut off (it now goes through the queue); and requests the arbitration reserve held back were reported as `query_failed` although they were never asked (now `query_budget_reserved`).
- _Measured 2026-09-23, same 500-region PHP run, 3,600 s, frozen at `1b28465`:_**281 of 406 questions completed, against 153 (hard cap) and 131 (7.8)**, in 2,987 s. Findings per family never fell: injection 24→33, output encoding 70→125, path 15→35, untrusted destination 16→31; 224 in total against 125. The declared target `classes/class.memberorder.php:936:getMemberOrderByCode` is found at `model_entailed`. The measured fixed start was 22–24 s throughout. **No portion was killed in the whole run** — every one answered whole — so the gain is the sizing, and the culprit path is covered by host controls but was NOT exercised end to end here.
- _Limits, stated:_ one subject, one budget. The census keeps ten examples per family, so it shows per-family counts did not fall, not that the earlier findings are a strict subset. The extra findings are unadjudicated: output encoding on PHP `echo` is the family most likely to carry false positives, and 224 is a count, not a precision. 99 taint questions were still held for arbitration, and access control, config and prototype were never reached (0 of 39) — the ranker puts them last, and the reserve cuts the tail. Completion is 69%, still short of the 95% M4 gate; this is development data and qualifies nothing. Evidence: `benchmarks/measurements/2026-09-23-measured-portioning.json`.
@@ -18,8 +18,8 @@ Derive a mechanism record from a pair by parsing the vulnerable excerpt with the
## Scope
- **In**: mechanism export from pairs; shape normalization; variant search as a proposer in MAP; leave-one-out measurement command and payload; `mechanisms` improve lever with validator rules; store hygiene (tier, provenance, dedupe, version); offline tests.
- **Out**: LLM-authored shapes; changing dispositions or evidence rungs; embedding retrieval changes (existing prove-order reuse stays); inter-file flow (reachability spec); any merge gate; vendoring more pairs (corpus spec).
- In: mechanism export from pairs; shape normalization; variant search as a proposer in MAP; leave-one-out measurement command and payload; `mechanisms` improve lever with validator rules; store hygiene (tier, provenance, dedupe, version); offline tests.
- Out: LLM-authored shapes; changing dispositions or evidence rungs; embedding retrieval changes (existing prove-order reuse stays); inter-file flow (reachability spec); any merge gate; vendoring more pairs (corpus spec).
## Boundary Candidates
@@ -35,13 +35,13 @@ Derive a mechanism record from a pair by parsing the vulnerable excerpt with the
## Upstream / Downstream
- **Upstream**: `PairCase`/`load_pair_catalog` with `review_tier`; `FileIR`; `MechanismStore`; `adjudicate`; `run_improvement` gate with per-profile holdout.
- **Downstream**: `reachability-flow-model` attaches paths to variant findings; `guard-dominance-regime` consumes the guard shapes as sanitizer candidates; reports show mechanism ids on findings.
- Upstream: `PairCase`/`load_pair_catalog` with `review_tier`; `FileIR`; `MechanismStore`; `adjudicate`; `run_improvement` gate with per-profile holdout.
- Downstream: `reachability-flow-model` attaches paths to variant findings; `guard-dominance-regime` consumes the guard shapes as sanitizer candidates; reports show mechanism ids on findings.
## Existing Spec Touchpoints
- **Extends**: `propose-adjudicate-prove` mechanism memory (additive fields, second write path), `three-stage-scan` MAP stage (one more proposer), `harnessx-self-improving-rulesets` improve loop (one more bounded lever).
- **Depends**: `pair-corpus-honesty` Req 9 (tiers) for which pairs may seed.
- Extends: `propose-adjudicate-prove` mechanism memory (additive fields, second write path), `three-stage-scan` MAP stage (one more proposer), `harnessx-self-improving-rulesets` improve loop (one more bounded lever).
- Depends: `pair-corpus-honesty` Req 9 (tiers) for which pairs may seed.
- `regress/candidate.py` — exposes `hotspot_from_finding`; variant findings enter the ranked candidate slice as zero-score hotspots from `cli.py` (after every scored promotion, inside the cap, through the safety check).
- `regress/candidate.py`: exposes `hotspot_from_finding`; variant findings enter the ranked candidate slice as zero-score hotspots from `cli.py` (after every scored promotion, inside the cap, through the safety check).
> **Status 2026-10-02:** complete, 17 of 17 tasks (spec.json phase corrected to `implementation-complete`). Its `[variants]` config keys were retired on 2026-10-02: the variant search they configured (`semantic/variants.py`) had already been deleted.
## Introduction
OpenUltraSAST's labeled pair corpus measures the detector but does not improve it, and its mechanism memory is empty because only sandbox-proven findings may write to it. This spec turns every trusted pair into a mechanism record, adds a structural variant search that proposes matches on any scanned tree, measures the corpus's own detection rate leave-one-out, and gives the improve loop a lever that can raise recall under the existing false-positive ceiling and per-profile holdout gate.
## Boundary Context
- **In scope**: mechanism export from pairs; shape normalization per language; variant search proposer in MAP; leave-one-out evaluation; `mechanisms` improve lever; store schema additions; reports showing mechanism ids; offline tests.
- **Out of scope**: taint, facts, walker, dispositions, evidence rungs; embedding retrieval changes; inter-file flow; LLM-authored shapes; catalogs and scorer changes beyond reading `PairCase`; any merge gate.
- **Adjacent expectations**: `Mechanism` gains additive fields only; `order_promotions` keeps working on old records; the improve loop's existing clauses and the per-profile holdout clause apply unchanged; `local` remains the only slice that can fail CI.
- In scope: mechanism export from pairs; shape normalization per language; variant search proposer in MAP; leave-one-out evaluation; `mechanisms` improve lever; store schema additions; reports showing mechanism ids; offline tests.
- Out of scope: taint, facts, walker, dispositions, evidence rungs; embedding retrieval changes; inter-file flow; LLM-authored shapes; catalogs and scorer changes beyond reading `PairCase`; any merge gate.
- Adjacent expectations: `Mechanism` gains additive fields only; `order_promotions` keeps working on old records; the improve loop's existing clauses and the per-profile holdout clause apply unchanged; `local` remains the only slice that can fail CI.
## Requirements
### Requirement 1: Mechanisms seeded from trusted pairs
**Objective:** As a maintainer, I want each reviewed pair to become a mechanism record, so that the corpus is a library the detector can search, not only a ruler.
**Objective:** As a maintainer, I want each reviewed pair to become a mechanism record, so that the corpus is both a library the detector can search and a ruler.
#### Acceptance Criteria
@@ -62,7 +64,7 @@ OpenUltraSAST's labeled pair corpus measures the detector but does not improve i
### Requirement 5: A recall-raising lever in the improve loop
**Objective:** As a maintainer, I want the improve loop to be able to admit mechanisms, so that it can raise recall and not only shadow noisy rules.
**Objective:** As a maintainer, I want the improve loop to be able to admit mechanisms, so that it can raise recall as well as shadow noisy rules.
- **Discovery Scope**: Extension of mechanism memory, MAP stage, pair eval, and the improve loop
- **Key Findings**:
- Feature: `corpus-seeded-mechanisms`
- Discovery Scope: Extension of mechanism memory, MAP stage, pair eval, and the improve loop
- Key Findings:
- `semantic/mechanisms.py` already has `Mechanism`, `MechanismStore` (append-only JSONL), `append_mechanism`, embedding cache, hard filters by language/CWE/tags, and `order_promotions`. It is written only by `cli._append_proven_mechanisms` after a sandbox trigger, so it is empty on every real tree.
- Clearwing's recall lever is the variant loop: a proven mechanism becomes a structural search over the messy tree. OpenUltraSAST copied the store and the retrieval, not the loop.
- `FileIR` call sites already carry callee name, argument texts, argument identifier lists, and constant flags; binds carry source text and names. That is enough for a shape without new parsing.
@@ -16,30 +16,30 @@
### Mechanism store today
- **Context**: Can the existing store carry corpus-derived records?
- **Findings**: `Mechanism` fields are summary, cwe, language, tags, keywords, what_made_it_exploitable, source_finding_id, source_repo. Identity is the pattern. JSONL is authoritative; vectors are a rebuildable cache keyed by id and model.
- **Implications**: add `origin` (`sandbox` | `corpus`), `review_tier`, `pairs`, `shape` (structured), `guard` as additive fields with defaults so old rows load. A second writer (`append_from_pair`) beside the sandbox writer.
- Context: Can the existing store carry corpus-derived records?
- Findings: `Mechanism` fields are summary, cwe, language, tags, keywords, what_made_it_exploitable, source_finding_id, source_repo. Identity is the pattern. JSONL is authoritative; vectors are a rebuildable cache keyed by id and model.
- Implications: add `origin` (`sandbox` | `corpus`), `review_tier`, `pairs`, `shape` (structured), `guard` as additive fields with defaults so old rows load. A second writer (`append_from_pair`) beside the sandbox writer.
- **Findings**: for the labeled function, the sink call site is the call whose trailing name equals the label's sink or whose line equals the label's line; source positions are argument indexes whose identifiers trace by binds to a fact source, a parameter, or a container read. Guard kind comes from statements on the fixed side absent on the vulnerable side: `if` with `== NULL`/`is None`/`!x` → `null_test`; comparison with `<`, `>`, `len(` → `bounds_test`; membership in a literal collection or `startswith`/regex → `allowlist_test`; identity read from an auth context (`request.state`, `req.user`, `session`) → `auth_check`; parameterized second argument → `parameterized_call`; changed declared type → `type_change`.
- **Implications**: `semantic/variants.py` implements `derive_shape(pair_ir_vuln, pair_ir_fixed, label)` and `match_shape(file_ir, shape)`. Guard classification is heuristic and closed; unknown → `none`.
- Findings: for the labeled function, the sink call site is the call whose trailing name equals the label's sink or whose line equals the label's line; source positions are argument indexes whose identifiers trace by binds to a fact source, a parameter, or a container read. Guard kind comes from statements on the fixed side absent on the vulnerable side: `if` with `== NULL`/`is None`/`!x` → `null_test`; comparison with `<`, `>`, `len(` → `bounds_test`; membership in a literal collection or `startswith`/regex → `allowlist_test`; identity read from an auth context (`request.state`, `req.user`, `session`) → `auth_check`; parameterized second argument → `parameterized_call`; changed declared type → `type_change`.
- Implications: `semantic/variants.py` implements `derive_shape(pair_ir_vuln, pair_ir_fixed, label)` and `match_shape(file_ir, shape)`. Guard classification is heuristic and closed; unknown → `none`.
### Where variant search sits
- **Context**: Propose, adjudicate, prove roles must stay separate.
- **Findings**: MAP already runs overlay and the tool hunter; variant search is a third proposer that reads `FileIR` and the store. Findings at `suspicion`, origin `variant`; when the overlay has a flow at that call site, merge (attach mechanism id to the overlay record). Candidate eligibility follows the hunter-eligibility path being added in `reachability-flow-model` or the sev-5 forced-hotspot path today.
- **Implications**: no disposition change; one new finding origin; manifest counters.
- Context: Propose, adjudicate, prove roles must stay separate.
- Findings: MAP already runs overlay and the tool hunter; variant search is a third proposer that reads `FileIR` and the store. Findings at `suspicion`, origin `variant`; when the overlay has a flow at that call site, merge (attach mechanism id to the overlay record). Candidate eligibility follows the hunter-eligibility path being added in `reachability-flow-model` or the sev-5 forced-hotspot path today.
- Implications: no disposition change; one new finding origin; manifest counters.
### Leave-one-out
- **Context**: Prove that corpus growth raises recall, honestly.
- **Findings**: pair eval materializes each side into a temp tree already; a leave-one-out runner seeds a temp store from the other trusted pairs and runs variant search on both sides, scoring with the pair scorer's detection and leak rules.
- **Implications**: `pairs.py` gains a read-only consumer path (`evaluate_loo`) or a sibling module `semantic/loo.py` to keep the scorer's boundary; report per slice, profile, mechanism; artifact only.
- Context: Prove that corpus growth raises recall, honestly.
- Findings: pair eval materializes each side into a temp tree already; a leave-one-out runner seeds a temp store from the other trusted pairs and runs variant search on both sides, scoring with the pair scorer's detection and leak rules.
- Implications: `pairs.py` gains a read-only consumer path (`evaluate_loo`) or a sibling module `semantic/loo.py` to keep the scorer's boundary; report per slice, profile, mechanism; artifact only.
## Architecture Pattern Evaluation
@@ -54,27 +54,27 @@
### Decision: Seed only from seeded and reviewed tiers
- **Context**: `title` pairs are unreviewed guesses; `advisory` pairs have correct fixes but unlabeled mechanisms.
- **Selected Approach**: export reads `review_tier`; `advisory`/`title` may be scored leave-one-out as held-out targets but never seed.
- **Trade-offs**: fewer seeds at first; honest about what the detector was taught.
- Context: `title` pairs are unreviewed guesses; `advisory` pairs have correct fixes but unlabeled mechanisms.
- Selected Approach: export reads `review_tier`; `advisory`/`title` may be scored leave-one-out as held-out targets but never seed.
- Trade-offs: fewer seeds at first; honest about what the detector was taught.
### Decision: Variant is a proposer at suspicion
- **Context**: Evidence honesty.
- **Selected Approach**: origin `variant`, `suspicion`; merge with overlay flow when present; sandbox may raise.
- **Trade-offs**: variant-only hits do not fail CI; correct.
- Context: Evidence honesty.
- Selected Approach: origin `variant`, `suspicion`; merge with overlay flow when present; sandbox may raise.
- Trade-offs: variant-only hits do not fail CI; correct.
### Decision: Lever admits records, not shapes
- **Context**: Keep the validator closed.
- **Selected Approach**: `mechanisms` lever edits reference record ids produced by the exporter; free-form shapes are rejected.
- Context: Keep the validator closed.
- Selected Approach: `mechanisms` lever edits reference record ids produced by the exporter; free-form shapes are rejected.
## Risks & Mitigations
- Coarse shapes over-match (`exec(x)` everywhere) — source-position requirement plus fact/parameter/container check; report leak rate on fixed sides leave-one-out; the per-profile gate rejects admissions that leak.
- Guard heuristics misclassify — closed set with `none`; reported, not used for demotion.
- Store growth — dedupe by shape; one record per shape with pair provenance list.
- Leave-one-out cost — one temp store per pair over tens of pairs; fine; cache parsed IR per excerpt.
- Coarse shapes over-match (`exec(x)` everywhere): source-position requirement plus fact/parameter/container check; report leak rate on fixed sides leave-one-out; the per-profile gate rejects admissions that leak.
- Guard heuristics misclassify: closed set with `none`; reported, not used for demotion.
- Store growth: dedupe by shape; one record per shape with pair provenance list.
- Leave-one-out cost: one temp store per pair over tens of pairs; fine; cache parsed IR per excerpt.
content, examples, thresholds, or promotion, the detection core shall mark the affected
population spent and shall refuse it for admission.
### Requirement 14: Consumer and re-parenting contract
The detection core shall be the single contract that existing and future detection specs
reuse.
1. The detection core shall define the interface through which the pre-push transaction
consumes evidence records and verdicts within its deadline.
2. The detection core shall define the interface through which proof-based blocking
evidence attaches to an existing claim.
3. When an existing arm's behavior is described, the detection core shall require that
description to be expressed over the family-neutral contract rather than as a separate
runner contract.
4. When a consumer receives bounded silence caused by missing semantic knowledge, the
detection core shall require the consumer to preserve the named coverage reasons.
5. When a family specification is created, the detection core shall require it to identify
its family contract, readiness fixtures, development population, and admission boundary.
### Requirement 15: Fixed measurement schema
The detection core shall require one fixed measurement schema for every detection number.
1. When any detection result is reported, the detection core shall require population and
freeze digests, claimed family and ecosystem scope, denominator policy, family decision
vocabulary, neutral outcomes, evidence bases, every rate with its sample count and
uncertainty interval, coverage, instrument failures, model-call failures, and runtime.
2. When semantic relations participate in an arm, the detection core shall require the
report to include sites considered, sites recognized, transitions completed, security
objects constructed, constraints evaluated, determinate outcomes, and unresolved counts
by named reason.
3. When a measurement changes population, denominator policy, claimed scope, family
contract, fact schema, model program, or outcome mapping, the detection core shall treat
the result as a new baseline and shall not present it as a trend against the previous
baseline.
4. When a zero or clean-looking result is reported, the detection core shall require proof
that every tracked input was read and that every prerequisite semantic stage executed.
5. When a later-stage metric is reported, the detection core shall include the results and
digests of every prerequisite readiness stage.
## Evaluation-Admission V3 Amendment
Requirements 16–20 extend Requirements 12 and 15 for a new evaluation-admission contract. Requirements 1–15 and completed version-two evidence retain their existing meaning, identities, and validation behavior; the amendment does not reinterpret or upgrade them.
The detection core shall provide a new evaluation-admission contract for exact-scope, multi-arm qualification without changing the meaning or validation of historical admission artifacts.
#### Acceptance Criteria
1. When a new evaluation admission is requested, the detection core shall validate it under a distinct contract version and shall keep every earlier admission contract independently replayable.
2. The detection core shall preserve the accepted bytes, digests, decisions, and replay behavior of existing version-two admission artifacts.
3. If pooled version-two evidence is presented as exact-scope version-three evidence, the detection core shall refuse the conversion and shall require a new baseline evaluated under the new contract.
4. If an admission artifact names an unknown, mixed, or incompatible contract version, the detection core shall refuse it rather than infer a compatible shape.
5. When a population, evaluation plan, arm identity, claimed scope, denominator policy, threshold set, judgment vocabulary, analyzer identity, fact identity, configuration identity, registered budget, execution identity, outcome mapping, or combination rule changes, the detection core shall require a new baseline identity.
6. The detection core shall prevent an insufficient result, impaired result, aggregate result, incomplete arm result, raw report, or other partial artifact from authorizing consumer use.
### Requirement 17: Reviewed multi-arm plan and atomic spend authority
The detection core shall bind every arm evaluated against one untouched population to a reviewed label-blind plan and one atomic admission spend.
#### Acceptance Criteria
1. When an evaluation plan is registered before label access, the detection core shall require a reviewed label-blind plan naming the population, input-set identity, every component arm, every combined arm, and every expected observation slot.
2. When an arm enters an evaluation plan, the detection core shall require its immutable baseline identity, exact claimed family-and-ecosystem scopes, denominator policy, thresholds, sufficiency rules, outcome mapping, and combination lineage where applicable.
3. When the label-blind plan is reviewed, the detection core shall evaluate threshold reachability and evidence sufficiency independently for every planned arm and exact claimed scope.
4. If any planned arm or claimed scope cannot meet its registered minimum-size, threshold-reachability, or interval-precision rule, the detection core shall report that arm and scope as insufficient and shall refuse the shared population spend.
5. When every planned observation is complete and before label access, the detection core shall require an admission authority reviewed independently from its producer that binds the feasible plan to the exact completed observation identities and the current population custody state.
6. If an expected observation is absent, unreadable, changed, associated with another arm, input, or plan, or inconsistent with the feasible plan, the detection core shall refuse admission authorization.
7. When shared labels are released, the detection core shall require one admission preparation, one exact monotonic population transition, and one resulting label-release receipt binding every planned arm.
8. If the custody head is stale, the transition is not the exact available-to-spent successor, the receipt is absent or mismatched, or availability is later restored, the detection core shall refuse label release and scoring.
9. The detection core shall not authorize a partial spend for a subset of planned arms or scopes.
10. While the population remains unspent, the detection core shall permit admission reuse only when raw observations and population-specific diagnostics remain unavailable to detector development and only closed process status, counts, and digests are exposed.
11. If raw observations or population-specific diagnostics are inspected, or an arm, plan, configuration, registered budget, or execution identity changes after an attempt, the detection core shall require a development or spent disposition and shall refuse that population for admission.
12. When a pre-spend attempt is retried, the detection core shall accept the retry only under the exact unchanged population, plan, arms, inputs, configuration, registered budgets, and execution identities.
### Requirement 18: Exact-scope denominators and whole-population coverage
The detection core shall derive qualification independently for each arm's exact vulnerability-family and ecosystem scope while preserving separate whole-population coverage.
#### Acceptance Criteria
1. When quality is evaluated, the detection core shall derive one complete standard metric set for every exact arm, vulnerability-family, and ecosystem scope declared in the reviewed plan.
2. When recall or pair correctness is derived for a scope, the detection core shall use only vulnerable and fixed cases independently verified for that exact family and ecosystem.
3. When ordinary false alarms are derived for an arm, the detection core shall count each eligible ordinary push once for every family claimed by that arm in the push's reviewed ecosystem and shall not project it into another ecosystem.
4. When precision is derived for a scope, the detection core shall use only findings and cases assigned to that exact arm, family, and ecosystem under the reviewed denominator policy.
5. When one underlying push contributes to multiple verified scoped denominators, the detection core shall count it once in whole-population coverage.
6. When a scoped metric is reported, the detection core shall include its numerator, denominator, value, sample count, and Wilson 95% interval under the fixed metric algebra.
7. Where an overall aggregate is reported, the detection core shall identify its constituent arms and scopes, sample counts, and weighting and shall not attach a gate or authorization decision to the aggregate.
8. If a post-spend scoped denominator is zero or differs from the reviewed label-blind census and feasible plan, the detection core shall refuse the evaluation as invalid rather than report insufficient, go, or no-go.
### Requirement 19: Version-three case release, arm results, and proof binding
The detection core shall validate post-spend evidence separately for every planned arm while preserving shared population custody and combination lineage.
#### Acceptance Criteria
1. When post-spend cases are released, the detection core shall require one version-three case-release graph reviewed independently from its producer and bound to the reviewed evaluation plan, shared spend, and exact completed observations.
2. When a case is released for an arm, the detection core shall bind its arm identity, exact scope, role, pair identity, result, evidence, change inputs, reviewed input manifest, and release review.
3. If a version-two case release is supplied to a version-three evaluation, the detection core shall refuse it rather than reinterpret it under the new contract.
4. When a combined arm is evaluated, the detection core shall preserve every component arm's observations, metrics, gates, and decision and shall report the registered combination lineage.
5. If a combined result omits, replaces, or contradicts a required component result, the detection core shall refuse the combined evaluation.
6. While ground-truth roles remain sealed, the detection core shall treat every label-blind proof candidate as non-authoritative for demonstrated status.
7. When a proof-backed status is derived after case release, the detection core shall require an exact validated proof attachment matching the released family, scope, pair roles, change inputs, finding location, differential record, control binding, producer, verification receipt, and trust policy.
8. When a combined arm retains proof evidence, the detection core shall require the identical validated proof candidate from the matching component observation and shall prevent the combined arm from creating or transferring proof evidence.
9. If proof is absent or inconclusive, the detection core shall retain the finding as not demonstrated.
10. If a proof attachment is role-swapped, cross-family, cross-scope, location-mismatched, or bound to unrelated inputs or controls, the detection core shall refuse the attachment as invalid and shall not count it as demonstrated.
11. The detection core shall not allow one planned arm's valid result to hide another planned arm's missing, invalid, or failed result.
### Requirement 20: Complete evaluation reports and replay authority
The detection core shall make the complete multi-arm evaluation replay the only admission artifact capable of authorizing downstream consumption.
#### Acceptance Criteria
1. When a planned arm completes evaluation, the detection core shall report one go or no-go decision for every exact claimed scope from that arm's validated instrument, budget, readiness, and quality gates.
2. The detection core shall reserve go and no-go for completed quality evaluations and shall keep pre-spend insufficiency, unread input, skipped stages, incomplete arms, and other instrument failures distinct from quality decisions.
3. When an evaluation report is produced, the detection core shall preserve per-arm scoped metrics separately from whole-population coverage, completed and deferred work, coverage funnels, instrument and model failures, runtime, and every degradation reason.
4. When every planned arm has a valid report, the detection core shall produce one complete evaluation replay binding the reviewed plans, feasibility results, admission authority, shared custody transition and receipt, the version-three case-release graph, independently valid per-arm scoped reports, and combination lineage.
5. If any planned arm report, mandatory gate, custody binding, case release, or replay component is missing or invalid, the detection core shall refuse evaluation completion.
6. When an evaluation replay is validated, the detection core shall rederive identities, counts, metrics, gates, and decisions from the canonical source artifacts rather than trust caller-supplied summaries.
7. When a consumer requests admission evidence, the detection core shall provide authorization only from a complete validated evaluation replay and shall not authorize from a per-arm report or aggregate alone.
8. When a zero or clean-looking arm result is included, the detection core shall require nonzero input-read evidence and execution evidence for every required stage before accepting the replay.
## Out of Scope
- The pre-push hook transaction, rollout policy, and capability registry.
- Vulnerability-family algorithms, prompts, examples, and optimization programs.
- Framework- or repository-specific semantic fact content.
- Analysis or indexing of vendored dependency source.
- Proof execution and sandbox orchestration.
- Enabling a vulnerability family for advisory or blocking use.
- Analysis-engine, frontend, graph, and family-internal implementation details.
- Population sourcing, human curation, permanent reservation, licensing, and lifecycle operations.
- Durable object-store or compare-and-swap adapter implementation, bucket administration, and retention configuration.
- Detector execution and isolation, observation storage and access enforcement, and retry orchestration.
- Workflow heartbeat and operator-status presentation, notification transport, and product rollout.
"approval_note":"2026-11-06: Requirements and the merged V3 design were auto-approved by the maintainer's explicit -y request. Additive V3 task groups 12-18 passed the task-plan review and an independent graph sanity review; existing groups 1-11 remain historical V2 context, and blocked task 11.4 is not a V3 prerequisite. Tasks are approved for implementation."
| Completed groups 1–7 and 8.1 | Retained as regression substrate; their former requirement mappings do not count toward v2 coverage |
| Previous task 8.2 | Permanently parked after admission NO-GO #3; no task below revives that arm |
| Previous tasks 8.3–8.4 | Superseded by v2 consumer and proof conformance work; no rollout or live capability enablement is included |
| Existing task 11.4 | Retained as blocked historical V2 validation; it is not a prerequisite for additive V3 tasks 12–18, which have explicit dependencies and their own compatibility gate |
## Task constraints
- New and amended code keeps McCabe complexity at most 10, functions at most 50 logical lines, and new modules at most 400 logical lines.
- Core owns contracts and pure validation only; analyzers, model execution, proof execution, persistence, acquisition, and rollout remain with adjacent owners.
- Every zero-result test first proves nonzero readable input; an unread input is an instrument failure.
- Historical v1 artifacts remain byte-identical. V2 creates a new baseline and is never trended against v1.
- The parked authorization arm remains parked. Admission orchestration uses synthetic/reference inputs only and does not consume a live population.
- Existing target facts may be schema-migrated without expanding vocabulary or behavior.
- Tasks 12–18 are the current additive V3 implementation scope; completed groups 1–11 remain historical V2 implementation context.
- V2 aggregate roots and public routes remain frozen; V3 uses a distinct schema and adds no converter or inferred projection.
- Benchmark-owned storage, compare-and-swap, execution, access, retry, and workflow orchestration remain outside core and are exercised only through synthetic or in-memory boundaries.
- [ ] 1. Establish migration-safe foundations
- [x] 1.1 Freeze historical v1 artifacts and the reviewed migration inventory
- Pin representative v1 trace and report bytes and hashes as immutable regression fixtures.
- Provide an independent schema-routed read-only decoder that validates committed v1 artifacts without importing the compatibility namespace, constructing v2 determinate outcomes, or exposing a writer or converter.
- Freeze the reviewed initial set of v1 readers and writers with stable entry identities, paths, symbols or capabilities, owner boundaries, and open migration status.
- Preserve every inventory identity permanently and allow only monotone transitions from open to `v2_migrated` or `retired`; fail unreadable or zero-byte sources, additions, deletions, and rekeying.
- Preserve v1 as migration input only and never treat it as v2 conformance or a comparable trend baseline.
- Done when fixture tests detect any byte or hash change, the decoder validates both historical schemas read-only, every inventory entry resolves to a nonzero source and valid target, and schema, status, or addition mutations fail.
- _Requirements: 14.3, 15.3_
- [x] 1.2 Build the strict canonical codec and resolver-backed artifact boundary
- Produce deterministic UTF-8 JSON with full SHA-256 identities, stable mapping order, and retained sequence order.
- [x] 2.1 (P) Represent family-neutral semantic relations and observations
- Cover registration, invocation, call, return, value transformation, operation effect, and constraint effect.
- Preserve role, location, conditions, provenance, and independent complete, incomplete, or unresolved status.
- Refuse conflicting complete candidates by producing explicit ambiguity rather than selecting one.
- Done when contract tests exercise every relation kind and all three relation statuses with canonical identities.
- _Requirements: 1.2, 4.1, 4.2, 4.3, 5.2_
- _Boundary: Semantic Evidence_
- _Depends: 1.2_
- [x] 2.2 (P) Validate complete input and decision provenance
- Stamp analyzer, family contract, fact set, acquisition, configuration, population, and change-input identities.
- Require the closed named-input role set and resolve every referenced input through the trusted artifact boundary.
- Keep model and proof identities outside the immutable deterministic packet.
- Done when every stamp field has a tamper test and unreadable or substituted named input is refused.
- _Requirements: 10.1, 10.4_
- _Boundary: Provenance and Lineage_
- _Depends: 1.2_
- [x] 2.3 Enforce append-only population custody
- Construct and validate contiguous reviewed-custodian receipt chains against the complete current log.
- Require compare-and-swap requests to name the expected log head and derive the next sequence and predecessor.
- Make spent status monotonic and reject old prefixes, omitted receipts, broken chains, and later availability claims.
- Provide deterministic in-memory custody-port and custodian fixtures for lineage and admission tests.
- Done when concurrent requests have one valid successor and no fixture can restore a spent population.
- _Requirements: 12.2, 13.1, 13.4_
- [x] 2.4 Bind lineage purposes to exact influenced artifacts
- Require each lineage use to name its purpose, influenced artifact, source population, label access, and current-chain receipt.
- Enforce required and optional use counts for prompts, examples, thresholds, fitted models, facts, matching behavior, mappings, and promotion.
- Distinguish fact ancestry that forbids all label access from development lineage that may use a different spent population.
- Refuse protected-population influence, redirected artifact identities, wrong purposes, and incomplete receipt history.
- Done when matching subjects cannot mask a wrong influenced artifact and contaminated populations are deterministically refused.
- _Requirements: 13.1, 13.3, 13.4_
- [x] 2.5 Bind immutable evidence packets to neutral decisions
- Preserve family identity, security object, analyzed scope, ordered semantics, observations, family fields, completeness, and provenance.
- Keep family decision codes separate from violation, satisfied, and indeterminate outcomes.
- Require bounded scope for satisfaction, named reasons for indeterminate outcomes, and a valid deterministic, model-corroborated, or execution-backed basis.
- Prevent incomplete or ambiguous required relations from producing violation or satisfaction.
- Done when positive, satisfied, and incomplete records round-trip canonically and every invalid packet/decision combination is refused.
- [x] 5.4 Revalidate fact-derived relations and enforce the dependency boundary
- Re-open referenced packs, facts, match evidence, review policy, lineage, custody, and resolved bytes for every fact-derived relation.
- Reject undeclared roles, invalid scope, inconsistent content, and fact provenance that cannot be reproduced.
- Require reviewed semantic facts for determinate behavior attributed to excluded dependencies.
- Preserve missing dependency knowledge as named incomplete evidence rather than product-derived safety.
- Done when product-only dependency behavior is refused, a missing fact stays indeterminate, and the reviewed structurally matched equivalent is accepted.
- _Requirements: 2.4, 4.4, 10.4, 11.4, 11.5_
- [ ] 6. Add bounded model handoff and semantic shortcut probes
- [x] 6.1 (P) Preserve reproducible model requests, results, and responses
- Present the canonical evidence, completeness snapshot, family declaration, and declared residual question without importing a model runtime.
- Freeze provider, model, program, prompt, examples, fitted component, configuration, adapter, decision mapping, and input-bundle identities before execution.
- Preserve normalized responses plus one bounded redacted inline value or trusted immutable artifact, including failures.
- Done when matching success/failure fixtures round-trip and substituted request, packet, response, program, or mapping identities are refused.
- _Requirements: 9.1, 9.2, 10.3_
- _Boundary: Model Handoff_
- _Depends: 2.4, 3.1_
- [x] 6.2 Apply model results without rewriting deterministic evidence
- Validate current prompt/example/fitted-model/mapping lineage against the evaluated population and declared artifacts.
- Reject proposer/adjudicator role collision and results for another packet, family, residual, or mapping.
- Allow model-corroborated output only when deterministic residual prerequisites are complete.
- Prevent model output from creating semantic relations, satisfaction over missing evidence, or execution-backed basis.
- Done when accepted derivation leaves the evidence packet byte-identical and every incomplete or contaminated path remains indeterminate.
- Define closed numerator/denominator rules for pair correctness, recall, fixed and ordinary false alarms, precision, semantic coverage, and model success.
- Require Wilson 95 intervals with numerator and denominator; serialize zero denominators as null and prevent them from passing.
- Treat any identity mutation as a new baseline and refuse cross-arm or cross-scope aggregation.
- Done when a baseline mutation matrix and two-arm isolation fixture distinguish every non-comparable result.
- _Requirements: 11.6, 15.1, 15.3_
- _Boundary: Measurement_
- _Depends: 4.1, 4.4_
- [x] 8.2 Build complete measurement records and typed repository-holdout reports
- Record family decision vocabulary, neutral outcomes, evidence bases, fixed metrics, semantic funnel, model outcomes/failures, instrument failures, runtime, and readiness ancestry.
- Require every later-stage record to carry current prerequisite readiness stage results and digests.
- Define a typed repository-holdout report with an explicit evidence-use role, training and holdout manifest references, repository identities, fitting lineage, and distinct measurement/report identities.
- Refuse zero or clean-looking results unless tracked input reads and prerequisite semantic stages passed.
- Done when missing metrics, counts, failure fields, readiness ancestry, or holdout-report identity fields invalidate the corresponding record.
- _Requirements: 13.2, 15.1, 15.2, 15.4, 15.5_
- [x] 8.3 Validate independently reviewed census and manifest payloads
- Canonically bind every substantive census and manifest field into separate review-subject digests.
- Done when only the exact spent successor authorizes scoring and every other receipt shape is refused without an external side effect.
- _Requirements: 12.2, 13.4_
- [x] 8.7 Validate the authenticated post-spend case release
- Define canonical released-case bindings over case identity, scope, role, scoped pair identity, exact result digest, evidence-input digests, change-input digests, and input-manifest digest.
- Bind the release to the population, manifest, authorization, preparation, exact current-chain spend receipt, custodian identity, and an independent review attestation.
- Validate the release only after the exact spend successor and require one canonical binding for every supplied case and no others.
- Reject missing, extra, duplicate, cross-scope, role-swapped, pair-swapped, result-swapped, or rehashed operational-payload substitutions before metric production.
- Keep case identities and labels out of the pre-label census and make the external custodian, not the runner, authoritative for their post-spend association.
- Done when the reproduced vulnerable/fixed payload swap and every release-field mutation fail before any admission metric is returned.
- _Requirements: 12.2, 12.5, 12.6, 13.4, 15.1_
- _Boundary: Admission Case Release_
- [x] 8.8 Derive scoped admission metrics from canonical case evidence
- Re-run shared evidence validation, instrument, budget, verdict, and proof validation for every case.
- Consume only cases that exactly match the authenticated post-spend release; use its roles and scoped pairs while reconciling aggregate counts to the reviewed census.
- Score quality only over pre-registered family/ecosystem scopes while retaining whole-population coverage.
- Compute paired behavior, ordinary false alarms, semantic coverage, model outcomes, and all standard metrics without trusting runner aggregates.
- Done when forged decisions, substituted contracts, invalid fact review, contaminated lineage, or case-count drift fail before metric production.
- _Requirements: 12.5, 12.6, 15.1, 15.2_
- _Depends: 8.7_
- [x] 8.9 Assemble evidence-backed gates and the final admission report
- Emit instrument, budget, readiness, and quality gates with reasons and supporting identities.
- Bind the report to the validation context, authorization, pre-label spend receipt, authenticated case release, census, and complete case-result set.
- Require every mandatory metric and gate before allowing GO; make model success mandatory when a model is baseline-configured or validated claimed-scope calls occur. Refuse undeclared claimed-scope calls before report production; make configured zero-call, missing-threshold, and failed declared-call cases NO-GO while ignoring out-of-scope-only calls.
- Return an immutable admission replay bundle carrying the exact manifest, denominator, readiness, preparation, spend receipt, case release, cases, and derived report; give the bundle a canonical transport digest without treating it as a validation token.
- Revalidate the replay against independently supplied current context, custody, and content resolution by re-running the complete derivation and requiring exact report and replay equality.
- Keep model-corroborated, deterministic, and execution-backed counts separate.
- Done when any missing/failed mandatory gate forces NO-GO, coordinated report/readiness/threshold/case substitutions are refused, and a complete passing synthetic fixture yields a reproducible replayed GO report.
- _Requirements: 12.6, 15.1, 15.2, 15.4, 15.5_
- _Depends: 8.8_
- [x] 8.10 Authorize fitted components only from replayed probes and disjoint evaluation
- Accept only the typed admission replay bundle from task 8.9, validate it first against current authoritative context/custody/content, and refuse raw admission reports or coordinated replay substitutions.
- Resolve the component’s unique model declaration, family contract, review policy, and custody policy from the admission-bound context.
- Re-evaluate every canonical probe input and require the derived result set to match the component evidence.
- Consume the typed repository-holdout report from task 8.2 and validate its evidence-use role, training/holdout references, report identity, and fitting lineage.
- Prove the report’s training and repository-held-out repository sets are disjoint.
- Require an admission GO that was untouched before its bound label-release spend.
- Done when fabricated summaries, caller-selected policy substitutes, overlapping repositories, contaminated lineage, or mismatched admission are refused.
- _Requirements: 13.1, 13.2, 13.4_
- _Depends: 8.2, 8.9_
- [ ] 9. Integrate adjacent producers and benchmark-owned orchestration
- [x] 9.1 (P) Translate the parked authorization trace conservatively into v2 evidence
- Move conversion out of the family trace domain and keep authorization vocabulary outside shared core modules.
- Map missing per-hop knowledge to incomplete relations and indeterminate outcomes without behavior lifting.
- Preserve existing parked-arm regression behavior and avoid any admission rerun or reactivation.
- Done when adapter conformance matches the frozen v1 behavior while the authorization domain no longer imports v2 core types directly.
- _Requirements: 3.5, 14.3_
- _Boundary: Authorization Adapter_
- _Depends: 3.3, 7.2_
- [x] 9.2 (P) Translate hunt-owned model execution through the bounded handoff
- Convert structured executor inputs and outputs to the canonical request/result contracts.
- Keep prompt creation, provider calls, caching, parsing, and redaction in the hunt subsystem.
- Reject mismatched declared results through core validation without importing DSPy into core.
- Done when a fake executor success/failure pair crosses the adapter and core’s import census contains no model-runtime dependency.
- Consume the immutable conformance fixture and manifest produced by the downstream family adapter without constructing its contract, packs, matches, authority, or target data in core.
- Re-run only the family-neutral fact-pack, fact-match, relation-application, full-evidence, canonical-reference, and import-boundary checks named by that handoff.
- Verify the manifest accounts for complete, name-only, partial, substituted, self-authored, protected-population-contaminated, and legacy-projection-only cases.
- Done when the downstream handoff is accepted by generic validators and core remains free of target imports; adapter implementation and target-specific fixtures remain downstream-owned.
- [x] 9.6 Integrate benchmark-owned admission scoring and guarded custody
- Delegate neutral scoring to core while retaining population persistence and run orchestration in the benchmark.
- Make the custodian the sole append-only writer and combine durable compare-and-swap spend plus label release as one guarded operation.
- After the winning spend, produce and independently attest the canonical case release over the exact immutable result, evidence, change, and input identities supplied to scoring.
- Supply expanded manifest, lineage, resolver, and validation-context identities without exposing labels early.
- Exercise durable append, label release, case-release production, crash-before-append, crash-after-append, and concurrent-winner behavior at this external boundary.
- Done when synthetic admission completes end to end, injected crash/concurrency cases prove labels reach at most the successful compare-and-swap winner, and swapped operational payloads are refused by core.
- [x] 10.1 Isolate every live v1 contract and writer behind the closed compatibility namespace
- Move all authorization-shaped v1 values into one compatibility contract owner and all live v1 serialization into its sole writer facade.
- Limit the compatibility namespace to the canonical primitive needed for byte stability and prevent every v2 module from importing it.
- Preserve the frozen v1 wire representation while eliminating independent contract or serializer implementations elsewhere.
- Done when old writes remain byte-identical, the compatibility namespace is the only live v1 contract and write owner, and ownership/import tests refuse any duplicate implementation.
- _Requirements: 14.3, 15.3_
- _Depends: 1.1_
- [x] 10.2 Convert the temporary public and module compatibility surfaces into explicit shims
- Keep the old named top-level API and designated compatibility modules as explicit imports from the isolated namespace during migration.
- Remove legacy definitions and re-exports from mixed v2 modules so each compatibility name has exactly one owner.
- Keep the canonical v2 paths private until the final atomic cutover.
- Done when the old named API remains compatible before cutover, direct imports of moved legacy symbols from mixed modules fail, and no shim owns an independent implementation.
- _Requirements: 14.3, 15.3_
- [x] 10.3 Build the syntax-only migration-boundary instrument
- Prove every scoped source file is readable and nonzero before interpreting a zero result.
- Report explicit compatibility-namespace imports, designated shim imports, named top-level v1 symbols, writer-facade references, and module-valued or dynamic core access without evaluating alias or data flow.
- Validate immutable inventory identities and their monotone open, `v2_migrated`, or `retired` status transitions.
- During migration, distinguish the designated compatibility implementation from the production-consumer scope; reserve repository-wide zero for post-deletion validation.
- Done when an adversarial syntax matrix reports the exact forbidden sites and `try`/`finally`, comprehension, assignment, and generic-call alias shapes cannot turn an explicit forbidden reference into a false zero.
- _Requirements: 14.3, 15.3_
- [x] 10.4 Migrate or retire remaining registered production source-tree consumers
- Move each registered non-benchmark production reader and writer to the canonical v2 contract or retire it within its owning boundary.
- Update the matching stable inventory entry to `v2_migrated` or `retired` in the same change without removing or rekeying it.
- Keep authorization analysis, model execution, proof execution, and other adjacent behavior in their existing owner boundaries while changing only their contract seam.
- Done when every source-tree entry owned by core or an adjacent adapter is terminal and the verified nonzero production-consumer scope outside the designated compatibility implementation has zero forbidden references.
- _Requirements: 14.1, 14.3, 15.3_
- _Boundary: Core and Adjacent Adapter Migration_
- _Depends: 10.3_
- [x] 10.5 Migrate benchmark readiness and admission consumers
- Move every registered benchmark reader and writer to v2 while retaining readiness sequencing, persistence, population custody, and admission orchestration in the benchmark boundary.
- Update each matching stable inventory entry to `v2_migrated` or `retired` without changing its identity.
- Preserve guarded spend, authenticated case release, fixed measurement, and bounded model behavior through the migration.
- Done when benchmark production scope has zero forbidden consumer references outside the designated compatibility implementation, every benchmark entry is terminal, and synthetic v2 readiness and admission complete end to end.
- _Requirements: 8.1, 12.6, 14.3, 15.3_
- _Boundary: Benchmark Readiness and Admission Adapters_
- _Depends: 9.5, 9.6, 10.3_
- [x] 10.6 Migrate or retire registered test and operational-tool consumers
- Move authorization assumptions into legacy or adapter tests, keep family-neutral contract tests on v2, and retain frozen v1 bytes only for decoder regression.
- Move or retire registered operational-tool consumers without adding a second compatibility or write path.
- Mark the corresponding stable inventory entries `v2_migrated` or `retired` and require every original entry to reach a terminal status.
- Done when the complete original inventory is terminal and the boundary instrument proves a nonzero repository scope with no forbidden consumer reference outside the designated compatibility implementation.
- _Requirements: 14.3, 15.3_
- _Boundary: Migration Tests and Operational Tooling_
- _Depends: 10.4, 10.5_
- [x] 10.7 Delete v1 write capability and publish the single v2 surface atomically
- Require every inventory entry to be terminal and the migration-time production-consumer boundary to be zero before deletion begins.
- Delete the compatibility namespace, v1 writer facade, and temporary shims while publishing the canonical v2 modules and top-level exports in the same integration change.
- Prove old top-level and submodule imports and writes fail and that no v1 serializer or converter remains.
- Retain the independent read-only decoder for frozen historical bytes and refuse comparison between v1 and v2 baseline identities.
- After deletion, require repository-wide zero with no compatibility exclusion.
- Done when isolated-tree and repository tests show old imports and writes fail, v2 imports and writes succeed, historical bytes still validate, and the final nonzero boundary scan remains zero.
- [x] 11.1 Run both reference families through the complete synthetic readiness path
- Start from a verified nonzero input and run recognition, transitions, security object/constraints, pair, development, reviewed synthetic admission, guarded spend, and consumer verdict.
- Produce violation, bounded satisfaction, and bounded silence for both unlike families.
- Require current prerequisite readiness identities at every later stage.
- Done when both authored repositories complete the path without target-specific core fields or a real admission population.
- [x] 11.3 Verify bounded validation cost and total consumer deadline accounting
- Measure canonicalization and validation across increasing relation, observation, receipt, and case counts.
- Exercise large transition sets for linear behavior and bounded memory without weakening semantic checks.
- Include optional model and proof processing in the same measured consumer budget.
- Publish sample count, percentile, workload, hardware, and derived budget through the fixed measurement contract.
- Done when performance fixtures satisfy the derived workload envelope and any excess fails the named budget gate.
- _Requirements: 1.4, 6.1, 6.2, 6.3, 6.4_
- [ ] 11.4 Run the full migration, security, and regression gate
- Verify core imports only standard-library and lower core layers and contains no target, DSPy, benchmark, proof-execution, or analyzer dependencies.
- Run all unit, adapter, migration, reference-family, end-to-end, admission, and historical compatibility suites.
- Enforce function/module complexity caps and scan for remaining v1 imports or writers.
- Prove each test instrument read nonzero fixture bytes before interpreting zero findings or failures.
- Done when the complete suite passes, boundary/import censuses are clean, historical bytes are unchanged, and all declared adapter and consumer seams pass conformance.
- _Requirements: 1.4, 10.4, 14.3, 15.3, 15.4_
- _Depends: 10.7_
- _Blocked: Benchmark population governance conflict: the permanent unseen-pool reservation guard and the later spent/retired training-only pilot policy disagree. The benchmark owner must approve the authoritative lifecycle and replacement/retirement plan before the full suite can pass._
## Evaluation-admission V3 amendment
Tasks 12–18 implement Requirements 16–20 as an additive predecessor for the approved
valid-detection-benchmark design. Existing groups 1–11 remain the historical V2
implementation record. The unresolved downstream-governance item 11.4 is not reopened and
is not a prerequisite for this V3 stream; V3 tasks depend explicitly on completed V2
contract work and use synthetic/in-memory workflow boundaries only.
- Represent one canonical arm-and-scope key for every policy, threshold, sufficiency rule, metric, gate, and decision.
- Bind exact denominator algebra, fixed Wilson intervals, sample counts, minimum sizes, interval precision, and rational aggregate weights.
- Include every arm identity input that must start a new baseline when changed, including judgment vocabulary, analyzer, facts, configuration, registered budget, execution, outcome mapping, and combination rule.
- Refuse duplicate, unsorted, unknown, cross-key, non-finite, or incomplete values through exact closed validation.
- Done when canonical contract tests accept one complete component and combined-arm configuration and reject every single-field identity or arm-scope mismatch.
- Reserve the exact evaluation-admission V3 schema identity and version-routing primitives without changing any V2 public name or route.
- Make V3 contracts exact-type only and prohibit V2-to-V3 conversion, subclass acceptance, and mixed-version child relationships as each later root is introduced.
- Define the shared non-authorizing marker for feasibility, insufficiency, raw reports, aggregates, incomplete arm results, and other partial values; leave complete root decoding to facade integration.
- Done when boundary tests preserve exact V2 routing, recognize only the reserved V3 identity on the V3 path, and prove that a representative partial V3 value has no authorization capability.
- Represent component and combined arm configurations with exact scopes, policies, thresholds, sufficiency rules, outcome mapping, and a closed declarative combination rule.
- Require component identities and a finite normalized decision table for every combined arm, with identical-component-candidate-only proof retention.
- Done when canonical component and combined configurations validate and every component, rule-row, policy, threshold, sufficiency, or outcome-mapping mutation changes the configuration identity.
- _Requirements: 16.5, 17.1, 17.2, 19.4_
- _Boundary: Evaluation Arm Configuration_
- _Depends: 12.3_
- [ ] 13.2 Build the non-circular plan declaration, baselines, and expected slots
- Derive the complete label-blind plan declaration before constructing per-arm baselines so the plan and baselines cannot form a digest cycle.
- Bind each baseline to the population, freeze, input set, declaration, and exact arm configuration so any listed change creates a new baseline.
- Define expected observation slots for every planned arm and input without including ground-truth roles or labels.
- Done when plan reordering is canonical, every substantive mutation changes the declaration and affected baselines, and a combined arm cannot name an absent component.
- _Requirements: 16.5, 17.1, 17.2_
- _Boundary: Evaluation Plan_
- _Depends: 13.1_
- [ ] 13.3 Validate the reviewed label-blind census and plan manifest
- Represent exact vulnerable/fixed scope capacities, ecosystem-specific ordinary-push capacities, semantic/model capacities, and deduplicated whole-population size without case labels.
- Require an independently reviewed plan payload that binds the population, input set, arm configurations, baselines, readiness, custody head, census, and expected slots.
- Resolve review evidence under the configured trust and byte limits and refuse self-review, copied attestations, exposed labels, missing scopes, or inconsistent population identities.
- Done when one reviewed multi-arm manifest validates with labels sealed and every review, census, scope, ordinary-ecosystem, or slot substitution is refused before feasibility.
- _Requirements: 17.1, 17.2, 17.3, 18.1, 18.3_
- _Boundary: Evaluation Plan Review_
- [ ] 13.4 Derive feasibility independently for every planned arm and exact scope
- Produce exactly one feasibility row for the canonical product of planned arms and their claimed scopes.
- Recompute minimum sample size, threshold reachability, and Wilson interval precision from the reviewed label-blind census for each row.
- Derive shared spend eligibility only when every row is feasible; keep insufficiency and its reasons distinct from GO and NO-GO.
- Refuse missing, extra, duplicated, or caller-summarized rows and prevent preparation when one row is insufficient.
- Done when a single insufficient scope blocks the shared spend without opening labels, while the complete feasible plan yields one reproducible preflight digest.
- _Requirements: 17.3, 17.4, 20.2_
- _Boundary: Evaluation Feasibility_
- [ ] 14. Seal planned observations and authorize one atomic spend
- [ ] 14.1 Validate exact nonzero observations against every expected slot
- Resolve every sealed observation through trusted content references and prove its bytes are nonzero before accepting its header.
- Require one-to-one equality between expected slots and sealed observations; refuse missing, extra, changed, unreadable, cross-arm, cross-input, or duplicate values.
- Done when the complete slot set validates and each single-slot removal, swap, truncation, zero-byte value, or identity mutation is refused before authority.
- Bind each attempt event to the complete retry identity, closed access policy, raw-observation and diagnostic access counts, public status projection, disposition, review, and predecessor.
- Resolve the reviewed predecessor chain from its declared head and require every unspent attempt to have the identical population, plan, arms, inputs, configurations, budgets, and execution identities.
- Permit public pre-spend projection only for closed status codes, counts, and digests; any raw/diagnostic access or identity change produces a terminal development-opened or spent disposition.
- Enforce terminal monotonicity so a later sealed event cannot restore admission eligibility.
- Done when an unchanged all-sealed retry chain validates and broken, forked, changed, exposed, self-reviewed, or terminal-then-sealed histories cannot authorize.
- _Requirements: 17.10, 17.11, 17.12_
- _Boundary: Evaluation Attempt History_
- _Depends: 13.3_
- [ ] 14.3 Validate the independently reviewed admission authority
- Bind the feasible plan to the exact sealed observation set, attempt-history head, current readiness, validation context, and current complete custody head.
- Require the admission-authority payload to be reviewed independently from its producer and resolve its review evidence before authorization.
- Revalidate feasibility, all observation bytes, attempt history, readiness, and custody immediately before producing authority.
- Refuse stale or restored availability, changed observations, missing review, and any difference from the reviewed plan rather than returning a partial authority.
- Done when one canonical authority is derived only from the complete reviewed graph and every stale, substituted, or incomplete source fails before spend.
- [ ] 14.4 Bind every planned arm to one preparation and custody receipt
- Construct one head-bound label-release request whose influenced artifact transitively commits to every planned baseline and exact authority.
- Validate only the exact monotonic available-to-spent successor and derive an explicit all-arm binding between preparation, receipt, and planned baselines.
- Reject subset preparations, missing or mismatched receipts, stale concurrent attempts, broken custody chains, and any later restoration of availability.
- Keep durable compare-and-swap and label release outside core while exercising the boundary with an in-memory custody port.
- Done when one receipt authorizes all and only the planned arms, concurrent/stale losers fail, and no partial spend is constructible.
- _Requirements: 17.7, 17.8, 17.9_
- _Boundary: Evaluation Shared Spend_
- _Depends: 14.3_
- [ ] 15. Authenticate V3 case release, combinations, and proof continuity
- [ ] 15.1 Build the arm-aware V3 case and independently reviewed release graph
- Bind each post-spend case to its arm-scope key, baseline, observation, underlying push, role, pair, result, evidence inputs, change inputs, input manifest, and proof-candidate identity.
- Bind one independently reviewed release to the plan, admission authority, preparation, exact shared spend, and complete canonical case set.
- Require exact planned-arm and released-case cardinality and exact V3 root types; refuse V2 releases and subclasses.
- Require nonzero read and required-stage execution evidence in every released case before it can enter scoring.
- Done when the complete V3 release validates and V2 substitution, missing arms, duplicate cases, wrong spend, or copied review fail before metric production.
- Recompute every case and binding digest from canonical evidence rather than accepting caller-supplied summaries.
- Require vulnerable/fixed roles and scoped pair identities to form exact pairs; prohibit pair identities on ordinary cases.
- Preserve the separately reviewed underlying-push identity for scoped projection and later whole-population deduplication.
- Refuse arm, scope, role, pair, observation, result, evidence, location, input, or manifest swaps before any count is derived.
- Done when a canonical multi-scope release passes and a mutation matrix proves every association mismatch is invalid rather than a failed quality gate.
- Recompute combined observation outcomes from the registered finite decision table and exact component observation digests.
- Preserve each component arm's observation identity and outcome independently of the combined result; defer completed metric, gate, decision, and report preservation to replay assembly.
- Refuse missing, extra, replaced, reordered, or contradictory component material and any combined arm whose lineage differs from the reviewed plan.
- Done when the registered combined arm replays deterministically from exact component observations and no component observation can hide another component's absence, contradiction, or lineage mismatch.
- _Requirements: 19.4, 19.5, 19.11_
- _Boundary: Combination Validation_
- _Depends: 15.1, 15.2_
- [ ] 15.4 Enforce proof-candidate identity and prevent proof transfer
- Keep every label-blind proof candidate non-authoritative until the V3 release establishes its exact arm, family, scope, roles, inputs, and location.
- Apply proof-backed status only through the existing trusted differential-proof validator and the identical released candidate.
- Permit a combined arm to retain only a candidate already present in its named component observation; prevent creation, substitution, and cross-arm transfer.
- Preserve absent or inconclusive proof as not demonstrated and refuse role-swapped, cross-family, cross-scope, location, input, control, producer, receipt, or trust-policy mismatches.
- Done when valid released proof attaches once, absent proof remains non-demonstrated, and the complete transfer/mismatch matrix is refused.
- [ ] 16.1 Partition quality evidence by exact arm, family, and ecosystem
- Select vulnerable and fixed cases only from the independently released exact scope and derive pair correctness without cross-scope reuse.
- Derive precision only from findings and cases assigned to the same arm-scope key under its reviewed denominator policy.
- Preserve family decision vocabulary, neutral outcomes, evidence bases, model outcomes, and semantic coverage independently for each key.
- Done when a multi-arm, multi-scope case set yields isolated partitions and any cross-arm, cross-family, or cross-ecosystem contribution is refused.
- _Requirements: 18.1, 18.2, 18.4_
- _Boundary: Evaluation Scoped Metrics_
- _Depends: 15.2, 15.4_
- [ ] 16.2 Project ordinary pushes by ecosystem and deduplicate whole coverage
- Count each eligible ordinary push once for every family claimed by the arm in the push's reviewed ecosystem and never in another ecosystem.
- Deduplicate one underlying push once in whole-population coverage even when it contributes to several verified scope denominators.
- Reconcile projected ordinary counts and global unique-push counts back to the reviewed census and released identities.
- Done when same-ecosystem multi-family projection produces the expected scoped counts, cross-ecosystem projection is absent, and whole coverage counts each push once.
- _Requirements: 18.3, 18.5_
- _Boundary: Evaluation Ordinary Projection_
- [ ] 16.3 Produce the complete scoped metric set and reject post-spend invalidity
- Derive every standard metric for every arm-scope key using the fixed algebra and Wilson 95% calculation.
- Publish numerator, denominator, sample count, value, and interval bounds for each scoped metric.
- Treat a post-spend zero denominator, census difference, missing standard metric, or inconsistent projection as invalid evaluation evidence rather than insufficiency, GO, or NO-GO.
- Done when complete nonzero scoped metrics match independent count fixtures and every zero/mismatch mutation fails before report construction.
- Derive instrument, budget, readiness, and quality gates for every exact arm-scope result from canonical evidence.
- Emit GO or NO-GO only when the scope evaluation is complete; keep insufficiency, unread input, skipped stage, incomplete arm, and instrument impairment outside quality decisions.
- Preserve per-arm completed and deferred case identities, stage funnel, instrument/model failures, runtime, and closed degradation reasons separately from shared whole coverage.
- Require nonzero input reads and required-stage execution before accepting a clean-looking result.
- Done when every completed scope receives one reproducible decision and every incomplete or impaired fixture is refused without a quality decision.
- _Requirements: 20.1, 20.2, 20.3, 20.8_
- _Boundary: Evaluation Report_
- [ ] 16.5 Derive optional aggregates that can never authorize
- Bind aggregate constituents, sample counts, canonical rational weights, and derived metric values to exact arm-scope metric identities.
- Require weights to be non-negative, canonical, one-to-one with constituents, and sum exactly to one through integer arithmetic.
- Exclude gates, GO/NO-GO, proof promotion, and every consumer-authorization capability from the aggregate contract.
- Done when a valid aggregate reproduces from its constituents and raw, incomplete, gated, decision-bearing, or differently weighted aggregates are refused as authorization evidence.
- _Requirements: 16.6, 18.7, 20.7_
- _Boundary: Evaluation Aggregate_
- [ ] 17. Close complete replay and replay-only consumer authorization
- [ ] 17.1 Assemble one complete replay over the shared evaluation graph
- Require exact equality between planned arms/scopes and case/report identities, including every component required by a combined arm.
- Prevent replay construction when any arm, scope, mandatory gate, custody binding, release, case, component, or lineage is missing or invalid.
- Done when the complete graph produces one deterministic replay digest and every omitted or substituted component prevents construction.
- _Requirements: 19.4, 19.5, 19.11, 20.4, 20.5_
- _Boundary: Evaluation Replay_
- _Depends: 15.3, 16.4, 16.5_
- [ ] 17.2 Revalidate replay from canonical source artifacts and current trust state
- Resolve the complete attempt chain and observation/review content again, revalidate current custody, and rederive every identity, count, metric, gate, decision, report, lineage, and replay digest.
- Require exact equality with the carried values and refuse coordinated rehashing, stale external state, unknown/mixed versions, and partial roots.
- Make the validated complete replay the only value that may cross a V3 authorization boundary.
- Done when independent replay validation reproduces the original reports exactly and every source, summary, or trust-state mutation is refused.
- _Requirements: 16.6, 20.5, 20.6, 20.7, 20.8_
- _Boundary: Evaluation Replay Validation_
- [ ] 17.3 Publish the integrated V3 preflight, authorization, scoring, and replay facade
- Wire the exact plan preflight, reviewed authority, shared-spend validation, score/replay, strict decoding, and replay validation through one public V3 contract surface.
- Preserve every V2 facade and import unchanged and expose no benchmark storage, execution, lifecycle, or presentation implementation from core.
- Reject wrong-phase values and make each returned artifact usable only by the next declared phase.
- Done when a client can execute the complete synthetic contract through the public V3 schema and V2/unknown/mixed artifacts fail at the first boundary.
- [ ] 17.4 (P) Authorize fitted components only from one exact replayed arm-scope result
- Validate the complete V3 replay before selecting exactly one planned arm-scope result and require that completed result to be GO.
- Preserve existing reversed-pair probe replay, fitting lineage, exact model declaration, and disjoint resolved repository-holdout checks.
- Refuse V2 replay, raw arm report, aggregate, incomplete replay, missing/ambiguous key, NO-GO scope, or substituted fitted evidence.
- Done when a matching fitted component authorizes only from its exact GO scope and every partial, aggregate, cross-arm, or probe/holdout mutation fails.
- _Requirements: 13.2, 16.6, 20.7_
- _Boundary: Fitted Consumer V3_
- _Depends: 17.2_
- [ ] 18. Validate the V3 predecessor end to end
- [ ] 18.1 Run a synthetic multi-arm, multi-family evaluation through one shared spend
- Use at least two component arms and one combined arm across at least two families and two ecosystems with multi-scope fixes and representative ordinary pushes.
- Preserve component observations/results and verify ordinary projection plus shared whole-population deduplication in the completed replay.
- Done when the public V3 facade produces one validated replay with every planned arm/scope and no V2 or benchmark-owned implementation is used as a substitute.
- [ ] 18.2 (P) Run the adversarial pre-spend and compatibility matrix
- Exercise V2 byte/digest/replay stability, V2-as-V3, unknown/mixed/subclass schemas, baseline mutations, impossible scopes, missing slots, zero/unreadable observations, self-review, forked attempt history, identity changes, raw access, stale custody, and subset spend.
- Prove every measurement instrument reads nonzero fixture bytes and fails loudly on unread input.
- Verify insufficiency leaves custody available and no refused case reaches detector execution or label release in the synthetic boundary.
- Done when every pre-spend mutation fails at its owning layer, V2 remains unchanged, and no failure produces a clean-looking result or quality decision.
- Verify invalid post-spend evidence never becomes insufficiency, GO, NO-GO, aggregate authority, or consumer eligibility.
- Revalidate every complete replay from source artifacts and current custody rather than trusting carried summaries.
- Done when the valid control replay reproduces exactly and every post-spend mutation is refused before authorization without hiding another arm's status.
- Verify dependency direction, no V2/V3 aggregate imports across the version boundary, standard-library-only core runtime, and absence of benchmark, analyzer, model-runtime, or proof-execution ownership in V3 modules.
- Enforce function and module complexity caps and measure validation growth over increasing arms, scopes, observations, attempt events, cases, and reports.
- Fail each measurement fixture unless its scoped inputs are readable and nonzero before interpreting timing, count, or clean-result output.
- Done when import/boundary checks are clean, amended code stays within complexity caps, and validation cost remains bounded by supplied artifact counts without weakening checks.
- _Requirements: 16.2, 20.6, 20.8_
- _Boundary: V3 Boundary and Performance Validation_
- _Depends: 18.1, 18.2, 18.3_
- [ ] 18.5 Run the final compatibility, regression, and schema-export gate
- Run the V2 golden suite, every V3 unit/integration/adversarial suite, and the existing core regression suite without consuming a real admission population.
- Prove all fixtures are readable and nonzero and the exact `openultrasast.detection-core.evaluation-admission.v3` schema is publicly exported.
- Done when all gates pass, V2 outputs remain frozen, validation stays bounded by supplied artifacts, and the validated predecessor is ready for downstream benchmark task generation.
- _Requirements: 16.1, 16.2, 16.4, 20.6, 20.8_
- _Boundary: V3 Final Validation Integration_
- _Depends: 18.4_
## Implementation Notes
- The frozen migration inventory is a reviewed worklist with stable identities, not a semantic-use census. It may only close entries as `v2_migrated` or `retired`; it never grows, deletes entries, or proves absence by interpreting Python flow.
- Cutover is proved by explicit syntax checks plus deletion of the compatibility and v1 write capability. No task extends the discarded alias/data-flow interpreter.
- Task 10.5 debug established that executable owner probes cannot compensate for blanket source-closure acceptance; resumed hunt attempts also require collision-resistant attempt identity and create-only persistence before handoff validation is authoritative.
- Task 10.6 debug established that exact-byte successor authority must require canonical repository-relative paths and strict resolved-root containment, including rejection of outside-root symlink targets, before module digests can prove closure.
- Human review reopened Task 10.6 for one focused remediation cycle covering canonical path syntax, resolved-root containment/symlink refusal, and the task-local formatting gate.
- Human review reopened Task 10.5 for one focused remediation cycle covering exact source-closure evidence, collision-resistant create-only attempt persistence, and complexity-cap conformance.
- Task 11.4 debug established that the 73 reserved-repository references are a benchmark-governance specification conflict, not a detection-core or instrument defect; historical artifacts and the reservation guard must remain unchanged pending an owning-spec decision.
Sized the LLM tier on banks (the clearest pilot miss) with the project's own DeepSeek client (`deepseek-flash`).
DeepSeek is reachable, and model id `deepseek-flash` is valid. **~$0.00016–0.00025 per
call, ~1.1 s latency.** This costs less than a cent per push even at 8 candidates × a few regions. The GLM/abliteration fallback
(`ABLITERATION_*`) currently errors `HTTP 422` on the `thinking` request param. A client fix (don't send
`thinking` to that endpoint) is needed before it can be used; DeepSeek works today so it is not blocking.
"2 guided by 1" works in this case: the LLM gives the correct verdict when given the real sink + source
context:
- For `_load` (the pool's SZZ-labelled site, 135–141), the model returned `vulnerable=false`. This is correct: `_load` reconstructs
paths from its own index, not the sink.
- Asked about the real sink `from_prompt_path`/`set()` line 44 (`path / f"{prompt.name}…"; write_text`) with the
context that `prompt.name` can be request-derived: **`vulnerable=true`**, *"prompt.name is joined into a
filesystem path and written without sanitization… path traversal or arbitrary file write."* This is a correct catch.
Two blockers to realising it in the live pipeline (both now concrete):
1. **Context, not single-shot.** The model only catches with the real sink + caller/source context. A single-shot
question on the ranker's region (which pointed at `_load`, and at the benign local-glob `_scan`) does not. This
requires the **agentic search-with-proof hunter** (explores callers/entry points to establish the trust
boundary), not the single CANDIDATE_QUESTION path. That needs more calls → their cost is the next measurement.
2. **Corpus-label soundness.** The pool's banks label is mislocated: `head_lines=[[30],[104],[138–141]]`,
`function=_load`. The real sink is line **44**, outside head_lines±10 and a different function. So (a) even a
correct finding at line 44 would not score as a catch under `location_match`, and (b) the sink pre-exists in base
(the change did not touch it), so a change-scoped pre-push hook **correctly stays quiet**. The pool even
annotates this bias ("delta hook can correctly stay quiet"). **Part of the pilot's engine 0/5 is this SZZ
mislocation, not a detection failure.**
The budget is approved. Keep the revised next steps staged:
- Validate the 5 pilot labels first (cheap, LLM-assisted corpus audit): for each, is the real sink inside the
changed lines, and is it change-introduced vs pre-existing-in-base? Only sound, change-located vulns are fair
targets. This separates true detection gaps from SZZ artifacts before spending on detection runs.
- Then run the agentic hunter (search-with-proof) on the sound targets and size its per-vuln call count/cost;
measure catch + false-alarm (on ordinary changes) vs regex 1/5. The single-shot path is insufficient; the hunter
is the real "#1".
- Feed confirmed source shapes (e.g. library param → path join) back as guarded static facts (#2).
## Prompt evolution is a CORE lever for the LLM arm (maintainer directive 2026-10-12)
**"What is key is that we evolve the prompts as well"**, and **"GEPA or comparable; this is an evolving field."**
The bedrock hunter ran (64 model calls, $0.0076), but its **current prompts** led it to flag adjacent concerns
(network-PUBLIC default, download_file). It never named the `install_packages` command injection. This was a
*prompting/targeting* miss, rather than a cost or capability limit. So the LLM arm needs a **prompt-optimisation loop** that reflects on misses/low-confidence
outputs, updates the prompt, and retries. Build it on **DSPy** (its GEPA / MIPRO optimisers). Keep it **pluggable**
behind an interface because the field changes quickly (GEPA, MIPRO, TextGrad, successors). This aligns with
[[learned-decisions-not-hand-tuning]] (learned decisions + local memory, not hand rules). It applies to both LLM
uses here: the search-with-proof detector and the corpus-label auditor.
The optimisation must learn which hints/annotations the model needs to reach the real sink. Rewording the
prompt alone is insufficient. bedrock needed the hint "`install_packages` runs `pip install` with caller-supplied package
names → command injection"; banks needed "`prompt.name` is request-derived and joined into a filesystem path →
traversal". DSPy optimises which evidence/hints to feed the signature so the model targets the true sink instead of
adjacent noise.
Use the project's existing framework knowledge for those hints. The semantic facts taxonomy
(`ruleset/semantic/*.toml`, `ruleset/frameworks.toml`) encodes sources/sinks/sanitizers per framework without
user annotations ([[php-detection-is-engine-only]] zero-annotation value). Where framework knowledge covers the
API, feed it as a hint. Where it does not (library APIs like banks/bedrock), the LLM proposes the trust boundary,
which becomes a fact after repeated confirmation (the #2 feedback loop). No new annotation burden on the
user. This is now part of the "#2 guided by #1" workstream and must not be deferred. (DeepSeek
`deepseek-flash` is the working default; GLM 5.3 "cyber" model is future-spec.)
## POOL-P1 LABEL AUDIT (LLM, 2026-10-12): 47% of labels are sound
The banks label-location check was extended to all **49 public** pool vulns via DeepSeek (advisory + head code +
pool label → sound/unsound). Full artifact: `benchmarks/measurements/2026-10-12-pool-p1-label-audit/`.
> **Status 2026-10-02:** dormant. Requirements approved and design generated on 2026-09-09; the design was never approved and no tasks exist. It depends on `contributor-scan`, which is paused; reopen it with that spec.
## Introduction
Nine engine defects were found in one session against two repositories. **Three were the same bug in three
matchers** — a spec token matched as a substring rather than a word: `resolve` inside `resolveUrl` in the
matchers**: a spec token matched as a substring rather than a word: `resolve` inside `resolveUrl` in the
sanitizer matcher, `user` inside `users` in the discharger matcher, `gets` inside `fgets` in the sink matcher.
Each was found by a measurement, fixed locally, and left no trace; the third survived until an 89k-line C
library made it visible as 26 identical false positives.
Nothing in the system remembers that a finding was wrong. There is no artifact saying *this site, at this
rung, is not a defect, and here is why* — so a class recurs, and each recurrence costs another measurement.
rung, is not a defect, and here is why*, so a class recurs, and each recurrence costs another measurement.
Two audiences want that artifact for different reasons. A maintainer wants the *propagation* automated: two of
the three matcher bugs needed no judgement at all, only the fix carried across. A developer wants to dismiss a
finding in their own repository, have it stay dismissed, and have the reason reviewable beside the code — a
finding in their own repository, have it stay dismissed, and have the reason reviewable beside the code; a
tool that cannot be told it is wrong is one they stop running.
This feature is bounded by a rule the project has held from the start and a leak that shows why. An optimiser
may evolve **the question** — the judge and residual prompts, which are how we ask. It may never author **the
facts** — tables, rulesets, the declared policy file, which are what we claim. On 2026-09-05 the improve lever
may evolve **the question**: the judge and residual prompts, which are how we ask. It may never author **the
facts**: tables, rulesets, the declared policy file, which are what we claim. On 2026-09-05 the improve lever
admitted candidate shapes because they "recover a currently missed holdout pair"; five of eleven were taught by
the holdout pairs they then recovered, and a holdout Youden of **−0.059 was reported as +0.059**.
@@ -36,11 +38,11 @@ recorded and my colleagues can see why.
#### Acceptance Criteria
1. The system shall read dismissals from a declarative file in the **target** repository, and shall never
write to it — the same rule the declared policy file already follows.
write to it, the same rule the declared policy file already follows.
2. A dismissal shall carry the site it concerns, the rung it was dismissed at, a reason, and an author.
3. The system shall reject a dismissal that carries no reason, because a suppression list without reasons is
a graveyard nobody can audit.
4. The system shall report a dismissed finding as dismissed, with its reason, rather than omitting it — so a
4. The system shall report a dismissed finding as dismissed, with its reason, rather than omitting it, so a
stale dismissal is visible rather than silent.
5. When a dismissal no longer matches any finding, the system shall say so, because a dismissal that has
stopped applying is either a fixed bug or a moved target and both are worth knowing.
@@ -52,20 +54,20 @@ cause is visible as a cause rather than as five separate bugs.
#### Acceptance Criteria
1. When a finding is contradicted — by a dismissal, by the judge with coverage, or by a declared contract —
1. When a finding is contradicted (by a dismissal, by the judge with coverage, or by a declared contract)
the system shall record the site, the rung, the witness, the spec tokens that produced it, and the commit.
2. Each ledger entry shall be runnable as a regression case: this site, at this commit, shall not be reported
at that rung.
3. The ledger's regression cases shall run offline in CI, so that re-introducing a fixed cause fails a test
rather than a repository scan four weeks later.
4. The system shall group ledger entries by what they share — spec token, matcher clause, family, witness
shape — and report the groups by size.
4. The system shall group ledger entries by what they share (spec token, matcher clause, family, witness
shape) and report the groups by size.
5. The ledger shall be committed evidence, not a local artifact.
### Requirement 3: A model may propose a cause, and only propose it
**Objective:** As a maintainer, I want the model to do the part that needs judgement — naming what a group of
counter-examples has in common — without it deciding anything.
**Objective:** As a maintainer, I want the model to do the part that needs judgement (naming what a group of
counter-examples has in common) without it deciding anything.
#### Acceptance Criteria
@@ -130,7 +132,7 @@ quietly loses recall is worse than none.
1. The evaluation set shall include whole repositories, not pairs alone: the pair corpus could not see any of
the nine defects that motivate this feature, and the gates stayed byte-identical through every fix.
2. The system shall state how many repositories its evidence base contains, because two — one of which has
never produced a true positive — is very little to tune against.
2. The system shall state how many repositories its evidence base contains, because two (one of which has
never produced a true positive) is very little to tune against.
3. Where an adoption is measured on one repository only, the system shall label it as such rather than
# Brief: flow-aware-ranking — spend the expensive query where the cheap evidence is strongest
# Brief: flow-aware-ranking, spend the expensive query where the cheap evidence is strongest
> **Status 2026-10-02:** dormant. Brief only and unapproved; its 2026-09-12 amendment placed ranking inside the pre-push safety net, where the ranker now lives.
**Status:** brief only, unapproved.
## Current amendment — 2026-09-12
## Current amendment, 2026-09-12
This section supersedes forward-looking interpretations below; the iteration history is retained.
The product goal is a scalable, actionable pre-push security safety net as AI accelerates code changes.
@@ -21,7 +23,7 @@ need explicit summaries or unresolved coverage, never automatic vendor reintrodu
- Latest recorded phase-2 budgets: PMPro **82**, WP Statistics **91**, MW WP Form **54**, VAmPI **16**.
The `<50` gate remains unmet. The evidence passes recorded no unanswered pairs. These are stored
ranking measurements, not fresh detection/precision results or a pre-push latency demonstration.
- **Equal-score ties are not the whole remaining problem.** At fixed scores, perfect tie-breaking
- Equal-score ties are not the whole remaining problem. At fixed scores, perfect tie-breaking
still needs at least **68** regions for PMPro and **86** for WP Statistics (67/85 strictly
higher-scored same-tier regions precede the targets). An LLM restricted to equal-score ties
cannot pass `<50`. Within-tier reordering across scores or a new feature is a distinct experiment.
@@ -93,7 +95,7 @@ Three regions from one plugin, with what the ranker sees today and what is actua
| `pmpro_rest_api_get_membership_level` | **0.80** | none — no SQL sink at all |
A region with no sink outranks two holding raw concatenated SQL. The discriminating signal is in the sink
call's own source text — interpolated versus bound — and nothing in the ranker reads it.
call's own source text (interpolated versus bound), and nothing in the ranker reads it.
One detail decides whether a keyword rule suffices, and it does not: `esc_sql` IS in scope for the CVE's
function, applied to a different statement. "A sanitizer is present, rank it lower" demotes precisely the
@@ -120,8 +122,8 @@ finding**, it decides what is looked at, and `regions_truncated` already reports
Each phase ships only if it beats the previous one **on a repository the change was not developed against**.
A phase that cannot show that is not an improvement, it is a story.
- **Phase 0 — the oracle and the baseline.** The evidence extractor (no model) and a scoring harness over the
pinned CVEs, reporting each one's rank position. Gate: reproduces today's numbers — CVE-2023-23488 at
- Phase 0, the oracle and the baseline. The evidence extractor (no model) and a scoring harness over the
pinned CVEs, reporting each one's rank position. Gate: reproduces today's numbers, CVE-2023-23488 at
position 390 of 4,463, 83.8% of regions in one tier. Without a baseline nothing later is measurable.
**Done 2026-09-10.**`benchmarks/ranking/position.py`, committed as
@@ -136,36 +138,36 @@ A phase that cannot show that is not an improvement, it is a story.
**And this corrects the cross-assessment above.** "Ranking's job is concentration, not inclusion" was
measured on `pmpro` alone, where the CVE is inside the budget. On the other two plugins the CVE sits at
512 and 503 —**outside a 500-region budget** — so the current ranker never examines it at all, and the
512 and 503,**outside a 500-region budget**, so the current ranker never examines it at all, and the
slices found those CVEs only because a slice has too few regions to exclude anything. Both jobs are
real: inclusion on two repositories, concentration on all three. The single metric `budget_at_recall`
covers both, which is why it is the one that matters.
The instrument is `evidenceOnly` mode on the taint query, recorded on every scan as `ModelScanResult.tiers`
and `tier_counts`, and acted on by nothing — so every later phase is measured against a record made by
and `tier_counts`, and acted on by nothing, so every later phase is measured against a record made by
scans that did not know they were being measured.
**Measured cost of the instrument**, 637-file graph, the scan's own 500-region request set:
2,500 pairs in 98.9 s including JVM load — about 40 ms per pair
2,500 pairs in 98.9 s including JVM load, about 40 ms per pair
tier 0 (no sink of that family in reach): 2,161 of 2,500 = 86%
Two attempts before that ran past an hour each without producing a payload, and the cause was mine both
times: repository-wide walks — the sink scan, the reachable-method BFS, `familyInRepo` — repeated per
times: repository-wide walks (the sink scan, the reachable-method BFS, `familyInRepo`) repeated per
request. Memoising them across the batch is what made "milliseconds per pair" true rather than claimed.
**And the same memoisation reaches the dataflow mode.** The identical 250-request set that cost 2,026.3 s
(8,105 ms per request) with both joins on now costs **287.0 s — 1,148 ms per request — for the same 5,779
(8,105 ms per request) with both joins on now costs **287.0 s (1,148 ms per request) for the same 5,779
rows.** Most of what contributor-scan 5.13 measured as "the base taint query" was the per-request scans,
not the dataflow. That correction is recorded there.
and nothing more. 1,177 requests at 6.63 s is still 2.2 hours, so this phase does NOT make the scan finish;
it halves the problem. Claiming completion here would send someone looking for a bug where there is none.
**Done 2026-09-10, gate exceeded, scan still does not finish — and the reason corrects phase 0's cost
**Done 2026-09-10, gate exceeded, scan still does not finish, and the reason corrects phase 0's cost
figure.** On the 637-file plugin: `tier_counts = {0: 2161, 1: 60, 2: 20, 3: 211, 4: 48}`, **2,161 of 2,500
requests pruned (86%)**, evidence pass 94.5 s. The remaining **339 requests still hit the 2,400 s ceiling**
— more than 7 s each.
requests pruned (86%)**, evidence pass 94.5 s. The remaining **339 requests still hit the 2,400 s ceiling**,
more than 7 s each.
**The scan finishes, 2026-09-11.** Not because of anything in this spec: contributor-scan 5.13 found
that every query batch had been recomputing Joern's dataflow overlay at load since PHP builds went
@@ -186,12 +188,12 @@ A phase that cannot show that is not an improvement, it is a story.
which 88% were tier 0 and returned instantly; the ~30 real requests in it cost about 9.5 s each. Pruning
removes exactly the cheap questions and keeps the expensive ones, so it cannot make those faster. The
honest per-request cost of a taint question that *can* have an answer is **7–10 s at `callDepth = 3`**,
and that is the number phase 2 and 5.13 must work against — not 1.1 s, and not 6.6 s either.
- **Phase 2 — heuristic tiers, still no model.** The vector and tiers defined below, ordered within a tier
by a published weighted sum. Gate: `budget_at_recall`— the smallest K holding every pinned CVE — **below
and that is the number phase 2 and 5.13 must work against, not 1.1 s, and not 6.6 s either.
- Phase 2, heuristic tiers, still no model. The vector and tiers defined below, ordered within a tier
by a published weighted sum. Gate: `budget_at_recall`(the smallest K holding every pinned CVE) **below
50 on every pinned repository**, and `pmpro`'s taint query completing at that K. This is the phase that
makes a scan finish, because the job is concentration: the CVE is already inside 500, the question is
whether it is inside 50. **If this passes, the model is a refinement, not the plan** — and knowing that is
whether it is inside 50. **If this passes, the model is a refinement, not the plan**, and knowing that is
worth more than assuming it.
**Implemented 2026-09-11, measurement in flight.**`Evidence.score` is the published weighted sum, and
every weight is in one table in `model/evidence.py`: per open sink +1.0 (capped at 5), per cleansed or
@@ -302,17 +304,17 @@ A phase that cannot show that is not an improvement, it is a story.
facts and a weight table can do; the ties are phase 3's. After that, within-tier order is phase 3's -- a model reading the code the ties cannot --
or one more exact path fact (`distance`), and the gate stays unmet until one of them moves it.
- **Phase 3 — the LLM ranker.** Same evidence, ordered by a model. Gate: beats phase 2 leave-one-repository-out.
- Phase 3, the LLM ranker. Same evidence, ordered by a model. Gate: beats phase 2 leave-one-repository-out.
If it does not, phase 2 ships and phase 3 does not.
- **Phase 4 — break the closed loop.** An exploration slice (~10% of budget sampled outside the top-K) and
- Phase 4, break the closed loop. An exploration slice (~10% of budget sampled outside the top-K) and
full labelling on repositories small enough to arbitrate every region. Gate: a measured estimate of the
selection bias in phase 3's numbers.
- **Phase 5 — prompt optimisation (GEPA / SIMBA / DSPy).** Only after 3 and 4. An optimiser run against a
- Phase 5, prompt optimisation (GEPA / SIMBA / DSPy). Only after 3 and 4. An optimiser run against a
censored metric optimises the censorship.
## What the prompt has to carry
The question put to the model is **not** "is this vulnerable" — that is adjudication, it is the arbiter's job,
The question put to the model is **not** "is this vulnerable". That is adjudication, the arbiter's job,
and it is the one thing a ranker must never be asked. It is:
> Given a fixed budget of expensive dataflow queries, which of these sites are worth spending it on?
@@ -324,14 +326,14 @@ reachable from a request handler.
## Constraints, from measurement and from prior scars
- **Cost.** One model call per region is 4,463 calls a scan. Batch over a prefilter, or it does not ship.
- **The loop is closed.** A region the ranker excludes is never arbitrated, never labelled, so the ranker
- Cost. One model call per region is 4,463 calls a scan. Batch over a prefilter, or it does not ship.
- The loop is closed. A region the ranker excludes is never arbitrated, never labelled, so the ranker
never learns it was wrong. Every run confirms the ordering it already had.
- **N projects.** Four pinned CVEs is not a training set. This project already has the closed-loop
- N projects. Four pinned CVEs is not a training set. This project already has the closed-loop
train-on-test scar; the reported figure must come from a repository the optimiser never saw.
- **Determinism.** Two runs of one repository must agree or a baseline diff is meaningless. The ranking is
- Determinism. Two runs of one repository must agree or a baseline diff is meaningless. The ranking is
recorded in the manifest.
- **Degradation.** With no model configured, phases 0–2 still rank, and the report says which ranker ordered
- Degradation. With no model configured, phases 0–2 still rank, and the report says which ranker ordered
the scan.
## Cross-assessment against the measurements
@@ -340,7 +342,7 @@ Read against what was actually measured today, the plan above has two errors and
here, corrected, rather than silently rewritten, because the corrections are themselves results.
**Phase 1's gate is wrong.** It says pruning makes `pmpro`'s taint query complete. It does not. Pruning
removes 53% of requests, and 1,177 × 6.63 s is still **2.2 hours** — nowhere near the 300 s default or the
removes 53% of requests, and 1,177 × 6.63 s is still **2.2 hours**, nowhere near the 300 s default or the
2,400 s that was tried. Pruning halves the problem; it does not solve it. The honest gate is "53% fewer
requests issued, measured", and completion belongs to phase 2. Stating otherwise would have sent someone
looking for a bug in phase 1 when phase 1 had done exactly what it can.
@@ -354,8 +356,8 @@ the numbers it had; the numbers were the instrument's.
**Ranking's job is concentration, not inclusion.** The brief implies the CVE is missed because the ranker
excluded it. It did not: CVE-2023-23488's region sits at position 390 of 4,463, inside the 500-region budget,
and was examined. It goes unreported because the budget is spent on 500 regions when 50 would do, and 500
is unaffordable at 6.63 s per family per region. So the ranker's success metric is not "is the CVE in the
budget" — it already is — but **"how small can the budget be while the CVE stays in it"**. That is a
is unaffordable at 6.63 s per family per region. So the ranker's success metric is
**"how small can the budget be while the CVE stays in it"**; the CVE is already in the budget. That is a
different objective, and it is the one that makes a scan finish.
**The gap: nothing above says what the evidence IS.** "Sink shape" and "cleansing on the call" are named,
@@ -364,24 +366,24 @@ check for language leakage. The rest of this brief defines it.
Two things the measurements confirm and the plan should lean on harder:
- **The joins are cheap and must survive any ranker.** 22% of query cost, two of three CVEs. A ranker that
- The joins are cheap and must survive any ranker. 22% of query cost, two of three CVEs. A ranker that
demotes field- or hook-carried regions because they show no direct source is a ranker that loses those
CVEs. The evidence vector below therefore carries the join's own stage-one result as a field.
- **84% of regions in one tier means any separation is progress**, and the separation does not need to be
- 84% of regions in one tier means any separation is progress, and the separation does not need to be
clever to be large. A tier 0 that is merely exact already reorders the budget more than the current ranker
ever has.
## The evidence vector, defined
The unit is a **(region, family) pair**. For each, a fixed vector of fields is computed. Every field is
defined in terms of **fact-table categories**— source, sink, sanitizer, dispatch, access — never in terms of
defined in terms of **fact-table categories**(source, sink, sanitizer, dispatch, access), never in terms of
a language's syntax, which is what makes the definition language-agnostic: any language with a fact table
gets the same vector, and a heuristic that mentions `$wpdb` or `wp_ajax_` has by construction left the
vector and must be rejected in review.
**Where it is computed matters as much as what it is.** The graph is already built for the arbiter, at 76 s
for 637 files. Extracting these fields is a CPG query **without dataflow**— call nodes, their arguments,
their enclosing methods, the call graph one hop out — which is milliseconds per call rather than 6.63 s per
for 637 files. Extracting these fields is a CPG query **without dataflow**(call nodes, their arguments,
their enclosing methods, the call graph one hop out), which is milliseconds per call rather than 6.63 s per
request. The extractor is therefore a sibling of `taint.sc` that never calls `reachableByFlows`:
> **Tiering is the CPG query without dataflow. Arbitration is the CPG query with it. Same graph, same
@@ -391,7 +393,7 @@ That is the whole architecture in one line, and it is why the evidence is langua
merely language-neutral-in-intent: the CPG normalises syntax across frontends, so "the argument of this call
is a literal" means the same thing in PHP, Python and C.
### Sink evidence — from `[[sink]]` facts of family F
### Sink evidence, from `[[sink]]` facts of family F
| field | definition | exactness |
|---|---|---|
@@ -399,7 +401,7 @@ is a literal" means the same thing in PHP, Python and C.
| `shape` | per sink call, one of: **`literal`** (the vulnerable-position argument is a constant), **`bound`** (the call is a declared-safe form — a `parameterized` sanitizer fact wraps it, or its arity/shape matches the bound form the taint query already recognises), **`computed`** (the argument is an expression: concatenation, interpolation, a variable, a call result) | exact for `literal` and `bound`; `computed` is the residue |
| `cleansed_on_call` | per sink call, whether a sanitizer of F is an **ancestor of the vulnerable-position argument in the AST** — not merely present in the same function | exact, and this is the field a text scan gets wrong: `esc_sql` in scope on a different statement is `false` here |
### Source evidence — from `[[source]]` facts and the entry-point contract
### Source evidence, from `[[source]]` facts and the entry-point contract
| field | definition | exactness |
|---|---|---|
@@ -409,7 +411,7 @@ is a literal" means the same thing in PHP, Python and C.
| `access` | the declared access level, **with provenance**: `declared` (a registration or route contract says so) or `inferred` (nothing found) — because "public" meaning "no decorator found" is the false positive the project has already paid for | exact |
| `carried` | the region reads a **field** whose assignment is fed by a source anywhere in the file, or a **dispatch** value whose registered producer is — i.e. stage one of the two-stage join, which is already computed and memoised, reused here as evidence | exact where computed; absent (not false) where the join was not run |
### Path evidence — cheap structural proxies for what dataflow would establish
### Path evidence, cheap structural proxies for what dataflow would establish
| field | definition |
|---|---|
@@ -436,14 +438,14 @@ is a literal" means the same thing in PHP, Python and C.
Two properties are load-bearing and must hold in every implementation:
1. **Only tier 0 excludes; every other tier orders.** So the only place exactness is *required* is tier 0,
1. Only tier 0 excludes; every other tier orders. So the only place exactness is *required* is tier 0,
and tier 0's predicate has a proof: a family with no sink in reach cannot produce a flow, in any language,
under any dataflow model. Above tier 0, being wrong costs position, and `regions_truncated` reports what
position cost.
2. **Tier 0's scope must match the arbiter's.** At `callDepth = 0` the scope is the region; at depth k it is
2. Tier 0's scope must match the arbiter's. At `callDepth = 0` the scope is the region; at depth k it is
the region plus its reachable methods. A tier 0 computed on the file when the arbiter will look k hops
out is not exact and must not be called so. The repository-wide form (`family_in_repo == false`) is exact
at every depth and is the cheap first cut — it alone removes all 487 `deserialization` requests on `pmpro`.
at every depth and is the cheap first cut. It alone removes all 487 `deserialization` requests on `pmpro`.
**Within a tier**, order by a score. In phase 2 that score is a fixed weighted sum over the vector, published
in the manifest so two runs agree. In phase 3 it is the model's ordering **of the same vector**, which is
@@ -454,23 +456,23 @@ what makes the two comparable: same input, different ordering function, one held
The vector above is exact wherever exactness was cheap. What remains genuinely uncertain is narrower than
"ranking":
- **Ambiguous cleansing.**`cleansed_on_call` is exact when the sanitizer wraps the argument directly. It is
`false`— correctly, but unhelpfully — when cleansing happens two statements earlier into a variable that is
- Ambiguous cleansing. `cleansed_on_call` is exact when the sanitizer wraps the argument directly. It is
`false`(correctly, but unhelpfully) when cleansing happens two statements earlier into a variable that is
then interpolated. Deciding whether *that* cleansing reaches *this* call is reading the code, and it is the
single largest source of tier-2/tier-3 misplacement.
- **Unfamiliar shapes.** A sink call whose form the `bound` rule does not recognise, in a framework with no
- Unfamiliar shapes. A sink call whose form the `bound` rule does not recognise, in a framework with no
facts yet. The model can say "this looks bound" as an *ordering* opinion; it cannot add a `bound` fact,
because a fact is policy.
- **Within-tier priority** when the fixed score ties, which on a real repository it will, constantly.
- Within-tier priority when the fixed score ties, which on a real repository it will, constantly.
And the boundaries:
- **The model never computes tier 0.** Exclusion is exact or it is a silent miss, and a model's "no sink
- The model never computes tier 0. Exclusion is exact or it is a silent miss, and a model's "no sink
here" is not a proof.
- **The model never sees the question "is this vulnerable".** It sees "which of these is worth an expensive
- The model never sees the question "is this vulnerable". It sees "which of these is worth an expensive
query", with the vector and the sink call's source text. Adjudication is the arbiter's, and keeping it
there is what keeps the gates byte-identical.
- **The model's input is the vector plus text, never the arbiter's verdict** on the same run — that is the
- The model's input is the vector plus text, never the arbiter's verdict on the same run. That is the
closed loop feeding on itself.
## Metrics, so every phase can fail
@@ -496,8 +498,8 @@ answer is visible:
1. ~~Where does the evidence extractor live?~~ **It is a CPG query without dataflow, a sibling of `taint.sc`
in `cpg/queries/`.** Not the mapper: the mapper reads text and would have to re-derive per language what
the graph already normalises. The vector is language-agnostic *because* the graph is.
2. ~~Per file or per reachable set?~~**Both, in that order.**`family_in_repo` first — exact at any depth,
free, a fifth of the budget on `pmpro` — then the reachable set at the arbiter's own `callDepth`. Never the
2. ~~Per file or per reachable set?~~**Both, in that order.**`family_in_repo` first (exact at any depth,
free, a fifth of the budget on `pmpro`), then the reachable set at the arbiter's own `callDepth`. Never the
file alone above depth zero, because that is not exact and must not be called tier 0.
3. How many pinned repositories before leave-one-out means anything? Four is not enough. This is the
corpus ask, and it is the same ask contributor-scan 5.8 made: plugins chosen by installed base, CVEs read
@@ -505,7 +507,7 @@ answer is visible:
4. Does the arbiter's verdict feed back as a label? **Only through the exploration slice.** A verdict on a
region the ranker chose is a label the policy selected, and training on it closes the loop. A verdict on
an exploration region is the only unbiased label the scan produces, and it is the one worth keeping.
5. **New:** the weighted sum in phase 2 has weights. Where do they come from before any model exists?
5. New: the weighted sum in phase 2 has weights. Where do they come from before any model exists?
Proposed: set by hand from the tier semantics, published in the manifest, and treated as a baseline for
phase 3 to beat — not tuned, because tuning four weights on four CVEs is the overfit the plan exists to
phase 3 to beat, not tuned, because tuning four weights on four CVEs is the overfit the plan exists to
| Composition seam | `make_docgen` in `docuharnessx/bundle.py` — the only HarnessX import site; `control \| stages_builder()`; returns model-less `HarnessConfig` | `build_sast()` in `harness/build_sast.py` (mirrors `examples/coding/build_coding`), lazy behind `_has_harnessx()` | **shared** |
| Model binding | CLI `prepare_run()`: `model_config.agentic(make_docgen(...))`; model on `harness.model_config.main`, never in config | `HxScanOrchestrator`: `asyncio.run(model_config.agentic(cfg).run(ScanTask(...)))` | **shared** |
| Processor base | `NoOpStage(MultiHookProcessor)`, content-neutral on `step_end`, real module-level classes for `_target_` resolution | `MultiHookProcessor(skip_model=True)` + `SlotContractMixin`, real module-level classes | **shared** |
(drops playwright + browser binary). Net ≈ 40–60% fewer packages, no browser binary.
## Residual risks (security-tool-specific) and mitigations
| Risk | Mitigation |
|---|---|
| `playwright install` fetches a Chromium binary out-of-band (unaudited, breaks hermetic builds) | Browser path is lazy and never invoked; document that `[harnessx]` does not trigger a browser download; lean subset deletes `browser.py` |
| `docker` SDK can reach the Docker daemon socket (privilege boundary) | Docker sandbox **default-off**, `LocalSandbox` default; run untrusted target code only inside the sandbox; SHA-pin docker/cryptography/cffi |
| FastAPI/uvicorn listening-socket surface in a security tool | `api/` server only starts on the explicit `serve` path; add a guard test asserting no server starts on import; lean subset removes `api/` |
| 118-package CVE surface in a supply-chain tool | SHA-pin the entire extra; keep core **zero-dep**; add a **zero-dep CI guard test** (importing `openultrasast` without the extra pulls nothing outside stdlib) |
| `recipe/verl` git submodule recursion | Init non-recursively; never in the accept-as-extra install path |
> Decision date 2026-06-22. The `pip install --dry-run` audit is itself the Phase-3 gate: it is
> re-run before the packaging task (5.2), and if the realized graph pulls a browser binary /
> Docker SDK / FastAPI stack as *mandatory* (non-lazy) deps judged unacceptable, the spec switches
> to the vendored lean subset under the same extra name.
**Purpose.** This feature splits OpenUltraSAST's single overloaded regex layer — which today is detector, severity authority, and volume knob at once — into three governed owners: **rules detect**, a **central CWE policy decides severity and scope**, and a **single project score** becomes the optimization target. On top of that governance plane it adopts the HarnessX runtime exactly where it pays off — the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds.
**Users.** OpenUltraSAST maintainers gain a staged, reversible HarnessX adoption and an automatic, gated rule/policy tuning loop. Security policy owners gain a versioned, central CWE-to-severity policy. Security reviewers and CI consumers gain a bounded 0-100 AppSec project score with a two-condition gate.
**Impact.** Today `findings.PatternRule` carries a string `severity`, `quick_scan_findings` reads a module-level `PATTERN_RULES`, `calibration.py` only demotes ranking (floored at `MIN_CALIBRATED_PRIORITY = 0.1`, keyed on `tags[0]`), `benchmark.py:368` writes an inert `next_improvement_candidate="rules_or_language_hunter"`, and `harness.HarnessRuntime.run_stage` is a sync dict-state driver. This design turns rules into TOML data, removes `PatternRule.severity`, sources severity from a vendored verycode `CWE_Score.tsv`, computes a project score, makes the benchmark actionable, and confines HarnessX behind an optional, SHA-pinned, lazy-imported extra — while `dependencies = []` and `requires-python = ">=3.11"` stay intact and the **≥90% recall / <10% FP** detection gate holds at every phase.
### Goals
- Rules become versioned data (`ruleset/<lang>/*.toml`) with a loop-controlled `status` and no rule-local severity.
- One CWE = one governed severity, sourced from `policy/CWE_Score.tsv` via `resolve_severity()`.
- A single 0-100 `project_score` from `policy.severity × reachability`, with a two-condition CI gate.
- HarnessX adopted at the hunter loop, the verifier, and the meta-harness, behind `openultrasast[harnessx]`.
- A bounded self-improvement loop (`rule`, `policy` levers) that auto-reverts byte-for-byte under the gate.
- The governance + scoring + benchmark planes stay **zero-dependency** and **byte-identical by default** through Phases 0-2.
### Non-Goals
- The loop authoring or rewriting detection pattern text (human PR only).
- The loop editing the authoritative 0-5 CWE severity (verycode-upstream authority).
- A full single-step HarnessX rewrite of all deterministic stages and weight-level RL (deferred to optional Phase 5+).
- Changing the reachability vocabulary, which stays exactly `reachable | inferred-file-surface | unknown`.
- Dynamic/DAST scoring — report-only for a static tool.
## Boundary Commitments
### This Spec Owns
- The **RulesetStore** (`ruleset/store.py` + `ruleset/<lang>/*.toml`): `load_ruleset()`, the `PatternRule` schema (detection-only), and the `enabled|shadow|disabled` status vocabulary.
- The **PolicyStore** (`policy/verycode.py` + `policy/CWE_Score.tsv`): `load_policy()`, `CwePolicy`, `resolve_severity()`, and the CWE-120→CWE-121 remap.
- The **ScoreModel** (`scoring/project_score.py`): `SEV_WEIGHT`, `REACH_MULT`, `finding_penalty`, `project_score`, the score artifact, and the two-condition CI gate.
- The **loop ledger** (`.openultrasast/calibration/rule_policy.json`): the loop-owned overlay of `status`/evidence-floor/threshold/precision.
- The **HarnessX seam**: `harness/build_sast.py`, `HxScanOrchestrator`, the `MultiHookProcessor` SAST processors, the `SlotContractMixin`, and the `improve/` meta-harness driver.
- The **benchmark→signal** conversion: `rule_id` attribution, per-rule precision/recall, `rule_policy_delta`, and `rule_signals.json`.
### Out of Boundary
- The detection-pattern **content** of rules (authored by humans via PR; the loop may only toggle status / raise floors).
- The authoritative **0-5 severity** numbers in `CWE_Score.tsv` (verycode-upstream; PR gaps there).
- **Stdlib only** for the governance/scoring/benchmark planes: `tomllib`, `csv`, `json`, `math`, `dataclasses`.
- **HarnessX** only behind the `[harnessx]` extra, pinned to a commit SHA, imported lazily behind `_has_harnessx()`.
- The existing `HarnessRuntime` as the top-level sync driver through Phases 0-4 (not removed by this spec until Phase 5).
- The existing reachability vocabulary emitted by `findings.py` (consumed, never redefined).
### Revalidation Triggers
- **Contract shape changes** to `PatternRule` (removing `severity`), `CwePolicy`, `VerificationResult`, `BenchmarkMiss`/`BenchmarkFalsePositive` (adding `rule_id`), or the `project_score` artifact JSON.
- **Data ownership change**: severity moving from rule → policy; calibration re-keying from `tags[0]` → `rule_id`.
- **Dependency direction change**: any new import of HarnessX outside a lazy `_has_harnessx()` guard; any addition to core `dependencies`.
- **Runtime prerequisite change**: the startup CWE-resolution invariant (fail-loud if an enabled rule's CWE is unmapped); the `pip install --dry-run` audit gate before Phase 3.
- **Lever surface change**: any addition to `journal._VALID_LEVERS` or the `EvolveValidator` bounds.
## Architecture
### Existing Architecture Analysis
| Existing element | File | Constraint to respect / debt to fix |
|---|---|---|
| `HarnessRuntime.run_stage(name, callback)` | `harness.py:74` | Sync, dict `state` (`{"degradations": []}`), single `HarnessEvent` (`scan_id`), `ProcessorSpec.reads/writes` allow-list. Incompatible with HarnessX async-generator + message-window contract — **bridge, do not map**. Stays as the top-level driver through Phase 4. |
| `PatternRule.severity: str` + module-level `PATTERN_RULES` | `findings.py:31,34` | The fused defect: regex carries its own severity string ("high"/"medium"/"low") and is the only volume control. **Remove `severity`; externalize patterns to TOML.** |
| `calibrate_rankings` demotion | `calibration.py:146` | `max(MIN_CALIBRATED_PRIORITY, calibrated - 0.5)`, keyed on `finding.tags[0]` (`calibration.py:262`). Demotes ranking, never the rule. **Re-key to `rule_id`; write status flips to ledger.** |
| `verify_findings` rejects on `reachability_status == "unknown"` | `verification.py:91` | Structural verifier. Keep as **zero-dep fallback**; add LLM-judge path. |
| `benchmark.py:368``next_improvement_candidate="rules_or_language_hunter"` | `benchmark.py` | Inert stub written to `calibration_records.json`. **Make actionable** via `rule_id` attribution + `rule_policy_delta`. |
| `cli.py``run_stage(...)` call chain | `cli.py:104-145` | The 13-stage sync pipeline. Stages migrate to HarnessX hooks incrementally; the hunter pool first. |
### Architecture Pattern & Boundary Map
**Selected pattern.** Three planes with one-directional authority: **governance data drives execution; execution emits signals; the improvement loop edits governance under a gate.** Nothing in execution mutates governance directly. The governance + scoring planes are **pure stdlib, zero HarnessX** (non-negotiable, so CI gating runs in the default install). HarnessX lives only in the execution and improvement planes, behind the extra.
**Boundaries.** RulesetStore detects, PolicyStore decides severity/scope, ScoreModel optimizes — three owners, three change channels (human PR for patterns/severity; loop-via-`EvolveValidator` for status/floors/score-constants).
**New components rationale.** PolicyStore/ScoreModel/RulesetStore exist to break the fused regex defect; `HxScanOrchestrator`/`VerifyProcessor` exist to host the agent loop and judge; `improve/evolve.py`+`EvolveValidator` exist to make adaptation safe (bounded, gated, reversible). **Steering compliance:** zero-dep default install preserved; detection gate kept as a hard constraint separate from the score reward.
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
- `calibration.py` — re-key from `tags[0]` (line 262) to `rule_id`; write `rule_policy.json` status flips, not only ranking demotion (line 146).
- `benchmark.py` — add `rule_id` to `BenchmarkMiss`/`BenchmarkFalsePositive`; per-rule precision/recall in `BenchmarkMetrics`; emit `rule_policy_delta`; replace the inert stub at line 368; add `BenchEvaluator` subclass.
- `verification.py` — add LLM-judge path (`VerifyProcessor` over `llm_judge` helpers); keep structural verifier (`reachability_status == "unknown"` reject, line 91) as zero-dep fallback.
The top-level driver stays sync (`HarnessRuntime`); only the `hunter_pool` stage opens an async surface via `asyncio.run`. Deterministic stages run as `skip_model=True` processors carrying work in typed slots; the `SlotContractMixin` validates `reads_slots`/`writes_slots` on every hook (preserving the `ProcessorSpec.reads/writes` discipline). If a HarnessX capability is unavailable at runtime, the stage falls back to its existing equivalent and records a degradation in the manifest.
J -->|no| K2["VerifyProcessor: llm_judge build/parse/call<br/>(structural fallback if extra absent)"]
K --> L["record_calibration: lesson/missing -> FalsePositiveLearning<br/>+ per-rule outcome -> rule_policy.json"]
K2 --> L
L --> M["on_task_end: ScoreProcessor -> project_score + gate"]
M --> N["report / sarif / manifest (incl. score block)"]
```
### Self-improvement loop
The benchmark is the **regression suite governing the data stores**. The objective is `maximize project_score subject to recall ≥ 90% AND FP < 10%` as a hard feasibility constraint — kept separate so the loop cannot "improve" the number by shadowing rules that hurt real security. Every change is `shadow`-staged first, novelty/evidence/replay-gated, and reverted byte-for-byte on any failure.
```mermaid
sequenceDiagram
participant B as BenchmarkHarness
participant O as improve/evolve.py
participant M as MetaAgent.evolve()
participant V as EvolveValidator (_rules/_policy)
participant J as journal (compute_attribution/fill_gating)
| 1.1, 1.4 | Hunter pool on real `Harness.run()`; async confined to one stage | `HxScanOrchestrator` | `run.py:HxScanOrchestrator.run()` | Scan pipeline |
- `status ∈ {enabled, shadow, disabled}` is the volume control; the loop overlays `status`/`min_evidence_level`/`precision_estimate` from `rule_policy.json`.
- `load_policy() -> dict[str, CwePolicy]` keyed `"CWE-NNN"` from `CWE_Score.tsv` (167 CWEs; severity column header is literally `"Flaw Severity "` with a trailing space — parse defensively).
- `resolve_severity(policy, rule.cwe, target, ranking)` replaces every literal `rule.severity` read. **Policy always wins**; any legacy rule severity string is discarded.
- **Startup invariant (fail-loud):** every `enabled` rule's `cwe` must resolve in PolicyStore, else the scan aborts. `dynamic`-only CWEs are report-only.
def assert_rules_resolve(rules, policy) -> None: # raises at startup if an enabled rule's CWE is unmapped
...
```
- **Invariants:** one CWE ⇒ one severity; two rules for the same CWE always receive identical severity.
### Scoring Layer
#### ScoreModel
| Field | Detail |
|-------|--------|
| Intent | Compute a 0-100 project score and the two-condition gate verdict |
| Requirements | 5.1-5.9 |
**Responsibilities & Constraints**
- Pure stdlib (`math`), always on, zero-dep — runs in the default install with no HarnessX.
- Excludes `shadow`-status findings before computing; assigns 0 penalty to dynamic-only/unmapped CWEs (reports them instead).
- The `REACH_MULT` keys are exactly the three values `findings.py` emits; the reachability multiplier is the **FP-calibration knob** (a confirmed FP lowers effective reachability, never deletes a rule).
- Emits the scan-level artifact and the manifest score block; computes the hard gate (sev5 + reachable) and the score gate (advisory-first).
def gate(findings, score, *, min_score: int, block_severity_reachable: int = 5,
blocking: bool) -> dict: ...
```
- **Postconditions:** artifact JSON (see Data Models) written next to per-finding output and merged into the manifest.
### Execution Layer
#### HxScanOrchestrator
| Field | Detail |
|-------|--------|
| Intent | Run the hunter pool on a real HarnessX agent loop, confining async to one stage |
| Requirements | 1.1, 1.4, 1.7, 1.8 |
**Responsibilities & Constraints**
- Invoked from the existing `hunter_pool``run_stage` callback (`cli.py:122`); wraps `asyncio.run(model_config.agentic(harness_config).run(ScanTask(...)))`.
- `build_sast()` mirrors `examples/coding/build_coding`; `ModelConfig(main=hunter, judge=…, ranker=…, patcher=…).agentic(cfg)` keeps model info out of `HarnessConfig`.
- Produces a `StatefulTrajectory` per target; rewards backfilled from verifier verdicts. On capability-missing, falls back to `run_hunter_pool()` and appends to manifest `degradations`.
- Hook mapping: `stage_start`→`on_step_start`, `stage_end`→`on_step_end`, `scan_id`→`session_id`/`run_id`; `before/after_model|tool` direct. `HARNESSX_CONTRACT_MODE` (message-window) + `OUS_SLOT_CONTRACT` (key-level via `SlotContractMixin`) split the old single `contract_mode`.
#### PolicyScoringProcessor
| Field | Detail |
|-------|--------|
| Intent | Resolve policy severity per finding and make the emission decision |
- Drops `shadow` findings from the reported set and the score input while recording their outcome for precision tracking.
- Applies `min_emit_priority`/`min_emit_precision` thresholds — the new volume controls that replace "every regex match becomes a finding".
**Contracts**: Event [x]
#### VerifyProcessor
| Field | Detail |
|-------|--------|
| Intent | LLM-judge verification with a zero-dep structural fallback |
| Requirements | 1.2 |
**Responsibilities & Constraints**
- Wraps `build_judge_prompt`/`parse_judge_response`/`_call_judge` (`llm_judge.py`) as a `judge`-role sub-harness; maps the strict-JSON verdict (`plausible|unsupported|hedging|…` + `lesson`/`missing`) to `VerificationResult`.
- **Fallback:** the existing structural verifier (`verification.py:91`, rejects on `reachability_status == "unknown"`) is used verbatim when the extra is absent or the judge is unavailable.
**Contracts**: Event [x]
### Improvement Layer
#### EvolveValidator (`_rules` / `_policy`)
| Field | Detail |
|-------|--------|
| Intent | Enforce the bounded-lever safety contract on every loop edit |
| Requirements | 3.2-3.8 |
**Responsibilities & Constraints** (the gate — runs in the *validity* phase beside `_contract`/`_replay`)
- `_rules`: edit parses; `status ∈ {enabled,shadow,disabled}`; `tighten`/`loosen` adjusts only threshold/evidence-floor **within configured bounds**; `add` doesn't duplicate a `rule_id`; **every `cwe` resolves in PolicyStore**; **pattern text unchanged** (reject otherwise); **no `enabled→disabled` jump** (require `shadow`).
- `_policy`: touches only evidence floor, scope, `K`/`MIN_SCORE` within bounds; **never the 0-5 severity**.
- On any failure: `raise StrictValidationError(kind="rule_change")` → the whole round is rejected.
| Intent | Drive benchmark→signal→evolve→re-benchmark→accept/revert under the gate |
| Requirements | 4.1-4.8, 6.3 |
**Responsibilities & Constraints**
- Converts `rule_policy_delta` + FP learnings → `rule_signals.json` (recall side `signal="miss"`; precision side `signal="verifier_reject"|"fp"`); calls `MetaAgent.evolve()`.
- **Acceptance gate (hard):** round accepted **only if** aggregate recall ≥ 90% **AND** FP < 10% **AND**`project_score` did not regress on the held set **AND** no unpredicted regression; otherwise `fill_gating(outcome="reverted")` restores config byte-for-byte.
- Precision-dragging rules are **auto-shadowed**, never deleted; coverage gaps nominate a rule via `rule_signals.json` (pattern authoring stays human-PR).
- Objective: `maximize project_score s.t. recall/FP gate` — score is the reward, the gate is the feasibility region; never folded together.
status = "enabled" # enabled | shadow | disabled (loop-controlled)
min_evidence_level = "static_corroboration" # loop may raise within bounds
precision_estimate = 0.62 # backfilled from benchmarks
version = "3"
```
Status semantics: `enabled` fires/scores/reports; `shadow` fires and records outcomes but is excluded from report and score (mandatory staging state for any loop-proposed change); `disabled` does not fire.
Loop ledger (`rule_policy.json`) — loop-owned overlay, re-keyed on `rule_id`:
- **Authority direction:** governance data is the source of truth; execution reads it, emits findings/signals; only the loop (via `EvolveValidator`) writes `status`/floors/score-constants back, never the pattern or the 0-5 severity.
- **Temporal/versioning:**`PatternRule.version` and `rule_policy.json.round` give per-rule audit; the journal records each lever edit; `fill_gating(reverted)` restores byte-for-byte.
- **Serialization (resume-safe):** all scan slots (`sast.targets`, `sast.findings`, `sast.rankings`, verifications) use plain dicts/dataclasses with `_target_` registration so they survive `wake()`.
## Error Handling
### Error Strategy
| Error type | Condition | Response |
|---|---|---|
| **Startup config error** | An `enabled` rule's `cwe` is unmapped in PolicyStore | `assert_rules_resolve` raises; **fail loud**, abort scan before any work (Req 2.7) |
| **Malformed ruleset/policy TOML** | `tomllib` parse failure | Fail loud at load; CI catches before merge |
| **Acceptance-gate failure** | recall<90% OR FP≥10% OR score regressed OR unpredicted regression | `fill_gating(reverted)` restores config byte-for-byte (Req 4.5) |
| **HarnessX capability missing at runtime** | Extra absent or sub-harness unavailable | Stage falls back to existing equivalent; record degradation in manifest (Req 1.8) |
| **Slot contract violation** | Stage reads/writes an undeclared slot | `SlotContractMixin` raises under `OUS_SLOT_CONTRACT=strict`; warns under `warn` (Req 1.5) |
| **Slot serialization loss on resume** | Non-`_target_` slot restores as `None` | Per-slot-type round-trip test guards; `_target_` registration required |
- The score artifact `unmapped_cwe` list surfaces policy-completeness drift; the journal changeset table surfaces every rule/policy lever edit per round.
## Testing Strategy
### Unit Tests
- `load_policy()` parses the `"Flaw Severity "` trailing-space header and yields 167 `CwePolicy` entries with correct `static`/`dynamic` flags.
- `finding_penalty()` returns 0.0 for dynamic-only and unmapped CWEs.
### Integration Tests
- **Byte-identical baseline (Req 6.2):** with the default ruleset and an empty `rule_policy.json`, `benchmark.py` output matches the pre-feature golden corpus exactly.
- **Startup invariant (Req 2.7):** an enabled rule with an unmapped CWE aborts the scan with a loud error.
- **Self-improvement round (Req 4.4/4.5):** a deliberately noisy rule is auto-shadowed when it drags precision; a round that regresses recall is reverted byte-for-byte.
- **Capability fallback (Req 1.8):** extra absent → `quick`/`standard` run through the sync driver unchanged; degradation recorded when a HX stage is requested but unavailable.
- **Slot round-trip (Req on resume):** each scan slot type serializes and restores without loss after `wake()`.
### E2E / Gate Tests
- **CI detection gate (Req 6.1):** the pipeline fails the merge if recall<90% or FP≥10% on the corpus.
- **Two-condition score gate (Req 5.7/5.8):** hard gate fails on a sev5+reachable finding; score gate is advisory until `K`/`MIN_SCORE` are calibrated, then blocking.
- **Zero-dep guard (Req 7.5):** with HarnessX absent, governance/scoring/benchmark planes operate and CI gating runs in the default install.
### Performance / Tolerance
- **HX tolerance (Req 6.4):** an enabled HarnessX-backed stage keeps recall/FP within the configured tolerance of the zero-dep baseline.
- Hunter-loop cost/token budgets enforced by `CostGuardProcessor`/`TokenBudgetProcessor`.
## Security Considerations
- **Supply-chain posture (Req 7):** core `dependencies = []` is preserved; HarnessX ships only as a SHA-pinned optional extra, lazy-imported behind `_has_harnessx()`. A `pip install --dry-run` audit runs before Phase 3; if the realized transitive graph pulls browser binaries / Docker SDK / a FastAPI web stack as *mandatory* deps and maintainers judge it unacceptable for a supply-chain auditing tool, the pre-specified **vendored lean subset** (Option C) is switched in under the same extra name — a packaging change, not a redesign. Submodule init is **non-recursive** so the large `verl` RL fork is never pulled.
- **Loop containment:** the meta-agent can never invent or rewrite detectors (no pattern edits) or rewrite the severity authority (no 0-5 edits); `shadow`-first staging, novelty/evidence/replay gates, and byte-for-byte revert bound its blast radius — directly preventing the "too influential" defect from re-entering via the loop.
- **Gate integrity:** the recall/FP gate is a hard constraint that is never folded into the score reward, so the loop cannot "improve" the number by shadowing rules that hurt real security.
## Migration Strategy
Every phase is independently shippable, test-gated, and reversible. Phases 0-2 deliver governance value at **zero dependency cost** and stay byte-identical by default. Phases 3-4 are the HarnessX adoption behind the extra. Phase 5 is the optional native end-state.
| **5 — Full harness + RL (optional)** | Migrate deterministic stages to single-step harness + `SlotContractMixin`; retire `HarnessRuntime`; promote score gate to blocking; optional `rl/`. | **opt-in** | golden-diff vs Phase 3 empty; gate held | per-stage revert |
**Rollback triggers.** Any phase that drops recall<90% or raises FP≥10% on the corpus blocks the merge; any self-improvement round failing the acceptance gate reverts byte-for-byte; an unacceptable `pip --dry-run` graph switches Phase 3 to the vendored subset.
OpenUltraSAST today fuses three responsibilities into one regex layer: detection patterns also carry their own severity strings and act as the sole volume control, so a noisy rule can dominate scan output, set its own criticality, and fire forever with no central governance. This feature splits those responsibilities into three governed owners — rules detect, a central CWE policy decides severity and scope, and a single project score becomes the optimization target — and adopts the HarnessX runtime exactly where it pays off: the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds.
The work spans five user asks. First, OpenUltraSAST shall adopt HarnessX directly at the hunter loop and migrate the existing scan flow stage-by-stage onto HarnessX processors. Second, detection rules shall become versioned data governed centrally, with severity sourced from the verycode CWE policy rather than from the rule itself. Third, the HarnessX meta-harness shall adapt rules and policy gates within bounded levers. Fourth, benchmark results shall drive that adaptation as actionable signals with a hard acceptance gate. Fifth, a Veracode-style project score shall be computed from policy-weighted, reachability-adjusted findings and gated in CI.
Two cross-cutting constraints govern every part of this feature. The existing detection gate of at least 90% recall and under 10% false-positive rate on the benchmark corpus must never regress, at any phase, for any change. And the HarnessX runtime must be adopted in a way that preserves the project's zero-dependency posture for the governance and scoring planes, so an explicit dependency-weight decision controls whether HarnessX ships as an optional extra or as a vendored lean subset.
## Boundary Context
- **In scope**: detection rules as versioned data; a central CWE-to-severity policy with static/dynamic scope; a single project score (0-100) with a two-condition CI gate; HarnessX adoption at the hunter loop, the verifier, and the self-improvement meta-harness; benchmark-to-signal conversion with per-rule attribution; bounded loop authority over rule status, evidence floors, thresholds, and score constants; a dependency-weight audit and decision; preservation of the recall/FP detection gate.
- **Out of scope**: authoring new detection pattern text by the loop (human pull-request only); editing the authoritative 0-5 CWE severity by the loop (verycode-upstream authority); a full single-step HarnessX rewrite of all deterministic stages and weight-level reinforcement learning (deferred, optional, later phase); changing the reachability vocabulary, which remains exactly `reachable | inferred-file-surface | unknown`; dynamic/DAST scoring, which is report-only for a static tool.
- **Adjacent expectations**: the governance and scoring planes shall run with no HarnessX present and `dependencies = []`; CI gating (score plus benchmark) shall run in the zero-dependency default install; HarnessX, when present, shall be confined behind a capability check and lazy imports so heavy code paths stay cold in the default install.
## Requirements
### Requirement 1: HarnessX Direct Adoption and Scan-Flow Migration
**Objective:** As an OpenUltraSAST maintainer, I want the scan flow to adopt the HarnessX runtime directly at the LLM hunter loop and migrate the existing stages onto HarnessX processors in a staged way, so that the project gains the HarnessX agent loop, journal, and trajectory machinery without a high-risk big-bang rewrite.
#### Acceptance Criteria
1. Where the HarnessX extra is installed, the Scan Orchestrator shall run the hunter pool stage on a real HarnessX `Harness.run()` loop with the HarnessX control, tool, and observability processors registered in the hunter sub-harness.
2. Where the HarnessX extra is installed, the Verify Processor shall produce verification verdicts using the HarnessX llm-judge helpers and shall retain the structural verifier as a zero-dependency fallback.
3. While the HarnessX extra is absent, the Scan Runtime shall execute `quick` and `standard` scans through the existing sync driver with behavior unchanged from the pre-adoption baseline.
4. When the hunter pool stage runs under HarnessX, the Scan Orchestrator shall confine all asynchronous execution to that single stage so that the deterministic stages remain synchronous.
5. The Scan Runtime shall preserve each stage's declared read and write slot allow-list and shall validate slot access on every lifecycle hook, so that the existing provenance and audit discipline is retained under the HarnessX runtime.
6. Where a deterministic stage runs under HarnessX, the Scan Runtime shall execute it with model invocation disabled so that no deterministic stage incurs a model call.
7. When migrating event names, the Scan Runtime shall map the existing `stage_start`/`stage_end` to the HarnessX step-level hooks and `scan_id` to the HarnessX run identifier without losing trace continuity.
8. If a HarnessX-backed stage is enabled and the corresponding HarnessX capability is unavailable at runtime, then the Scan Runtime shall fall back to the existing equivalent stage and shall record the degradation in the scan manifest.
### Requirement 2: Central Security-Affine Ruleset Governance
**Objective:** As a security policy owner, I want detection rules to be versioned data whose severity and scope come from a central CWE policy rather than from the rule itself, so that one CWE maps to one governed severity and no rule can be too influential by setting its own criticality or volume.
#### Acceptance Criteria
1. The Ruleset Store shall load detection rules from versioned data files in which each rule declares a pattern, a CWE, tags, and a loop-controlled `status`, and shall not carry any rule-local severity field.
2. When resolving the severity of a finding, the Policy Scoring Processor shall read severity exclusively from the Policy Store keyed on the rule's CWE and shall discard any severity string carried in legacy rule data.
3. While two enabled rules map to the same CWE, the Policy Scoring Processor shall assign both findings the identical policy-governed severity so that two rules for one CWE can no longer disagree.
4. The Ruleset Store shall support exactly three rule statuses — `enabled`, `shadow`, and `disabled` — where `enabled` fires and scores and reports, `shadow` fires and records outcomes but is excluded from the report and the score, and `disabled` does not fire.
5. When a finding's rule status is `shadow`, the Policy Scoring Processor shall exclude that finding from the reported findings and from the project-score input while still recording its outcome for precision tracking.
6. The Policy Store shall expose, per CWE, an authoritative 0-5 severity plus a static scope flag and a dynamic scope flag, and shall mark dynamic-only CWEs as report-only and never scored by the static tool.
7. If an enabled rule's CWE does not resolve in the Policy Store at startup, then the Scan Runtime shall fail loud and shall not proceed with the scan.
8. The Ruleset Store shall remap the C/C++ buffer-handling rules that reference the unmapped CWE-120 to CWE-121 (severity 5) so that those rules contribute a nonzero penalty to the project score.
9. The Ruleset Store and the Policy Store shall be loaded with the standard library only and shall require no third-party dependency.
### Requirement 3: Bounded Rule and Policy Adaptation in the Self-Improvement Loop
**Objective:** As an OpenUltraSAST maintainer, I want the HarnessX meta-harness to adapt rules and policy gates only through two bounded levers, so that the loop can suppress, stage, or raise the bar on noisy rules without ever inventing detectors or rewriting the severity authority.
#### Acceptance Criteria
1. The Self-Improvement Loop shall expose exactly two new edit levers: a `rule` lever covering rule status, evidence floor, and threshold, and a `policy` lever covering evidence floor, scope, and the score constants `K` and `MIN_SCORE`.
2. The Rule Edit Validator shall reject any loop edit that modifies detection pattern text, so that pattern authoring remains human-pull-request only.
3. The Policy Edit Validator shall reject any loop edit that modifies the authoritative 0-5 CWE severity, so that the severity authority remains upstream-governed.
4. If the loop proposes moving a rule directly from `enabled` to `disabled`, then the Rule Edit Validator shall reject the edit and require `shadow` as a mandatory intermediate staging state.
5. When validating a proposed edit, the Rule Edit Validator shall assert that the edit parses, that `status` is one of `enabled`/`shadow`/`disabled`, that any threshold or evidence-floor tightening or loosening stays within configured bounds, that an added rule does not duplicate an existing rule identifier, and that every referenced CWE resolves in the Policy Store.
6. If the loop re-proposes an edit that was previously reverted without supplying a retry rationale, then the Self-Improvement Loop shall block the re-proposal via the novelty gate.
7. Before accepting any round, the Self-Improvement Loop shall run a replay smoke gate that proves the edited ruleset and policy still load and boot.
8. When any loop edit is rejected during validation, the Self-Improvement Loop shall raise a rule-change validation error and reject the entire round.
9. The Self-Improvement Loop shall record each rule and policy edit, its lever, and its outcome in the journal so that every change is attributable and reversible.
### Requirement 4: Benchmark-Driven Iteration with a Hard Acceptance Gate
**Objective:** As an OpenUltraSAST maintainer, I want benchmark results to become actionable per-rule signals that drive the self-improvement loop under a hard acceptance gate, so that detection quality improves automatically while regressions are reverted byte-for-byte.
#### Acceptance Criteria
1. The Benchmark Harness shall attribute every recorded miss and every recorded false positive to a `rule_id` and shall aggregate per-rule precision and recall in its metrics.
2. The Benchmark Harness shall emit a per-rule recommendation artifact alongside its records, recommending enable, disable, shadow, threshold, or evidence-floor changes per rule.
3. When the orchestrator converts benchmark output to loop input, it shall write recall-side items as miss signals and precision-side items as verifier-reject or false-positive signals into the loop's signal file.
4. After the meta-agent edits rules or policy, the Self-Improvement Loop shall re-run the benchmark and shall accept the round only if aggregate recall is at least 90% AND aggregate false-positive rate is under 10% AND the project score did not regress on the held set AND no unpredicted regression occurred.
5. If a re-run round fails any acceptance condition, then the Self-Improvement Loop shall revert the round and restore the rule and policy configuration byte-for-byte.
6. If a rule drags aggregate precision below the gate, then the Self-Improvement Loop shall auto-shadow that rule rather than delete it, so that recall is preserved while the false-positive cost is removed.
7. When a coverage gap surfaces as a persistent miss, the Self-Improvement Loop shall nominate a new or loosened rule through the signal file while leaving pattern authoring to a human pull request.
8. The Self-Improvement Loop shall treat maximizing the project score as the optimization objective and the recall/FP gate as a separate hard feasibility constraint, so that the loop cannot improve the score by shadowing rules in a way that harms real security.
### Requirement 5: Project Scores via verycode Policies
**Objective:** As a security reviewer, I want a single 0-100 project score computed from policy-weighted, reachability-adjusted findings and gated in CI, so that a scan produces a bounded AppSec quality signal alongside its findings.
#### Acceptance Criteria
1. The Score Model shall compute a single 0-100 project score from policy-governed severity weights multiplied by a reachability multiplier over the reported findings, using the standard library only.
2. The Score Model shall use reachability multipliers keyed exactly on the three emitted reachability values `reachable`, `inferred-file-surface`, and `unknown`.
3. When a finding maps to a CWE that is dynamic-only or unmapped in the Policy Store, the Score Model shall assign that finding a zero score penalty and shall report it as out-of-scope or unmapped rather than scoring it.
4. The Score Model shall exclude findings whose rule status is `shadow` from the score input before computing the project score.
5. The Score Model shall emit a scan-level score artifact containing the project score, the maximum severity, the total penalty, a by-category breakdown, the out-of-scope dynamic-only count, the unmapped CWE list, and the gate verdict, and shall include the score block in the scan manifest.
6. When confirming a false positive, the Calibration subsystem shall lower the finding's effective reachability multiplier rather than delete the rule, so that recall stays high while the score stops over-penalizing.
7. If any reported finding maps to policy severity 5 AND has `reachability_status` equal to `reachable`, then the CI Score Gate shall fail the build.
8. Where the score gate is configured as blocking, the CI Score Gate shall fail the build when the project score is below `MIN_SCORE`; until the score constants are calibrated against the corpus, the CI Score Gate shall compute and report the score advisory-first without blocking.
9. The Score Model and the CI Score Gate shall run in the zero-dependency default install with no HarnessX present.
### Requirement 6: Detection Gate Non-Regression
**Objective:** As an OpenUltraSAST maintainer, I want the existing detection gate of at least 90% recall and under 10% false positives on the benchmark corpus to hold across every phase and every change, so that no governance, scoring, runtime, or self-improvement change can silently degrade detection quality.
#### Acceptance Criteria
1. The CI Pipeline shall evaluate aggregate recall and aggregate false-positive rate on the benchmark corpus on every merge and shall fail the merge if recall is below 90% or the false-positive rate is at or above 10%.
2. While the default ruleset and an empty loop ledger are in effect, the Benchmark Harness shall produce results byte-identical to the pre-feature baseline for the governance and rules-as-data phases.
3. When the self-improvement loop proposes any rule or policy change, the Self-Improvement Loop shall require the recall/FP gate to pass on the re-benchmarked candidate before the change can be accepted.
4. If a HarnessX-backed stage is enabled, then the CI Pipeline shall confirm recall and false-positive rate stay within the configured tolerance of the equivalent zero-dependency baseline.
5. The CI Pipeline shall treat the recall/FP gate as a hard constraint that is never folded into the project-score reward, so that improving the score can never substitute for meeting the detection gate.
6. If any change would reduce recall below 90% or raise the false-positive rate to 10% or above, then the CI Pipeline shall block the change regardless of any project-score improvement.
**Objective:** As an OpenUltraSAST maintainer, I want an explicit dependency-weight decision that keeps the core install zero-dependency and chooses between an optional HarnessX extra and a vendored lean subset based on an audited dependency graph, so that a supply-chain auditing tool never silently inherits an unacceptable transitive dependency footprint.
#### Acceptance Criteria
1. The Packaging Configuration shall keep the core `dependencies` list empty and shall expose HarnessX only as an optional extra pinned to a specific commit revision.
2. The Runtime shall guard every HarnessX import behind a capability check and shall import HarnessX lazily so that heavy code paths stay cold in the default install.
3. Before adopting the HarnessX execution stages, the maintainer shall run a dependency dry-run audit and shall record the realized transitive dependency graph.
4. If the audited transitive graph pulls a browser-binary, container-SDK, or web-server stack as a mandatory dependency and the maintainers judge it unacceptable for a supply-chain auditing tool, then the Packaging Configuration shall switch to a vendored lean subset of HarnessX under the same extra name with the heavy subtrees removed.
5. While HarnessX is absent, the CI Pipeline shall run a zero-dependency guard test confirming the governance, scoring, and benchmark planes operate with no HarnessX present.
6. Where the project is developed via a submodule, the Runtime setup shall initialize HarnessX non-recursively so that the large reinforcement-learning fork subtree is not pulled.
7. The Packaging Configuration shall keep the vendored lean subset pre-specified so that switching from the optional extra to the vendored subset is a packaging change and not a redesign.
**Purpose**: Capture the discovery findings and architectural investigation behind the `harnessx-self-improving-rulesets` spec. This is the background-notes companion to `design.md` — it records *what was read*, *what was found in the source*, and *why* the three resolved decisions came out the way they did. `design.md` carries the committed architecture; this file carries the evidence and the rejected alternatives.
---
## Summary
- **Feature**: `harnessx-self-improving-rulesets`
- **Discovery Scope**: Complex Integration (adopt an external agent-runtime library + re-architect the in-house rule/policy/scoring planes)
- **Key Findings**:
- HarnessX and OpenUltraSAST (OUS) already both have a thing called a "harness" with "processors" and "events", but they are **incompatible at every load-bearing point**: HarnessX is async-generator / message-window-contract / transcript-state; OUS is sync / declared-`reads`/`writes`-contract / dict-state. There is no drop-in.
- HarnessX's real, reusable value for OUS is concentrated in three places: the **LLM hunter agent loop** (control/tools/observability processors), the **llm-judge verifier** (`build_judge_prompt`/`parse_judge_response`/`_call_judge`), and the **meta-harness self-improvement engine** (journal / novelty / evidence / replay / attribution). Everything else (lab web UI, browser tools, sandboxes, office skills) is unused weight.
- OUS today fuses **detector + severity authority + volume control** into one regex layer (`PATTERN_RULES` in `findings.py`); its self-improvement loop only demotes *ranking* (floored at `0.1`, keyed on `tags[0]`), never the rules; and its benchmark already names the ruleset as the thing to fix (`next_improvement_candidate="rules_or_language_hunter"`) but writes it to an inert JSON. The signals exist; the loop back onto the rules does not.
- The `verycode` policy data (`CWE_Score.tsv`, 167 CWEs, severity 0-5 + static/dynamic scope) is **complete and authoritative**, but the `verycode-score` scoring *algorithm was never implemented* (`// scorer code here` placeholder). The score formula in the design is a labeled, defensible reconstruction.
- OUS today has **zero runtime dependencies** (`dependencies = []`); HarnessX's `dependencies` list is ~30 heavyweight packages (Playwright, Docker SDK, FastAPI/uvicorn, multi-LLM SDKs, office-doc suite) with **no lean extra** to pip-install a subset — a direct hard dependency would invert the project's supply-chain posture for a security tool.
## Research Log
### HarnessX core composition & runtime model
- **Context**: We need to know whether OUS's `harness.HarnessRuntime` can be replaced by, or interoperate with, HarnessX's `Harness`, and how an external project builds and runs a HarnessX harness.
- **Sources Consulted**: `/tmp/HarnessX/harnessx/core/{harness,builder,processor,events,runloop,state,contract_check,config_schema,config_store,model_config,trajectory}.py` (read in full); `/home/mc/Source/OpenUltraSAST/src/openultrasast/harness.py`.
- **Findings**:
- Public composition is three objects + one method: `HarnessBuilder.build() -> HarnessConfig`; `ModelConfig.agentic(HarnessConfig) -> Harness`; `await Harness.run(task) -> HarnessResult`. **`HarnessConfig` carries NO model info** — model is always supplied separately via `ModelConfig` (key→provider map, `main` required, non-`main` keys auto-get sub-harnesses).
- `HarnessBuilder` is an immutable fluent factory: `add(proc)`, `slot(**kw)`, `add_tool`, `plugin(source)`, `__or__`/`merge` (collects conflicts → `HarnessConflictError`), `build()` (topo-sorts processors per hook by `_order` then Kahn within a bucket by `_after`). Declarative alt: `build_from_config(dict)` from a flat `processors:` list of `_target_` dicts.
- The run loop (`run_loop`) is a fixed async agent loop: `task_start` → `while True:` (step: `step_start` → `before_model` → call provider → `after_model` → per-tool `before_tool`/exec/`after_tool` → `step_end`) → `task_end`. `get_procs(key)` runs `"*"` (MultiHook) processors on every hook, then hook-specific lists.
- State is **two message tracks** (`raw_messages` append-only, `messages` effective context) + typed `slots: dict[str, StateSlot]` (the nearest analogue to OUS's flat `state` dict, but typed and `_target_`-serializable). `StatefulTrajectory` is a first-class product (`s_t = (z_t, Δz_t, a_t, o_t, e_t, r_t)`), built inline during `run_loop`, with `backfill_rewards` and `to_training_records`/`to_rl_records`.
- Hook-name alignment vs OUS: 5/8 match exactly (`task_start`, `before_model`, `after_model`, `before_tool`, `after_tool`, `task_end`); the gap is `stage_start`/`stage_end` ↔ `step_start`/`step_end`. Semantically an OUS "stage" is a coarse pipeline phase wrapping a callback (`run_stage(name, callback)`), whereas a HarnessX "step" is one model-call iteration — not the same concept.
- **Implications**: A whole-pipeline swap is high risk. The cheapest correct adoption confines HarnessX's async loop to the one OUS stage that *is* an agent loop (the hunter pool) and keeps the sync driver for deterministic stages. This directly motivates Decision D1 (phased seam, not big-bang).
### The two incompatible processor contracts
- **Context**: OUS's defining feature is a declared `reads`/`writes` per-key state contract. We must know if HarnessX preserves it.
- HarnessX `Processor` is `async def process(event) -> AsyncIterator[Event]` — an async generator that yields transformed events. The recommended base is `MultiHookProcessor` with `on_*` hook methods, an `@on(EventClass)` decorator, and class-attr ordering hints (`_order`, `_after`, `_singleton_group`, `_hook`). There is **no `spec`, no `handles`, no `reads`/`writes`.**
- HarnessX's contract governs **message-window mutation**, not state keys: enforced in `ProcessorChain.process` via `_validate_messages_contract` (system stays at index 0 on `step_start`; `before_model` net length ∈ {0,+1}; appended message must be role `user`; no out-of-window edits) and per-chain `check_post_hook_invariants`. Mode is the env var `HARNESSX_CONTRACT_MODE ∈ {warn(default), strict}` — not a constructor arg.
- OUS `Processor` is **synchronous**, `def handle(event, state: dict) -> dict | None`, with `spec: ProcessorSpec(name, version, handles, reads, writes)`; `_run_processor` enforces *state-key* discipline (reads must pre-exist; writes must be declared; return must be a dict). `ContractMode` is a `"strict"|"warn"` constructor arg.
- **Implications**: The two contracts do not map onto each other. OUS's per-key allow-list (its provenance/audit discipline) has **no home in HarnessX**. To keep that discipline we must reimplement it as a `SlotContractMixin` on our HarnessX processors (`reads_slots`/`writes_slots` class attrs validated in a base `on_*` wrapper), gated by our own `OUS_SLOT_CONTRACT` env. This is Decision D7.
### HarnessX built-in processors — reuse map
- **Context**: Which HarnessX processors can OUS register directly, which are reusable as patterns/helpers, and which are irrelevant?
- **Register-direct only inside the hunter loop** (the one real agent loop): `LoopDetectionProcessor`, `CostGuardProcessor`/`TokenBudgetProcessor`, `ToolFailureGuard`, `SelfVerifyProcessor`, `ToolFilterProcessor`/`ToolWhitelistProcessor`, `ModelRouterProcessor`, `EpisodeMetricsProcessor`/`OTelProcessor`/`CheckpointProcessor`.
- **Reuse as helper functions** for `verify`: the `llm_judge` code (`build_judge_prompt`, `parse_judge_response` with the verdict whitelist, the two-attempt repair loop `_call_judge`) plus the strict-JSON verdict schema (`verdict/confidence/cause/missing_capability`/`lesson`/`missing`). The judge is explicitly told "ground truth is intentionally withheld" — matching OUS's no-oracle scanning.
- **Reuse as pattern** for `calibrate`/`record_calibration`: `MemoryRetrievalProcessor` + `MemoryPolicy` (retrieve/add, should_retrieve/should_store) is the clean interface to refactor the FP ledger toward; `MemoryExtractionProcessor` mirrors distilling verifier outcomes into durable learnings. Storage stays a custom JSON ledger.
- **Keep custom** for every SAST-domain stage: `static_mapping`, `preprocess`, `entry_point_mapping`, `quick_findings`, `report`, `sarif`, `manifest`, and the benchmark recall/FP scoring (HarnessX `EvalResult.score` is a single float — too coarse for the recall/FP gate).
- **Implications**: Adoption is bimodal — register HarnessX classes directly only in the hunter sub-harness; everywhere else port logic as helpers or patterns. This is the backbone of the §3 flow-migration table in `design.md`.
### HarnessX self-improvement (meta_harness) and the rule-adaptation seam
- **Context**: Can an external signal (a verifier rejection or a benchmark miss) drive a *detection-rule* change through HarnessX's self-improvement loop, and how does it compare to OUS's current calibration?
- HarnessX self-improves a *harness* round by round. The unit of change is a `HarnessConfig` (the meta-agent literally rewrites the inner agent's config/processors/templates as files); the unit of memory is a markdown **journal** with per-`## Round N` YAML frontmatter; the unit of safety is a multi-phase **validation gate**.
- The journal classifies every change by a **lever** from `_VALID_LEVERS = {configuration, control, action, instruction}` and tracks `predicted_affected` task_ids plus a `gating_attribution` (`flipped/still_F/regressed/still_T/absent`). `build_context` renders a per-lever **time-decayed Beta posterior** scoreboard (`_DECAY=0.9`) and an explicit "Reverted hypotheses — do not re-propose" list.
- `EvolveValidator.run` enforces validity (canonicalize → contract → diff → replay smoke gate), policy (novelty + evidence, only when the diff is non-empty), advisory. `check_novelty` hard-blocks re-proposing a reverted hypothesis or reverted `(levers, predicted_affected)` signature unless a `retry_rationale` is present. `_evidence` requires a `candidates.md``## Candidate C-N` cross-referenced by the journal. `run_synthetic_task_smoke_gate` proves the evolved config boots end-to-end. `ContractAutoCheckProcessor` gives in-session `[AUTO-CHECK]` feedback on every authored `processors/*.py`/`tools/*.py` edit.
- **The concrete seam to drive a rule change**: add a lever value to `journal._VALID_LEVERS`; carry the external signal in via a `rule_signals.json` read-context artifact + `_render_task_brief` deliverable; add a validity-phase check (`EvolveValidator._rules`) that raises `StrictValidationError(kind="rule_change")`; add changeset buckets (`compute_changeset` + `journal._CHANGESET_KEYS_ORDER`); extend `contract_autocheck` to classify rule files. The novelty/evidence/replay/attribution machinery then applies for free.
- **OUS by comparison**: detection rules in `findings.py` are static and never modified. Calibration (`calibration.py`) only does `calibrated = max(0.1, calibrated - 0.5)` on a *file's* ranking priority (keyed on `tags[0]`, not `rule_id`) plus a prompt-constraint string — it never disables/tightens/adds a rule, has no novelty gate, no attribution. Benchmark misses produce a `BenchmarkCalibrationRecord` whose `next_improvement_candidate` is hard-coded to `"rules_or_language_hunter"` and written to an **inert**`calibration_records.json` that nothing consumes. The recall side is a fully open loop.
- **Implications**: OUS has exactly the *signals* HarnessX needs (verifier rejections = precision side; benchmark misses = recall side) but no loop closing back onto the rules. The meta_harness is the engine that closes it. This motivates Decision D3 (which levers, and their hard bounds) and the §5/§6 design.
### The verycode policy & scoring model
- **Context**: The design needs a central CWE→severity authority and a project score. What does `verycode` actually provide?
- `CWE_Score.tsv` is the real artifact: 167 data rows, one per CWE, columns `Flaw Category | CWE ID | CWE Name | Flaw Severity (0-5) | Static (X/empty) | Dynamic (X/empty)`. Severity distribution: 12×5, 15×4, 97×3, 31×2, 0×1, 12×0. Scope: 154 Static, 45 Dynamic, 13 Dynamic-only, none neither. (Header column 4 is literally `"Flaw Severity "` with a trailing space.)
- **The scoring algorithm does not exist.**`norandom/verycode-score` is the unmodified `the-gophers/go-action` "Tweeter" template with `// scorer code here` and a `--policy policies/standard.json` flag; `norandom/verycode-lib` is a one-line stub README. The data shape + the "GitHub Advanced Security findings, weighted by CWE severity" descriptions imply a Veracode-style 0-100 pipeline; the design reconstructs the formula and labels it as such.
- **Real coverage gap**: CWE-120 (Classic Buffer Overflow) is **absent from the TSV**, but 3 high-value C/C++ rules use it (`c-unsafe-copy`, `c-unsafe-memory-copy`, `c-unsafe-scanf`). Unmapped → those rules would contribute 0 to the score. CWE-121 (Stack Buffer Overflow) is present at severity 5 and is the precise classic-overflow CWE.
- **Severity mismatch**: OUS rule strings disagree with policy (rules label CWE-78/95/134 "high"; policy rates them 5/critical; rules label CWE-94/502 "high"; policy rates them 3/medium). The policy is the authoritative source.
- The OUS reachability vocabulary is exactly `reachable | inferred-file-surface | unknown` (verified in `findings.py`), which maps cleanly onto a `REACH_MULT = {reachable:1.0, inferred-file-surface:0.6, unknown:0.4}` exploitability multiplier — and that multiplier is the natural FP-calibration knob (lower effective reachability instead of deleting a rule).
- **Implications**: Severity must flow from PolicyStore (`CWE_Score.tsv`), not from the rule. Day-one fixes: remap the 3 CWE-120 rules to CWE-121 (Decision D5); discard rule string-severities (Decision D4). The score is a stdlib-only reconstruction (Decision D8 ships it advisory-first until `K`/`MIN_SCORE` are calibrated).
### Installability & dependency weight
- **Context**: How does an external project depend on HarnessX, and what does it cost a zero-dependency security tool?
- HarnessX is **library-first** and **effectively not on PyPI** (`name="harnessx"`, `version="0.1.0"`, Beta; consumed from Git). `requires-python = ">=3.11"` — exact match with OUS, no floor conflict. Depend via a SHA-pinned Git URL (the version string won't move).
- Custom stages map cleanly onto HarnessX's plugin model: each SAST stage is a `MultiHookProcessor`; group stages + tools + skills into a `HarnessPlugin` mounted via `builder.plugin(...)`; load in-process (instance/class) rather than via directory/`plugin.json`.
- **Dependency-weight is the headline risk.** HarnessX's `dependencies` (always-installed, not extras) is ~30 heavyweight packages: `anthropic`/`openai`/`litellm`/`tiktoken`, `fastapi`/`uvicorn`/`websockets`, `playwright` (ships browser binaries), `docker`/`e2b`, `hydra-core`/`omegaconf`/`structlog`/`loguru`, `opentelemetry-*`, `mcp`, `python-pptx`/`python-docx`/`openpyxl`/`reportlab`/`pdfplumber`. **There is no lean extra** — `pip install harnessx` pulls everything. OUS currently has `dependencies = []`.
- **Mitigating fact**: HarnessX uses **pervasive lazy imports** (heavy modules imported inside functions). So the heavy deps are *installed* but only *imported on demand* — which makes both an optional-extra opt-in and a vendored subset genuinely viable.
- **Submodule caveat**: `.gitmodules` declares a nested `recipe/verl_harnessX/verl` → a large verl RL fork OUS does not need; never `--recursive` init.
- **Implications**: A hard dependency is off the table (it inverts the zero-dep stance and pins to an unmovable Beta). Options narrow to (B) optional extra and (C) vendored lean subset. The lazy-import architecture is what makes B keep heavy code cold by default and makes C a surgical packaging switch. This is Decision D2.
### Current OUS rule-system problem analysis
- **Context**: Establish, in code terms, why the user's complaint ("rules are too influential and not centrally managed") is true.
1. **Pattern matches dominate output.** In quick mode, every non-comment regex match becomes a finding (`quick_scan_findings`); the only gate is language. Ranking only sorts; verification rejects only on `reachability_status == "unknown"`. A noisy regex sets output volume and FP rate directly — the rule *is* the result.
2. **Severity is rule-local.**`PatternRule.severity` is a string literal copied straight onto the finding (`_finding_from_match`). Two rules for the same CWE can disagree; nothing reconciles them.
3. **No central store.** The ruleset is a Python tuple compiled into the package; `config.py` has no rule/policy section. You cannot enable/disable/tune a rule without editing source and cutting a release.
4. **Calibration demotes ranking, not rules.**`calibrate_ranking` only subtracts `learning.demotion` (default 0.5) from a *file's*`priority`, floored at `MIN_CALIBRATED_PRIORITY = 0.1`, keyed on `tags[0]`. A chronically wrong rule fires at full severity forever; learning targets the wrong object.
5. **No CWE-policy linkage.**`cwe` is free-text used only for rationale text and loose benchmark matching. No table maps CWE → governed severity/enablement/evidence floor.
6. **No project-score weighting.** Severity and emission are global constants; the ranking signal (`composite_priority`) only re-sorts. A low-severity crypto finding in a fuzzable network parser is never promoted; a high-severity finding in dead code is never demoted.
- **Implications**: The regex layer is simultaneously **detector, severity authority, and volume knob**, while ranking/verification/calibration/benchmark can only act *around* the rules, never *on* them. The fix is to split those three roles into three owners (rules detect / policy decides severity & scope / score is the optimization target) with a one-directional authority model (governance drives execution; execution emits signals; the loop edits governance under a gate). This is the organizing principle of `design.md`.
## Architecture Pattern Evaluation
These are the three "architect angles" that were considered for the overall shape. Each is internally coherent; the synthesis in `design.md` borrows from all three and resolves their three disagreements (D1/D2/D3).
| **Incremental seam** | Keep the OUS sync driver; run HarnessX only inside the `hunter_pool` callback via `asyncio.run`. Rules→data + policy + score added around the existing pipeline. Dependency = optional extra (B) with vendored (C) as escape hatch. Loop edits a `"rule"` lever (status/threshold). | Smallest blast radius; confines the async surface to one stage; keeps the recall/FP gate green per-merge; preserves zero-dep default. | Two harness contracts coexist for a while; `HarnessRuntime` lingers; deterministic stages don't gain HarnessX trajectories yet. | Chosen as the base trajectory (Phases 0-4). |
| **Native rewrite** | Delete `HarnessRuntime` entirely; the *whole* scan becomes a single-step HarnessX harness with `skip_model=True` + slot dataflow; vendored lean subset (C) as the committed default. Loop edits a `"rule"` lever; patterns stay human-PR'd. | Cleanest end-state; one runtime, one contract; full trajectory/journal coverage of every stage; smallest auditable dep graph. | Touches every stage at once — too much blast radius to keep the gate green per-merge; vendoring is permanent fork-maintenance burden. | Demoted to optional Phase 5+; its end-state (single-step harness, retire `HarnessRuntime`) is the long-term target. |
| **Policy-first** | The whole scan runs as processors on a fixed loop; central CWE policy is the centerpiece; loop edits `"rule"` + `"policy"` levers (policy = evidence floors/scope, *not* the 0-5 CWE severity). Dependency = optional extra (B) default, vendored (C) fallback. | Strongest governance story; two levers give the loop the right surface to "stop rules being too influential" without inventing detectors or rewriting severity. | Same whole-pipeline-at-once risk as native if taken literally; needs careful bounds so the loop can't corrupt the ruleset. | Contributed the two-lever model (D3) and the bounded-authority safety rules. |
## Design Decisions
The three architect angles agreed on the destination (rules→governed data; severity from a central CWE policy; a gated project score; HarnessX hosting the hunter loop and a rule-adapting self-improvement loop). They disagreed on three load-bearing choices. Each is resolved below with the recall/FP gate as the deciding tie-breaker. (D4-D9 are smaller decisions that follow.)
### Decision D1: Scope of the HarnessX migration
- **Context**: How much of the scan moves onto HarnessX, and how fast?
- **Alternatives Considered**:
1. Incremental seam — sync driver stays; HarnessX only in the hunter callback.
3. Policy-first whole-loop — whole scan as processors on a fixed loop.
- **Selected Approach**: **Phased.** Adopt HarnessX directly *at the hunter loop* first (Phase 3); migrate deterministic stages onto a single-step harness only later and optionally (Phase 5+), retiring `HarnessRuntime` only after the seam is proven and the gate has held.
- **Rationale**: The sync→async + contract mismatch is the single highest-risk item in the whole effort. The native rewrite touches every stage at once — too much blast radius to keep ≥90% recall / <10% FP green per-merge. The incremental seam ships the *actual* HarnessX value (agent loop, journal, `StatefulTrajectory`) at exactly the one stage that is already an agent loop.
- **Trade-offs**: Two harness contracts coexist through Phases 3-4 and `HarnessRuntime` lingers; in exchange every phase is independently shippable, test-gated, and reversible.
- **Follow-up**: Golden-output diff gate at each phase; confine `asyncio.run` to the hunter callback; `SlotContractMixin` to preserve key-level discipline.
### Decision D2: Dependency strategy
- **Context**: A zero-dependency security tool considering a ~30-package agent runtime with no lean extra.
- **Alternatives Considered**:
1. (A) Hard dependency in main deps.
2. (B) Optional extra `openultrasast[harnessx]`, SHA-pinned, lazy-imported.
3. (C) Vendored lean subset under `src/openultrasast/_vendor/harnessx/`.
- **Selected Approach**: **(B) is the shipping default; (C) is a pre-committed, triggered fallback** gated by an explicit `pip install --dry-run` audit before Phase 3. Core `dependencies = []` stays; all HarnessX imports are lazy behind a `_has_harnessx()` check.
- **Rationale**: HarnessX's pervasive lazy imports mean the heavy deps are *installed* but only *imported on demand*, so a clean opt-in extra already keeps heavy code paths cold in the default install — with the least maintenance burden, while tracking upstream. (A) permanently inverts the project's stance and pins to an unmovable Beta `0.1.0`. (C) adds permanent fork-maintenance and should only be paid for when the *installed* graph (not the *imported* graph) is itself unacceptable.
- **Trade-offs**: With B, enabling the extra still inherits the full HarnessX dep set (no lean upstream extra exists); every HarnessX import must be guarded.
- **Follow-up**: Run the dry-run audit and record the realized transitive graph. **Decision rule**: if it pulls Playwright browser binaries / Docker SDK / a FastAPI web stack as *mandatory* transitive deps and maintainers judge that unacceptable, switch to (C) — vendor `harnessx/{core,plugins,processors,bundles,meta_harness,workspace,tools/base}` + the `llm_judge`/`journal`/`validate_workflow`/`agent`/`contract_autocheck` modules, delete `lab/`/browser/sandbox/office subtrees, re-pin a minimal set. The lazy-import design makes the deletion surgical. Never use the recursive submodule path (`recipe/verl_harnessX/verl`).
### Decision D3: What the self-improvement loop is allowed to edit
- **Context**: The user's core complaint is that rules are "too influential." The loop must be able to *govern* rules without becoming a new way for them to run wild.
- **Rationale**: This is the precise answer to "stop rules being too influential" — the loop may *suppress / stage / raise-the-bar*, never *invent detectors* or *rewrite the severity authority*. Bounds + mandatory `shadow` staging + auto-revert keep loop authority safe; the meta_harness novelty/evidence/replay/attribution gates apply for free.
- **Trade-offs**: The loop cannot fix a bad pattern itself (it can only `shadow` it and nominate a human PR), and cannot re-weight a CWE — both are deliberate guardrails.
- **Follow-up**: Implement `EvolveValidator._rules`/`._policy` raising `StrictValidationError(kind="rule_change")`; enforce the four "may NOT" bounds (no pattern edits, no 0-5 severity edits, no `enabled→disabled` without `shadow`, no reverted re-proposal without `retry_rationale`).
### Decision D4: Severity authority
- **Context**: Two rules for the same CWE can disagree on severity today.
- **Selected Approach**: PolicyStore (`CWE_Score.tsv`) is authoritative; `resolve_severity(policy, rule.cwe, target, ranking)` replaces the literal `rule.severity` read; `PatternRule.severity` is removed. Policy always wins; rule string-severities are discarded.
- **Rationale**: One CWE = one governed severity, reconciled centrally; ends rule disagreement.
- **Trade-offs**: Loses the per-rule severity hint (acceptable — it was an authoring accident, not a governed decision).
### Decision D5: The CWE-120 coverage gap
- **Context**: 3 C/C++ buffer rules use CWE-120, which is absent from `CWE_Score.tsv`.
- **Selected Approach**: Remap `c-unsafe-copy`/`c-unsafe-memory-copy`/`c-unsafe-scanf` to **CWE-121** (Stack Buffer Overflow, severity 5) in the ruleset TOML.
- **Rationale**: CWE-121 is the precise classic-overflow CWE and is already at severity 5; this avoids forking the upstream policy file. Otherwise those high-value rules contribute 0 to the score.
- **Follow-up**: Add a hard CI invariant — every `enabled` rule's `cwe` must resolve in PolicyStore (fail loud at startup); log `unmapped_cwe` in the score artifact; PR genuine gaps upstream to verycode-policies.
### Decision D6: Response to a confirmed false positive
- **Context**: How does the system react to an FP without hurting recall?
- **Selected Approach**: Lower the finding's effective **reachability multiplier** and/or `shadow` the rule — **never delete it**.
- **Rationale**: The score stops over-penalizing while the rule keeps firing (tracked, just not scored/reported), preserving ≥90% recall and pushing FP <10%. This is exactly the recall/FP balance the project gates on.
### Decision D7: Where OUS's `reads`/`writes` contract lives
- **Context**: HarnessX has no per-key state contract; OUS's defining discipline must survive.
- **Selected Approach**: Reimplement as a `SlotContractMixin` on our `MultiHookProcessor`s (`reads_slots`/`writes_slots` class attrs validated in a base `on_*` wrapper), gated by `OUS_SLOT_CONTRACT=strict|warn`.
- **Rationale**: The two contracts are incompatible; the mixin preserves provenance/audit discipline inside HarnessX's runtime without trying to bend the message-window contract.
### Decision D8: Project-score gate rollout
- **Context**: The verycode score formula is a reconstruction (no upstream implementation).
- **Selected Approach**: Ship the score gate **advisory-first** (computed and reported, not blocking); promote to blocking only after `K`/`MIN_SCORE` are calibrated against the benchmark corpus. Score computation is stdlib-only and always on (zero-dep), so CI gating runs in the default install with no HarnessX present.
- **Rationale**: The formula needs corpus calibration before it can safely block a merge; keeping it zero-dep keeps the CI gate independent of the optional harness.
### Decision D9: RL adoption
- **Context**: HarnessX has an `rl/` subsystem (weight-level improvement) alongside the config-level meta_harness.
- **Selected Approach**: Optional, Phase 5+.
- **Rationale**: Config-level adaptation via the meta_harness is sufficient to close the rule/policy loop; weight-level RL over hunter trajectories is purely additive and not on the critical path.
## Risks & Mitigations
- **Sync↔async + contract incompatibility (High)** — confine the async surface to the hunter stage via `asyncio.run` in its callback (Phase 3); `SlotContractMixin` for key-level discipline; deterministic stages run `skip_model`; golden-output diff gate at each phase.
- **Dependency posture inversion (High)** — governance/scoring planes stay zero-dep; HarnessX strictly behind the extra + lazy imports + pinned SHA; CI zero-dep guard test; `pip --dry-run` audit before Phase 3; vendored-subset fallback pre-specified.
- **Benchmark-gate regression from rule auto-tuning (High)** — the recall/FP gate is the per-round acceptance test (aggregate + per-rule, hard constraint); `shadow` before `disabled`; novelty + attribution block thrash and reverted re-proposals; auto-revert byte-for-byte.
- **Score gaming the gate (High)** — keep the gate as a *hard constraint*, never part of the reward; track unpredicted regressions; add a score-regression check to acceptance.
- **Meta-agent corrupts the ruleset, re-introducing "too influential" (Medium)** — bounded levers (D3): no pattern rewrites, no severity edits, mandatory `shadow` staging, no `enabled→disabled` jump; recall gate auto-reverts; every change journaled and novelty-checked.
- **Policy completeness / CWE drift (Medium)** — day-one CWE-120→CWE-121 remap; hard CI invariant that every enabled rule's CWE resolves; `unmapped_cwe` logged; treat verycode-policies as authoritative-upstream and PR gaps there.
- **Frozen Beta upstream (Medium)** — pin SHA; document in `VENDOR.md`; keep the meta_harness patch minimal/isolated; rule lever degrades gracefully (absent patch = no rule lever, loop still does ranking demotion); vendored subset removes upstream-instability exposure for depended modules.
- **Slot serialization loss on resume (Medium)** — all scan slots use plain dicts/dataclasses with `_target_` registration; per-slot-type round-trip serialization test (non-serializable content restores as `None` on `wake()`).
- **Two-store coordination drift (Low-Med)** — a single `resolve_severity` owns the rule-status + policy-gate combination; the changeset records both lever edits per round; the benchmark regression suite catches interaction regressions.
This plan splits the fused regex layer into three governed owners (rules detect, CWE policy decides severity/scope, project score is the optimization target) and adopts HarnessX behind an optional extra. Tasks are grouped into four phases. Each phase is independently shippable and keeps the ≥90% recall / <10% FP detection gate green. Phases 1-2 (Foundation) and the early Core work stay zero-dependency and byte-identical by default; the HarnessX execution and self-improvement work (Integration) lives behind a lazy, SHA-pinned extra; Validation locks the cross-cutting gates and the dependency-weight decision.
- [x] 1. Stand up the central CWE policy authority
- [x] 1.1 Vendor the verycode CWE severity data and build the Policy Store loader
- Vendor the verycode CWE-to-severity dataset into the governance plane and parse it with the standard library only, tolerating the trailing-space severity column header.
- Expose, per CWE, an authoritative 0-5 severity, a static scope flag, and a dynamic scope flag; mark dynamic-only CWEs as report-only.
- Observable completion: loading the policy yields the full set of CWE policy entries with correct static/dynamic flags and zero third-party imports, verified by a unit test.
- Resolve a finding's severity exclusively from the Policy Store keyed on the rule's CWE, discarding any severity string carried in legacy rule data.
- Guarantee that two findings whose rules map to the same CWE receive identical severity.
- Observable completion: a resolution call returns the policy severity (never a legacy string) and is identical for two rules sharing a CWE, proven by a unit test.
- _Requirements: 2.2, 2.3_
- _Boundary: PolicyStore_
- [x] 1.3 Enforce the fail-loud startup CWE-resolution invariant
- Assert at startup that every enabled rule's CWE resolves in the Policy Store; abort the scan loudly before any work when an enabled rule's CWE is unmapped.
- Observable completion: an enabled rule with an unmapped CWE aborts the scan with a loud error and no findings are produced, proven by an integration test.
- _Requirements: 2.7_
- _Boundary: PolicyStore_
- _Depends: 1.1_
- [x] 2. Build the project Score Model and CI gates (zero-dep)
- [x] 2.1 Compute the 0-100 policy-weighted, reachability-adjusted project score
- Compute a single 0-100 project score from policy-governed severity weights multiplied by a reachability multiplier over reported findings, using the standard library only.
- Key the reachability multipliers exactly on the three emitted reachability values (`reachable`, `inferred-file-surface`, `unknown`) without redefining the vocabulary.
- Assign a zero score penalty to findings whose CWE is dynamic-only or unmapped, reporting them as out-of-scope or unmapped instead of scoring them.
- Observable completion: a no-penalty corpus scores 100, a single severity-5 reachable finding scores roughly 43, and dynamic-only/unmapped findings contribute zero penalty, proven by boundary unit tests.
- _Requirements: 5.1, 5.2, 5.3, 5.9_
- _Boundary: ScoreModel_
- _Depends: 1.1_
- [x] 2.2 Emit the scan-level score artifact and manifest score block
- Emit a score artifact containing the project score, the maximum severity, the total penalty, a by-category breakdown, the out-of-scope dynamic-only count, the unmapped-CWE list, and the gate verdict; include the score block in the scan manifest.
- Observable completion: a scan writes a score artifact with every required field and merges the score block into the manifest, verified against the documented artifact schema.
- _Requirements: 5.5_
- _Boundary: ScoreModel_
- _Depends: 2.1_
- [x] 2.3 Implement the two-condition CI score gate (advisory-first)
- Fail the build when any reported finding maps to policy severity 5 and has reachability `reachable`.
- When configured as blocking, fail the build when the project score is below the minimum; until the score constants are calibrated against the corpus, compute and report the score advisory-first without blocking.
- Observable completion: the hard gate fails on a severity-5 reachable finding while the score gate reports advisory-only by default, proven by gate tests.
- Add configuration entries for the ruleset, policy, and score data locations plus the gate thresholds and emission floors used by later phases.
- Observable completion: the scan reads the score constants and emission thresholds from configuration rather than hardcoded literals, confirmed by a config-driven test.
- [x] 3. Convert detection rules into versioned, governed data
- [x] 3.1 Externalize detection patterns into a versioned Ruleset Store
- Load detection rules from versioned data files in which each rule declares a pattern, a CWE, tags, and a loop-controlled status, carrying no rule-local severity field.
- Remove the rule-local severity field from the in-memory rule model so severity can only come from policy.
- Observable completion: the existing pattern rules load from versioned data files with no severity attribute present, proven by a unit test asserting the absence of the field.
- _Requirements: 2.1, 2.9_
- _Boundary: RulesetStore_
- [x] 3.2 Implement the three-status volume control
- Support exactly three rule statuses — `enabled`, `shadow`, `disabled` — where enabled fires/scores/reports, shadow fires and records outcomes but is excluded from report and score, and disabled does not fire.
- Exclude shadow-status findings from the reported set and from the score input while still recording their outcome for precision tracking.
- Observable completion: an enabled rule appears in report and score, a shadow rule is excluded from both yet records an outcome, and a disabled rule never fires, proven by a unit test across all three statuses.
- _Requirements: 2.4, 2.5, 5.4_
- _Boundary: RulesetStore, ScoreModel_
- _Depends: 3.1, 2.1_
- [x] 3.3 Remap the unmapped C/C++ buffer rules to a scored CWE
- Remap the C/C++ buffer-handling rules that reference the unmapped CWE-120 to CWE-121 so they resolve in policy and contribute a nonzero penalty.
- Observable completion: the remapped buffer rules resolve to a severity-5 policy entry and produce a nonzero score penalty, verified by a unit test.
- _Requirements: 2.8_
- _Boundary: RulesetStore_
- _Depends: 3.1, 1.1_
- [x] 3.4 Wire findings and severity resolution through the governance plane
- Pass the loaded ruleset into the finding-detection stage instead of reading an implicit module-level rule list, and resolve every finding's severity through the Policy Store.
- Apply emission thresholds so not every regex match becomes a reported finding, replacing the old "rule is its own volume control" behavior.
- Observable completion: a scan produces findings with policy-resolved severity and shadow findings filtered out, while the default ruleset keeps benchmark output byte-identical to the pre-feature baseline.
- [x] 4. Make benchmark output an actionable per-rule signal
- [x] 4.1 Attribute misses and false positives to rules
- Attribute every recorded miss and every recorded false positive to a rule identifier and aggregate per-rule precision and recall in the benchmark metrics.
- Observable completion: a benchmark run reports per-rule precision/recall and every miss and false positive carries a rule identifier, verified by a metrics test.
- _Requirements: 4.1_
- _Boundary: Benchmark Harness_
- _Depends: 3.1_
- [x] 4.2 Emit a per-rule recommendation delta artifact
- Replace the inert improvement-candidate stub with a per-rule recommendation artifact recommending enable, disable, shadow, threshold, or evidence-floor changes per rule.
- Observable completion: a benchmark run writes a recommendation artifact with per-rule suggested actions alongside its records, verified against the documented delta schema.
- _Requirements: 4.2_
- _Boundary: Benchmark Harness_
- _Depends: 4.1_
- [x] 4.3 Introduce the loop-owned ledger and re-key calibration to rules
- Add the loop-owned overlay ledger that records per-rule status, evidence floor, threshold, and precision, applied over the ruleset at load time.
- Re-key calibration from the first-tag key to the rule identifier and write status flips into the ledger rather than only demoting ranking.
- Observable completion: an empty ledger leaves behavior byte-identical to the prior phase while a ledger entry overlays a rule's status at load, proven by an integration test.
- _Requirements: 2.4, 5.6, 6.2_
- _Boundary: RulesetStore, Calibration subsystem_
- _Depends: 3.1, 3.2_
- [x] 4.4 Convert false-positive confirmation into a reachability knob
- When confirming a false positive, lower the finding's effective reachability multiplier rather than delete the rule, so recall stays high while the score stops over-penalizing.
- Observable completion: confirming a false positive lowers the finding's effective reachability multiplier and the rule still fires, proven by a calibration test.
- [x] 5. Gate and package the HarnessX dependency-weight decision
- [x] 5.1 Run the dependency dry-run audit and record the transitive graph
- Before adopting the HarnessX execution stages, run a dependency dry-run audit and record the realized transitive dependency graph.
- If the audited graph pulls a browser-binary, container-SDK, or web-server stack as a mandatory dependency and it is judged unacceptable, select the pre-specified vendored lean subset under the same extra name with the heavy subtrees removed.
- Observable completion: the audit produces a recorded transitive-dependency graph and a documented accept-extra-or-vendor-subset decision before any HarnessX execution code is wired in.
- _Requirements: 7.3, 7.4, 7.7_
- _Boundary: Packaging Configuration_
- [x] 5.2 Configure HarnessX as a SHA-pinned optional extra with a capability guard
- Keep the core dependency list empty and expose HarnessX only as an optional extra pinned to a specific commit revision; initialize any submodule non-recursively so the large RL fork subtree is not pulled.
- Guard every HarnessX import behind a capability check and import it lazily so heavy code paths stay cold in the default install.
- Observable completion: the default install resolves with an empty core dependency list and HarnessX symbols are only imported after the capability check passes, verified by an import-cold test.
- _Requirements: 7.1, 7.2, 7.6_
- _Boundary: Packaging Configuration_
- _Depends: 5.1_
- [x] 6. Adopt HarnessX at the hunter loop and verifier
- [x] 6.1 Run the hunter pool stage on a real HarnessX agent loop
- Run the hunter pool stage on a real HarnessX run loop with the control, tool, and observability processors registered in the hunter sub-harness, invoked from the existing hunter-pool stage callback.
- Confine all asynchronous execution to this single stage so the deterministic stages remain synchronous.
- Observable completion: with the extra installed, the hunter pool produces per-target trajectories from a real agent loop while all other stages stay synchronous, verified by an integration test.
- Map the existing stage-start/stage-end events to the HarnessX step-level hooks and the scan identifier to the HarnessX run identifier without losing trace continuity.
- Observable completion: a HarnessX-backed run preserves trace continuity with stage events mapped to step hooks and the scan id mapped to the run id, verified by inspecting the emitted trace.
- _Requirements: 1.7_
- _Boundary: HxScanOrchestrator_
- _Depends: 6.1_
- [x] 6.3 (P) Run deterministic stages as model-disabled HarnessX processors with slot contracts
- Run each deterministic stage under HarnessX with model invocation disabled so no deterministic stage incurs a model call.
- Preserve each stage's declared read/write slot allow-list and validate slot access on every lifecycle hook.
- Observable completion: deterministic stages execute under HarnessX with no model calls and a slot-contract violation is raised on undeclared slot access, proven by a slot-contract test.
- Implemented zero-dep in `slot_contract.py` (`SlotContractMixin`/`SlotPipeline`, validates reads/writes per lifecycle hook, enforces `skip_model`) + `stage_processors.py` (the deterministic stages as `skip_model` slot processors; `run_quick_pipeline` is byte-identical to the quick path). Per "bridge, do not map", the sync `HarnessRuntime` stays the top-level driver and these slot processors are the model-disabled representation; `host_under_harnessx` is the lazy, capability-gated bridge that disables the model on HarnessX's `BeforeModelEvent.skip_model`. Module names differ from the design's `harness/{slot_contract,processors}.py` because `harness` is a module, not a package. Tests: `tests/test_slot_contract.py`.
- _Requirements: 1.5, 1.6_
- _Boundary: HxScanOrchestrator_
- _Depends: 6.1_
- [x] 6.4 (P) Add the LLM-judge verifier with a structural fallback
- Produce verification verdicts using the HarnessX llm-judge helpers when the extra is present, mapping the strict-JSON verdict to a verification result.
- Retain the existing structural verifier (which rejects on unknown reachability) verbatim as the zero-dependency fallback.
- Observable completion: with the extra present the judge produces verdicts and with it absent the structural verifier produces equivalent verdicts, proven by a paired verifier test.
- _Requirements: 1.2_
- _Boundary: VerifyProcessor_
- _Depends: 5.2_
- [x] 6.5 Fall back gracefully when a HarnessX capability is unavailable
- When a HarnessX-backed stage is enabled but its capability is unavailable at runtime, fall back to the existing equivalent stage and record the degradation in the scan manifest.
- Observable completion: requesting a HarnessX stage whose capability is missing runs the existing equivalent and records a degradation entry in the manifest, proven by a fallback test.
- _Requirements: 1.8_
- _Boundary: HxScanOrchestrator_
- _Depends: 6.1, 6.4_
- [x] 7. Stand up the bounded self-improvement loop
- [x] 7.1 Add the two bounded edit levers
- Expose exactly two new edit levers: a rule lever covering rule status, evidence floor, and threshold, and a policy lever covering evidence floor, scope, and the score constants.
- Observable completion: the loop's valid-lever set includes exactly the rule and policy levers and rejects any edit outside their declared fields, verified by a lever test.
- _Requirements: 3.1_
- _Boundary: EvolveValidator, improve/evolve.py_
- _Depends: 5.2_
- [x] 7.2 Enforce the rule/policy edit safety bounds in the validator
- Reject any loop edit that modifies detection pattern text and any edit that modifies the authoritative 0-5 CWE severity.
- Reject moving a rule directly from enabled to disabled, requiring shadow as a mandatory intermediate staging state.
- Assert that each edit parses, that status is one of the three literals, that threshold/evidence-floor changes stay within configured bounds, that an added rule does not duplicate an existing identifier, and that every referenced CWE resolves in policy; raise a rule-change validation error and reject the whole round on any failure.
- Observable completion: each prohibited edit (pattern change, severity change, enabled-to-disabled jump, duplicate identifier, unmapped CWE, out-of-bounds threshold) is rejected and rejects the round, proven by validator unit tests.
- _Requirements: 3.2, 3.3, 3.4, 3.5, 3.8_
- _Boundary: EvolveValidator_
- _Depends: 7.1, 1.1_
- [x] 7.3 Add the novelty, replay, and journal gates
- Block re-proposing a previously reverted edit that lacks a retry rationale via the novelty gate.
- Run a replay smoke gate before any benchmark re-run proving the edited ruleset and policy still load and boot.
- Record each rule and policy edit, its lever, and its outcome in the journal so every change is attributable and reversible.
- Observable completion: a reverted re-proposal without rationale is blocked, a non-booting edited ruleset is rejected before re-benchmark, and every edit appears in the journal with its lever and outcome, proven by gate tests.
- _Requirements: 3.6, 3.7, 3.9_
- _Boundary: EvolveValidator, improve/evolve.py_
- _Depends: 7.2_
- [x] 7.4 Convert benchmark deltas into loop signals
- Convert the per-rule recommendation delta and false-positive learnings into a loop signal file, writing recall-side items as miss signals and precision-side items as verifier-reject or false-positive signals.
- Observable completion: a benchmark run produces a signal file whose recall-side entries are miss signals and precision-side entries are verifier-reject/false-positive signals, verified against the documented signal schema.
- _Requirements: 4.3_
- _Boundary: improve/evolve.py_
- _Depends: 4.2, 7.1_
- [x] 7.5 Implement the hard acceptance gate with byte-for-byte revert
- After the meta-agent edits rules or policy, re-run the benchmark and accept the round only if aggregate recall is at least 90% AND false-positive rate is under 10% AND the project score did not regress on the held set AND no unpredicted regression occurred.
- On any failed acceptance condition, revert the round and restore the rule and policy configuration byte-for-byte.
- Observable completion: a round that regresses recall is reverted and the configuration is restored byte-for-byte, while a compliant round is accepted and persisted, proven by a self-improvement integration test.
- _Requirements: 4.4, 4.5, 6.3_
- _Boundary: improve/evolve.py_
- _Depends: 7.3, 7.4, 4.1_
- [x] 7.6 Auto-shadow precision-draggers, nominate coverage gaps, and separate the objective
- Auto-shadow a rule that drags aggregate precision below the gate rather than delete it, preserving recall while removing the false-positive cost.
- Nominate a new or loosened rule through the signal file when a coverage gap surfaces as a persistent miss, leaving pattern authoring to a human pull request.
- Treat maximizing the project score as the optimization objective and the recall/FP gate as a separate hard feasibility constraint that is never folded into the score reward.
- Observable completion: a deliberately noisy rule is auto-shadowed (not deleted) when it drags precision, a persistent miss emits a rule nomination, and the loop cannot raise the score by shadowing rules that breach the gate, proven by integration tests.
- [x] 8. Lock the detection-gate non-regression checks into CI
- [x] 8.1 Enforce the per-merge recall/FP detection gate
- Evaluate aggregate recall and aggregate false-positive rate on the benchmark corpus on every merge and fail the merge if recall is below 90% or the false-positive rate is at or above 10%.
- Treat the recall/FP gate as a hard constraint that is never folded into the project-score reward, blocking any change that breaches the gate regardless of score improvement.
- Observable completion: a CI run fails the merge when recall drops below 90% or false-positive rate reaches 10%, and a score improvement cannot unblock a gate-breaching change, proven by pipeline tests.
- _Requirements: 6.1, 6.5, 6.6_
- _Boundary: CI Pipeline_
- _Depends: 4.1_
- [x] 8.2 (P) Assert byte-identical benchmark baseline for the zero-dep phases
- With the default ruleset and an empty loop ledger, assert the benchmark output is byte-identical to the pre-feature baseline for the governance and rules-as-data phases.
- Observable completion: a golden-diff test confirms byte-identical benchmark output with the default ruleset and empty ledger.
- When a HarnessX-backed stage is enabled, confirm recall and false-positive rate stay within the configured tolerance of the equivalent zero-dependency baseline.
- Observable completion: a CI run with a HarnessX-backed stage enabled keeps recall/FP within the configured tolerance of the baseline, proven by a tolerance test.
- _Requirements: 6.4_
- _Boundary: CI Pipeline, HxScanOrchestrator_
- _Depends: 6.1, 8.1_
- [x] 8.4 (P) Add the zero-dependency guard test
- Run a guard test confirming that with HarnessX absent the governance, scoring, and benchmark planes operate and CI gating runs in the default install.
- Observable completion: the guard test passes with HarnessX uninstalled and fails if any governance/scoring/benchmark path imports HarnessX.
- _Requirements: 7.5_
- _Boundary: CI Pipeline_
- _Depends: 8.1_
- [x] 9. Validate scan-pipeline integrity and serialization
- Confirm that with the extra absent, quick and standard scans execute through the existing sync driver with behavior unchanged from the pre-adoption baseline, and that a requested-but-unavailable HarnessX stage records a degradation.
- Observable completion: an extra-absent scan matches the pre-adoption baseline and a missing-capability request records a manifest degradation, proven by an end-to-end fallback test.
- _Requirements: 1.3, 1.8_
- _Boundary: HxScanOrchestrator, CI Pipeline_
- _Depends: 6.5_
- [x] 9.2 (P) Verify slot serialization round-trips on resume
- Confirm each scan slot type serializes and restores without loss after a resume, guarding against non-serializable slots restoring as empty.
- Observable completion: a per-slot-type round-trip test passes for every scan slot after a resume cycle.
- Done for the current artifact-persistence model (findings/ranking/file-target/verification/score round-trip + the real findings.json reload path in `tests/test_scan_pipeline_integrity.py`). The HarnessX processor slot-contract serialization variant remains deferred with task 6.3.
A --> G[hunt.program<br/>dspy.ReAct + HuntSignature]
G --> R[hunter_arm.run<br/>records per change]
R --> S[unseen.score + valid.metrics]
S --> I[importance table,<br/>acceptance decision, report]
O[dspy optimisers<br/>GEPA, MIPROv2] --> G
I --> O
```
- Selected pattern: a table-centred pipeline; the program is a replaceable consumer of the table.
- Existing patterns preserved: the runner's record shape and resumability; the scorer; folds by repository; the
instrument proof (bytes read, base and head verified).
### Technology stack (changed layers only)
| Layer | Tool | Role |
|---|---|---|
| Program runtime | `dspy==3.4.*` (pulls litellm, openai, gepa, diskcache) as extra `[hunt]` | signature, ReAct tool loop, optimisers, cache, usage |
| Model access | `dspy.LM("openai/deepseek-flash", api_base=DEEPSEEK_BASE_URL, api_key=env)` | one client for the research side; prices registered from `model/endpoint.py:55` |
| Table | `pandas` as extra `[hunt]` | the feature table, masks, joins with labels and scores |
| Statistics | `learn.experiments` now; `scipy`/`statsmodels` after the statistics item | paired bootstrap, McNemar |
| FeatureTable | one row per candidate location with named columns | 3.1, 3.5, 4.2 | pandas | `columns: feature.<producer>.<name>`, `label`, `provenance.*`, `fold`, `schema_version` |
| producers | compute facts from the repository only | 3.2, 3.3, 3.4, 3.7 | source, semantic, mapping | `compute(root, change) -> list[dict]`; absent feature = `None` + `provenance.<name>.reason` |
| HuntProgram | the search | 3.5, 3.6, 5.1 | dspy, producers (rendered row) | input: rendered row + hunks; output: list of `{path, line, family ∈ FAMILY_IDS, confidence, rationale, cited_lines}` |
| search_metric | per-change score and feedback | 4.3, 7.1 | unseen.score tolerance | hit on fix site with right family = 1; ordinary push suspicions counted; feedback text names misses |
| semantic | `feature.semantic.roles` (source/sink/sanitizer per identifier), `.def_use` (binds and calls in the function), `.obligations` | semantic facts |
| taxonomy | `feature.taxonomy.families` (closed list with one-line definitions) | ruleset/families.toml |
| A. Extend `learn/program.py` with a hunter signature | one harness, our caller and cache | no new dependency | the compile loop has no train-vs-holdout rule, no tool loop, no optimiser beyond demos+instructions; everything the maintainer asked for (statistical mutation of prompts) is to build | rejected |
| B. New `hunt/` package on DSPy, feature table as the contract, harness reused for labels, folds, leaks, paired analysis | the optimisers, the tool loop, caching and usage come with the library; our code is the feature producers, the table, the acceptance rule, the scoring | one heavy research dependency; two caches (DSPy's and `Caller`'s) until the classifier migrates | selected |
| C. Rewrite the classifier onto DSPy too | one runtime | large rewrite of a parked line | later, if B earns it |
## Design decisions
### Decision: DSPy runs the hunter program; the acceptance rule stays ours
- Alternatives: A above; a hand-written optimiser.
- Selected: `dspy.ReAct` over a `HuntSignature` whose inputs are the rendered feature row plus the changed hunks;
`dspy.GEPA` or `MIPROv2` propose candidates; `dspy.Evaluate` scores them on compile folds with our metric; the
candidate is accepted only by our rule on held-out folds (R6), implemented outside DSPy.
- Trade-offs: DSPy's cache is request-keyed, not content-addressed; a run pins its cache directory and copies it
with the run record. Cost comes from `dspy.track_usage()` with our prices registered in litellm.
### Decision: the feature table is the contract between producers, ablation and the program
- Selected: one row per (change, path, line) candidate with columns `feature.<producer>.<name>`, `label`,
`provenance.*`, `fold`; producers are functions `(root, change, path, line) -> dict`; an ablation arm is a
column mask recorded in the manifest; the program sees only the unmasked columns, rendered in a fixed order.
- Trade-offs: a table per run is larger than nested records; it is also what pandas, scipy and the maintainer read.
### Decision: producers are a fixed list, switched by name, no registry
- [x] 1. Foundation: the feature table and its producers
- [x] 1.1 Create the feature table with a versioned column schema, masks and digests
- One row per candidate location (change, path, line) with key, label, feature, provenance and fold columns
- Rows read and write as JSONL and parquet; a table digest covers rows and schema version
- A mask drops named feature columns and is recorded with the table it was applied to
- Done when a round-trip test shows the same digest after write and read, and a masked table lists its mask
- _Requirements: 3.1, 3.5, 4.2, 5.1_
- [x] 1.2 Produce the four existing annotations from the repository only (P)
- Diff hunks at head with on-changed-line and distance; enclosing function, span, callers, entry points from the source facts; roles, def-use and obligations from the semantic facts; the closed family list with one-line definitions
- An annotation a producer cannot compute is recorded as absent with a reason, never defaulted
- Done when a fixture repository yields every column, and a repository with a planted advisory id and a security-worded commit message yields no feature carrying either
- _Requirements: 3.2, 3.3, 3.4, 3.7, 3.8_
- _Boundary: producers_
- [x] 1.3 Label the rows by the benchmark's rule (P)
- Fix-site rows of positives label 1 with the change's family and fix span; ordinary-push rows label 0; other rows on a positive label null and are excluded from scoring
- Each positive row carries the verification record reference next to the label; a row whose source cannot be traced is rejected
- Done when a fixture manifest with one positive and two ordinary pushes produces the expected labels and the null rows are excluded
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6_
- _Boundary: FeatureTable, training_set.py_
- [x] 1.4 Install the research extra and pin the program runtime
- Optional dependency group for the loop with the program runtime and the table library, pinned to a minor version; the core install stays unchanged
- Done when the extra installs into the research environment and the core test suite collects without it
- _Requirements: 5.1, 9.5_
- [x] 2. Core: the hunter as a program
- [x] 2.1 Define the hunt signature and the program with the three repository tools
- Inputs are the rendered unmasked feature columns plus the hunks; the output is the classifier's answer structure with a family from the closed list and cited lines inside the shown text
- The tool loop is bounded by the run budget; an answer outside the vocabulary or citing unseen lines is recorded unknown
- Done when a scripted model drives the program through the tools on a fixture and the parsed findings match the fixture's expectation
- _Requirements: 3.5, 3.6, 5.1, 5.2, 5.4_
- [x] 2.2 Score a change: the search metric with feedback (P)
- A suspicion within the scorer's tolerance of the fix site with the right family counts as a hit; ordinary-push suspicions are counted per push; the feedback text names the missed site or the off-diff suspicions
- Done when the metric reproduces the pilot's hit list for the 12 positives from their recorded suspicions
- _Requirements: 1.4, 4.3, 7.1, 7.2_
- _Boundary: search_metric_
- [x] 2.3 Record a run as a content-addressed artifact (P)
- Program id, schema version, mask, data digests, every scored configuration, spend, calls, replay hits, wall time, library versions; the saved program and its cache directory travel with it
- A run that exhausts its budget writes no artifact and reports the spend so far
- Done when two identical runs produce the same artifact id and a budget-exhausted run leaves only the partial report
- [x] 3.1 Run the program through the existing hunter-arm runner
- The runner takes a program artifact and a mask, computes the table rows for each change at head, renders the inputs, runs the program and writes the same record shape as today, resumable by change id, input digest, program id and mask
- The pilot manifest is refused by digest unless the run is explicitly a pilot report run
- Done when a run over a five-change fixture manifest with the scripted model writes records the existing scorer accepts
- _Requirements: 2.4, 8.4, 9.3, 9.4_
- [x] 3.2 Prove the plumbing: baseline program on the 12 pilot positives
- The baseline instruction equals today's hunter prompt plus the hunks; one live, budgeted run over the 12 positives only, as a pilot report run
- Done when the search metric over the new records is compared with the pilot's recorded hunter hits and the difference is explained in the measurement record
- _Requirements: 8.1, 8.4, 8.5, 8.6_
- _Depends: 2.2, 2.3_
- [ ] 4. Core: the training set, folds and ablation
- [ ] 4.1 Source the training set apart from the pilot
- About one hundred advisories through the fix-site catalog and verifier, pilot repositories excluded, at least 40 accepted positives across two languages, 20 ordinary pushes per positive from the same repositories; the sourcing run is recorded
- Done when the frozen training manifest lists its positives, negatives, languages and the pilot exclusion check
- _Requirements: 2.1, 2.2, 2.3_
- [x] 4.2 Fold by repository and audit every annotation for leaks
- Compile folds and held-out folds by repository group; the leak audit runs over every feature column and reports any that separates positives from negatives on its own
- Done when the folds are disjoint by repository and the audit report lists each column with its separation score
- _Requirements: 2.5, 2.6_
- [x] 4.3 Register and run a paired ablation
- A manifest fixes the table digest, the masks, the folds, the metric, the sample size, the budget and the decision rule before anything runs; each arm runs over the same changes with the same program and sampling
- Results per arm carry the paired effect on search recall and suspicions per ordinary push, per family and pooled, with intervals from the repository-clustered bootstrap and a paired test
- Done when an ablation over a fixture manifest with the scripted model yields two arms, the paired statistics, and the full recall-against-suspicions curve
- _Requirements: 4.1, 4.2, 4.3, 4.7, 4.8_
- [x] 4.4 Build the importance table and apply the keep/drop rule
- For every annotation, its effect when removed from the full set and when added to the baseline, with intervals and counts; an annotation stays only on a held-out effect whose interval excludes zero or when a kept annotation needs it; one that raises suspicions without raising recall is dropped and recorded
- Done when the table is written for a fixture ablation and the rule's decisions match the fixture's expectations
- _Requirements: 4.4, 4.5, 4.6_
- [x] 4.5 Contamination check
- A training repository later found in the pilot or in a benchmark report marks every run that used it as contaminated and excludes those runs from any gain
- Done when a fixture run is marked contaminated after its repository is added to a report list
- _Requirements: 2.7_
- [x] 5. Core: proposals and acceptance
- [x] 5.1 Propose candidates with the program runtime's optimisers
- One change per round: an instruction proposal, a demonstration set, or a mask; proposals score on compile folds with the search metric; the proposer never sees held-out results; rejected candidates stay visible to the next round
- Done when a round on a fixture produces a candidate with its compile-fold score and the proposer's inputs are shown to exclude held-out data
- _Requirements: 5.4, 6.6, 6.8_
- [x] 5.2 Accept or reject by the rule
- Compile-fold and held-out results per family; accepted only when both are non-negative, one strictly positive, no family below its regression budget, instruction under the length cap; ties rejected; the comparison is recorded in the new artifact
- Done when every clause of the rule has a synthetic fold-result test that flips the decision
- [x] 6.1 Run an accepted program once over the held-out pilot and report the gain
- Same scorer and metrics as the pilot; before-and-after of search recall, suspicions per push, push-level recall, false alarms per 100 and precision, per family, with intervals; the proof arm's workload per 100 pushes; the gate distance read off the curve; the run recorded next to the pilot's measurement, at most once per accepted program
- Done when the measurement record exists with the before-and-after table and the instrument proof for every change
# Brief: ir-consolidation — stop maintaining an IR we did not need, and adopt one we do
# Brief: ir-consolidation, stop maintaining an IR we did not need, and adopt one we do
> **Status 2026-10-02:** dormant. Brief only and unapproved. The subtraction it asks for still needs a measured recall comparison before the flat IR goes.
**Status:** brief only, unapproved.
@@ -10,16 +12,16 @@
Two different questions live inside that, and they have opposite answers. Keeping them apart is most of the
value of this brief.
## Question 1 — the source IR we already replaced and never retired
## Question 1, the source IR we already replaced and never retired
`semantic/` is **3,218 lines, 16% of the codebase**: a hand-written IR (`ir.py`, `cst.py`) and a flat taint
engine (`taint.py`), plus the overlay that runs them. `model-grounded-detection` measured that IR against
Joern's CPG and the result was decisive —**2.2% entailment against 41.3%** — which is why the CPG was
Joern's CPG and the result was decisive,**2.2% entailment against 41.3%**, which is why the CPG was
adopted for the model layer. The flat IR was never removed, and still runs as a proposer in every scan.
So the outgrowth is not something we are about to add. It is something already carried.
**Vine and P-Code do not help here.** Both are *binary* IRs — Vine lifts executables (BitBlaze, UC Berkeley,
**Vine and P-Code do not help here.** Both are *binary* IRs. Vine lifts executables (BitBlaze, UC Berkeley,
GPL-2.0, last significant activity 2009), and P-Code lifts machine code. Our source languages are Python,
PHP, JavaScript and Java, and the source IR we should adopt is the one we already have. The work is
subtraction, not substitution:
@@ -30,16 +32,16 @@ subtraction, not substitution:
A candidate ~1,000-line deletion with a number attached, which is the only kind worth doing.
## Question 2 — the abstraction we are missing, where an existing IR is exactly right
## Question 2, the abstraction we are missing, where an existing IR is exactly right
`c/memory` has produced **zero true positives** and three distinct classes of false one. The reason is
structural and now stated in the report (Req 5.6): it models unbounded string CALLS, while real C overflows
are *arithmetic*— an index, a pointer offset, a `memcpy` whose size is computed. Every one of libpng's is,
are *arithmetic*(an index, a pointer offset, a `memcpy` whose size is computed). Every one of libpng's is,
and its library code never calls a modelled string API at all.
"Can this expression exceed this allocation" is a value question, not a reachability one. **Rolf Rolles'
GhidraPAL carries a three-valued-logic abstract interpreter over Ghidra P-Code** — each bit definite 0,
definite 1, or unknown, with abstract transformers for `INT_ADD`, `INT_MULT`, `INT_AND`, the comparisons —
GhidraPAL carries a three-valued-logic abstract interpreter over Ghidra P-Code**, with each bit definite 0,
definite 1, or unknown, and with abstract transformers for `INT_ADD`, `INT_MULT`, `INT_AND`, the comparisons,
validated against concrete execution in its own tests. That is precisely the shape of analysis the C family
lacks, and it exists.
@@ -53,13 +55,13 @@ lacks, and it exists.
**angr is the strongest candidate and Vine is not one.** Three things decide it:
* **It is Python.** This codebase is Python with `dependencies = []` in the core, and every heavyweight
* It is Python. This codebase is Python with `dependencies = []` in the core, and every heavyweight
capability so far has been reached across a subprocess boundary rather than imported. angr fits that
pattern as an extra, the way `harnessx` and the tree-sitter grammars already do.
* **Value-set analysis is the abstraction we are missing**, by name. "Can this index exceed this allocation"
* Value-set analysis is the abstraction we are missing, by name. "Can this index exceed this allocation"
is a VSA question, and VSA is abstract interpretation rather than SMT -- which is the choice
`model-grounded-detection` already made and gave its reasons for.
* **BSD-2-Clause**, against Vine's GPL-2.0. This project has a standing rule about never vendoring code
* BSD-2-Clause, against Vine's GPL-2.0. This project has a standing rule about never vendoring code
without a stated licence, and a GPL dependency is a decision about the whole tool rather than about one
analysis.
@@ -69,19 +71,19 @@ thing to read before writing any abstract domain here, whether the platform unde
It is also the honest answer to a decision this project already took. `model-grounded-detection` chose
abstract interpretation over SMT, and chose to build the model from code because execution cannot be
assumed. A bit-level abstract interpreter over P-Code is *still abstract interpretation* and still static —
what changes is the artifact: it needs a **built** library rather than its source.
assumed. A bit-level abstract interpreter over P-Code is *still abstract interpretation* and still static.
What changes is the artifact: it needs a **built** library rather than its source.
## What this brief must decide
1. Whether the source-IR subtraction is safe, measured rather than assumed.
2. Whether the C arithmetic class is worth a second pipeline at all, or whether `c/memory` stays scoped to
string misuse and says so — which is where task 2.16 left it, deliberately.
string misuse and says so, which is where task 2.16 left it, deliberately.
3. If it is worth it: **angr** first, on language, licence and the fact that value-set analysis is the
named abstraction -- with GhidraPAL read as the worked example of a bit-precise domain rather than as a
platform. Vine is dormant since 2009 and is not a candidate.
4. What a second pipeline costs in the one currency this project has been strict about — **not lines, but
the number of things that can silently return nothing.** This session found three separate quiet failures
4. What a second pipeline costs in the one currency this project has been strict about,
**the number of things that can silently return nothing rather than lines.** This session found three separate quiet failures
in one PHP scan. A binary pipeline adds a build step, a lifter, and a domain, each of which can produce
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
