The plane's memory store no longer names one vendor. `MinioStore`/`SdkClient`/
`minio_settings`/`MINIO_KEYS` become `S3Store`/`S3Client`/`s3_settings`/`S3_KEYS`;
the URL scheme is `s3://<bucket>[/<prefix>]` (`s3://` alone uses `S3_BUCKET`); the
settings are `S3_ENDPOINT` (a full URL), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`,
optional `AWS_SESSION_TOKEN` and `S3_REGION`. The client is boto3 (extra `s3`,
imported lazily): SigV4, path-style addressing, SDK checksum trailers off, paginated
listing, versions and delete markers listed and deleted by version (`purge`, used by
the contract tests' teardown so the bucket keeps nothing of a run).
Behaviour is unchanged: the store never configures its bucket and verifies
versioning, the `runs/` expiry rule, S3 Select and tag reads at startup; no local
fallback; every row carries every queryable field with "" placeholders. The admin
hints read `--endpoint-url "$S3_ENDPOINT"`. The fake client still reproduces RustFS's
two Select failures (unbound leading-row field, null-typed column).
Docs: rustfs.md stays the tested server's setup page (policy JSON unchanged, env
names updated, any S3-compatible store with versioning, lifecycle, tags and Select
works); memory.md, ops/ax/README.md, README.md, steering and the two current spec
designs use the S3 naming. Retired specs and committed measurements are untouched.
Real server (RustFS, OUSAST_MEMORY_TEST_S3=1): test_plane_memory + test_learn_compile
75 passed, 1 skipped. Store copy of the live local store to a temporary prefix: every
count matched (776 row objects, 2982 rows, 22 facts, 788 excerpts, 2601 responses,
2 programs, 1793 index entries), 4190 versions purged afterwards.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
RustFS's S3 Select infers an object's schema from its first 1000 rows: a where
field absent there is unbound (EvaluatorBindingDoesNotExist), and a column null
in all of them but set later fails the whole object (JSONParsingError, measured:
999 leading nulls answer, 1000 fail). plane/memory.py now declares per kind the
fields a where may name (QUERY_FIELDS: facts candidates_digest, verdict
candidate/final, example profile, plus the ROW_FIELDS every row has) and writes
their union on every row at write time as strings, "" where one does not apply
(null measured to break). rows() refuses an unknown kind, an undeclared where
field and where = null before any read. `ousast plane memory-normalise
[--store URL] [--dry-run]` rewrites pre-rule rows once, idempotently, with
counts. The fake object client mimics both RustFS failures.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer decision "Neutralise both for v1".
1. Phantom candidates. A pair side's candidates are now only the functions that
side declares (plane/harvest.py `declares`, the excerpt builder's
function_bounds); a labelled function absent from a side is listed under
`absent_on_side` and never asked, and a side declaring none of its verify
family's functions is no verify unit. The recorded harvest units held 58
phantom verify candidates (53 fixed, 5 vulnerable sides). build_features.py
skips label rows whose side does not declare the function and drops their
earlier rows (MemoryStore.drop_rows); rebuilt the pair records from the
recorded signals, no model call: 126 label rows absent_on_side (117 fixed,
9 vulnerable) -- access_control 13, config_secrets 4, deserialization 20,
injection 28, memory 4, output_encoding 16, path 15, untrusted_destination
26 -- so 252 feature rows dropped (1712 -> 1460); no other row changed.
2. Engine withheld. Input profiles in learn/schema.py (`v1` withholds the
engine instrument, `full` withholds nothing); ProgramSpec.inputs defaults to
v1 and compile.toml records it. Neither the engine's state nor its values
reach the candidate, examples, demonstrations or the instruction proposer's
error summary, and the Gower retrieval distance leaves the engine block out.
Stored records are unchanged. design.md Risks records the decision and that
the engine returns only through a pre-registered A/B on assumed-benign and
benign-control negatives.
3. `ousast learn audit-leaks` (learn/leaks.py): per feature and per instrument
state, separation = (pairs the signal alone puts on the right side - on the
wrong side) / pairs, over all pairs and per family; flagged at >= 0.20 over
>= 10 pairs. Counts under
benchmarks/measurements/2026-10-01-decision-engine-leak-audit/ (before and
after the rebuild). After: facts.function_lines (0.80 untrusted_destination,
0.75 config_secrets, 0.26 injection; higher on fixed), verify.turns_mean
0.67 config_secrets, eng.* 0.35-0.40 output_encoding, verify.flag_a/votes/
site_offset and agree.final 0.20-0.30 on 10-12 pair families. Before the
rebuild the phantom fixed sides drove state.source/state.roles/ms.flagged
to 1.0 and state.engine to 0.95 on deserialization.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
`ousast plane harvest` (plane/harvest.py) writes two Runs over a label snapshot:
verify a/b + agree on the verify-family units (both sides of each labelled pair as
an opaque files-only Workspace, candidates outside the hunt's root, no declared
sites; v1 and dev-php pins with repo-facts first) and model roles on the candidate
files (pair excerpts grouped, one Task per population pin). Budgets sum under the
ceiling by construction. New roles task template; the generator re-pins it.
Spent by attribution: verify $2.54 (1961 calls, 150 units, 182 hunts per pass),
roles $1.40 (1034 calls, 799 files, 0 unclassified chunks); DeepSeek balance
19.63 -> 18.23. Engine on the pair corpus (benchmarks/learn/engine_pairs.py,
frozen e6ed785, one 3g container) is PARTIAL: 384 of 686 sides at commit, still
running; unfinished sides stay missing. benchmarks/learn/build_features.py built
1712 static+plane records into the memory store; counts-only record under
benchmarks/measurements/2026-10-01-decision-engine-harvest/.
source/roles parts are `none`, not `failed`, for a language without a declaration
pattern (C). ops/ax/README: Substrate golden-actor disk leak and the inline
Workspace limit (~20 KB).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/excerpt.py: the labelled function at its pin as numbered lines, bounded
(80 lines / 4,000 chars for candidates, 40 / 2,000 for memory examples),
centred on focus lines; advisory ids anywhere and comments with security
wording (labels.security_reason, multilingual) become <redacted>; owner,
owner/name, path and commit ids are replaced; a delta unit appends the
base->head diff (<= 40 lines). C/C++ and other brace languages without a
repo-facts declaration pattern use a brace matcher (346 c_cpp pair rows
had no excerpt without it).
learn/examples.py + `ousast learn memory build`: label + feature record +
excerpt -> `example` rows (no rationale), excerpts as content-addressed
blobs; refuses origin: user and sources outside sources.toml (exit 2);
conditional labels skipped; exit 2 when >10% of a source's recorded
candidates have no excerpt. A fixed side that was read but no longer
declares the labelled function is counted apart (absent_fixed_side): a
label defect, not an unread clone.
plane/memory.py: `example` in KINDS; put_blob/get_blob/blob_names on the
MemoryStore base, so FileStore and MinioStore share them (tested on both,
MinIO through the fake object client).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/sources.toml is the fail-closed source list: every read goes through one guard
that refuses an unlisted path or an [[excluded]] file name before opening it, and a
source whose evaluation record is missing cannot label. Population v3 is named only
as excluded, by file name, and is never opened (tested with a v3-named fixture and
open/read_text tracked).
learn/labels.py and `ousast learn labels` apply the design's rules: v2 declared
sites and v1 fix-range functions positive, the same function on the fixed pin
negative only when the fix changed it, introduce/repair delta units, benign control
pushes as conditional negatives; pairs vulnerable/fixed (unscorable and title tier
out, title as a sensitivity arm); dev-php recipe known functions; recorded
adjudications with their reasons. v2 rows are labels marked design_informed, as the
design says. Repository groups: normalised owner/name, merged by shared CVE/GHSA,
shared fix commits and [[alias]] rows.
assumed_benign (maintainer decision): ordinary commits of the population
repositories, >= 90 days from any fix, no later security commit touching their
files, weight 0.5, conditional, reported separately and never merged with verified
negatives. The security filter is multilingual (CJK, Korean, Russian, European and
other languages) plus CVE/GHSA/CWE references and dependency bumps; widened after
reading accepted pushes (validation, passwords, guards).
First snapshot, counts only (benchmarks/measurements/2026-09-30-decision-engine-
labels/labels.json): verified pin labels in positive/negative repository groups
injection 44/41, untrusted_destination 35/34, deserialization 20/20 (per-family),
output_encoding 15/15, path 15/14, access_control 12/12 (pooled), config_secrets 5/5,
memory 5/5 (insufficient). assumed_benign: 969 pushes in 27 groups from 36,000
commits scanned.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Quick mode had no PHP rules, so a PHP quick scan read the files and reported a plausible zero.
ruleset/php/rules.toml adds 13 line/statement rules built on semantic/php.toml's sinks, sources and
sanitizers (sql/wpdb, command, code_eval, unserialize, echo, file_inclusion, filesystem,
outbound_request, redirect). 9 are enabled and 4 are shadow (sql-call-composition, code-eval,
unserialize, xss-echo-input). A rule is shadow when it fired on the corpus but never at a known site.
precision_estimate is the measured lower bound.
Measured in-sample on the development corpus only (pmpro introduce/repair/benign, population-v1 PHP
cases; population-v2 was not read). Known site hit on the vulnerable pin in 4 of 5 cases (LearnPress
missed). The hit disappears on the fixed pin for pmpro and YesWiki. Numbers, per-pin files/bytes read
and the engine comparison are in benchmarks/measurements/2026-09-30-php-quick-rules/.
- fixtures php-vulnerable / php-benign plus manifests: ousast benchmark covers PHP
(9/14 reported, 13/14 including shadow; the benign twin records 2 known false alerts)
- PHP `#` comments are comments; `#[...]` attributes are not
- the alerts task now reports PHP as quick coverage; alerts-engine keeps running the engine for PHP
next to the quick rules (ENGINE_ALONGSIDE_QUICK)
- other languages' benchmark findings and metrics are unchanged
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Quick mode has rules only for c/cpp, groovy, java, javascript/typescript and
python, so every PHP case's "0 alerts" read as clean. The `alerts` task now
records per language of the files read whether quick mode covers it
(derived from the bundled ruleset, not a list): coverage quick | none, with
file counts per pin, and `uncovered`. `remember` turns that into `coverage`
memory rows (a new kind; the Run passes the alerts summary.json as input),
loop-measure reports coverage per language, and improve/memory.py's M1/M2
skip a repository+pin whose coverage for the rule's language is only
`none` (counted under `coverage_none` in the proposal's excluded).
`ousast plane alerts-engine <Run.yaml> --source <frozen export>`
(plane/engine_alerts.py) does the alerts task on the host for cases whose
files include a language the engine covers but quick mode does not (PHP):
exports both pins from the case cache, runs the quick scan and the
openultrasast:dev engine container (finding_dump.py, --network none,
--memory 3g, one at a time, per-pin deadline), writes engine findings in
the alerts.jsonl row shape (rule_id engine:<family>, source engine,
in_fix_range from case.json) with a summary (files read, questions,
completed, seconds, degradations per pin), and marks the task done via
reconciler.mark_done. No result, no files read, or questions == 0 fails
loudly and leaves the run untouched. Refuses a git checkout as source.
Runner image rebuilt and re-pinned (sha256:ebb40a9a...); validation-46
regenerated (digest only). Live, recorded in
benchmarks/measurements/2026-09-30-php-engine-alerts/: froxlor 366 files,
130/130 questions per pin, 6 alerts per pin, 0 in the fix range; wp-gopay
19 files, 101/101 questions, 0 alerts. The loop: 32 run rows (coverage 8,
php=engine), 0 proposals, gate no_proposals, $0.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The standard scan now runs run_hunter_pool, verify_findings and the
deterministic fuse_findings directly; the HarnessX hunter orchestrator,
the LLM judge dispatcher (verify_judge.py, deleted with its test) and the
LLM fusion panels are gone, and so are their extra-absent degradations.
host_under_harnessx is dropped from stage_processors.
config: [models] verifier and [fusion] panel_model/decider_model raise
RetiredConfigError naming the plane replacement (the CLI prints it and
exits 2); a [harnessx] section loads with one warning and is otherwise
ignored. harness.json keeps "verifier": null. Tests that asserted the
fallback are renamed to the behaviour they assert; the module-audit
manifest drops verify_judge. Nothing outside the two modules deleted in
task 6 imports harness_ext or hunter_harness.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
`ousast plane workspaces --validation-set ... --loop` appends the loop to a Run:
per case `alerts` (plane/tasks/alerts.py: the quick-mode scan on the vulnerable
and the fixed pin, a fixed-pin Workspace per case, alerts attributed to their
enclosing function by the repo-facts patterns, `in_fix_range` from the fix's
old-side and new-side ranges, failing on an unread tree) feeding `remember`,
then `memory-snapshot` -> `loop-measure` -> `loop-propose` -> `loop-improve`
(plane/tasks/loop.py). Every new task binds no Model and has budget
{usd: 0, calls: 0}. The snapshot is the store's rows after the train-on-test
guard, written on the host by memory.seed (a sandboxed run of it fails loudly);
propose runs M1/M2 against this repository at the commit under test; improve
runs one unchanged evolve.run_round and delivers gate.json, journal.json, the
sidecar, rule_policy.json only when accepted, and proposal_outcome rows that
the host ingests. Adoption stays a maintainer commit.
`remember` is now a Run entry per case (the image contains it), with the facts
row keyed by the image the facts were computed on (repo-facts records it in
its summary; unknown -> the row is omitted, never guessed). Alert rows carry
their own pin; improve/memory.py counts a repeated alert once.
Runner image rebuilt (+0.39 MB: only the new modules; the quick scan needs no
extra and the rulesets were already in the package), pushed, plane/ re-pinned
and regenerated (validation-46: 105 entries). Fixture test runs the whole loop
graph through the fake ax, whose new `exec` mode runs the real task modules.
Live: one loop Run over three validation-46 cases on the local ax, recorded in
benchmarks/measurements/2026-09-30-harnessx-removal-loop/ (all done, $0 new
spend, 0 proposals, gate verdict no_proposals).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
improve/memory.py: rules M1 (enabled->shadow on repeated false alerts across
>=2 repositories, no agreed or site_match hit) and M2 (shadow->enabled on
>=2 agreed declared sites no enabled rule catches); M3 disputed families are
advisory (signals.json) and never reach the validator. The train-on-test
guard drops rows of --qualify-population, gated benchmark cases and holdout
pair repositories before any rule sees a row. run_round(proposals=...) adds
memory edits after the benchmark's through the unchanged validator and gate;
provenance (row ids, object keys and versions, counts, exclusions, index
digest) goes to memory_proposals.jsonl and the journal's evidence field;
proposal_outcome rows are written back after a non-dry run.
ousast improve --memory is opt-in; default improve output is byte-identical
to the task-1 baseline.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
plane/memory.py: the MemoryStore interface with FileStore (index.jsonl, facts by
content hash, repos/<host>__<owner>__<name>/<pin>.jsonl, idempotent ingest,
deterministic row ids, refusal below 1 GiB free naming the path) and MinioStore
(object metadata and tags, S3 Select pushdown probed once per process with a
logged local fallback, object version ids on every row read, versioning and
runs/ lifecycle helper, presigned URLs). OUSAST_MEMORY selects the backend;
MINIO_* settings come from .env; the minio SDK is the optional extra `minio`.
plane/tasks/remember.py: model-free rows (facts, verdict, unit_cost, alert)
from a case's artifacts; remember_run computes them on the host for a recorded
run without writing into it. reconciler.mark_done (477 lines total), seed
before and ingest after `ousast plane run`, `ousast plane remember <run>`.
The generator annotates each repo-facts Task with openultrasast.io/memory-key
(repo, pin, candidates digest, image digest) and the Run with its population
and split; plane/ regenerated.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
agree matched declared sites through benchmarks/independent/evaluate.py,
which the runner image does not carry, so site matching would have been
"not assessed". The generator now resolves the fix's old-side ranges
(evaluate.hunks, line for line) into case.json, with the declared sites in
the set and the reference's cost, and agree.matches_declared reproduces
matches_v2; a parity test runs both over the recorded outputs of the 15
validation cases (200 probes, identical verdicts) and agree on those outputs
returns the reference's 16 of 20 and $0.022.
agree reports cost_per_candidate (plane spend) and
cost_per_candidate_with_recorded_triage; Requirement 6.2 is judged on the
latter (design.md). --runner-image re-pins the templates from ops/ax/up.sh's
pin file; the committed manifests are regenerated at the current digest.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Model deepseek-flash (declared provider extension, endpoint prices in
parameters, secretKey naming the DEEPSEEK_API_KEY secret), digest-pinned
Task templates repo-facts / verify / agree, and a generator
(plane/generate.py, `ousast plane workspaces --validation-set ...`) that
emits per case the vulnerable-pin Workspace, a files-only inputs Workspace
(recorded candidates after the recorded triage, declared sites, triage),
per-case Tasks and one Run validation-46: facts -> verify a, verify b ->
agree for the 15 cases, 46 candidates (43 kept), budgets 1.5x the recorded
per-pass share. A Run entry cannot add a binding or env, and Run.inputs
names only task artifacts, so the case is fixed by per-case Tasks.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Requirements 3.1, 3.2, 3.3, 3.4, 3.5, 2.2, 7.1, 7.3 of ai-service-plane.
- plane/reconciler.py (492 lines, test-counted): a Run's tasks submitted to ax
in dependency order through the injectable `ax` CLI (apply / get task /
delete task), serialize labels never overlapping, --workers, state.json with
pending|running|suspended|done|failed|unfinished, a PID lock per run, done
tasks skipped on rerun, unfinished tasks holding their dependants, a failed
task halting the run. Budgets, inputs, output dir, artifact URL and the bound
Model (annotation openultrasast.io/model; identity and priced parameters) go
to the task as env in the rendered Task; consumed artifacts travel as files
of a generated Workspace bound under /workspace. A threaded HTTP receiver
takes each task's tar delivery, validating run/task and rejecting traversal.
Outcome comes from the delivered summary.json (done only when every unit
finished), never from ax's phase alone. status prints the token attribution
table per task and Model from the summaries' usage fields (n/a, never an
estimate) and writes attribution.json; doctor runs the four host checks.
- plane/workspaces.py: one Workspace manifest per case pin of a population.
- CLI: ousast plane run|status|doctor|workspaces.
- tests/test_plane_reconciler.py: a fake ax executable drives the DAG;
tests/test_plane_ax_live.py (marker ax) applies ops/ax/smoke-task.yaml on a
live cluster and skips unless doctor passes.
- Module audit rows and regenerated manifest.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
b4b9308 was committed with two failing tests: the module audit read
`from . import profiling` as an import of the package, called profiling
orphaned, and the committed audit manifest drifted. My check read tail's
exit status, not pytest's. Now a from-import; manifest regenerated (109
load-bearing, 0 orphaned); 1,444 pass with pytest's own exit checked.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The cache fsynced every payload and manifest -- two per entry, ~18 ms, and a
push publishes one entry per question (~22,000 on a WordPress plugin). A
read already rejects any entry whose manifest receipt, payload size or
payload SHA-256 does not check, so a torn write is a miss, never a wrong
answer; the fsyncs bought nothing the verification does not.
The broader problem: three quadratic walls in this replay were found one
per two-hour run, because all were Python bookkeeping between engine calls
and the only record was the engine log. `OUSAST_SAMPLE_PROFILE=<path>`
starts an in-process sampler that rewrites the hottest stacks every 30 s,
so a run killed at its deadline still names where its time went. The host
cannot ptrace the container, hence in-process. The replay harness turns it
on per case.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The 13.3 replay on 7cf256b crashed on case 1: `pre-push` ran past its
7,200 s deadline plus the harness's 60 s grace, `subprocess.run` raised
TimeoutExpired, the harness died without running cases 2 and 3, and the
case's captured output died with it. Each case now streams stdout and
stderr to files with INFO logging on; an overrun kills the case's process
group, is recorded as `deadline_overrun` with its size -- the product
failing its cancellation contract -- and the next case still runs.
`--only` reruns selected cases. CLI log lines now carry a timestamp, so a
log can say where the time went.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The stage costs added for PHP diagnosis never appeared: nothing in this project
configures logging, so the default root level of WARNING dropped every
informational line and a PMPro case ran for an hour producing none.
That is the same failure shape the project keeps relearning. A run that reports
nothing is indistinguishable from a run with nothing to report, and it is how four
PHP investigations were spent reading a later stage's timeout as the cause.
Logging is now configured from OUSAST_LOG_LEVEL, quiet by default and sent to
stderr when asked, so a hook stays silent while a diagnostic run says where its
budget went. Verified end to end: six stage lines with durations on a real replay.
1,410 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The M1b exit run established the comparison but could not show a developer-readable
consequence or repair direction, because both belong to an evaluated capability
declaration and the installed registry is deliberately empty.
The experimental evaluation output now renders, for each change-attributed finding,
its family, exact location, witness, change attribution, provenance, engine method
and operation, together with a consequence and repair direction produced from a
supplied declaration through the same grounded-template path the admitted case
uses. No prose is invented: when no declaration matches, or a template is not
grounded in the operation and context, or the declaration does not cover the
operation symbol, the explanation is absent and the reason is stated.
Declarations reach this rendering only. The runner never passes them to admission,
so nothing in that file can enable a capability or emit an alert however it is
written, and a control asserts that even a declaration claiming PASS admits
nothing. The flag requires explicit base/head replay and the recorded-veto flag.
The evaluation instrument generates its own unreviewed declaration bound to the
run's analysis semantics, asserting that it is neither enabled nor PASS, so the
demonstration cannot drift into a qualified claim.
1,377 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Add group 9 to the approved task plan as the bounded owner amendment release
milestone M1a requires, implement the task-local recorded-veto evaluation flag,
and run the three pinned NodeGoat revisions under it.
The flag re-runs the production comparison and admission over a lifted copy of
the same evidence and records each veto instead of applying it. Recognized
completeness vetoes are lifted and logged; failures never are, because silence
from a failure is not a quiet fixed twin. It is off by default, accepted only on
explicit base/head replay, refused in Git stdin mode, labels every payload it
produces, and admits no defect even when a lifted candidate otherwise would.
M1a outcome, inside its one-week box and with expectations frozen on disk first:
the three entailed <lambda>2 witnesses are present at contributions.js 32-34; the
fixed twin was genuinely analyzed and quiet, answering its target question with
zero rows and no deadline reason while the same findings appear as base-only; the
unchanged revision was analyzed and recorded unchanged. The kill criterion did
not trigger.
Five vetoes fired and were recorded. M1a also corrected the diagnosis: with all
five lifted the detection still stops at operation_correspondence_unresolved,
because a newly introduced operation sits on a changed line and has no base
anchor. The unchanged revision resolves its base location on the same lines and
classifies as unchanged, which isolates correspondence as the mechanism. That
fifth veto was not in the earlier four-veto diagnosis and is now M1b's central
repair.
Diagnosis only. No defect admitted, no capability enabled, no gate changed, no
latency claim; the 900-second development budget cannot measure the hook gate.
Rollout and task 8.3 remain NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A dropped file is dropped from the ONLY graph there is, so every question
about it has no answer -- which is how a 637-file plugin became a scan of
nothing. But those files build perfectly well alone: the failure is
per-invocation, not per-file. So a file the main build refuses is excluded
from it, given a second CPG, and both are queried with rows concatenated per
request id.
pmpro two-file slice before: cpg_empty, 0 findings
after: 18 entailed incl. CVE-2023-23488 at :936,
reported with cpg_sharded
What this does NOT buy is a flow that crosses the shard boundary: a source in
the excluded file whose sink is in the rest of the tree is invisible to every
question asked, and no merging of results recovers it. That is why the split
is REPORTED rather than quietly served as if it were whole.
The excluded-file tree is symlinks at the original relative paths. A region
asks about `classes/class.memberorder.php` and the queries match a filename by
suffix, so a flattened copy would cheerfully answer about a file nobody asked
about.
Reports gain "What could not be analysed", rendering cpg_sharded, cpg_empty,
files_unparsed, query_failed and budget_exhausted in the reader's terms. A
degradation is a fact about coverage, not a warning to file away: silence in
the coverage table means "no fact table covered this", and silence about an
empty or split graph means something else entirely.
Also removes the WordPress vocabulary from the query, which was the other half
of this request. `apply_filters` was a hardcoded Set in taint.sc and
`add_action|add_filter|add_shortcode` a hardcoded alternation in mapping.py.
Both are now a `[[dispatch]]` fact:
[[dispatch]]
id = "wordpress_hooks"
register = ["add_action", "add_filter", "add_shortcode"]
apply = ["apply_filters", ..., "do_action", ...]
The shape is general -- a registry keyed by a string literal at both ends, as
Django signals, jQuery events and Symfony's dispatcher all are -- so a second
framework is a row in a TOML rather than an edit to a query. `dispatch_apply`
is empty for every language that declares none, and the hook half then does
nothing at all rather than looking for names that mean nothing there.
848 tests pass; gates byte-identical. wpstatistics 3/2 with the CVE at :225 on
the vulnerable side only, mwwpform 1/0, pmpro 18/1, VAmPI 4 unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.9. Task 2.4 let a region's question follow the call
graph, and the moment it did, many entry points could reach one shared
sink: libpng reported the same site twenty-six times. A contributor shown
the same defect twenty-six times learns to scroll past it.
Findings are deduplicated on (site, family), keeping the strongest rung
and counting the regions that reached it -- "reached from 26 regions" is
the useful half of that observation and one row is the readable half. Two
families at one site stay two findings, because `echo $_GET[...]` is an
injection question and an output-encoding question with different answers.
The duplication that motivated this was removed by another route: the 26
rows were all the `fgets`-matched-as-`gets` false positive, and the sink
matcher fix took them out. So this changes no current number and is a
structural guarantee rather than a measured reduction -- the mechanism that
produced them is untouched, and only the finding that made it visible is
gone.
+25 lines for a guarantee, in a session that has been asked to watch its
growth. The audit is unchanged at 97 modules, 90 load-bearing and 7
standalone, with no orphans.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.16, implementing the approved Req 5.6. `c/memory`
scored zero true positives across 89k lines of C and three distinct
classes of false one, for a structural reason: it models string CALLS and
libpng overflows through arithmetic.
The task allowed retiring the family. Kept, scoped and stated instead --
`strcpy` misuse is real in plenty of C and the family earns its place for
that. Retiring it would have removed the disclosure along with the
coverage, leaving a C scan silent for a reason nobody could read.
**Its own description was the quiet-failure risk.** It claimed "bounds,
lifetime, arithmetic", which is exactly what it does not decide. It now
names the APIs it models, and a new `limits` field in families.toml says
what it cannot: an overflow expressed as an index, a pointer offset, or a
memcpy whose size is computed -- which is how most real C overflows are
written, and how every one of libpng's is.
A limit recorded in a fact table nobody reads is no better than silence, so
the report renders it. **What was analysed** lists every family offered to
a region whether or not it found anything, because a family that ran and
reported nothing is the case a reader is most likely to misread, and it
states plainly that silence means no fact table covered the code rather
than that the code is safe.
The counts are derived from the fact tables the arbiter actually used, so
they cannot drift from what the engine looked for; only the limit sentence
is authored, because "cannot decide an overflow expressed as an index" is a
statement about an abstraction and no table can produce it.
A C scan that finds nothing now says:
| c | `memory` | 121 | 9 |
**`memory` (c)** — Models CALLS, not arithmetic...
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.10. Every one of libpng's twenty-five entry-point
regions was a `main()` -- twenty-three under contrib/, plus example.c and
pngtest.c -- and those regions carried the fifty interprocedural taint
requests that dominated a 300s query phase. A library has no entry points,
so entry-point-driven analysis had nothing to anchor to except its own
example programs.
The task said to derive this rather than hardcode `contrib/`, and a project
already declares it. Makefile.am names installed targets in
`lib_LTLIBRARIES` and `bin_PROGRAMS`, each target's `<target>_SOURCES`
names its files, and `check_PROGRAMS`/`noinst_PROGRAMS` are automake's own
word for "not installed" -- exactly the distinction wanted. Taking every
`_SOURCES` would have called libpng's whole test suite shipped code, since
contrib/libtests declares its sources the same way the library does.
interprocedural regions 25 -> 1
taint query 300.1s -> 125.0s
total query 311.9s -> 138.5s
regions 121 -> 121, unchanged by design
The surviving interprocedural region is contrib/tools/pngfix.c:main, which
libpng genuinely installs.
Two rules keep it honest. **Silence is not exclusion**: a project with no
build file this can read gets None back and nothing changes, so vampi's
numbers are untouched. **Not shipped is not unscanned**: those regions are
still arbitrated function-locally, because a contributor editing a test
must still get an answer -- what they lose is the expensive treatment and
first claim on the budget.
Entailed findings went 0 -> 3, and all three are false positives at
contrib/gregbook/wpng.c:374, `strcpy(outname+len, ".png")`. It is safe:
outname is char[256] and line 325 guards with
`if ((len = strlen(inname)) > 250)`. They were not introduced so much as
uncovered -- a different callDepth meant a different sink set, and the 26
fgets false positives were masking them.
The structural cause is recorded as 2.12: c/injection and c/memory carry
ZERO sanitizers, so no C flow can ever be discharged and every
`argv -> strcpy` entails however carefully it is bounded. The discharge
here is not a call but a guard that dominates the sink, which is precisely
what dominance.sc already does for access control and is wired to that one
family.
Also adds Group 7, five tasks for stopping the same false positive twice.
Two of today's three false-positive classes were one bug in three
matchers -- `resolve` in `resolveUrl`, `user` in `users`, `gets` in
`fgets` -- each fixed locally and each next one surviving because nobody
carried the fix across. A model may group counter-examples by cause and
draft an edit; only a split-respecting measurement may adopt one, because
the predecessor's improve lever admitted shapes taught by the holdout pairs
they then recovered and turned a -0.059 Youden into +0.059.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.2 for libpng and the 2.3 decision it exists to inform.
89,082 lines of C across 93 files, under a 2GB heap cap:
CPG build 16.1s (25s cold)
Joern queries 250.0s
arbitration 122.0s (105 calls, $0.0122)
whole scan 446s
peak RSS 911MB
regions 121, all arbitrated, 32 unasked
CVE-2025-66293 out_of_scope, as the recipe declares
Memory is not the constraint a fleet runner would hit: 911MB leaves room
for the rest of the job on a 2-vCPU/7GB runner. The memory pressure seen
while building this was concurrent JVMs on one desktop, not repository
size.
**Server mode: no.** The query phase splits taint 243.3s / config 10.7s
over two invocations, and fixed overhead measured on the same CPG is ~13s
per invocation. So ~26s of the 250s is JVM startup and ~224s is CPGQL
evaluation: a warm server removes 6% of the scan and none of the part that
costs. The driver now records query seconds per kind, because a single
total cannot tell startup from evaluation and the two answer this
oppositely.
The small repository pointed the other way. vampi's 34.3s over three kinds
is roughly three startups and almost no evaluation, which made a warm JVM
look worth having; the ratio inverts with size. That inversion is why the
task was sequenced after a real repository rather than decided on a
520-line file.
On whether one whole-repository CPG is the only shape for a horizontal
fleet: no, and it is also not where the pressure is. The build
parallelises and costs 16s; the query does not shard and costs 250s, so
splitting a repository across runners divides the cheap phase -- and after
2.4 a shard boundary hides exactly the cross-module flows that matter.
Scoping to a contributor's diff cuts the expensive phase instead. 911MB
for 89k lines extrapolates to roughly 5GB at a million lines, which would
exceed a standard runner, so the ceiling is real and further out than this
checkout can locate.
Two defects recorded rather than fixed, both invisible at vampi scale:
2.9, all 26 entailed findings are the same site reported 26 times, because
regions that share a sink each emit their own; and 2.10, 22 of the 30
highest-ranked regions are libpng's `contrib/` example programs rather than
the library.
Also ignores Joern's `workspace/`, which a maintainer querying a CPG by
hand from the repository root will create.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.2. VAmPI at a pinned commit, scanned whole rather than
as excerpts, with benchmarks/repos/measure.py recording build time, query
time, arbitration time, peak memory, cost, unjudged regions, and whether
each known vulnerability was found and at which rung.
The corpus could not have found any of this. Every defect below needs a
repository to become visible, and three of them were invisible behind a
passing test suite.
**The batch path was unreachable.** query_batch was implemented, tested and
committed while CpgResult exposed only `run`, so the driver's capability
check found nothing and fell back to one JVM per region-family: the first
attempt at this measurement ran eighteen minutes on 520 lines, issuing
`--param function=` one at a time. Every test called query_batch directly,
so none of them noticed. CpgResult now carries run_batch and a test asserts
the capability is reachable through the object the driver is handed.
The earlier 275s->106s improvement attributed to batching came from region
deduplication alone.
**The budget stopped free work.** It bounds model calls, but was enforced
by breaking the region loop -- spending a paid limit to stop unpaid CPG
arbitration. 6 of 22 regions were examined and VAmPI's BOLA, which the
graph entails for nothing, was never reached. It now withholds the judge
only. 22 of 22 regions arbitrated, 17 counted as regions_unasked (silent
because unaffordable is not silent because decided), and the BOLA is
entailed by the model layer.
**A region with no function asked about the whole repository.** Sending
function="" matched every hit in the tree and the driver attributed that
answer to whoever asked: one `host='0.0.0.0'` in app.py became eight
identical entailed findings in eight files that do not contain it. Queries
now carry the region's file. Entailed findings 14 -> 7, query time
70.9s -> 40.9s. The single-request form stays unscoped so the committed
pair measurements remain reproducible.
**The JVM had no heap bound.** Joern claimed a quarter of physical RAM and
runs were killed under memory pressure at different points, so the numbers
described the host rather than the engine. -J-Xmx2048m by default,
OPENULTRASAST_CPG_HEAP_MB to raise it.
Also: arbitrated findings carried `function:line` where the report layer
read `path:line`, so every entailed finding reached a contributor with a
function name where its file belonged; and the witness line was parsed by
splitting on a comma, which produced sites like
`config.py:17): permissive literal '0.0.0.0':?`.
Reading: GO on one CPG per repository -- 6.2s to build, 177s end to end,
869MB, $0.0067, nothing infeasible. NO-GO on region-scoped taint as the
only shape: VAmPI's SQL injection runs from a handler parameter into a
function in another module, and a per-function region sees neither end.
That is the ordinary shape of a real injection bug, so it is recorded as
task 2.4 along with three smaller defects this run exposed.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The pair corpus scores excerpts: a vulnerable function and its fixed twin,
handed to the engine with the answer already narrowed down. Nothing in it
exercises a repository -- region ranking, the budget, one CPG over
thousands of files, or whether a verdict that holds on fifteen lines still
holds when the source, the sink and the guard live in three modules.
benchmarks/repos/<name>.toml names a real checkout: url, a full 40-char
commit SHA, licence, and every known vulnerability with its file, function
and line. `ousast repos` lists them; `--fetch` materialises one; CI does
neither, and conftest clears the switch and redirects the cache so a
developer with warm checkouts cannot read them by accident.
Three rules the loader enforces rather than documents:
- A branch or tag is rejected. Either would make the same recipe mean
different code next month while the baseline it produced kept the old
numbers.
- `in_scope` must be stated per vulnerability, never defaulted. It decides
whether "not found" is a miss or a non-question, which is the most
important thing about the result.
- A recipe that declares `detection` must name something in scope to
detect.
`ousast repos` does not trust a recipe either: against a materialised
checkout it checks HEAD against the pin, that the licence file is there,
and that each declared function appears above its declared line. That
earned its place immediately -- the first libpng recipe named LICENSE.md,
which is the filename at HEAD today and not at the pinned commit, where it
is still LICENSE.
Two checkouts, verified against the real repositories:
- vampi (MIT, 520 lines) is the DETECTION checkout. Excerpts of it are
already in the pair corpus, including the update_password miss diagnosed
earlier, so a whole-repository run says directly whether a verdict
survives the move off the excerpt. Its routes register through an
OpenAPI operationId, so it exercises registration-based entry-point
mapping rather than the decorator path.
- libpng (libpng-2.0, 91k lines of C) is the ENVELOPE checkout, pinned at
the parent of the commit that added the missing clamp. Its CVE is marked
out of scope: an out-of-bounds read is not a family this engine
arbitrates, and recording "not found" against it as a miss is the
easiest way to make a measurement lie.
Nothing is vendored; the recipe records the licence so an artifact can say
what it ran against.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
`model-grounded-detection` finished 16/16 with the model layer having zero
production callers. This closes that. A real scan of a Flask file now reports:
by_rung: {"model_entailed": 3, "model_corroborated": 0, "suspicion": 0}
model:injection:ping:10 <- os.system("ping -c 1 " + request.args["host"])
The stage is the fifth MAP proposer, mirroring _check_obligations, and is
additive: pattern, overlay and obligation findings all keep working, and without
a CPG engine it records cpg_unavailable and changes nothing else. Two scan
integrity tests pin the exact degradation set and both broke -- correctly. Their
`requested == baseline` assertion, that findings are byte-identical, still
passed, so the only change is that the scan now SAYS the arbiter was absent
instead of being silently without one. The guards are updated to assert the
model degradation specifically rather than being loosened.
ModelLayerConfig, not ModelConfig: that name is already the [models] block, and
the type checker caught the collision immediately. Two meanings of one name is
how a scan silently reads the wrong settings.
Every payload carries the same shape, skipped ones included. A payload that is
sometimes a full record and sometimes a single "skipped" key is how a report
ends up quietly omitting a section.
unjudged_sample names the lowest-ranked regions the budget did not reach.
Counting them is the honesty requirement; naming them is what lets a maintainer
write a large-repository exclusion list from evidence rather than guesswork.
The run also produced the finding behind new task 1.4: 275 seconds for a
ten-line file. cost_usd is 0.0, so those 21 invocations were not model calls --
they were Joern query launches, one per region per family, each a JVM start.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
99 source modules -> 81. 832 tests -> 646. The whole learning/ package (16
modules, ~4,100 lines) and six semantic/ modules removed, with the three gates
byte-identical to 2026-09-06-gate-baseline.txt at every step.
The sequence is what made it safe: vocabularies ported first (1.1), importers
detached second (1.2-1.3), deletion third (1.4). Every import break during the
deletion was therefore a detachment miss in the importer and was fixed there --
no module was restored to satisfy one. Both misses were the same shape and were
fixed as a class: operations.py and shapes.py each imported trailing_name from
the deleted variants.py, so it moved to semantic/ir.py, its natural home, and
both importers changed in one edit.
Ported before deleting: the closed guard vocabulary (verbatim, nothing else
holds it) into model/specs.py, which derives TaintSpec and DominanceSpec from
the retained ruleset facts rather than copying them. Re-homed: families ->
model/taxonomy, candidates -> model/candidates, endpoint -> model/endpoint,
plus Region, labeled_family and labeled_spans. labeled_family no longer falls
back to the LLM classifier -- an unlabeled pair is honestly unknown.
Four deviations, each recorded in the spec notes with its reason. improve/
evolve.py and validator.py were stripped, not deleted: each mixes the mechanism
lever (gone) with the deterministic rule-status loop `ousast improve` still
runs, which has no LLM in it; a test now asserts the lever's absence by symbol
so it cannot creep back. The learning CLI group became `model candidates`,
because deleting it outright would have orphaned the retained enumerator by
removing its only entry point. thinking moved from [learning] to [models].
A gap named rather than papered over: no retained sink fact routes by CWE to
path, output_encoding, untrusted_destination or prototype -- the same hole the
entailment ceiling diagnosed. The test asserts the seeded set and the gap set
exactly, so task 4.2 must update it when it authors those sinks.
The audit found exactly one genuine orphan: vectorstore.py, a bakeoff harness
with no production caller and no CLI, alive only through its own test. The five
named subsystems each carry an explicit decision. Manifest committed, with a
test asserting the tree still matches it.
Group 2 (the Joern go/no-go) has not started.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The meta-agent round journalled `no_proposal`. The real cause was an
authentication failure: `_model_config` hard-coded the anthropic provider
while the detector was talking to DeepSeek and no ANTHROPIC_API_KEY exists
here. The composition itself was fine -- MetaAgent constructed, evolve
started, budget $5 -- and the trace was thrown away.
`no_proposal` covered two different worlds: a proposer that considered the
facts and declined, and one that could not start at all. The journal now
carries the proposer's own reason, and the vendor comes from
`[harnessx] provider` instead of being assumed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The 176-pair vfc baseline hit its four-hour timeout and wrote nothing. Scores
were accumulated in memory and persisted once at the end, so roughly four
hours of paid API calls were lost. I fixed the exception path this morning --
a provider hiccup costs one pair -- and missed that a timeout, a kill or a
machine going away cost all of them.
Each pair is now appended to `scores.jsonl` and flushed as it is measured, and
`--resume` keeps what a previous run of that exact baseline already bought.
This also gives the per-pair outcomes the round-3 review asked for, which make
the noise floor auditable after the fact instead of only in aggregate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Group 1 complete. Tasks 1.2 and 1.3: `candidates.ceiling`, `ousast learning
candidates`, and the stop-or-go reading. No model is called anywhere in this
group, so the condition that could have ended the feature was reachable for
free.
Verdict: go.
agent-vfc (deciding) 89.7% ceiling against 44.4% round-zero detection
vfc (deciding) 84.7%, baseline still running, non-gating
vfc-js (deciding) 52.9% -- see below
github (deciding) 100.0%
vibe-py (development)100.0% against 83.3%
The margin is largest on agent-vfc, which is the slice the tool is actually
for.
The measurement forced a shape correction before the detector existed.
config_secrets first read 63.6% with a bind-and-literal shape, because a
permissive default is as often an argument to a call -- cors({origin: '*'}),
app.use(session({secure: false})) -- as it is a binding. Adding call sites to
that family's shape took it to 100% and recovered four agent-vfc rows.
Carry-forward, and not this feature's to fix: vfc-js sits at 52.9% because
eight of seventeen rows label a function the JavaScript CST names `<anon>` --
`module.exports.publish = function () {}` and its relatives. That is a
function-naming gap, it caps the existing hunter on that slice exactly as
much, and it is the same class as the `_declarator_start` defects found in
learning-harness task 4.1.
The ruleset arm of `ceiling` is kept so the 12.1% that rejected it stays a
measurement rather than a remembered number.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
records whether its decision beat noise
Two things the maintainer was right about.
Thinking was disabled so `temperature: 0` would not be silently ignored --
a trade that only pays if it buys determinism. Round zero says it did not:
11 of 20 injection pairs disagreed with themselves across five runs, so the
variance is in the agentic path, not the decoder. We paid the reasoning cost
on a reasoning model and got nothing. It is now `[learning] thinking`,
recorded in the artifact and part of its key so the two modes cannot be read
as one number. The default stays off only because that is what round zero
was measured under.
And the deeper point: the spec built `sign_test` and `reliable_change` and
the acceptance rule calls neither -- `decide` compares raw deltas. On a
detector with that floor a raw delta is mostly noise, and a rule that
brittle is exactly what makes a deterministic decoder look necessary. It is
not; the alternative is an acceptance rule that models the score as noisy,
the way reflective-evolution and Bayesian prompt optimizers do. Changing
Req 9.5 is a maintainer's call, so this does not change the decision: every
round now journals the two-sided sign test over the pairs that moved, and
after four rounds the journal will say whether any decision could be told
from a coin flip.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Keyed by model alone, the memory-safety run over the vfc slice silently
replaced the run over the web families, and the published table would have
shown one of them as if it were everything. The artifact key and the
comparison label now carry the slice selection when it is not the default.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The chat client meters itself and nothing passed the meter to the baseline
or the round, so every published cost would have been 0.00 -- and the
round's cost cap, which reads the same meter, could never have fired.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
`--slice vfc` selected the vfc rows and `run_baseline` filtered them all out
again against its own default of the two web slices, so the memory-safety
baseline Req 8.4 asks for produced an empty report and said nothing about it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Three defects, two of them the same shape as ones already rejected once:
- The meta-agent proposer still could not propose. `MetaAgent(inner_model=)`
takes a ModelConfig; it was handed a model name, a `# type: ignore` hid
that from mypy, and a bare `except Exception` turned the resulting
AttributeError into `meta_agent_failed` -- which reads exactly like the
model declining. The composition is now a seam a test can pin, and a
construction error is re-raised rather than reported as a quiet round.
- `ousast pairs --hunter` crashed in its documented form: the K floor landed
with `--k-runs` still defaulting to 1. The covering test always passed
`--k-runs 4`, which is why nothing noticed.
- 28 rows (12% of the corpus) resolved a span on the vulnerable side and
none on the fixed side, so every fixed-side finding read as outside the
labeled function and the pair scored as silence whatever the detector
said. `fix_function` now reaches the catalog and the scorer, and the
generator derives it from the fixed excerpt for the 25 trap rows that
never declared one. Measured after: 0 rows.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The catalog generator fills `fix_date` from the recipe at the precision the
recipe has -- a full ISO day when one was recorded, otherwise the year.
Every harvested recipe carries a year and almost none carries a day, and
writing `-01-01` would turn a year into a claim about a month. A row counts
as post-cutoff only when its earliest possible date is after the cutoff, so
a coarse date can under-claim the clean slice but never over-claim it.
Each family now reports the same numbers over the rows fixed after the
cutoff, beside the whole number and never instead of it, with the undated
count on its own line. Without a configured cutoff every scoreboard says so
rather than inventing one: guessing a model's training cutoff and then
publishing a contamination number from the guess would be worse than
reporting none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Eight of nine group-3 tasks were rejected. Each defect was hiding a number,
and the numbers are why the loop was not learning:
- The fixed side was judged against the *vulnerable* side's function
ranges. 191 of 227 pairs with a resolvable labeled span (84%) have
different spans on the two sides, so a leak was recorded as silence,
always in the direction that flatters Youden.
- Round zero measured every family with the generalist. `family:unknown`
earns no credit against any real family, so every family but `unknown`
baselined at a structural zero and every noise floor with it.
- The cross-family sweep counted absolute failures instead of negative
flips. On a corpus where most pairs fail at round zero that rejects
every round for damage it did not do. Unscorable rows voted too.
- K runs was declared and never threaded; `pairs --k-runs` was parsed and
discarded, and its only test grepped `--help`.
- The meta-agent proposer was a stub, so `learning round` could never
propose anything. It now composes MetaAgent.evolve against a scratch
copy of the family directory, and `--proposals FILE` runs a recorded
sequence offline.
- `refused_holdout` was a vocabulary word no code path could produce, and
the covering test accepted either outcome.
- Cost was metered once, after everything had been paid for; a stage that
raised left no record at all.
- The corrected lever number was published beside the earlier one rather
than in place of it, and the test named for that property never checked.
- Refusals were `continue` statements: `Refusal.degradation()` existed and
nothing called it. A train-split identical twin was reported as
`split:train`, which is untrue.
Test repairs that each changed a number: two split tests measured an
`x = 1` fixture that teaches nothing whatever the rule says; two CLI
scoreboard tests asserted against all-zero output produced by a stub that
answers twice and puts its finding outside the labeled function.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.9. Five entry points make the classified detector set reachable:
classify a corpus and report the classifier against itself, score a catalog by
family, run round zero, run one evolve round, and publish every number from its
artifacts. The pairs command gains a runs-per-pair flag, since the family scorer
needs more than one.
Every step degrades with a reason rather than failing: with no chat endpoint the
run says so and still records what it could measure without a detector, and a
missing configuration directory is written from the current hunter prompt rather
than being an error. Artifacts are keyed by model throughout, so comparing a
second detector is a flag rather than a rewrite.
Both matrices are green and all three gate outputs stay byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.4. A standard scan now classifies each hotspot region and runs the
detectors of every admitted family plus the generalist, reading the
configurations round zero wrote. A scan never creates them: a missing directory
is a recorded reason and the stage is skipped, which is also what happens when
no chat endpoint resolves.
Every claim passes through its family's verifier and carries what produced it:
the family, the detector version and what the verifier said. Nothing rises above
suspicion here, because the verifier writes a tag and the evidence ladder
belongs to another boundary. Reports gain a section naming the three, SARIF
gains the matching properties, and the manifest gains a learning block with the
taxonomy version, the per-family counts, the classifier report and the round.
The feature is off unless a project turns it on, so an existing scan is
unchanged, and all three gate outputs stay byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
