Roadmap "Where we are" table said Baseline (G0.2) = "Not yet, run on kube" -- stale.
Record 07 (G0 exit) was completed right after on the docker lane (e901640, 1300/1300,
catch 3.3% / FP 3.1%, regex/quick-rule floor, engine_covered=0). G0 is exited; the kube
`ax` lane was never needed for the baseline. Reframed the readiness plan: steps 1-4 done,
the `ax` lane moves to G1 (engine arm at scale, gated by spec tasks 4.4/5.4).
Also fixed the ax binary in the coordinator commands: use $HOME/go/bin/ax, NOT
$HOME/.local/ax-v0.3.1/ax (the latter hits the AX-lane dispatch blocker recorded in the
unseen spec tasks.md). README.md + benchmarks/unseen/README.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Same CPG, only the search changes. Baseline (shipped path spec) resolves 0
prompt->write_text flows on banks; injecting the two annotations an LLM steer
would supply (parameter_sources=True + write_text/read_text path sinks) makes the
engine resolve prompt -> path-join -> prompt_file.write_text(prompt.raw). First
measured instance of "#2 guided by #1" closing on an unseen target; fix needed
exactly the param-source + file-write-sink annotations.
Files the instrument bug the first MVP hit: single-query cpg.run("taint",
request_params(...)) is dead -- request_params emits 5 keys (guards/fixedOrigin/
originAnchors/quotedSanitizers/fieldParameterSources) absent from taint.sc's exec
signature; both single-query sub-routes reject them (silent None). Batch path
(run_batch->requestsFile JSON) unaffected; push/scan route uses batch.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Verified root cause: the pilot repos are libraries whose untrusted input is public-API
parameters, not HTTP requests. Python taint sources (ruleset/semantic/python.toml) are web/CLI
only; the parameter-as-source mode (_parameters_are_input) fires only for recognised entry
points, and the pilot vuln regions are entry:false -- so their params are deliberately not
sources. Widening that statically would reintroduce the WordPress/PMPro false-SQL-injection
flood the docstring documents. So the static engine cannot safely source library APIs on its
own -- which is exactly the seam for the maintainer's '2 guided by 1' directive: the LLM
adjudicates which library parameter is a trust boundary, the taint engine traces that
source->sink as proof (not LLM say-so), and repeated source shapes graduate to guarded static
facts. First increment (pending budget): LLM-guided-source mode on the 5 pilot regions,
measuring source naming, traced finding, catch lift vs regex 1/5, and false-alarm on ordinary
changes; staged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Inspecting head.findings (not ranker evidence) overturns the previous commit's read:
the dataflow/taint tier produced ZERO findings in the vuln families (path/injection/deser)
on all 5 repos -- only config_secrets (banks 1, socketio 2; launch-editor/graphiti/bedrock 0).
The tier1/score2.0 'sink' I cited was the RANKER's static candidate, not a confirmed taint
finding. taint_evidence_not_applicable = no source->sink flow was traced (banks: sink found,
no source). So ambiguous_line_correspondence is downstream and moot -- there are no
vuln-family findings to anchor, and the region-level witness fix (Option A) is refuted before
implementation. The LLM adjudication tier was disabled (model_calls=0).
Reprioritised: #1 LLM-tier pilot L1/L2 (now primary, staged, costs money), #2 taint
source/sink coverage + region ranking, #3 change-anchoring (only after tiers yield findings).
Instrument rule: caught the ranker-vs-finding misread before building the wrong fix.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Five public pool repos, vuln change each, clean host, 300s. The shipped engine added NO
catches over the regex floor. 3/5 repos finished their region queue (deferred=0) and still
emitted no finding, so change-scoping is not the path to catch. The engine ranked the vuln
region in all 5 and found the right sink on 2 (score 2.0), but withholds any finding it
cannot anchor 1:1 to changed lines -- all 5 hit ambiguous_line_correspondence ->
witness_unresolved -> coverage incomplete, suppressing even the scored sinks. Confirmed not a
shallow-clone artifact (correspondence is computed from both blobs).
Reprioritised engine levers: #1 change-anchoring robustness (region-level witness fallback),
#2 family ranking/taint coverage, #3 the LLM adjudication tier (the pilot is the case for it).
Release read: the pre-push net is NOT releasable on detection yet.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
On a clean host the shipped engine runs dataflow (query 0->35s, selected 0->356 regions).
The evidence/taint batch (~225s) is invariant to warm session and graph preload; importCpg
is cheap (~35s). The real barrier is taint.sc's ~160s per-batch flow work, so build+evidence
~255s of a 300s budget defers 1009/1365 regions. Lever = change-scope the family/file set the
taint batch walks, not session/preload. Corrects the prior ~200s graph-load wall diagnosis.
Next decisive experiment: engine-on unseen pilot vs record-07 regex floor (catch 3/92).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Verified on the records: the push replay arms hardcode model="disabled",
max_model_calls=0, cost_usd=0 (NO LLM) and engine_covered=0 (engine never ran,
~200s wall). So the measured catch~2.5% is the quick pattern-rule tier ALONE --
grep-with-structure. Pattern rules alone will never be enough.
Reprioritize G2 after record 07 to an efficiency barrier analysis over the two
tiers that carry detection and both sat out:
- engine (Joern dataflow): E1 split the ~150s evidence-fixed (JVM/graph-load/
census); E2 does a warm session + graph preload at build kill it (decisive);
E3 analysis per-pair; E4 build vs repo size.
- LLM adjudication: L1 model latency + calls per push; L2 recall lift over
regex+dataflow on the pool.
Deliverable: a barrier table (budget x tier combo -> fits? measured recall/FP)
for a realistic pre-push window (<=120s target, 300s hard) -> minimal releasable
config. Change-scoping branch is a late add-on (trims only the per-pair tail),
not the lever. Also fixed a stray </content> tag in the brief.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Track the engine-detection lever found while driving G0. Live diagnosis on the
frozen pool: the engine's "no finding" on unseen changes is usually
analysis-never-runs -- it ranks repo-wide candidates and defers them all
(deadline_exhausted) before analyzing the pushed change (bedrock injection,
303s, selected=[], ranked sites unrelated to the change). Lever: seed analysis
from the change's declarations + dataflow neighbourhood (rank.py/mapping.py) so
it runs within budget.
- .kiro/specs/engine-change-scoped-analysis/brief.md: problem (measured),
hypothesis, anchors, and checkable increments (0 confirm build-vs-analysis
cost split -> 1 locate selection/budget -> 2 seed from change -> 3 soundness
+ tests -> 4 live re-test -> 5 measure delta on the engine arm).
- roadmap.md G2: active workstreams (change-scoped analysis; the done-but-
necessary-not-sufficient line-correspondence fix), for progress tracking.
Honest scope: G2, gated on the pool metric; not part of the G0 quick-rule
record-07. Coverage also needs the capability registry (Task 8.3); arm-scale
measurement needs the kube lane.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Fold the night's learnings into the roadmap, steering and the
unseen-repo-evaluation spec so the process is reproducible and the
project is "ready now":
- roadmap.md: a 2026-10-11 state table (pool p1 frozen; harvest hardened
+ bucket-first; replay-image provenance pinned; baseline-on-kube next)
and a 5-step readiness plan to record 07.
- storage-streaming.md: RustFS-over-local as the default (local disk is
the scarce resource); the implemented ousast-corpus layout; replay
outputs belong on RustFS; the AX store endpoint is operator-owned.
- design.md: an "Amendments (2026-10-11)" section (bucket-first
persistence, ~2.2% rate-bound yield, machine-label record 07, pinned
replay image provenance).
- tasks.md: a folded-in amendment block (pool p1 frozen + freeze recipe;
bucket-first; CI replay image; machine-label baseline; kube lane +
ax/store as operator-owned infra).
Facts stated are verified (measured yield, freeze digest, GHCR anon pull,
batch.py's endpoint check); infra capability/policy is left to the
operator, not assumed. The frozen pool artifacts themselves are not
committed here (a separate samples/licensing decision).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Maintainer 2026-10-04: the safety net is for unseen code and the A/B tests exist
to prove the approach generalises; infrastructure work had drifted from that.
Runs are designed for AX on kube-ax, kind is the fallback.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A4: roadmap-2026-09-04.md moves to .kiro/archive/steering/ so it no longer
loads as steering (roadmap.md and safety-net.md replaced it); roadmap.md's
pointer follows it. overview.md's spec table is recounted from tasks.md and
spec.json (adds ai-service-plane, harnessx-removal, learned-decision-engine,
pre-push-safety-net; drops the stale "active one"), and the module line now
cites the regenerated audit: 150 modules, 145 load-bearing, 5 standalone,
0 orphaned; 1,964 tests collected.
A5: a dated status note at the top of the requirements (or brief) of 16
specs: retired, superseded, dormant or complete, and for
model-grounded-detection the removal of judge/calibrate/execution (closing
the Req 10 execution-tier option). spec.json phase corrected to
implementation-complete where every task is checked (model-grounded-
detection, propose-adjudicate-prove, tree-sitter-overlay-extra,
corpus-seeded-mechanisms, pair-corpus-honesty); openrouter-sast-harness gets
a spec.json with status retired. Nothing deleted, no spec moved.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
All three pinned NodeGoat revisions replayed through the production rules with
exact pins and reopened source receipts.
The vulnerable revision reports all three sites new with
operation_absent_from_comparable_base, and admission now rejects on
capability_unavailable alone: dependency_unresolved is gone, which verifies 10.7
on real data since those gaps belonged to unrelated questions. The fixed twin is
analyzed and quiet with three base-only findings. The unchanged revision reports
all three sites unchanged with base locations resolved to the same lines.
Every finding carries its question, the engine-reported method and its witness.
The head census is the complete 44 files and all 436 base questions complete.
Zero admitted defects on every revision.
All five vetoes M1a recorded are repaired, plus two the repairs exposed. What
remains is recorded rather than repaired: 231 head questions still report an
incomplete change context from families whose queries export none, which did not
block this case and whose families stay experimental, and the vendor exclusion
and correspondence ambiguity remain reported as the limits they are.
M1b is not yet verified. The milestone also asks for a developer-readable finding
carrying a supported security consequence and a concrete repair direction, and
both are owned by an evaluated capability declaration that is deliberately empty.
Writing that text without a declaration is what admission exists to prevent, so
task 10.10 renders it through an explicitly experimental declaration with
eligibility still disabled.
Runtime, independent qualification and rollout all remain NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Rerunning the three pinned revisions after tasks 10.1-10.6 shows the production
comparison classifying all of them with no lifting, and agreeing exactly with the
recorded view. That agreement is the point: the vetoes M1a recorded were repaired
rather than bypassed.
The vulnerable revision is new with operation_absent_from_comparable_base, and the
three eval witnesses at contributions.js lines 32 to 34 are attributed to the
change that introduced them. The fixed twin is analyzed and quiet with three
base-only findings. The unchanged revision is unchanged. In M1a every one of the
436 questions was demoted on both sides and production reported
witness_identity_unresolved on every finding.
All three artifacts still report finding_status none with zero admitted defects.
Admission rejects on capability_unavailable, the deliberate eligibility gate, and
on dependency_unresolved, which the rerun exposed as coarse plumbing: the runner
passes every coverage reason as every candidate's dependency gaps. That is now
task 10.7, ahead of M4, since it would block admission forever once capabilities
qualify. It is not a comparison failure and not an M1b exit blocker.
Three comparisons on one inspected teaching repository under a development budget.
No capability enabled, no alert emitted, no latency claim. Runtime, independent
qualification and rollout all remain NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Rerunning the three NodeGoat revisions after tasks 10.1-10.3 shows the recorded
view now classifies the case: the vulnerable revision reaches new with
operation_absent_from_comparable_base, and the unchanged revision reaches
unchanged with its base location resolved to the same lines. Both previously
stopped at operation_correspondence_unresolved and witness_identity_unresolved.
The rerun also caught a defect in the repair itself: the new novelty reason was
missing from the admission allowlist, so admission rejected the candidate as
change_unsupported. Fixed with a control.
The production path is still blocked by the last two vetoes, and the measurement
shows neither can be scoped or relaxed as the plan described. A taint answer
reports traced flows, so an absent row means either that the operation does not
exist in the base or that no traced flow reached it.
dynamic_external_or_depth_context_unresolved exists to stop the second being read
as the first, and removing it would turn a wrong absence into a wrong new
finding. It is also computed over every in-scope method, so on any real Express
application it is always true and can never be discharged, which is why the tool
reports nothing on real projects.
The resolution is new evidence rather than a relaxed rule: export the operations
the query actually enumerated, separately from the flows it traced, so absence is
established from the inventory while a traced-flow claim still requires complete
reachability. That is now task 10.4, ahead of the two vetoes it unblocks, and it
adds no detector, alias resolver or second source IR.
1,348 tests passed, nine skipped. No capability enabled; rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Add group 9 to the approved task plan as the bounded owner amendment release
milestone M1a requires, implement the task-local recorded-veto evaluation flag,
and run the three pinned NodeGoat revisions under it.
The flag re-runs the production comparison and admission over a lifted copy of
the same evidence and records each veto instead of applying it. Recognized
completeness vetoes are lifted and logged; failures never are, because silence
from a failure is not a quiet fixed twin. It is off by default, accepted only on
explicit base/head replay, refused in Git stdin mode, labels every payload it
produces, and admits no defect even when a lifted candidate otherwise would.
M1a outcome, inside its one-week box and with expectations frozen on disk first:
the three entailed <lambda>2 witnesses are present at contributions.js 32-34; the
fixed twin was genuinely analyzed and quiet, answering its target question with
zero rows and no deadline reason while the same findings appear as base-only; the
unchanged revision was analyzed and recorded unchanged. The kill criterion did
not trigger.
Five vetoes fired and were recorded. M1a also corrected the diagnosis: with all
five lifted the detection still stops at operation_correspondence_unresolved,
because a newly introduced operation sits on a changed line and has no base
anchor. The unchanged revision resolves its base location on the same lines and
classifies as unchanged, which isolates correspondence as the mechanism. That
fifth veto was not in the earlier four-veto diagnosis and is now M1b's central
repair.
Diagnosis only. No defect admitted, no capability enabled, no gate changed, no
latency claim; the 900-second development budget cannot measure the hook gate.
Rollout and task 8.3 remain NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Add release-milestones.md as the executable outcome contract: prove one useful
Node detection, make it fast, verify PHP/Python transfer, qualify on untouched
cases, then enable a narrow advisory rollout. Link it from the roadmap, safety-net
steering, requirements, tasks, status and spec.json.
Split M1 after review. The raw NodeGoat <lambda>2 witness already exists in the
2026-09-14 post-census bundle; four admission vetoes discard it. M1a runs the
vulnerable/fixed/unchanged revisions under a task-local experimental flag that
records each veto instead of applying it, with a one-week box and a kill
criterion if the fixed twin is not quiet. M1b repairs only the vetoes that fired
and reruns through production rules. Existing approvals and thresholds unchanged.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Feeds the day's measurements back into the spec and reorders the roadmap
around what is actually blocking.
STATUS. Three CVEs found on real WordPress plugins -- CVE-2023-23488,
CVE-2023-6559, CVE-2022-25148 -- each entailed by the graph with no model in
the loop and each absent from its fixed side. Two of them require the
two-stage join and would be invisible without it. The detector is not the
problem.
The repository is. A 637-file plugin builds in 76s to a 4.25MB graph,
dominance and config answer, taint does not finish. Three blockers in the
order they bite: cost (6.63 s/request), ranking (84% of regions share one rank,
ordered alphabetically), coverage (access_control never asked for PHP,
output_encoding untrustworthy while the sanitizer list is flat).
And what is NOT a blocker, contrary to most of what this file said this
morning: php2cpg. One genuine defect, 0.2% of files, detected and excluded and
reported. Everything else attributed to the frontend was a local php shim
proxying stdio, which cost four wrong diagnoses in a day.
NEW TASK 5.14, and it is the cheapest thing on the board. 53% of taint
requests are put to a family whose sinks are not in scope -- all 487 of PMPro's
deserialization requests go to a repository that never calls unserialize.
Skipping them cannot lose a finding, because a family with no sink cannot
produce a flow. 4.6 hours -> 2.2, before any ranking work.
ROADMAP reordered around one question: the shortest path to a scan of a real
plugin that reports its CVE. Prune the impossible questions, then rank (an LLM
job, and safe as one because ranking cannot manufacture a finding, with the
closed-loop problem treated as a design requirement rather than a caveat), then
attack the per-request cost only if those two are not enough.
857 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Three directions recorded, each grounded in what this session measured.
**WordPress by flaw class, not by plugin** (tasks 5.8, 5.9). One plugin is
a checkout; a class is a claim. WordPress's recurring shapes are narrow and
each maps to a family that already exists -- unauthenticated wp_ajax_nopriv
reaching $wpdb, a missing capability check, unescaped output, arbitrary
file operations. Two of the four are blocked on gaps this session found:
access control cannot be asked at all because PHP has no obligation facts
(5.9), and unescaped output is blocked on the flat sanitizer list (5.6).
The observable is a class-by-class table across several plugins saying
found, missed, or out of scope -- a single found CVE is not the artifact
worth teaching from.
**Why a corpus of benchmarks is not enough on its own**, written into
overview.md as the finding it is rather than an opinion. Twelve engine
defects in one session against three real repositories, and the three gates
stayed byte-identical through every fix. A gate that does not move is a
corpus that could not see the bug. The reason is structural: a pair is a
labelled function with a declared family and a known answer, so it cannot
exercise which region to look at, what to spend, whether a capability is
reachable, whether two entry points share a sink, whether the question fits
in an argv slot, or whether a flow crosses a module. Three of the twelve
were quiet failures where the tool reported clean and every honesty counter
agreed -- which a benchmark score cannot express, because the score is
computed from the verdicts that exist. The corpus is the holdout that stops
per-repository tuning, and the wrong instrument for finding out whether the
tool works.
**ir-consolidation** (brief, unapproved) separates two questions the IR
suggestion contains, which have opposite answers. The SOURCE IR is already
carried rather than about to be added: semantic/ is 3218 lines, 16% of the
codebase, measured at 2.2% entailment against the CPG's 41.3% and still
running as a proposer -- and Vine and P-Code are binary IRs, so the work
there is subtraction. The MISSING abstraction is C arithmetic, and there
GhidraPAL's three-valued-logic abstract interpreter over Ghidra P-Code is
exactly the right shape: "can this index exceed this allocation" is a value
question, which is what a bit-level TVL domain answers and what every
libpng overflow needs. Vine is dormant since 2009 and is not the candidate.
The constraint is recorded as an ordering: question 1 lands before question
2 starts, or adopting an IR makes the growth problem worse. And the cost
that matters is not lines but the number of things that can silently return
nothing -- this session found three such in a single PHP scan, and a binary
pipeline adds a build, a lifter and a domain, each able to produce an empty
result that looks like a clean one.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The named release targets are, today, exactly the two cases that do not
work -- and they fail for different reasons, which is what makes them a
good pair rather than one.
WordPress fails because `taint_specs(language="php")` returns zero
families: Joern's php2cpg parses it and we say nothing. That is the
shortest path from here to a named target, and the case this arbiter suits
best, because PHP web bugs are the classic taint families.
libpng fails because c/memory models string functions while libpng
overflows through arithmetic, and its library never calls a sink we model.
That needs either a second abstraction -- size arithmetic, a constraint
problem -- or an honest stated limit.
So "ready for" means two different things and the criteria say which
applies where. Where a family can decide the class, find the known
vulnerability. Where it cannot, run inside the envelope, report nothing
false, and STATE the limit. The second is not a consolation prize: a tool
that says "I do not model this class" is usable, and one that silently
finds nothing is what teaches developers to distrust SAST in the first
place.
Seven v0.1 criteria, each tied to a requirement that already exists:
unattended runs inside a stated envelope; zero known-false entailed
findings on every pinned checkout; the known vulnerability found wherever a
family can decide it (Req 4.6); every family's limit stated in the report
(Req 5.6); a proportionate report (Req 5.5); installs without the user
having Joern, a JVM or PHP; and a committed regression baseline per target.
That reorders the middle of the roadmap. PHP moves up, and the c/memory
decision moves up with it, because criterion 4 needs that family either
earning its place or being scoped -- and scoping it is cheap. The vibe-code
repository stays valuable as the guard against tuning everything on vampi,
but drops below PHP because it is not a named target.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Fourteen specs have accumulated and the roadmap still described a
pre-Joern world -- "adjudication is still Python-ast plus a tree-sitter CLI
stub", "Joern-as-required deferred" -- which is stale enough to mislead
someone reading it to decide what to do next. It is kept as
roadmap-2026-09-04.md rather than deleted.
**overview.md** is the synthesis above the specs: what the tool is, how one
pass works, what has been measured, and what is not true yet. Every number
cites a committed artifact. The thesis is stated plainly -- the
differentiator is not the dataflow engine, since CodeQL and Semgrep have
one and developers do not turn it on, but the rung: a claim you can act on
without writing a query, and an explicit admission when the engine could
not decide.
It says what does not work as clearly as what does. The C story is
structurally broken, not undertuned. One language is demonstrated. PHP,
Ruby and Go have zero families, so Joern would parse them and the engine
would say nothing. The evidence base is two repositories and one has never
produced a true positive. Control dependence is unavailable in both CPGs.
execution_confirmed has no production caller.
And it records the lesson that shaped the current specs: nine defects found
by pointing the tool at two real repositories, three of them the same bug
in three matchers, with the three gates byte-identical through every fix --
so a corpus of function-level pairs could not have caught any of them.
**roadmap.md** is ordered by the standing test, that the tool must find
bugs or it is SAST noise. Which gives the ordering principle: make the
output worth reading, then make it find more. A better arbiter behind an
unreadable report changes nothing, and a second language before the report
is fixed doubles the noise.
It also says what is NOT being worked and why -- authorization-obligations
and reachability-flow-model were largely delivered by other routes and
should be reconciled or retired rather than carried as approved-not-started,
which misrepresents both.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
`publish` keyed baselines by model alone, so the vibe-py run silently
replaced the agent-vfc run and the table showed hand-written teaching code as
if it were the corpus. That is precisely the failure the constrained-detector
requirements exist to prevent, live in a published artifact.
Third time today that a key was fixed in one place and not another --
`compare_baselines` got the model+slice label and `_baselines` did not, the
same way the exception path got checkpointing and the timeout path did not.
The table now carries a slice column and one row per slice, and the artifact
is keyed by what identifies a run. Republished: agent-vfc's six families are
back, including access_control at 0.000 recall on four pairs, which is the
number that was missing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Eight of nine group-3 tasks were rejected. Each defect was hiding a number,
and the numbers are why the loop was not learning:
- The fixed side was judged against the *vulnerable* side's function
ranges. 191 of 227 pairs with a resolvable labeled span (84%) have
different spans on the two sides, so a leak was recorded as silence,
always in the direction that flatters Youden.
- Round zero measured every family with the generalist. `family:unknown`
earns no credit against any real family, so every family but `unknown`
baselined at a structural zero and every noise floor with it.
- The cross-family sweep counted absolute failures instead of negative
flips. On a corpus where most pairs fail at round zero that rejects
every round for damage it did not do. Unscorable rows voted too.
- K runs was declared and never threaded; `pairs --k-runs` was parsed and
discarded, and its only test grepped `--help`.
- The meta-agent proposer was a stub, so `learning round` could never
propose anything. It now composes MetaAgent.evolve against a scratch
copy of the family directory, and `--proposals FILE` runs a recorded
sequence offline.
- `refused_holdout` was a vocabulary word no code path could produce, and
the covering test accepted either outcome.
- Cost was metered once, after everything had been paid for; a stage that
raised left no record at all.
- The corrected lever number was published beside the earlier one rather
than in place of it, and the test named for that property never checked.
- Refusals were `continue` statements: `Refusal.degradation()` existed and
nothing called it. A train-split identical twin was reported as
`split:train`, which is untrue.
Test repairs that each changed a number: two split tests measured an
`x = 1` fixture that teaches nothing whatever the rule says; two CLI
scoreboard tests asserted against all-zero output produced by a stub that
answers twice and puts its finding outside the labeled function.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Absence bugs as obligations rather than flows: obligated operations and
discharger kinds as closed facts, obligation shapes learned from trusted
pairs, the application's policy recovered from sibling-handler
consistency and an optional declared policy file, path-aware evaluation
with a function-local degraded mode, honest rungs below proof, an
`obligations` improve lever under the holdout gate. Depends on
reachability-flow-model and guard-dominance-regime; the roadmap marks
reachability as the critical path for two consumers.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
- tasks.md: all tasks checked; rounds 1-6 recorded with RED evidence,
measured leave-one-out and admission numbers, and the recorded design
deviations.
- spec.json: completed 2026-09-05.
- roadmap: corpus-seeded-mechanisms marked completed; the loop is closed
end to end (export -> improve admits under the holdout clause -> scan
searches -> reports name the mechanism).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
- tasks.md: round-12 verdict and the decision to assemble the redaction
test's example key at runtime and rewrite the unpushed series so no
commit carries a contiguous key literal; two dated notes marked as
superseded by tasks 8.1 and 8.4.
- roadmap: the folded inventory-vs-overlay Youden item is checked.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
- tasks.md: all tasks checked; the two escalations are closed (5.1 by the
Req 10 pointer rows, 5.3 by the Req 9 review tiers); round 11 recorded
with the carry-forwards for corpus-seeded-mechanisms and
overlay-ir-completeness.
- roadmap: measured state after tiers and pointer slices (vibe-py 115,
agent-vfc 59, gates byte-identical to pre-spec); pair-corpus-honesty
marked completed.
- spec.json: completed 2026-09-05 after eleven review rounds.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Harvest 176 isolated vuln-vs-fix functions (109 CVEs, 352 training rows)
into benchmarks/pairs/vfc with provenance and a JSONL training manifest.
Score the slice as an honesty dashboard (ousast pairs --slice vfc);
keep the pair gate on the local fixtures only.
Add openultrasast[semantic] with official grammar wheels and one CST walker
into FileIR. quick stays extra-free. Host CLI is not overlay-ready. Default CI
stays extra-free; a semantic job runs pytest -m semantic.
