Feeds the day's measurements back into the spec and reorders the roadmap
around what is actually blocking.
STATUS. Three CVEs found on real WordPress plugins -- CVE-2023-23488,
CVE-2023-6559, CVE-2022-25148 -- each entailed by the graph with no model in
the loop and each absent from its fixed side. Two of them require the
two-stage join and would be invisible without it. The detector is not the
problem.
The repository is. A 637-file plugin builds in 76s to a 4.25MB graph,
dominance and config answer, taint does not finish. Three blockers in the
order they bite: cost (6.63 s/request), ranking (84% of regions share one rank,
ordered alphabetically), coverage (access_control never asked for PHP,
output_encoding untrustworthy while the sanitizer list is flat).
And what is NOT a blocker, contrary to most of what this file said this
morning: php2cpg. One genuine defect, 0.2% of files, detected and excluded and
reported. Everything else attributed to the frontend was a local php shim
proxying stdio, which cost four wrong diagnoses in a day.
NEW TASK 5.14, and it is the cheapest thing on the board. 53% of taint
requests are put to a family whose sinks are not in scope -- all 487 of PMPro's
deserialization requests go to a repository that never calls unserialize.
Skipping them cannot lose a finding, because a family with no sink cannot
produce a flow. 4.6 hours -> 2.2, before any ranking work.
ROADMAP reordered around one question: the shortest path to a scan of a real
plugin that reports its CVE. Prune the impossible questions, then rank (an LLM
job, and safe as one because ranking cannot manufacture a finding, with the
closed-loop problem treated as a design requirement rather than a caveat), then
attack the per-request cost only if those two are not enough.
857 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Three directions recorded, each grounded in what this session measured.
**WordPress by flaw class, not by plugin** (tasks 5.8, 5.9). One plugin is
a checkout; a class is a claim. WordPress's recurring shapes are narrow and
each maps to a family that already exists -- unauthenticated wp_ajax_nopriv
reaching $wpdb, a missing capability check, unescaped output, arbitrary
file operations. Two of the four are blocked on gaps this session found:
access control cannot be asked at all because PHP has no obligation facts
(5.9), and unescaped output is blocked on the flat sanitizer list (5.6).
The observable is a class-by-class table across several plugins saying
found, missed, or out of scope -- a single found CVE is not the artifact
worth teaching from.
**Why a corpus of benchmarks is not enough on its own**, written into
overview.md as the finding it is rather than an opinion. Twelve engine
defects in one session against three real repositories, and the three gates
stayed byte-identical through every fix. A gate that does not move is a
corpus that could not see the bug. The reason is structural: a pair is a
labelled function with a declared family and a known answer, so it cannot
exercise which region to look at, what to spend, whether a capability is
reachable, whether two entry points share a sink, whether the question fits
in an argv slot, or whether a flow crosses a module. Three of the twelve
were quiet failures where the tool reported clean and every honesty counter
agreed -- which a benchmark score cannot express, because the score is
computed from the verdicts that exist. The corpus is the holdout that stops
per-repository tuning, and the wrong instrument for finding out whether the
tool works.
**ir-consolidation** (brief, unapproved) separates two questions the IR
suggestion contains, which have opposite answers. The SOURCE IR is already
carried rather than about to be added: semantic/ is 3218 lines, 16% of the
codebase, measured at 2.2% entailment against the CPG's 41.3% and still
running as a proposer -- and Vine and P-Code are binary IRs, so the work
there is subtraction. The MISSING abstraction is C arithmetic, and there
GhidraPAL's three-valued-logic abstract interpreter over Ghidra P-Code is
exactly the right shape: "can this index exceed this allocation" is a value
question, which is what a bit-level TVL domain answers and what every
libpng overflow needs. Vine is dormant since 2009 and is not the candidate.
The constraint is recorded as an ordering: question 1 lands before question
2 starts, or adopting an IR makes the growth problem worse. And the cost
that matters is not lines but the number of things that can silently return
nothing -- this session found three such in a single PHP scan, and a binary
pipeline adds a build, a lifter and a domain, each able to produce an empty
result that looks like a clean one.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The named release targets are, today, exactly the two cases that do not
work -- and they fail for different reasons, which is what makes them a
good pair rather than one.
WordPress fails because `taint_specs(language="php")` returns zero
families: Joern's php2cpg parses it and we say nothing. That is the
shortest path from here to a named target, and the case this arbiter suits
best, because PHP web bugs are the classic taint families.
libpng fails because c/memory models string functions while libpng
overflows through arithmetic, and its library never calls a sink we model.
That needs either a second abstraction -- size arithmetic, a constraint
problem -- or an honest stated limit.
So "ready for" means two different things and the criteria say which
applies where. Where a family can decide the class, find the known
vulnerability. Where it cannot, run inside the envelope, report nothing
false, and STATE the limit. The second is not a consolation prize: a tool
that says "I do not model this class" is usable, and one that silently
finds nothing is what teaches developers to distrust SAST in the first
place.
Seven v0.1 criteria, each tied to a requirement that already exists:
unattended runs inside a stated envelope; zero known-false entailed
findings on every pinned checkout; the known vulnerability found wherever a
family can decide it (Req 4.6); every family's limit stated in the report
(Req 5.6); a proportionate report (Req 5.5); installs without the user
having Joern, a JVM or PHP; and a committed regression baseline per target.
That reorders the middle of the roadmap. PHP moves up, and the c/memory
decision moves up with it, because criterion 4 needs that family either
earning its place or being scoped -- and scoping it is cheap. The vibe-code
repository stays valuable as the guard against tuning everything on vampi,
but drops below PHP because it is not a named target.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Fourteen specs have accumulated and the roadmap still described a
pre-Joern world -- "adjudication is still Python-ast plus a tree-sitter CLI
stub", "Joern-as-required deferred" -- which is stale enough to mislead
someone reading it to decide what to do next. It is kept as
roadmap-2026-09-04.md rather than deleted.
**overview.md** is the synthesis above the specs: what the tool is, how one
pass works, what has been measured, and what is not true yet. Every number
cites a committed artifact. The thesis is stated plainly -- the
differentiator is not the dataflow engine, since CodeQL and Semgrep have
one and developers do not turn it on, but the rung: a claim you can act on
without writing a query, and an explicit admission when the engine could
not decide.
It says what does not work as clearly as what does. The C story is
structurally broken, not undertuned. One language is demonstrated. PHP,
Ruby and Go have zero families, so Joern would parse them and the engine
would say nothing. The evidence base is two repositories and one has never
produced a true positive. Control dependence is unavailable in both CPGs.
execution_confirmed has no production caller.
And it records the lesson that shaped the current specs: nine defects found
by pointing the tool at two real repositories, three of them the same bug
in three matchers, with the three gates byte-identical through every fix --
so a corpus of function-level pairs could not have caught any of them.
**roadmap.md** is ordered by the standing test, that the tool must find
bugs or it is SAST noise. Which gives the ordering principle: make the
output worth reading, then make it find more. A better arbiter behind an
unreadable report changes nothing, and a second language before the report
is fixed doubles the noise.
It also says what is NOT being worked and why -- authorization-obligations
and reachability-flow-model were largely delivered by other routes and
should be reconciled or retired rather than carried as approved-not-started,
which misrepresents both.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
`publish` keyed baselines by model alone, so the vibe-py run silently
replaced the agent-vfc run and the table showed hand-written teaching code as
if it were the corpus. That is precisely the failure the constrained-detector
requirements exist to prevent, live in a published artifact.
Third time today that a key was fixed in one place and not another --
`compare_baselines` got the model+slice label and `_baselines` did not, the
same way the exception path got checkpointing and the timeout path did not.
The table now carries a slice column and one row per slice, and the artifact
is keyed by what identifies a run. Republished: agent-vfc's six families are
back, including access_control at 0.000 recall on four pairs, which is the
number that was missing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Eight of nine group-3 tasks were rejected. Each defect was hiding a number,
and the numbers are why the loop was not learning:
- The fixed side was judged against the *vulnerable* side's function
ranges. 191 of 227 pairs with a resolvable labeled span (84%) have
different spans on the two sides, so a leak was recorded as silence,
always in the direction that flatters Youden.
- Round zero measured every family with the generalist. `family:unknown`
earns no credit against any real family, so every family but `unknown`
baselined at a structural zero and every noise floor with it.
- The cross-family sweep counted absolute failures instead of negative
flips. On a corpus where most pairs fail at round zero that rejects
every round for damage it did not do. Unscorable rows voted too.
- K runs was declared and never threaded; `pairs --k-runs` was parsed and
discarded, and its only test grepped `--help`.
- The meta-agent proposer was a stub, so `learning round` could never
propose anything. It now composes MetaAgent.evolve against a scratch
copy of the family directory, and `--proposals FILE` runs a recorded
sequence offline.
- `refused_holdout` was a vocabulary word no code path could produce, and
the covering test accepted either outcome.
- Cost was metered once, after everything had been paid for; a stage that
raised left no record at all.
- The corrected lever number was published beside the earlier one rather
than in place of it, and the test named for that property never checked.
- Refusals were `continue` statements: `Refusal.degradation()` existed and
nothing called it. A train-split identical twin was reported as
`split:train`, which is untrue.
Test repairs that each changed a number: two split tests measured an
`x = 1` fixture that teaches nothing whatever the rule says; two CLI
scoreboard tests asserted against all-zero output produced by a stub that
answers twice and puts its finding outside the labeled function.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Absence bugs as obligations rather than flows: obligated operations and
discharger kinds as closed facts, obligation shapes learned from trusted
pairs, the application's policy recovered from sibling-handler
consistency and an optional declared policy file, path-aware evaluation
with a function-local degraded mode, honest rungs below proof, an
`obligations` improve lever under the holdout gate. Depends on
reachability-flow-model and guard-dominance-regime; the roadmap marks
reachability as the critical path for two consumers.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
- tasks.md: all tasks checked; rounds 1-6 recorded with RED evidence,
measured leave-one-out and admission numbers, and the recorded design
deviations.
- spec.json: completed 2026-09-05.
- roadmap: corpus-seeded-mechanisms marked completed; the loop is closed
end to end (export -> improve admits under the holdout clause -> scan
searches -> reports name the mechanism).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
- tasks.md: round-12 verdict and the decision to assemble the redaction
test's example key at runtime and rewrite the unpushed series so no
commit carries a contiguous key literal; two dated notes marked as
superseded by tasks 8.1 and 8.4.
- roadmap: the folded inventory-vs-overlay Youden item is checked.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
- tasks.md: all tasks checked; the two escalations are closed (5.1 by the
Req 10 pointer rows, 5.3 by the Req 9 review tiers); round 11 recorded
with the carry-forwards for corpus-seeded-mechanisms and
overlay-ir-completeness.
- roadmap: measured state after tiers and pointer slices (vibe-py 115,
agent-vfc 59, gates byte-identical to pre-spec); pair-corpus-honesty
marked completed.
- spec.json: completed 2026-09-05 after eleven review rounds.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Harvest 176 isolated vuln-vs-fix functions (109 CVEs, 352 training rows)
into benchmarks/pairs/vfc with provenance and a JSONL training manifest.
Score the slice as an honesty dashboard (ousast pairs --slice vfc);
keep the pair gate on the local fixtures only.
Add openultrasast[semantic] with official grammar wheels and one CST walker
into FileIR. quick stays extra-free. Host CLI is not overlay-ready. Default CI
stays extra-free; a semantic job runs pytest -m semantic.
