The named release targets are, today, exactly the two cases that do not
work -- and they fail for different reasons, which is what makes them a
good pair rather than one.
WordPress fails because `taint_specs(language="php")` returns zero
families: Joern's php2cpg parses it and we say nothing. That is the
shortest path from here to a named target, and the case this arbiter suits
best, because PHP web bugs are the classic taint families.
libpng fails because c/memory models string functions while libpng
overflows through arithmetic, and its library never calls a sink we model.
That needs either a second abstraction -- size arithmetic, a constraint
problem -- or an honest stated limit.
So "ready for" means two different things and the criteria say which
applies where. Where a family can decide the class, find the known
vulnerability. Where it cannot, run inside the envelope, report nothing
false, and STATE the limit. The second is not a consolation prize: a tool
that says "I do not model this class" is usable, and one that silently
finds nothing is what teaches developers to distrust SAST in the first
place.
Seven v0.1 criteria, each tied to a requirement that already exists:
unattended runs inside a stated envelope; zero known-false entailed
findings on every pinned checkout; the known vulnerability found wherever a
family can decide it (Req 4.6); every family's limit stated in the report
(Req 5.6); a proportionate report (Req 5.5); installs without the user
having Joern, a JVM or PHP; and a committed regression baseline per target.
That reorders the middle of the roadmap. PHP moves up, and the c/memory
decision moves up with it, because criterion 4 needs that family either
earning its place or being scoped -- and scoping it is cheap. The vibe-code
repository stays valuable as the guard against tuning everything on vampi,
but drops below PHP because it is not a named target.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Fourteen specs have accumulated and the roadmap still described a
pre-Joern world -- "adjudication is still Python-ast plus a tree-sitter CLI
stub", "Joern-as-required deferred" -- which is stale enough to mislead
someone reading it to decide what to do next. It is kept as
roadmap-2026-09-04.md rather than deleted.
**overview.md** is the synthesis above the specs: what the tool is, how one
pass works, what has been measured, and what is not true yet. Every number
cites a committed artifact. The thesis is stated plainly -- the
differentiator is not the dataflow engine, since CodeQL and Semgrep have
one and developers do not turn it on, but the rung: a claim you can act on
without writing a query, and an explicit admission when the engine could
not decide.
It says what does not work as clearly as what does. The C story is
structurally broken, not undertuned. One language is demonstrated. PHP,
Ruby and Go have zero families, so Joern would parse them and the engine
would say nothing. The evidence base is two repositories and one has never
produced a true positive. Control dependence is unavailable in both CPGs.
execution_confirmed has no production caller.
And it records the lesson that shaped the current specs: nine defects found
by pointing the tool at two real repositories, three of them the same bug
in three matchers, with the three gates byte-identical through every fix --
so a corpus of function-level pairs could not have caught any of them.
**roadmap.md** is ordered by the standing test, that the tool must find
bugs or it is SAST noise. Which gives the ordering principle: make the
output worth reading, then make it find more. A better arbiter behind an
unreadable report changes nothing, and a second language before the report
is fixed doubles the noise.
It also says what is NOT being worked and why -- authorization-obligations
and reachability-flow-model were largely delivered by other routes and
should be reconciled or retired rather than carried as approved-not-started,
which misrepresents both.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
`publish` keyed baselines by model alone, so the vibe-py run silently
replaced the agent-vfc run and the table showed hand-written teaching code as
if it were the corpus. That is precisely the failure the constrained-detector
requirements exist to prevent, live in a published artifact.
Third time today that a key was fixed in one place and not another --
`compare_baselines` got the model+slice label and `_baselines` did not, the
same way the exception path got checkpointing and the timeout path did not.
The table now carries a slice column and one row per slice, and the artifact
is keyed by what identifies a run. Republished: agent-vfc's six families are
back, including access_control at 0.000 recall on four pairs, which is the
number that was missing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Eight of nine group-3 tasks were rejected. Each defect was hiding a number,
and the numbers are why the loop was not learning:
- The fixed side was judged against the *vulnerable* side's function
ranges. 191 of 227 pairs with a resolvable labeled span (84%) have
different spans on the two sides, so a leak was recorded as silence,
always in the direction that flatters Youden.
- Round zero measured every family with the generalist. `family:unknown`
earns no credit against any real family, so every family but `unknown`
baselined at a structural zero and every noise floor with it.
- The cross-family sweep counted absolute failures instead of negative
flips. On a corpus where most pairs fail at round zero that rejects
every round for damage it did not do. Unscorable rows voted too.
- K runs was declared and never threaded; `pairs --k-runs` was parsed and
discarded, and its only test grepped `--help`.
- The meta-agent proposer was a stub, so `learning round` could never
propose anything. It now composes MetaAgent.evolve against a scratch
copy of the family directory, and `--proposals FILE` runs a recorded
sequence offline.
- `refused_holdout` was a vocabulary word no code path could produce, and
the covering test accepted either outcome.
- Cost was metered once, after everything had been paid for; a stage that
raised left no record at all.
- The corrected lever number was published beside the earlier one rather
than in place of it, and the test named for that property never checked.
- Refusals were `continue` statements: `Refusal.degradation()` existed and
nothing called it. A train-split identical twin was reported as
`split:train`, which is untrue.
Test repairs that each changed a number: two split tests measured an
`x = 1` fixture that teaches nothing whatever the rule says; two CLI
scoreboard tests asserted against all-zero output produced by a stub that
answers twice and puts its finding outside the labeled function.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Absence bugs as obligations rather than flows: obligated operations and
discharger kinds as closed facts, obligation shapes learned from trusted
pairs, the application's policy recovered from sibling-handler
consistency and an optional declared policy file, path-aware evaluation
with a function-local degraded mode, honest rungs below proof, an
`obligations` improve lever under the holdout gate. Depends on
reachability-flow-model and guard-dominance-regime; the roadmap marks
reachability as the critical path for two consumers.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
- tasks.md: all tasks checked; rounds 1-6 recorded with RED evidence,
measured leave-one-out and admission numbers, and the recorded design
deviations.
- spec.json: completed 2026-09-05.
- roadmap: corpus-seeded-mechanisms marked completed; the loop is closed
end to end (export -> improve admits under the holdout clause -> scan
searches -> reports name the mechanism).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
- tasks.md: round-12 verdict and the decision to assemble the redaction
test's example key at runtime and rewrite the unpushed series so no
commit carries a contiguous key literal; two dated notes marked as
superseded by tasks 8.1 and 8.4.
- roadmap: the folded inventory-vs-overlay Youden item is checked.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
- tasks.md: all tasks checked; the two escalations are closed (5.1 by the
Req 10 pointer rows, 5.3 by the Req 9 review tiers); round 11 recorded
with the carry-forwards for corpus-seeded-mechanisms and
overlay-ir-completeness.
- roadmap: measured state after tiers and pointer slices (vibe-py 115,
agent-vfc 59, gates byte-identical to pre-spec); pair-corpus-honesty
marked completed.
- spec.json: completed 2026-09-05 after eleven review rounds.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Harvest 176 isolated vuln-vs-fix functions (109 CVEs, 352 training rows)
into benchmarks/pairs/vfc with provenance and a JSONL training manifest.
Score the slice as an honesty dashboard (ousast pairs --slice vfc);
keep the pair gate on the local fixtures only.
Add openultrasast[semantic] with official grammar wheels and one CST walker
into FileIR. quick stays extra-free. Host CLI is not overlay-ready. Default CI
stays extra-free; a semantic job runs pytest -m semantic.
