4.1 is the arbiter for the class no crash oracle reaches. An access-control bug
is the ABSENCE of a guard: no flow to follow, no crash to reproduce. What the
model can establish is a consistency violation -- an obligated operation no
discharging guard governs, in a file where its siblings are governed by one.
"Governed" turned out not to be one relation, and asking pure CFG dominance of
the operation gets two of three shapes wrong. Measured against the real engine:
identity constraint the guard is INSIDE the operation's arguments
filter_by(owner_id = current_user.id)
identity branch abort(403).controlledBy == [note.owner_id != current_user.id]
-- the check runs AFTER the fetch, so it never dominates
the operation, but it controls whether the value escapes
dominating guard a discharger-naming call that dominates the operation
That is why the query uses controlledBy rather than dominatedBy for shape 2.
Two bugs found by running it, both the same class as the sanitizer bug in group
2 -- part of a fact flattened, the part carrying the signal dropped:
* dominance_specs used only `calls` and `decorators`, omitting
identity_sources (current_user, request.user) and constraint_params
(owner_id, user_id) -- which is how an identity constraint is actually
written. The arbiter reported "no guard" on handlers that plainly had one.
request_sources and permissive_values stay excluded: they describe the
vulnerable shape, and including them would let the arbiter read a bug as
its own fix.
* matching operations by code containment yielded four or five desugared rows
per operation (the bare `tmp1.filter_by` receiver among them, carrying none
of the guards the real call has), so a guarded operation could be flagged.
Matching is by call name now.
Live, deterministic, zero model calls:
leaky -> model_entailed no discharging guard, while 2 sibling handlers are guarded
edit_note -> no finding (ownership check controls the denial)
list_notes -> no finding (identity constraint in the query)
4.2 authors the sink facts the go/no-go measured as its largest miss class (11
of 23): Django ORM raw SQL, and the four families that previously had no fact
routing to them by CWE at all -- path, output_encoding, untrusted_destination,
prototype. Each entry is a documented dangerous API of the named library with
its provenance recorded inline, and rule_ids left empty because a sink fact is
not a detection rule. Flagged for maintainer review as data, given the standing
"no LLM-authored patterns" constraint. The exact-set assertion written in task
1.1 to hold this gap open is updated rather than deleted, so it still fails if a
family ever loses its seed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
3.1 adds StaticFinding.rung beside evidence_level rather than renaming it. The
existing evidence_level carries `static_corroboration`, emitted by the flat IR
overlay -- a path Req 4.4 demotes to the suspicion band. Restating that as
model_corroborated would claim a CPG arbitrated something it never saw, and
rewriting every legacy finding to suspicion would be a product-visible
regression nobody asked for. So `rung` defaults to suspicion and is raised only
by ladder.at_rung(finding, verdict), which is the structural form of Req 5.4:
nothing rises above suspicion without a Verdict behind it. The witness travels
with the rung, because a rung without its evidence is just a louder assertion.
It reaches markdown, JSON and SARIF.
3.2 is the judge, and it is defined as much by what it lacks. Three exhaustive
outcomes: the model entails (report it, never call the model), the model
contradicts (drop the claim, record why), the model cannot decide (report
suspicion). The LLM gets one bounded typed question with tools=[] so it cannot
search for the site, asked only about the residual the graph cannot settle --
whether a sanitizer on a REAL flow suffices. Its answer can only confirm a flow
the CPG already found; it can never conjure one.
Req 8.5 is asserted structurally: a test parses judge.py and fails on any AST
name like k_runs/majority/vote/average/acceptance/floor/budget, any for/while
loop, or an import from the deleted round machinery. That test first matched
the word "votes" in the module's own docstring, which is why it now inspects
code rather than text.
Verified live against a real CPG, not only scripted rows:
vuln rung=model_entailed request.args["name"] -> db.execute(...)
fixed rung=suspicion no source reaches this sink
model calls made: 0
Both sides arbitrated by the graph with the LLM never consulted -- the thing
the prior architecture could not do at any price.
678 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Injection slice, 50 pairs, ~30 minutes of real engine time.
model_entailed on the vulnerable side 54% (27/50) vs the flat IR's 2.2%
... on minimal-fix pairs only 61.5% (16/26)
pairs distinguished 32% overall, 38.5% on minimal-fix
determinism (Req 6.4) every verdict identical on re-run
cost median 32s/pair, max 58s
CPG build failures 1
The engine is not the constraint: all 23 misses trace to our own data model --
11 sinks absent from the fact tables (Django ORM objects.raw, self.query,
GraphQL resolvers), 11 where the sink is modelled but no source reaches it, and
one C translation unit that does not parse standalone, counted separately so it
can never read as "no vulnerability". That is exactly the result that justifies
adopting a CPG rather than building one.
The finding that changes the design: classifying what the 50 injection FIXES
actually do gives other_or_validation 37, bound_parameters 10, argv_list 2,
sanitizer_call 1. The dominant fix is a validation guard on the path, not a
sanitizer and not a sink shape -- and taint reachability can never separate
those pairs, because the flow still reaches the sink and what changed is that a
guard now dominates the path. The arbiter must be TAINT + DOMINANCE, which is
the same primitive task 4.1 builds for access_control, with the guard
vocabulary ported in task 1.1 as its lookup list. Two planned mechanisms
collapse into one, and 4.1 should now precede the remaining TaintSpecs.
Reported per stratum because only 26 of 50 pairs are genuine minimal-fix pairs:
12 are different programs (local-python-vulnerable is 7% similar with 83
changed lines) and 12 are large rewrites. A raw distinguished_rate would
measure corpus quality and report it as model quality.
Also corrected in the notes: the Flask-shaped source table is a real gap but a
minor lever -- adding Django patterns would gain a source for exactly one pair.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The whole engine integration is one subprocess boundary: joern-parse builds a
cpg.bin, joern --script runs a shipped CPGQL query that prints its JSON between
fence markers, because Joern's stdout carries a banner and a REPL prompt around
whatever a script emits. Nothing is imported in-process, so the core install
keeps dependencies = [] -- a test walks the seam's AST to assert it imports
only the standard library and names no JVM binding.
Fail closed everywhere. A build or query that errors, times out, or prints
something unparseable yields no verdict and the caller reports suspicion. The
taint verdict distinguishes this from absence deliberately: a query returning
None means "the engine could not decide", which is not "there is no flow", and
conflating them is how a missing engine silently becomes a clean bill of health.
The verdict is where the ladder earns its keep. An unsanitized source->sink
path is model_entailed -- the graph establishes it and the LLM's claim is no
longer load-bearing. A path a sanitizer may break is model_corroborated: the
flow is real, but whether the sanitizer suffices is the semantic residual the
LLM answers, checked against this verdict rather than averaged over runs. The
chosen flow is picked by a total order (shortest, then lexicographic), so two
runs over one CPG cannot disagree -- Req 6.4 asserted directly.
The query itself is the part our flat IR could not do: reachableByFlows walks
the PDG, so `q = "..." + request.args["x"]; execute(q)` is found, which is
exactly the one-hop limit that held measured entailment at 2.2%.
Capability detection reuses semantic/engines.joern_available rather than
growing a second notion of "installed". The audit manifest is regenerated; the
new modules are named by Req 4/5/6 rather than counted as orphans.
Task 2.3, the go/no-go measurement, needs Joern installed and has not run.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
99 source modules -> 81. 832 tests -> 646. The whole learning/ package (16
modules, ~4,100 lines) and six semantic/ modules removed, with the three gates
byte-identical to 2026-09-06-gate-baseline.txt at every step.
The sequence is what made it safe: vocabularies ported first (1.1), importers
detached second (1.2-1.3), deletion third (1.4). Every import break during the
deletion was therefore a detachment miss in the importer and was fixed there --
no module was restored to satisfy one. Both misses were the same shape and were
fixed as a class: operations.py and shapes.py each imported trailing_name from
the deleted variants.py, so it moved to semantic/ir.py, its natural home, and
both importers changed in one edit.
Ported before deleting: the closed guard vocabulary (verbatim, nothing else
holds it) into model/specs.py, which derives TaintSpec and DominanceSpec from
the retained ruleset facts rather than copying them. Re-homed: families ->
model/taxonomy, candidates -> model/candidates, endpoint -> model/endpoint,
plus Region, labeled_family and labeled_spans. labeled_family no longer falls
back to the LLM classifier -- an unlabeled pair is honestly unknown.
Four deviations, each recorded in the spec notes with its reason. improve/
evolve.py and validator.py were stripped, not deleted: each mixes the mechanism
lever (gone) with the deterministic rule-status loop `ousast improve` still
runs, which has no LLM in it; a test now asserts the lever's absence by symbol
so it cannot creep back. The learning CLI group became `model candidates`,
because deleting it outright would have orphaned the retained enumerator by
removing its only entry point. thinking moved from [learning] to [models].
A gap named rather than papered over: no retained sink fact routes by CWE to
path, output_encoding, untrusted_destination or prototype -- the same hole the
entailment ceiling diagnosed. The test asserts the seeded set and the gap set
exactly, so task 4.2 must update it when it authors those sinks.
The audit found exactly one genuine orphan: vectorstore.py, a bakeoff harness
with no production caller and no CLI, alive only through its own test. The five
named subsystems each carry an explicit decision. Manifest committed, with a
test asserting the tree still matches it.
Group 2 (the Joern go/no-go) has not started.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The analogue of the candidate ceiling, one rung up: not "can a site be
enumerated" (96.6%) but "can a deterministic abstract-interpretation model
DECIDE the vulnerability without trusting the LLM". Offline, no model calls.
model_entailed today 2.2% (2/92 web/logic pairs)
model_corroborated, one-hop taint 24% (22/92)
taint reachable, multi-hop 35% (26/74 flow families)
candidate site enumerable 96.6% (for comparison)
The gap between 96.6% and 2.2% is the gap between enumerating and arbitrating,
and it is the finding. The model cannot be a pure arbiter at v1. But the
measurement diagnoses depth, not impossibility: one-hop source resolution
(real taint sits 1-3 binds upstream: q = '...'+id; execute(q)), a closed
literal sink table (misses project wrappers and path/xss/prototype sinks), and
a function-naming gap that eats 20/50 injection pairs. Deepening the taint to
the candidate enumerator's depth-4 chain already moved injection 24% -> 48%.
So the course is a layered arbiter, not a pure one: report model_entailed
where the model proves it (deterministic, no LLM); let the LLM judge but check
its claim against the resolved taint on the ~35% corroboration band (noise
removed there); honest suspicion for the rest, which is where the deferred
execution/differential tier earns its place. Modeling confirmed as abstract
interpretation; SMT reserved for the memory/C tier; concolic avoided.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Round 5 re-ran round 1's content through the lever it should have used --
counterexamples, rendered to the detector as "Worked examples" rather than
"Known non-issues, do not report these" -- and was rejected because one pair
of eight moved, at p = 1.000.
So the experiment I designed to give the demo lever a fair test could not
give it one. Correcting my authoring error changed the reason for the verdict
and not the verdict, because the rule cannot tell a lever that helps from a
lever that does nothing.
Five rounds, $2.17, zero decisions distinguishable from chance.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Four rounds on injection, $1.76. Not one decision carried evidence, in either
direction. Round 2 rejected a change because one pair of eight moved, at
p = 1.000 -- the maximum the sign test can return. Round 3 accepted one at
p = 0.500 and that acceptance is now versioned as v1. On a family where 11 of
20 pairs disagree with themselves between runs of the unchanged detector,
both readings are what changing nothing looks like.
I corrected an overstatement in the notes: after two rejections I wrote that
the rule was near-incapable of accepting. Round 3 accepted. The accurate
statement is symmetric and worse -- it accepts on noise as readily as it
rejects on it.
Escalated, not applied: Req 9.5's threshold rule should be replaced by an
acceptance that survives the measured variance. The Pareto archive and the
sign test both exist and neither decides anything. The honest caveat is that
at a minibatch of 8 a reliability-gated rule would accept almost nothing
either -- the corpus is the binding constraint, not the rule.
And the finding that outranks the rounds, in the maintainer's words: the
error is not the model, the error is the elsewhere. Eighteen defects this
session, every one in this repository, three of them the same shape of fixing
an instance and missing the class. The default assumption for an unexplained
number is now that we broke it -- which puts pro's 46.7% against flash's
83.3% on our side of the ledger until shown otherwise.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
`publish` keyed baselines by model alone, so the vibe-py run silently
replaced the agent-vfc run and the table showed hand-written teaching code as
if it were the corpus. That is precisely the failure the constrained-detector
requirements exist to prevent, live in a published artifact.
Third time today that a key was fixed in one place and not another --
`compare_baselines` got the model+slice label and `_baselines` did not, the
same way the exception path got checkpointing and the timeout path did not.
The table now carries a slice column and one row per slice, and the artifact
is keyed by what identifies a run. Republished: agent-vfc's six families are
back, including access_control at 0.000 recall on four pairs, which is the
number that was missing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Group 1 complete. Tasks 1.2 and 1.3: `candidates.ceiling`, `ousast learning
candidates`, and the stop-or-go reading. No model is called anywhere in this
group, so the condition that could have ended the feature was reachable for
free.
Verdict: go.
agent-vfc (deciding) 89.7% ceiling against 44.4% round-zero detection
vfc (deciding) 84.7%, baseline still running, non-gating
vfc-js (deciding) 52.9% -- see below
github (deciding) 100.0%
vibe-py (development)100.0% against 83.3%
The margin is largest on agent-vfc, which is the slice the tool is actually
for.
The measurement forced a shape correction before the detector existed.
config_secrets first read 63.6% with a bind-and-literal shape, because a
permissive default is as often an argument to a call -- cors({origin: '*'}),
app.use(session({secure: false})) -- as it is a binding. Adding call sites to
that family's shape took it to 100% and recovered four agent-vfc rows.
Carry-forward, and not this feature's to fix: vfc-js sits at 52.9% because
eight of seventeen rows label a function the JavaScript CST names `<anon>` --
`module.exports.publish = function () {}` and its relatives. That is a
function-naming gap, it caps the existing hunter on that slice exactly as
much, and it is the same class as the `_declarator_start` defects found in
learning-harness task 4.1.
The ruleset arm of `ceiling` is kept so the 12.1% that rejected it stays a
measurement rather than a remembered number.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
A candidate-driven detector can only judge sites something else enumerates,
so the enumerator caps recall. Measured over the 58 scorable web pairs before
building anything:
pattern ruleset: 12.1% ceiling -- and the misses are not near-misses, the
ruleset finds nothing at all in most of these files
semantic IR: 96.6% ceiling, a median of ten call sites and bindings
inside the labeled function
The ruleset would have capped us at 12.1% against the 64.9% the unconstrained
loop already reaches. That design was one commit from being built.
Two corpus defects found on the way, both mine from task 4.1. The overleaf
excerpt started mid-import, because `_declarator_start` walks back to the
nearest `}` and Overleaf writes JavaScript without semicolons, so the nearest
one was inside `import { expressify }`. It then ended mid-object-literal,
because registration mentions were appended as bare lines. Neither parsed,
and a pair that cannot be parsed cannot be scored. Registration mentions now
expand to the statement that contains them; the ceiling moved 94.8% -> 96.6%.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Twelve injection pairs, both sides, K=5, temperature 0, thinking disabled,
recording every tool call. $0.23.
The statistic I designed for could not be computed: across 120 runs there was
not one repeated trajectory -- 4.67 and 4.92 distinct trajectories per side
out of 5, and 78 distinct tool-name sequences ignoring arguments entirely.
The detector is not sampling one measurement five times; it is running five
different investigations and voting on the results.
As a gradient the relationship is monotone: runs whose trajectories were
similar (>=0.75 Jaccard) disagreed 0% of the time, mid-similarity 4.2%,
low-similarity 30.9%.
Three more facts. The search is grep: 555 grep_repo calls against 181
read_file, 71 entry_points, 43 find_refs, 13 read_definition -- the model is
guessing patterns to locate sinks the CST and the entry-point mapper already
know. All the instability is on the vulnerable side; every pair produced
exactly one finding set on the fixed side, so the noise is in detection and
none of it in false positives. And findings converge much more than
trajectories do: 78 trajectories, one to three finding sets per pair.
The floor is therefore not measuring uncertainty about a question. It is
measuring how often an unconstrained search happens to reach the sink.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The three gate outputs are captured at 71efc9c -- the commit before the
first line of this feature -- and a test reproduces them byte for byte in
both matrices. Byte-identity was something a reviewer checked by hand
against a worktree; it is now mechanical and runs in CI.
Six absences are each exercised once with the reason they record: HarnessX,
the chat endpoint (the run still writes its artifact), the family
configurations, the network for pointer pairs, the semantic extra (named,
not guessed), and the sandbox -- so a claim nothing could run is not
confused with a claim that failed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The five byte-identical vibe-py twins now say so in the catalog instead of
only hashing to it, and a declared limit that names one of the unscorable
vocabulary's own reasons keeps that spelling, so `identical_twin` stays one
fact. Leave-one-out over vibe-py consequently runs on 30 rows, not 35.
The difficulty check measures three axes the corpus varies on: binding hops
from the identity source to the constrained use, registration distance, and
guard distance. Measured first, and it failed: the shared obligations
fixture reached (1, 0, 2) where the vibe-py absence rows reach (5, 2, 2).
Every fixture handler carried its own route decorator, so nothing in the
suite exercised a handler registered from another document and no binding
chain was longer than one hop -- the checker was passing on a simpler world
than the one it meets. `from_spec` closes that with the connexion shape
those rows really have. The verdict is committed beside the numbers.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Eight of the ten vendored access-control rows had no named handler at all,
so every one of them would have scored `missing_context` forever rather
than as a detector miss. After the re-harvest nine are named and the tenth
states its reason.
The handler-context mode now carries the class its framework registers,
multi-line decorators, and commented-out guards -- `#@token_required`
above a ThreatByte handler is the entire bug. Where the registration lives
in another document (connexion's `operationId`, an express router), a
recipe names `context = [...]`, the harvester writes that stanza per side,
and the pair materializes both files at their repository-relative paths so
the entry-point mapper sees the layout the project has. `harvest_recipe`
fetches every blob before the first write and the driver takes `--report`,
so a failed fetch leaves a reviewed excerpt exactly as it was.
Two defects the re-harvest exposed: `.mjs` was not a known suffix, so the
one pair vendored as an ES module had no file target at all; and
`_python_route_access` matched auth words inside decorator prose, reading
`@ns.doc(description='... without proper authorization checks')` as a
guard. Recorded in the hygiene artifact: nine of the ten duplicate groups
sit on both sides of the split, which the repository-level split cannot
see.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.1. A teacher is a vendored pair, at a review tier a maintainer stands
behind, on the train split, and scorable. Every learning path now asks the same
module that one question, so the split cannot be enforced in one place and
forgotten in another. A refusal is data rather than an exception: the path
records which holdout pairs a candidate touched and moves on, so a leak is
visible in the run instead of silently accepted.
The exporter refuses to seed from a holdout pair, leave-one-out teaches from the
train split while still scoring every pair, and the mechanism lever refuses a
candidate a holdout pair taught, because such a shape can only recover the pair
that produced it.
Measured and committed: on the vibe-py holdout the lever's own number falls from
seven detected and a Youden of +0.059 to five detected and -0.059 once holdout
pairs stop teaching, and the candidate set shrinks from ten shapes to six. That
gap is the inflation this rule removes.
Two existing tests encoded the leak and now encode the truth. The documented
export-to-improve flow had a single holdout pair teaching itself, and now needs
a train pair to teach and a holdout pair to recover. The obligation export
claimed the corpus seeds at least one obligation shape, which was true only
because holdout pairs were teaching: every vibe-py absence pair that yields an
obligation lesson sits on the holdout side, so the honest number today is zero.
The derivation stays covered by fixtures; the corpus gap belongs to the absence
re-harvest and to a maintainer pass over the splits.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.3. An expected row may carry a family label, stored opaquely because the
classifier owns the taxonomy and the corpus module never imports the learning
package. Every pair carries a computed reason when it cannot be scored: a
declared known limit wins, otherwise identical twin when the two excerpts are
the same code once the provenance header is stripped. No row is ever dropped.
Two corpus defects the new accounting surfaces, committed as an artifact beside
the numbers they cap. Nineteen repositories sit on both sides of the split, so a
learning path could be taught by one file of a repository and evaluated on
another. Ten groups of pairs share a vulnerable excerpt byte for byte, mostly
the same fix harvested twice at different granularity.
The split helper keeps every declared split, lets rows that named none inherit
their repository's side, and otherwise assigns a whole repository at once,
oldest first, so nothing straddles by our doing. Pair signals can now be
restricted to one split, which is what lets a learning path teach from train
rows only.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Measured 2026-09-06 on the vibe-py holdout: the existing tool hunter, with no
learning, scores 12/14 pair-correct (Youden +0.857, no leaks) once detection
and silence are keyed on the labeled class and function and the byte-identical
twins are excluded; the overlay scores 5/14 on the same rows. The published
hunter number (0/17, Youden -0.588) was a scoring artifact: the scorer wanted
the literal CWE id in the finding text and the prompt never asked for it.
The brief proposes an auto-classifier as the routing key, one evolved hunter
configuration per class, a class-aware scorer with unscorable denominators,
the split enforced by construction, and the HarnessX evolve loop with per-class
attribution as the round primitive. Raw measurements are committed under
reports/measurements.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
