contributor-scan 2.4, closing the no-go task 2.2 recorded.
VAmPI's SQL injection is a request parameter reaching `get_by_username`
in api_views/users.py, passed to `User.get_user` in models/user_model.py,
interpolated into a query string and executed there. Source and sink are
in different functions and different modules. Regions were per-function,
so neither region saw both ends and the engine was silent on the most
ordinary shape a real injection bug has.
`reachableByFlows` was never the limitation -- it follows calls already.
The limitation was the QUESTION. Sinks were filtered to the labeled
function's own body, so a crossing flow had no sink to end at.
Three changes, each with a reason it is sound rather than convenient:
**An entry point is the trust boundary.** parameter_sources was off for
repository scans because a repository's parameters are not all attacker
input. An entry point's are, by definition -- that is what makes it an
entry point. The driver turns it on only for regions the mapper found and
named, and leaves it off for walker fallbacks and for named functions that
are not entry points.
**Sinks may sit below the handler.** taint.sc admits sinks in methods
reachable from the labeled method through the call graph, bounded at
ENTRY_POINT_CALL_DEPTH = 3: handler -> service -> data access, which is
where these bugs live. Unbounded would make the question about most of the
repository rather than about this entry point, and make it unaffordable.
**The finding names the sink, not the asker.** A crossing flow ends in
another file and another function, so Verdict carries the evidence's own
`path:line:function` and the site uses it. Reporting
`models/user_model.py:73:get_by_username` would name a place that does not
exist: line 73 is inside get_user.
Also: labeled methods are now resolved by name AND file. `get_user` exists
in more than one module of a real repository, which a corpus of one-file
excerpts could never surface.
Measured on the same checkout and budget as 2.2:
VAmPI-SQLI missed -> model_entailed at models/user_model.py:73:get_user
known found 0 of 2 before 2.2's fixes, 1 of 2 in 2.2, 2 of 2 now
entailed 7 -> 8
query time 40.9s -> 37.4s
peak RSS 869MB -> 827MB
cost $0.0067, unchanged
Asking the bigger question cost nothing measurable here: file scoping and
call-graph following pull in opposite directions and roughly cancel at this
size. That is one 520-line repository, and VAmPI is now the repository this
change was developed against -- the exact overfitting shape Req 8 exists to
catch. The depth bound and the entry-point rule mean nothing general until
a checkout nobody tuned against says so.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.2. VAmPI at a pinned commit, scanned whole rather than
as excerpts, with benchmarks/repos/measure.py recording build time, query
time, arbitration time, peak memory, cost, unjudged regions, and whether
each known vulnerability was found and at which rung.
The corpus could not have found any of this. Every defect below needs a
repository to become visible, and three of them were invisible behind a
passing test suite.
**The batch path was unreachable.** query_batch was implemented, tested and
committed while CpgResult exposed only `run`, so the driver's capability
check found nothing and fell back to one JVM per region-family: the first
attempt at this measurement ran eighteen minutes on 520 lines, issuing
`--param function=` one at a time. Every test called query_batch directly,
so none of them noticed. CpgResult now carries run_batch and a test asserts
the capability is reachable through the object the driver is handed.
The earlier 275s->106s improvement attributed to batching came from region
deduplication alone.
**The budget stopped free work.** It bounds model calls, but was enforced
by breaking the region loop -- spending a paid limit to stop unpaid CPG
arbitration. 6 of 22 regions were examined and VAmPI's BOLA, which the
graph entails for nothing, was never reached. It now withholds the judge
only. 22 of 22 regions arbitrated, 17 counted as regions_unasked (silent
because unaffordable is not silent because decided), and the BOLA is
entailed by the model layer.
**A region with no function asked about the whole repository.** Sending
function="" matched every hit in the tree and the driver attributed that
answer to whoever asked: one `host='0.0.0.0'` in app.py became eight
identical entailed findings in eight files that do not contain it. Queries
now carry the region's file. Entailed findings 14 -> 7, query time
70.9s -> 40.9s. The single-request form stays unscoped so the committed
pair measurements remain reproducible.
**The JVM had no heap bound.** Joern claimed a quarter of physical RAM and
runs were killed under memory pressure at different points, so the numbers
described the host rather than the engine. -J-Xmx2048m by default,
OPENULTRASAST_CPG_HEAP_MB to raise it.
Also: arbitrated findings carried `function:line` where the report layer
read `path:line`, so every entailed finding reached a contributor with a
function name where its file belonged; and the witness line was parsed by
splitting on a comma, which produced sites like
`config.py:17): permissive literal '0.0.0.0':?`.
Reading: GO on one CPG per repository -- 6.2s to build, 177s end to end,
869MB, $0.0067, nothing infeasible. NO-GO on region-scoped taint as the
only shape: VAmPI's SQL injection runs from a handler parameter into a
function in another module, and a per-function region sees neither end.
That is the ordinary shape of a real injection bug, so it is recorded as
task 2.4 along with three smaller defects this run exposed.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The pair corpus scores excerpts: a vulnerable function and its fixed twin,
handed to the engine with the answer already narrowed down. Nothing in it
exercises a repository -- region ranking, the budget, one CPG over
thousands of files, or whether a verdict that holds on fifteen lines still
holds when the source, the sink and the guard live in three modules.
benchmarks/repos/<name>.toml names a real checkout: url, a full 40-char
commit SHA, licence, and every known vulnerability with its file, function
and line. `ousast repos` lists them; `--fetch` materialises one; CI does
neither, and conftest clears the switch and redirects the cache so a
developer with warm checkouts cannot read them by accident.
Three rules the loader enforces rather than documents:
- A branch or tag is rejected. Either would make the same recipe mean
different code next month while the baseline it produced kept the old
numbers.
- `in_scope` must be stated per vulnerability, never defaulted. It decides
whether "not found" is a miss or a non-question, which is the most
important thing about the result.
- A recipe that declares `detection` must name something in scope to
detect.
`ousast repos` does not trust a recipe either: against a materialised
checkout it checks HEAD against the pin, that the licence file is there,
and that each declared function appears above its declared line. That
earned its place immediately -- the first libpng recipe named LICENSE.md,
which is the filename at HEAD today and not at the pinned commit, where it
is still LICENSE.
Two checkouts, verified against the real repositories:
- vampi (MIT, 520 lines) is the DETECTION checkout. Excerpts of it are
already in the pair corpus, including the update_password miss diagnosed
earlier, so a whole-repository run says directly whether a verdict
survives the move off the excerpt. Its routes register through an
OpenAPI operationId, so it exercises registration-based entry-point
mapping rather than the decorator path.
- libpng (libpng-2.0, 91k lines of C) is the ENVELOPE checkout, pinned at
the parent of the commit that added the missing clamp. Its CVE is marked
out of scope: an out-of-bounds read is not a family this engine
arbitrates, and recording "not found" against it as a miss is the
easiest way to make a measurement lie.
Nothing is vendored; the recipe records the licence so an artifact can say
what it ran against.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
JVM startup dominated a scan: a call per region per family put a ten-line
file at 275s over three regions, and would have put a thousand regions at
roughly fifty hours. The queries themselves are milliseconds once the CPG
is loaded, so the cost was almost entirely process starts.
scan_repository() is now collect -> batch -> arbitrate. The three .sc
scripts accept a `requests` JSON map and answer all of them in one run,
keeping the single-request form so the committed measurements stay
reproducible byte-for-byte. Arbiters are untouched: each is handed a
CpgResult whose `run` returns its own prefetched rows.
The batcher builds parameters by calling request_params() extracted from
taint.py, dominance.py and config_value.py rather than reconstructing
them. A second copy that drifts from the arbiter's is the exact shape of
the five family-assumption bugs the predecessor spec spent a session
finding; there is now one definition per family and the batcher borrows
it.
Regions are also deduplicated by (path, function), collapsing the
nameless entry-point rows that were producing `model:injection:?:10`.
Measured on the same ten-line file: 275s -> 106s, three duplicate
findings -> one `model:injection:ping:10`, rung unchanged at
model_entailed. Gates byte-identical.
Server mode is recorded as task 2.3, deliberately sequenced after the
repository measurement: whether the remaining ~100s floor is worth a
process model, a weakened subprocess seam and a Scala-evaluating HTTP
endpoint depends on a build-time-to-query-time ratio only 2.2 produces.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
`model-grounded-detection` finished 16/16 with the model layer having zero
production callers. This closes that. A real scan of a Flask file now reports:
by_rung: {"model_entailed": 3, "model_corroborated": 0, "suspicion": 0}
model:injection:ping:10 <- os.system("ping -c 1 " + request.args["host"])
The stage is the fifth MAP proposer, mirroring _check_obligations, and is
additive: pattern, overlay and obligation findings all keep working, and without
a CPG engine it records cpg_unavailable and changes nothing else. Two scan
integrity tests pin the exact degradation set and both broke -- correctly. Their
`requested == baseline` assertion, that findings are byte-identical, still
passed, so the only change is that the scan now SAYS the arbiter was absent
instead of being silently without one. The guards are updated to assert the
model degradation specifically rather than being loosened.
ModelLayerConfig, not ModelConfig: that name is already the [models] block, and
the type checker caught the collision immediately. Two meanings of one name is
how a scan silently reads the wrong settings.
Every payload carries the same shape, skipped ones included. A payload that is
sometimes a full record and sometimes a single "skipped" key is how a report
ends up quietly omitting a section.
unjudged_sample names the lowest-ranked regions the budget did not reach.
Counting them is the honesty requirement; naming them is what lets a maintainer
write a large-repository exclusion list from evidence rather than guesswork.
The run also produced the finding behind new task 1.4: 275 seconds for a
ten-line file. cost_usd is 0.0, so those 21 invocations were not model calls --
they were Joern query launches, one per region per family, each a JVM start.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Pair scoring could rely on a labelled function; a contributor cannot supply one
-- which code is risky is the question they are asking. Regions now come from
entry points, with a file-level fallback that says it is one, and each carries
only the families its shape and language admit. Ranking uses what
EntryPointRecord already decided (access_level, trust_boundary) rather than a
second opinion computed from the same evidence.
A family is offered only where its arbiter could answer: access_control needs a
handler, so a library file with no entry point is not asked about obligations it
has no siblings to be inconsistent with.
The driver makes the two costs that were free at pair scale explicit. ONE CPG
per repository, reused across regions -- a build is 5-30s on a forty-line
excerpt and per-region builds would make this unusable. And a budget that is a
TOTAL for the run rather than the enumerator's per-region cap of eight, spent
highest-rank-first, with whatever it does not reach counted into
regions_unjudged. A tool that silently analyses a tenth of a repository and
reports a clean result is worse than one that says what it skipped.
Two flaws the tests caught, both mine and both about ownership: the budget
counted calls by reading a client's own usage log, so it silently never tripped
for a client that keeps none -- the driver now counts its own spend through a
wrapper; and the driver trusted its caller to have sorted, when the order the
budget is spent in belongs to whoever holds the budget.
The third was worse and only surfaced because a test failed for the right
reason: the driver passed candidates=() and never enumerated, so a repository
scan would have reported only what the arbiters establish unaided and never
populated the suspicion band -- which supplied 15 of 26 findings in the pair
measurement. It enumerates now.
Sibling audit (Req 8.5) done before writing: family routing lives in exactly one
production place, pipeline._arbitrate, dispatching on spec type. regions_for
returns family names and the driver resolves specs, so that stays true.
750 passing, mypy clean over 96 files, gates byte-identical, zero orphans.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Asked whether the prior spec is finished. Audited rather than counted, and the
answer is: all 16 tasks are complete as written, and not all requirements are
satisfied in the product.
Four are built but unreachable, so the requirement is not met where it matters:
model/report.py implements the per-slice and overfitting-gap discipline (Req
11.1) and has no production caller; resolve_execution_backend returns
NullExecutionBackend on BOTH branches so the execution_confirmed rung (Req 10.1)
can never be reached; harvest_recipe (Req 9.4) is a seam with nothing plugged
into it; and the pipeline that measures 90.0% pair-correct is called only by a
measurement harness, never by ousast scan.
That is a spec defect rather than an execution one. No task ever said "make this
reachable", and Req 11.2 -- the gates stay byte-identical -- is satisfied
PRECISELY BECAUSE nothing is wired. I read that as reassurance for a whole
session when it was evidence.
The spec is not reopened; it remains the complete record of building and
measuring the arbiter. contributor-scan takes all four: its Req 1 closes 8.x and
11.1 in the product, Req 7.1 forces the memory decision 10.1 defers, and Req 9
covers the harvest 9.4 left as a seam.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Requirements approved, design and tasks generated. 47 criteria, all traced;
13 tasks in 6 groups.
The design is deliberately conservative. The model layer joins as ONE STAGE
INSIDE MAP, beside the obligations checker it most resembles -- which already
takes entry points, produces findings and merges them. Findings are added, never
substituted; the pattern and overlay paths keep working; a machine without Joern
scans exactly as it does today. Nothing here is allowed to make the tool worse
for someone who does not have the engine.
Two decisions carry the most weight. One CPG per scan rather than per region: a
build is 5-30s on an excerpt and is the dominant cost at repository scale, so
per-region builds would make this unusable. And one total budget rather than the
enumerator's per-region cap of eight, which is meaningless across thousands of
regions; the budget is spent highest-rank-first and whatever it does not reach is
counted and reported, never silently truncated.
Task 2.2 is a go/no-go and comes before shipping: the first scan of a pinned
known-vulnerable checkout is a measurement nobody has taken, and if one CPG per
repository proves infeasible the region and budget design changes shape. Better
to learn that before groups 3-6 are built on it.
Group 5 (corpus) is last on purpose. The binding constraint today is
reachability, not corpus size -- 198 of 238 OWASP pairs went unvendored because
volume without readability makes every number worse.
The task preamble carries the predecessor's two hard-won habits: audit sibling
paths in the same commit as any routing or matching change (five bugs shared
that shape, each caught by a measurement rather than a check), and run the whole
CI list rather than the parts one remembers (CI was red for a session because
`ruff check` passing was taken as evidence for `ruff format --check`).
Not approved for implementation.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
10 requirements, 46 criteria. A NEW spec rather than an amendment to
model-grounded-detection, which is complete and closed.
Req 1-8 cover the gap that prompted it: the model layer has zero production
callers, so ousast scan must use it (1), regions must come from entry points
rather than a labelled function (2), ranking and budget must work at repository
scale where the enumerator's per-region cap of eight is meaningless (3), the
performance envelope must be measured on a real checkout (4), output must be
actionable (5), packaging must be built and verified rather than merely defined
(6), the two deferred decisions must close (7), and the honesty disciplines
carry forward (8).
Req 9 and 10 were added after the maintainer asked whether their requests had
been captured. They had not been -- I folded the corpus asks implicitly into
"run against a known-vulnerable repository", which is not the same thing. Now
explicit: a PHP/WordPress slice (php2cpg works once an interpreter is present,
which the image provides, and plugin CVEs are overwhelmingly the families the
model arbitrates well); real-world C library CVEs -- libpng, image libraries,
OpenSSH, libpam, kernel modules -- gated on the memory decision, because until
it closes they would join 173 pairs already idle; more vibe-code pairs, since
that slice is where the stated audience works and it is only 35 pairs; and
regression tuning over a pinned large-repository baseline, which is the thing
pair scoring structurally cannot see.
Req 9.4 carries today's lesson: verify on a sample that the model can READ a
slice before vendoring it. The OWASP harvest found 238 pairs and vendored 40
for exactly that reason.
Not approved.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Asked what the spec state was and whether anything was forgotten. It was, and
it is the largest thing outstanding: model-grounded-detection is 16/16 tasks
complete and its arbiter measures 86.7% pair-correct at 6.7% leak, but the
model layer has ZERO production callers. `ousast scan` still runs
preprocess -> map -> overlay -> obligations -> regress -> report and touches
none of it. Every number quoted today came from a measurement harness in a
scratch directory.
That is a scope gap in the predecessor spec rather than a slip in executing it:
there was never a task saying "make this reachable", and its Req 11.2 (gates
stay byte-identical) is satisfied PRECISELY BECAUSE nothing is wired. The spec
is complete and the feature is unshipped, both at once.
A new spec rather than an amendment, because the predecessor is coherent and
closed -- it built an arbiter and measured it -- while this is a different
question: making it something a developer runs on a repository they are about
to contribute to.
What it covers, in the order it bites: scan integration; whole-repo scanning
(entry-point region selection, ranking across thousands of candidates, a budget
policy -- the enumerator caps at 8 per region, meaningless at repo scale -- and
a performance envelope never measured); running against a known-vulnerable
checkout, since the 176 real-world CVE pairs are single-function excerpts that
exercise none of it; the deferred memory decision, with 173 of those pairs idle
behind a stub tier; and retiring evidence_level.
It also records what was built outside any spec: the pipeline itself, the ops
work (one image, compose, shell wrappers, pyinfra -- with the Docker image
never actually built), and the corpus additions.
And it carries the known limits forward rather than losing them: the flat
permissive_values list that cannot tell debug=True from httponly=True,
interprocedural resolution as the most repeated miss cause, and the five bugs
that shared one shape -- a taint-specific assumption applied to every family,
each caught by a measurement rather than a check.
Not approved.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The rung this project deliberately does not build. Clearwing already runs
sandbox lifecycles, container pools, sanitizer images, PoC replay and stability
classification, and the corrective brief concluded -- from this project's own
defect history -- that reimplementing that machinery is where we go wrong. So
model/execution.py decides eligibility, calls an adopted tool, and records what
happened. A test asserts structurally that it imports no container or
subprocess machinery and defines no build/run/compile/patch function, because
the failure mode to prevent is this file quietly growing a sandbox.
Eligibility is narrow in both directions: only a family a static model cannot
arbitrate (memory safety in C, where the bug turns on runtime layout -- Req
10.3 forbids any web/logic family depending on this tier, and each now has
taint, dominance or constant abstraction instead), and only a finding the model
left undecided. The tier escalates uncertainty; it never re-checks a verdict.
One deliberate asymmetry: a failure to reproduce is NOT a refutation. A sandbox
that does not trigger a bug has not shown the bug is absent, only that this
attempt did not trigger it, so the candidate stays where the model left it.
Reading a failed reproduction as a clean bill of health is the same error class
as reading an absent engine as "no vulnerability".
With no adopted tool present -- the normal case -- the rung is never reached and
clearwing_unavailable is recorded.
All 16 tasks of model-grounded-detection are now done. 720 passing, mypy/ruff
clean, gates byte-identical, zero orphaned modules.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The boxes were left unticked and the notes unwritten when group 5 was
committed. Records the closure fix and the three bugs it uncovered, each of
which would have raised the number while making the answer untrue, and the
correction that req.url was not a wrong diagnosis but an untestable one -- the
scoping bug blocked everything downstream of it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Full model layer over 92 pairs: 39.1% model_entailed (36/92), 45.6% reaching
any verdict, 23.9% distinguished -- an 18x lift in entailment from an arbiter
that makes no model calls at all.
Per family: untrusted_destination 0.80 entailed, injection 0.58 (0.36
distinguished), deserialization 0.33, prototype 0.14, config_secrets 0.10,
path 0.00. access_control reaches 6 of 8 pairs at 0.38 distinguished -- the
class no crash oracle can reach, arbitrated deterministically.
A hypothesis tested and discarded. From the group-2 misses I diagnosed path's
0/8 as a missing req.url source token and said so. The probe -- the same 8
pairs with and without the token -- returned 0/8 -> 0/8. Falsified. The real
cause is that the sink sits lexically inside an anonymous callback, so its
enclosing method is the closure (<lambda>0) and the sinkMethod == function
filter drops it; verdict(function="") on the same CPG returns model_entailed.
And the obvious fix for that is wrong. Relaxing the filter raises the number
with a FALSE witness: the flows found in the closure are `res -> readFileSync`
and `this -> ...`, neither of which is the vulnerability, whose real path
crosses a helper (req.url -> url -> resolveUrl -> possibleFilename). Turning a
miss into a wrong entailment is worse than the miss -- it is the arbiter
asserting something untrue, the one failure mode this architecture exists to
prevent. The correct fix is two-part and recorded: scope a sink to any method
lexically nested in the labeled function, and scope parameter sources to that
function's own parameters, then re-measure counting only real witnesses.
Every miss is classified and none is architectural: 23 sink absent from the
excerpt or unmodelled, 13 no recognisable source in the excerpt, 3 source
unmodelled with the idiom named.
Two honesty items in the artifact. 16 of 92 pairs carry no function label and
are arbitrated over the whole file, so a verdict can credit a different finding
than the labeled one -- owasp-python-hash (CWE-327) is credited for a
permissive set_cookie elsewhere in its file, on both sides. And config_secrets
conflates three abstractions: permissive setting values (implemented), weak
algorithm choice (facts already carry weak_literals, unwired) and hardcoded
secrets (needs a credential-literal check).
694 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
3.1 adds StaticFinding.rung beside evidence_level rather than renaming it. The
existing evidence_level carries `static_corroboration`, emitted by the flat IR
overlay -- a path Req 4.4 demotes to the suspicion band. Restating that as
model_corroborated would claim a CPG arbitrated something it never saw, and
rewriting every legacy finding to suspicion would be a product-visible
regression nobody asked for. So `rung` defaults to suspicion and is raised only
by ladder.at_rung(finding, verdict), which is the structural form of Req 5.4:
nothing rises above suspicion without a Verdict behind it. The witness travels
with the rung, because a rung without its evidence is just a louder assertion.
It reaches markdown, JSON and SARIF.
3.2 is the judge, and it is defined as much by what it lacks. Three exhaustive
outcomes: the model entails (report it, never call the model), the model
contradicts (drop the claim, record why), the model cannot decide (report
suspicion). The LLM gets one bounded typed question with tools=[] so it cannot
search for the site, asked only about the residual the graph cannot settle --
whether a sanitizer on a REAL flow suffices. Its answer can only confirm a flow
the CPG already found; it can never conjure one.
Req 8.5 is asserted structurally: a test parses judge.py and fails on any AST
name like k_runs/majority/vote/average/acceptance/floor/budget, any for/while
loop, or an import from the deleted round machinery. That test first matched
the word "votes" in the module's own docstring, which is why it now inspects
code rather than text.
Verified live against a real CPG, not only scripted rows:
vuln rung=model_entailed request.args["name"] -> db.execute(...)
fixed rung=suspicion no source reaches this sink
model calls made: 0
Both sides arbitrated by the graph with the LLM never consulted -- the thing
the prior architecture could not do at any price.
678 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Injection slice, 50 pairs, ~30 minutes of real engine time.
model_entailed on the vulnerable side 54% (27/50) vs the flat IR's 2.2%
... on minimal-fix pairs only 61.5% (16/26)
pairs distinguished 32% overall, 38.5% on minimal-fix
determinism (Req 6.4) every verdict identical on re-run
cost median 32s/pair, max 58s
CPG build failures 1
The engine is not the constraint: all 23 misses trace to our own data model --
11 sinks absent from the fact tables (Django ORM objects.raw, self.query,
GraphQL resolvers), 11 where the sink is modelled but no source reaches it, and
one C translation unit that does not parse standalone, counted separately so it
can never read as "no vulnerability". That is exactly the result that justifies
adopting a CPG rather than building one.
The finding that changes the design: classifying what the 50 injection FIXES
actually do gives other_or_validation 37, bound_parameters 10, argv_list 2,
sanitizer_call 1. The dominant fix is a validation guard on the path, not a
sanitizer and not a sink shape -- and taint reachability can never separate
those pairs, because the flow still reaches the sink and what changed is that a
guard now dominates the path. The arbiter must be TAINT + DOMINANCE, which is
the same primitive task 4.1 builds for access_control, with the guard
vocabulary ported in task 1.1 as its lookup list. Two planned mechanisms
collapse into one, and 4.1 should now precede the remaining TaintSpecs.
Reported per stratum because only 26 of 50 pairs are genuine minimal-fix pairs:
12 are different programs (local-python-vulnerable is 7% similar with 83
changed lines) and 12 are large rewrites. A raw distinguished_rate would
measure corpus quality and report it as model quality.
Also corrected in the notes: the Flask-shaped source table is a real gap but a
minor lever -- adding Django patterns would gain a source for exactly one pair.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The first live run produced the finding that shapes this design: taint
reachability alone does not distinguish a fix from a bug. On the canonical SQL
pair the flow request.args["name"] -> name -> execute exists on BOTH sides --
vuln db.execute("select ... '" + name + "'") one argument, interpolated
fixed db.execute("select ... = %s", (name,)) two arguments, bound
-- so the model entailed the fixed twin exactly as readily as the vulnerable
one, and arbitrated nothing. What separates them is the SHAPE of the sink call,
which is precisely what safe_shape_sinks preserved from the earlier sanitizer
fix. The query now reports sinkArity and sinkArg0Literal, and a flow ending at
a sink written in its declared-safe form is not a vulnerability.
Two bugs in that shape test, both found only against the real engine:
Joern's `call.argument` includes the RECEIVER at argumentIndex 0, so `db` was
counted as an argument -- making a one-argument interpolated call look like a
two-argument bound one, the exact inversion of the test -- and the "first
argument" whose literalness decides a format-string sink was the receiver
rather than the format. Both fixed with argumentIndexGt(0).
The rule is narrow on purpose: only sinks named in safe_shape_sinks are subject
to it (os.system(x) is not safe merely because it takes one argument), and a
row carrying no shape fields is judged on the flow alone -- an unknown shape
must never be read as a safe one.
End to end on the pair, deterministic and with no model call:
vuln -> model_entailed request.args["name"] -> db.execute(...) (line 4)
fixed -> None
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
99 source modules -> 81. 832 tests -> 646. The whole learning/ package (16
modules, ~4,100 lines) and six semantic/ modules removed, with the three gates
byte-identical to 2026-09-06-gate-baseline.txt at every step.
The sequence is what made it safe: vocabularies ported first (1.1), importers
detached second (1.2-1.3), deletion third (1.4). Every import break during the
deletion was therefore a detachment miss in the importer and was fixed there --
no module was restored to satisfy one. Both misses were the same shape and were
fixed as a class: operations.py and shapes.py each imported trailing_name from
the deleted variants.py, so it moved to semantic/ir.py, its natural home, and
both importers changed in one edit.
Ported before deleting: the closed guard vocabulary (verbatim, nothing else
holds it) into model/specs.py, which derives TaintSpec and DominanceSpec from
the retained ruleset facts rather than copying them. Re-homed: families ->
model/taxonomy, candidates -> model/candidates, endpoint -> model/endpoint,
plus Region, labeled_family and labeled_spans. labeled_family no longer falls
back to the LLM classifier -- an unlabeled pair is honestly unknown.
Four deviations, each recorded in the spec notes with its reason. improve/
evolve.py and validator.py were stripped, not deleted: each mixes the mechanism
lever (gone) with the deterministic rule-status loop `ousast improve` still
runs, which has no LLM in it; a test now asserts the lever's absence by symbol
so it cannot creep back. The learning CLI group became `model candidates`,
because deleting it outright would have orphaned the retained enumerator by
removing its only entry point. thinking moved from [learning] to [models].
A gap named rather than papered over: no retained sink fact routes by CWE to
path, output_encoding, untrusted_destination or prototype -- the same hole the
entailment ceiling diagnosed. The test asserts the seeded set and the gap set
exactly, so task 4.2 must update it when it authors those sinks.
The audit found exactly one genuine orphan: vectorstore.py, a bakeoff harness
with no production caller and no CLI, alive only through its own test. The five
named subsystems each carry an explicit decision. Manifest committed, with a
test asserting the tree still matches it.
Group 2 (the Joern go/no-go) has not started.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The prior commit's edit did not match and 7.4 was left unclaimed. model/specs.py
is read-only data -- no LLM and no optimiser writes it -- and task 1.1 now
carries that criterion with a test asserting the module exposes no writer. All
49 criteria trace to a task.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Design approved. All 49 criteria trace to a task.
The tasks carry an agent-guidance preamble aimed squarely at the escalations
this session hit: the eighteen-defect lesson (fix the class, not the instance),
the gate-baseline as the safety net (the whole learning/ package is off the
gate path, so its deletion cannot move a gate -- a moved gate means you touched
a retained path), the zero-dep capability-detection pattern to copy verbatim,
and -- most important for the subtraction -- a full coupling map naming every
touchpoint of the removed code (cli, pairs hunter path, config, reports,
semantic seed/loo, the mechanism loop across improve/ and semantic/). The
subtraction is not discovery by import error.
Group 1 is pure gate-protected subtraction, sequenced so vocabularies are
ported (1.1) and importers detached (1.2-1.3) before anything is deleted (1.4),
then tests reduced in the same spirit (1.5) and the tree audited for orphans
(1.6). Group 2 is a Joern go/no-go: task 2.3 reads go or stop, and groups 3-6
do not begin until it is green -- building past a red 2.3 is the mistake the
whole spec exists to prevent. Parallelism is deliberately absent; the
subtraction is sequential and the spike gates the rest.
Not approved for implementation.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Requirements approved. All 49 criteria trace to a component.
The shape: a Code Property Graph (Joern) is the arbiter; taint reachability and
guard dominance produce evidence-ladder verdicts (suspicion / corroborated /
entailed / execution_confirmed); the LLM answers one bounded question per
candidate and its claim is checked against the CPG, never averaged.
Two design disciplines carried from the requirements:
Phase 1 is subtraction, and it removes ~5,000 lines before a line of Joern is
written. The learning/ noise architecture is absent from the gate path, so
rounds/acceptance/proposer/journal, the K-run/floor/sign-test in scoring, and
the mechanism loop delete without moving a gate -- proven by the committed
gate-baseline run before and after, and by an import-graph check that fails the
build if a removed module is still imported. The guard and sink vocabularies
are ported into model/specs.py *before* their modules are deleted. Tests are
reduced in the same change, with a test->requirement map. A module-audit
manifest classifies all 99 modules and names fusion/mcp/skills/harness_ext/
hunter_harness for an explicit keep-or-remove decision.
Phase 2 is a Joern spike on the injection family alone -- the CpgBackend seam
(subprocess to joern-parse + a CPGQL query emitting JSON, out of process, one
isolated touch-point) and one TaintSpec, end to end, with a determinism test.
It can end the design before the integration cost is paid, the same way the
candidate ceiling could have in the prior spec. Joern is capability-detected on
PATH like the semantic extra; without it every verdict degrades to suspicion
and the core stays zero-dependency.
What survives re-homes under model/: the taxonomy, the candidate enumerator
(suspicion band), the endpoint client, and the obligation facts feeding the
dominance queries.
Not approved.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
11 requirements, 49 criteria. The outgrowth is addressed head-on and it leads,
because building on a superseded 99-module / 832-test surface is how this
session logged eighteen defects.
Req 1-3 are subtraction and come first:
1 remove the noise architecture by name -- rounds, acceptance, proposer,
journal, the K-run/floor/sign-test in scoring, the mechanism lever and
shape-search loop -- preserving the security *models* they carried. The
whole learning/ noise layer is absent from the gate path, so it deletes
without moving a gate.
2 reduce the 832 tests to what protects surviving behavior: a test that
pinned a deleted mechanism dies with it, every survivor maps to a
surviving requirement, and the "does nothing / greps --help / passes with
the capability absent" tests are removed outright.
3 audit all 99 modules -- load-bearing / standalone / orphaned -- and delete
the orphans, with fusion/mcp/skills/harness_ext/hunter_harness named for
an explicit keep-or-remove decision.
Req 4-8 adopt the CPG arbiter: Joern behind an optional extra (core stays
zero-dep, degrades to suspicion without it), the evidence ladder as the verdict
type, deterministic taint-reachability and guard-dominance verdicts, our
taxonomy and source/sink/sanitizer/guard models ported to CPG queries, and the
LLM reduced to a proposer whose claim the model checks -- no K-runs, no floor,
no acceptance rule, no evolve loop. Req 9 makes the corpus the model's
calibration set (a pair the model can't split is a named model gap). Req 10
keeps execution as a deferred, adopted-from-Clearwing tier for memory/C only.
Not approved.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Two design questions answered by the entailment measurement.
IL, not AST. Our semantic/ir.py is a lossy AST projection -- flat call/bind
lists, no CFG, no PDG -- which is *why* entailment is 2.2%. Guard dominance is
a dominator-tree question and path-sensitive taint needs branch structure;
a flat bind list expresses neither. Real abstract interpretation is a fixpoint
over a CFG, so the substrate is a Code Property Graph: AST + CFG + PDG + call
graph in one queryable graph, taint as source->sink-minus-sanitizer
reachability, absence as dominance.
Adopt, not build. Building a multi-language CFG/PDG/taint engine on tree-sitter
is reinventing CodeQL/Semgrep/Joern -- the maximal case of this session's
failure mode. Joern is Apache-2.0, covers our exact corpus
(C/C++/Java/JS/Python/Kotlin/binary), has a taint engine, is embeddable, and a
CPG+LLM MCP bridge already exists. The 2.2% on our flat IR is the measured
distance to a real CPG, and closing it by hand is reimplementing Joern.
The flow: source -> Joern CPG -> taint + dominance -> deterministic
entailed/corroborated verdict -> LLM judges the residual, checked against the
CPG. What stays ours is the contribution: the taxonomy and the
source/sink/sanitizer/guard models for the web/logic CWEs, ported to CPG
queries, plus the LLM-checked-against-the-model orchestration. Joern is a
workbench with no opinion on these classes; that opinion is our layer. Named
cost: Joern is JVM/Scala, so server-mode or the MCP bridge, not in-process.
Three-layer adoption now: Joern CPG for the web/logic arbiter, Clearwing for
the memory/C execution tier, and our security modelling + LLM orchestration as
the layer that binds them.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The analogue of the candidate ceiling, one rung up: not "can a site be
enumerated" (96.6%) but "can a deterministic abstract-interpretation model
DECIDE the vulnerability without trusting the LLM". Offline, no model calls.
model_entailed today 2.2% (2/92 web/logic pairs)
model_corroborated, one-hop taint 24% (22/92)
taint reachable, multi-hop 35% (26/74 flow families)
candidate site enumerable 96.6% (for comparison)
The gap between 96.6% and 2.2% is the gap between enumerating and arbitrating,
and it is the finding. The model cannot be a pure arbiter at v1. But the
measurement diagnoses depth, not impossibility: one-hop source resolution
(real taint sits 1-3 binds upstream: q = '...'+id; execute(q)), a closed
literal sink table (misses project wrappers and path/xss/prototype sinks), and
a function-naming gap that eats 20/50 injection pairs. Deepening the taint to
the candidate enumerator's depth-4 chain already moved injection 24% -> 48%.
So the course is a layered arbiter, not a pure one: report model_entailed
where the model proves it (deterministic, no LLM); let the LLM judge but check
its claim against the resolved taint on the ~35% corroboration band (noise
removed there); honest suspicion for the rest, which is where the deferred
execution/differential tier earns its place. Modeling confirmed as abstract
interpretation; SMT reserved for the memory/C tier; concolic avoided.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Maintainer correction: execution isn't possible for all SAST targets, and the
bugs this project owns -- IDOR, missing authz, injection -- don't crash
anything. Execution is the easy arbiter only for memory-safety-in-C, which is
not our world.
What execution provided was an independent, deterministic arbiter of whether a
claim is real -- not the same LLM that made it, not a noisy vote. That arbiter
can be built statically, from the code, as a program model the claim is checked
against. LLM proposes; the model disposes. Same architecture, same repair of
the root cause, no runnable target required.
The evidence ladder becomes static and deterministic:
suspicion -> model_corroborated (claim consistent with resolved dataflow /
a guard its siblings carry is absent) -> model_entailed (the model alone
establishes it) -> execution_confirmed (deferred tier, C memory-safety only).
model_corroborated and above are deterministic -- run the checker twice, same
verdict -- which removes the run-to-run noise the same way a crash would, with
no K-runs and no target to build. Our IR candidate enumerator (96.6% ceiling),
binding chains and obligation checker stop being prompt context and become the
arbiter itself.
The course: promote those assets from inputs to arbiter, delete the noise
layer, make the LLM a proposer the model audits, keep the corpus as the
model's calibration set (the fix diff is what the model should see), and defer
execution to Clearwing for the memory-safety classes only. Build the arbiter we
can build from code; adopt the one we cannot, when a class demands it.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Written after five evolve rounds produced zero decisions distinguishable from
noise, and after reading Clearwing, Big Sleep, and the AIxCC 2025 finals.
The premise we got wrong: OpenUltraSAST treats the LLM as a noisy oracle to
average (K runs), floor (noise budget), gate (sign test) and slowly improve
(evolve loop). 120 runs produced 120 distinct trajectories -- the variance is
the loop's freedom to search, not the model's uncertainty. And the premise is
empirically false as a claim about LLMs: Big Sleep found a real SQLite zero-day
in Nov 2024; AIxCC finalists uncovered 18 real 0-days across 54M LOC, with the
2nd-place system using non-reasoning LLMs at 90% accuracy and $181/point. The
lever is architecture, not prompt optimization.
What the winners share, and Clearwing is the open instance of: the LLM is a
judgement layer on an execution substrate, and truth comes from the sandbox --
a crash, a sanitizer report, a patch that stops the crash -- not from a score
against a labeled corpus. Clearwing's poc_runner (apply diff, recompile,
re-run, does-the-crash-still-happen) replaces our entire noise architecture. We
built K-runs because we had no ground truth; they generate their own.
The course: adopt Clearwing as the substrate (the maintainer already forks it),
delete the noise architecture rather than fix it, and contribute the one thing
Clearwing lacks -- the web/logic detection path. Our IR candidate enumerator
(12.1% ruleset vs 96.6% IR, measured) and our obligation/absence-bug checker
reach the dangerous web bugs that never crash a sandbox, which Clearwing's
crash-oracle model cannot. The open research problem is a differential oracle
for those classes: the vulnerable-parent/fix-commit pair as ground truth, with
a request the fixed version rejects and the vulnerable one serves.
Recommendation is explicitly not to rebuild: eighteen defects this session,
all ours, against a 19k-line battle-tested MIT substrate. Not approved.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Round 5 re-ran round 1's content through the lever it should have used --
counterexamples, rendered to the detector as "Worked examples" rather than
"Known non-issues, do not report these" -- and was rejected because one pair
of eight moved, at p = 1.000.
So the experiment I designed to give the demo lever a fair test could not
give it one. Correcting my authoring error changed the reason for the verdict
and not the verdict, because the rule cannot tell a lever that helps from a
lever that does nothing.
Five rounds, $2.17, zero decisions distinguishable from chance.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Four rounds on injection, $1.76. Not one decision carried evidence, in either
direction. Round 2 rejected a change because one pair of eight moved, at
p = 1.000 -- the maximum the sign test can return. Round 3 accepted one at
p = 0.500 and that acceptance is now versioned as v1. On a family where 11 of
20 pairs disagree with themselves between runs of the unchanged detector,
both readings are what changing nothing looks like.
I corrected an overstatement in the notes: after two rejections I wrote that
the rule was near-incapable of accepting. Round 3 accepted. The accurate
statement is symmetric and worse -- it accepts on noise as readily as it
rejects on it.
Escalated, not applied: Req 9.5's threshold rule should be replaced by an
acceptance that survives the measured variance. The Pareto archive and the
sign test both exist and neither decides anything. The honest caveat is that
at a minibatch of 8 a reliability-gated rule would accept almost nothing
either -- the corpus is the binding constraint, not the rule.
And the finding that outranks the rounds, in the maintainer's words: the
error is not the model, the error is the elsewhere. Eighteen defects this
session, every one in this repository, three of them the same shape of fixing
an instance and missing the class. The default assumption for an unexplained
number is now that we broke it -- which puts pro's 46.7% against flash's
83.3% on our side of the ledger until shown otherwise.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Group 1 complete. Tasks 1.2 and 1.3: `candidates.ceiling`, `ousast learning
candidates`, and the stop-or-go reading. No model is called anywhere in this
group, so the condition that could have ended the feature was reachable for
free.
Verdict: go.
agent-vfc (deciding) 89.7% ceiling against 44.4% round-zero detection
vfc (deciding) 84.7%, baseline still running, non-gating
vfc-js (deciding) 52.9% -- see below
github (deciding) 100.0%
vibe-py (development)100.0% against 83.3%
The margin is largest on agent-vfc, which is the slice the tool is actually
for.
The measurement forced a shape correction before the detector existed.
config_secrets first read 63.6% with a bind-and-literal shape, because a
permissive default is as often an argument to a call -- cors({origin: '*'}),
app.use(session({secure: false})) -- as it is a binding. Adding call sites to
that family's shape took it to 100% and recovered four agent-vfc rows.
Carry-forward, and not this feature's to fix: vfc-js sits at 52.9% because
eight of seventeen rows label a function the JavaScript CST names `<anon>` --
`module.exports.publish = function () {}` and its relatives. That is a
function-naming gap, it caps the existing hunter on that slice exactly as
much, and it is the same class as the `_declarator_start` defects found in
learning-harness task 4.1.
The ruleset arm of `ceiling` is kept so the 12.1% that rejected it stays a
measurement rather than a remembered number.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Design approved in session. Every one of the 36 criteria is claimed by a task.
Group 1 calls no model at all. The candidate enumerator, the ceiling report
and the stop-or-go reading are offline, so the condition that ends this
feature -- a deciding slice whose ceiling sits below its recorded detection
rate -- is reachable before a cent is spent. The ruleset variant of this same
design would have failed that condition at 12.1% against 64.9%.
Group 2 is the only work that touches scoring.py, and its observable is that
nothing changes: the recorded round-zero artifacts are the pin, and if they
cannot be reproduced exactly then the refactor is wrong, not the numbers.
Group 3 builds the detector, and task 3.3 is the test that would have caught
the defect this spec exists for -- two runs must make an identical sequence of
model calls. It lives in the suite and runs repeatedly, not in a one-off
measurement.
Group 4 is the maintainer's emphasis as code: deciding slices, development
slices, the cross-slice rejection by name, and the overfitting gap printed
beside every headline number.
Group 5 compares the two detectors per slice, re-measures the trajectory count
and the noise floor, and names -- without amending -- the learning-harness
requirements that a collapsed floor would retire.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Requirements approved in session. All 36 criteria trace to a component.
The shape: a deterministic pass enumerates candidate sites from the semantic
IR, shaped by family -- call sites for injection and path, operations with
their guards for access control, bindings and literals for configuration. The
model answers one bounded typed question per candidate with no tools and no
choice of what to look at. The predicates that decide whether a finding counts
are applied while it is being made, with one retry carrying the violation, and
they come from a single definition that the scorer also uses, so the two
cannot drift.
Determinism is a design property, not a hope: candidates are ordered by
(path, line, name) and batched at a fixed size, so two runs make an identical
sequence of model calls. That is the direct answer to 120 runs producing 120
distinct trajectories.
Phase 1 is the candidate ceiling, offline, no model call, measured on the
real-project slices as well as the teaching one. If the ceiling on a deciding
slice comes in below that slice's current detection rate, the design stops
there having spent nothing. Building that in as the first phase is deliberate:
the ruleset variant of this design would have capped recall at 12.1% against a
64.9% baseline, and only a measurement caught it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Renamed from detector-optimizer: the brief's own conclusion is that the
optimizer is not the next thing to build, and a spec should be named for what
it delivers.
Seven requirements, 37 criteria. The load-bearing ones:
Req 1 -- candidates come from the semantic IR, never the pattern ruleset,
and the enumerator's ceiling is published as the limit on recall. Measured
before building: ruleset 12.1%, IR 96.6%.
Req 2 -- one bounded typed question per candidate, no tools, no choice of
what to look at, and a test asserts two runs make an identical sequence of
model calls.
Req 3 is the maintainer's emphasis and it decides acceptance. The slices
harvested from real projects decide; hand-written vibe-py is a development
slice that cannot on its own justify a change; a change that improves vibe-py
and regresses a deciding slice is rejected by name; and the overfitting gap
-- 83.3% against 44.4% at baseline -- is published beside every headline
number. The candidate ceiling is measured separately on real-world slices,
because their functions are larger and the enumerator's behaviour there is
not implied by its behaviour on teaching code.
Req 5 -- the K-run and noise-floor machinery is re-measured against the
constrained detector and deleted if the floor collapses, naming the
learning-harness requirements that would be amended.
Not approved; design not started.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
learning-harness divides cleanly. The measurement layer stands and is worth
more than before -- it is what made a $0.23 experiment able to answer 'is
our own instability structural?'. What was answering the wrong question:
Req 3.5's K (an ensemble of distinct search procedures, not a repeated
measurement), Req 8.2's noise floor (it measures how often an unconstrained
search reaches the sink, so the regression budget grows with the search's own
wandering), Req 9.5's acceptance rule, and the sign test added this session.
And the highest-leverage change in either spec turns out not to be a prompt,
a lever or an optimizer: stop asking the model to locate the sink. 555 grep
calls against 71 entry_points, while the CST and the entry-point mapper
already know where the sinks are.
Limits stated in the brief: twelve pairs, one family, one model; the
high-similarity bucket is two run pairs and its 0% is decoration.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Twelve injection pairs, both sides, K=5, temperature 0, thinking disabled,
recording every tool call. $0.23.
The statistic I designed for could not be computed: across 120 runs there was
not one repeated trajectory -- 4.67 and 4.92 distinct trajectories per side
out of 5, and 78 distinct tool-name sequences ignoring arguments entirely.
The detector is not sampling one measurement five times; it is running five
different investigations and voting on the results.
As a gradient the relationship is monotone: runs whose trajectories were
similar (>=0.75 Jaccard) disagreed 0% of the time, mid-similarity 4.2%,
low-similarity 30.9%.
Three more facts. The search is grep: 555 grep_repo calls against 181
read_file, 71 entry_points, 43 find_refs, 13 read_definition -- the model is
guessing patterns to locate sinks the CST and the entry-point mapper already
know. All the instability is on the vulnerable side; every pair produced
exactly one finding set on the fixed side, so the noise is in detection and
none of it in false positives. And findings converge much more than
trajectories do: 78 trajectories, one to three finding sets per pair.
The floor is therefore not measuring uncertainty about a question. It is
measuring how often an unconstrained search happens to reach the sink.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The harness treats the detector as a black box emitting a noisy scalar, and
K runs, majority voting, the measured floor, the regression budget and the
sign test all follow from that. It is the stance you take when you cannot
see inside what you are measuring, and we can.
Temperature is already 0 and thinking is disabled, and 11 of 20 injection
pairs still disagree with themselves. The freedom is in the trajectory --
which files get read, which greps get run -- and we built statistics to
average over it instead of removing it.
Three DSPy constructs attack the source: constraints enforced as predicates
at generation time (we already compute every one of them, after the fact, at
scoring time), typed decomposition of the single ReAct loop into steps whose
deterministic parts we already answer without a model, and best-of-N against
a checkable reward instead of a majority vote that discards the
disagreement. The cheaper the detector model, the more of the return sits
there and the less in prompt evolution -- the opposite of the order this
brief first proposed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
SIMBA ranks instances by max-to-min score gap across variants, which turns
our noise floor from an obstacle into an instance sampler: the 11 injection
pairs that disagree with themselves are the pairs where a rule change can
flip an outcome. Its two alternating strategies -- append a demo, append a
rule -- are the split our levers already have.
GEPA's reflection meta-prompt is a specification for the feedback function,
not just a prompt: it asks the reflector to extract domain facts the model
cannot re-derive later, which means feedback must carry which sink, which
span and which side leaked, not a verdict.
And the finding that outranks both: GEPA splits a trainset, SIMBA batches 32.
Injection has 12 train teachers. These algorithms are sample-efficient in
rollouts, not in instances, and they will not shortcut a corpus of twelve.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The learning harness built the measurement honestly and then hill-climbed on
it by hand. Four review rounds' worth of defects in that loop were defects of
its shape, not its code.
Read from GEPA's Algorithm 1 and 2 rather than from a summary, three things
bear directly on ours: the acceptance gate is deliberately cheap and noisy,
because the Pareto frontier -- not the gate -- is what protects against a bad
accept; selection is stochastic and diversity-preserving, which is what stops
the search collapsing onto one lineage; and the expensive full evaluation is
spent only on candidates that already passed the cheap gate.
Two consequences for us are honesty findings, not performance ones. Our
acceptance rule reads the holdout every round, so repeated rounds overfit the
test set -- the same leak class this project already measured once, one level
up. And the proposer is handed a score where the published work hands it
textual feedback from the evaluation trace, which our evaluator produces and
then discards.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Round zero and one evolve round run end to end with the scripted client over
the whole vibe-py slice: per-family and per-slice blocks, twins out by
reason, a floor each, a proposal applied and reverted byte for byte, the
sweep reporting zero negative flips, the rejected version archived. The only
thing the live run adds is a detector that can see.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Measured on the corpus under the split and review-tier rules, access_control
has 1 train teacher and 4 holdout pairs: every agent-vfc row sits at tier
`title` and the vibe-py absence rows that could teach are trap pairs whose
two sides are different handlers, so the train split teaches zero obligation
shapes. injection has 12 teachers and 10 holdout pairs, the only family with
enough train evidence to tell a real change from noise.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Three defects, two of them the same shape as ones already rejected once:
- The meta-agent proposer still could not propose. `MetaAgent(inner_model=)`
takes a ModelConfig; it was handed a model name, a `# type: ignore` hid
that from mypy, and a bare `except Exception` turned the resulting
AttributeError into `meta_agent_failed` -- which reads exactly like the
model declining. The composition is now a seam a test can pin, and a
construction error is re-raised rather than reported as a quiet round.
- `ousast pairs --hunter` crashed in its documented form: the K floor landed
with `--k-runs` still defaulting to 1. The covering test always passed
`--k-runs 4`, which is why nothing noticed.
- 28 rows (12% of the corpus) resolved a span on the vulnerable side and
none on the fixed side, so every fixed-side finding read as outside the
labeled function and the pair scored as silence whatever the detector
said. `fix_function` now reaches the catalog and the scorer, and the
generator derives it from the fixed excerpt for the 25 trap rows that
never declared one. Measured after: 0 rows.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The three gate outputs are captured at 71efc9c -- the commit before the
first line of this feature -- and a test reproduces them byte for byte in
both matrices. Byte-identity was something a reviewer checked by hand
against a worktree; it is now mechanical and runs in CI.
Six absences are each exercised once with the reason they record: HarnessX,
the chat endpoint (the run still writes its artifact), the family
configurations, the network for pointer pairs, the semantic extra (named,
not guessed), and the sandbox -- so a claim nothing could run is not
confused with a claim that failed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The catalog generator fills `fix_date` from the recipe at the precision the
recipe has -- a full ISO day when one was recorded, otherwise the year.
Every harvested recipe carries a year and almost none carries a day, and
writing `-01-01` would turn a year into a claim about a month. A row counts
as post-cutoff only when its earliest possible date is after the cutoff, so
a coarse date can under-claim the clean slice but never over-claim it.
Each family now reports the same numbers over the rows fixed after the
cutoff, beside the whole number and never instead of it, with the undated
count on its own line. Without a configured cutoff every scoreboard says so
rather than inventing one: guessing a model's training cutoff and then
publishing a contamination number from the guess would be worse than
reporting none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The five byte-identical vibe-py twins now say so in the catalog instead of
only hashing to it, and a declared limit that names one of the unscorable
vocabulary's own reasons keeps that spelling, so `identical_twin` stays one
fact. Leave-one-out over vibe-py consequently runs on 30 rows, not 35.
The difficulty check measures three axes the corpus varies on: binding hops
from the identity source to the constrained use, registration distance, and
guard distance. Measured first, and it failed: the shared obligations
fixture reached (1, 0, 2) where the vibe-py absence rows reach (5, 2, 2).
Every fixture handler carried its own route decorator, so nothing in the
suite exercised a handler registered from another document and no binding
chain was longer than one hop -- the checker was passing on a simpler world
than the one it meets. `from_spec` closes that with the connexion shape
those rows really have. The verdict is committed beside the numbers.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Eight of nine group-3 tasks were rejected. Each defect was hiding a number,
and the numbers are why the loop was not learning:
- The fixed side was judged against the *vulnerable* side's function
ranges. 191 of 227 pairs with a resolvable labeled span (84%) have
different spans on the two sides, so a leak was recorded as silence,
always in the direction that flatters Youden.
- Round zero measured every family with the generalist. `family:unknown`
earns no credit against any real family, so every family but `unknown`
baselined at a structural zero and every noise floor with it.
- The cross-family sweep counted absolute failures instead of negative
flips. On a corpus where most pairs fail at round zero that rejects
every round for damage it did not do. Unscorable rows voted too.
- K runs was declared and never threaded; `pairs --k-runs` was parsed and
discarded, and its only test grepped `--help`.
- The meta-agent proposer was a stub, so `learning round` could never
propose anything. It now composes MetaAgent.evolve against a scratch
copy of the family directory, and `--proposals FILE` runs a recorded
sequence offline.
- `refused_holdout` was a vocabulary word no code path could produce, and
the covering test accepted either outcome.
- Cost was metered once, after everything had been paid for; a stage that
raised left no record at all.
- The corrected lever number was published beside the earlier one rather
than in place of it, and the test named for that property never checked.
- Refusals were `continue` statements: `Refusal.degradation()` existed and
nothing called it. A train-split identical twin was reported as
`split:train`, which is untrue.
Test repairs that each changed a number: two split tests measured an
`x = 1` fixture that teaches nothing whatever the rule says; two CLI
scoreboard tests asserted against all-zero output produced by a stub that
answers twice and puts its finding outside the labeled function.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Eight of the ten vendored access-control rows had no named handler at all,
so every one of them would have scored `missing_context` forever rather
than as a detector miss. After the re-harvest nine are named and the tenth
states its reason.
The handler-context mode now carries the class its framework registers,
multi-line decorators, and commented-out guards -- `#@token_required`
above a ThreatByte handler is the entire bug. Where the registration lives
in another document (connexion's `operationId`, an express router), a
recipe names `context = [...]`, the harvester writes that stanza per side,
and the pair materializes both files at their repository-relative paths so
the entry-point mapper sees the layout the project has. `harvest_recipe`
fetches every blob before the first write and the driver takes `--report`,
so a failed fetch leaves a reviewed excerpt exactly as it was.
Two defects the re-harvest exposed: `.mjs` was not a known suffix, so the
one pair vendored as an ES module had no file target at all; and
`_python_route_access` matched auth words inside decorator prose, reading
`@ns.doc(description='... without proper authorization checks')` as a
guard. Recorded in the hygiene artifact: nine of the ten duplicate groups
sit on both sides of the split, which the repository-level split cannot
see.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The maintainer settled the conflict the group 2 review escalated: a detector
configuration has no skills set. The skill router belongs to the tool's earlier
shape, and a family's prompt, checklist, tools and counterexamples already carry
what a round evolves.
The README described the project as an editor-agent harness whose skills steer
it, and pointed users at the maintainer's spec directory. The tool is a CLI now
and the agent integrations are optional, so the opening says that, the skills
section describes what it actually is, and the spec pointer is gone from the
user-facing text.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The maintainer settled both items the group 1 review escalated.
The family that covers server-side request forgery is named untrusted
destination and also covers open redirect, because both are untrusted input
choosing where a request goes and the corpus carries 34 open-redirect rows that
would otherwise be unclassifiable. The requirement, the brief and the design
example now say so, and the approval note records the amendment.
The shipped taxonomy is flat, so no answer can earn hierarchical partial credit.
Rather than leave that as prose, two new acceptance criteria require every
agreement figure and every family figure to say it, and both report types now
carry the flag, computed from the taxonomy rather than asserted. A taxonomy that
later declares a parent flips it without any other change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
