The prior commit's edit did not match and 7.4 was left unclaimed. model/specs.py
is read-only data -- no LLM and no optimiser writes it -- and task 1.1 now
carries that criterion with a test asserting the module exposes no writer. All
49 criteria trace to a task.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Design approved. All 49 criteria trace to a task.
The tasks carry an agent-guidance preamble aimed squarely at the escalations
this session hit: the eighteen-defect lesson (fix the class, not the instance),
the gate-baseline as the safety net (the whole learning/ package is off the
gate path, so its deletion cannot move a gate -- a moved gate means you touched
a retained path), the zero-dep capability-detection pattern to copy verbatim,
and -- most important for the subtraction -- a full coupling map naming every
touchpoint of the removed code (cli, pairs hunter path, config, reports,
semantic seed/loo, the mechanism loop across improve/ and semantic/). The
subtraction is not discovery by import error.
Group 1 is pure gate-protected subtraction, sequenced so vocabularies are
ported (1.1) and importers detached (1.2-1.3) before anything is deleted (1.4),
then tests reduced in the same spirit (1.5) and the tree audited for orphans
(1.6). Group 2 is a Joern go/no-go: task 2.3 reads go or stop, and groups 3-6
do not begin until it is green -- building past a red 2.3 is the mistake the
whole spec exists to prevent. Parallelism is deliberately absent; the
subtraction is sequential and the spike gates the rest.
Not approved for implementation.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Requirements approved. All 49 criteria trace to a component.
The shape: a Code Property Graph (Joern) is the arbiter; taint reachability and
guard dominance produce evidence-ladder verdicts (suspicion / corroborated /
entailed / execution_confirmed); the LLM answers one bounded question per
candidate and its claim is checked against the CPG, never averaged.
Two design disciplines carried from the requirements:
Phase 1 is subtraction, and it removes ~5,000 lines before a line of Joern is
written. The learning/ noise architecture is absent from the gate path, so
rounds/acceptance/proposer/journal, the K-run/floor/sign-test in scoring, and
the mechanism loop delete without moving a gate -- proven by the committed
gate-baseline run before and after, and by an import-graph check that fails the
build if a removed module is still imported. The guard and sink vocabularies
are ported into model/specs.py *before* their modules are deleted. Tests are
reduced in the same change, with a test->requirement map. A module-audit
manifest classifies all 99 modules and names fusion/mcp/skills/harness_ext/
hunter_harness for an explicit keep-or-remove decision.
Phase 2 is a Joern spike on the injection family alone -- the CpgBackend seam
(subprocess to joern-parse + a CPGQL query emitting JSON, out of process, one
isolated touch-point) and one TaintSpec, end to end, with a determinism test.
It can end the design before the integration cost is paid, the same way the
candidate ceiling could have in the prior spec. Joern is capability-detected on
PATH like the semantic extra; without it every verdict degrades to suspicion
and the core stays zero-dependency.
What survives re-homes under model/: the taxonomy, the candidate enumerator
(suspicion band), the endpoint client, and the obligation facts feeding the
dominance queries.
Not approved.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
11 requirements, 49 criteria. The outgrowth is addressed head-on and it leads,
because building on a superseded 99-module / 832-test surface is how this
session logged eighteen defects.
Req 1-3 are subtraction and come first:
1 remove the noise architecture by name -- rounds, acceptance, proposer,
journal, the K-run/floor/sign-test in scoring, the mechanism lever and
shape-search loop -- preserving the security *models* they carried. The
whole learning/ noise layer is absent from the gate path, so it deletes
without moving a gate.
2 reduce the 832 tests to what protects surviving behavior: a test that
pinned a deleted mechanism dies with it, every survivor maps to a
surviving requirement, and the "does nothing / greps --help / passes with
the capability absent" tests are removed outright.
3 audit all 99 modules -- load-bearing / standalone / orphaned -- and delete
the orphans, with fusion/mcp/skills/harness_ext/hunter_harness named for
an explicit keep-or-remove decision.
Req 4-8 adopt the CPG arbiter: Joern behind an optional extra (core stays
zero-dep, degrades to suspicion without it), the evidence ladder as the verdict
type, deterministic taint-reachability and guard-dominance verdicts, our
taxonomy and source/sink/sanitizer/guard models ported to CPG queries, and the
LLM reduced to a proposer whose claim the model checks -- no K-runs, no floor,
no acceptance rule, no evolve loop. Req 9 makes the corpus the model's
calibration set (a pair the model can't split is a named model gap). Req 10
keeps execution as a deferred, adopted-from-Clearwing tier for memory/C only.
Not approved.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Two design questions answered by the entailment measurement.
IL, not AST. Our semantic/ir.py is a lossy AST projection -- flat call/bind
lists, no CFG, no PDG -- which is *why* entailment is 2.2%. Guard dominance is
a dominator-tree question and path-sensitive taint needs branch structure;
a flat bind list expresses neither. Real abstract interpretation is a fixpoint
over a CFG, so the substrate is a Code Property Graph: AST + CFG + PDG + call
graph in one queryable graph, taint as source->sink-minus-sanitizer
reachability, absence as dominance.
Adopt, not build. Building a multi-language CFG/PDG/taint engine on tree-sitter
is reinventing CodeQL/Semgrep/Joern -- the maximal case of this session's
failure mode. Joern is Apache-2.0, covers our exact corpus
(C/C++/Java/JS/Python/Kotlin/binary), has a taint engine, is embeddable, and a
CPG+LLM MCP bridge already exists. The 2.2% on our flat IR is the measured
distance to a real CPG, and closing it by hand is reimplementing Joern.
The flow: source -> Joern CPG -> taint + dominance -> deterministic
entailed/corroborated verdict -> LLM judges the residual, checked against the
CPG. What stays ours is the contribution: the taxonomy and the
source/sink/sanitizer/guard models for the web/logic CWEs, ported to CPG
queries, plus the LLM-checked-against-the-model orchestration. Joern is a
workbench with no opinion on these classes; that opinion is our layer. Named
cost: Joern is JVM/Scala, so server-mode or the MCP bridge, not in-process.
Three-layer adoption now: Joern CPG for the web/logic arbiter, Clearwing for
the memory/C execution tier, and our security modelling + LLM orchestration as
the layer that binds them.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The analogue of the candidate ceiling, one rung up: not "can a site be
enumerated" (96.6%) but "can a deterministic abstract-interpretation model
DECIDE the vulnerability without trusting the LLM". Offline, no model calls.
model_entailed today 2.2% (2/92 web/logic pairs)
model_corroborated, one-hop taint 24% (22/92)
taint reachable, multi-hop 35% (26/74 flow families)
candidate site enumerable 96.6% (for comparison)
The gap between 96.6% and 2.2% is the gap between enumerating and arbitrating,
and it is the finding. The model cannot be a pure arbiter at v1. But the
measurement diagnoses depth, not impossibility: one-hop source resolution
(real taint sits 1-3 binds upstream: q = '...'+id; execute(q)), a closed
literal sink table (misses project wrappers and path/xss/prototype sinks), and
a function-naming gap that eats 20/50 injection pairs. Deepening the taint to
the candidate enumerator's depth-4 chain already moved injection 24% -> 48%.
So the course is a layered arbiter, not a pure one: report model_entailed
where the model proves it (deterministic, no LLM); let the LLM judge but check
its claim against the resolved taint on the ~35% corroboration band (noise
removed there); honest suspicion for the rest, which is where the deferred
execution/differential tier earns its place. Modeling confirmed as abstract
interpretation; SMT reserved for the memory/C tier; concolic avoided.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Maintainer correction: execution isn't possible for all SAST targets, and the
bugs this project owns -- IDOR, missing authz, injection -- don't crash
anything. Execution is the easy arbiter only for memory-safety-in-C, which is
not our world.
What execution provided was an independent, deterministic arbiter of whether a
claim is real -- not the same LLM that made it, not a noisy vote. That arbiter
can be built statically, from the code, as a program model the claim is checked
against. LLM proposes; the model disposes. Same architecture, same repair of
the root cause, no runnable target required.
The evidence ladder becomes static and deterministic:
suspicion -> model_corroborated (claim consistent with resolved dataflow /
a guard its siblings carry is absent) -> model_entailed (the model alone
establishes it) -> execution_confirmed (deferred tier, C memory-safety only).
model_corroborated and above are deterministic -- run the checker twice, same
verdict -- which removes the run-to-run noise the same way a crash would, with
no K-runs and no target to build. Our IR candidate enumerator (96.6% ceiling),
binding chains and obligation checker stop being prompt context and become the
arbiter itself.
The course: promote those assets from inputs to arbiter, delete the noise
layer, make the LLM a proposer the model audits, keep the corpus as the
model's calibration set (the fix diff is what the model should see), and defer
execution to Clearwing for the memory-safety classes only. Build the arbiter we
can build from code; adopt the one we cannot, when a class demands it.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Written after five evolve rounds produced zero decisions distinguishable from
noise, and after reading Clearwing, Big Sleep, and the AIxCC 2025 finals.
The premise we got wrong: OpenUltraSAST treats the LLM as a noisy oracle to
average (K runs), floor (noise budget), gate (sign test) and slowly improve
(evolve loop). 120 runs produced 120 distinct trajectories -- the variance is
the loop's freedom to search, not the model's uncertainty. And the premise is
empirically false as a claim about LLMs: Big Sleep found a real SQLite zero-day
in Nov 2024; AIxCC finalists uncovered 18 real 0-days across 54M LOC, with the
2nd-place system using non-reasoning LLMs at 90% accuracy and $181/point. The
lever is architecture, not prompt optimization.
What the winners share, and Clearwing is the open instance of: the LLM is a
judgement layer on an execution substrate, and truth comes from the sandbox --
a crash, a sanitizer report, a patch that stops the crash -- not from a score
against a labeled corpus. Clearwing's poc_runner (apply diff, recompile,
re-run, does-the-crash-still-happen) replaces our entire noise architecture. We
built K-runs because we had no ground truth; they generate their own.
The course: adopt Clearwing as the substrate (the maintainer already forks it),
delete the noise architecture rather than fix it, and contribute the one thing
Clearwing lacks -- the web/logic detection path. Our IR candidate enumerator
(12.1% ruleset vs 96.6% IR, measured) and our obligation/absence-bug checker
reach the dangerous web bugs that never crash a sandbox, which Clearwing's
crash-oracle model cannot. The open research problem is a differential oracle
for those classes: the vulnerable-parent/fix-commit pair as ground truth, with
a request the fixed version rejects and the vulnerable one serves.
Recommendation is explicitly not to rebuild: eighteen defects this session,
all ours, against a 19k-line battle-tested MIT substrate. Not approved.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Round 5 re-ran round 1's content through the lever it should have used --
counterexamples, rendered to the detector as "Worked examples" rather than
"Known non-issues, do not report these" -- and was rejected because one pair
of eight moved, at p = 1.000.
So the experiment I designed to give the demo lever a fair test could not
give it one. Correcting my authoring error changed the reason for the verdict
and not the verdict, because the rule cannot tell a lever that helps from a
lever that does nothing.
Five rounds, $2.17, zero decisions distinguishable from chance.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Four rounds on injection, $1.76. Not one decision carried evidence, in either
direction. Round 2 rejected a change because one pair of eight moved, at
p = 1.000 -- the maximum the sign test can return. Round 3 accepted one at
p = 0.500 and that acceptance is now versioned as v1. On a family where 11 of
20 pairs disagree with themselves between runs of the unchanged detector,
both readings are what changing nothing looks like.
I corrected an overstatement in the notes: after two rejections I wrote that
the rule was near-incapable of accepting. Round 3 accepted. The accurate
statement is symmetric and worse -- it accepts on noise as readily as it
rejects on it.
Escalated, not applied: Req 9.5's threshold rule should be replaced by an
acceptance that survives the measured variance. The Pareto archive and the
sign test both exist and neither decides anything. The honest caveat is that
at a minibatch of 8 a reliability-gated rule would accept almost nothing
either -- the corpus is the binding constraint, not the rule.
And the finding that outranks the rounds, in the maintainer's words: the
error is not the model, the error is the elsewhere. Eighteen defects this
session, every one in this repository, three of them the same shape of fixing
an instance and missing the class. The default assumption for an unexplained
number is now that we broke it -- which puts pro's 46.7% against flash's
83.3% on our side of the ledger until shown otherwise.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
`publish` keyed baselines by model alone, so the vibe-py run silently
replaced the agent-vfc run and the table showed hand-written teaching code as
if it were the corpus. That is precisely the failure the constrained-detector
requirements exist to prevent, live in a published artifact.
Third time today that a key was fixed in one place and not another --
`compare_baselines` got the model+slice label and `_baselines` did not, the
same way the exception path got checkpointing and the timeout path did not.
The table now carries a slice column and one row per slice, and the artifact
is keyed by what identifies a run. Republished: agent-vfc's six families are
back, including access_control at 0.000 recall on four pairs, which is the
number that was missing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Group 1 complete. Tasks 1.2 and 1.3: `candidates.ceiling`, `ousast learning
candidates`, and the stop-or-go reading. No model is called anywhere in this
group, so the condition that could have ended the feature was reachable for
free.
Verdict: go.
agent-vfc (deciding) 89.7% ceiling against 44.4% round-zero detection
vfc (deciding) 84.7%, baseline still running, non-gating
vfc-js (deciding) 52.9% -- see below
github (deciding) 100.0%
vibe-py (development)100.0% against 83.3%
The margin is largest on agent-vfc, which is the slice the tool is actually
for.
The measurement forced a shape correction before the detector existed.
config_secrets first read 63.6% with a bind-and-literal shape, because a
permissive default is as often an argument to a call -- cors({origin: '*'}),
app.use(session({secure: false})) -- as it is a binding. Adding call sites to
that family's shape took it to 100% and recovered four agent-vfc rows.
Carry-forward, and not this feature's to fix: vfc-js sits at 52.9% because
eight of seventeen rows label a function the JavaScript CST names `<anon>` --
`module.exports.publish = function () {}` and its relatives. That is a
function-naming gap, it caps the existing hunter on that slice exactly as
much, and it is the same class as the `_declarator_start` defects found in
learning-harness task 4.1.
The ruleset arm of `ceiling` is kept so the 12.1% that rejected it stays a
measurement rather than a remembered number.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Design approved in session. Every one of the 36 criteria is claimed by a task.
Group 1 calls no model at all. The candidate enumerator, the ceiling report
and the stop-or-go reading are offline, so the condition that ends this
feature -- a deciding slice whose ceiling sits below its recorded detection
rate -- is reachable before a cent is spent. The ruleset variant of this same
design would have failed that condition at 12.1% against 64.9%.
Group 2 is the only work that touches scoring.py, and its observable is that
nothing changes: the recorded round-zero artifacts are the pin, and if they
cannot be reproduced exactly then the refactor is wrong, not the numbers.
Group 3 builds the detector, and task 3.3 is the test that would have caught
the defect this spec exists for -- two runs must make an identical sequence of
model calls. It lives in the suite and runs repeatedly, not in a one-off
measurement.
Group 4 is the maintainer's emphasis as code: deciding slices, development
slices, the cross-slice rejection by name, and the overfitting gap printed
beside every headline number.
Group 5 compares the two detectors per slice, re-measures the trajectory count
and the noise floor, and names -- without amending -- the learning-harness
requirements that a collapsed floor would retire.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Requirements approved in session. All 36 criteria trace to a component.
The shape: a deterministic pass enumerates candidate sites from the semantic
IR, shaped by family -- call sites for injection and path, operations with
their guards for access control, bindings and literals for configuration. The
model answers one bounded typed question per candidate with no tools and no
choice of what to look at. The predicates that decide whether a finding counts
are applied while it is being made, with one retry carrying the violation, and
they come from a single definition that the scorer also uses, so the two
cannot drift.
Determinism is a design property, not a hope: candidates are ordered by
(path, line, name) and batched at a fixed size, so two runs make an identical
sequence of model calls. That is the direct answer to 120 runs producing 120
distinct trajectories.
Phase 1 is the candidate ceiling, offline, no model call, measured on the
real-project slices as well as the teaching one. If the ceiling on a deciding
slice comes in below that slice's current detection rate, the design stops
there having spent nothing. Building that in as the first phase is deliberate:
the ruleset variant of this design would have capped recall at 12.1% against a
64.9% baseline, and only a measurement caught it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Renamed from detector-optimizer: the brief's own conclusion is that the
optimizer is not the next thing to build, and a spec should be named for what
it delivers.
Seven requirements, 37 criteria. The load-bearing ones:
Req 1 -- candidates come from the semantic IR, never the pattern ruleset,
and the enumerator's ceiling is published as the limit on recall. Measured
before building: ruleset 12.1%, IR 96.6%.
Req 2 -- one bounded typed question per candidate, no tools, no choice of
what to look at, and a test asserts two runs make an identical sequence of
model calls.
Req 3 is the maintainer's emphasis and it decides acceptance. The slices
harvested from real projects decide; hand-written vibe-py is a development
slice that cannot on its own justify a change; a change that improves vibe-py
and regresses a deciding slice is rejected by name; and the overfitting gap
-- 83.3% against 44.4% at baseline -- is published beside every headline
number. The candidate ceiling is measured separately on real-world slices,
because their functions are larger and the enumerator's behaviour there is
not implied by its behaviour on teaching code.
Req 5 -- the K-run and noise-floor machinery is re-measured against the
constrained detector and deleted if the floor collapses, naming the
learning-harness requirements that would be amended.
Not approved; design not started.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
learning-harness divides cleanly. The measurement layer stands and is worth
more than before -- it is what made a $0.23 experiment able to answer 'is
our own instability structural?'. What was answering the wrong question:
Req 3.5's K (an ensemble of distinct search procedures, not a repeated
measurement), Req 8.2's noise floor (it measures how often an unconstrained
search reaches the sink, so the regression budget grows with the search's own
wandering), Req 9.5's acceptance rule, and the sign test added this session.
And the highest-leverage change in either spec turns out not to be a prompt,
a lever or an optimizer: stop asking the model to locate the sink. 555 grep
calls against 71 entry_points, while the CST and the entry-point mapper
already know where the sinks are.
Limits stated in the brief: twelve pairs, one family, one model; the
high-similarity bucket is two run pairs and its 0% is decoration.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Twelve injection pairs, both sides, K=5, temperature 0, thinking disabled,
recording every tool call. $0.23.
The statistic I designed for could not be computed: across 120 runs there was
not one repeated trajectory -- 4.67 and 4.92 distinct trajectories per side
out of 5, and 78 distinct tool-name sequences ignoring arguments entirely.
The detector is not sampling one measurement five times; it is running five
different investigations and voting on the results.
As a gradient the relationship is monotone: runs whose trajectories were
similar (>=0.75 Jaccard) disagreed 0% of the time, mid-similarity 4.2%,
low-similarity 30.9%.
Three more facts. The search is grep: 555 grep_repo calls against 181
read_file, 71 entry_points, 43 find_refs, 13 read_definition -- the model is
guessing patterns to locate sinks the CST and the entry-point mapper already
know. All the instability is on the vulnerable side; every pair produced
exactly one finding set on the fixed side, so the noise is in detection and
none of it in false positives. And findings converge much more than
trajectories do: 78 trajectories, one to three finding sets per pair.
The floor is therefore not measuring uncertainty about a question. It is
measuring how often an unconstrained search happens to reach the sink.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The harness treats the detector as a black box emitting a noisy scalar, and
K runs, majority voting, the measured floor, the regression budget and the
sign test all follow from that. It is the stance you take when you cannot
see inside what you are measuring, and we can.
Temperature is already 0 and thinking is disabled, and 11 of 20 injection
pairs still disagree with themselves. The freedom is in the trajectory --
which files get read, which greps get run -- and we built statistics to
average over it instead of removing it.
Three DSPy constructs attack the source: constraints enforced as predicates
at generation time (we already compute every one of them, after the fact, at
scoring time), typed decomposition of the single ReAct loop into steps whose
deterministic parts we already answer without a model, and best-of-N against
a checkable reward instead of a majority vote that discards the
disagreement. The cheaper the detector model, the more of the return sits
there and the less in prompt evolution -- the opposite of the order this
brief first proposed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
SIMBA ranks instances by max-to-min score gap across variants, which turns
our noise floor from an obstacle into an instance sampler: the 11 injection
pairs that disagree with themselves are the pairs where a rule change can
flip an outcome. Its two alternating strategies -- append a demo, append a
rule -- are the split our levers already have.
GEPA's reflection meta-prompt is a specification for the feedback function,
not just a prompt: it asks the reflector to extract domain facts the model
cannot re-derive later, which means feedback must carry which sink, which
span and which side leaked, not a verdict.
And the finding that outranks both: GEPA splits a trainset, SIMBA batches 32.
Injection has 12 train teachers. These algorithms are sample-efficient in
rollouts, not in instances, and they will not shortcut a corpus of twelve.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The learning harness built the measurement honestly and then hill-climbed on
it by hand. Four review rounds' worth of defects in that loop were defects of
its shape, not its code.
Read from GEPA's Algorithm 1 and 2 rather than from a summary, three things
bear directly on ours: the acceptance gate is deliberately cheap and noisy,
because the Pareto frontier -- not the gate -- is what protects against a bad
accept; selection is stochastic and diversity-preserving, which is what stops
the search collapsing onto one lineage; and the expensive full evaluation is
spent only on candidates that already passed the cheap gate.
Two consequences for us are honesty findings, not performance ones. Our
acceptance rule reads the holdout every round, so repeated rounds overfit the
test set -- the same leak class this project already measured once, one level
up. And the proposer is handed a score where the published work hands it
textual feedback from the evaluation trace, which our evaluator produces and
then discards.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Round zero and one evolve round run end to end with the scripted client over
the whole vibe-py slice: per-family and per-slice blocks, twins out by
reason, a floor each, a proposal applied and reverted byte for byte, the
sweep reporting zero negative flips, the rejected version archived. The only
thing the live run adds is a detector that can see.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Measured on the corpus under the split and review-tier rules, access_control
has 1 train teacher and 4 holdout pairs: every agent-vfc row sits at tier
`title` and the vibe-py absence rows that could teach are trap pairs whose
two sides are different handlers, so the train split teaches zero obligation
shapes. injection has 12 teachers and 10 holdout pairs, the only family with
enough train evidence to tell a real change from noise.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Three defects, two of them the same shape as ones already rejected once:
- The meta-agent proposer still could not propose. `MetaAgent(inner_model=)`
takes a ModelConfig; it was handed a model name, a `# type: ignore` hid
that from mypy, and a bare `except Exception` turned the resulting
AttributeError into `meta_agent_failed` -- which reads exactly like the
model declining. The composition is now a seam a test can pin, and a
construction error is re-raised rather than reported as a quiet round.
- `ousast pairs --hunter` crashed in its documented form: the K floor landed
with `--k-runs` still defaulting to 1. The covering test always passed
`--k-runs 4`, which is why nothing noticed.
- 28 rows (12% of the corpus) resolved a span on the vulnerable side and
none on the fixed side, so every fixed-side finding read as outside the
labeled function and the pair scored as silence whatever the detector
said. `fix_function` now reaches the catalog and the scorer, and the
generator derives it from the fixed excerpt for the 25 trap rows that
never declared one. Measured after: 0 rows.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The three gate outputs are captured at 71efc9c -- the commit before the
first line of this feature -- and a test reproduces them byte for byte in
both matrices. Byte-identity was something a reviewer checked by hand
against a worktree; it is now mechanical and runs in CI.
Six absences are each exercised once with the reason they record: HarnessX,
the chat endpoint (the run still writes its artifact), the family
configurations, the network for pointer pairs, the semantic extra (named,
not guessed), and the sandbox -- so a claim nothing could run is not
confused with a claim that failed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The catalog generator fills `fix_date` from the recipe at the precision the
recipe has -- a full ISO day when one was recorded, otherwise the year.
Every harvested recipe carries a year and almost none carries a day, and
writing `-01-01` would turn a year into a claim about a month. A row counts
as post-cutoff only when its earliest possible date is after the cutoff, so
a coarse date can under-claim the clean slice but never over-claim it.
Each family now reports the same numbers over the rows fixed after the
cutoff, beside the whole number and never instead of it, with the undated
count on its own line. Without a configured cutoff every scoreboard says so
rather than inventing one: guessing a model's training cutoff and then
publishing a contamination number from the guess would be worse than
reporting none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The five byte-identical vibe-py twins now say so in the catalog instead of
only hashing to it, and a declared limit that names one of the unscorable
vocabulary's own reasons keeps that spelling, so `identical_twin` stays one
fact. Leave-one-out over vibe-py consequently runs on 30 rows, not 35.
The difficulty check measures three axes the corpus varies on: binding hops
from the identity source to the constrained use, registration distance, and
guard distance. Measured first, and it failed: the shared obligations
fixture reached (1, 0, 2) where the vibe-py absence rows reach (5, 2, 2).
Every fixture handler carried its own route decorator, so nothing in the
suite exercised a handler registered from another document and no binding
chain was longer than one hop -- the checker was passing on a simpler world
than the one it meets. `from_spec` closes that with the connexion shape
those rows really have. The verdict is committed beside the numbers.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Eight of nine group-3 tasks were rejected. Each defect was hiding a number,
and the numbers are why the loop was not learning:
- The fixed side was judged against the *vulnerable* side's function
ranges. 191 of 227 pairs with a resolvable labeled span (84%) have
different spans on the two sides, so a leak was recorded as silence,
always in the direction that flatters Youden.
- Round zero measured every family with the generalist. `family:unknown`
earns no credit against any real family, so every family but `unknown`
baselined at a structural zero and every noise floor with it.
- The cross-family sweep counted absolute failures instead of negative
flips. On a corpus where most pairs fail at round zero that rejects
every round for damage it did not do. Unscorable rows voted too.
- K runs was declared and never threaded; `pairs --k-runs` was parsed and
discarded, and its only test grepped `--help`.
- The meta-agent proposer was a stub, so `learning round` could never
propose anything. It now composes MetaAgent.evolve against a scratch
copy of the family directory, and `--proposals FILE` runs a recorded
sequence offline.
- `refused_holdout` was a vocabulary word no code path could produce, and
the covering test accepted either outcome.
- Cost was metered once, after everything had been paid for; a stage that
raised left no record at all.
- The corrected lever number was published beside the earlier one rather
than in place of it, and the test named for that property never checked.
- Refusals were `continue` statements: `Refusal.degradation()` existed and
nothing called it. A train-split identical twin was reported as
`split:train`, which is untrue.
Test repairs that each changed a number: two split tests measured an
`x = 1` fixture that teaches nothing whatever the rule says; two CLI
scoreboard tests asserted against all-zero output produced by a stub that
answers twice and puts its finding outside the labeled function.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Eight of the ten vendored access-control rows had no named handler at all,
so every one of them would have scored `missing_context` forever rather
than as a detector miss. After the re-harvest nine are named and the tenth
states its reason.
The handler-context mode now carries the class its framework registers,
multi-line decorators, and commented-out guards -- `#@token_required`
above a ThreatByte handler is the entire bug. Where the registration lives
in another document (connexion's `operationId`, an express router), a
recipe names `context = [...]`, the harvester writes that stanza per side,
and the pair materializes both files at their repository-relative paths so
the entry-point mapper sees the layout the project has. `harvest_recipe`
fetches every blob before the first write and the driver takes `--report`,
so a failed fetch leaves a reviewed excerpt exactly as it was.
Two defects the re-harvest exposed: `.mjs` was not a known suffix, so the
one pair vendored as an ES module had no file target at all; and
`_python_route_access` matched auth words inside decorator prose, reading
`@ns.doc(description='... without proper authorization checks')` as a
guard. Recorded in the hygiene artifact: nine of the ten duplicate groups
sit on both sides of the split, which the repository-level split cannot
see.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The maintainer settled the conflict the group 2 review escalated: a detector
configuration has no skills set. The skill router belongs to the tool's earlier
shape, and a family's prompt, checklist, tools and counterexamples already carry
what a round evolves.
The README described the project as an editor-agent harness whose skills steer
it, and pointed users at the maintainer's spec directory. The tool is a CLI now
and the agent integrations are optional, so the opening says that, the skills
section describes what it actually is, and the spec pointer is gone from the
user-facing text.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The maintainer settled both items the group 1 review escalated.
The family that covers server-side request forgery is named untrusted
destination and also covers open redirect, because both are untrusted input
choosing where a request goes and the corpus carries 34 open-redirect rows that
would otherwise be unclassifiable. The requirement, the brief and the design
example now say so, and the approval note records the amendment.
The shipped taxonomy is flat, so no answer can earn hierarchical partial credit.
Rather than leave that as prose, two new acceptance criteria require every
agreement figure and every family figure to say it, and both report types now
carry the flag, computed from the taxonomy rather than asserted. A taxonomy that
later declares a parent flips it without any other change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.4. Per family, and per slice when asked: outcome counts, recall, silence,
Youden, a directional-bias index and recall at a fixed false-positive ceiling.
Rows that cannot be scored are listed by reason and leave every denominator, and
a family with nothing scorable reports no rate rather than a perfect one.
Two numbers exist because Youden cannot tell apart a detector that flags
everything from one that says everything is safe: the directional-bias index
separates them, and recall at a fixed ceiling is forfeited entirely once a
family leaks more than the budget allows.
Change against a baseline is not a difference of means. Flips are counted per
pair in both directions and a two-sided exact sign test decides whether the
change is bigger than the noise, so one flip in ten reads as noise and six reads
as a real regression.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Foundation: taxonomy, DeepSeek endpoint, corpus fields, TypeScript grammar,
curated hunter tools. Core: classifier tiers and measurement, per-pair family
scoring and aggregation, detector configurations and runner, verifier registry
with oracle callback, canary templates and second judge, journal and archive.
Integration: one teacher rule, family scoring on the pairs hunter path, round
zero with per-model noise floors, scan-path tags and manifest block, proposers,
acceptance and revert, round orchestration, publish, subcommands. Corpus
repairs and validation last. Both sanity reviews' findings recorded in the
plan; K-run evaluation honors requirement 3.5 at every stage.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
A learning package layered families -> classify -> scoring -> split ->
detectors and verifiers -> rounds -> publish, composing the existing seams:
the tool hunter gains prompt, context and tool parameters; the hunter path of
the pair scorer takes an injected family matcher so pairs never imports
learning; every learning path selects teachers through one train-only rule;
the regress verdict takes an optional canary oracle; the chat endpoint resolves
DeepSeek before OpenRouter with thinking disabled and reasoning replayed, while
embeddings keep their own endpoint; HarnessX proposes behind the seam with a
scripted proposer for offline runs. Fifty-nine acceptance criteria traced;
boundary sections, file structure plan and migration phases populated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Scope: catch the ten taxonomy families reliably on real-world and generated
application code, including forgotten authorization checks; not browser
engines, interpreters or memory-safety fuzzing. Decisions: verifier-aligned
families, a measured per-family noise floor as the regression budget, a cheap
model on an independent chat endpoint with embeddings elsewhere, canary-oracle
verifiers for the four most dangerous families first, the obligations lever
folded into the access-control family's first round, web families before the
C slice. Eleven requirements in EARS form; awaiting approval.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
AIxCC cyber reasoning systems, self-improving harness methods (GEPA, DGM,
Self-Harness, SkillOpt, AHE), vulnerability-dataset pitfalls and CWE taxonomies,
and routing/regression-control literature. The design changes they force:
coarse verifier-aligned families instead of fine CWE, multi-label routing with
an unknown class, pair-wise outcomes with a directional-bias index, K-run
reliable-change tests, a per-class seed-noise floor as the regression budget,
a no-regression acceptance rule with ties rejected, structural edits over prose
edits, a proposer blind to the holdout, and verifier code and gold labels
outside every editable surface.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Measured 2026-09-06 on the vibe-py holdout: the existing tool hunter, with no
learning, scores 12/14 pair-correct (Youden +0.857, no leaks) once detection
and silence are keyed on the labeled class and function and the byte-identical
twins are excluded; the overlay scores 5/14 on the same rows. The published
hunter number (0/17, Youden -0.588) was a scoring artifact: the scorer wanted
the literal CWE id in the finding text and the prompt never asked for it.
The brief proposes an auto-classifier as the routing key, one evolved hunter
configuration per class, a class-aware scorer with unscorable denominators,
the split enforced by construction, and the HarnessX evolve loop with per-class
attribution as the round primitive. Raw measurements are committed under
reports/measurements.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Six task groups in the design's migration order: closed facts and
obligation shapes; operations, siblings, declared policy, dominance
protocol and the checker; MAP wiring, reports, ranking and hunter intent;
corpus harvest mode, labels, leave-one-out and relabeling; the
obligations lever; validation. All 44 acceptance criteria covered; no
task waits on the unimplemented upstream specs (path-aware evaluation
enters through the Dominance protocol and an optional PathRecord input).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Requirements approved by the maintainer in session. Design: closed
obligation facts (operations and dischargers), obligation shapes as a
second family in the mechanism store, sibling-set consistency and an
optional declared policy as the policy sources, a Dominance protocol
with an order-based default so the checker works function-local before
reachability and guard dominance land, findings at suspicion carrying
consistency_violation / declared_policy_violation / function_local as
data, reports, manifest and ranking fields, handler_context harvest
mode, leave-one-out per obligation kind, and an obligations lever with
pending policy clauses under the existing gate. All 44 acceptance
criteria traced.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Absence bugs as obligations rather than flows: obligated operations and
discharger kinds as closed facts, obligation shapes learned from trusted
pairs, the application's policy recovered from sibling-handler
consistency and an optional declared policy file, path-aware evaluation
with a function-local degraded mode, honest rungs below proof, an
`obligations` improve lever under the holdout gate. Depends on
reachability-flow-model and guard-dominance-regime; the roadmap marks
reachability as the critical path for two consumers.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
- tasks.md: all tasks checked; rounds 1-6 recorded with RED evidence,
measured leave-one-out and admission numbers, and the recorded design
deviations.
- spec.json: completed 2026-09-05.
- roadmap: corpus-seeded-mechanisms marked completed; the loop is closed
end to end (export -> improve admits under the holdout clause -> scan
searches -> reports name the mechanism).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Review rounds 3-5 remediations (Req 5, 7).
- ousast improve resolves candidates from the target's calibration dir,
then the working directory's (where `mechanisms export` writes), and
admits into <target>/.openultrasast/calibration/mechanisms.jsonl,
where a scan of that target reads; --mechanism-candidates,
--mechanism-store, --no-mechanisms; --dry-run redirects the store to
scratch; accepted rounds print admit/retract <id>.
- run_improvement threads the store paths; RoundOutcome carries the
mechanism edits; reverted mechanism keys are blocked outright; a
no-gain admission is rejected as mechanism_no_gain; a retraction may
name an admitted record that left the candidate set.
- leave-one-out forwards variant-search degradations and pins the
containment and label rules; mechanisms export defaults to the
candidates file so it never re-admits a tombstone.
- design §Lever rewritten to the implemented gate; README and docs show
export -> improve -> scan.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 4.2 (Req 5.3, 5.4, 5.5).
- propose_mechanism_edits: retract admitted records that leak on a
holdout fixed side; admit a candidate only when a trial store holding
it alone recovers a currently missed holdout pair without leaking.
- run_round(mechanism_candidates=, mechanism_store=): validates, applies
after the rule gate passed, re-scores the holdout pairs per profile with
variant search (evaluate_mechanism_profiles), and rejects with a
byte-for-byte revert on mechanism_leak, profile_regression or
mechanism_no_gain; the journal records each mechanism edit with its
pair provenance and the metrics before and after.
- The rule gate and its per-profile clause are unchanged.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
