The meta-agent round journalled `no_proposal`. The real cause was an
authentication failure: `_model_config` hard-coded the anthropic provider
while the detector was talking to DeepSeek and no ANTHROPIC_API_KEY exists
here. The composition itself was fine -- MetaAgent constructed, evolve
started, budget $5 -- and the trace was thrown away.
`no_proposal` covered two different worlds: a proposer that considered the
facts and declined, and one that could not start at all. The journal now
carries the proposer's own reason, and the vendor comes from
`[harnessx] provider` instead of being assumed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
`publish` keyed baselines by model alone, so the vibe-py run silently
replaced the agent-vfc run and the table showed hand-written teaching code as
if it were the corpus. That is precisely the failure the constrained-detector
requirements exist to prevent, live in a published artifact.
Third time today that a key was fixed in one place and not another --
`compare_baselines` got the model+slice label and `_baselines` did not, the
same way the exception path got checkpointing and the timeout path did not.
The table now carries a slice column and one row per slice, and the artifact
is keyed by what identifies a run. Republished: agent-vfc's six families are
back, including access_control at 0.000 recall on four pairs, which is the
number that was missing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The 176-pair vfc baseline hit its four-hour timeout and wrote nothing. Scores
were accumulated in memory and persisted once at the end, so roughly four
hours of paid API calls were lost. I fixed the exception path this morning --
a provider hiccup costs one pair -- and missed that a timeout, a kill or a
machine going away cost all of them.
Each pair is now appended to `scores.jsonl` and flushed as it is measured, and
`--resume` keeps what a previous run of that exact baseline already bought.
This also gives the per-pair outcomes the round-3 review asked for, which make
the noise floor auditable after the fact instead of only in aggregate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Group 1 complete. Tasks 1.2 and 1.3: `candidates.ceiling`, `ousast learning
candidates`, and the stop-or-go reading. No model is called anywhere in this
group, so the condition that could have ended the feature was reachable for
free.
Verdict: go.
agent-vfc (deciding) 89.7% ceiling against 44.4% round-zero detection
vfc (deciding) 84.7%, baseline still running, non-gating
vfc-js (deciding) 52.9% -- see below
github (deciding) 100.0%
vibe-py (development)100.0% against 83.3%
The margin is largest on agent-vfc, which is the slice the tool is actually
for.
The measurement forced a shape correction before the detector existed.
config_secrets first read 63.6% with a bind-and-literal shape, because a
permissive default is as often an argument to a call -- cors({origin: '*'}),
app.use(session({secure: false})) -- as it is a binding. Adding call sites to
that family's shape took it to 100% and recovered four agent-vfc rows.
Carry-forward, and not this feature's to fix: vfc-js sits at 52.9% because
eight of seventeen rows label a function the JavaScript CST names `<anon>` --
`module.exports.publish = function () {}` and its relatives. That is a
function-naming gap, it caps the existing hunter on that slice exactly as
much, and it is the same class as the `_declarator_start` defects found in
learning-harness task 4.1.
The ruleset arm of `ceiling` is kept so the 12.1% that rejected it stays a
measurement rather than a remembered number.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.1. The detector stops searching: something deterministic decides what
it looks at, ordered by (path, line, name) and bounded by a constant, so the
batches a later task forms are determined by the input alone.
A call candidate carries what the model used to grep for -- the arguments,
and the binding chain the IR resolved for them, nearest first. An access
control candidate is the operation with the guards in scope, because an
absence bug has no sink.
Two things the fixture taught while writing it. The IR emits a chained call
twice at one line, once bare and once carrying the keyword arguments, so
deduplication keeps the richest: a bare `filter_by` and
`filter_by(user_id=owner)` are the same site and only one of them says what
the constraint was. And `leaky` and `constrained` in the shared obligations
fixture both discharge through `filter_by` -- what separates them is that one
chain ends at `request.args.get('owner')` and the other at `request.user.id`.
That contrast is now the test, and it is exactly the judgement the model is
being narrowed down to make.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Design approved in session. Every one of the 36 criteria is claimed by a task.
Group 1 calls no model at all. The candidate enumerator, the ceiling report
and the stop-or-go reading are offline, so the condition that ends this
feature -- a deciding slice whose ceiling sits below its recorded detection
rate -- is reachable before a cent is spent. The ruleset variant of this same
design would have failed that condition at 12.1% against 64.9%.
Group 2 is the only work that touches scoring.py, and its observable is that
nothing changes: the recorded round-zero artifacts are the pin, and if they
cannot be reproduced exactly then the refactor is wrong, not the numbers.
Group 3 builds the detector, and task 3.3 is the test that would have caught
the defect this spec exists for -- two runs must make an identical sequence of
model calls. It lives in the suite and runs repeatedly, not in a one-off
measurement.
Group 4 is the maintainer's emphasis as code: deciding slices, development
slices, the cross-slice rejection by name, and the overfitting gap printed
beside every headline number.
Group 5 compares the two detectors per slice, re-measures the trajectory count
and the noise floor, and names -- without amending -- the learning-harness
requirements that a collapsed floor would retire.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Requirements approved in session. All 36 criteria trace to a component.
The shape: a deterministic pass enumerates candidate sites from the semantic
IR, shaped by family -- call sites for injection and path, operations with
their guards for access control, bindings and literals for configuration. The
model answers one bounded typed question per candidate with no tools and no
choice of what to look at. The predicates that decide whether a finding counts
are applied while it is being made, with one retry carrying the violation, and
they come from a single definition that the scorer also uses, so the two
cannot drift.
Determinism is a design property, not a hope: candidates are ordered by
(path, line, name) and batched at a fixed size, so two runs make an identical
sequence of model calls. That is the direct answer to 120 runs producing 120
distinct trajectories.
Phase 1 is the candidate ceiling, offline, no model call, measured on the
real-project slices as well as the teaching one. If the ceiling on a deciding
slice comes in below that slice's current detection rate, the design stops
there having spent nothing. Building that in as the first phase is deliberate:
the ruleset variant of this design would have capped recall at 12.1% against a
64.9% baseline, and only a measurement caught it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Renamed from detector-optimizer: the brief's own conclusion is that the
optimizer is not the next thing to build, and a spec should be named for what
it delivers.
Seven requirements, 37 criteria. The load-bearing ones:
Req 1 -- candidates come from the semantic IR, never the pattern ruleset,
and the enumerator's ceiling is published as the limit on recall. Measured
before building: ruleset 12.1%, IR 96.6%.
Req 2 -- one bounded typed question per candidate, no tools, no choice of
what to look at, and a test asserts two runs make an identical sequence of
model calls.
Req 3 is the maintainer's emphasis and it decides acceptance. The slices
harvested from real projects decide; hand-written vibe-py is a development
slice that cannot on its own justify a change; a change that improves vibe-py
and regresses a deciding slice is rejected by name; and the overfitting gap
-- 83.3% against 44.4% at baseline -- is published beside every headline
number. The candidate ceiling is measured separately on real-world slices,
because their functions are larger and the enumerator's behaviour there is
not implied by its behaviour on teaching code.
Req 5 -- the K-run and noise-floor machinery is re-measured against the
constrained detector and deleted if the floor collapses, naming the
learning-harness requirements that would be amended.
Not approved; design not started.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
A candidate-driven detector can only judge sites something else enumerates,
so the enumerator caps recall. Measured over the 58 scorable web pairs before
building anything:
pattern ruleset: 12.1% ceiling -- and the misses are not near-misses, the
ruleset finds nothing at all in most of these files
semantic IR: 96.6% ceiling, a median of ten call sites and bindings
inside the labeled function
The ruleset would have capped us at 12.1% against the 64.9% the unconstrained
loop already reaches. That design was one commit from being built.
Two corpus defects found on the way, both mine from task 4.1. The overleaf
excerpt started mid-import, because `_declarator_start` walks back to the
nearest `}` and Overleaf writes JavaScript without semicolons, so the nearest
one was inside `import { expressify }`. It then ended mid-object-literal,
because registration mentions were appended as bare lines. Neither parsed,
and a pair that cannot be parsed cannot be scored. Registration mentions now
expand to the statement that contains them; the ceiling moved 94.8% -> 96.6%.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
learning-harness divides cleanly. The measurement layer stands and is worth
more than before -- it is what made a $0.23 experiment able to answer 'is
our own instability structural?'. What was answering the wrong question:
Req 3.5's K (an ensemble of distinct search procedures, not a repeated
measurement), Req 8.2's noise floor (it measures how often an unconstrained
search reaches the sink, so the regression budget grows with the search's own
wandering), Req 9.5's acceptance rule, and the sign test added this session.
And the highest-leverage change in either spec turns out not to be a prompt,
a lever or an optimizer: stop asking the model to locate the sink. 555 grep
calls against 71 entry_points, while the CST and the entry-point mapper
already know where the sinks are.
Limits stated in the brief: twelve pairs, one family, one model; the
high-similarity bucket is two run pairs and its 0% is decoration.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Twelve injection pairs, both sides, K=5, temperature 0, thinking disabled,
recording every tool call. $0.23.
The statistic I designed for could not be computed: across 120 runs there was
not one repeated trajectory -- 4.67 and 4.92 distinct trajectories per side
out of 5, and 78 distinct tool-name sequences ignoring arguments entirely.
The detector is not sampling one measurement five times; it is running five
different investigations and voting on the results.
As a gradient the relationship is monotone: runs whose trajectories were
similar (>=0.75 Jaccard) disagreed 0% of the time, mid-similarity 4.2%,
low-similarity 30.9%.
Three more facts. The search is grep: 555 grep_repo calls against 181
read_file, 71 entry_points, 43 find_refs, 13 read_definition -- the model is
guessing patterns to locate sinks the CST and the entry-point mapper already
know. All the instability is on the vulnerable side; every pair produced
exactly one finding set on the fixed side, so the noise is in detection and
none of it in false positives. And findings converge much more than
trajectories do: 78 trajectories, one to three finding sets per pair.
The floor is therefore not measuring uncertainty about a question. It is
measuring how often an unconstrained search happens to reach the sink.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The harness treats the detector as a black box emitting a noisy scalar, and
K runs, majority voting, the measured floor, the regression budget and the
sign test all follow from that. It is the stance you take when you cannot
see inside what you are measuring, and we can.
Temperature is already 0 and thinking is disabled, and 11 of 20 injection
pairs still disagree with themselves. The freedom is in the trajectory --
which files get read, which greps get run -- and we built statistics to
average over it instead of removing it.
Three DSPy constructs attack the source: constraints enforced as predicates
at generation time (we already compute every one of them, after the fact, at
scoring time), typed decomposition of the single ReAct loop into steps whose
deterministic parts we already answer without a model, and best-of-N against
a checkable reward instead of a majority vote that discards the
disagreement. The cheaper the detector model, the more of the return sits
there and the less in prompt evolution -- the opposite of the order this
brief first proposed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
SIMBA ranks instances by max-to-min score gap across variants, which turns
our noise floor from an obstacle into an instance sampler: the 11 injection
pairs that disagree with themselves are the pairs where a rule change can
flip an outcome. Its two alternating strategies -- append a demo, append a
rule -- are the split our levers already have.
GEPA's reflection meta-prompt is a specification for the feedback function,
not just a prompt: it asks the reflector to extract domain facts the model
cannot re-derive later, which means feedback must carry which sink, which
span and which side leaked, not a verdict.
And the finding that outranks both: GEPA splits a trainset, SIMBA batches 32.
Injection has 12 train teachers. These algorithms are sample-efficient in
rollouts, not in instances, and they will not shortcut a corpus of twelve.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The learning harness built the measurement honestly and then hill-climbed on
it by hand. Four review rounds' worth of defects in that loop were defects of
its shape, not its code.
Read from GEPA's Algorithm 1 and 2 rather than from a summary, three things
bear directly on ours: the acceptance gate is deliberately cheap and noisy,
because the Pareto frontier -- not the gate -- is what protects against a bad
accept; selection is stochastic and diversity-preserving, which is what stops
the search collapsing onto one lineage; and the expensive full evaluation is
spent only on candidates that already passed the cheap gate.
Two consequences for us are honesty findings, not performance ones. Our
acceptance rule reads the holdout every round, so repeated rounds overfit the
test set -- the same leak class this project already measured once, one level
up. And the proposer is handed a score where the published work hands it
textual feedback from the evaluation trace, which our evaluator produces and
then discards.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
records whether its decision beat noise
Two things the maintainer was right about.
Thinking was disabled so `temperature: 0` would not be silently ignored --
a trade that only pays if it buys determinism. Round zero says it did not:
11 of 20 injection pairs disagreed with themselves across five runs, so the
variance is in the agentic path, not the decoder. We paid the reasoning cost
on a reasoning model and got nothing. It is now `[learning] thinking`,
recorded in the artifact and part of its key so the two modes cannot be read
as one number. The default stays off only because that is what round zero
was measured under.
And the deeper point: the spec built `sign_test` and `reliable_change` and
the acceptance rule calls neither -- `decide` compares raw deltas. On a
detector with that floor a raw delta is mostly noise, and a rule that
brittle is exactly what makes a deterministic decoder look necessary. It is
not; the alternative is an acceptance rule that models the score as noisy,
the way reflective-evolution and Bayesian prompt optimizers do. Changing
Req 9.5 is a maintainer's call, so this does not change the decision: every
round now journals the two-sided sign test over the pairs that moved, and
after four rounds the journal will say whether any decision could be told
from a coin flip.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
OpenAI-shaped providers put `logprobs` beside the message rather than
inside it, so the confidence signal the classifier is meant to abstain on
was always None -- a knob that reads 'never unsure' whatever the model said.
Measured against DeepSeek while checking: it returns the structure but not
the values. Every chosen token comes back at logprob 0.0 and every
alternative at -9999.0, on a deliberately uncertain prompt as much as a
certain one. The field is there; the information is not.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Keyed by model alone, the memory-safety run over the vfc slice silently
replaced the run over the web families, and the published table would have
shown one of them as if it were everything. The artifact key and the
comparison label now carry the slice selection when it is not the default.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Round zero is the most expensive thing this harness does. An error two
thirds of the way through raised straight out of `run_baseline` and took
everything already measured and paid for with it, so the only honest
response to a flaky endpoint was to run the whole thing again and hope. The
pair is now recorded as unscorable with the reason `detector_unreachable`
and the run finishes.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Measured live against DeepSeek on one pair: the model spends its step budget
on tools and then answers in a paragraph that names the sink, the argument
and the line -- which parses as nothing. A detector that found the bug and
reported zero is indistinguishable from one that found nothing, and every
family in round zero would have baselined at zero for this reason alone.
The loop now takes one more turn when the hunt ends in prose or runs out of
steps, asking the provider for JSON and handing back the reasoning the model
already produced. It formats an answer that was reached; it never asks for a
new one, and a hunt that called no tool never gets there.
Same pair, same model, before and after: recall 0.00 -> 1.00, Youden 0.00 ->
1.00 over three runs per side.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The chat client meters itself and nothing passed the meter to the baseline
or the round, so every published cost would have been 0.00 -- and the
round's cost cap, which reads the same meter, could never have fired.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Round zero and one evolve round run end to end with the scripted client over
the whole vibe-py slice: per-family and per-slice blocks, twins out by
reason, a floor each, a proposal applied and reverted byte for byte, the
sweep reporting zero negative flips, the rejected version archived. The only
thing the live run adds is a detector that can see.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
`--slice vfc` selected the vfc rows and `run_baseline` filtered them all out
again against its own default of the two web slices, so the memory-safety
baseline Req 8.4 asks for produced an empty report and said nothing about it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Measured on the corpus under the split and review-tier rules, access_control
has 1 train teacher and 4 holdout pairs: every agent-vfc row sits at tier
`title` and the vibe-py absence rows that could teach are trap pairs whose
two sides are different handlers, so the train split teaches zero obligation
shapes. injection has 12 teachers and 10 holdout pairs, the only family with
enough train evidence to tell a real change from noise.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Three defects, two of them the same shape as ones already rejected once:
- The meta-agent proposer still could not propose. `MetaAgent(inner_model=)`
takes a ModelConfig; it was handed a model name, a `# type: ignore` hid
that from mypy, and a bare `except Exception` turned the resulting
AttributeError into `meta_agent_failed` -- which reads exactly like the
model declining. The composition is now a seam a test can pin, and a
construction error is re-raised rather than reported as a quiet round.
- `ousast pairs --hunter` crashed in its documented form: the K floor landed
with `--k-runs` still defaulting to 1. The covering test always passed
`--k-runs 4`, which is why nothing noticed.
- 28 rows (12% of the corpus) resolved a span on the vulnerable side and
none on the fixed side, so every fixed-side finding read as outside the
labeled function and the pair scored as silence whatever the detector
said. `fix_function` now reaches the catalog and the scorer, and the
generator derives it from the fixed excerpt for the 25 trap rows that
never declared one. Measured after: 0 rows.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The three gate outputs are captured at 71efc9c -- the commit before the
first line of this feature -- and a test reproduces them byte for byte in
both matrices. Byte-identity was something a reviewer checked by hand
against a worktree; it is now mechanical and runs in CI.
Six absences are each exercised once with the reason they record: HarnessX,
the chat endpoint (the run still writes its artifact), the family
configurations, the network for pointer pairs, the semantic extra (named,
not guessed), and the sandbox -- so a claim nothing could run is not
confused with a claim that failed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The catalog generator fills `fix_date` from the recipe at the precision the
recipe has -- a full ISO day when one was recorded, otherwise the year.
Every harvested recipe carries a year and almost none carries a day, and
writing `-01-01` would turn a year into a claim about a month. A row counts
as post-cutoff only when its earliest possible date is after the cutoff, so
a coarse date can under-claim the clean slice but never over-claim it.
Each family now reports the same numbers over the rows fixed after the
cutoff, beside the whole number and never instead of it, with the undated
count on its own line. Without a configured cutoff every scoreboard says so
rather than inventing one: guessing a model's training cutoff and then
publishing a contamination number from the guess would be worse than
reporting none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The five byte-identical vibe-py twins now say so in the catalog instead of
only hashing to it, and a declared limit that names one of the unscorable
vocabulary's own reasons keeps that spelling, so `identical_twin` stays one
fact. Leave-one-out over vibe-py consequently runs on 30 rows, not 35.
The difficulty check measures three axes the corpus varies on: binding hops
from the identity source to the constrained use, registration distance, and
guard distance. Measured first, and it failed: the shared obligations
fixture reached (1, 0, 2) where the vibe-py absence rows reach (5, 2, 2).
Every fixture handler carried its own route decorator, so nothing in the
suite exercised a handler registered from another document and no binding
chain was longer than one hop -- the checker was passing on a simpler world
than the one it meets. `from_spec` closes that with the connexion shape
those rows really have. The verdict is committed beside the numbers.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Eight of nine group-3 tasks were rejected. Each defect was hiding a number,
and the numbers are why the loop was not learning:
- The fixed side was judged against the *vulnerable* side's function
ranges. 191 of 227 pairs with a resolvable labeled span (84%) have
different spans on the two sides, so a leak was recorded as silence,
always in the direction that flatters Youden.
- Round zero measured every family with the generalist. `family:unknown`
earns no credit against any real family, so every family but `unknown`
baselined at a structural zero and every noise floor with it.
- The cross-family sweep counted absolute failures instead of negative
flips. On a corpus where most pairs fail at round zero that rejects
every round for damage it did not do. Unscorable rows voted too.
- K runs was declared and never threaded; `pairs --k-runs` was parsed and
discarded, and its only test grepped `--help`.
- The meta-agent proposer was a stub, so `learning round` could never
propose anything. It now composes MetaAgent.evolve against a scratch
copy of the family directory, and `--proposals FILE` runs a recorded
sequence offline.
- `refused_holdout` was a vocabulary word no code path could produce, and
the covering test accepted either outcome.
- Cost was metered once, after everything had been paid for; a stage that
raised left no record at all.
- The corrected lever number was published beside the earlier one rather
than in place of it, and the test named for that property never checked.
- Refusals were `continue` statements: `Refusal.degradation()` existed and
nothing called it. A train-split identical twin was reported as
`split:train`, which is untrue.
Test repairs that each changed a number: two split tests measured an
`x = 1` fixture that teaches nothing whatever the rule says; two CLI
scoreboard tests asserted against all-zero output produced by a stub that
answers twice and puts its finding outside the labeled function.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Eight of the ten vendored access-control rows had no named handler at all,
so every one of them would have scored `missing_context` forever rather
than as a detector miss. After the re-harvest nine are named and the tenth
states its reason.
The handler-context mode now carries the class its framework registers,
multi-line decorators, and commented-out guards -- `#@token_required`
above a ThreatByte handler is the entire bug. Where the registration lives
in another document (connexion's `operationId`, an express router), a
recipe names `context = [...]`, the harvester writes that stanza per side,
and the pair materializes both files at their repository-relative paths so
the entry-point mapper sees the layout the project has. `harvest_recipe`
fetches every blob before the first write and the driver takes `--report`,
so a failed fetch leaves a reviewed excerpt exactly as it was.
Two defects the re-harvest exposed: `.mjs` was not a known suffix, so the
one pair vendored as an ES module had no file target at all; and
`_python_route_access` matched auth words inside decorator prose, reading
`@ns.doc(description='... without proper authorization checks')` as a
guard. Recorded in the hygiene artifact: nine of the ten duplicate groups
sit on both sides of the split, which the repository-level split cannot
see.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.9. Five entry points make the classified detector set reachable:
classify a corpus and report the classifier against itself, score a catalog by
family, run round zero, run one evolve round, and publish every number from its
artifacts. The pairs command gains a runs-per-pair flag, since the family scorer
needs more than one.
Every step degrades with a reason rather than failing: with no chat endpoint the
run says so and still records what it could measure without a detector, and a
missing configuration directory is written from the current hunter prompt rather
than being an error. Artifacts are keyed by model throughout, so comparing a
second detector is a flag rather than a rewrite.
Both matrices are green and all three gate outputs stay byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.8. A number in a roadmap that nobody can regenerate is a claim, not a
measurement. One entry point now writes them all from the committed artifacts
and puts the denominator beside each: how many pairs could be scored, why the
rest could not, which taxonomy version scored them, the noise the family shows
against itself, and what the run cost per pair it got right. Every table says
whether hierarchical partial credit could apply, which today it cannot.
The roadmap section is replaced in place between markers rather than appended,
so the file cannot accumulate a stale number beside a fresh one, and running the
command twice without new artifacts changes nothing. With no baseline recorded
it says so instead of printing an empty table.
The corrected mechanism-lever number travels with the rest: detected five of
seventeen at a negative Youden once only train-split pairs teach, against seven
when every pair taught, with the difference named as what holdout pairs stopped
teaching. Round zero now records what it spent so the cost column has a source.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.7. A round proposes one change for one family and then lets the evidence
decide whether it stays. The stages run cheapest first: a train minibatch, then
the family's held-out pairs, then every other family. Each stage scores several
runs per pair, and asking for fewer than three is refused outright, because a
single run misses most real changes.
The family directory is snapshotted before the change, so every outcome other
than acceptance leaves the tree exactly as it was found: a tie, a regression on
either side, another family below its floor, a proposal that reaches beyond its
directory, or a round that crosses its cost cap. Acceptance bumps only that
family's version and records which configuration won which pair.
Every outcome is journalled, including having no proposal at all, and the round
directory holds the proposal, the scores, the pair identities behind the
attribution counts and the detector trajectories, all redacted, so a round can
be replayed and its attribution audited rather than taken on trust. A round
number is never reused.
Recorded while writing the sweep: with per-family directories a change cannot
reach another family, so the sweep is a floor check over every other family at
the moment this round would land, not a causal claim about this change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.6. A round is kept only when the family it targets does not go backwards
on either the train or the held-out pairs, at least one of them moves forward,
and no other family flips more pairs than that family showed against itself in
round zero. A tie is a rejection: accepting one would let the loop wander on
noise and call it progress. The rule lives in one place so no caller can soften
it.
A family with no measured floor gets no budget, because unmeasured is not the
same as unlimited and the alternative is a family that never ran in round zero
quietly absorbing every regression. A family marked as never gating cannot
reject a round however much it moves.
Undoing is a directory restored byte for byte: files the round added are
removed, files it deleted are put back, and a digest per family shows that every
directory other than the target is identical before and after, whatever the
outcome. That is what makes the freeze structural rather than promised.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The round-zero test decided which materialized side it was looking at by
searching the whole path for a substring, and the random temporary directory
name sometimes contained it. The detector then "found" the bug on the fixed side
too, the pair flipped, and the model comparison failed about one run in eight.
The check is now the directory name alone. This matters beyond the test: a noise
floor exists to measure how much a detector disagrees with itself, and noise
coming from the harness would be recorded as the detector's own and inflate
every family's regression budget.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.5. A proposer is handed facts, never the scores it is about to be judged
on: the misses and leaks of one family on the train split, what each run
actually did, and the hypotheses this family already tried and lost. A holdout
pair never appears in those facts. Self-diagnosis from free narrative
confabulates; structured facts are what make a proposal answerable.
A proposal is one hypothesis, one lever and one file inside one family's
directory, and the narrowness is the safety property: a round can only touch
what it targets, so every other family is frozen by construction rather than by
promise. A proposal that names two files, an unknown file, a path leaving the
directory, or a prompt over the cap is refused by name, and the write roots
exclude the verifier code and the corpus.
The scripted proposer runs out rather than repeating itself. The HarnessX
proposer reports its write roots and degrades with a recorded reason when the
extra is absent; its evolve call lands with the live rounds.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.4. A standard scan now classifies each hotspot region and runs the
detectors of every admitted family plus the generalist, reading the
configurations round zero wrote. A scan never creates them: a missing directory
is a recorded reason and the stage is skipped, which is also what happens when
no chat endpoint resolves.
Every claim passes through its family's verifier and carries what produced it:
the family, the detector version and what the verifier said. Nothing rises above
suspicion here, because the verifier writes a tag and the evidence ladder
belongs to another boundary. Reports gain a section naming the three, SARIF
gains the matching properties, and the manifest gains a learning block with the
taxonomy version, the per-family counts, the classifier report and the round.
The feature is off unless a project turns it on, so an existing scan is
unchanged, and all three gate outputs stay byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.3. Before anything learns, every family gets the same detector and is
measured against itself. Running a pair several times at the same settings does
not give the same answer, because the chat provider offers no seed, so the
spread between runs is real and unavoidable. That spread is the family's noise
floor and it becomes the budget a later round must stay inside before a
cross-family change counts as a regression. Choosing a budget instead of
measuring one would be a guess dressed as a threshold.
A pair the corpus already marked unscorable is never run: spending model calls
on a row that cannot be scored buys nothing and would put its noise into a floor
where it does not belong.
Artifacts are keyed by detector model, so a second model is baselined by
changing the model and running again, and the comparison table reads both side
by side. A model swap never touches a configuration, which is what makes the two
numbers comparable and reconsidering the model cheap. The web slices are the
default selection and the memory-safety family is measured with its floor marked
as never gating a round.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.2. The model-driven hunter is now scored by family and function. A
detection is a finding of the labeled family inside the labeled function, a
finding of another family or outside it is counted and never called a leak, and
the words in a finding are never read. A pair that cannot be scored is reported
with its reason and the detector is never run on it. The scoreboard gains a
per-family block with its own denominator, taxonomy version and the note that
hierarchical partial credit could not apply.
The text-token matcher this path used is deleted rather than left dormant. It
was the reason the hunter's published number said zero of seventeen while the
model was finding the right bugs in the right functions, and leaving it in the
file would invite its return.
The test that pinned the old rule now pins the new one: the same finding with
the same words counts when it carries a family tag and does not when it carries
none. The overlay and inventory paths and all three gate outputs are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The maintainer settled the conflict the group 2 review escalated: a detector
configuration has no skills set. The skill router belongs to the tool's earlier
shape, and a family's prompt, checklist, tools and counterexamples already carry
what a round evolves.
The README described the project as an editor-agent harness whose skills steer
it, and pointed users at the maintainer's spec directory. The tool is a CLI now
and the agent integrations are optional, so the opening says that, the skills
section describes what it actually is, and the spec pointer is gone from the
user-facing text.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.3: two of the five unscorable reasons had no code path, so an unparsable
row was reported as a corpus labelling defect when the true cause was a tooling
gap. They have different owners and different fixes, and conflating them is the
dishonesty this spec exists to remove. The design's unscorable_reason is
reinstated with the parse result as an input, and a handler that arrived without
its registration is now named as missing context for the families whose bug
shape is a missing check.
Task 2.6: a verification stamped its own word for proof onto the shared evidence
level, which is not a value the project's ladder defines, and three existing
helpers raised on it. Nothing in production called it yet, so the suite stayed
green while the wiring in the next group would have broken a scan. A
verification now writes only its tag, and a caller that wants to move a claim up
the ladder goes through an explicit mapping that can only produce a level the
ladder has.
Task 2.7: the injection canary planted its row in a private in-memory database
the target could never reach, so it could not fire; the fake sandbox handed the
token back regardless of the snippet, which is why nothing noticed. It is now an
execution canary whose marker only an interpreter could write, and each template
is tested by running it for real against a vulnerable fixture and its fixed
twin. That test immediately caught a second flaw: a handler that merely echoes
its input printed the oracle token and satisfied the oracle without any
vulnerability, so an injected payload now carries a value unrelated to the token
the oracle looks for.
Also from the review: a configuration with no tools is refused instead of
producing a detector that silently reports nothing, the classifier abstains when
its fallback call fails too, and the archive docstring says which version wins.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The maintainer settled both items the group 1 review escalated.
The family that covers server-side request forgery is named untrusted
destination and also covers open redirect, because both are untrusted input
choosing where a request goes and the corpus carries 34 open-redirect rows that
would otherwise be unclassifiable. The requirement, the brief and the design
example now say so, and the approval note records the amendment.
The shipped taxonomy is flat, so no answer can earn hierarchical partial credit.
Rather than leave that as prose, two new acceptance criteria require every
agreement figure and every family figure to say it, and both report types now
carry the flag, computed from the taxonomy rather than asserted. A taxonomy that
later declares a parent flips it without any other change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.1. A teacher is a vendored pair, at a review tier a maintainer stands
behind, on the train split, and scorable. Every learning path now asks the same
module that one question, so the split cannot be enforced in one place and
forgotten in another. A refusal is data rather than an exception: the path
records which holdout pairs a candidate touched and moves on, so a leak is
visible in the run instead of silently accepted.
The exporter refuses to seed from a holdout pair, leave-one-out teaches from the
train split while still scoring every pair, and the mechanism lever refuses a
candidate a holdout pair taught, because such a shape can only recover the pair
that produced it.
Measured and committed: on the vibe-py holdout the lever's own number falls from
seven detected and a Youden of +0.059 to five detected and -0.059 once holdout
pairs stop teaching, and the candidate set shrinks from ten shapes to six. That
gap is the inflation this rule removes.
Two existing tests encoded the leak and now encode the truth. The documented
export-to-improve flow had a single holdout pair teaching itself, and now needs
a train pair to teach and a holdout pair to recover. The obligation export
claimed the corpus seeds at least one obligation shape, which was true only
because holdout pairs were teaching: every vibe-py absence pair that yields an
obligation lesson sits on the holdout side, so the honest number today is zero.
The derivation stays covered by fixtures; the corpus gap belongs to the absence
re-harvest and to a maintainer pass over the splits.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.8. The journal is append-only, one line per round, and a round number is
never reused, so the record of what was tried cannot be quietly rewritten by the
thing that is trying. Every round writes a line whether it was accepted or not,
because a rejection is evidence too, and the rejected hypotheses are what stop
the proposer from being handed the same idea twice.
The archive keeps, per family, which configuration version wins which pair,
rather than one champion. Keeping only the best-overall candidate throws away a
version that is the only one solving some pair, which is how a search collapses
onto a single lineage. A later version takes a pair only by winning it.
Nothing is written without passing redaction, because a hypothesis or a
trajectory may quote the code it was reasoning about.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.7. A canary is how a family says the vulnerability fired. Injection
plants a row in an in-memory database that only an unconstrained query returns,
file path plants a file under the scratch mount that only a traversal reaches,
and deserialization plants an object that only an unsafe load reconstructs. Each
prints its token on that path and no other, and the token is derived from the
claim, so one family's oracle can never answer for another and one claim's
canary can never answer for the next. A snippet that cannot reach the function
prints nothing, which is a suspicion rather than a proof.
Every template passes the existing snippet safety check before it can run, and
the candidate is always imported from the read-only source mount.
Nothing reaches proven on a machine's say-so alone. A second, independent
judgement reads the snippet and what the sandbox printed and either confirms or
drops the claim back to corroboration; a judge that cannot answer never promotes
anything. The judge is told the claim and the output, never the answer key.
Templates and oracle code live in the package, outside every directory a round
may write.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.6. A detector's claim is a suspicion until something other than the model
agrees with it, and what counts as agreement depends on the family. The regress
verdict now takes an optional oracle over the sandbox result: when a family
supplies one it decides triggerability instead of the exit code, because a crash
and a demonstrated injection are different evidence and one family's oracle must
not answer for another. Without an oracle the exit code decides exactly as
before, and an inconclusive run stays inconclusive either way.
The registry answers by the verifier kind the taxonomy declares. Access control
gets static corroboration when the obligations checker looks at the same
function and agrees the obligation is undischarged, and refuses to corroborate a
handler that owes nothing. A family whose verifier does not exist yet reports at
suspicion rather than pretending, which is the honest default rather than a gap.
A verification is written onto a copy of the claim as a tag, never onto the
original.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.5. A detector is a configuration, not code: a system prompt, an evidence
checklist, the curated tools it may call, its step and cost budget, and the hard
negatives and counterexamples earlier rounds left it. The loader refuses a
directory that is not a family, a prompt over the length cap, a tool outside the
curated set and a configuration with no version, each by name. An unbounded
prompt is the documented signature of overfitting, so the cap is enforced at
load rather than trusted.
Round zero writes one directory per family from the same prompt, so the baseline
is controlled and every later round rewrites exactly one directory and bumps
that family's version; every other family is frozen by construction because a
round can only touch what it targets.
The runner gives a detector the whole file rather than an excerpt around a
hotspot, since context loss is a documented cause of misjudging patched code,
and tags each finding with exactly one family and the configuration version
that produced it. A region runs every family the classifier admitted plus the
generalist, and never a tool outside the curated set.
Deviation from the file plan: the parts live in one family.toml plus the
markdown and JSON-lines files, because two TOML files with three keys between
them was more surface than value and the version had no home in the plan's list.
Configurations live under the project's learning directory, not inside the
package, because a round writes them.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
A mechanism claimed by two families now fails as loudly as a weakness claimed
twice, with a named offender. Two behaviours that were correct but unasserted
are pinned: the obligations tool refuses a path outside the repository, and a
provider that refuses to build degrades to no endpoint instead of raising into
a scan. The TypeScript parse assertion is now exact rather than a quarter-of-
the-slice allowance, which surfaces one real corpus defect: a TSX excerpt
harvested as a truncated destructuring parameter list is not valid TypeScript
on its own, and belongs to the re-harvest rather than to the grammar.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Review of task 1.3 found the twin detector stripping every leading comment
line, which in C and C++ swallows include and define directives, and in any
language swallows an ordinary leading comment. Two files whose only difference
was a buffer size on a define hashed equal and were marked unscorable, which
removed a real pair from every denominator: the exact dishonesty this spec
exists to end, running backwards. Measured before the fix: twelve file sides
across six vendored pairs had directives inside the stripped prefix, including
the Juliet stack-overflow pair whose whole point is the buffer size.
The strip is now anchored to the grammar the harvester writes: a leading
comment line counts as header only when it carries one of the provenance keys
followed by a colon, and the scan stops at the first line that is code. Block
and hash comment forms both work. Three regression tests pin the boundary in
both directions, and a fourth asserts no vendored excerpt loses a directive.
The five identical twins, the ten duplicate groups and the nineteen straddling
repositories are unchanged, so the committed hygiene artifact still holds.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.4. Per family, and per slice when asked: outcome counts, recall, silence,
Youden, a directional-bias index and recall at a fixed false-positive ceiling.
Rows that cannot be scored are listed by reason and leave every denominator, and
a family with nothing scorable reports no rate rather than a perfect one.
Two numbers exist because Youden cannot tell apart a detector that flags
everything from one that says everything is safe: the directional-bias index
separates them, and recall at a fixed ceiling is forfeited entirely once a
family leaks more than the budget allows.
Change against a baseline is not a difference of means. Flips are counted per
pair in both directions and a two-sided exact sign test decides whether the
change is bigger than the noise, so one flip in ten reads as noise and six reads
as a real regression.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.3. A pair is scored once per family. Detection means a finding whose
family tag relates to the labeled family and whose line sits inside the labeled
function; the words in a finding are never read, because a text token is exactly
the proxy that made the old hunter number meaningless. Silence means no such
finding on the fixed side, and findings of other families or outside the labeled
function are counted separately and never called leaks. A fabricated family id
counts as neither, and is recorded as noise worth naming.
Every evaluation runs a pair several times, because a single run misses most
real changes. The recorded outcome is the majority of the runs and the runs that
disagree with it are the pair's own flakiness, which is what the regression
budget will be built from. A pair that cannot be scored keeps its reason and is
never run, and a labeled function no parsed range names is unscorable rather
than a silent miss.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.2. The report covers the whole corpus and scores the classifier only
where a maintainer label exists. Agreement uses the classifier's first answer,
the oracle number uses the best of its answers, and the random baseline is a
uniform pick over the nine real families, so a router error is visible on its
own rather than hidden inside a detector number. Credit is hierarchical: full
for the same family, half for a parent or child, none for a lateral or
fabricated id.
When it is being measured, the classifier no longer reads the declared label it
is scored against, otherwise agreement would be the classifier reading back its
own answer sheet. With no maintainer labels in the corpus yet the report says
the reference set is empty instead of inventing a score, and every disagreement
is queued once with both answers and the deciding tier for a human to settle.
Measured on the 277 vendored pairs: tier one places 275 of them without a single
model call, only two land in unknown, and the 176 memory-safety rows resolve to
one family instead of reading as an unclassified catch-all.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.1. Tier one decides from what the repository already knows: a declared
family label, an obligation kind, a mechanism id, the rule a finding came from,
or the labeled weakness, and it never spends a model call. Tier two asks one
closed question in JSON mode with thinking disabled, and abstains when the
answer names an id outside the taxonomy or the reply's mean token
log-probability says the model was guessing. Tier three is abstention. Every
answer records which tier decided it.
Answers are multi-label and come back in taxonomy order, so two runs of the
same question compare equal. The classifier proposes and never writes a label,
and it owns validation of the labels the corpus carries, because the corpus
stores a family as an opaque string.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.5. Four tools join the three the hunter already had. Resolving a
definition by symbol returns its file, line span and body, decorators included,
so a detector stops guessing what a callee or a route handler does. Listing
entry points, flows and obligations answers with identifiers and line spans
only, so a structural tool can never become a second way to dump the
repository. Every path a caller supplies is clamped to the scan root first.
The loop now takes the system prompt, the user prompt, the files to start from,
the tool subset, the character limit and the tags its findings carry, each
defaulting to today's behaviour, and it replays the provider's reasoning field
on the next turn because the provider otherwise drops its own chain of thought
across a tool call. A detector configuration is exactly this set of knobs.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.4. The optional extra gains the TypeScript grammar wheel, which ships two
grammars: a file with JSX needs the JSX-aware one and plain TypeScript needs the
other, so the parser picks by file suffix while the parsed file keeps the
language it was given. The walker already treated TypeScript as JavaScript, so
nothing else changes.
Eighteen agent-vfc rows and every TypeScript absence row were counted as
unsupported before this and could not be scored at all. They now parse on both
sides. The core install stays zero-dependency and the extra-free suite still
skips these tests.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.3. An expected row may carry a family label, stored opaquely because the
classifier owns the taxonomy and the corpus module never imports the learning
package. Every pair carries a computed reason when it cannot be scored: a
declared known limit wins, otherwise identical twin when the two excerpts are
the same code once the provenance header is stripped. No row is ever dropped.
Two corpus defects the new accounting surfaces, committed as an artifact beside
the numbers they cap. Nineteen repositories sit on both sides of the split, so a
learning path could be taught by one file of a repository and evaluated on
another. Ten groups of pairs share a vulnerable excerpt byte for byte, mostly
the same fix harvested twice at different granularity.
The split helper keeps every declared split, lets rows that named none inherit
their repository's side, and otherwise assigns a whole repository at once,
oldest first, so nothing straddles by our doing. Pair signals can now be
restricted to one split, which is what lets a learning path teach from train
rows only.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.2. Detectors, the classifier's model tier, the proposer and the judge
resolve one chat endpoint: an explicit override first, then the scripted client
flag, then an endpoint named in the config, then DeepSeek, then OpenRouter.
Prove-order retrieval keeps its own embedding endpoint untouched, because the
chat provider we default to offers none.
The adapter handles the provider's documented quirks: thinking is on by default
there and silently ignores temperature, so it is disabled explicitly; the
reasoning field is kept on the reply for the caller to replay across tool turns;
JSON mode has no schema and occasionally returns empty content, so one retry is
built in, never a loop; token log-probabilities are requested only when the
caller wants them, and their mean is what lets the classifier abstain.
Cost comes from the provider's own usage fields, cache hits priced separately,
and a model with no recorded price reports no cost rather than a made-up one.
The endpoint record deliberately holds no key, so it is safe to journal.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.1. Ten families are the only key that routes work and scores a pair;
CWE ids and mechanism ids are attributes reached through a family. Each family
names what can confirm a claim in it: a sandbox canary for injection, file path
and deserialization, static corroboration for access control, nothing for the
rest. The unknown family carries the mechanisms that span families and the
weaknesses this project deliberately does not target, so a declined row is
visible instead of silently unresolved.
Every labeled corpus weakness resolves, no weakness id is claimed twice, and
every mechanism in the closed vocabulary belongs to exactly one family. Open
redirect joins server-side request forgery under one untrusted-destination
family because both are untrusted input choosing a request destination; that
places 34 real rows instead of discarding them. The relation lookup answers
same, parent-child, lateral or fabricated, and abstention earns no partial
credit: unknown against a real family is lateral.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Foundation: taxonomy, DeepSeek endpoint, corpus fields, TypeScript grammar,
curated hunter tools. Core: classifier tiers and measurement, per-pair family
scoring and aggregation, detector configurations and runner, verifier registry
with oracle callback, canary templates and second judge, journal and archive.
Integration: one teacher rule, family scoring on the pairs hunter path, round
zero with per-model noise floors, scan-path tags and manifest block, proposers,
acceptance and revert, round orchestration, publish, subcommands. Corpus
repairs and validation last. Both sanity reviews' findings recorded in the
plan; K-run evaluation honors requirement 3.5 at every stage.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
A learning package layered families -> classify -> scoring -> split ->
detectors and verifiers -> rounds -> publish, composing the existing seams:
the tool hunter gains prompt, context and tool parameters; the hunter path of
the pair scorer takes an injected family matcher so pairs never imports
learning; every learning path selects teachers through one train-only rule;
the regress verdict takes an optional canary oracle; the chat endpoint resolves
DeepSeek before OpenRouter with thinking disabled and reasoning replayed, while
embeddings keep their own endpoint; HarnessX proposes behind the seam with a
scripted proposer for offline runs. Fifty-nine acceptance criteria traced;
boundary sections, file structure plan and migration phases populated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Scope: catch the ten taxonomy families reliably on real-world and generated
application code, including forgotten authorization checks; not browser
engines, interpreters or memory-safety fuzzing. Decisions: verifier-aligned
families, a measured per-family noise floor as the regression budget, a cheap
model on an independent chat endpoint with embeddings elsewhere, canary-oracle
verifiers for the four most dangerous families first, the obligations lever
folded into the access-control family's first round, web families before the
C slice. Eleven requirements in EARS form; awaiting approval.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
AIxCC cyber reasoning systems, self-improving harness methods (GEPA, DGM,
Self-Harness, SkillOpt, AHE), vulnerability-dataset pitfalls and CWE taxonomies,
and routing/regression-control literature. The design changes they force:
coarse verifier-aligned families instead of fine CWE, multi-label routing with
an unknown class, pair-wise outcomes with a directional-bias index, K-run
reliable-change tests, a per-class seed-noise floor as the regression budget,
a no-regression acceptance rule with ties rejected, structural edits over prose
edits, a proposer blind to the holdout, and verifier code and gold labels
outside every editable surface.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Measured 2026-09-06 on the vibe-py holdout: the existing tool hunter, with no
learning, scores 12/14 pair-correct (Youden +0.857, no leaks) once detection
and silence are keyed on the labeled class and function and the byte-identical
twins are excluded; the overlay scores 5/14 on the same rows. The published
hunter number (0/17, Youden -0.588) was a scoring artifact: the scorer wanted
the literal CWE id in the finding text and the prompt never asked for it.
The brief proposes an auto-classifier as the routing key, one evolved hunter
configuration per class, a class-aware scorer with unscorable denominators,
the split enforced by construction, and the HarnessX evolve loop with per-class
attribution as the round primitive. Raw measurements are committed under
reports/measurements.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.6. The entry-point mapper named a handler only from a decorator on the
function, so projects that register routes elsewhere had no named entry point
and every obligation on them was unreachable. It now also names Python handlers
registered through add_url_rule and add_resource, the HTTP-verb methods of a
Resource, MethodView, APIView or ViewSet subclass, the handler identifier on a
JavaScript router registration with its middleware as access evidence, and the
exported HTTP-verb functions of a Next.js route module.
Access classification, evidence, conditions and the record shape are unchanged,
and the map gate output is byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 4.2. For an obligation-labeled pair, leave-one-out runs the checker on both
sides with the held-out store's obligation shapes. The pair counts as detected
only when a finding inside the labeled function carries both the labeled
obligation and a known fix from a store record, so the checker's own facts-only
finding is never mistaken for the corpus teaching. Recall, silence and Youden
are reported per obligation kind beside the flow mechanisms.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 4.1. A new excerpt mode keeps the labeled handler with its decorators and
every statement outside it that registers the handler by name, so guards living
in decorators, routers and middleware travel with the pair. Comment and string
mentions never count.
The closed mechanism vocabulary gains the three obligation ids, catalog rows
accept an additive obligation field validated against the closed operation
kinds, and the pair scorer counts a finding tagged with that obligation inside
the labeled function as a detection. Rows without the label run the same code
as before.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.3. For every finding that carries a route the hunter answers one closed
question, is this route meant to be public, and the answer is recorded as
public, protected or unknown with its rationale. Nothing else changes: not the
label, not the evidence, not the evidence level, and the hunter never authors a
pattern or a policy. The prompt carries the handler's own lines only and is
redacted before it leaves the process. Without a client one degradation is
recorded and the findings are identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.2. Each obligation finding prints its operation, the missing discharger
with the provenance that failed, the evidence behind the label, the known fix
learned from the corpus and the hunter's intent verdict; the Obligations section
lists the sibling sets evaluated and the declared policy version. SARIF carries
the same as properties.
Obligation findings rank by resource sensitivity times label weight, declared
above consistency above function-local, and stay strictly below every proven
finding. Proof rungs live on verdicts rather than on static findings, so the
cap takes the proven ids from the run's triggerable verdicts.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.1. The obligations proposer runs after variant search in standard and
deep scans, reads the declared policy from the target, and never writes it.
Its findings are appended after every other finding and never enter the sandbox
candidate set. The manifest block carries the sibling sets, the under-populated
count, the operations examined, the findings by label, the policy version, the
stage degradations, and the sibling sets themselves.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.5. An obligated operation that is reached and has no dominating
discharger is reported at suspicion, labeled by why the obligation exists:
a declared policy clause, the practice of sibling handlers, or, in the degraded
no-path mode, the operation fact alone. With path records and neither a clause
nor a sibling norm nothing justifies the obligation and nothing is reported.
A declared public route waives the path guard wherever it is required, not only
where siblings practice it (Req 4.4). A malformed store row is recorded as a
degradation naming the record instead of being skipped silently.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.2. Handlers on the same resource under the same module are siblings;
when at least min_siblings of them discharge an obligation and one does not,
that one is the anomaly and the discharging siblings are its evidence (Req 3.2).
A set with fewer members than the minimum is under-populated and never reported
(Req 3.3). A public route among authenticated siblings is a missing path guard
with the mapper's access classification as evidence (Req 3.4).
The sibling key is the module path without its suffix, so two same-named files
in different directories are never merged. Witness validity (an identity
constraint counts only when it binds the authenticated context) is one helper
shared by operations, siblings and the checker.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.4 (Req 5.1, 5.2). Decorator and router witnesses dominate the
handler; a constraint or a non-permissive value dominates the operation
it sits on; a guard call before the operation dominates by contract;
other statement witnesses need an early exit between them and the
operation; along a path record an earlier hop dominates. Guard dominance
replaces the implementation behind the same protocol later.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.3 (Req 4.1, 4.2). load_declared_policy: closed top-level,
resource and route fields; access public|authenticated|role; identity
source restricted to the facts' authenticated-context sources; comment-
stripped version hash; trailing-* route globs; unknown fields and values
rejected by name; the tool never writes the file.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.1 (Req 3.1, 5.4). find_operations, find_discharges (decorators,
guard and check calls, identity constraints with their provenance,
non-permissive settings) and covers; chained call sites deduped; the
resource token is the receiver head, the fact's resource argument, or a
SQL table name.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.3 (Req 2.1, 2.2, 2.4).
- seed.pair_lessons derives obligation shapes for absence-mechanism rows
and falls back to the sink shape when no obligation is learned, so the
flow store is unchanged; skip reasons name the accepted dischargers.
- append_from_pair accepts both shape families (guard carries the
discharger kind); search_tree ignores obligation rows;
obligation_mechanisms selects them.
- Measured on vibe-py: the two vampi pairs seed protected_read /
identity_constraint / authenticated_context.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.2 (Req 2.1, 2.3, 2.5).
- semantic/obligations/shapes.py: ObligationShape (family obligation,
text-free key), derive_obligation and explain_skip. The lesson is any
discharged operation of the target's walker-visible kind, fixed side
first and then the vulnerable function's own safe branch, since
Real-Vuln twins are flag-toggled identical functions or unrelated trap
functions; keyword constraints are read from the call's source line
(positional-only IR call sites); an identity constraint bound from the
request or a constant discharges nothing.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Six task groups in the design's migration order: closed facts and
obligation shapes; operations, siblings, declared policy, dominance
protocol and the checker; MAP wiring, reports, ranking and hunter intent;
corpus harvest mode, labels, leave-one-out and relabeling; the
obligations lever; validation. All 44 acceptance criteria covered; no
task waits on the unimplemented upstream specs (path-aware evaluation
enters through the Dominance protocol and an optional PathRecord input).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Requirements approved by the maintainer in session. Design: closed
obligation facts (operations and dischargers), obligation shapes as a
second family in the mechanism store, sibling-set consistency and an
optional declared policy as the policy sources, a Dominance protocol
with an order-based default so the checker works function-local before
reachability and guard dominance land, findings at suspicion carrying
consistency_violation / declared_policy_violation / function_local as
data, reports, manifest and ranking fields, handler_context harvest
mode, leave-one-out per obligation kind, and an obligations lever with
pending policy clauses under the existing gate. All 44 acceptance
criteria traced.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Absence bugs as obligations rather than flows: obligated operations and
discharger kinds as closed facts, obligation shapes learned from trusted
pairs, the application's policy recovered from sibling-handler
consistency and an optional declared policy file, path-aware evaluation
with a function-local degraded mode, honest rungs below proof, an
`obligations` improve lever under the holdout gate. Depends on
reachability-flow-model and guard-dominance-regime; the roadmap marks
reachability as the critical path for two consumers.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
- tasks.md: all tasks checked; rounds 1-6 recorded with RED evidence,
measured leave-one-out and admission numbers, and the recorded design
deviations.
- spec.json: completed 2026-09-05.
- roadmap: corpus-seeded-mechanisms marked completed; the loop is closed
end to end (export -> improve admits under the holdout clause -> scan
searches -> reports name the mechanism).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Review rounds 3-5 remediations (Req 5, 7).
- ousast improve resolves candidates from the target's calibration dir,
then the working directory's (where `mechanisms export` writes), and
admits into <target>/.openultrasast/calibration/mechanisms.jsonl,
where a scan of that target reads; --mechanism-candidates,
--mechanism-store, --no-mechanisms; --dry-run redirects the store to
scratch; accepted rounds print admit/retract <id>.
- run_improvement threads the store paths; RoundOutcome carries the
mechanism edits; reverted mechanism keys are blocked outright; a
no-gain admission is rejected as mechanism_no_gain; a retraction may
name an admitted record that left the candidate set.
- leave-one-out forwards variant-search degradations and pins the
containment and label rules; mechanisms export defaults to the
candidates file so it never re-admits a tombstone.
- design §Lever rewritten to the implemented gate; README and docs show
export -> improve -> scan.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 5.1 (Req 6.1, 6.3).
- reports.py: the markdown and SARIF writers take the mechanisms a run
cites; a finding cites one through its mechanism:<id> tag (variant) or
its merged overlay record's mechanism_id; markdown shows Mechanism,
Known fix guard and Learned from pairs per finding plus a Mechanisms
section, SARIF properties carry mechanism_id/summary/guard/pairs;
variant findings keep suspicion, merged inventory findings keep their
own evidence level; reports without mechanisms are unchanged.
- cli: _search_variants returns the cited mechanisms from the store and
the scan passes them to both writers.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 4.2 (Req 5.3, 5.4, 5.5).
- propose_mechanism_edits: retract admitted records that leak on a
holdout fixed side; admit a candidate only when a trial store holding
it alone recovers a currently missed holdout pair without leaking.
- run_round(mechanism_candidates=, mechanism_store=): validates, applies
after the rule gate passed, re-scores the holdout pairs per profile with
variant search (evaluate_mechanism_profiles), and rejects with a
byte-for-byte revert on mechanism_leak, profile_regression or
mechanism_no_gain; the journal records each mechanism edit with its
pair provenance and the metrics before and after.
- The rule gate and its per-profile clause are unchanged.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 4.1 (Req 5.1, 5.2, 5.5).
- validator: MechanismEdit (admit | retract by record id) under the
`mechanisms` lever; validate_mechanism accepts only records from the
exporter candidate set with origin corpus, a closed guard and source
kinds, and identifier-only shape text; free-form shapes are rejected.
- apply_mechanism_edits appends the record (admit) or a retracted
tombstone (retract) so the log stays append-only; StoreSnapshot.restore
reverts byte for byte. MechanismStore.load folds tombstones away.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.1 (Req 4.1-4.4).
- semantic/loo.py: every scorable pair is held out once; a temporary
store is seeded from the other seeded/reviewed pairs (advisory and
title pairs are scored as targets, never teach; known_limit skipped);
both excerpts are searched for variants and scored with the pair rules
(detected inside the labeled function naming the labeled mechanism or
sink; silent when the fixed side yields no hit); found_by and leaked_by
name the records and the pairs that taught them; metrics per slice,
profile and mechanism; score_pair_with_store is shared with the lever.
- cli: `pairs --slice S --loo [--loo-out P]` writes loo.json; never a gate.
- First measurement: vibe-py 35 pairs, 11 detected, 21 silent, Youden
-0.086, 16 teaching pairs.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
- variant_search.search_tree skips a malformed store row with a
variants_store_row_invalid degradation instead of aborting the scan;
a hit on a demoted or flowless overlay record is never merged (test).
- seed.pair_lessons is the one shape derivation shared by the exporter
and leave-one-out; labels pass line and sink together; pairs without a
labeled function are reported as no_labeled_function.
- config parses [variants].max_mechanisms with the shared int helper.
- regress.candidate.hotspot_from_finding is the one zero-score hotspot
helper for forced inventory and variant findings.
- design synced (SearchResult, hits_to_findings summaries, candidate.py).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.3 (Req 3.4, 3.5, 3.6, 6.2).
- [variants] enabled / max_mechanisms config; MAP runs variant search
after the overlay in standard and deep scans only, appends the variant
findings, rewrites overlay.json with merged ids, and the manifest
records mechanisms_searched, files_searched, findings and
merged_into_overlay.
- REGRESS: variant findings join the candidate list after the promotions
as zero-score hotspots, under the existing safety check and cap.
- Spec notes for group 2 and the design synced (variant_search.py,
seed.py, derive_shape signature).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.2 (Req 3.1-3.3, 3.6).
- variants.match_shapes: call sites whose trailing callee name and arity
match a shape and whose source positions all carry a source of any
kind; constants and arity mismatches never match.
- semantic/variant_search.py: search_tree over a tree's parsed files with
the store's corpus shapes (bounded by max_mechanisms, language
degradations recorded); hits_to_findings merges a hit into an overlay
promote/coverage record at the same call site (mechanism_id, no
duplicate) and otherwise emits variant:<id>:<path>:<line> at suspicion
with the pair provenance in the rationale.
- OverlayRecord.mechanism_id (additive).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.1 (Req 1.1, 1.2, 1.5, 2.5).
- semantic/seed.py: export_mechanisms seeds only seeded/reviewed,
non-known-limit, parsing pairs with cached excerpts; every skip carries
a reason (tier, known_limit, parse_failed, language unsupported with a
degradation, pointer not cached, no labeled sink call site); runs
offline over the vendored catalog.
- variants.derive_shape: labels that name only a function and a mechanism
resolve to the fact-sink call inside that function (CWE match and a
non-constant argument first); line and sink intersect when both exist;
container reads are subscript or .get( only.
- append_from_pair keeps the strongest review tier and accumulates
provenance tags when a shape is re-seeded.
- CLI: `ousast mechanisms export --slice S [--store P] [--json]`.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.2 (Req 1.3, 1.4).
- Mechanism gains origin, review_tier, pairs, shape, guard with defaults;
rows written before this spec load unchanged.
- MechanismStore.load folds the append-only log by id (latest row wins)
so a re-seeded shape carries every pair that taught it.
- append_from_pair writes origin=corpus rows with a deterministic id
derived from the shape key; corpus_mechanisms selects searchable rows.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
