Measured live against DeepSeek on one pair: the model spends its step budget
on tools and then answers in a paragraph that names the sink, the argument
and the line -- which parses as nothing. A detector that found the bug and
reported zero is indistinguishable from one that found nothing, and every
family in round zero would have baselined at zero for this reason alone.
The loop now takes one more turn when the hunt ends in prose or runs out of
steps, asking the provider for JSON and handing back the reasoning the model
already produced. It formats an answer that was reached; it never asks for a
new one, and a hunt that called no tool never gets there.
Same pair, same model, before and after: recall 0.00 -> 1.00, Youden 0.00 ->
1.00 over three runs per side.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Eight of nine group-3 tasks were rejected. Each defect was hiding a number,
and the numbers are why the loop was not learning:
- The fixed side was judged against the *vulnerable* side's function
ranges. 191 of 227 pairs with a resolvable labeled span (84%) have
different spans on the two sides, so a leak was recorded as silence,
always in the direction that flatters Youden.
- Round zero measured every family with the generalist. `family:unknown`
earns no credit against any real family, so every family but `unknown`
baselined at a structural zero and every noise floor with it.
- The cross-family sweep counted absolute failures instead of negative
flips. On a corpus where most pairs fail at round zero that rejects
every round for damage it did not do. Unscorable rows voted too.
- K runs was declared and never threaded; `pairs --k-runs` was parsed and
discarded, and its only test grepped `--help`.
- The meta-agent proposer was a stub, so `learning round` could never
propose anything. It now composes MetaAgent.evolve against a scratch
copy of the family directory, and `--proposals FILE` runs a recorded
sequence offline.
- `refused_holdout` was a vocabulary word no code path could produce, and
the covering test accepted either outcome.
- Cost was metered once, after everything had been paid for; a stage that
raised left no record at all.
- The corrected lever number was published beside the earlier one rather
than in place of it, and the test named for that property never checked.
- Refusals were `continue` statements: `Refusal.degradation()` existed and
nothing called it. A train-split identical twin was reported as
`split:train`, which is untrue.
Test repairs that each changed a number: two split tests measured an
`x = 1` fixture that teaches nothing whatever the rule says; two CLI
scoreboard tests asserted against all-zero output produced by a stub that
answers twice and puts its finding outside the labeled function.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.1. Tier one decides from what the repository already knows: a declared
family label, an obligation kind, a mechanism id, the rule a finding came from,
or the labeled weakness, and it never spends a model call. Tier two asks one
closed question in JSON mode with thinking disabled, and abstains when the
answer names an id outside the taxonomy or the reply's mean token
log-probability says the model was guessing. Tier three is abstention. Every
answer records which tier decided it.
Answers are multi-label and come back in taxonomy order, so two runs of the
same question compare equal. The classifier proposes and never writes a label,
and it owns validation of the labels the corpus carries, because the corpus
stores a family as an opaque string.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.5. Four tools join the three the hunter already had. Resolving a
definition by symbol returns its file, line span and body, decorators included,
so a detector stops guessing what a callee or a route handler does. Listing
entry points, flows and obligations answers with identifiers and line spans
only, so a structural tool can never become a second way to dump the
repository. Every path a caller supplies is clamped to the scan root first.
The loop now takes the system prompt, the user prompt, the files to start from,
the tool subset, the character limit and the tags its findings carry, each
defaulting to today's behaviour, and it replays the provider's reasoning field
on the next turn because the provider otherwise drops its own chain of thought
across a tool call. A detector configuration is exactly this set of knobs.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.2. Detectors, the classifier's model tier, the proposer and the judge
resolve one chat endpoint: an explicit override first, then the scripted client
flag, then an endpoint named in the config, then DeepSeek, then OpenRouter.
Prove-order retrieval keeps its own embedding endpoint untouched, because the
chat provider we default to offers none.
The adapter handles the provider's documented quirks: thinking is on by default
there and silently ignores temperature, so it is disabled explicitly; the
reasoning field is kept on the reply for the caller to replay across tool turns;
JSON mode has no schema and occasionally returns empty content, so one retry is
built in, never a loop; token log-probabilities are requested only when the
caller wants them, and their mean is what lets the classifier abstain.
Cost comes from the provider's own usage fields, cache hits priced separately,
and a model with no recorded price reports no cost rather than a made-up one.
The endpoint record deliberately holds no key, so it is safe to journal.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
