A4: roadmap-2026-09-04.md moves to .kiro/archive/steering/ so it no longer
loads as steering (roadmap.md and safety-net.md replaced it); roadmap.md's
pointer follows it. overview.md's spec table is recounted from tasks.md and
spec.json (adds ai-service-plane, harnessx-removal, learned-decision-engine,
pre-push-safety-net; drops the stale "active one"), and the module line now
cites the regenerated audit: 150 modules, 145 load-bearing, 5 standalone,
0 orphaned; 1,964 tests collected.
A5: a dated status note at the top of the requirements (or brief) of 16
specs: retired, superseded, dormant or complete, and for
model-grounded-detection the removal of judge/calibrate/execution (closing
the Req 10 execution-tier option). spec.json phase corrected to
implementation-complete where every task is checked (model-grounded-
detection, propose-adjudicate-prove, tree-sitter-overlay-extra,
corpus-seeded-mechanisms, pair-corpus-honesty); openrouter-sast-harness gets
a spec.json with status retired. Nothing deleted, no spec moved.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
README: "Configuring the HarnessX agentic plane" becomes "The agentic
plane (ax)" (ousast plane run|status|doctor|remember, per-task Model,
budget and egress, pointer to ops/ax/README.md, one tagged retirement
line); "The HarnessX self-improving cycle" becomes "The self-improving
cycle", and its MetaAgent paragraph is replaced by the memory proposer
(`ousast improve --memory`, provenance, train-on-test guard). The fusion
panel_model/decider_model lines and the [harnessx] spend caps are gone.
The README reference test now passes without its strict xfail.
docs/examples.md section 6 runs the plane; docs/threat-model.md describes
the per-task EgressPolicy, the usd/calls budgets and the start-request
credential path. RELEASE_NOTES.md gets an Unreleased entry: HarnessX
removed, the retired keys and their errors, the silent loss of the
second hunter for users who set [models] hunter, and the plane as the
agentic path.
Dated retirement notes (2026-09-30) head the requirements, design and
brief of harnessx-self-improving-rulesets, learning-harness,
three-stage-scan and openrouter-sast-harness, and follow the spec table
in steering/overview.md; history is unchanged. All tasks are checked, so
spec.json moves to implementation-complete.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
DeepSeek announced V4.1 Flash on 2026-09-10 and routes V4 Pro requests to it
from 12:00 Beijing time on 2026-09-14, billed at Flash's price.
The consequence worth writing down is not the id change. It is that the
detector and the judge are now the SAME model, where the design assumed the
judge was a stronger and more expensive second opinion. Every claim in this
repo that a judge call is a higher-quality check is now a claim about the
PROMPT and the evidence put to it, not about model class -- and the residual
question a judge is asked ("does this sanitizer suffice for this weakness")
has to carry its own weight.
Legacy ids stay in the price table so a recorded run from before the migration
still costs out; a manifest that cannot price its own model is one nobody can
audit.
The code for this landed in 83ab533, swept in by `git add -A` under a docs
message about the ranker -- the second time today. This commit is the record
it should have had.
855 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Maintainer correction: execution isn't possible for all SAST targets, and the
bugs this project owns -- IDOR, missing authz, injection -- don't crash
anything. Execution is the easy arbiter only for memory-safety-in-C, which is
not our world.
What execution provided was an independent, deterministic arbiter of whether a
claim is real -- not the same LLM that made it, not a noisy vote. That arbiter
can be built statically, from the code, as a program model the claim is checked
against. LLM proposes; the model disposes. Same architecture, same repair of
the root cause, no runnable target required.
The evidence ladder becomes static and deterministic:
suspicion -> model_corroborated (claim consistent with resolved dataflow /
a guard its siblings carry is absent) -> model_entailed (the model alone
establishes it) -> execution_confirmed (deferred tier, C memory-safety only).
model_corroborated and above are deterministic -- run the checker twice, same
verdict -- which removes the run-to-run noise the same way a crash would, with
no K-runs and no target to build. Our IR candidate enumerator (96.6% ceiling),
binding chains and obligation checker stop being prompt context and become the
arbiter itself.
The course: promote those assets from inputs to arbiter, delete the noise
layer, make the LLM a proposer the model audits, keep the corpus as the
model's calibration set (the fix diff is what the model should see), and defer
execution to Clearwing for the memory-safety classes only. Build the arbiter we
can build from code; adopt the one we cannot, when a class demands it.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Written after five evolve rounds produced zero decisions distinguishable from
noise, and after reading Clearwing, Big Sleep, and the AIxCC 2025 finals.
The premise we got wrong: OpenUltraSAST treats the LLM as a noisy oracle to
average (K runs), floor (noise budget), gate (sign test) and slowly improve
(evolve loop). 120 runs produced 120 distinct trajectories -- the variance is
the loop's freedom to search, not the model's uncertainty. And the premise is
empirically false as a claim about LLMs: Big Sleep found a real SQLite zero-day
in Nov 2024; AIxCC finalists uncovered 18 real 0-days across 54M LOC, with the
2nd-place system using non-reasoning LLMs at 90% accuracy and $181/point. The
lever is architecture, not prompt optimization.
What the winners share, and Clearwing is the open instance of: the LLM is a
judgement layer on an execution substrate, and truth comes from the sandbox --
a crash, a sanitizer report, a patch that stops the crash -- not from a score
against a labeled corpus. Clearwing's poc_runner (apply diff, recompile,
re-run, does-the-crash-still-happen) replaces our entire noise architecture. We
built K-runs because we had no ground truth; they generate their own.
The course: adopt Clearwing as the substrate (the maintainer already forks it),
delete the noise architecture rather than fix it, and contribute the one thing
Clearwing lacks -- the web/logic detection path. Our IR candidate enumerator
(12.1% ruleset vs 96.6% IR, measured) and our obligation/absence-bug checker
reach the dangerous web bugs that never crash a sandbox, which Clearwing's
crash-oracle model cannot. The open research problem is a differential oracle
for those classes: the vulnerable-parent/fix-commit pair as ground truth, with
a request the fixed version rejects and the vulnerable one serves.
Recommendation is explicitly not to rebuild: eighteen defects this session,
all ours, against a 19k-line battle-tested MIT substrate. Not approved.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Round 5 re-ran round 1's content through the lever it should have used --
counterexamples, rendered to the detector as "Worked examples" rather than
"Known non-issues, do not report these" -- and was rejected because one pair
of eight moved, at p = 1.000.
So the experiment I designed to give the demo lever a fair test could not
give it one. Correcting my authoring error changed the reason for the verdict
and not the verdict, because the rule cannot tell a lever that helps from a
lever that does nothing.
Five rounds, $2.17, zero decisions distinguishable from chance.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Four rounds on injection, $1.76. Not one decision carried evidence, in either
direction. Round 2 rejected a change because one pair of eight moved, at
p = 1.000 -- the maximum the sign test can return. Round 3 accepted one at
p = 0.500 and that acceptance is now versioned as v1. On a family where 11 of
20 pairs disagree with themselves between runs of the unchanged detector,
both readings are what changing nothing looks like.
I corrected an overstatement in the notes: after two rejections I wrote that
the rule was near-incapable of accepting. Round 3 accepted. The accurate
statement is symmetric and worse -- it accepts on noise as readily as it
rejects on it.
Escalated, not applied: Req 9.5's threshold rule should be replaced by an
acceptance that survives the measured variance. The Pareto archive and the
sign test both exist and neither decides anything. The honest caveat is that
at a minibatch of 8 a reliability-gated rule would accept almost nothing
either -- the corpus is the binding constraint, not the rule.
And the finding that outranks the rounds, in the maintainer's words: the
error is not the model, the error is the elsewhere. Eighteen defects this
session, every one in this repository, three of them the same shape of fixing
an instance and missing the class. The default assumption for an unexplained
number is now that we broke it -- which puts pro's 46.7% against flash's
83.3% on our side of the ledger until shown otherwise.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
learning-harness divides cleanly. The measurement layer stands and is worth
more than before -- it is what made a $0.23 experiment able to answer 'is
our own instability structural?'. What was answering the wrong question:
Req 3.5's K (an ensemble of distinct search procedures, not a repeated
measurement), Req 8.2's noise floor (it measures how often an unconstrained
search reaches the sink, so the regression budget grows with the search's own
wandering), Req 9.5's acceptance rule, and the sign test added this session.
And the highest-leverage change in either spec turns out not to be a prompt,
a lever or an optimizer: stop asking the model to locate the sink. 555 grep
calls against 71 entry_points, while the CST and the entry-point mapper
already know where the sinks are.
Limits stated in the brief: twelve pairs, one family, one model; the
high-similarity bucket is two run pairs and its 0% is decoration.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Round zero and one evolve round run end to end with the scripted client over
the whole vibe-py slice: per-family and per-slice blocks, twins out by
reason, a floor each, a proposal applied and reverted byte for byte, the
sweep reporting zero negative flips, the rejected version archived. The only
thing the live run adds is a detector that can see.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Measured on the corpus under the split and review-tier rules, access_control
has 1 train teacher and 4 holdout pairs: every agent-vfc row sits at tier
`title` and the vibe-py absence rows that could teach are trap pairs whose
two sides are different handlers, so the train split teaches zero obligation
shapes. injection has 12 teachers and 10 holdout pairs, the only family with
enough train evidence to tell a real change from noise.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Three defects, two of them the same shape as ones already rejected once:
- The meta-agent proposer still could not propose. `MetaAgent(inner_model=)`
takes a ModelConfig; it was handed a model name, a `# type: ignore` hid
that from mypy, and a bare `except Exception` turned the resulting
AttributeError into `meta_agent_failed` -- which reads exactly like the
model declining. The composition is now a seam a test can pin, and a
construction error is re-raised rather than reported as a quiet round.
- `ousast pairs --hunter` crashed in its documented form: the K floor landed
with `--k-runs` still defaulting to 1. The covering test always passed
`--k-runs 4`, which is why nothing noticed.
- 28 rows (12% of the corpus) resolved a span on the vulnerable side and
none on the fixed side, so every fixed-side finding read as outside the
labeled function and the pair scored as silence whatever the detector
said. `fix_function` now reaches the catalog and the scorer, and the
generator derives it from the fixed excerpt for the 25 trap rows that
never declared one. Measured after: 0 rows.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The three gate outputs are captured at 71efc9c -- the commit before the
first line of this feature -- and a test reproduces them byte for byte in
both matrices. Byte-identity was something a reviewer checked by hand
against a worktree; it is now mechanical and runs in CI.
Six absences are each exercised once with the reason they record: HarnessX,
the chat endpoint (the run still writes its artifact), the family
configurations, the network for pointer pairs, the semantic extra (named,
not guessed), and the sandbox -- so a claim nothing could run is not
confused with a claim that failed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The catalog generator fills `fix_date` from the recipe at the precision the
recipe has -- a full ISO day when one was recorded, otherwise the year.
Every harvested recipe carries a year and almost none carries a day, and
writing `-01-01` would turn a year into a claim about a month. A row counts
as post-cutoff only when its earliest possible date is after the cutoff, so
a coarse date can under-claim the clean slice but never over-claim it.
Each family now reports the same numbers over the rows fixed after the
cutoff, beside the whole number and never instead of it, with the undated
count on its own line. Without a configured cutoff every scoreboard says so
rather than inventing one: guessing a model's training cutoff and then
publishing a contamination number from the guess would be worse than
reporting none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The five byte-identical vibe-py twins now say so in the catalog instead of
only hashing to it, and a declared limit that names one of the unscorable
vocabulary's own reasons keeps that spelling, so `identical_twin` stays one
fact. Leave-one-out over vibe-py consequently runs on 30 rows, not 35.
The difficulty check measures three axes the corpus varies on: binding hops
from the identity source to the constrained use, registration distance, and
guard distance. Measured first, and it failed: the shared obligations
fixture reached (1, 0, 2) where the vibe-py absence rows reach (5, 2, 2).
Every fixture handler carried its own route decorator, so nothing in the
suite exercised a handler registered from another document and no binding
chain was longer than one hop -- the checker was passing on a simpler world
than the one it meets. `from_spec` closes that with the connexion shape
those rows really have. The verdict is committed beside the numbers.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Eight of nine group-3 tasks were rejected. Each defect was hiding a number,
and the numbers are why the loop was not learning:
- The fixed side was judged against the *vulnerable* side's function
ranges. 191 of 227 pairs with a resolvable labeled span (84%) have
different spans on the two sides, so a leak was recorded as silence,
always in the direction that flatters Youden.
- Round zero measured every family with the generalist. `family:unknown`
earns no credit against any real family, so every family but `unknown`
baselined at a structural zero and every noise floor with it.
- The cross-family sweep counted absolute failures instead of negative
flips. On a corpus where most pairs fail at round zero that rejects
every round for damage it did not do. Unscorable rows voted too.
- K runs was declared and never threaded; `pairs --k-runs` was parsed and
discarded, and its only test grepped `--help`.
- The meta-agent proposer was a stub, so `learning round` could never
propose anything. It now composes MetaAgent.evolve against a scratch
copy of the family directory, and `--proposals FILE` runs a recorded
sequence offline.
- `refused_holdout` was a vocabulary word no code path could produce, and
the covering test accepted either outcome.
- Cost was metered once, after everything had been paid for; a stage that
raised left no record at all.
- The corrected lever number was published beside the earlier one rather
than in place of it, and the test named for that property never checked.
- Refusals were `continue` statements: `Refusal.degradation()` existed and
nothing called it. A train-split identical twin was reported as
`split:train`, which is untrue.
Test repairs that each changed a number: two split tests measured an
`x = 1` fixture that teaches nothing whatever the rule says; two CLI
scoreboard tests asserted against all-zero output produced by a stub that
answers twice and puts its finding outside the labeled function.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Eight of the ten vendored access-control rows had no named handler at all,
so every one of them would have scored `missing_context` forever rather
than as a detector miss. After the re-harvest nine are named and the tenth
states its reason.
The handler-context mode now carries the class its framework registers,
multi-line decorators, and commented-out guards -- `#@token_required`
above a ThreatByte handler is the entire bug. Where the registration lives
in another document (connexion's `operationId`, an express router), a
recipe names `context = [...]`, the harvester writes that stanza per side,
and the pair materializes both files at their repository-relative paths so
the entry-point mapper sees the layout the project has. `harvest_recipe`
fetches every blob before the first write and the driver takes `--report`,
so a failed fetch leaves a reviewed excerpt exactly as it was.
Two defects the re-harvest exposed: `.mjs` was not a known suffix, so the
one pair vendored as an ES module had no file target at all; and
`_python_route_access` matched auth words inside decorator prose, reading
`@ns.doc(description='... without proper authorization checks')` as a
guard. Recorded in the hygiene artifact: nine of the ten duplicate groups
sit on both sides of the split, which the repository-level split cannot
see.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The maintainer settled the conflict the group 2 review escalated: a detector
configuration has no skills set. The skill router belongs to the tool's earlier
shape, and a family's prompt, checklist, tools and counterexamples already carry
what a round evolves.
The README described the project as an editor-agent harness whose skills steer
it, and pointed users at the maintainer's spec directory. The tool is a CLI now
and the agent integrations are optional, so the opening says that, the skills
section describes what it actually is, and the spec pointer is gone from the
user-facing text.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The maintainer settled both items the group 1 review escalated.
The family that covers server-side request forgery is named untrusted
destination and also covers open redirect, because both are untrusted input
choosing where a request goes and the corpus carries 34 open-redirect rows that
would otherwise be unclassifiable. The requirement, the brief and the design
example now say so, and the approval note records the amendment.
The shipped taxonomy is flat, so no answer can earn hierarchical partial credit.
Rather than leave that as prose, two new acceptance criteria require every
agreement figure and every family figure to say it, and both report types now
carry the flag, computed from the taxonomy rather than asserted. A taxonomy that
later declares a parent flips it without any other change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.4. Per family, and per slice when asked: outcome counts, recall, silence,
Youden, a directional-bias index and recall at a fixed false-positive ceiling.
Rows that cannot be scored are listed by reason and leave every denominator, and
a family with nothing scorable reports no rate rather than a perfect one.
Two numbers exist because Youden cannot tell apart a detector that flags
everything from one that says everything is safe: the directional-bias index
separates them, and recall at a fixed ceiling is forfeited entirely once a
family leaks more than the budget allows.
Change against a baseline is not a difference of means. Flips are counted per
pair in both directions and a two-sided exact sign test decides whether the
change is bigger than the noise, so one flip in ten reads as noise and six reads
as a real regression.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Foundation: taxonomy, DeepSeek endpoint, corpus fields, TypeScript grammar,
curated hunter tools. Core: classifier tiers and measurement, per-pair family
scoring and aggregation, detector configurations and runner, verifier registry
with oracle callback, canary templates and second judge, journal and archive.
Integration: one teacher rule, family scoring on the pairs hunter path, round
zero with per-model noise floors, scan-path tags and manifest block, proposers,
acceptance and revert, round orchestration, publish, subcommands. Corpus
repairs and validation last. Both sanity reviews' findings recorded in the
plan; K-run evaluation honors requirement 3.5 at every stage.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
A learning package layered families -> classify -> scoring -> split ->
detectors and verifiers -> rounds -> publish, composing the existing seams:
the tool hunter gains prompt, context and tool parameters; the hunter path of
the pair scorer takes an injected family matcher so pairs never imports
learning; every learning path selects teachers through one train-only rule;
the regress verdict takes an optional canary oracle; the chat endpoint resolves
DeepSeek before OpenRouter with thinking disabled and reasoning replayed, while
embeddings keep their own endpoint; HarnessX proposes behind the seam with a
scripted proposer for offline runs. Fifty-nine acceptance criteria traced;
boundary sections, file structure plan and migration phases populated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Scope: catch the ten taxonomy families reliably on real-world and generated
application code, including forgotten authorization checks; not browser
engines, interpreters or memory-safety fuzzing. Decisions: verifier-aligned
families, a measured per-family noise floor as the regression budget, a cheap
model on an independent chat endpoint with embeddings elsewhere, canary-oracle
verifiers for the four most dangerous families first, the obligations lever
folded into the access-control family's first round, web families before the
C slice. Eleven requirements in EARS form; awaiting approval.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
AIxCC cyber reasoning systems, self-improving harness methods (GEPA, DGM,
Self-Harness, SkillOpt, AHE), vulnerability-dataset pitfalls and CWE taxonomies,
and routing/regression-control literature. The design changes they force:
coarse verifier-aligned families instead of fine CWE, multi-label routing with
an unknown class, pair-wise outcomes with a directional-bias index, K-run
reliable-change tests, a per-class seed-noise floor as the regression budget,
a no-regression acceptance rule with ties rejected, structural edits over prose
edits, a proposer blind to the holdout, and verifier code and gold labels
outside every editable surface.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Measured 2026-09-06 on the vibe-py holdout: the existing tool hunter, with no
learning, scores 12/14 pair-correct (Youden +0.857, no leaks) once detection
and silence are keyed on the labeled class and function and the byte-identical
twins are excluded; the overlay scores 5/14 on the same rows. The published
hunter number (0/17, Youden -0.588) was a scoring artifact: the scorer wanted
the literal CWE id in the finding text and the prompt never asked for it.
The brief proposes an auto-classifier as the routing key, one evolved hunter
configuration per class, a class-aware scorer with unscorable denominators,
the split enforced by construction, and the HarnessX evolve loop with per-class
attribution as the round primitive. Raw measurements are committed under
reports/measurements.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
