spec/steering: the decision strategy corrected: software features do not decide safety; the trace does, the pool measures and never trains

Maintainer's finding of 2026-10-14 recorded in hunter-learning-loop (a dated section that overrides the shape
annotations as the model's input), its design (amendment pointing to the prover's trace) and the steering
overview's current state.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-14 19:40:12 +02:00
co-authored by Claude Fable 5.1
parent 03b114cbeb
commit 03d925fffa
3 changed files with 46 additions and 0 deletions
@@ -1,5 +1,12 @@
# Design: hunter-learning-loop # Design: hunter-learning-loop
> **Amendment 2026-10-14.** The feature table below is the shape of the first round and is superseded as the
> model's input by the ordered trace of `access-control-prover` (design sections "trace: the ordered
> serialisation" and "cfg and CfgDominance"). The feature table remains the screen's instrument and the
> ablation's bookkeeping; the decision strategy is the one stated in requirements.md, "Decision strategy,
> corrected 2026-10-14": a rule over the trace, a zero-shot judge for the residual question, the pool for
> measurement only.
## Overview ## Overview
**Purpose**: turn the LLM hunter from a hand-written prompt into a measured search stage whose inputs are an **Purpose**: turn the LLM hunter from a hand-written prompt into a measured search stage whose inputs are an
engineered feature table and whose prompt is optimised by a program runtime, judged on held-out folds. engineered feature table and whose prompt is optimised by a program runtime, judged on held-out folds.
@@ -65,6 +65,34 @@ annotation set is kept or thrown away. It reports in the benchmark's own terms:
not retrained here; the earlier freeze on hunter changes ("no hunter prompt, tool or configuration may not retrained here; the earlier freeze on hunter changes ("no hunter prompt, tool or configuration may
change before that measurement") is lifted by the pilot measurement, which is the baseline. change before that measurement") is lifted by the pilot measurement, which is the baseline.
## Decision strategy, corrected 2026-10-14
The maintainer's finding after the second access-control screen: the annotations this loop selected and
screened were software features, not security features. Counts of callers, entry points, calls, binds,
function length, diff shape, language and framework describe the code around a change; none of them
determines that the code is unsafe, and a decision model fitted on them learns the pool (the deletion count
separates reversed fixes at 0.90) and generalises to nothing. "If you build a safety net and only send
information unrelated to security, the expected result is nothing."
What decides safety for the access-control family is a property of a path, in order: an entry an attacker
can reach, request-derived data reaching a protected operation along that path, and no required check
dominating the operation by control flow. That object, the ordered trace with argument provenance and
control-flow dominance, is specified and built in `access-control-prover` (requirements 9 and 10). From
this date:
1. The hunter's input is the trace, not the feature table. Shape annotations may be rendered for context but
are not decision features and the ablation shall not credit them as such.
2. The pool measures; it never trains. No model is fitted on the pool's labels, no prompt is compiled on the
pool, no threshold is tuned on it. The verdict is a rule over the trace, the same on every repository;
the facts naming entries, operations and checks are closed data with a stated source.
3. Where a judgement remains (is this entry meant to be public; is this operation protected somewhere the
facts do not model), the judge reads the trace zero-shot with general security knowledge. The pool
reports how often it is right; the unseen pilot says whether that holds elsewhere.
4. The ablation's remaining use is selection, not fitting: which trace components the judge needs to see,
with intervals, read with the same suspicion as any in-sample number.
5. Requirements 3.3 and 4 below stand as written for the record of the first round; where they name shape
annotations as the model's input, this section overrides them.
## Requirements ## Requirements
### Requirement 1: Labels from fix sites and ordinary pushes ### Requirement 1: Labels from fix sites and ordinary pushes
+11
View File
@@ -20,6 +20,17 @@ Everything below this section is dated history; where it disagrees with this sec
the LLM tier on pre-push once the search is good. (3) `search-with-proof` / `proof-arm-opencode`: proof on the LLM tier on pre-push once the search is good. (3) `search-with-proof` / `proof-arm-opencode`: proof on
what survives the search, as the block gate. (4) `engine-change-scoped-analysis`: the deterministic facts what survives the search, as the block gate. (4) `engine-change-scoped-analysis`: the deterministic facts
that feed the hunter's annotations. that feed the hunter's annotations.
- **Corrected 2026-10-14, decision strategy.** The hunter's annotations were software features (callers,
entry-point counts, calls, binds, diff shape, language, framework); they describe code around a change and
cannot decide unsafety, and anything fitted on them learns the pool (the deletion count separates reversed
fixes at 0.90). The security object is the ordered path: an attacker-reachable entry, request-derived data
reaching a protected operation, no required check dominating it by control flow. That is the access-control
prover (`.kiro/specs/access-control-prover`, deliveries A to D in the tree): entries per framework,
handler-named operations, enforcement witnesses, path records from the caller index, the trace with argument
provenance, a control-flow graph per function deciding dominance; a deterministic rule, no model call, 15 of
23 fix changes on the training pool with 15 of 15 flagged fixes correct. Rules from here: the pool measures
and never trains; no fitted model, no compiled prompt, no tuned threshold on it; a judge, where one is needed,
reads the trace zero-shot; the ablation selects trace components, it does not fit.
- **Everything else is history or parked.** Retired spec directories were removed from the tree on 2026-10-13 - **Everything else is history or parked.** Retired spec directories were removed from the tree on 2026-10-13
(`git show <commit>:.kiro/specs/<name>/` still has them): `harnessx-removal` (51d064e), (`git show <commit>:.kiro/specs/<name>/` still has them): `harnessx-removal` (51d064e),
`harnessx-self-improving-rulesets` and `openrouter-sast-harness` (c8d2fef), `learning-harness`, `harnessx-self-improving-rulesets` and `openrouter-sast-harness` (c8d2fef), `learning-harness`,