diff --git a/.kiro/specs/hunter-learning-loop/design.md b/.kiro/specs/hunter-learning-loop/design.md index a26e06f..067c7ea 100644 --- a/.kiro/specs/hunter-learning-loop/design.md +++ b/.kiro/specs/hunter-learning-loop/design.md @@ -1,5 +1,12 @@ # Design: hunter-learning-loop +> **Amendment 2026-10-14.** The feature table below is the shape of the first round and is superseded as the +> model's input by the ordered trace of `access-control-prover` (design sections "trace: the ordered +> serialisation" and "cfg and CfgDominance"). The feature table remains the screen's instrument and the +> ablation's bookkeeping; the decision strategy is the one stated in requirements.md, "Decision strategy, +> corrected 2026-10-14": a rule over the trace, a zero-shot judge for the residual question, the pool for +> measurement only. + ## Overview **Purpose**: turn the LLM hunter from a hand-written prompt into a measured search stage whose inputs are an engineered feature table and whose prompt is optimised by a program runtime, judged on held-out folds. diff --git a/.kiro/specs/hunter-learning-loop/requirements.md b/.kiro/specs/hunter-learning-loop/requirements.md index ed0d98f..e435874 100644 --- a/.kiro/specs/hunter-learning-loop/requirements.md +++ b/.kiro/specs/hunter-learning-loop/requirements.md @@ -65,6 +65,34 @@ annotation set is kept or thrown away. It reports in the benchmark's own terms: not retrained here; the earlier freeze on hunter changes ("no hunter prompt, tool or configuration may change before that measurement") is lifted by the pilot measurement, which is the baseline. +## Decision strategy, corrected 2026-10-14 + +The maintainer's finding after the second access-control screen: the annotations this loop selected and +screened were software features, not security features. Counts of callers, entry points, calls, binds, +function length, diff shape, language and framework describe the code around a change; none of them +determines that the code is unsafe, and a decision model fitted on them learns the pool (the deletion count +separates reversed fixes at 0.90) and generalises to nothing. "If you build a safety net and only send +information unrelated to security, the expected result is nothing." + +What decides safety for the access-control family is a property of a path, in order: an entry an attacker +can reach, request-derived data reaching a protected operation along that path, and no required check +dominating the operation by control flow. That object, the ordered trace with argument provenance and +control-flow dominance, is specified and built in `access-control-prover` (requirements 9 and 10). From +this date: + +1. The hunter's input is the trace, not the feature table. Shape annotations may be rendered for context but + are not decision features and the ablation shall not credit them as such. +2. The pool measures; it never trains. No model is fitted on the pool's labels, no prompt is compiled on the + pool, no threshold is tuned on it. The verdict is a rule over the trace, the same on every repository; + the facts naming entries, operations and checks are closed data with a stated source. +3. Where a judgement remains (is this entry meant to be public; is this operation protected somewhere the + facts do not model), the judge reads the trace zero-shot with general security knowledge. The pool + reports how often it is right; the unseen pilot says whether that holds elsewhere. +4. The ablation's remaining use is selection, not fitting: which trace components the judge needs to see, + with intervals, read with the same suspicion as any in-sample number. +5. Requirements 3.3 and 4 below stand as written for the record of the first round; where they name shape + annotations as the model's input, this section overrides them. + ## Requirements ### Requirement 1: Labels from fix sites and ordinary pushes diff --git a/.kiro/steering/overview.md b/.kiro/steering/overview.md index 33975fb..328546d 100644 --- a/.kiro/steering/overview.md +++ b/.kiro/steering/overview.md @@ -20,6 +20,17 @@ Everything below this section is dated history; where it disagrees with this sec the LLM tier on pre-push once the search is good. (3) `search-with-proof` / `proof-arm-opencode`: proof on what survives the search, as the block gate. (4) `engine-change-scoped-analysis`: the deterministic facts that feed the hunter's annotations. +- **Corrected 2026-10-14, decision strategy.** The hunter's annotations were software features (callers, + entry-point counts, calls, binds, diff shape, language, framework); they describe code around a change and + cannot decide unsafety, and anything fitted on them learns the pool (the deletion count separates reversed + fixes at 0.90). The security object is the ordered path: an attacker-reachable entry, request-derived data + reaching a protected operation, no required check dominating it by control flow. That is the access-control + prover (`.kiro/specs/access-control-prover`, deliveries A to D in the tree): entries per framework, + handler-named operations, enforcement witnesses, path records from the caller index, the trace with argument + provenance, a control-flow graph per function deciding dominance; a deterministic rule, no model call, 15 of + 23 fix changes on the training pool with 15 of 15 flagged fixes correct. Rules from here: the pool measures + and never trains; no fitted model, no compiled prompt, no tuned threshold on it; a judge, where one is needed, + reads the trace zero-shot; the ablation selects trace components, it does not fit. - **Everything else is history or parked.** Retired spec directories were removed from the tree on 2026-10-13 (`git show :.kiro/specs//` still has them): `harnessx-removal` (51d064e), `harnessx-self-improving-rulesets` and `openrouter-sast-harness` (c8d2fef), `learning-harness`,