mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
spec/steering: the decision strategy corrected: software features do not decide safety; the trace does, the pool measures and never trains
Maintainer's finding of 2026-10-14 recorded in hunter-learning-loop (a dated section that overrides the shape annotations as the model's input), its design (amendment pointing to the prover's trace) and the steering overview's current state. Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
03b114cbeb
commit
03d925fffa
@@ -1,5 +1,12 @@
|
||||
# Design: hunter-learning-loop
|
||||
|
||||
> **Amendment 2026-10-14.** The feature table below is the shape of the first round and is superseded as the
|
||||
> model's input by the ordered trace of `access-control-prover` (design sections "trace: the ordered
|
||||
> serialisation" and "cfg and CfgDominance"). The feature table remains the screen's instrument and the
|
||||
> ablation's bookkeeping; the decision strategy is the one stated in requirements.md, "Decision strategy,
|
||||
> corrected 2026-10-14": a rule over the trace, a zero-shot judge for the residual question, the pool for
|
||||
> measurement only.
|
||||
|
||||
## Overview
|
||||
**Purpose**: turn the LLM hunter from a hand-written prompt into a measured search stage whose inputs are an
|
||||
engineered feature table and whose prompt is optimised by a program runtime, judged on held-out folds.
|
||||
|
||||
@@ -65,6 +65,34 @@ annotation set is kept or thrown away. It reports in the benchmark's own terms:
|
||||
not retrained here; the earlier freeze on hunter changes ("no hunter prompt, tool or configuration may
|
||||
change before that measurement") is lifted by the pilot measurement, which is the baseline.
|
||||
|
||||
## Decision strategy, corrected 2026-10-14
|
||||
|
||||
The maintainer's finding after the second access-control screen: the annotations this loop selected and
|
||||
screened were software features, not security features. Counts of callers, entry points, calls, binds,
|
||||
function length, diff shape, language and framework describe the code around a change; none of them
|
||||
determines that the code is unsafe, and a decision model fitted on them learns the pool (the deletion count
|
||||
separates reversed fixes at 0.90) and generalises to nothing. "If you build a safety net and only send
|
||||
information unrelated to security, the expected result is nothing."
|
||||
|
||||
What decides safety for the access-control family is a property of a path, in order: an entry an attacker
|
||||
can reach, request-derived data reaching a protected operation along that path, and no required check
|
||||
dominating the operation by control flow. That object, the ordered trace with argument provenance and
|
||||
control-flow dominance, is specified and built in `access-control-prover` (requirements 9 and 10). From
|
||||
this date:
|
||||
|
||||
1. The hunter's input is the trace, not the feature table. Shape annotations may be rendered for context but
|
||||
are not decision features and the ablation shall not credit them as such.
|
||||
2. The pool measures; it never trains. No model is fitted on the pool's labels, no prompt is compiled on the
|
||||
pool, no threshold is tuned on it. The verdict is a rule over the trace, the same on every repository;
|
||||
the facts naming entries, operations and checks are closed data with a stated source.
|
||||
3. Where a judgement remains (is this entry meant to be public; is this operation protected somewhere the
|
||||
facts do not model), the judge reads the trace zero-shot with general security knowledge. The pool
|
||||
reports how often it is right; the unseen pilot says whether that holds elsewhere.
|
||||
4. The ablation's remaining use is selection, not fitting: which trace components the judge needs to see,
|
||||
with intervals, read with the same suspicion as any in-sample number.
|
||||
5. Requirements 3.3 and 4 below stand as written for the record of the first round; where they name shape
|
||||
annotations as the model's input, this section overrides them.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Labels from fix sites and ordinary pushes
|
||||
|
||||
@@ -20,6 +20,17 @@ Everything below this section is dated history; where it disagrees with this sec
|
||||
the LLM tier on pre-push once the search is good. (3) `search-with-proof` / `proof-arm-opencode`: proof on
|
||||
what survives the search, as the block gate. (4) `engine-change-scoped-analysis`: the deterministic facts
|
||||
that feed the hunter's annotations.
|
||||
- **Corrected 2026-10-14, decision strategy.** The hunter's annotations were software features (callers,
|
||||
entry-point counts, calls, binds, diff shape, language, framework); they describe code around a change and
|
||||
cannot decide unsafety, and anything fitted on them learns the pool (the deletion count separates reversed
|
||||
fixes at 0.90). The security object is the ordered path: an attacker-reachable entry, request-derived data
|
||||
reaching a protected operation, no required check dominating it by control flow. That is the access-control
|
||||
prover (`.kiro/specs/access-control-prover`, deliveries A to D in the tree): entries per framework,
|
||||
handler-named operations, enforcement witnesses, path records from the caller index, the trace with argument
|
||||
provenance, a control-flow graph per function deciding dominance; a deterministic rule, no model call, 15 of
|
||||
23 fix changes on the training pool with 15 of 15 flagged fixes correct. Rules from here: the pool measures
|
||||
and never trains; no fitted model, no compiled prompt, no tuned threshold on it; a judge, where one is needed,
|
||||
reads the trace zero-shot; the ablation selects trace components, it does not fit.
|
||||
- **Everything else is history or parked.** Retired spec directories were removed from the tree on 2026-10-13
|
||||
(`git show <commit>:.kiro/specs/<name>/` still has them): `harnessx-removal` (51d064e),
|
||||
`harnessx-self-improving-rulesets` and `openrouter-sast-harness` (c8d2fef), `learning-harness`,
|
||||
|
||||
Reference in New Issue
Block a user