Open_Ultra_SAST/.kiro/specs/pre-push-safety-net/strategy-review-2026-09-28.md

Strategy review after the independent evaluations (2026-09-28)

Status: for maintainer decision. Nothing below changes the approved specs until a direction is chosen.

1. What was measured

Two populations of real vulnerabilities that no rule had seen, each with its fix and a benign control, scored by protocols committed before the first scan (benchmarks/independent/).

	v1 (11 cases, 2026-09-24)	v2 (17 cases, 2026-09-28)
Analyzer	13180a1	a75b9f6 (v1 plus every retune learned from v1)
Recall	1 / 11	0 / 17 (Wilson 95% 0–0.18)
Sampled precision	1 / 12	0 / 17 (0–0.18)
Question completion	near zero	12% (13,206 / 108,374)
Benign false alerts	—	3
Run-to-run stability	not measured	2 of 100 findings differ

Between v1 and v2 the analyzer received every fix v1 suggested: file-owned graph gaps, .gitignore semantics, Node vm sinks, Django sources, quoted-only SQL escapes, fixed-origin redirects, guards and allowlists. On the v1 cases those retunes raised detection from 1 to 4 of 11. On untouched cases they transferred to nothing.

2. Why, per class of cause (observed in v2, from source)

1. Entry points are framework-shaped, and our sources are token lists. Request data arrives as filter_input(INPUT_GET, …) and the WordPress REST $request (GoPay, Directorist), plugin helpers such as get_url_var() (Groundhogg), FastAPI and NestJS typed or decorated handler parameters (qwed, ReactPress, LMDeploy, ContextForge), and Koa's ctx.request (Budibase). A vocabulary grows one framework at a time, and each new population brings frameworks it lacks.
2. Flows cross module boundaries the frontends do not resolve. JavaScript required modules (FUXA's route to its storage module), dependency-injected receivers (YesWiki, v1), registry dispatch (OpenCVE, v1), parser combinators (alerta, v1).
3. Sinks are matched by name, so precision fails where names collide. Every sampled v2 finding was false: a regex .match()/.exec() read as injection, header() with an integer cast, parameterised ORM statements, client-SDK code, a process's own argv, a generated database name.
4. Scale. Large repositories complete a few percent of their questions inside a 40-minute budget (pgAdmin 2%, ContextForge 6%, Budibase 1%); one repository crashes the taint batch on every scan (LMDeploy).

Classes 1–3 are breadth problems: each fix is correct and local, and the next repository needs another. That is the pattern both evaluations show, and the reason more patching on the same design is unlikely to reach M4.

3. What we know about the alternative

The tool hunter (a model with read_file / grep_repo / find_refs) was measured once, on 2026-09-06, on the vibe-py pair holdout: 12 of 14 detected, Youden +0.857, no leaks, against the overlay's 5 of 14 on the same pairs (class-aware scoring; benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-*.json). Caveats: one language, curated excerpts, a pair corpus rather than whole repositories, and never run on an independent population. Classes 1–3 above are exactly where a model reading the code has an advantage over token lists; class 4 (cost, scope) is where it has a disadvantage.

4. Options

A. Continue engine-first. Model framework entry points, fix the sink-collision classes, raise completion, then qualify on a v3 population. Every item is sound; the evidence (v1 → v2) says the breadth does not converge at the pace one population teaches. Highest effort, weakest evidence.

B. Model proposes, engine proves. The hunter supplies recall; the CPG, where it can reach, supplies the witness and admission evidence; findings the engine cannot confirm are reported as unconfirmed rather than dropped. This inverts propose-adjudicate-prove's current out-of-scope line ("making an LLM the adjudicator") only partly: the model proposes, it does not adjudicate. Needs a cost and scope design (which files a pre-push scan hands the model) and a run of the unchanged hunter on an untouched population before any tuning.

C. Narrow the product. Qualify one ecosystem first (for example WordPress plugins, the declared v0.1 target), where a framework model is finite, and ship only that capability (M5 already allows per-capability enablement). Compatible with A or B.

5. Recommendation

Run the decisive, cheap experiment before choosing: the unchanged tool hunter on the v2 pins, scored with protocol v2's matching, within a stated model-cost budget. v2 is spent for the engine, but the hunter was never tuned on it; the rule is that no hunter prompt, tool or configuration may change before that measurement is recorded, and the scope handed to it must be decided and written down first (it must not use the fix diff or the declared sites, which would leak the answer).

• If the hunter reaches useful recall on v2 with few fixed-side and benign alerts, re-plan M2–M4 around B, narrowed by C.
• If it does not, pursue A narrowed by C, starting with the WordPress entry-point model and the sink-collision precision classes, and qualify on a v3 population.

6. Decisions needed

1. Direction: run the hunter experiment first (recommended), or choose A, B or C now.
2. For the experiment: the scope handed to the model per pin, the model, and the cost ceiling.
3. Whether the roadmap's M1b/M2 work (rendering, runtime) pauses until the direction is chosen.

7. Update 2026-09-29: what the experiment showed

Recorded in benchmarks/independent/results-v2-model-pipeline.json (exploratory; v2 is spent for tuning).

Instrument first. Four defects had made every model-backed result since 2026-09-10 unmeasured: the default model id was the product name and the API rejected it (cf57995); the hunter reached the embeddings provider, not the chat provider (cf57995); the final-answer turn used the provider's JSON mode and the answers were lost (3ffb8b4); one grep match in a minified bundle overflowed the model's context (8794884). The "hunter 1/17" number in section 5's experiment was produced through the broken answer turn and is void.

Scope selection, source only. The declared vulnerable function is among the candidates for: the ranker's top-20 entry points 1/17; the shipped sink vocabulary 2/17; a model reading every product file 15/17 ($14). The vocabulary route (option A) is confirmed not to converge on this population.

Verification. Per-file triage, one 6-step tool hunt per file, two independent passes, report only agreement:

• validation set: 16 of 20 declared sites agreed, 12 of 15 cases; the 26 pgAdmin candidates that flickered between single runs reduce to 9 stable, 4 disputed, 13 quiet;
• full run, 7 of 17 cases before the account emptied ($23): detected 4/7 (pgAdmin, qwed, LMDeploy, ReactPress); ContextForge and FUXA found in one pass of two; winml missed because a CORS literal is triaged "constant" -- the config family needs its own question, not the reachability one;
• cost: $0.009-0.022 per candidate for two passes (old single-pass design: $0.017); a full two-pass run of the population is roughly $45-50.

Precision is the open problem. 196 agreed findings across 7 cases (pgAdmin alone 126 of 875 verified). Five pgAdmin findings read against source: three true (privileged; raw {{ data.* }} in SQL templates, unescaped stored schema_res), two false with rationales that contradict their own verdict (an ORM call the model itself calls parameterized; a pickle from the signed session). Agreement cannot catch a mistake both passes make. The proposed next filter is a no-tools judge over each agreed finding's rationale and code (~$0.002 per finding), which was not run. Two-pass agreement is itself not stable between runs (ContextForge, FUXA).

Standing. Recall: the model route reaches 12/15 candidate-site cases where the engine reached 0/17 and the vocabulary 2/17; that decides between options A and B in the model's favour for recall. Precision, fixed-pin behaviour and cost at product scale are unmeasured. No gate is met. Any qualification needs population v3 and a pre-registered protocol that also states the judge stage, the pass rule and the config-family question.