Marius CiepluchandClaude Opus 5.5
afde23d3cf
Maintainer direction: A/B testing to extrapolate detection instead of hard-coding it, and a trained decision engine. Instruments become signal producers; labels from ground truth with provenance; a simple calibrated engine with population folds; experiments as plane Runs; extrapolation with intervals, checked once on frozen v3; the engine is what scan/pre-push/plane report. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2.8 KiB
Brief: a trained decision engine and A/B experiments instead of hand-coded detection
Maintainer, 2026-09-30: "you need to go to a different approach with a b testing to extrapolate detection rather than hardcoding it and train a decision engine".
Why
Every detection gain so far was a hand edit made after a miss: regex rules, taint vocabulary, guards, prompt lines, a tie-break rule. Each edit was tuned on the cases that exposed it and had to be re-qualified on a fresh population; v1 -> v2 showed engine tuning does not transfer (0/17). The plane changed what is possible: every task records per-candidate outcomes, costs and repository facts into one memory store, and runs are cheap to repeat.
Direction
- Instruments become signal producers: quick rules, the Joern engine, model sink classification, verify passes, tie-break, repo facts (callers, reachability), triage. None decides alone.
- A decision engine, trained on labelled outcomes, turns the signals for a candidate into a calibrated probability that it is a real vulnerability of its family; operating points (block, report, drop) are thresholds on it.
- Changes to the pipeline (prompts, models, pass counts, rule sets, features) are variants compared by A/B experiments run as plane Runs with pre-registered metrics and confidence intervals, never adopted on a single run's point estimate.
- Detection on unseen code is extrapolated from measured learning curves and leave-one-population-out results, reported with uncertainty, and checked once on the frozen v3.
Data on hand (2026-09-30)
- Pair corpus: 317 vulnerable/fixed pairs, 342 labelled findings (
ousast pairs), with train/holdout splits. - Populations: v1 (spent), v2 validation set (46 candidates, 20 declared sites; plane runs recorded), PHP development corpus (pmpro, v1 PHP, MediaWiki, WP Statistics). v3 (PHP, 16 cases) is being frozen now and stays untouched until the final check.
- Memory store rows: facts, per-pass verdicts, disputes, tie-breaks, costs, alerts, coverage.
Risks
- Small labelled data: the engine must be simple and calibrated (regularised linear or shallow trees), with population-level folds; a deep model would memorise.
- Leakage: labels and features must never come from the population a number is reported on (the train-on-test rule; see the closed-loop leak).
- Label quality: "rejected by the plane" is not "safe"; negatives come from fixed and benign pins and adjudicated findings, not from absence of an alert.
Relation to earlier specs
Replaces the hand-edit loop of improve for detection decisions (the memory proposer's M1/M2 status edits become
one variant family among others). learning-harness and harnessx-self-improving-rulesets are retired with
HarnessX. finding-feedback-loop, constrained-detector and model-grounded-detection are inputs to read in
design, not dependencies.