From f6f9591cd0caa6c892f0ef62bc399782259d02b8 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Wed, 30 Sep 2026 18:13:27 +0200 Subject: [PATCH] spec(learned-decision-engine): design for review Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo --- .kiro/specs/learned-decision-engine/design.md | 915 ++++++++++++++++++ .kiro/specs/learned-decision-engine/spec.json | 6 +- 2 files changed, 918 insertions(+), 3 deletions(-) create mode 100644 .kiro/specs/learned-decision-engine/design.md diff --git a/.kiro/specs/learned-decision-engine/design.md b/.kiro/specs/learned-decision-engine/design.md new file mode 100644 index 0000000..b7c2494 --- /dev/null +++ b/.kiro/specs/learned-decision-engine/design.md @@ -0,0 +1,915 @@ +# Design Document + +## Overview + +Detection decisions move out of hand edits and into a trained, calibrated model. The instruments stay and emit +signals. A model-free feature builder turns those signals into one record per candidate. A label builder attaches +ground truth from sources we can defend. A small logistic model, evaluated only on repositories it never saw, +turns a record into a probability, and two thresholds on that probability (BLOCK, ADVISORY) are what `pre-push`, +`scan` and plane Runs report. Pipeline changes are variants in pre-registered, paired experiments. + +Six facts from the repository and the recorded results shape the design: + +- **The deployment target is a repository nobody trained on.** Maintainer: "the tool will never run against the + test cases it's being trained with. it's purpose is a repo safety net." (Req 7). The primary unit is a candidate + in changed code (`ousast pre-push`, base..head), and whole-repository `scan` is the secondary mode. Every + reported number therefore comes from folds grouped by repository across all corpora. Features are + repository-agnostic by an allow-list. Fit to a user's own repository is an opt-in layer outside the reported + numbers. +- **Labels are scarce, and full plane signals exist on 43 rows.** The only recorded plane run with + verify/agree/tie-break is `validation-46`: 15 cases, 43 candidates after triage, 19 of them with + `site_match` true, all from population v2 + (`~/ousast-results/plane/validation-46/*-final/agreed.json`). The pair corpus is larger (427 pairs and 452 + labelled findings in the catalog today, 317/342 in the 2026-09-30 baseline + `benchmarks/measurements/2026-09-30-harnessx-removal-baseline/pairs.json` `overall`). Measured by + repository rather than by row, it is 146 groups, and two families are nearly single-repository (section 3). A + model over verify signals cannot be trained today. The design trains on the signals every labelled row can + have, and harvests the rest at a stated cost. +- **The core install is PyYAML only** (`pyproject.toml:11`). numpy, scipy and scikit-learn are not installed in + `.venv`. Inference must be pure Python, and training should be too (section 4). +- **`pre-push` already has the decision seam.** `admit_candidates` (`push/policy.py:879`) applies one evidence + bar to `CandidateDelta`s with `novelty` (`push/policy.py:61-74`). Its capability registry is empty by design + (`push/policy.py:802-823`: "No shipped declaration is enabled"), so `pre-push` blocks nothing today. The + engine's operating point replaces the capability-eligibility reasons, and the structural reasons stay. +- **The memory store is the system of record and has the needed shape.** Rows are keyed by repository and pin, + with a closed `KINDS` tuple (`plane/memory.py:52`). There are two backends (FileStore, MinioStore) behind one + interface (`plane/memory.py:176-291`), content-addressed blobs (`put_facts`, `plane/memory.py:249-255`), and a + train-on-test `Guard` (`improve/memory.py:95-133`). Features, labels, decisions, model artifacts and experiment + outcomes become new row kinds and blob prefixes. No second store is added. +- **Hand-written framework knowledge is embedded in the detectors** (Req 8). Examples are WordPress `$wpdb`, + hooks and `maybe_unserialize` in `ruleset/semantic/php.toml` and `ruleset/php/rules.toml`, and Flask and + Django names in `ruleset/semantic/python.toml`. Section 2 tags each entry. Priors become optional features + that are off by default. Framework roles are inferred per repository instead. + +Decisions in one place: + +| Question | Decision | +| --- | --- | +| Candidate key | `(repo, pin, path, function, family)` in the row envelope only; `pin` is the head for a delta | +| Features | allow-listed, typed, repository-agnostic; explicit `null` + instrument state for missing; two profiles (`static`, `plane`) | +| Where features are built | plane task `features` (no model) after `agree`/`final` and `alerts`; host path `learn.features.build_for_scan` for `scan`/`pre-push` | +| Labels | host-only `ousast learn labels`; site/fix-range matcher for populations, vulnerable/fixed side for pairs, benign deltas, adjudications; fail-closed source allow-list keeps v3 out | +| Model | L2-regularised logistic regression, pure Python (Newton/IRLS), per family at >= 20 positive and >= 20 negative repositories, pooled at >= 10, otherwise "insufficient data" | +| Calibration | Platt scaling on inner grouped out-of-fold scores; checked per outer fold (reliability, slope, intercept); if it fails, ADVISORY only | +| Folds | outer grouped K-fold by repository (repeated), plus leave-one-source-out and leave-one-framework-out; nested inner folds for lambda, calibration, thresholds | +| Operating points | BLOCK: lowest threshold whose inner-fold Wilson lower bound on precision is >= 0.95; ADVISORY: threshold reaching 0.90 inner-fold recall; reported with repository-bootstrap intervals | +| Artifacts | content-addressed JSON under `models/` in the store; adopted version exported into the package by a maintainer commit | +| Experiments | manifest `plane/experiments/.yaml` + Run annotation `openultrasast.io/experiment`; paired; repository-cluster bootstrap CI (primary), exact McNemar (secondary); O'Brien-Fleming two-look stopping | +| First experiment | `exp-001-known-callers`: callers line on vs off, development corpus (v1, pmpro, MediaWiki, WP Statistics, full-repository pointer pairs), about $8 expected, $15 ceiling | +| Framework knowledge | tagged `framework`/`library` priors, off by default; roles inferred by a `roles` plane task (model) + deterministic wrapper inference; leave-one-framework-out evaluation | + +## Architecture + +``` + instruments (signal producers, unchanged as standalone tools) + quick rules Joern engine repo-facts roles (new) verify a/b/c + agree + alerts task engine_alerts / callers, entry model role labels (plane only, Model-bound) + pre-push scan distance + wrapper inference + \ | | | / + +--------------+------------------+-------------------+--------------------+ + | + features (new, no model): allow-listed record per candidate + plane: task after final + alerts host: learn.features.build_for_scan + | + +---------------------+----------------------+ + | | + labels (new, host only, never in a sandbox) decide (new, no model) + populations: matches_v2 / fix ranges load adopted model -> p, operating point, + pairs: vulnerable vs fixed side top features -> decision rows / report fields + benign deltas, adjudications ^ + source allow-list: v3 absent | + | | + learn train/evaluate (host, pure Python) ---- model artifact (models/.json) ---- adopt (maintainer) + grouped folds, Platt, thresholds, curves | + | packaged: ruleset/decision/-.json + experiments (plane Runs with arms) -> arm_outcome rows -> analysis -> adopt | reject | inconclusive +``` + +A plane Run per case becomes `facts -> va, vb -> agree -> vc -> final -> alerts -> roles? -> features -> decide +-> remember`. `roles` is optional and Model-bound. `features` and `decide` are model-free with budget +`{usd: 0, calls: 0}`, as `remember` and the loop steps are (`plane/tasks/loop.py:1-5`). `remember` ingests the new +`features` and `decision` rows. Labels are never computed in a Run. They are built on the host from ground-truth +files the sandbox never receives, so no task that produces a feature can read a label. + +New code: `src/openultrasast/learn/` (`schema.py`, `features.py`, `roles.py`, `labels.py`, `folds.py`, +`train.py`, `calibrate.py`, `evaluate.py`, `decide.py`, `experiments.py`, `sources.toml`), +`src/openultrasast/plane/tasks/{features,decide,roles}.py`, `ruleset/frameworks.toml`, +`ruleset/decision/`, and CLI groups `ousast learn ...` and `ousast plane experiment ...`. The reconciler is not +touched. Its 500-line budget stays (`tests/test_plane_reconciler.py`). + +## Components and Interfaces + +### 1. Candidate key and feature record (Requirements 1, 7.2) + +**Candidate.** A candidate is `(repo, pin, path, function, family)`. `repo` is `memory.repo_key` +(`plane/memory.py:69`), `function` is the enclosing declaration by the `repo-facts` patterns +(`plane/tasks/repo_facts.py:131`, `` outside any function), and `family` is a `families.toml` id. It is the +key that already joins `alert` and `verdict` rows (`improve/memory.py:7`: "repo, pin and `path::function`"). For a +delta the pin is the head, and the record carries `base` beside it. The key lives in the row **envelope**, for +joining labels and grouping folds. It is never a feature. + +A candidate exists when at least one instrument emitted a signal of that family in that function: a quick-rule +alert, an engine finding, a `roles` sink, or a verify candidate. A labelled positive with no candidate is a +**reach miss**. It counts against recall and is reported separately as `reach`, so the engine is never credited +for sites no instrument surfaced. + +**Delta mapping (Req 7.3).** For base..head, changed hunks (`git diff -U0 base head`) map to the head functions +that enclose a changed line. Functions whose engine operation `compare_evidence` marks `new`/`worsened` +(`push/policy.py:489`, `CandidateDelta.novelty`) are added. Each such function yields one candidate per family +that has a signal there. The base side of the same function supplies the `delta.*` features. A whole-repository +scan is the same with no base (`delta.*` is `not_applicable`). + +**Features.** The feature set is closed and listed in `learn/schema.py` as `FEATURES: tuple[FeatureSpec, ...]`. +Each spec has a name, a source instrument, a type (`bool`, `int` with a cap, `float`, or `enum` with a closed +vocabulary), a profile (`static` or `plane`), and whether it is a prior. There are no free-text values, no +identifiers and no raw counts of repository-specific tokens. + +| Feature | Source (file) | Type | Profile | +| --- | --- | --- | --- | +| `lang.*` | `preprocess.detect_language` | one-hot over the languages the engine or quick mode covers | static | +| `qr.enabled_hits`, `qr.shadow_hits` | quick scan (`plane/tasks/alerts.py:117`, `findings.quick_scan_findings`) | int, cap 20 | static | +| `qr.mechanism.` | the hit rules' `tags` mapped to the closed mechanism vocabulary (`benchmarks/pairs/mechanisms.toml` ids) | int, cap 10 | static | +| `qr.max_precision_estimate` | `PatternRule.precision_estimate` (`ruleset/store.py:33`) | float | static | +| `qr.prior_hits` | hits of rules tagged `framework`/`library` (section 2) | int, cap 10 | static, **prior** | +| `eng.findings` | engine findings in the function (`engine_alerts.engine_rows`, `plane/engine_alerts.py:160`; pre-push scan) | int, cap 10 | static | +| `eng.rung_max` | finding rung | enum `suspicion, model_entailed, entailed` | static | +| `eng.witness_steps_min` | witness path length | int, cap 50 | static | +| `eng.sink_kind` | sink entry id mapped to an operation kind | enum (sql, command, code_eval, deserialize, file_path, file_inclusion, outbound_request, redirect, html_output, other) | static | +| `eng.source_kind` | source entry kind | enum (request, server, raw_body, argv_stdin, inferred_role, prior, other) | static | +| `eng.sink_origin`, `eng.source_origin` | where the vocabulary entry came from | enum (language, inferred_wrapper, model_role, prior) | static | +| `eng.sanitizer_on_path` | discharge seen but not accepted | bool | static | +| `eng.completion` | questions completed / total for the file | float | static | +| `eng.degraded`, `eng.unstable` | degradations; present in only one of two runs | bool | static | +| `facts.callers` | `repo-facts` call sites (`repo_facts.py:151`, cap 40) | int, cap 40 | static | +| `facts.caller_files` | distinct caller files | int, cap 20 | static | +| `facts.entry_distance` | caller-graph hops to an entry point (`mapping.analyze_entry_points`, `mapping.py:123`) | int, cap 6 | static | +| `facts.is_global` | candidate is `` | bool | static | +| `facts.function_lines` | declaration span | int, cap 500 | static | +| `roles.sink_confidence`, `roles.source_in_function` | `roles` task (section 2) | float / bool | static (deterministic part), plane (model part) | +| `delta.novelty` | `CandidateDelta.novelty` or diff of base/head signals | enum `new, worsened, unchanged, unknown, not_applicable` | static | +| `delta.changed_lines` | changed lines inside the function | int, cap 200 | static | +| `delta.sanitizer_removed` | a sanitizer call present at base is absent at head | bool | static | +| `verify.flag_a`, `verify.flag_b` | pass verdicts (`units.jsonl`, `plane/tasks/verify.py`) | bool | plane | +| `verify.flag_c` | tie-break pass (asked only when a/b disagree) | bool | plane | +| `verify.votes` | flags over the passes asked | int 0-3 | plane | +| `verify.turns_mean`, `verify.site_offset` | tool turns; lines between the model's reported site and the candidate line | float / int cap 200 | plane | +| `agree.final` | `agreed, disputed, rejected` (`plane/tasks/remember.py:73`) | enum | plane | +| `ms.flagged`, `ms.operation` | model sink classification (`benchmarks/independent/model_sinks.py`, moved into `roles`) | bool / enum | plane | + +**Deliberately excluded from features** (each is label information or an identity): + +- `site_match` (`plane/tasks/agree.py:200`) and `in_fix_range` (`plane/tasks/alerts.py:137`). Both are computed + from declared sites and fix ranges. +- Every fixed-pin alert. A deployed scan has no fixed pin. +- Declared-site counts and `case.json`. +- Costs (`usd`), which vary with provider pricing, not with the code. +- Rule ids, paths, file and function names, commit ids and repository names. + +Rule ids enter only through closed mechanism buckets and the prior count. A rule written after one case's miss +would otherwise identify that case. + +**Missing is not negative (Req 1.2).** Each record carries `instruments: {name: {state, version}}` with `state` in +`ran | none | failed | not_applicable`, and `x: {feature: value | null}`. `none` means no coverage for the language +(`plane/tasks/alerts.py` `coverage`, `remember.coverage_rows`, `plane/tasks/remember.py:79-96`). `failed` means +the instrument broke or did not read its input: the engine-alert rule "no result file, no `read N files` line, or +`questions == 0` fails the case loudly" (`plane/engine_alerts.py` docstring) maps here, never to zero findings. +`not_applicable` covers two cases: `delta.*` on a whole-repository scan, and `facts.*` on a pair excerpt, where no +repository exists. Every feature of an instrument whose state is not `ran` is `null`. At training time a `null` +becomes 0 plus a per-instrument indicator `missing.` (one indicator per instrument, not per feature, +to keep the parameter count down). + +**Profiles.** `static` is what `pre-push` and `scan` can compute locally without a model: quick rules, engine, +facts, deterministic roles and delta. `plane` adds verify, agree and model roles. A model is trained per +(family, profile). A `static` model never sees verify features, so it cannot learn to lean on an instrument that +is absent in deployment. + +**Enforcement (Req 7.2).** The builder emits only `FEATURES` names. `validate_record` rejects any other key, any +string value outside an enum's vocabulary, and any number outside its type. Tests: + +- `test_learn_schema.py::test_features_are_allow_listed`: every emitted key is in `FEATURES`, and no spec's type + is free text. +- `::test_record_is_name_invariant`: two fixture repositories that differ only in repository name, paths, file + names, function names and commit ids give byte-identical `x`. +- `::test_label_fields_never_reach_features`: `site_match`, `in_fix_range`, fixed-pin rows and `case.json` + fields in the inputs change nothing in `x`. +- `::test_missing_is_not_zero`: `coverage: none` gives `null` + `state: none`, and a covered language with no hit + gives `0` + `state: ran`. + +**Versioning.** `schema_version` (an integer, bumped on any spec change) and `feature_set_digest` (sha256 of the +canonical `FEATURES` list). `instruments.*.version` records the ruleset digest, the engine image id +(`engine_alerts._image_id`, `plane/engine_alerts.py:177`), the runner image digest (`memory.image_digest`), the +verify model id and a prompt digest, and the `roles` model id. That makes every record rebuildable (Req 1.3). + +**Where records live.** A new memory row kind `features` (`KINDS` at `plane/memory.py:52`), per repository and +pin like every other row: +`{candidate, family, language, profile, schema_version, feature_set_digest, x, instruments, unit: "pin"|"delta", +base?}`. The row id is `row_id("features", run, task, candidate+family)`. + +**Who builds them.** + +- *Plane*: task `features` (`plane/tasks/features.py`), no Model, one per case, after `final` and `alerts` (and + `roles` when present). Inputs are `facts.json`, the pass `units.jsonl`, `agreed.json` (with `site_match` dropped + on read), `alerts.jsonl` (vulnerable pin only, `in_fix_range` dropped), the alerts `summary.json` coverage, and + `roles.json`. It writes `features.jsonl`, which `remember` turns into rows (one more input in + `remember.rows_for`, `plane/tasks/remember.py:99`). +- *Host*: `learn.features.build_for_scan(root, findings, engine_result, *, base=None)`, called by `scan` and + `pre-push` (section 8) and by `ousast learn features ` to build records for the pair corpus and the + development corpus without a Run. + +### 2. Framework knowledge: tagged priors and inferred roles (Requirement 8) + +**Rule.** Hand-written knowledge stays at the language level: superglobals, built-ins, process execution, raw +database drivers (DB-API `execute`, `mysqli_*`, `pg_*`, JDBC), deserialisers and the standard library. +Framework and application roles (request objects, query builders, hooks, sanitisers, guards) are either +**inferred per repository** or kept as **tagged priors**, which are features that are off by default. + +**Tagging.** Every `[[source]]`, `[[sink]]`, `[[sanitizer]]` and `[[dispatch]]` entry in `ruleset/semantic/*.toml` +and every quick `[[rule]]` gains an optional `framework = ""` or `library = ""`. An entry that mixes +language-level and framework calls is split, so a tag always covers a whole entry. `ruleset/frameworks.toml` +lists the ids and, per id, a lexicon of symbols. The lexicon has two uses only: the lint test, which fails on an +untagged entry containing a lexicon symbol, and repository stratification (below). It never feeds detection. +Loaders take `priors: Literal["off", "all"] | frozenset[str]`. The default for the engine model is `off`. Today's +`scan` behaviour (fallback, Req 6.2) keeps `all`. + +Current framework-specific entries and their disposition: + +| Entry | Where | Tag / action | +| --- | --- | --- | +| `wpdb` sink (`$wpdb->get_var`, ...) | `ruleset/semantic/php.toml:47-58` | `framework = "wordpress"` | +| `unserialize` sink: `unserialize` + `maybe_unserialize` | `php.toml:136-138` | split: `unserialize` language; `maybe_unserialize` wordpress | +| `sql_escape`: `mysqli_real_escape_string`, ..., `esc_sql` | `php.toml:169-170` | split: `esc_sql` wordpress | +| `wpdb_prepare` sanitizer | `php.toml:194-196` | wordpress | +| `coerce`: `intval`, ..., `absint` | `php.toml:207-208` | split: `absint` wordpress | +| `wordpress_hooks` dispatch (`add_action`, `apply_filters`, ...) | `php.toml:225-242` | wordpress | +| layout `php_composer_and_wordpress` | `php.toml:244-247` | product filter (vendored/test paths), not detection; renamed `php_layout`, untagged | +| rule `php-wpdb-sql-composition` | `ruleset/php/rules.toml:20-29` | wordpress | +| rule `php-unserialize` (pattern includes `maybe_unserialize`) | `php/rules.toml:114-125` | split into `php-unserialize` (language) and `php-maybe-unserialize` (wordpress) | +| Flask `request` source (`request.args`, `.form`, `.json`, ...) | `ruleset/semantic/python.toml:5-17` | flask | +| `django-request` source | `python.toml:24-30` | django | +| `orm.raw` (`objects.raw`, `RawSQL`, `extra`; `text`, `from_statement`) | `python.toml:102-104` | split: django / sqlalchemy (library) | +| `path.open` (`open`, `os.*`, `shutil.*`; `send_file`, `send_from_directory`; `FileResponse`) | `python.toml:110-112` | split: language / flask / django | +| `html.unescaped` (`render_template_string`, `Markup`; `mark_safe`, `format_html`, `HttpResponse`) | `python.toml:119-121` | flask / django | +| `outbound.request` (`requests.*`, `httpx.*`; `urlopen`, `urlretrieve`) | `python.toml:127-129` | split: library requests/httpx / language | +| `redirect` (`redirect`, `HttpResponseRedirect`, `url_for`) | `python.toml:134-136` | flask / django | +| `url_allowlist` (`url_has_allowed_host_and_scheme`, `is_safe_url`) | `python.toml:145-146` | django | +| `pickle.loads` entry incl. `yaml.load` | `python.toml:55-57` | split: `yaml.load` library pyyaml | +| Express `req` source (`req.query`, `req.body`, ...) | `ruleset/semantic/javascript.toml:5-6` | express | +| `html.response` (`res.send`; `res.write`/`res.end`) | `javascript.toml:80-82` | split: express / language (Node core) | +| `path.open` (`sendFile`; `readFile`, ...) | `javascript.toml:87-89` | split: express / language | +| `redirect` (`redirect`; `location.*`) | `javascript.toml:129-131` | split: express / language (DOM) | +| `prototype.merge`, `outbound.request`, `html.escape` | `javascript.toml:57-59, 109-111, 148-149` | library (lodash, axios/needle, escape-html/DOMPurify) | +| `servlet` source, `response.write` | `ruleset/semantic/java.toml:5-6, 47-49` | `jakarta-servlet` | +| `outbound.request` (`getForObject`, `exchange`; `sendRedirect`; `openConnection`) | `java.toml:54-56` | split: spring / jakarta-servlet / language | +| `java.escape` (`ESAPI.encoder`, `StringEscapeUtils`; `getCanonicalPath`, `normalize`) | `java.toml:61-62` | split: library / language | +| hook-as-source rule | `cpg/queries/taint.sc:918-949`, `semantic/facts.py:73-90` | general code driven by the dispatch fact ("The registry's vocabulary arrives as a FACT", `taint.sc:921`); with priors off it is inert unless `roles` infers a registry | + +The maintainer's term counts (php.toml 24, php rules 9, python 16, javascript 5, java 3, 15 WordPress mentions +in `semantic/*.py` and `cpg/queries/*.sc`) are covered by these entries. The remaining mentions in `.sc` and +`.py` are comments (`taint.sc:161, 249, 322-327, 588, 911, 1296`; `dominance.sc:130`; `census.sc:9`; +`facts.py:345`). The tagging commit's lint test, not this table, is the authority. + +**Role inference (Req 8.1).** Two parts, both producing per-repository vocabulary with provenance: + +1. *Wrapper inference* (deterministic, `learn/roles.py`, no model). A function whose parameter reaches, within its + body, a call already in the role set becomes a derived sink or sanitizer of the same operation kind, with + `depth + 1`. The role set starts from language-level entries plus any inferred roles. Derivation iterates over + the `repo-facts` caller map to depth 3. When dependency source is present (`vendor/`, `node_modules/`, + site-packages in the checkout), it is scanned the same way, so `$wpdb->query` becomes a sink because + WordPress's own `wpdb::query` reaches `mysqli_query`, not because a table says so. That holds only when WordPress + core is present. A plugin checkout usually lacks it, which is why part 2 exists. +2. *Model roles* (plane task `roles`, Model-bound, `deepseek-flash`). The source-only classification of + `benchmarks/independent/model_sinks.py` moves into the package: per product file chunk, which functions and + call sites perform an operation of a closed kind on non-constant data. The v2 record: declared site among the + candidates in 15/17 cases, fix or site file in 16/17, $14.42 for 17 repositories + (`selection-v2-model-sinks.json`). The output extends to all roles: `sink`, `source`, `sanitizer`, `guard`, + `dispatch` (a register/apply pair such as a hook registry), with `operation`, line and a one-line evidence + quote. A chunk whose call fails is `unclassified`, never "no roles" (the `model_sinks.py` rule). In `pre-push` + only the changed files and the definitions they call are classified. + +Roles reach the engine as a per-scan vocabulary overlay in the shape of `ruleset/semantic/.toml` entries +with `origin = "inferred_wrapper" | "model_role"`. The engine already takes vocabulary as facts +(`taint.sc:921`). They also reach the feature record as `eng.*_origin` and `roles.*`. Roles are signals. A +model-labelled sink alone is not a finding. + +**Stratification metadata.** Each repository gets `frameworks: [ids]` from its dependency manifests +(`composer.json`, a WordPress plugin header, `requirements*.txt`/`pyproject.toml`, `package.json`, +`pom.xml`/`build.gradle`) matched against `frameworks.toml`. This is envelope metadata for folds and reports, +never a feature. + +**Evaluation (Req 8.2-8.4).** The engine is always trained and reported with `priors=off`. A prior group (one +framework id) is kept only through an engine-level A/B (section 6): priors on vs off, paired on the same outer +folds, metric recall at the BLOCK operating point. It is adopted only if the repository-bootstrap CI excludes zero. +Every report carries, per family, the metric with priors off, with the adopted priors, and the difference: the +share of the result that depends on priors. Leave-one-framework-out folds (section 4) give the "framework unseen +in training" row. + +### 3. Label builder (Requirement 2) + +`ousast learn labels` (`learn/labels.py`) runs on the host only. It writes `label` rows (a new kind) and +`labels/.jsonl` snapshots. + +**Sources are an allow-list, fail-closed.** `learn/sources.toml` names each usable source, the file that proves +its evaluation was recorded, and the date. A population not listed cannot produce a label. The builder never +opens a population file that is not listed, so population v3 (`population-v3-php.toml`, reserved) is excluded +without being read. It becomes usable only when a later commit adds it after the v3 result is recorded +(Req 2.3). + +| Source | Evaluation record | Status | +| --- | --- | --- | +| `population-v1` | `benchmarks/independent/results-v1.json` | spent (retuned on, `population-v2.toml:9-11`) | +| `population-v2` | `results-v2.json`, `results-v2-hunter.json`, `plane-increment-2.json` | spent for tuning (`model_sinks.py`: "v2 is spent for tuning"); its fold is marked *design-informed* (section 4) | +| `dev-php` | `benchmarks/push/inputs.json` (pmpro introduce/repair/benign), MediaWiki and WP Statistics checkouts | development | +| `pairs` | `benchmarks/pairs/**/catalog.toml` via `load_pair_catalog` (`pairs.py:229`) | development; catalog split recorded, not used for folds | +| `adjudications` | `results-v1.json` `precision_review`, `results-v2.json` / `results-v2-hunter.json` `precision_sample`, `verifier-check-2026-09-29.json` | recorded verdicts | + +**Rules per source.** + +- *Populations.* Positive: a candidate on the vulnerable pin that matches the case. Matching uses v2 declared sites + through `agree.matches_declared` (`plane/tasks/agree.py:45`, the parity-proven copy of `evaluate.matches_v2`). + v1 has no declared sites (0 in `population-v1.toml`), so it uses the fix-range rule `evaluate.matches` + (`benchmarks/independent/evaluate.py:162`). Negative: the same `(path, function)` on the fixed pin when the fix + changed that function (inside the new-side `fixed_ranges`, `plane/generate.py:106, 227-229`). A function the fix + did not touch is the same code at both pins and is not a negative. Benign: a candidate in a function changed + between the benign base and tip pins that has a finding at the tip and none at the base. It is negative + **by protocol assumption** (protocol v2 "Benign false alert"), with source `benign_control`. A candidate on a + vulnerable pin that matches no site is **unlabelled**, never negative. +- *Pairs.* Positive: the labelled function (`ExpectedFinding.function`/`line`, `benchmark.py:20-30`; 407 of 452 + labels name a function) on the vulnerable side. Negative: the same function on the fixed side. Pairs marked + `unscorable` (identical twins, known limits, `pairs.py:130`; 7 today) give no labels. Tier `title` + (`agent-vfc`, 59 pairs, labels read from advisory titles; labelled recall 3.6% in the baseline) is excluded by + default and used only as a sensitivity arm. +- *Deltas.* Every population case and every pair also gives delta units. fixed..vulnerable is the introducing + direction, and its labelled function is a positive delta candidate. vulnerable..fixed is a repair, whose + candidates are negative for BLOCK. benign base..tip is a benign push. These are the primary units for the + safety net (Req 7.3). +- *Benign history (optional, flagged).* Non-security commits from the history of each training repository at + least 90 days away from any fix date give benign deltas at no model cost. They are negative by assumption, carry + source `benign_history` and a weight of 0.5, and every metric is reported with and without them. This is the + only cheap way to estimate false blocks per benign push. +- *Adjudications.* `True` and `True, privileged` are positive (privileged is kept as a label attribute), and + `False, *` is negative. Each keeps the adjudicator (maintainer, or a judge task with its evidence) and the + recorded text. +- A plane verdict (`agreed`, `rejected`, `disputed`) is never a label (Req 2.2). It enters only as `agree.final`. + +**Label row.** `{candidate, family, label: 1|0, source, source_ref (case id / pair name / adjudication index), +unit: pin|delta, direction?, weight, split (catalog split or population split), group (repository group key), +frameworks, created, evidence}`. `group` is the normalised repository, the lower-cased `owner/name` as the guard +uses (`improve/memory.py:89-92`). Candidates that share an advisory id (CVE/GHSA) across corpora are merged into +one group, so a repository mirrored under two names cannot straddle folds. + +**What exists today** (counted 2026-09-30 with a scratch script over `load_pair_catalog` + +`split_by_repository`; the label-snapshot commit makes the count reproducible): + +| Family | Pair repos (positive) | Pairs | Population repos (v1 + v2 + dev) | Adjudicated | Per-family model? | +| --- | --- | --- | --- | --- | --- | +| injection | 43 | 56 | 7 + 10 + 3 | most of the 43 (per-family split not counted) | yes | +| untrusted_destination | 41 | 41 | 3 + 6 + 0 | some | yes | +| path | 24 | 24 | 0 | 0 | yes (Python-heavy) | +| config_secrets | 23 | 24 | 1 + 1 | some | yes, borderline | +| deserialization | 23 | 23 | 0 | 0 | yes (all Python) | +| access_control | 21 | 24 | 0 | 0 | yes, borderline | +| output_encoding | 17 | 58 (40 from one repository, OWASP BenchmarkJava) | 0 | 0 | pooled | +| memory | 7 | 176 (132 openssl, 40 curl) | 0 | 0 | **insufficient** | +| prototype | 6 | 7 | 0 | 0 | **insufficient** | + +Totals: 427 pairs and 452 labels in 146 repository groups (142 named). Tiers: advisory 250, seeded 115, title 59, +reviewed 3. Provenance: human 237, agent 139, synthetic 51. 59 repositories straddle the catalog's train/holdout +split (`split_by_repository` report). The engine ignores that split for folds and records it on the label. + +By language, the strata with at least 10 positive repositories are Python (untrusted_destination 30, injection +23, deserialization 23, path 17, access_control 15, output_encoding 15) and JavaScript injection (16). **PHP has +no pairs.** Its labels are v1 (4 repositories), v2 (6) and dev-php (3): about 9 injection and 4 +untrusted_destination repositories. + +Plane-profile rows (verify/agree present) exist for 43 candidates in 15 repositories, all v2. The adjudicated +rows are mostly negative: v1 1/12 true, v2 0/17, v2 hunter 2/14 (`results-v1.json` `summary.precision`, +`results-v2*.json` `precision`). + +What this means: + +- The `static` profile can be trained for six families, pooled for output_encoding, and not at all for memory or + prototype. +- The `plane` profile cannot be trained until verify has run on labelled units. The harvest Run in the commit + sequence (step 5) runs passes a and b on the 78 scorable, non-title pairs in the verify families (injection, + untrusted_destination, config_secrets; `verify.OPERATIONS`, `plane/tasks/verify.py:38-50`) on both sides, and on + the development cases. That is about 312 + 110 hunts at $0.014 each (validation-46: $0.8567 over 61 hunts), + **about $6**. +- PHP is reported as its own stratum only where it reaches 10 positive repositories. Today that is injection at + the boundary, and untrusted_destination is "insufficient data". + +### 4. The decision engine (Requirements 3, 7.1, 7.3, 7.4) + +**Model family.** L2-regularised logistic regression on standardised features plus the instrument-missing +indicators. There are about 40 parameters per model. Reasons: + +- With 20-60 repositories per family, a shallow gradient-boosted model fits interactions the folds cannot + validate and calibrates worse on small data. +- A linear model's contributions are its explanation (Req 3.5). +- Inference is a dot product, so `scan` and `pre-push` need no dependency. + +Gradient-boosted trees stay a possible future arm of an engine-level experiment, not a default. + +**Dependency trade-off.** + +- *scikit-learn as an optional extra* (`openultrasast[learn]`) brings numpy, scipy and joblib, about 100 MB in + the runner image, and a second numerical stack whose version changes model bytes. +- *Pure Python (chosen)*: `learn/train.py` fits by Newton/IRLS with a dense solve of a matrix of about 40x40 + (Gaussian elimination with partial pivoting). At about 2,000 rows that is milliseconds. It is deterministic with + no seeds inside the fit, and it is about 200 lines. + +scikit-learn is a **test-only** extra: `test_learn_train.py::test_matches_sklearn` (skipped unless installed) +checks coefficients against `LogisticRegression(penalty="l2", solver="lbfgs")` to 1e-6. Training and inference +stay pure Python in core. + +**Per family vs pooled.** A (family, profile) model is trained when the training data holds at least 20 +repositories with a positive and 20 with a negative in that family. A pooled model (all families, with a family +one-hot) serves families with 10-19. Below 10 the family is "insufficient data": no engine decision, and the +fallback applies (section 8). The same 10-repository floor applies to any reported stratum (family x language, +family x framework). + +**Weights.** One repository must not dominate (openssl has 132 pairs, OWASP BenchmarkJava 43). A row weighs +`w_source * min(1, 5 / n_rows(repo, family, label))`, so each repository contributes at most 5 effective positives +and 5 effective negatives per family. `benign_history` has `w_source = 0.5`. + +**Folds (Req 3.3, 7.1).** Every split is by `group` (repository), across all corpora at once. A repository in an +evaluation fold contributes no row, from any corpus, to training. + +- *Outer*: grouped K-fold with K = min(10, repositories in the family), stratified by family and label, repeated 5 + times with seeds 0-4. This gives the reported out-of-repository numbers. +- *Leave-one-source-out*: each population (v1, v2, dev-php) and each pair slice (vfc, vibe-py, owasp, vfc-js, + agent-vfc, sast, github, local) held out whole, with its repositories removed from training. This is the + "population-level fold" of Req 3.3 and the data behind the learning curve over the number of sources. +- *Leave-one-framework-out*: all repositories tagged with framework F held out, including WordPress, Django, + Flask and Express wherever that framework has at least 10 repositories. This gives the "framework unseen" row of + Req 8.3. +- *Inner* (inside each outer training set): grouped 5-fold. It selects lambda from {0.01, 0.1, 1, 10} by grouped + log-loss, produces the out-of-fold scores for calibration, and picks thresholds. Nothing is tuned on an outer + fold. +- The v2 fold is marked *design-informed*: the tie-break and the callers line were chosen after seeing v2 + (`plane-increment-2.json` `reading`). Plane-profile numbers are reported with and without it. + +**Calibration (Req 3.1, 7.4).** Platt scaling (a two-parameter logistic on the model's logit) is fit on the inner +out-of-fold logits. Isotonic regression is not used: too few positives per family. The check is per outer fold +and pooled: + +- a reliability table with 5 quantile bins; +- calibration slope and intercept (a logistic of the label on the calibrated logit), with repository-bootstrap + 95% CIs; +- the expected calibration error. + +Calibration **holds** for a family when the slope CI contains 1, the intercept CI contains 0, and the pooled +out-of-fold expected calibration error is at most 0.05. If it fails, the model is recalibrated once on grouped +out-of-fold scores of the full training set. If it still fails, the family gets **ADVISORY only**, the artifact +records `block.offered = false, reason = "calibration"`, and the report says so. + +**Prevalence.** Training mixes are enriched: pairs are 1:1, and validation-46 is 19/43. In a push, real +vulnerabilities are rare: v2 has 25 declared sites among 4,757 model-sink candidates. So the probability is +reported as calibrated **on the labelled mix**. The safety-net figures that do not depend on the mix are primary: + +- recall at an operating point, and +- false BLOCKs per 100 benign deltas (population benign pins, pmpro-benign, benign history). + +Precision at BLOCK is also reported at the benign-delta false-alarm rate, as `precision(pi) = pi*TPR / +(pi*TPR + (1-pi)*FPR)` for pi in {0.01, 0.05}. + +**Operating points (Req 3.4, 7.3).** Both are chosen on inner out-of-fold scores only: + +- **BLOCK**: the lowest threshold whose inner-fold precision has a Wilson 95% lower bound of at least 0.95, among + delta units. If none exists (expected today for most families), `block.offered = false, reason = + "precision_unreachable"`. +- **ADVISORY**: the highest threshold whose inner-fold recall is at least 0.90 (the M4 recall target). Its + precision is reported, not assumed. +- **DROP**: below ADVISORY. + +Both points are reported on outer folds with recall, precision and false BLOCKs per 100 benign deltas. Each has a +repository-cluster bootstrap 95% interval (2,000 resamples of repositories) and a Wilson interval as a +cross-check when each repository gives one unit. + +**Explanations (Req 3.5).** A feature's contribution is `coef_i * (x_i - mean_i) / sd_i`, and a missing +instrument contributes through its indicator. The top 5 by absolute contribution, with sign and the feature's +display value, are stored in the `decision` row and on the finding. Missing-instrument indicators are shown as, +for example, "engine: no coverage". + +### 5. Model artifacts and versioning (Requirement 3.2) + +A trained model is one canonical JSON (sorted keys, floats rendered with `repr`), stored as a content-addressed +blob `models/.json`. The store gains `put_blob(prefix, data) -> sha` and `get_blob(prefix, sha)`, +generalising `put_facts`/`get_facts` (`plane/memory.py:219-255`). Both backends get it unchanged. On MinIO the +object has tags `kind=model`, `family` and `profile`, and bucket versioning records the version id. +`models/index.jsonl` lists every version. `models/adopted.json` maps (family, profile) to a sha: a single +overwritten object (a new version under MinIO versioning, and history kept in `index.jsonl` on FileStore). + +``` +{"model_id": "", "schema_version": 1, "feature_set_digest": "...", "family": "injection" | "pooled", + "profile": "static" | "plane", "kind": "logistic-l2", "lambda": 1.0, "features": [{"name", "mean", "sd"}], + "coef": {...}, "intercept": ..., "calibration": {"method": "platt", "a": ..., "b": ..., "check": {...}}, + "operating_points": {"block": {"threshold", "offered", "reason", "target_precision": 0.95}, + "advisory": {"threshold", "target_recall": 0.90}}, + "priors": "off" | ["wordpress", ...], "roles": "off" | "deterministic" | "model", + "data": {"store_index_digest", "label_set_digest", "sources": [...], "excluded": {...}, "rows", "repositories"}, + "evaluation": {"folds_digest", "outer": {...}, "leave_one_source_out": {...}, "leave_one_framework_out": {...}, + "strata": {...}, "curves": "curves/.json"}, + "instruments": {"ruleset_digest", "engine_image", "runner_image", "verify_model", "prompt_digest", "roles_model"}, + "seed": 0, "code_commit": "", "created": ""} +``` + +`store_index_digest` is `improve.memory.index_digest` (`improve/memory.py:351`), and `label_set_digest` is the +sha256 of the sorted label rows used. Training reads rows through the store, so the same digests with the same +code commit give the same bytes (Req 3.2). `test_learn_train.py::test_reproducible` asserts this on both +backends, with the MinIO contract test gated as today. + +**Adoption** is a maintainer commit. `ousast learn adopt ` writes `models/adopted.json` and exports the +artifact to `src/openultrasast/ruleset/decision/-.json`, the copy that ships to users who have no +store. The mirror is the existing ledger rule: "adopting an accepted ledger stays a maintainer commit" +(`plane/tasks/loop.py:22-23`). A model whose feature set enables a prior group, or changes `roles`, can be adopted +only with a recorded `adopt` result of the engine-level experiment that justified it (section 6). + +### 6. A/B experiments (Requirement 4) + +**Manifest.** `plane/experiments/.yaml`, a companion file, is referenced by the Run annotation +`openultrasast.io/experiment: `. Each arm's Tasks carry `openultrasast.io/arm: A|B`. Annotations are stripped +before ax (`plane/manifests.py:4-6`), as `memory-key` is. + +``` +id: exp-001-known-callers +hypothesis: removing the known-callers line changes the rate at which verify flags a true site +arms: {A: {task_env: {}}, B: {task_env: {OUSAST_VERIFY_CALLERS: "off"}}} # everything else identical +unit: candidate # candidate | file | repository +pairing: paired # every unit in both arms +cluster: repository # resampling unit for intervals +order: {randomise: per_case, seed: 1} # which arm runs first per case +metric: {primary: "flag_any_pass_rate(positive units)", secondary: ["flag_any_pass_rate(fixed-pin negatives)", "usd_per_candidate"]} +test: {primary: "cluster_bootstrap_paired_diff", secondary: "exact_mcnemar", alpha: 0.05, sides: 2} +mde: 0.15 +stopping: {looks: [0.5, 1.0], boundary: "obrien_fleming"} +budget_usd: {total: 15, per_arm: 7.5} +units_file: plane/experiments/exp-001-known-callers.units.jsonl # frozen list, digest recorded +``` + +`ousast plane experiment register ` checks the manifest, freezes the units file, and records the sha256 of +both as an `experiment` row (a new kind, with repo `experiments/`). It refuses unless both files are committed +at `HEAD`. `ousast plane run` refuses a Run with an experiment annotation whose manifest or units digest differs +from the registered one. + +**Randomisation and pairing.** Paired is the default: every unit runs in both arms, and a seeded per-case coin +decides which arm's tasks run first, so provider drift over a run does not align with arms. Only when a variant +cannot run on the same unit (for example, a change to candidate generation itself) are repositories assigned to +arms by a seeded, family-stratified shuffle. Assignment is recorded in the units file. + +**Outcomes.** The `experiment-outcome` task (no model), after the arms, writes one `arm_outcome` row per (unit, +arm): outcome, usd from `unit_cost`, and pass details. Spend per arm is also in the plane's attribution table +(`ousast plane status`). + +**Statistics.** + +- *Primary*: the paired difference B - A in the per-unit outcome, with a 95% CI from a repository-cluster + bootstrap (10,000 resamples of repositories, with the units inside each). Units in one repository are + correlated, so a unit-level test would overstate certainty. +- *Secondary*: the exact McNemar test on discordant units (a binomial test with p = 0.5), reported beside it. +- *Sequential*: two looks, at half and all of the repositories (in a pre-shuffled order), with O'Brien-Fleming + boundaries (|z| >= 2.797 at the first look, 1.977 at the final). Stopping for futility is not binding. +- *Decision*: **adopt B** if the final CI excludes 0 in B's favour, and **reject B** if it excludes 0 against it. + For a cost-only change, **equivalent** means the TOST 90% CI of the detection difference lies within +-0.05 and + the cost CI shows a saving. Otherwise **inconclusive**, and the incumbent stays. +- *Engine-level experiments* (priors on/off, roles, feature groups, model variants) cost no model spend. The arms + are two training configurations evaluated on the same outer folds, and the unit is the repository. The metric is + recall at BLOCK (or at ADVISORY where BLOCK is not offered), plus false BLOCKs per 100 benign deltas. + +**Where results land.** An `experiment_result` row, and `benchmarks/experiments//result.json` (arms, units, +the estimates at each look, decision, spend, manifest digest, code commit) in the same commit that records the +decision. Adoption of a winning pipeline variant is a maintainer commit that changes the default (a Task +template's env, or a `[models]` value) and cites the result's digest. The memory proposer's M1/M2 status edits +(`improve/memory.py`) become one variant family: a proposed status change runs as an engine-level experiment +instead of going straight to the benchmark gate. + +**First experiment, fully specified (Req 4.4).** The question: `plane-increment-2.json` records declared sites +flagged in any pass as 17 for the plane vs 19 for the reference, of 20. The plane added the known-callers line +(`plane/tasks/verify.py:1-9, 164`, `CALLERS_CAP = 8` at :34), and the comparison is confounded (other changes, +stochastic passes, v2 is where it was seen). + +- **Arms.** A is `verify` as today. B is `verify` with `OUSAST_VERIFY_CALLERS=off`, a new switch (about 3 lines) + that makes `callers_line` return empty. Same model (`deepseek-flash`), `PER_HUNT = 6`, `MAX_STEPS = 6`, same + candidates, same `facts.json`, same runner image digest, two passes (a, b) per arm. +- **Units: the development corpus, not v2** (v2 generated the hypothesis): + - population v1, 11 cases, whose engine scans are recorded under `~/ousast-results/independent-v1/scans/` (44 + scans); + - pmpro, MediaWiki, WP Statistics (dev-php); + - the 27 full-repository pointer pairs in the verify families (`recipe` pairs of vibe-py and agent-vfc, + tier != title). + + That is 41 repositories. No development repository has recorded verify candidates today. They are generated + model-free and frozen in the units file: per pin, the fix-changed functions, plus the top engine- and + quick-alert functions, up to 8 candidates per pin, on the vulnerable and the fixed pin. Only units where arm A's + callers line is non-empty are *treated*. The primary analysis is restricted to them, and the rest are reported. +- **Metric.** Primary: the per-positive-unit rate of "flagged in any pass" (the 17-vs-19 metric), with positives + from the label builder (fix-range functions on the vulnerable pin). Secondary: the flag rate on fixed-pin + negatives (false flags), and usd per candidate. +- **Sample size and MDE.** About 55 positive units across 41 repositories (about 28 dev + 27 pointer pairs). With + a discordance rate of about 0.15, alpha 0.05 and power 0.8, the paired MDE is `(1.96 + 0.84) * sqrt(0.15) / + sqrt(55)`, about **0.15**. The observed gap of 0.10 needs about 115 positive units, so an inconclusive result is + the likely and honest outcome. It is registered as such in advance, and a follow-up needs new units, not a + continuation. +- **Budget.** Hunts per pass per arm: 14 dev repositories x 2 pins x up to 3 files (84) + 27 pairs x 2 sides (54) + = 138. Four arm-passes give 552 hunts x $0.014, **about $7.7 expected**. The ceiling is **$15 total, $7.5 per + arm**, enforced by the task budgets (`plane/budget.py`). Candidate generation reuses the recorded v1 engine + scans. For dev-php it runs the engine once per pin on the host, one container at a time, from a frozen source + export. + +### 7. Extrapolation and the one-time v3 check (Requirement 5) + +**Learning curves (5.1).** `ousast learn curve --family F --profile P` works in two directions: + +- *By repositories*: training on random subsets of 20, 40, 60, 80 and 100% of the outer-training repositories, + 20 subsets per size, each evaluated on the same outer folds. +- *By sources*: training on 1..S of the leave-one-source-out sources, in all orders, capped at 50. + +Per point: recall at ADVISORY, recall at BLOCK (where offered), PR-AUC, Brier score, and false BLOCKs per 100 +benign deltas. Bands are the 2.5/97.5% quantiles over subsets x repository bootstrap. An inverse power law +`m(n) = a - b * n^-c` is fit to each curve. The projection to 1.5x and 2x the repositories is reported only as a +band, and only where the fit's own bootstrap CI for `c` excludes 0. + +**Expected detection on unseen code (5.2).** Per family and per language (and per framework, Req 8.3): the +outer-fold and leave-one-source-out intervals at both operating points, with priors off and with adopted priors. +A stratum with fewer than 10 positive repositories or 10 negative repositories, or whose 95% interval on recall is +wider than 0.5, is printed as **"insufficient data (n repositories)"**, with no number. Today that applies to +memory, prototype, PHP untrusted_destination, and every plane-profile family until the harvest Run. + +**The v3 check (5.3), exactly once.** + +1. *Before.* The engine and every operating point are adopted, and population v3 is still reserved and unread by + this spec's code (it is absent from `learn/sources.toml`). `ousast learn predict --population-name + population-v3-php` writes `benchmarks/independent/prediction-v3.json` and it is committed: + - the adopted model ids and shas per (family, profile); + - `feature_set_digest`, priors and roles settings; + - the operating points; + - the frozen analyzer source hash (`benchmarks/push/freeze_source.sh`); + - for each family v3 may contain, the predicted recall and precision intervals from the PHP stratum where + sufficient, else the pooled-language interval with the label "no PHP stratum"; + - the predicted false BLOCKs per 100 benign deltas; + - the reference to `protocol-v3.md`, written by the population's owner, not by this spec. + + Only the population's name is needed; its file is not opened. +2. *Run.* v3 is scored under protocol v3, once, by its owner's runner, with the engine's decisions added to the + recorded outputs. +3. *After.* `benchmarks/independent/result-v3-engine.json` holds, per family, the measured point and interval + beside the predicted interval, `inside | outside`, the calibration check on v3, and the reach misses. v3 is + then added to `learn/sources.toml` as spent, and the next model may train on it. Its numbers never again + qualify anything. + +### 8. Integration (Requirements 6, 7.3, 7.5) + +**Lookup.** The model comes from, in order: + +1. `[decision] model` in the config (a path); +2. the store's `models/adopted.json` when `OUSAST_MEMORY` is set; +3. the packaged `ruleset/decision/-.json`; +4. none. + +`learn/decide.py` is pure Python: `load_models(...) -> Models` and `decide(record, models) -> Decision(p, +operating_point, model_id, top, offered)`. + +**`ousast scan`.** A `decide` stage runs after the findings are complete and before `findings.json` is written +(`cli.py:517-520`). It builds `static` records with `build_for_scan` from the quick findings, the engine result, +`analyze_entry_points` and the `repo-facts` functions (source-only), plus deterministic roles. It scores each +candidate and attaches `decision` to the finding: `{p, operating_point: block|advisory|drop, model_id, +top_features, offered}`. Candidates at or above ADVISORY are reported. Dropped ones go to `dropped_by_engine.json`, +beside `shadow_findings.json`. Shadow-status findings enter as candidates too, so a rule's enabled/shadow status +changes the report only if the decision changes (Req 6.3). The shadow split at `cli.py:503-508` applies only in +the fallback. `--fail-on blocked` is a new choice (exit 1 on any BLOCK), and `findings` keeps its meaning. +Markdown, SARIF (`properties.decision`) and `manifest.json` (`decision: {model_ids, fallback, reason}`) carry the +fields. + +**`ousast pre-push`.** Candidates are the `CandidateDelta`s from `compare_evidence`, enriched with quick-rule +hits and deterministic roles on the changed functions of head and base. `admit_candidates` (`push/policy.py:879`) +gains `engine: Models | None`: + +- With a model, the capability-eligibility reasons `capability_unavailable`, `capability_ambiguous`, + `capability_disabled`, `capability_unevaluated` and `operation_semantics_mismatch` are replaced by the engine's + reasons: `engine_below_block`, `engine_block_not_offered` (family insufficient or calibration failed), or none. +- Every structural reason stays a hard requirement: `comparison_provenance_mismatch`, `insufficient_rung`, + `unchanged`, `change_unsupported`, `witness_unresolved`, `location_unresolved`, `family_semantics_mismatch`, + `claim_discharged` and `dependency_unresolved` (`push/policy.py:909-960`). `PushReport` invariants + (`push/report.py:48-97`) require witnesses, locations, consequence and repair for actionable defects. +- With no capability, consequence and repair come from family-level templates shipped beside the model: + language-level text, never framework text. +- `--mode blocking` blocks only on BLOCK. `--mode advisory` lists ADVISORY and above. Each disposition carries + `{p, operating_point, model_id, top_features}` in the artifact and one line in the compact report + (`render_report`, `push/report.py:214`). +- The 30 s default deadline (`cli.py`, `--deadline 30.0`) covers feature building. Everything in the `static` + profile is source-only or already computed by the scan. Model roles run only with `--model-config`. + +**Plane Runs.** The `decide` task (no model) writes `decisions.jsonl`, `remember` stores `decision` rows, and +`ousast plane status` prints BLOCK/ADVISORY counts per case. + +**Fallback (6.2).** With no model, or a family below the floor, today's behaviour applies unchanged: rule +statuses, quick/engine findings as they are, pre-push with the empty capability registry. The report says so in +one line: "decision engine: not used (no adopted model | family insufficient data | calibration failed) -- +findings are today's rule output". A family whose BLOCK is not offered still gets ADVISORY decisions. + +**User adaptation (7.5)** is opt-in: `[decision] adapt = true`. It keeps the user's dismissals in +`.openultrasast/decisions/dismissals.jsonl` in their repository, as finding fingerprints with the model id, and +applies them after the engine: a dismissed fingerprint is suppressed, and optionally a per-repository intercept +offset is fitted on at least 10 dismissals. It never writes to the memory store. The label builder refuses any row +with `origin: user`. It is evaluated separately: on held-out training repositories, by replaying the first half +of their labels as dismissals and measuring the second half. It is reported as "adaptation", never merged into the +generalisation numbers. + +## Data Models + +- `features` row: envelope (`id, kind, repo, pin, run, task, population, split, image`) + `{candidate, family, + language, profile, schema_version, feature_set_digest, unit, base?, x, instruments}`. +- `label` row: envelope + `{candidate, family, label, source, source_ref, unit, direction?, weight, split, group, + frameworks, created, evidence}`. Written only by `ousast learn labels`. +- `decision` row: envelope + `{candidate, family, profile, model_id, p, operating_point, offered, top: [{feature, + value, contribution}]}`. +- `experiment` row: `{experiment, manifest_sha256, units_sha256, registered, code_commit}`. `arm_outcome` row: + `{experiment, arm, unit, outcome, passes, usd}`. `experiment_result` row: `{experiment, look, estimate, ci, + mcnemar_p, decision, spend}`. +- `KINDS` (`plane/memory.py:52`) gains `features, label, decision, experiment, arm_outcome, experiment_result`, + and `TAG_FIELDS` is unchanged (`family` is already a tag). +- Blobs: `models/.json`, `models/index.jsonl`, `models/adopted.json`, `curves/.json`, + `labels/.jsonl`. +- Files: `learn/sources.toml`, `ruleset/frameworks.toml`, `ruleset/decision/*.json`, + `plane/experiments/.yaml` + `.units.jsonl`, `benchmarks/experiments//result.json`, + `benchmarks/independent/prediction-v3.json`, `result-v3-engine.json`. + +## Error Handling + +- Every failure of the feature builder is loud. An instrument that did not read its input records `state: + failed`, and a Run's `features` task fails when every instrument failed for a case. A plausible all-zero record + is impossible because zero requires `state: ran`. +- `validate_record` rejects unknown features, out-of-vocabulary enums and label fields, naming the feature and the + source file. +- The label builder exits 2 on a source that is not in `sources.toml`, a population whose status is not recorded, + or a group that straddles an outer fold after the merges. The error names the repository. +- Training refuses a (family, profile) below the floor and writes an "insufficient data" artifact stub, so + `decide` can say why. It refuses when a label's `group` appears in both an outer training and evaluation fold, + as an assertion before fitting. Newton non-convergence after 50 iterations fails with the lambda and the + condition number. +- `decide` with a model whose `schema_version` or `feature_set_digest` differs from the builder's falls back, + with the reason in the report, and never scores a mismatched vector. +- Experiments: a registration mismatch refuses the Run. A budget ceiling ends an arm `unfinished` (`budget.py`), + and the analysis then reports "incomplete: arm B stopped at $x" and records no decision. +- `put_blob` on FileStore applies the 1 GiB free-space refusal (`plane/memory.py:341`). + +## Testing Strategy + +- `tests/test_learn_schema.py`: allow-list, name invariance, label fields excluded, missing vs zero (section 1). +- `tests/test_learn_labels.py`, on fixture populations and pairs: + - positives and negatives per rule, including a fixed-pin function the fix did not touch (not a negative), and + an unmatched vulnerable-pin candidate (unlabelled); + - `unscorable` and title-tier exclusion; + - an adjudication mapping; + - a source not in `sources.toml`, which cannot label, checked with a fixture population named like v3 and a test + that asserts the builder never opens its file (`open` monkeypatched); + - CVE-based group merge. +- `tests/test_learn_folds.py`: + - no group in both train and test, for outer, leave-one-source-out and leave-one-framework-out folds; + - inner folds nested; + - determinism by seed. +- `tests/test_learn_train.py`: + - Newton fit on a separable-with-noise fixture; + - `test_matches_sklearn` (optional extra); + - reproducible bytes from the same store snapshot on FileStore and on MinioStore (gated as today); + - weights cap per repository; + - floor refusals. +- `tests/test_learn_calibrate.py`: Platt on a known miscalibration, the slope/intercept check, and the + ADVISORY-only downgrade. +- `tests/test_learn_decide.py`: operating points, explanations (top-5 order and signs), the missing-instrument + display, and version-mismatch fallback. +- `tests/test_learn_experiments.py`: + - registration digest and refusal; + - paired bootstrap CI on a fixture with a known difference; + - exact McNemar values against hand-computed binomials; + - O'Brien-Fleming boundaries; + - adopt / reject / inconclusive / equivalent decisions. +- `tests/test_semantic_priors.py`: + - the lint test (no untagged entry contains a `frameworks.toml` symbol); + - `priors="off"` removes exactly the tagged entries; + - the split entries keep their language-level calls. +- `tests/test_learn_roles.py`: wrapper inference to depth 3 on a fixture, and unclassified chunks are never "no + roles". +- `tests/test_plane_features.py`, `test_plane_decide.py`: fixture run directories from the validation-46 shapes. + `features` drops `site_match`/`in_fix_range`, and `remember` stores `features`/`decision` rows. +- `tests/test_push_decision.py`: admission with an engine replaces only the capability reasons, and every + structural reason still blocks admission. `PushReport` invariants hold with engine templates. +- `tests/test_scan_decision.py`: the scan fallback line, `dropped_by_engine.json`, and that a shadow/enabled + status flip without a decision change leaves the report unchanged. +- Regression net: the existing gate, pair and plane tests. `test_benchmark_output_is_byte_identical_to_golden_baseline` + holds with no model adopted (the fallback is today's path). +- Joern-gated tests run only with the mount recipe (memory note "Joern-gated tests run nowhere"). The engine-alert + harvest is verified by its own read-count check, not assumed. + +## Requirement Traceability + +| Requirement | Design section | Verified by | +| --- | --- | --- | +| 1.1 one record per candidate from every instrument | 1 | `test_plane_features.py`, `test_learn_schema.py` | +| 1.2 missing is not negative | 1 (instrument state, `null`) | `test_missing_is_not_zero` | +| 1.3 versions and digests for rebuild | 1 (versioning), 5 | `test_learn_train.py::test_reproducible` | +| 2.1 positives/negatives by source | 3 | `test_learn_labels.py` | +| 2.2 plane verdict is a feature, not a label | 1, 3 | `test_label_fields_never_reach_features`; label rules | +| 2.3 provenance; reserved populations unusable | 3 (`sources.toml`, fail-closed) | `test_learn_labels.py` (v3-named fixture never opened) | +| 3.1 simple, calibrated, per family or pooled | 4 | `test_learn_train.py`, `test_learn_calibrate.py` | +| 3.2 reproducible, versioned | 5 | reproducibility test on both backends | +| 3.3 population-level folds, no repository in both | 4 (folds) | `test_learn_folds.py` + pre-fit assertion | +| 3.4 operating points on training folds | 4 | `test_learn_decide.py` | +| 3.5 explanations stored | 4, 8 | `test_learn_decide.py`, `test_scan_decision.py` | +| 4.1 pre-registered experiment | 6 (manifest, register) | `test_learn_experiments.py` | +| 4.2 paired, assignment and spend recorded | 6 | `arm_outcome` rows; `ousast plane status` | +| 4.3 adopt only on a CI excluding zero or equivalence | 6 (decision rule) | `test_learn_experiments.py` | +| 4.4 known-callers experiment first | 6 (exp-001) | commit 7 record | +| 5.1 learning curves with bands | 7 | `ousast learn curve` output; unit test on a fixture | +| 5.2 intervals per family and language; insufficient data | 7 | `test_learn_decide.py` (floor), report fixture | +| 5.3 v3 checked once, after freezing | 7 | `prediction-v3.json` committed before `result-v3-engine.json` | +| 6.1 probability, operating point, top features in reports | 8 | `test_scan_decision.py`, `test_push_decision.py`, `test_plane_decide.py` | +| 6.2 fallback said in report | 8 | `test_scan_decision.py` | +| 6.3 rule status no longer changes output alone | 8 | status-flip test | +| 7.1 repository-grouped folds across corpora | 3 (group, CVE merge), 4 | `test_learn_folds.py` | +| 7.2 repository-agnostic allow-list with test | 1 | `test_learn_schema.py` | +| 7.3 BLOCK/ADVISORY on unseen repositories; delta unit | 1 (delta), 4, 8 | outer-fold report; `test_push_decision.py` | +| 7.4 calibration checked out of repository; else ADVISORY only | 4 | `test_learn_calibrate.py` | +| 7.5 opt-in adaptation outside the numbers | 8 | label builder refuses `origin: user`; separate report | +| 8.1 language-level knowledge; roles inferred | 2 | `test_learn_roles.py` | +| 8.2 tagged priors, off by default, kept by A/B | 2, 6 | `test_semantic_priors.py`; engine-level experiment records | +| 8.3 leave-one-framework-out, per framework | 2, 4 | `test_learn_folds.py`; report strata | +| 8.4 share depending on priors | 2, 7 | report columns priors off / adopted / difference | + +## Commit sequence, mapped to future tasks + +Each commit is gated on the full suite's own exit code, `ruff` and `mypy`. None changes a reported number until +commit 9 adopts a model, and without an adopted model every path is today's. + +1. **Schema and features** (1.1-1.3, 7.2): `learn/schema.py`, `learn/features.py` (host builder), the new + `KINDS`, and `plane/tasks/features.py` + the `remember` input, with tests. +2. **Framework tags** (8.2, 8.4): `ruleset/frameworks.toml`, the tags and splits of section 2, `priors=` in the + semantic and quick loaders (default `all` for today's scan), and the lint test. Verify that + `test_benchmark_output_is_byte_identical_to_golden_baseline` is unchanged with `priors=all`. +3. **Roles** (8.1): `learn/roles.py` (wrapper inference), `plane/tasks/roles.py` (model roles from + `model_sinks.py`), and the vocabulary overlay for the engine. Fixture tests, no model call in tests. +4. **Labels** (2.1-2.3, 7.1): `learn/sources.toml`, `learn/labels.py`, `ousast learn labels`, group + normalisation and CVE merge, and a first label snapshot committed as a measurement record (counts only). +5. **Harvest** (1.1, data): a plane Run of verify a/b on the 78 verify-family pairs and the development cases + (about $6, ceiling $10), model roles on the candidate files (ceiling $10), and engine features for the pair + corpus on the host (frozen source, one container at a time, in the background). +6. **Train, calibrate, evaluate** (3.1-3.5, 5.1-5.2, 7.3-7.4, 8.3): `learn/{folds,train,calibrate,evaluate}.py`, + the model blob API in `plane/memory.py`, `ousast learn train|evaluate|curve`, and the first evaluation report + (priors off), committed under `benchmarks/measurements/-decision-engine-v1/`. +7. **Experiments** (4.1-4.4): `learn/experiments.py`, the `register` command, `experiment-outcome`, the Run + annotation checks, `OUSAST_VERIFY_CALLERS`, and exp-001 registered, run and recorded. +8. **Integration** (6.1-6.3, 7.3, 7.5): the `decide` task, the `scan` stage, `pre-push` admission with the engine, + report fields, the fallback line, and the opt-in adaptation layer. +9. **Adopt** (3.2, 8.2): `ousast learn adopt`, the packaged models for families that pass, and the priors-on/off + engine experiment per framework with a recorded decision. +10. **v3 check** (5.3): `prediction-v3.json` is committed first, then v3 is run by its owner under protocol v3, and + `result-v3-engine.json` is committed. Nothing between the two commits may touch the engine, features, priors + or thresholds. + +## Risks + +- **Label scarcity decides what can be trained.** The status of each family today: + - memory (7 repositories, 172 of its 176 pairs from openssl and curl) and prototype (6) are **insufficient + data**. Users get today's behaviour there, labelled as such. + - PHP has no pairs, and only about 13 population repositories. The v3 prediction will mostly be the pooled + interval, marked "no PHP stratum", and it will be wide. + - The plane profile has 43 rows from one population until the harvest Run. + - BLOCK at 95% precision is likely **not reachable** for most families on this data. The design reports "BLOCK + not offered" instead of lowering the target. +- **Weak signals.** The pair baseline has quick-rule labelled recall 0.24 and Youden -0.013 overall + (`pairs.json` `overall`). The v2 engine recall is 0/17 (`results-v2.json`). A calibrated model over weak signals + is honest but may be close to the base rate. Recall at ADVISORY 0.90 may need an ADVISORY threshold so low that + precision is poor. That is reported, not tuned away on outer folds. +- **Leakage paths and their guards.** + - Label fields in features (`site_match`, `in_fix_range`, fixed-pin alerts): the allow-list and test. + - The same repository under two names or corpora: normalised group plus CVE merge. + - Hand rules and framework priors written after seeing a training repository (the `$wpdb` rule from PMPro, the + hook source from WP Statistics, `taint.sc:911-949`): the priors-off default, and leave-one-framework-out. + Rules may also carry `taught_by = [repo]`, which masks their features in folds that evaluate those + repositories. It is backfilled best-effort from comments and git history, and its absence is stated. + - The mechanism exporter's train-on-test leak (`closed-loop-train-on-test-leak.md`): no exporter output is a + feature, and mechanism buckets come from the fixed vocabulary. + - Verify and tie-break design fitted on v2: the v2 fold is marked design-informed. + - Threshold and lambda selection on outer folds: nested folds, and a pre-fit assertion. + - v3: fail-closed sources, never opened. + - User dismissals: refused by the label builder. +- **Prevalence shift.** Calibration on 1:1 pairs overstates the probability in a real push. The primary safety-net + figures (recall; false BLOCKs per 100 benign deltas) do not depend on the mix. The benign-delta sample is small: + 28 population benign deltas plus pmpro, so the 95% upper bound for 0 false BLOCKs is about 0.12 per push. Benign + history deltas are the planned remedy and are reported with and without. +- **Framework inference is itself a model.** Model roles cost money and can be wrong. `scan` without `[models]` + gets only wrapper inference, whose reach on framework-heavy code without dependency source is poor. The report + says which role source ran. Both are measured by the roles-on/off engine experiment. Nothing assumes they help. +- **Cost on the 7 GB host.** + - Model spend is small: the harvest about $6, exp-001 about $8, model roles on candidate files about $10, all + under task ceilings. + - Engine features are the time cost: about 427 pairs x 2 sides x 30-60 s of JVM and CPG per excerpt is 7-14 + hours serial, plus dev-php full-repository scans of 200-2,000 s each (`results-v1.json` `seconds`). They run + once per engine image, from a frozen source export, one container at `--memory 3g` at a time, never beside the + kind cluster's verify pods. Results are cached by (content sha, image id) in the store. The home volume was at + 98%, and records are small (about 1 KB per candidate), but each engine run's scratch output must be cleaned + per case. +- **Experiments will often be inconclusive.** With 41 development repositories the MDE is about 15 points. + Pre-registering that and recording "inconclusive" is the intended behaviour. The failure mode to avoid is + extending an experiment after looking. +- **Pure-Python numerics.** Newton on near-separable data (a feature that alone separates a small family) can + diverge. L2 with lambda >= 0.01 and the iteration cap bound it, and the sklearn cross-check catches drift. diff --git a/.kiro/specs/learned-decision-engine/spec.json b/.kiro/specs/learned-decision-engine/spec.json index fd2ec6a..9b8a0c8 100644 --- a/.kiro/specs/learned-decision-engine/spec.json +++ b/.kiro/specs/learned-decision-engine/spec.json @@ -1,16 +1,16 @@ { "feature_name": "learned-decision-engine", "created_at": "2026-09-30T14:00:00Z", - "updated_at": "2026-09-30T14:30:00Z", + "updated_at": "2026-09-30T16:00:00Z", "language": "en", - "phase": "requirements-approved", + "phase": "design-generated", "approvals": { "requirements": { "generated": true, "approved": true }, "design": { - "generated": false, + "generated": true, "approved": false }, "tasks": {