mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 12:39:28 +00:00
Marius CiepluchandClaude Opus 5.5
7b17eacd44
6.2: the 724 static examples' 616 distinct excerpts embedded with openai/text-embedding-3-small via OpenRouter, 198,819 tokens, $0.0040 metered (ceiling $1). 6.6, family injection only, memory = every family (at most 2 contrast examples per prompt), input profile v1: smoke fixed k = 5 (repeat prefix-cache hit share 0.96); one compile (program c1237a49, proposed instruction 5, balanced Brier 0.128 on 13 C_val candidates vs 0.199 baseline); outer grouped 5-fold, 110 out-of-repository candidates in 29 groups: ADVISORY recall 0.90 [0.82, 0.97], precision 0.54 [0.44, 0.65], AUC 0.74; BLOCK not offered (ECE 0.064 > 0.05; no fold reached a Wilson lower bound of 0.95); canary re-ask agreement 0.83. $0.91 metered at list price; DeepSeek balance $18.20 -> $17.90. Instrument fixes found on the way, each with a test: - memory build: pair sides join the record stored under their harvest unit pin (--units), and only the built profile's records are read (static and plane shared a key; 85 -> 724 examples); - retrieval: stage 1 per side (family / contrast) and at most 2 contrast examples, never padded; - program: the JSON answer contract is part of the signature, appended to proposed instructions (DeepSeek JSON mode refused a prompt without "json"); - compile: candidate_families for a one-family compile; compile.toml k = 5 from the smoke rule. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
1256 lines
102 KiB
Markdown
1256 lines
102 KiB
Markdown
# Design Document
|
|
|
|
## Overview
|
|
|
|
Detection decisions move out of hand edits and into an AI classifier with local memory. The instruments stay and
|
|
emit signals. A model-free feature builder turns those signals into one record per candidate. A label builder
|
|
attaches ground truth from sources we can defend, and labelled cases become a local memory of examples. A
|
|
language-model program, compiled DSPy-style in-house (Req 3, changed 2026-10-01: "We will not be able to scale up
|
|
data"), reads the candidate's code, its signals and its inferred roles together with similar labelled cases
|
|
retrieved from memory, and answers a verdict with a confidence. That confidence is calibrated only on repositories
|
|
the program never saw, and two thresholds on it (BLOCK, ADVISORY) are what `pre-push`, `scan` and plane Runs report.
|
|
No statistical model is fitted. Pipeline changes, program variants included, are pre-registered, paired
|
|
experiments.
|
|
|
|
Six facts from the repository and the recorded results shape the design:
|
|
|
|
- **The deployment target is a repository nobody trained on.** Maintainer: "the tool will never run against the
|
|
test cases it's being trained with. it's purpose is a repo safety net." (Req 7). The primary unit is a candidate
|
|
in changed code (`ousast pre-push`, base..head), and whole-repository `scan` is the secondary mode. Every
|
|
reported number therefore comes from folds grouped by repository across all corpora. Features are
|
|
repository-agnostic by an allow-list. Fit to a user's own repository is an opt-in layer outside the reported
|
|
numbers.
|
|
- **Labels are scarce, and full plane signals exist on 43 rows.** The only recorded plane run with
|
|
verify/agree/tie-break is `validation-46`: 15 cases, 43 candidates after triage, 19 of them with
|
|
`site_match` true, all from population v2
|
|
(`~/ousast-results/plane/validation-46/*-final/agreed.json`). The pair corpus is larger (427 pairs and 452
|
|
labelled findings in the catalog today, 317/342 in the 2026-09-30 baseline
|
|
`benchmarks/measurements/2026-09-30-harnessx-removal-baseline/pairs.json` `overall`). Measured by
|
|
repository rather than by row, it is 146 groups, and two families are nearly single-repository (section 3). A
|
|
classifier over verify signals cannot be evaluated today. The design starts from the signals every labelled row
|
|
can have, harvests the rest at a stated cost, and accepts that data will stay small: the engine learns by
|
|
memory and compilation, not by fitting parameters.
|
|
- **The core install is PyYAML only** (`pyproject.toml:11`). numpy, scipy and scikit-learn are not installed in
|
|
`.venv`. Retrieval (a Gower distance and a cosine over at most 40 vectors) and calibration (two-parameter Platt)
|
|
stay pure Python; the model calls go through the plane's metered client (section 4).
|
|
- **`pre-push` already has the decision seam.** `admit_candidates` (`push/policy.py:879`) applies one evidence
|
|
bar to `CandidateDelta`s with `novelty` (`push/policy.py:61-74`). Its capability registry is empty by design
|
|
(`push/policy.py:802-823`: "No shipped declaration is enabled"), so `pre-push` blocks nothing today. The
|
|
engine's operating point replaces the capability-eligibility reasons, and the structural reasons stay.
|
|
- **The memory store is the system of record and has the needed shape.** Rows are keyed by repository and pin,
|
|
with a closed `KINDS` tuple (`plane/memory.py:52`). There are two backends (FileStore, MinioStore) behind one
|
|
interface (`plane/memory.py:176-291`), content-addressed blobs (`put_facts`, `plane/memory.py:249-255`), and a
|
|
train-on-test `Guard` (`improve/memory.py:95-133`). Features, labels, decisions, model artifacts and experiment
|
|
outcomes become new row kinds and blob prefixes. No second store is added.
|
|
- **Hand-written framework knowledge is embedded in the detectors** (Req 8). Examples are WordPress `$wpdb`,
|
|
hooks and `maybe_unserialize` in `ruleset/semantic/php.toml` and `ruleset/php/rules.toml`, and Flask and
|
|
Django names in `ruleset/semantic/python.toml`. Section 2 tags each entry. Priors become optional features
|
|
that are off by default. Framework roles are inferred per repository instead.
|
|
|
|
Decisions in one place:
|
|
|
|
| Question | Decision |
|
|
| --- | --- |
|
|
| Candidate key | `(repo, pin, path, function, family)` in the row envelope only; `pin` is the head for a delta |
|
|
| Features | allow-listed, typed, repository-agnostic; explicit `null` + instrument state for missing; two profiles (`static`, `plane`) |
|
|
| Where features are built | plane task `features` (no model) after `agree`/`final` and `alerts`; host path `learn.features.build_for_scan` for `scan`/`pre-push` |
|
|
| Labels | host-only `ousast learn labels`; site/fix-range matcher for populations, vulnerable/fixed side for pairs, benign deltas, adjudications; fail-closed source allow-list keeps v3 out |
|
|
| Engine | a language-model program, in-house DSPy-style: signature (code excerpt, signals, roles, family -> verdict, family, confidence, rationale with cited lines), modules `Retrieve -> Classify(k)`, every call through `MeteredClient` bound to a Model per task; no statistical model is fitted |
|
|
| Models | DeepSeek (`deepseek-flash`) for classification; OpenRouter (`openai/text-embedding-3-small`) for embeddings only |
|
|
| Memory | `example` rows (label + record + excerpt) in the MemoryStore; stage 1 Gower nearest neighbours on the signal profile with explicit missing, stage 2 code-embedding re-rank, 6 examples under a label/family/repository balance rule; an evaluation-boundary filter with a test |
|
|
| Compilation | pre-registered balanced Brier score on a compile split of repositories never evaluated; bootstrap few-shot (3 demonstrations) and instruction search (6 proposed + baseline); ceiling $3 per compile; responses cached for byte-identical replay |
|
|
| Families | one program for all families; calibration and operating points per family at >= 20 positive and >= 20 negative repositories, pooled at >= 10, otherwise "insufficient data" |
|
|
| Calibration | Platt scaling of the program's score, cross-fitted over outer folds; checked per fold (reliability, slope, intercept, ECE); if it fails, ADVISORY only |
|
|
| Folds | outer grouped 5-fold by repository (one repetition), plus leave-one-source-out and leave-one-framework-out; compile split disjoint from every evaluation fold |
|
|
| Operating points | BLOCK: lowest threshold whose Wilson lower bound on precision is >= 0.95, chosen on the other folds' predictions, never on a majority `unsure`; ADVISORY: threshold reaching 0.90 recall; reported with repository-bootstrap intervals |
|
|
| Artifacts | content-addressed compiled-program JSON under `programs/` in the store; adopted version exported into the package by a maintainer commit |
|
|
| Cost | ~$0.003 per candidate (k = 1), ~$0.0045 (k = 5, cached); ~$2.2 per compile; ~$3.4-5.2 per full evaluation; first slice ~$5, $7 ceiling, within today's balance |
|
|
| Experiments | manifest `plane/experiments/<id>.yaml` + Run annotation `openultrasast.io/experiment`; paired; repository-cluster bootstrap CI (primary), exact McNemar (secondary); O'Brien-Fleming two-look stopping |
|
|
| First experiment | `exp-001-known-callers`: callers line on vs off, development corpus (v1, pmpro, MediaWiki, WP Statistics, full-repository pointer pairs), about $8 expected, $15 ceiling |
|
|
| Framework knowledge | tagged `framework`/`library` priors, off by default; roles inferred by a `roles` plane task (model) + deterministic wrapper inference; leave-one-framework-out evaluation |
|
|
|
|
## Architecture
|
|
|
|
```
|
|
instruments (signal producers, unchanged as standalone tools)
|
|
quick rules Joern engine repo-facts roles (new) verify a/b/c + agree
|
|
alerts task engine_alerts / callers, entry model role labels (plane only, Model-bound)
|
|
pre-push scan distance + wrapper inference
|
|
\ | | | /
|
|
+--------------+------------------+-------------------+--------------------+
|
|
|
|
|
features (new, no model): allow-listed record per candidate
|
|
plane: task after final + alerts host: learn.features.build_for_scan
|
|
|
|
|
+---------------------+----------------------+
|
|
| |
|
|
labels (new, host only, never in a sandbox) embed (new, Model: OpenRouter embeddings)
|
|
populations: matches_v2 / fix ranges |
|
|
pairs: vulnerable vs fixed side decide (new, Model: DeepSeek, metered)
|
|
benign deltas, adjudications Retrieve (signals -> embeddings, boundary filter)
|
|
source allow-list: v3 absent -> Classify(k) -> calibrated p, operating point,
|
|
| rationale -> decision rows / report fields
|
|
memory build/embed (host) -> example rows, ^ ^ seeded, boundary-filtered memory
|
|
excerpts, embeddings in the store ---------------------+-----------+
|
|
| |
|
|
learn compile/evaluate (host, metered) ---- program artifact (programs/<sha>.json) ---- adopt (maintainer)
|
|
compile split, bootstrap demos, instruction search, |
|
|
grouped folds, cross-fitted Platt, thresholds, curves packaged: ruleset/decision/program-<profile>.json
|
|
experiments (plane Runs with arms) -> arm_outcome rows -> analysis -> adopt | reject | inconclusive
|
|
```
|
|
|
|
A plane Run per case becomes `facts -> va, vb -> agree -> vc -> final -> alerts -> roles? -> features -> embed ->
|
|
decide -> remember`. `roles` is optional and Model-bound. `features` is model-free with budget `{usd: 0, calls: 0}`,
|
|
as `remember` and the loop steps are (`plane/tasks/loop.py:1-5`). `embed` and `decide` are Model-bound (one Model
|
|
per Task, `plane/egress.py:87`) and metered. `remember` ingests the new `features` and `decision` rows. Labels are
|
|
never computed in a Run. They are built on the host from ground-truth files the sandbox never receives, so no task
|
|
that produces a feature can read a label.
|
|
|
|
New code: `src/openultrasast/learn/` (`schema.py`, `features.py`, `roles.py`, `labels.py`, `sources.toml` -- done --
|
|
and `excerpt.py`, `examples.py` (example store), `retrieve.py`, `program.py`, `compile.py`, `folds.py`,
|
|
`calibrate.py`, `evaluate.py`, `decide.py`, `experiments.py`, `compile.toml`),
|
|
`src/openultrasast/plane/tasks/{features,roles,embed,decide}.py`, `plane/models/openrouter-embedding.yaml`,
|
|
`ruleset/frameworks.toml`, `ruleset/decision/`, and CLI groups `ousast learn ...` and `ousast plane experiment ...`.
|
|
The reconciler is not touched. Its 500-line budget stays (`tests/test_plane_reconciler.py`).
|
|
|
|
## Components and Interfaces
|
|
|
|
### 1. Candidate key and feature record (Requirements 1, 7.2)
|
|
|
|
**Candidate.** A candidate is `(repo, pin, path, function, family)`. `repo` is `memory.repo_key`
|
|
(`plane/memory.py:69`), `function` is the enclosing declaration by the `repo-facts` patterns
|
|
(`plane/tasks/repo_facts.py:131`, `<global>` outside any function), and `family` is a `families.toml` id. It is the
|
|
key that already joins `alert` and `verdict` rows (`improve/memory.py:7`: "repo, pin and `path::function`"). For a
|
|
delta the pin is the head, and the record carries `base` beside it. The key lives in the row **envelope**, for
|
|
joining labels and grouping folds. It is never a feature.
|
|
|
|
A candidate exists when at least one instrument emitted a signal of that family in that function: a quick-rule
|
|
alert, an engine finding, a `roles` sink, or a verify candidate. A labelled positive with no candidate is a
|
|
**reach miss**. It counts against recall and is reported separately as `reach`, so the engine is never credited
|
|
for sites no instrument surfaced.
|
|
|
|
**Delta mapping (Req 7.3).** For base..head, changed hunks (`git diff -U0 base head`) map to the head functions
|
|
that enclose a changed line. Functions whose engine operation `compare_evidence` marks `new`/`worsened`
|
|
(`push/policy.py:489`, `CandidateDelta.novelty`) are added. Each such function yields one candidate per family
|
|
that has a signal there. The base side of the same function supplies the `delta.*` features. A whole-repository
|
|
scan is the same with no base (`delta.*` is `not_applicable`).
|
|
|
|
**Features.** The feature set is closed and listed in `learn/schema.py` as `FEATURES: tuple[FeatureSpec, ...]`.
|
|
Each spec has a name, a source instrument, a type (`bool`, `int` with a cap, `float`, or `enum` with a closed
|
|
vocabulary), a profile (`static` or `plane`), and whether it is a prior. There are no free-text values, no
|
|
identifiers and no raw counts of repository-specific tokens.
|
|
|
|
| Feature | Source (file) | Type | Profile |
|
|
| --- | --- | --- | --- |
|
|
| `lang.*` | `preprocess.detect_language` | one-hot over the languages the engine or quick mode covers | static |
|
|
| `qr.enabled_hits`, `qr.shadow_hits` | quick scan (`plane/tasks/alerts.py:117`, `findings.quick_scan_findings`) | int, cap 20 | static |
|
|
| `qr.mechanism.<m>` | the hit rules' `tags` mapped to the closed mechanism vocabulary (`benchmarks/pairs/mechanisms.toml` ids) | int, cap 10 | static |
|
|
| `qr.max_precision_estimate` | `PatternRule.precision_estimate` (`ruleset/store.py:33`) | float | static |
|
|
| `qr.prior_hits` | hits of rules tagged `framework`/`library` (section 2) | int, cap 10 | static, **prior** |
|
|
| `eng.findings` | engine findings in the function (`engine_alerts.engine_rows`, `plane/engine_alerts.py:160`; pre-push scan) | int, cap 10 | static |
|
|
| `eng.rung_max` | finding rung | enum `suspicion, model_entailed, entailed` | static |
|
|
| `eng.witness_steps_min` | witness path length | int, cap 50 | static |
|
|
| `eng.sink_kind` | sink entry id mapped to an operation kind | enum (sql, command, code_eval, deserialize, file_path, file_inclusion, outbound_request, redirect, html_output, other) | static |
|
|
| `eng.source_kind` | source entry kind | enum (request, server, raw_body, argv_stdin, inferred_role, prior, other) | static |
|
|
| `eng.sink_origin`, `eng.source_origin` | where the vocabulary entry came from | enum (language, inferred_wrapper, model_role, prior) | static |
|
|
| `eng.sanitizer_on_path` | discharge seen but not accepted | bool | static |
|
|
| `eng.completion` | questions completed / total for the file | float | static |
|
|
| `eng.degraded`, `eng.unstable` | degradations; present in only one of two runs | bool | static |
|
|
| `facts.callers` | `repo-facts` call sites (`repo_facts.py:151`, cap 40) | int, cap 40 | static |
|
|
| `facts.caller_files` | distinct caller files | int, cap 20 | static |
|
|
| `facts.entry_distance` | caller-graph hops to an entry point (`mapping.analyze_entry_points`, `mapping.py:123`) | int, cap 6 | static |
|
|
| `facts.is_global` | candidate is `<global>` | bool | static |
|
|
| `facts.function_lines` | declaration span | int, cap 500 | static |
|
|
| `roles.sink_confidence`, `roles.source_in_function` | `roles` task (section 2) | float / bool | static (deterministic part), plane (model part) |
|
|
| `delta.novelty` | `CandidateDelta.novelty` or diff of base/head signals | enum `new, worsened, unchanged, unknown, not_applicable` | static |
|
|
| `delta.changed_lines` | changed lines inside the function | int, cap 200 | static |
|
|
| `delta.sanitizer_removed` | a sanitizer call present at base is absent at head | bool | static |
|
|
| `verify.flag_a`, `verify.flag_b` | pass verdicts (`units.jsonl`, `plane/tasks/verify.py`) | bool | plane |
|
|
| `verify.flag_c` | tie-break pass (asked only when a/b disagree) | bool | plane |
|
|
| `verify.votes` | flags over the passes asked | int 0-3 | plane |
|
|
| `verify.turns_mean`, `verify.site_offset` | tool turns; lines between the model's reported site and the candidate line | float / int cap 200 | plane |
|
|
| `agree.final` | `agreed, disputed, rejected` (`plane/tasks/remember.py:73`) | enum | plane |
|
|
| `ms.flagged`, `ms.operation` | model sink classification (`benchmarks/independent/model_sinks.py`, moved into `roles`) | bool / enum | plane |
|
|
|
|
**Deliberately excluded from features** (each is label information or an identity):
|
|
|
|
- `site_match` (`plane/tasks/agree.py:200`) and `in_fix_range` (`plane/tasks/alerts.py:137`). Both are computed
|
|
from declared sites and fix ranges.
|
|
- Every fixed-pin alert. A deployed scan has no fixed pin.
|
|
- Declared-site counts and `case.json`.
|
|
- Costs (`usd`), which vary with provider pricing, not with the code.
|
|
- Rule ids, paths, file and function names, commit ids and repository names.
|
|
|
|
Rule ids enter only through closed mechanism buckets and the prior count. A rule written after one case's miss
|
|
would otherwise identify that case.
|
|
|
|
**Missing is not negative (Req 1.2).** Each record carries `instruments: {name: {state, version}}` with `state` in
|
|
`ran | none | failed | not_applicable`, and `x: {feature: value | null}`. `none` means no coverage for the language
|
|
(`plane/tasks/alerts.py` `coverage`, `remember.coverage_rows`, `plane/tasks/remember.py:79-96`). `failed` means
|
|
the instrument broke or did not read its input: the engine-alert rule "no result file, no `read N files` line, or
|
|
`questions == 0` fails the case loudly" (`plane/engine_alerts.py` docstring) maps here, never to zero findings.
|
|
`not_applicable` covers two cases: `delta.*` on a whole-repository scan, and `facts.*` on a pair excerpt, where no
|
|
repository exists. Every feature of an instrument whose state is not `ran` is `null`. The engine never imputes a
|
|
`null`: retrieval compares instrument states block by block (section 4.2), and the prompt shows "no coverage",
|
|
"failed" or "not applicable" in words.
|
|
|
|
**Profiles.** `static` is what `pre-push` and `scan` can compute locally without a model: quick rules, engine,
|
|
facts, deterministic roles and delta. `plane` adds verify, agree and model roles. A program is compiled and
|
|
calibrated per profile. A `static` program never sees verify features, in its prompt or its memory, so it cannot
|
|
learn to lean on an instrument that is absent in deployment.
|
|
|
|
**Enforcement (Req 7.2).** The builder emits only `FEATURES` names. `validate_record` rejects any other key, any
|
|
string value outside an enum's vocabulary, and any number outside its type. Tests:
|
|
|
|
- `test_learn_schema.py::test_features_are_allow_listed`: every emitted key is in `FEATURES`, and no spec's type
|
|
is free text.
|
|
- `::test_record_is_name_invariant`: two fixture repositories that differ only in repository name, paths, file
|
|
names, function names and commit ids give byte-identical `x`.
|
|
- `::test_label_fields_never_reach_features`: `site_match`, `in_fix_range`, fixed-pin rows and `case.json`
|
|
fields in the inputs change nothing in `x`.
|
|
- `::test_missing_is_not_zero`: `coverage: none` gives `null` + `state: none`, and a covered language with no hit
|
|
gives `0` + `state: ran`.
|
|
|
|
**Versioning.** `schema_version` (an integer, bumped on any spec change) and `feature_set_digest` (sha256 of the
|
|
canonical `FEATURES` list). `instruments.*.version` records the ruleset digest, the engine image id
|
|
(`engine_alerts._image_id`, `plane/engine_alerts.py:177`), the runner image digest (`memory.image_digest`), the
|
|
verify model id and a prompt digest, and the `roles` model id. That makes every record rebuildable (Req 1.3).
|
|
|
|
**Where records live.** A new memory row kind `features` (`KINDS` at `plane/memory.py:52`), per repository and
|
|
pin like every other row:
|
|
`{candidate, family, language, profile, schema_version, feature_set_digest, x, instruments, unit: "pin"|"delta",
|
|
base?}`. The row id is `row_id("features", run, task, candidate+family)`.
|
|
|
|
**Who builds them.**
|
|
|
|
- *Plane*: task `features` (`plane/tasks/features.py`), no Model, one per case, after `final` and `alerts` (and
|
|
`roles` when present). Inputs are `facts.json`, the pass `units.jsonl`, `agreed.json` (with `site_match` dropped
|
|
on read), `alerts.jsonl` (vulnerable pin only, `in_fix_range` dropped), the alerts `summary.json` coverage, and
|
|
`roles.json`. It writes `features.jsonl`, which `remember` turns into rows (one more input in
|
|
`remember.rows_for`, `plane/tasks/remember.py:99`).
|
|
- *Host*: `learn.features.build_for_scan(root, findings, engine_result, *, base=None)`, called by `scan` and
|
|
`pre-push` (section 8) and by `ousast learn features <corpus>` to build records for the pair corpus and the
|
|
development corpus without a Run.
|
|
|
|
### 2. Framework knowledge: tagged priors and inferred roles (Requirement 8)
|
|
|
|
**Rule.** Hand-written knowledge stays at the language level: superglobals, built-ins, process execution, raw
|
|
database drivers (DB-API `execute`, `mysqli_*`, `pg_*`, JDBC), deserialisers and the standard library.
|
|
Framework and application roles (request objects, query builders, hooks, sanitisers, guards) are either
|
|
**inferred per repository** or kept as **tagged priors**, which are features that are off by default.
|
|
|
|
**Tagging.** Every `[[source]]`, `[[sink]]`, `[[sanitizer]]` and `[[dispatch]]` entry in `ruleset/semantic/*.toml`
|
|
and every quick `[[rule]]` gains an optional `framework = "<id>"` or `library = "<id>"`. An entry that mixes
|
|
language-level and framework calls is split, so a tag always covers a whole entry. `ruleset/frameworks.toml`
|
|
lists the ids and, per id, a lexicon of symbols. The lexicon has two uses only: the lint test, which fails on an
|
|
untagged entry containing a lexicon symbol, and repository stratification (below). It never feeds detection.
|
|
Loaders take `priors: Literal["off", "all"] | frozenset[str]`. The default for the engine model is `off`. Today's
|
|
`scan` behaviour (fallback, Req 6.2) keeps `all`.
|
|
|
|
Current framework-specific entries and their disposition:
|
|
|
|
| Entry | Where | Tag / action |
|
|
| --- | --- | --- |
|
|
| `wpdb` sink (`$wpdb->get_var`, ...) | `ruleset/semantic/php.toml:47-58` | `framework = "wordpress"` |
|
|
| `unserialize` sink: `unserialize` + `maybe_unserialize` | `php.toml:136-138` | split: `unserialize` language; `maybe_unserialize` wordpress |
|
|
| `sql_escape`: `mysqli_real_escape_string`, ..., `esc_sql` | `php.toml:169-170` | split: `esc_sql` wordpress |
|
|
| `wpdb_prepare` sanitizer | `php.toml:194-196` | wordpress |
|
|
| `coerce`: `intval`, ..., `absint` | `php.toml:207-208` | split: `absint` wordpress |
|
|
| `wordpress_hooks` dispatch (`add_action`, `apply_filters`, ...) | `php.toml:225-242` | wordpress |
|
|
| layout `php_composer_and_wordpress` | `php.toml:244-247` | product filter (vendored/test paths), not detection; renamed `php_layout`, untagged |
|
|
| rule `php-wpdb-sql-composition` | `ruleset/php/rules.toml:20-29` | wordpress |
|
|
| rule `php-unserialize` (pattern includes `maybe_unserialize`) | `php/rules.toml:114-125` | split into `php-unserialize` (language) and `php-maybe-unserialize` (wordpress) |
|
|
| Flask `request` source (`request.args`, `.form`, `.json`, ...) | `ruleset/semantic/python.toml:5-17` | flask |
|
|
| `django-request` source | `python.toml:24-30` | django |
|
|
| `orm.raw` (`objects.raw`, `RawSQL`, `extra`; `text`, `from_statement`) | `python.toml:102-104` | split: django / sqlalchemy (library) |
|
|
| `path.open` (`open`, `os.*`, `shutil.*`; `send_file`, `send_from_directory`; `FileResponse`) | `python.toml:110-112` | split: language / flask / django |
|
|
| `html.unescaped` (`render_template_string`, `Markup`; `mark_safe`, `format_html`, `HttpResponse`) | `python.toml:119-121` | flask / django |
|
|
| `outbound.request` (`requests.*`, `httpx.*`; `urlopen`, `urlretrieve`) | `python.toml:127-129` | split: library requests/httpx / language |
|
|
| `redirect` (`redirect`, `HttpResponseRedirect`, `url_for`) | `python.toml:134-136` | flask / django |
|
|
| `url_allowlist` (`url_has_allowed_host_and_scheme`, `is_safe_url`) | `python.toml:145-146` | django |
|
|
| `pickle.loads` entry incl. `yaml.load` | `python.toml:55-57` | split: `yaml.load` library pyyaml |
|
|
| Express `req` source (`req.query`, `req.body`, ...) | `ruleset/semantic/javascript.toml:5-6` | express |
|
|
| `html.response` (`res.send`; `res.write`/`res.end`) | `javascript.toml:80-82` | split: express / language (Node core) |
|
|
| `path.open` (`sendFile`; `readFile`, ...) | `javascript.toml:87-89` | split: express / language |
|
|
| `redirect` (`redirect`; `location.*`) | `javascript.toml:129-131` | split: express / language (DOM) |
|
|
| `prototype.merge`, `outbound.request`, `html.escape` | `javascript.toml:57-59, 109-111, 148-149` | library (lodash, axios/needle, escape-html/DOMPurify) |
|
|
| `servlet` source, `response.write` | `ruleset/semantic/java.toml:5-6, 47-49` | `jakarta-servlet` |
|
|
| `outbound.request` (`getForObject`, `exchange`; `sendRedirect`; `openConnection`) | `java.toml:54-56` | split: spring / jakarta-servlet / language |
|
|
| `java.escape` (`ESAPI.encoder`, `StringEscapeUtils`; `getCanonicalPath`, `normalize`) | `java.toml:61-62` | split: library / language |
|
|
| hook-as-source rule | `cpg/queries/taint.sc:918-949`, `semantic/facts.py:73-90` | general code driven by the dispatch fact ("The registry's vocabulary arrives as a FACT", `taint.sc:921`); with priors off it is inert unless `roles` infers a registry |
|
|
|
|
The maintainer's term counts (php.toml 24, php rules 9, python 16, javascript 5, java 3, 15 WordPress mentions
|
|
in `semantic/*.py` and `cpg/queries/*.sc`) are covered by these entries. The remaining mentions in `.sc` and
|
|
`.py` are comments (`taint.sc:161, 249, 322-327, 588, 911, 1296`; `dominance.sc:130`; `census.sc:9`;
|
|
`facts.py:345`). The tagging commit's lint test, not this table, is the authority.
|
|
|
|
**Role inference (Req 8.1).** Two parts, both producing per-repository vocabulary with provenance:
|
|
|
|
1. *Wrapper inference* (deterministic, `learn/roles.py`, no model). A function whose parameter reaches, within its
|
|
body, a call already in the role set becomes a derived sink or sanitizer of the same operation kind, with
|
|
`depth + 1`. The role set starts from language-level entries plus any inferred roles. Derivation iterates over
|
|
the `repo-facts` caller map to depth 3. When dependency source is present (`vendor/`, `node_modules/`,
|
|
site-packages in the checkout), it is scanned the same way, so `$wpdb->query` becomes a sink because
|
|
WordPress's own `wpdb::query` reaches `mysqli_query`, not because a table says so. That holds only when WordPress
|
|
core is present. A plugin checkout usually lacks it, which is why part 2 exists.
|
|
2. *Model roles* (plane task `roles`, Model-bound, `deepseek-flash`). The source-only classification of
|
|
`benchmarks/independent/model_sinks.py` moves into the package: per product file chunk, which functions and
|
|
call sites perform an operation of a closed kind on non-constant data. The v2 record: declared site among the
|
|
candidates in 15/17 cases, fix or site file in 16/17, $14.42 for 17 repositories
|
|
(`selection-v2-model-sinks.json`). The output extends to all roles: `sink`, `source`, `sanitizer`, `guard`,
|
|
`dispatch` (a register/apply pair such as a hook registry), with `operation`, line and a one-line evidence
|
|
quote. A chunk whose call fails is `unclassified`, never "no roles" (the `model_sinks.py` rule). In `pre-push`
|
|
only the changed files and the definitions they call are classified.
|
|
|
|
Roles reach the engine as a per-scan vocabulary overlay in the shape of `ruleset/semantic/<lang>.toml` entries
|
|
with `origin = "inferred_wrapper" | "model_role"`. The engine already takes vocabulary as facts
|
|
(`taint.sc:921`). They also reach the feature record as `eng.*_origin` and `roles.*`. Roles are signals. A
|
|
model-labelled sink alone is not a finding.
|
|
|
|
**Stratification metadata.** Each repository gets `frameworks: [ids]` from its dependency manifests
|
|
(`composer.json`, a WordPress plugin header, `requirements*.txt`/`pyproject.toml`, `package.json`,
|
|
`pom.xml`/`build.gradle`) matched against `frameworks.toml`. This is envelope metadata for folds and reports,
|
|
never a feature.
|
|
|
|
**Evaluation (Req 8.2-8.4).** The engine is always compiled and reported with `priors=off` (prior features hidden
|
|
from the prompt and from the retrieval distance). A prior group (one
|
|
framework id) is kept only through an engine-level A/B (section 6): priors on vs off, paired on the same outer
|
|
folds, metric recall at the BLOCK operating point. It is adopted only if the repository-bootstrap CI excludes zero.
|
|
Every report carries, per family, the metric with priors off, with the adopted priors, and the difference: the
|
|
share of the result that depends on priors. Leave-one-framework-out folds (section 4) give the "framework unseen
|
|
in training" row.
|
|
|
|
### 3. Label builder (Requirement 2)
|
|
|
|
`ousast learn labels` (`learn/labels.py`) runs on the host only. It writes `label` rows (a new kind) and
|
|
`labels/<digest>.jsonl` snapshots.
|
|
|
|
**Sources are an allow-list, fail-closed.** `learn/sources.toml` names each usable source, the file that proves
|
|
its evaluation was recorded, and the date. A population not listed cannot produce a label. The builder never
|
|
opens a population file that is not listed, so population v3 (`population-v3-php.toml`, reserved) is excluded
|
|
without being read. It becomes usable only when a later commit adds it after the v3 result is recorded
|
|
(Req 2.3).
|
|
|
|
| Source | Evaluation record | Status |
|
|
| --- | --- | --- |
|
|
| `population-v1` | `benchmarks/independent/results-v1.json` | spent (retuned on, `population-v2.toml:9-11`) |
|
|
| `population-v2` | `results-v2.json`, `results-v2-hunter.json`, `plane-increment-2.json` | spent for tuning (`model_sinks.py`: "v2 is spent for tuning"); its fold is marked *design-informed* (section 4) |
|
|
| `dev-php` | `benchmarks/push/inputs.json` (pmpro introduce/repair/benign), MediaWiki and WP Statistics checkouts | development |
|
|
| `pairs` | `benchmarks/pairs/**/catalog.toml` via `load_pair_catalog` (`pairs.py:229`) | development; catalog split recorded, not used for folds |
|
|
| `adjudications` | `results-v1.json` `precision_review`, `results-v2.json` / `results-v2-hunter.json` `precision_sample`, `verifier-check-2026-09-29.json` | recorded verdicts |
|
|
|
|
**Rules per source.**
|
|
|
|
- *Populations.* Positive: a candidate on the vulnerable pin that matches the case. Matching uses v2 declared sites
|
|
through `agree.matches_declared` (`plane/tasks/agree.py:45`, the parity-proven copy of `evaluate.matches_v2`).
|
|
v1 has no declared sites (0 in `population-v1.toml`), so it uses the fix-range rule `evaluate.matches`
|
|
(`benchmarks/independent/evaluate.py:162`). Negative: the same `(path, function)` on the fixed pin when the fix
|
|
changed that function (inside the new-side `fixed_ranges`, `plane/generate.py:106, 227-229`). A function the fix
|
|
did not touch is the same code at both pins and is not a negative. Benign: a candidate in a function changed
|
|
between the benign base and tip pins that has a finding at the tip and none at the base. It is negative
|
|
**by protocol assumption** (protocol v2 "Benign false alert"), with source `benign_control`. A candidate on a
|
|
vulnerable pin that matches no site is **unlabelled**, never negative.
|
|
- *Pairs.* Positive: the labelled function (`ExpectedFinding.function`/`line`, `benchmark.py:20-30`; 407 of 452
|
|
labels name a function) on the vulnerable side. Negative: the same function on the fixed side. Pairs marked
|
|
`unscorable` (identical twins, known limits, `pairs.py:130`; 7 today) give no labels. Tier `title`
|
|
(`agent-vfc`, 59 pairs, labels read from advisory titles; labelled recall 3.6% in the baseline) is excluded by
|
|
default and used only as a sensitivity arm.
|
|
- *Deltas.* Every population case and every pair also gives delta units. fixed..vulnerable is the introducing
|
|
direction, and its labelled function is a positive delta candidate. vulnerable..fixed is a repair, whose
|
|
candidates are negative for BLOCK. benign base..tip is a benign push. These are the primary units for the
|
|
safety net (Req 7.3).
|
|
- *Benign history (optional, flagged).* Non-security commits from the history of each training repository at
|
|
least 90 days away from any fix date give benign deltas at no model cost. They are negative by assumption, carry
|
|
source `benign_history` and a weight of 0.5, and every metric is reported with and without them. This is the
|
|
only cheap way to estimate false blocks per benign push.
|
|
- *Adjudications.* `True` and `True, privileged` are positive (privileged is kept as a label attribute), and
|
|
`False, *` is negative. Each keeps the adjudicator (maintainer, or a judge task with its evidence) and the
|
|
recorded text.
|
|
- A plane verdict (`agreed`, `rejected`, `disputed`) is never a label (Req 2.2). It enters only as `agree.final`.
|
|
|
|
**Label row.** `{candidate, family, label: 1|0, source, source_ref (case id / pair name / adjudication index),
|
|
unit: pin|delta, direction?, weight, split (catalog split or population split), group (repository group key),
|
|
frameworks, created, evidence}`. `group` is the normalised repository, the lower-cased `owner/name` as the guard
|
|
uses (`improve/memory.py:89-92`). Candidates that share an advisory id (CVE/GHSA) across corpora are merged into
|
|
one group, so a repository mirrored under two names cannot straddle folds.
|
|
|
|
**What exists today** (counted 2026-09-30 with a scratch script over `load_pair_catalog` +
|
|
`split_by_repository`; the label-snapshot commit makes the count reproducible):
|
|
|
|
| Family | Pair repos (positive) | Pairs | Population repos (v1 + v2 + dev) | Adjudicated | Per-family model? |
|
|
| --- | --- | --- | --- | --- | --- |
|
|
| injection | 43 | 56 | 7 + 10 + 3 | most of the 43 (per-family split not counted) | yes |
|
|
| untrusted_destination | 41 | 41 | 3 + 6 + 0 | some | yes |
|
|
| path | 24 | 24 | 0 | 0 | yes (Python-heavy) |
|
|
| config_secrets | 23 | 24 | 1 + 1 | some | yes, borderline |
|
|
| deserialization | 23 | 23 | 0 | 0 | yes (all Python) |
|
|
| access_control | 21 | 24 | 0 | 0 | yes, borderline |
|
|
| output_encoding | 17 | 58 (40 from one repository, OWASP BenchmarkJava) | 0 | 0 | pooled |
|
|
| memory | 7 | 176 (132 openssl, 40 curl) | 0 | 0 | **insufficient** |
|
|
| prototype | 6 | 7 | 0 | 0 | **insufficient** |
|
|
|
|
Totals: 427 pairs and 452 labels in 146 repository groups (142 named). Tiers: advisory 250, seeded 115, title 59,
|
|
reviewed 3. Provenance: human 237, agent 139, synthetic 51. 59 repositories straddle the catalog's train/holdout
|
|
split (`split_by_repository` report). The engine ignores that split for folds and records it on the label.
|
|
|
|
By language, the strata with at least 10 positive repositories are Python (untrusted_destination 30, injection
|
|
23, deserialization 23, path 17, access_control 15, output_encoding 15) and JavaScript injection (16). **PHP has
|
|
no pairs.** Its labels are v1 (4 repositories), v2 (6) and dev-php (3): about 9 injection and 4
|
|
untrusted_destination repositories.
|
|
|
|
Plane-profile rows (verify/agree present) exist for 43 candidates in 15 repositories, all v2. The adjudicated
|
|
rows are mostly negative: v1 1/12 true, v2 0/17, v2 hunter 2/14 (`results-v1.json` `summary.precision`,
|
|
`results-v2*.json` `precision`).
|
|
|
|
What this means:
|
|
|
|
- The `static` profile can be evaluated for six families, pooled for output_encoding, and not at all for memory or
|
|
prototype.
|
|
- The `plane` profile cannot be evaluated until verify has run on labelled units. The harvest Run in the commit
|
|
sequence (step 5) runs passes a and b on the 78 scorable, non-title pairs in the verify families (injection,
|
|
untrusted_destination, config_secrets; `verify.OPERATIONS`, `plane/tasks/verify.py:38-50`) on both sides, and on
|
|
the development cases. That is about 312 + 110 hunts at $0.014 each (validation-46: $0.8567 over 61 hunts),
|
|
**about $6**.
|
|
- PHP is reported as its own stratum only where it reaches 10 positive repositories. Today that is injection at
|
|
the boundary, and untrusted_destination is "insufficient data".
|
|
|
|
### 4. The decision engine: an AI classifier with local memory (Requirements 3, 7.1, 7.3, 7.4)
|
|
|
|
The engine is a declared language-model program, compiled DSPy-style, not a fitted statistical model (Req 3.1).
|
|
Maintainer, 2026-10-01: "We will not be able to scale up data" and "we need to have the decision engine on AI
|
|
classifier not ML. with local memory, dspy style". Choices: DSPy-style **in-house** (the DSPy library is not a
|
|
dependency), and retrieval by **signals + code embeddings**. DeepSeek is the LLM; OpenRouter serves embeddings
|
|
only (memory note "Model calling was broken 2026-09").
|
|
|
|
What it builds on:
|
|
|
|
- the metered client `MeteredClient` (`plane/budget.py:66-152`): usd and calls ceilings, `BudgetExhausted` ->
|
|
`unfinished`, `AccountError` -> `failed`, prices from a Model manifest (`prices_from`, `plane/budget.py:45-54`;
|
|
`plane/models/deepseek-flash.yaml`: $0.014 cache hit, $0.44 miss, $1.32 output per million tokens);
|
|
- the Model-bound task pattern of `plane/tasks/roles.py:141-241` (injected client, units log, restart skip, exit
|
|
codes 0/2/3), and `build_client` (`plane/tasks/verify.py:335`);
|
|
- `DeepSeekChatClient` (`model/endpoint.py:91-150`): thinking disabled so temperature is honoured, JSON mode with
|
|
one empty-content retry, optional logprobs (`ChatResponse.mean_logprob`, `tool_hunter.py:160, 287`);
|
|
- `OpenRouterEmbeddingClient.embed` (`provider/openrouter.py:137-170`) and `[embeddings] model` /
|
|
`OPENROUTER_EMBEDDING_MODEL` (`config.py:85-88, 294-298`; `.env.example`: `openai/text-embedding-3-small`);
|
|
- the feature record (`learn/features.py:78-115` `record`, `learn/schema.py:231` `validate_record`, the
|
|
`EXCLUDED_FIELDS` deny-list at `learn/schema.py:88-95`), roles (`learn/roles.py`), labels and groups
|
|
(`learn/labels.py:296-320` `Label`, `:182` `Groups`, floors at `:53-54`), and `function_span`
|
|
(`learn/features.py:234`) for excerpts;
|
|
- the store (`plane/memory.py:182-298`, `KINDS` at `:55-58`, `put_facts`/`get_facts` at `:225-262`, the 1 GiB
|
|
free-space refusal at `:347`) and the host seed of a Run's memory snapshot (`memory.seed`, `plane/memory.py:642`;
|
|
the sandbox cannot read the store, `plane/tasks/loop.py:8-11`).
|
|
|
|
#### 4.1 The program
|
|
|
|
**Signature** (`learn/program.py`, a frozen dataclass rendered into the prompt and used to parse the answer):
|
|
|
|
| Field | Direction | Type and bound |
|
|
| --- | --- | --- |
|
|
| `code` | in | the candidate function, numbered lines, at most 80 lines and 4,000 characters (section 4.2 excerpt rules); for a `delta` unit, also the base->head diff of the function, at most 40 lines |
|
|
| `signals` | in | the allow-listed record `x` and `instruments` states, rendered as `name: value` lines, `null` shown as "<instrument>: no coverage / failed / not applicable"; validated by `validate_record` before rendering; nothing from `EXCLUDED_FIELDS` |
|
|
| `roles` | in | inferred roles of the function (`learn/roles.py` wrapper roles; model roles in the `plane` profile): role, operation, line, origin (`ORIGINS`, `learn/schema.py:83`) |
|
|
| `family` | in | the candidate's family (from the rule or engine family; `unknown` allowed) |
|
|
| `verdict` | out | `vulnerable` \| `not_vulnerable` \| `unsure` |
|
|
| `family` | out | one of the families, or `none` |
|
|
| `confidence` | out | float in [0, 1], the model's stated probability that its verdict is right |
|
|
| `rationale` | out | at most 3 sentences; must cite at least one line number of `code` (`cited_lines`), and may name signals |
|
|
|
|
The answer is JSON (`json_object=True`). A reply that does not parse, or cites no line in range, is retried once
|
|
at temperature 0; if it still fails it becomes `unsure` with `parse_failed: true` in the decision row, never
|
|
`not_vulnerable` ("verify the instrument": a broken answer is not a clean result).
|
|
|
|
**Modules**, composed as `Program = Retrieve -> Classify(k)`:
|
|
|
|
- `Retrieve(record, excerpt, boundary) -> examples` (section 4.2), model-free except for one embedding of the
|
|
candidate excerpt (cached).
|
|
- `Classify(k)`: the prompt is `instruction + fixed demonstrations` (the compiled prefix, identical for every
|
|
candidate so the provider's prefix cache hits) `+ retrieved examples + the candidate`. Sample 1 runs at
|
|
temperature 0; samples 2..k at temperature 0.7, issued concurrently. `k` is a compiled setting in {1, 3, 5}.
|
|
- The raw score is `s = mean_j p_j` with `p_j = confidence_j` for `vulnerable`, `1 - confidence_j` for
|
|
`not_vulnerable`, and 0.5 for `unsure`/`parse_failed`. The reported verdict is the majority (ties -> `unsure`);
|
|
a majority `unsure` can never reach BLOCK.
|
|
|
|
**Client plumbing (small, explicit changes).** `MeteredClient.complete` (`plane/budget.py:117-136`) forwards only
|
|
`json_object`; it gains `temperature` and `logprobs` pass-through. `DeepSeekChatClient._call`
|
|
(`model/endpoint.py:127-142`) adds `temperature` to `extra` beside `thinking`. No provider seed is relied on.
|
|
Embedding calls get a metered wrapper `MeteredEmbeddingClient` in `plane/budget.py` that reads the response's
|
|
`usage.prompt_tokens` (OpenRouter returns it; `parse_embedding_response`, `provider/openrouter.py:173`, keeps only
|
|
the vectors today and is extended to keep usage) and prices it from an embedding Model manifest.
|
|
|
|
**Plane tasks.** One Model per Task (`egress.policy_for(task, workspaces, model)`, `plane/egress.py:87`), so:
|
|
|
|
- `embed` (new, Model `openrouter-embedding`, egress `openrouter.ai`): embeds the case's candidate excerpts
|
|
missing from the cache, writes `embeddings.jsonl`. Budget default `{usd: 0.05, calls: 20}` per case.
|
|
- `decide` (new, Model `deepseek-flash`): reads `features`, `roles`, `embeddings.jsonl` and the **seeded memory
|
|
snapshot** (the host writes the boundary-filtered examples and the adopted program into the Run's inputs, the
|
|
`loop snapshot` pattern), runs the program, writes `decisions.jsonl`. Budget default `{usd: 0.25, calls: 300}`
|
|
per case. Exit codes as `roles`.
|
|
|
|
A plane Run per case becomes `facts -> va, vb -> agree -> vc -> final -> alerts -> roles? -> features -> embed ->
|
|
decide -> remember`.
|
|
|
|
**Host path.** `learn/decide.py: decide_all(records, excerpts, program, memory, client, embedder, budget)` for
|
|
`scan` and `pre-push`, with the same program code. It needs `[models]` (a DeepSeek key) and either a store
|
|
(`OUSAST_MEMORY`) or the packaged program; otherwise the fallback (section 8).
|
|
|
|
#### 4.2 Local memory retrieval (Req 3.2, 3.4, 3.6)
|
|
|
|
**The example store.** One `example` row (a new kind in `KINDS`) per labelled candidate that has both a label
|
|
(`ousast learn labels`, `learn/labels.py`) and a feature record: envelope + `{candidate, family, language, profile,
|
|
label, source, group, frameworks, unit, x, instruments, roles, excerpt_sha, label_set_digest}`. The excerpt is a
|
|
content-addressed blob `excerpts/<sha>.txt`; the embedding is `embeddings/<model-slug>/<excerpt_sha>.json` (float32
|
|
vector in base64, model id, dimensions, tokens, created). Both use the new `put_blob(prefix, data) -> sha` /
|
|
`get_blob(prefix, sha)`, generalising `put_facts`/`get_facts`. Examples are built on the host by `ousast learn
|
|
memory build` (model-free) and `ousast learn memory embed` (metered). The snapshot: `verified_pin_labels` 424
|
|
positive / 436 negative rows in 105 groups, plus conditional negatives (benign_control 72, assumed_benign 4,671)
|
|
that become examples only once their condition (a candidate signal at the tip and none at the base) is evaluated
|
|
(`benchmarks/measurements/2026-09-30-decision-engine-labels/labels.json`). Rows name functions and hold third-party
|
|
code: the store is local, nothing is committed but counts.
|
|
|
|
**Excerpt rules** (`learn/excerpt.py`): the function body at the label's pin from the clone (`labels.Clone`,
|
|
`learn/labels.py:365`) via `function_span`; numbered lines; no path, repository or commit; comments kept, but
|
|
advisory ids (`_ADVISORY`, `learn/labels.py:59`) and lines matching `security_reason` (`learn/labels.py:277`)
|
|
inside comments are replaced by `<redacted>`. Candidate excerpts: at most 80 lines / 4,000 characters, centred on
|
|
the changed lines of a delta or the engine/quick hit line. Memory examples: at most 40 lines / 2,000 characters.
|
|
Memory examples carry **no rationale**: code, signals, roles, family and label only (cheaper, and no generated text
|
|
that could leak the label; only the compiled demonstrations carry rationales, section 4.3).
|
|
|
|
**Stage 1: signal-profile nearest neighbours.** A Gower distance over the allow-listed features of the active
|
|
profile (`features_for(profile)`, `learn/schema.py:163`), computed per instrument block so that "missing" is
|
|
explicit rather than imputed:
|
|
|
|
- For instrument `i`, if both records have `state == ran`: `d_i` = mean over its features of `d_f`, where a
|
|
numeric feature is `|a - b| / cap` (the `FeatureSpec.cap`; floats in [0, 1] use 1), a bool or enum is `0` if
|
|
equal else `1`, null in both is `0` and null in one is `1`.
|
|
- If the states differ (`ran` vs `none`/`failed`/`not_applicable`): `d_i = 1`. If both are the same non-`ran`
|
|
state: `d_i = 0`. `failed` vs `none` counts as different (`1`).
|
|
- `D = sum_i w_i d_i / sum_i w_i`, with `w_i = 1` for every instrument block in the profile (language is one
|
|
block). Prior features (`FeatureSpec.prior`) are dropped when `priors = off`. Weights are a compile-time
|
|
setting only if an A/B shows a gain; the default is uniform.
|
|
- Candidates in the eligible pool (after the boundary filter) are ranked by `D`, ties by `sha256(example_id +
|
|
seed)`; the first `n1 = 40` of the candidate's family and the first `n1` of the other families go to stage 2
|
|
(per side since 2026-10-01, so a minority family reaches stage 2; Risks, "First paid slice").
|
|
|
|
**Stage 2: code-embedding re-rank.** Cosine similarity of the candidate's excerpt embedding with each stage-1
|
|
example's. The re-rank score is `r = lam * (1 - D) + (1 - lam) * cos`, with `lam` a compiled setting in {0, 0.5}
|
|
(default 0.5). An example or candidate without an embedding (budget, provider failure) keeps its stage-1 order,
|
|
and the decision row records `retrieval: "signals_only"`. Embedding model: `openai/text-embedding-3-small` via
|
|
OpenRouter (1,536 dimensions; $0.02 per million input tokens at list price, confirmed by the smoke run's `usage`).
|
|
Cache key `sha256(normalised excerpt)` per embedding model, so the store is embedded once (~5,600 excerpts x ~900
|
|
tokens = ~5.0 M tokens, **about $0.10**) and each new candidate costs ~1,200 tokens (**$0.000024**).
|
|
|
|
**Balance rule** (`k_ret = 6`, a compiled setting in {4, 6, 8}): walk the re-ranked list and take examples subject
|
|
to: at most `k_ret/2` per label, at least 2 of each label when the pool has them; at most 2 per repository group;
|
|
at least `k_ret - 2` of the candidate's family when the pool has them (the rest from other families as contrast).
|
|
If the pool cannot satisfy a minimum, take what exists and record `balance: "short"`.
|
|
|
|
**The evaluation boundary (Req 3.4, 7.1).** `learn/retrieve.py: eligible(example, target, fold)` is the only way
|
|
an example enters a prompt, for retrieved examples and for the compiled demonstrations alike:
|
|
|
|
- `example.group` is not in the fold's evaluation groups (all of them, not just the target's), and not the
|
|
target's group after the CVE/alias merge (`Groups`, `learn/labels.py:182`);
|
|
- leave-one-source-out: `example.source` is not the held-out source;
|
|
- leave-one-framework-out on F: `F not in example.frameworks`; a compiled demonstration tagged F is swapped for its
|
|
pre-computed alternate (section 4.3);
|
|
- in evaluation, an example whose embedding cosine with the target is >= 0.98 is also dropped as a near-duplicate
|
|
(a vendored copy or fork the group merge missed); the count dropped is reported. In deployment it is kept.
|
|
- In deployment (`scan`, `pre-push`), a repository whose normalised name matches a corpus group excludes that group
|
|
from retrieval, and the report says "repository is in the corpus: its own cases excluded".
|
|
|
|
`tests/test_learn_retrieve.py::test_boundary` builds fixture folds (outer, leave-one-source-out,
|
|
leave-one-framework-out) and asserts, for every target and fold, that every retrieved id and every demonstration id
|
|
in the rendered prompt is eligible; a second test asserts that `memory.seed` for a plane Run writes no example of
|
|
the case's group. A pre-call assertion in `Classify` repeats the check on the ids actually rendered.
|
|
|
|
**New labels enter memory (Req 3.6).** Adding an example row changes retrieval at once; nothing is recompiled. Each
|
|
decision records the `memory_snapshot_digest` (`improve.memory.index_digest`, `improve/memory.py:351`) so the
|
|
effect of new labels is an A/B with two memory snapshots as arms (section 6).
|
|
|
|
#### 4.3 Compilation, DSPy-style and in-house (Req 3.3)
|
|
|
|
`ousast learn compile --profile P --spec learn/compile.toml` produces a compiled program from a frozen data
|
|
snapshot. The compile spec is committed before the compile runs (pre-registration).
|
|
|
|
**Compile split.** 25% of repository groups (seeded, stratified by family) form the compile split `C`. `C` is
|
|
never an evaluation fold; it remains memory for the evaluation of other repositories. Inside `C`, groups are split
|
|
60/40 into `C_boot` (demonstration source) and `C_val` (scoring). One compile per profile, not one per outer fold:
|
|
nested compiles per fold would cost five times as much, and `C` being disjoint from every evaluation fold keeps the
|
|
reported numbers out-of-repository.
|
|
|
|
**Metric (pre-registered, `compile.toml`).** Primary: the **balanced Brier score** of the raw score `s` on `C_val`,
|
|
each (family, label) cell weighted equally and each repository capped at 5 effective rows per cell (the weight rule
|
|
of the earlier design). Lower is better; `unsure` and `parse_failed` score as 0.5. Secondary (reported, not
|
|
optimised): recall at precision 0.90 and the share of `unsure`. Tie (within 0.005): fewer prompt tokens wins.
|
|
|
|
**Bootstrap few-shot.** Run the program with no demonstrations (retrieval on, `k = 1`, temperature 0) on up to 150
|
|
candidates of `C_boot`. Keep as demonstration candidates those whose verdict is right with `confidence >= 0.7`
|
|
and whose rationale cites a line in range and names no identity (repository, path, CVE id: scanned and refused).
|
|
The rationale was produced without seeing the label, so a demonstration teaches reasoning, not the answer. From the
|
|
pool, draw 4 demonstration sets of `m = 3` (one positive, one negative, one of either; families mixed), seeded. For
|
|
every demonstration, an alternate from a different framework is stored for leave-one-framework-out folds.
|
|
|
|
**Instruction search.** One proposal call gives the model the signature, a glossary of the signals, and 20
|
|
summarised bootstrap errors (signals and verdicts, no code), and asks for 6 candidate instructions. With the
|
|
hand-written baseline instruction that is 7. Scoring on `C_val` (60 candidates, `k = 1`): the 7 instructions with
|
|
demo set 1, then the best instruction with demo sets 2-4: 10 configurations x 60 = 600 calls. Then `k` in {1, 3, 5}
|
|
and `lam` in {0, 0.5} are **not** searched in the compile: they are A/B variants (section 6), so compile spend stays
|
|
bounded.
|
|
|
|
**Budget.** About $2.2 per compile (section 4.6); ceiling **$3.00** per compile through `MeteredClient`. A compile
|
|
that hits its ceiling records `unfinished` and produces no artifact.
|
|
|
|
**Reproducibility (Req 3.3).** Every model and embedding response is cached as `responses/<sha>.json`, keyed by
|
|
`sha256(model id, Model parameters digest, messages, temperature, sample index, json_object)`. Every random choice
|
|
(compile split, demo draws, tie-breaks, subsets) is seeded from `compile.toml`. Re-running a compile on the same
|
|
store snapshot with the cache reproduces the artifact byte for byte at $0 (`test_learn_compile.py::
|
|
test_reproducible_from_cache`, on FileStore and on MinioStore gated as today). Without the cache, temperature 0 is
|
|
not a determinism guarantee at the provider; the artifact records the cache digest so a re-run can say whether it
|
|
was replayed or re-asked.
|
|
|
|
#### 4.4 Folds and calibration (Req 3.5, 7.1, 7.4)
|
|
|
|
**Folds.** Every split is by `group` (repository), across all corpora; the label builder's merges hold.
|
|
|
|
- *Outer*: grouped K = 5 over the groups outside `C`, stratified by family and label, **one** repetition (cost);
|
|
intervals come from the repository-cluster bootstrap, not from repetitions.
|
|
- *Leave-one-source-out* and *leave-one-framework-out* as in the earlier design (each population and pair slice
|
|
held out whole; framework F held out where F has >= 10 groups). The label snapshot has no framework at 10 groups
|
|
today (`by_framework`: express 4, flask 2, django 1), so leave-one-framework-out is expected to print
|
|
"insufficient data" until the corpus grows; it costs nothing until then.
|
|
- The v2 fold stays marked *design-informed*.
|
|
- Families below the floors (`PER_FAMILY_FLOOR = 20`, `POOLED_FLOOR = 10`, `learn/labels.py:53-54`) are not
|
|
evaluated at all and get no engine decision: memory, config_secrets and unknown today (385 rows not spent on).
|
|
|
|
**Calibration.** The raw score `s` is mapped by Platt scaling on `logit(clip(s, 0.01, 0.99))`. It is
|
|
**cross-fitted** over outer folds: the map applied to fold j is fitted on the out-of-fold predictions of the other
|
|
folds (each of which was itself made with memory excluding its own fold), so no map is fitted on the fold it is
|
|
checked on. Per family at the per-family floor; pooled with a per-family intercept between the floors. Isotonic and
|
|
binned maps are not used: too few positives per family. The adopted map is refitted on all out-of-fold
|
|
predictions. The check is unchanged: reliability table (5 quantile bins), slope and intercept with
|
|
repository-bootstrap CIs, expected calibration error. **Holds** when the slope CI contains 1, the intercept CI
|
|
contains 0 and ECE <= 0.05; otherwise ADVISORY only, recorded as `block.offered = false, reason = "calibration"`.
|
|
The probability stays "calibrated on the labelled mix"; the prevalence-free figures (recall at a point, false
|
|
BLOCKs per 100 benign deltas, precision at pi in {0.01, 0.05}) remain primary.
|
|
|
|
Confidence source alternatives (each an A/B variant, not a default): the verdict token's probability from
|
|
`logprobs` (no extra cost), and the verbal confidence alone at `k = 1`. The default is the self-consistency mean
|
|
at `k = 5` if the smoke run measures prefix-cache hits on samples 2..k (hit share >= 0.8 of their prompt tokens);
|
|
otherwise `k = 1` with verbal confidence, because uncached repeats cost 5x (section 4.6).
|
|
|
|
#### 4.5 Operating points and explanations (Req 7.3, 6.1)
|
|
|
|
Operating points are read from the calibrated probability, chosen on the cross-fitted predictions of the folds
|
|
other than the one reported (nested, as before):
|
|
|
|
- **BLOCK**: the lowest threshold whose precision has a Wilson 95% lower bound >= 0.95 among delta units, and the
|
|
majority verdict is not `unsure`. If none exists, `block.offered = false, reason = "precision_unreachable"`.
|
|
- **ADVISORY**: the highest threshold reaching recall >= 0.90. Its precision is reported, not assumed.
|
|
- **DROP**: below ADVISORY.
|
|
|
|
Both are reported on outer folds with recall, precision and false BLOCKs per 100 benign deltas, each with a
|
|
repository-cluster bootstrap 95% interval (2,000 resamples).
|
|
|
|
Each decision carries: the calibrated `p`, the operating point, the program id, the majority `rationale` and its
|
|
`cited_lines`, the signals the rationale names (checked against the record; names not in the record are dropped),
|
|
the retrieved neighbours as counts by label and family (never identities), and `retrieval` mode. These replace the
|
|
logistic "top contributing features" of the earlier design in reports (Req 6.1).
|
|
|
|
#### 4.6 Cost model
|
|
|
|
Prices: `deepseek-flash` $0.014 / $0.44 / $1.32 per million cache-hit / miss / output tokens; embeddings $0.02 per
|
|
million. Token sizes are estimates to be replaced by the smoke run's measured `usage`:
|
|
|
|
| Prompt part | Tokens | Cached |
|
|
| --- | --- | --- |
|
|
| instruction | ~500 | yes (shared prefix) |
|
|
| 3 compiled demonstrations (40-line excerpt, signals, label, rationale) | ~2,250 | yes |
|
|
| 6 retrieved examples (~750 each) | ~4,500 | no |
|
|
| candidate (80-line excerpt ~1,100, signals and roles ~300) | ~1,400 | no |
|
|
| output (JSON, rationale <= 3 sentences) | ~200 | -- |
|
|
|
|
- **Per candidate, `k = 1`**: 2,750 x 0.014 + 5,950 x 0.44 + 200 x 1.32 per million = **$0.0029**.
|
|
- **Each extra sample** (same prompt, fully cached): 8,700 x 0.014 + 200 x 1.32 per million = **$0.0004**. So
|
|
`k = 5` is **$0.0045** with cache hits, and $0.0146 if the cache does not hit (the smoke run decides the default).
|
|
- **Embedding**: $0.000024 per candidate; the whole store about $0.10 once.
|
|
- **Per compile**: bootstrap 150 x $0.0029 = $0.44; proposals ~$0.01; scoring 600 x $0.0029 = $1.74; total
|
|
**~$2.2, ceiling $3.00**.
|
|
- **Per full evaluation** of one program, evaluated families only (475 verified rows outside memory,
|
|
config_secrets and unknown) plus a 200-unit benign-delta sample for false BLOCKs:
|
|
- outer folds: 675 candidates, **$2.0 at k = 1, $3.0 at k = 5** (cached);
|
|
- leave-one-source-out: 475 more, **$1.4 / $2.1**;
|
|
- leave-one-framework-out: $0 today (no framework at the floor);
|
|
- total **~$3.4 (k = 1) to $5.2 (k = 5)**, ceiling $7.
|
|
- **Learning curves** (section 7): 6 memory-size points + 3 source-count points on a fixed 200-candidate subset at
|
|
`k = 1`: 9 x 200 x $0.0029 = **~$5.2**, ceiling $7.
|
|
- **A program-variant A/B** on 300 paired candidates: arm A replays cached responses from the evaluation ($0);
|
|
arm B costs **$0.9 (k = 1) to $1.35 (k = 5)**.
|
|
- **Deployment**: `pre-push` with ~5 candidates is ~$0.02 per push at `k = 5`; a plane Run case with ~30 candidates
|
|
~$0.14.
|
|
|
|
**Against the balance.** DeepSeek was ~$19.6 before the harvest; the harvest's verify passes are ~$6 expected and
|
|
model roles up to their $10 ceiling, so **~$4-14 remains** depending on what the harvest actually spends (read the
|
|
provider balance after it ends; do not assume). Affordable now, as one go-ahead of **~$4.5-5.5 expected, $7 ceiling**:
|
|
the smoke run (20 candidates, ~$0.10, measures cache hits and real token counts), embedding the store (~$0.10,
|
|
OpenRouter, separate account), one compile (~$2.2), and the outer-fold evaluation at the cheaper confirmed `k`
|
|
(~$2-3). Leave-one-source-out, the learning curves, program A/Bs and exp-001 (~$7.7) together need **~$18-20 more**
|
|
and a top-up. No single run may start whose ceiling exceeds the balance read just before it.
|
|
|
|
### 5. Compiled program artifacts and versioning (Requirement 3.3)
|
|
|
|
A compiled program is one canonical JSON (sorted keys, floats with `repr`), a content-addressed blob
|
|
`programs/<sha256>.json`, with `programs/index.jsonl` (every version) and `programs/adopted.json` (profile -> sha;
|
|
a single overwritten object, versioned under MinIO, history in the index on FileStore). On MinIO the object is
|
|
tagged `kind=program` and `profile`.
|
|
|
|
```
|
|
{"program_id": "<sha256>", "kind": "llm-program", "schema_version": 1, "feature_set_digest": "...",
|
|
"profile": "static" | "plane", "signature": {"digest": "...", "fields": [...]},
|
|
"instruction": "<text>", "instruction_source": "baseline" | "proposed:<n>",
|
|
"demos": [{"example_id", "excerpt_sha", "label", "family", "rationale", "alternate": "<example_id>"}],
|
|
"retrieval": {"distance": "gower-v1", "n1": 40, "k": 6, "lam": 0.5, "balance": {...},
|
|
"embedding_model": "openai/text-embedding-3-small", "near_duplicate_cos": 0.98,
|
|
"excerpt": {"candidate_lines": 80, "candidate_chars": 4000, "example_lines": 40, "example_chars": 2000}},
|
|
"classify": {"model": "deepseek-flash", "model_params_digest": "...", "k": 5, "temperature": [0.0, 0.7],
|
|
"confidence": "vote_mean" | "verbal" | "logprob", "max_output_tokens": 300},
|
|
"calibration": {"method": "platt-crossfit", "per_family": {...}, "check": {...}},
|
|
"operating_points": {"<family>": {"block": {"threshold", "offered", "reason", "target_precision": 0.95},
|
|
"advisory": {"threshold", "target_recall": 0.90}}},
|
|
"priors": "off" | ["wordpress", ...], "roles": "off" | "deterministic" | "model",
|
|
"data": {"memory_snapshot_digest", "label_set_digest", "compile_groups_digest", "sources": [...], "excluded": {...}},
|
|
"compile": {"spec_sha256", "metric": "balanced_brier_v1", "configurations": [{"instruction", "demo_set", "score"}],
|
|
"usd", "calls", "response_cache_digest", "replayed": true | false},
|
|
"evaluation": {"folds_digest", "outer": {...}, "leave_one_source_out": {...}, "leave_one_framework_out": {...},
|
|
"strata": {...}, "curves": "curves/<sha>.json"},
|
|
"instruments": {"ruleset_digest", "engine_image", "runner_image", "verify_model", "prompt_digest", "roles_model"},
|
|
"seed": 0, "code_commit": "<git sha>", "created": "<iso>"}
|
|
```
|
|
|
|
The memory itself is not in the artifact; `memory_snapshot_digest` pins which memory the evaluation used. A
|
|
decision made later with a different memory snapshot records its own digest.
|
|
|
|
**Adoption** stays a maintainer commit: `ousast learn adopt <program_id>` writes `programs/adopted.json` and exports
|
|
the artifact to `src/openultrasast/ruleset/decision/program-<profile>.json`. The packaged copy holds demonstrations
|
|
with third-party code excerpts: only demonstrations from repositories with a recorded permissive licence are
|
|
exported; others are replaced by their alternates or dropped, and the exported program is re-evaluated as its own
|
|
variant (the memory-size-0 configuration of section 7 if no store ships). A program whose priors or roles setting
|
|
differs from the incumbent is adopted only with a recorded A/B result (section 6).
|
|
|
|
### 6. A/B experiments (Requirement 4)
|
|
|
|
**Manifest.** `plane/experiments/<id>.yaml`, a companion file, is referenced by the Run annotation
|
|
`openultrasast.io/experiment: <id>`. Each arm's Tasks carry `openultrasast.io/arm: A|B`. Annotations are stripped
|
|
before ax (`plane/manifests.py:4-6`), as `memory-key` is.
|
|
|
|
```
|
|
id: exp-001-known-callers
|
|
hypothesis: removing the known-callers line changes the rate at which verify flags a true site
|
|
arms: {A: {task_env: {}}, B: {task_env: {OUSAST_VERIFY_CALLERS: "off"}}} # everything else identical
|
|
unit: candidate # candidate | file | repository
|
|
pairing: paired # every unit in both arms
|
|
cluster: repository # resampling unit for intervals
|
|
order: {randomise: per_case, seed: 1} # which arm runs first per case
|
|
metric: {primary: "flag_any_pass_rate(positive units)", secondary: ["flag_any_pass_rate(fixed-pin negatives)", "usd_per_candidate"]}
|
|
test: {primary: "cluster_bootstrap_paired_diff", secondary: "exact_mcnemar", alpha: 0.05, sides: 2}
|
|
mde: 0.15
|
|
stopping: {looks: [0.5, 1.0], boundary: "obrien_fleming"}
|
|
budget_usd: {total: 15, per_arm: 7.5}
|
|
units_file: plane/experiments/exp-001-known-callers.units.jsonl # frozen list, digest recorded
|
|
```
|
|
|
|
`ousast plane experiment register <id>` checks the manifest, freezes the units file, and records the sha256 of
|
|
both as an `experiment` row (a new kind, with repo `experiments/<id>`). It refuses unless both files are committed
|
|
at `HEAD`. `ousast plane run` refuses a Run with an experiment annotation whose manifest or units digest differs
|
|
from the registered one.
|
|
|
|
**Randomisation and pairing.** Paired is the default: every unit runs in both arms, and a seeded per-case coin
|
|
decides which arm's tasks run first, so provider drift over a run does not align with arms. Only when a variant
|
|
cannot run on the same unit (for example, a change to candidate generation itself) are repositories assigned to
|
|
arms by a seeded, family-stratified shuffle. Assignment is recorded in the units file.
|
|
|
|
**Outcomes.** The `experiment-outcome` task (no model), after the arms, writes one `arm_outcome` row per (unit,
|
|
arm): outcome, usd from `unit_cost`, and pass details. Spend per arm is also in the plane's attribution table
|
|
(`ousast plane status`).
|
|
|
|
**Statistics.**
|
|
|
|
- *Primary*: the paired difference B - A in the per-unit outcome, with a 95% CI from a repository-cluster
|
|
bootstrap (10,000 resamples of repositories, with the units inside each). Units in one repository are
|
|
correlated, so a unit-level test would overstate certainty.
|
|
- *Secondary*: the exact McNemar test on discordant units (a binomial test with p = 0.5), reported beside it.
|
|
- *Sequential*: two looks, at half and all of the repositories (in a pre-shuffled order), with O'Brien-Fleming
|
|
boundaries (|z| >= 2.797 at the first look, 1.977 at the final). Stopping for futility is not binding.
|
|
- *Decision*: **adopt B** if the final CI excludes 0 in B's favour, and **reject B** if it excludes 0 against it.
|
|
For a cost-only change, **equivalent** means the TOST 90% CI of the detection difference lies within +-0.05 and
|
|
the cost CI shows a saving. Otherwise **inconclusive**, and the incumbent stays.
|
|
- *Engine-level experiments* compare **compiled program variants**: priors on/off, roles, `k` in {1, 3, 5}, the
|
|
confidence source (vote mean, verbal, logprob), `lam` in {0, 0.5}, `k_ret`, comments kept vs stripped in
|
|
excerpts, two instructions, two memory snapshots (Req 3.6: the effect of new labels), or a different chat model.
|
|
They are no longer free. The arms are two programs run on the same candidates in the same outer folds (paired),
|
|
the unit is the candidate and the resampling cluster the repository. Arm A replays the incumbent's cached
|
|
responses from its evaluation ($0); arm B is paid: about $0.9-1.35 per 300 paired candidates (section 4.6), with
|
|
the arm's ceiling in the manifest's `budget_usd`. The metric is recall at BLOCK (or at ADVISORY where BLOCK is
|
|
not offered), plus false BLOCKs per 100 benign deltas; the balanced Brier score is secondary. A variant that only
|
|
changes compile-time choices is compiled on the same compile split with the same seed, so the two arms differ in
|
|
exactly the declared setting. Because model calls are stochastic, each arm records its response-cache digest; a
|
|
re-analysis replays both arms at $0.
|
|
|
|
**Where results land.** An `experiment_result` row, and `benchmarks/experiments/<id>/result.json` (arms, units,
|
|
the estimates at each look, decision, spend, manifest digest, code commit) in the same commit that records the
|
|
decision. Adoption of a winning pipeline variant is a maintainer commit that changes the default (a Task
|
|
template's env, or a `[models]` value) and cites the result's digest. The memory proposer's M1/M2 status edits
|
|
(`improve/memory.py`) become one variant family: a proposed status change runs as an engine-level experiment
|
|
instead of going straight to the benchmark gate.
|
|
|
|
**First experiment, fully specified (Req 4.4).** The question: `plane-increment-2.json` records declared sites
|
|
flagged in any pass as 17 for the plane vs 19 for the reference, of 20. The plane added the known-callers line
|
|
(`plane/tasks/verify.py:1-9, 164`, `CALLERS_CAP = 8` at :34), and the comparison is confounded (other changes,
|
|
stochastic passes, v2 is where it was seen).
|
|
|
|
- **Arms.** A is `verify` as today. B is `verify` with `OUSAST_VERIFY_CALLERS=off`, a new switch (about 3 lines)
|
|
that makes `callers_line` return empty. Same model (`deepseek-flash`), `PER_HUNT = 6`, `MAX_STEPS = 6`, same
|
|
candidates, same `facts.json`, same runner image digest, two passes (a, b) per arm.
|
|
- **Units: the development corpus, not v2** (v2 generated the hypothesis):
|
|
- population v1, 11 cases, whose engine scans are recorded under `~/ousast-results/independent-v1/scans/` (44
|
|
scans);
|
|
- pmpro, MediaWiki, WP Statistics (dev-php);
|
|
- the 27 full-repository pointer pairs in the verify families (`recipe` pairs of vibe-py and agent-vfc,
|
|
tier != title).
|
|
|
|
That is 41 repositories. No development repository has recorded verify candidates today. They are generated
|
|
model-free and frozen in the units file: per pin, the fix-changed functions, plus the top engine- and
|
|
quick-alert functions, up to 8 candidates per pin, on the vulnerable and the fixed pin. Only units where arm A's
|
|
callers line is non-empty are *treated*. The primary analysis is restricted to them, and the rest are reported.
|
|
- **Metric.** Primary: the per-positive-unit rate of "flagged in any pass" (the 17-vs-19 metric), with positives
|
|
from the label builder (fix-range functions on the vulnerable pin). Secondary: the flag rate on fixed-pin
|
|
negatives (false flags), and usd per candidate.
|
|
- **Sample size and MDE.** About 55 positive units across 41 repositories (about 28 dev + 27 pointer pairs). With
|
|
a discordance rate of about 0.15, alpha 0.05 and power 0.8, the paired MDE is `(1.96 + 0.84) * sqrt(0.15) /
|
|
sqrt(55)`, about **0.15**. The observed gap of 0.10 needs about 115 positive units, so an inconclusive result is
|
|
the likely and honest outcome. It is registered as such in advance, and a follow-up needs new units, not a
|
|
continuation.
|
|
- **Budget.** Hunts per pass per arm: 14 dev repositories x 2 pins x up to 3 files (84) + 27 pairs x 2 sides (54)
|
|
= 138. Four arm-passes give 552 hunts x $0.014, **about $7.7 expected**. The ceiling is **$15 total, $7.5 per
|
|
arm**, enforced by the task budgets (`plane/budget.py`). Candidate generation reuses the recorded v1 engine
|
|
scans. For dev-php it runs the engine once per pin on the host, one container at a time, from a frozen source
|
|
export.
|
|
|
|
### 7. Extrapolation and the one-time v3 check (Requirement 5)
|
|
|
|
**Learning curves over memory size (5.1).** `ousast learn curve --profile P` varies what the program may
|
|
retrieve, not what it was compiled with (the compiled instruction and demonstrations stay fixed, so the curve
|
|
measures memory):
|
|
|
|
- *By memory size*: the eligible memory restricted to a random share of its repository groups, 0%, 25%, 50% and
|
|
100%, with two seeded draws at 25% and 50% (6 points). 0% is the program with its demonstrations only, which is
|
|
also the configuration a user without a store gets (section 5).
|
|
- *By sources*: memory restricted to 1, 2 and all of the label sources (pairs, populations, adjudications), 3
|
|
points.
|
|
- Every point is evaluated on the same fixed, stratified 200-candidate subset of the outer folds, at `k = 1`, with
|
|
the boundary filter applied (about $5.2 for all 9 points, ceiling $7; section 4.6).
|
|
|
|
Per point: recall at ADVISORY, recall at BLOCK (where offered), PR-AUC, balanced Brier score, false BLOCKs per 100
|
|
benign deltas, and the share of `unsure`, per family where the family has the floor. Bands are the 2.5/97.5%
|
|
quantiles over the draws x a repository-cluster bootstrap. An inverse power law `m(n) = a - b * n^-c` in the
|
|
number of memory groups is fit to each curve; a projection to 1.5x and 2x the memory is reported only as a band,
|
|
and only where the bootstrap CI for `c` excludes 0. A flat curve is a finding: it says memory does not help that
|
|
family and the demonstrations carry the result.
|
|
|
|
**Expected detection on unseen code (5.2).** Per family and per language (and per framework, Req 8.3): the
|
|
outer-fold and leave-one-source-out intervals at both operating points, with priors off and with adopted priors.
|
|
A stratum with fewer than 10 positive repositories or 10 negative repositories, or whose 95% interval on recall is
|
|
wider than 0.5, is printed as **"insufficient data (n repositories)"**, with no number. Today that applies to
|
|
memory, prototype, PHP untrusted_destination, and every plane-profile family until the harvest Run.
|
|
|
|
**The v3 check (5.3), exactly once.**
|
|
|
|
1. *Before.* The engine and every operating point are adopted, and population v3 is still reserved and unread by
|
|
this spec's code (it is absent from `learn/sources.toml`). `ousast learn predict --population-name
|
|
population-v3-php` writes `benchmarks/independent/prediction-v3.json` and it is committed:
|
|
- the adopted program id per profile, its `memory_snapshot_digest` and the chat and embedding model ids;
|
|
- `feature_set_digest`, priors and roles settings;
|
|
- the operating points;
|
|
- the frozen analyzer source hash (`benchmarks/push/freeze_source.sh`);
|
|
- for each family v3 may contain, the predicted recall and precision intervals from the PHP stratum where
|
|
sufficient, else the pooled-language interval with the label "no PHP stratum";
|
|
- the predicted false BLOCKs per 100 benign deltas;
|
|
- the reference to `protocol-v3.md`, written by the population's owner, not by this spec.
|
|
|
|
Only the population's name is needed; its file is not opened.
|
|
2. *Run.* v3 is scored under protocol v3, once, by its owner's runner, with the engine's decisions added to the
|
|
recorded outputs. The engine's spend on v3 is a model cost (about $0.0045 per candidate at `k = 5`); its
|
|
ceiling is fixed in `prediction-v3.json` from the candidate count the owner reports, never by opening v3 files
|
|
from this spec's code, and needs the maintainer's go-ahead.
|
|
3. *After.* `benchmarks/independent/result-v3-engine.json` holds, per family, the measured point and interval
|
|
beside the predicted interval, `inside | outside`, the calibration check on v3, and the reach misses. v3 is
|
|
then added to `learn/sources.toml` as spent, and its cases may enter memory. Its numbers never again
|
|
qualify anything.
|
|
|
|
### 8. Integration (Requirements 6, 7.3, 7.5)
|
|
|
|
**Lookup.** The compiled program comes from, in order:
|
|
|
|
1. `[decision] program` in the config (a path);
|
|
2. the store's `programs/adopted.json` when `OUSAST_MEMORY` is set (with the store's memory for retrieval);
|
|
3. the packaged `ruleset/decision/program-<profile>.json` (demonstrations only, no retrieval memory unless a store is
|
|
configured; its own calibration, section 5);
|
|
4. none.
|
|
|
|
The program also needs a chat Model: `[models]` with a DeepSeek key on the host, the bound Model in a plane Run.
|
|
Embeddings are optional: without an OpenRouter key retrieval runs `signals_only` and says so.
|
|
`learn/decide.py`: `load_program(...) -> Program` and `decide_all(records, excerpts, program, memory, client,
|
|
embedder, budget) -> list[Decision(p, operating_point, program_id, verdict, rationale, cited_lines, signals,
|
|
neighbours, retrieval, offered)]`, all calls through `MeteredClient`.
|
|
|
|
**`ousast scan`.** A `decide` stage runs after the findings are complete and before `findings.json` is written
|
|
(`cli.py:517-520`). It builds `static` records with `build_for_scan` from the quick findings, the engine result,
|
|
`analyze_entry_points` and the `repo-facts` functions (source-only), plus deterministic roles. It scores each
|
|
candidate and attaches `decision` to the finding: `{p, operating_point: block|advisory|drop, program_id, verdict,
|
|
rationale, cited_lines, signals, offered}`. Candidates at or above ADVISORY are reported. Dropped ones go to
|
|
`dropped_by_engine.json`, beside `shadow_findings.json`. Shadow-status findings enter as candidates too, so a rule's
|
|
enabled/shadow status changes the report only if the decision changes (Req 6.3). The shadow split at
|
|
`cli.py:503-508` applies only in the fallback. `--fail-on blocked` is a new choice (exit 1 on any BLOCK), and
|
|
`findings` keeps its meaning. Markdown, SARIF (`properties.decision`) and `manifest.json` (`decision: {program_id,
|
|
usd, fallback, reason}`) carry the fields.
|
|
|
|
**`ousast pre-push`.** Candidates are the `CandidateDelta`s from `compare_evidence`, enriched with quick-rule
|
|
hits and deterministic roles on the changed functions of head and base. `admit_candidates` (`push/policy.py:879`)
|
|
gains `engine: Models | None`:
|
|
|
|
- With a model, the capability-eligibility reasons `capability_unavailable`, `capability_ambiguous`,
|
|
`capability_disabled`, `capability_unevaluated` and `operation_semantics_mismatch` are replaced by the engine's
|
|
reasons: `engine_below_block`, `engine_block_not_offered` (family insufficient or calibration failed), or none.
|
|
- Every structural reason stays a hard requirement: `comparison_provenance_mismatch`, `insufficient_rung`,
|
|
`unchanged`, `change_unsupported`, `witness_unresolved`, `location_unresolved`, `family_semantics_mismatch`,
|
|
`claim_discharged` and `dependency_unresolved` (`push/policy.py:909-960`). `PushReport` invariants
|
|
(`push/report.py:48-97`) require witnesses, locations, consequence and repair for actionable defects.
|
|
- With no capability, consequence and repair come from family-level templates shipped beside the model:
|
|
language-level text, never framework text.
|
|
- `--mode blocking` blocks only on BLOCK. `--mode advisory` lists ADVISORY and above. Each disposition carries
|
|
`{p, operating_point, program_id, verdict, rationale, cited_lines}` in the artifact and one line in the compact report
|
|
(`render_report`, `push/report.py:214`).
|
|
- The 30 s default deadline (`cli.py`, `--deadline 30.0`) covers feature building and `decide`. Everything in the
|
|
`static` profile is source-only or already computed by the scan. Model roles run only with `--model-config`.
|
|
`decide` issues candidates and their `k` samples concurrently (at most 8 requests in flight); a candidate not
|
|
decided by the deadline is reported `undecided: deadline` and listed as ADVISORY-unscored, never dropped and never
|
|
blocked. A push's decide budget defaults to `{usd: 0.10, calls: 120}`.
|
|
|
|
**Plane Runs.** The `embed` and `decide` tasks (Model-bound, section 4.1) write `embeddings.jsonl` and
|
|
`decisions.jsonl`; the host seeds `decide` with the adopted program and the case's boundary-filtered memory;
|
|
`remember` stores `decision` rows, and `ousast plane status` prints BLOCK/ADVISORY counts per case.
|
|
|
|
**Fallback (6.2).** With no adopted program, no chat Model or key, an exhausted budget, or a family below the
|
|
floor, today's behaviour applies unchanged: rule
|
|
statuses, quick/engine findings as they are, pre-push with the empty capability registry. The report says so in
|
|
one line: "decision engine: not used (no adopted program | no model key | budget exhausted | family insufficient data |
|
|
calibration failed) --
|
|
findings are today's rule output". A family whose BLOCK is not offered still gets ADVISORY decisions.
|
|
|
|
**User adaptation (7.5)** is opt-in: `[decision] adapt = true`. It keeps the user's dismissals in
|
|
`.openultrasast/decisions/dismissals.jsonl` in their repository, as finding fingerprints with the program id and the
|
|
excerpt sha. Dismissed fingerprints are suppressed after the engine, and, with at least 10 dismissals, they enter a
|
|
**user-local memory** (examples labelled `not_vulnerable`, `origin: user`) that `Retrieve` reads for that repository
|
|
only, beside the corpus memory. It never writes to the memory store; the label builder and the example builder
|
|
refuse any row with `origin: user`. It is evaluated separately: on held-out repositories, by replaying the first half
|
|
of their labels as dismissals and measuring the second half, and reported as "adaptation", never merged into the
|
|
generalisation numbers.
|
|
|
|
## Data Models
|
|
|
|
- `features` row: envelope (`id, kind, repo, pin, run, task, population, split, image`) + `{candidate, family,
|
|
language, profile, schema_version, feature_set_digest, unit, base?, x, instruments}`.
|
|
- `label` row: envelope + `{candidate, family, label, source, source_ref, unit, direction?, weight, split, group,
|
|
frameworks, created, evidence}`. Written only by `ousast learn labels`.
|
|
- `example` row: envelope + `{candidate, family, language, profile, label, source, group, frameworks, unit, x,
|
|
instruments, roles, excerpt_sha, label_set_digest}`. Written only by `ousast learn memory build`; refuses
|
|
`origin: user` and any source not in `sources.toml`.
|
|
- `decision` row: envelope + `{candidate, family, profile, program_id, memory_snapshot_digest, p, raw_score,
|
|
operating_point, offered, verdict, votes, rationale, cited_lines, signals, neighbours: {label: n, family: n},
|
|
retrieval: "signals+embeddings" | "signals_only", parse_failed, usd, calls}`.
|
|
- `experiment` row: `{experiment, manifest_sha256, units_sha256, registered, code_commit}`. `arm_outcome` row:
|
|
`{experiment, arm, unit, outcome, passes, usd, response_cache_digest}`. `experiment_result` row: `{experiment,
|
|
look, estimate, ci, mcnemar_p, decision, spend}`.
|
|
- `KINDS` (`plane/memory.py:55-58`) already holds `features, label, decision, experiment, arm_outcome,
|
|
experiment_result` (task 1) and gains `example`. `TAG_FIELDS` is unchanged (`family` is already a tag).
|
|
- Blobs: `programs/<sha>.json`, `programs/index.jsonl`, `programs/adopted.json`, `excerpts/<sha>.txt`,
|
|
`embeddings/<model-slug>/<sha>.json`, `responses/<sha>.json`, `curves/<sha>.json`, `labels/<sha>.jsonl`.
|
|
- Models: `plane/models/openrouter-embedding.yaml` (provider extension `openrouter`, egress `openrouter.ai`,
|
|
`secretKey` OPENROUTER_API_KEY, `parameters.input_per_m: 0.02`, cache and output rates 0).
|
|
- Files: `learn/sources.toml`, `learn/compile.toml`, `ruleset/frameworks.toml`, `ruleset/decision/program-*.json`,
|
|
`plane/experiments/<id>.yaml` + `.units.jsonl`, `benchmarks/experiments/<id>/result.json`,
|
|
`benchmarks/independent/prediction-v3.json`, `result-v3-engine.json`.
|
|
|
|
## Error Handling
|
|
|
|
- Every failure of the feature builder is loud. An instrument that did not read its input records `state:
|
|
failed`, and a Run's `features` task fails when every instrument failed for a case. A plausible all-zero record
|
|
is impossible because zero requires `state: ran`.
|
|
- `validate_record` rejects unknown features, out-of-vocabulary enums and label fields, naming the feature and the
|
|
source file. The prompt renderer refuses a record that did not pass it.
|
|
- The label builder exits 2 on a source that is not in `sources.toml`, a population whose status is not recorded,
|
|
or a group that straddles an outer fold after the merges. The error names the repository.
|
|
- The example builder fails a candidate whose excerpt is empty or whose function span is not found (it never
|
|
stores an empty excerpt as an example), and reports the count per source; `memory build` exits 2 if more than 10%
|
|
of a source's labels have no excerpt (an unread clone is not a small memory).
|
|
- Compile and evaluation refuse a family below the floor and write an "insufficient data" stub, so `decide` can say
|
|
why. They assert before every call that no rendered example or demonstration is outside the boundary (section
|
|
4.2), and before a compile that the compile split and the evaluation folds share no group.
|
|
- A model answer that does not parse is retried once, then recorded `unsure, parse_failed`. A run where more than 5%
|
|
of answers are `parse_failed` is reported as an instrument failure, not as a result.
|
|
- A smoke check precedes every paid run: 3 candidates, the provider's `usage` present and non-zero, the reply
|
|
parsed, the embedding dimension as declared. A run whose first calls report zero tokens stops (memory note
|
|
"Model calling was broken 2026-09": a working-looking zero).
|
|
- `BudgetExhausted` ends a compile or evaluation `unfinished` with no artifact and no number; `AccountError` ends it
|
|
`failed` (`plane/budget.py:22-38`).
|
|
- `decide` with a program whose `schema_version`, `feature_set_digest` or signature digest differs from the
|
|
builder's falls back, with the reason in the report, and never renders a mismatched record.
|
|
- Experiments: a registration mismatch refuses the Run. A budget ceiling ends an arm `unfinished` (`budget.py`),
|
|
and the analysis then reports "incomplete: arm B stopped at $x" and records no decision.
|
|
- `put_blob` on FileStore applies the 1 GiB free-space refusal (`plane/memory.py:347`).
|
|
|
|
## Testing Strategy
|
|
|
|
No test makes a model or embedding call: the chat and embedding clients are scripted fakes (the
|
|
`tool_hunter.ChatClient` protocol, `index.EmbeddingClient` at `index.py:56`) that record what was asked.
|
|
|
|
- `tests/test_learn_schema.py`: allow-list, name invariance, label fields excluded, missing vs zero (section 1).
|
|
- `tests/test_learn_labels.py`, on fixture populations and pairs (done, task 4):
|
|
- positives and negatives per rule, including a fixed-pin function the fix did not touch (not a negative), and
|
|
an unmatched vulnerable-pin candidate (unlabelled);
|
|
- `unscorable` and title-tier exclusion;
|
|
- an adjudication mapping;
|
|
- a source not in `sources.toml`, which cannot label, checked with a fixture population named like v3 and a test
|
|
that asserts the builder never opens its file (`open` monkeypatched);
|
|
- CVE-based group merge.
|
|
- `tests/test_learn_excerpt.py`: bounds (lines, characters), centring on changed lines, no path or repository in
|
|
the excerpt, advisory ids and security wording in comments redacted, delta diff rendering.
|
|
- `tests/test_learn_retrieve.py`:
|
|
- Gower distance on hand-computed fixtures: both `ran`, state mismatch = 1, same non-`ran` state = 0, null in one
|
|
= 1, prior features dropped with `priors = off`;
|
|
- re-rank with `lam` in {0, 0.5}, `signals_only` without embeddings;
|
|
- the balance rule (label, family and per-repository caps; `short` recorded);
|
|
- `test_boundary`: for outer, leave-one-source-out and leave-one-framework-out fixture folds, every rendered
|
|
example and demonstration is eligible; `memory.seed` writes no example of the case's group; the near-duplicate
|
|
drop in evaluation only; the deployment self-exclusion.
|
|
- `tests/test_learn_program.py`: the rendered prompt contains no `EXCLUDED_FIELDS` name or label value; the shared
|
|
prefix is byte-identical across candidates; parsing (valid, invalid -> one retry -> `unsure, parse_failed`;
|
|
cited line out of range); majority and score `s`; `unsure` cannot BLOCK; `MeteredClient` receives `temperature`
|
|
and stops at its ceiling; the response cache key and replay.
|
|
- `tests/test_learn_compile.py`: bootstrap keeps only correct, confident, identity-free rationales; instruction
|
|
candidates scored by the balanced Brier metric with ties to fewer tokens; compile split disjoint from evaluation
|
|
folds; `test_reproducible_from_cache` (byte-identical artifact on FileStore, and on MinioStore gated as today);
|
|
ceiling -> `unfinished`, no artifact.
|
|
- `tests/test_learn_folds.py`: no group in both an evaluation fold and the compile split or retrieval pool, for
|
|
outer, leave-one-source-out and leave-one-framework-out folds; determinism by seed.
|
|
- `tests/test_learn_calibrate.py`: Platt on a known miscalibration, cross-fitting (fold j's map never saw fold j),
|
|
the slope/intercept/ECE check, and the ADVISORY-only downgrade.
|
|
- `tests/test_learn_decide.py`: operating points from cross-fitted predictions, explanation fields (signals named in
|
|
the rationale but absent from the record are dropped), version-mismatch fallback, deadline -> `undecided`.
|
|
- `tests/test_learn_experiments.py`:
|
|
- registration digest and refusal;
|
|
- paired bootstrap CI on a fixture with a known difference;
|
|
- exact McNemar values against hand-computed binomials;
|
|
- O'Brien-Fleming boundaries;
|
|
- adopt / reject / inconclusive / equivalent decisions;
|
|
- a program-variant experiment whose arm A replays cached responses at $0.
|
|
- `tests/test_semantic_priors.py` (done, task 2): the lint test, `priors="off"` removes exactly the tagged entries,
|
|
the split entries keep their language-level calls.
|
|
- `tests/test_learn_roles.py` (done, task 3): wrapper inference to depth 3 on a fixture, and unclassified chunks are
|
|
never "no roles".
|
|
- `tests/test_plane_features.py`, `test_plane_embed.py`, `test_plane_decide.py`: fixture run directories from the
|
|
validation-46 shapes. `features` drops `site_match`/`in_fix_range`; `embed` and `decide` meter, honour budgets and
|
|
exit 0/2/3; `remember` stores `features`/`decision` rows.
|
|
- `tests/test_push_decision.py`: admission with an engine replaces only the capability reasons, and every
|
|
structural reason still blocks admission. `PushReport` invariants hold with engine templates.
|
|
- `tests/test_scan_decision.py`: the scan fallback line (no program, no key, budget), `dropped_by_engine.json`, and
|
|
that a shadow/enabled status flip without a decision change leaves the report unchanged.
|
|
- Regression net: the existing gate, pair and plane tests. `test_benchmark_output_is_byte_identical_to_golden_baseline`
|
|
holds with no program adopted (the fallback is today's path).
|
|
- Joern-gated tests run only with the mount recipe (memory note "Joern-gated tests run nowhere"). The engine-alert
|
|
harvest is verified by its own read-count check, not assumed.
|
|
|
|
## Requirement Traceability
|
|
|
|
| Requirement | Design section | Verified by |
|
|
| --- | --- | --- |
|
|
| 1.1 one record per candidate from every instrument | 1 | `test_plane_features.py`, `test_learn_schema.py` |
|
|
| 1.2 missing is not negative | 1 (instrument state, `null`), 4.2 (distance) | `test_missing_is_not_zero`; `test_learn_retrieve.py` |
|
|
| 1.3 versions and digests for rebuild | 1 (versioning), 5 | `test_learn_compile.py::test_reproducible_from_cache` |
|
|
| 2.1 positives/negatives by source | 3 | `test_learn_labels.py` |
|
|
| 2.2 plane verdict is a feature, not a label | 1, 3 | `test_label_fields_never_reach_features`; label rules |
|
|
| 2.3 provenance; reserved populations unusable | 3 (`sources.toml`, fail-closed) | `test_learn_labels.py` (v3-named fixture never opened) |
|
|
| 3.1 declared LM program on the metered client; no statistical model | 4.1 | `test_learn_program.py`, `test_plane_decide.py` |
|
|
| 3.2 retrieval by signals re-ranked by code embeddings; no identities | 4.2 | `test_learn_retrieve.py`, `test_learn_excerpt.py` |
|
|
| 3.3 compiled on training folds; versioned; reproducible | 4.3, 5 | `test_learn_compile.py` |
|
|
| 3.4 retrieval never crosses the evaluation boundary | 4.2 (boundary), 4.4 | `test_learn_retrieve.py::test_boundary`, `test_learn_folds.py`, pre-call assertion |
|
|
| 3.5 confidence calibrated on held-out repositories; else ADVISORY only | 4.4 | `test_learn_calibrate.py` |
|
|
| 3.6 new labels improve via memory, measured by A/B | 4.2, 6 | memory-snapshot experiment; `memory_snapshot_digest` on decisions |
|
|
| 4.1 pre-registered experiment | 6 (manifest, register) | `test_learn_experiments.py` |
|
|
| 4.2 paired, assignment and spend recorded | 6 | `arm_outcome` rows; `ousast plane status` |
|
|
| 4.3 adopt only on a CI excluding zero or equivalence | 6 (decision rule) | `test_learn_experiments.py` |
|
|
| 4.4 known-callers experiment first | 6 (exp-001) | task 7 record |
|
|
| 5.1 learning curves over memory size and sources, with bands | 7 | `ousast learn curve` output; unit test on a fixture |
|
|
| 5.2 intervals per family and language; insufficient data | 7 | `test_learn_decide.py` (floor), report fixture |
|
|
| 5.3 v3 checked once, after freezing | 7 | `prediction-v3.json` committed before `result-v3-engine.json` |
|
|
| 6.1 probability, operating point, contributing evidence in reports | 4.5, 8 | `test_scan_decision.py`, `test_push_decision.py`, `test_plane_decide.py` |
|
|
| 6.2 fallback said in report | 8 | `test_scan_decision.py` |
|
|
| 6.3 rule status no longer changes output alone | 8 | status-flip test |
|
|
| 7.1 repository-grouped folds across corpora | 3 (group, CVE merge), 4.4 | `test_learn_folds.py` |
|
|
| 7.2 repository-agnostic allow-list with test | 1, 4.2 (excerpt rules) | `test_learn_schema.py`, `test_learn_excerpt.py` |
|
|
| 7.3 BLOCK/ADVISORY on unseen repositories; delta unit | 1 (delta), 4.5, 8 | outer-fold report; `test_push_decision.py` |
|
|
| 7.4 calibration checked out of repository; else ADVISORY only | 4.4 | `test_learn_calibrate.py` |
|
|
| 7.5 opt-in adaptation outside the numbers | 8 | builders refuse `origin: user`; separate report |
|
|
| 8.1 language-level knowledge; roles inferred | 2 | `test_learn_roles.py` |
|
|
| 8.2 tagged priors, off by default, kept by A/B | 2, 6 | `test_semantic_priors.py`; program-variant experiment records |
|
|
| 8.3 leave-one-framework-out, per framework | 2, 4.4 | `test_learn_folds.py`; report strata |
|
|
| 8.4 share depending on priors | 2, 7 | report columns priors off / adopted / difference |
|
|
|
|
## Commit sequence, mapped to future tasks
|
|
|
|
Each commit is gated on the full suite's own exit code, `ruff` and `mypy`. None changes a reported number until
|
|
commit 9 adopts a program, and without an adopted program every path is today's. Steps that spend model money
|
|
start only after the maintainer confirms their budget against the balance read just before.
|
|
|
|
1. **Schema and features** (1.1-1.3, 7.2) -- done.
|
|
2. **Framework tags** (8.2, 8.4) -- done.
|
|
3. **Roles** (8.1) -- done.
|
|
4. **Labels** (2.1-2.3, 7.1) -- done.
|
|
5. **Harvest** (1.1, data) -- in progress: verify a/b on the verify-family pairs and the development cases (about
|
|
$6, ceiling $10), model roles on the candidate files (ceiling $10), engine features on the host.
|
|
6. **Program, memory, compile, first evaluation** (3.1-3.6, 5.1-5.2, 7.3-7.4, 8.3): excerpts and the example
|
|
store, metered embeddings and the cache, retrieval with the boundary test, the program and response cache, folds,
|
|
compile, cross-fitted calibration; then the paid slice (smoke, embed, compile, outer evaluation, ~$5, ceiling $7)
|
|
and, after a second go-ahead, leave-one-source-out and the memory-size curves (~$7, ceiling $10), committed under
|
|
`benchmarks/measurements/<date>-decision-engine-v1/`.
|
|
7. **Experiments** (4.1-4.4): `learn/experiments.py`, `register`, `experiment-outcome`, the Run annotation checks,
|
|
`OUSAST_VERIFY_CALLERS`, exp-001 registered, run and recorded, and the program-variant experiment machinery.
|
|
8. **Integration** (6.1-6.3, 7.3, 7.5): the `embed` and `decide` tasks with the seeded memory, the `scan` stage,
|
|
`pre-push` admission with the engine, report fields, the fallback line, and the opt-in adaptation layer.
|
|
9. **Adopt** (3.3, 8.2): `ousast learn adopt`, the packaged program with the licence filter, the priors-on/off and
|
|
`k` experiments with recorded decisions.
|
|
10. **v3 check** (5.3): `prediction-v3.json` is committed first, then v3 is run by its owner under protocol v3, and
|
|
`result-v3-engine.json` is committed. Nothing between the two commits may touch the program, memory snapshot,
|
|
features, priors or thresholds.
|
|
|
|
## Risks
|
|
|
|
- **Label scarcity decides what can be evaluated.** The status of each family today:
|
|
- memory (5 repository groups, 173/173 rows, almost all openssl and curl), config_secrets (5) and prototype are
|
|
**insufficient data** (`labels.json` `verified_pin_labels`). Users get today's behaviour there, labelled as
|
|
such, and no money is spent evaluating them.
|
|
- PHP has no pairs and 13 positive population groups. The v3 prediction will mostly be the pooled interval,
|
|
marked "no PHP stratum", and it will be wide.
|
|
- The plane profile has 43 rows from one population until the harvest Run.
|
|
- BLOCK at 95% precision is likely **not reachable** for most families on this data. The design reports "BLOCK
|
|
not offered" instead of lowering the target.
|
|
- **Weak signals, strong code.** The pair baseline has quick-rule labelled recall 0.24 and Youden -0.013 overall
|
|
(`pairs.json` `overall`); the v2 engine recall is 0/17 (`results-v2.json`). The program reads the code, so it
|
|
can beat the signals, but the verify history says the model over-flags (v1 1/12, v2 0/17 adjudicated true). A
|
|
calibrated classifier that mostly says "advisory" is honest; recall at ADVISORY 0.90 may need a threshold with
|
|
poor precision. That is reported, not tuned away on outer folds.
|
|
- **LLM nondeterminism.** The same prompt can answer differently, and temperature 0 is not a provider guarantee.
|
|
Guards: every response cached and replayable (numbers reproduce at $0); sample 1 at temperature 0; the vote mean
|
|
over `k` samples as the score; the A/B unit is paired and both arms' caches are recorded; a re-ask agreement rate
|
|
on a fixed 30-candidate canary set is reported with each evaluation.
|
|
- **Label leakage into prompts.**
|
|
- Through rationales: memory examples carry no rationale; demonstration rationales are produced without the label
|
|
and scanned for identities and advisory ids; the candidate's prompt never holds `EXCLUDED_FIELDS` (test).
|
|
- Through code: fixed-side excerpts can carry fix comments ("prevent XSS", a CVE id). Advisory ids and security
|
|
wording in comments are redacted; a comments-stripped arm measures what remains. Delta excerpts show the diff of
|
|
the change under test, never the later fix.
|
|
- Through memorisation: the chat model may have seen public CVE fixes in pre-training. Not removable; the v3
|
|
check (unpublished fixes are rarer there) and per-source numbers expose it, and the report names it.
|
|
- **Retrieval leakage.** Same repository under another name or corpus: group normalisation, CVE merge, the
|
|
near-duplicate drop at cosine >= 0.98 in evaluation, and `test_boundary`. Demonstrations from the compile split
|
|
are, by construction, from repositories never evaluated. The seed for plane Runs filters on the host, where the
|
|
labels are; the sandbox never receives an example of its own case's group.
|
|
- **Label-tracking signals (maintainer decision 2026-10-01: "Neutralise both for v1").** The harvest found two
|
|
signals that follow the pair label instead of the code: verify candidates on fixed sides named functions the side
|
|
no longer declares (now never produced: a side's candidates are only the functions it declares, the rest counted
|
|
`absent_on_side`), and the engine's "read the file, asked no question" state, which a fixed side reaches by losing
|
|
its sink. The v1 program withholds the engine instrument entirely -- state and values -- from prompts,
|
|
demonstrations, the instruction proposer and the retrieval distance (input profile `v1`, `ProgramSpec.inputs`);
|
|
the stored record keeps it. The engine returns only through a pre-registered A/B evaluated on assumed-benign and
|
|
benign-control negatives, where it cannot track a fix. `ousast learn audit-leaks` reports how well every signal
|
|
alone separates the sides of a pair and flags any above 0.20 (counts under `benchmarks/measurements/`).
|
|
- **Earlier leakage paths still apply.** Hand rules and framework priors written after seeing a training repository
|
|
(the `$wpdb` rule from PMPro, the hook source from WP Statistics, `taint.sc:911-949`): priors off by default,
|
|
leave-one-framework-out, best-effort `taught_by = [repo]`. The mechanism exporter's train-on-test leak
|
|
(`closed-loop-train-on-test-leak.md`): no exporter output is a feature. Verify and tie-break fitted on v2: the v2
|
|
fold is design-informed. Thresholds and instructions chosen on reported folds: cross-fitting and the compile
|
|
split. v3: fail-closed sources, never opened. User dismissals: refused by both builders.
|
|
- **Cost.** Model spend is now per candidate and per evaluation, not one-off. Measured (section 4.6) the first
|
|
slice fits today's balance; everything after it needs a top-up. Guards: `MeteredClient` ceilings on every task
|
|
and command, a smoke check that measures real tokens and cache hits before any paid run, cached replay for
|
|
re-analysis, unevaluable families skipped, `k = 1` if the prefix cache does not hit. Deployment cost (~$0.02 per
|
|
push) is a user-facing change: `pre-push` without a key falls back, and the report states the spend.
|
|
- **Model and version drift.** `deepseek-flash` is an alias whose weights can change, and so can OpenRouter's
|
|
embedding route. The artifact records model ids, the Model parameters digest and the response `model` field;
|
|
before adoption and before the v3 prediction the canary set is re-asked, and agreement below 0.9 with the
|
|
cached answers, or an embedding dimension change, invalidates the evaluation (re-embed, re-evaluate) rather than
|
|
being averaged in. Prices are data in the Model manifest; a price change changes only cost reports.
|
|
- **Latency.** A model call per candidate is slower than a dot product. The 30 s `pre-push` deadline is met by
|
|
concurrency and the `undecided: deadline` state; a push with many candidates gets partial decisions, stated.
|
|
- **Packaging third-party code.** Demonstrations and memory hold excerpts of corpus repositories. Only
|
|
permissively licensed excerpts may ship in the package; the store is never shipped. A user without a store gets
|
|
the demonstrations-only program, whose own curve point (memory 0%) and calibration are what they are told.
|
|
- **Framework inference is itself a model.** Model roles cost money and can be wrong. `scan` without `[models]`
|
|
gets only wrapper inference, whose reach on framework-heavy code without dependency source is poor. The report
|
|
says which role source ran. Both are measured by the roles-on/off experiment. Nothing assumes they help.
|
|
- **Host time on the 7 GB host.** Engine features remain the time cost (7-14 hours serial for the pair corpus,
|
|
dev-php scans of 200-2,000 s each), run once per engine image from a frozen source export, one container at
|
|
`--memory 3g` at a time, never beside the kind cluster's verify pods, cached by (content sha, image id). Model
|
|
calls add wall time but no memory pressure; excerpts and embeddings are small (~8 KB per example with a 1,536-float
|
|
vector, ~45 MB for 5,600 examples).
|
|
- **Experiments will often be inconclusive.** With 41 development repositories the MDE is about 15 points.
|
|
Pre-registering that and recording "inconclusive" is the intended behaviour. The failure mode to avoid is
|
|
extending an experiment after looking.
|
|
|
|
- **First paid slice, 2026-10-01 (tasks 6.2, 6.6; injection only).** Four instrument defects surfaced before any
|
|
number was believed, each fixed with a test:
|
|
- *The memory build joined no pair.* The harvest stores a pair side's record under its unit pin (the blob sha1 of
|
|
the side's excerpt), not under `label_pin`; `ousast learn memory build --units <harvest units.json>` joins them.
|
|
And a candidate's `static` and `plane` records share a key, so reading both let one overwrite the other and
|
|
silently drop about half the examples; the build now reads its own profile only. 85 -> 724 examples.
|
|
- *Contrast was unbounded.* The balance rule padded a short pick with other families (up to 5 of 6), and stage 1
|
|
took the 40 nearest of all families, so a minority family often never reached stage 2. Now stage 1 takes the
|
|
`n1` nearest of the candidate's family and the `n1` nearest others, and at most 2 contrast examples are shown
|
|
whenever the pool holds the family (maintainer: "allow 2 contrast"); never padded.
|
|
- *Proposed instructions dropped the answer contract*, and DeepSeek's JSON mode refuses a prompt without the word
|
|
"json" (HTTP 400, the first compile stopped there). The contract (`ANSWER_FORMAT`) is now part of the signature
|
|
and appended to any instruction that lacks it; the baseline prefix is byte-identical.
|
|
- The adjudication source fails the 10% excerpt gate: 9 of its 43 candidates are anonymous functions (`<lambda>N`,
|
|
`<module>`) that `function_span` cannot find by name; the clones were read. Open.
|
|
The memory is small: 724 static examples (only harvested candidates have feature records; the 97 advisory-fix pairs
|
|
have none yet), injection 140 (75/65), so `C_val` had 13 injection candidates and the compile metric is noisy.
|
|
The measured ranking signal is real but weak at the recall target: AUC 0.74 out of repository, ADVISORY at recall
|
|
0.90 has precision 0.54 (prevalence 0.53); the majority `vulnerable` verdict alone has precision 0.86, recall 0.43.
|
|
Canary re-ask agreement at temperature 0 was 0.83, below the 0.9 the adoption check requires.
|
|
|
|
## Maintainer decisions at design approval (2026-09-30)
|
|
|
|
- Design approved as written.
|
|
- Benign pushes: ordinary non-security commits are mined as **assumed-benign** pushes (no security keyword, no later
|
|
fix touching the same lines), kept as their own label source (`assumed_benign`), reported separately from the
|
|
verified negatives, and never merged with them.
|
|
|
|
## Maintainer decisions for the Requirement 3 change (2026-10-01)
|
|
|
|
- The decision engine is an AI classifier with local memory, not ML: "We will not be able to scale up data" and
|
|
"we need to have the decision engine on AI classifier not ML. with local memory, dspy style".
|
|
- DSPy-style **in-house** (signature, modules, bootstrap few-shot, instruction search on the plane's metered
|
|
client), not the DSPy library.
|
|
- Retrieval by **signals + code embeddings**: signal-profile nearest neighbours re-ranked by OpenRouter code
|
|
embeddings. DeepSeek is the LLM; OpenRouter is used for embeddings only.
|
|
|
|
Open for the maintainer at design approval:
|
|
|
|
- The first paid slice of task 6 (smoke, embed, one compile, outer evaluation): ~$5 expected, $7 ceiling, against
|
|
the balance read after the harvest.
|
|
- The second slice (leave-one-source-out, memory-size curves): ~$7, ceiling $10, and the top-up it needs.
|
|
- Which corpus licences allow demonstrations to ship in the package (section 5).
|
|
|
|
|
|
- 2026-10-01: revised design approved; packaged demonstrations limited to permissive licences (MIT, BSD, Apache-2.0,
|
|
ISC, zlib); copyleft and unknown-licence code is used for evaluation and local memory only, never packaged.
|