diff --git a/README.md b/README.md index a985c4a..7caf5cb 100644 --- a/README.md +++ b/README.md @@ -251,6 +251,8 @@ shell, Docker or a free-form command. Documentation site with flow diagrams (`docs/`, `mkdocs.yml`): `uv sync --extra docs && uv run mkdocs serve`. +- [docs/where-we-stand.md](docs/where-we-stand.md): the state in one page: capability against the + gates, the approach step by step, ax on the server, token economics, open items. - [docs/architecture.md](docs/architecture.md): the scan pipeline, evidence ladder, CWE policy and project score, and the self-improving loops. - [docs/evaluation.md](docs/evaluation.md): independent populations, the qualification gates, diff --git a/docs/decision-engine.md b/docs/decision-engine.md index ebe17cb..83f4e72 100644 --- a/docs/decision-engine.md +++ b/docs/decision-engine.md @@ -7,11 +7,9 @@ design is the `learned-decision-engine` specification (maintainer tooling); the ## Why -Every detection gain so far was a hand edit made after a miss: a rule, a taint vocabulary entry, -a guard, a prompt line, a tie-break. Each was tuned on the cases that exposed it, and engine -tuning measured on population v1 did not transfer to v2 (see [evaluation.md](evaluation.md)). -The engine replaces that with a decision learned from labelled outcomes and checked out of -repository. +Every detection gain before the engine was a hand edit tuned on the cases that exposed it, and +engine tuning measured on population v1 did not transfer to v2. The reasoning, and the step-by-step +chain that replaced hand edits, are on [Where we stand](where-we-stand.md#2-our-approach-to-detection-step-by-step). ## Design @@ -100,16 +98,19 @@ bootstrap intervals (2,000 resamples); input profile v1, priors off, k = 5. evaluation; demonstrations are now swapped or dropped like retrieved examples, and the stopped part was replayed after the fix. -**Not adopted.** Still ahead: leave-one-source-out, leave-one-framework-out, the remaining -learning-curve points (memory size 0/25/50%, number of sources), integration into scan and -pre-push, adoption, and the one-time v3 check. +**Not adopted.** What is still ahead is listed once, on +[Where we stand](where-we-stand.md#5-open-items). ## Experiments -Changes to prompts, models, pass counts or features are compared by controlled experiments, so -nothing is adopted on one run's number. One is registered and has not run: exp-002 (variance -reduction of the score). No result exists; this page carries one only once a record under -`benchmarks/measurements/` does. +Changes to prompts, models, pass counts, features or sampling are compared by controlled +experiments from a pre-registered manifest (`plane/experiments/.yaml`), so nothing is adopted +on one run's number. Two are registered. exp-001 (known callers) has not run. exp-002 (a retrieval +ensemble against today's temperature sampling) ran on 160 paired candidates in 56 repository +groups and was **rejected**: within-pair AUC 0.815 against 0.741 and $0.0021 against $0.0051 per +candidate, with re-run agreement 0.875 against 0.906 and an interval spanning zero +(`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`). The reading is on +[Where we stand](where-we-stand.md#2-our-approach-to-detection-step-by-step). ## Commands diff --git a/docs/evaluation.md b/docs/evaluation.md index ee303a3..8f26a35 100644 --- a/docs/evaluation.md +++ b/docs/evaluation.md @@ -112,10 +112,10 @@ gate: at least 16 of 20 declared sites agreed, at a cost per candidate under $0. ## The decision engine -Two slices of the learned decision engine have been measured out of repository: injection alone -(2026-10-01) and the six evaluable families (2026-10-02: pooled AUC 0.77 to 0.96 per family, -within-pair AUC 0.72 to 0.93, BLOCK offered for none). The numbers and their records are in -[decision-engine.md](decision-engine.md). The engine is not adopted. +Two slices have been measured out of repository (injection, 2026-10-01; six families, +2026-10-02). The engine is not adopted. The state against the gates, in one place with the plane +increment and the populations above, is on [Where we stand](where-we-stand.md#1-detection-capability-and-goals-no-go-measured); +the per-family tables are on [decision-engine.md](decision-engine.md). ## Removal of the earlier agentic extra diff --git a/docs/index.md b/docs/index.md index c8d98d3..12b939c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -45,6 +45,9 @@ flowchart LR ## Where to go next +- The current state in one page, every figure with its record: capability against the M4 gates, the + step-by-step approach, ax on the server, token economics before and after, and the open items: + [Where we stand](where-we-stand.md). - Run a scan: [Examples](examples.md), and the commands in [Scanning](scanning.md). - Understand a finding's evidence and score: [Architecture](architecture.md). - Operate the plane and its store: [ax on this host](ops/ax/README.md) and diff --git a/docs/token-ergonomics.md b/docs/token-ergonomics.md index e56b965..9ea9e5d 100644 --- a/docs/token-ergonomics.md +++ b/docs/token-ergonomics.md @@ -10,6 +10,10 @@ paid for; the development half is tooling that keeps bulk file contents out of t Every number on this page comes from a committed record, named next to it. Records are counts only; none holds code, prompts or responses. +The before-and-after summary (script pipeline against plane, and the development share) is on +[Where we stand](where-we-stand.md#4-token-economics-before-and-after); this page holds the +mechanisms and the detail table. + ## A. Runtime: the plane's token economy ```mermaid diff --git a/docs/where-we-stand.md b/docs/where-we-stand.md new file mode 100644 index 0000000..b51cfa4 --- /dev/null +++ b/docs/where-we-stand.md @@ -0,0 +1,258 @@ +# Where we stand + +State as of 2026-10-02 (v2.0.0). This page answers five questions in one place. How far is +detection from its gates. How does the project now try to close that gap. How does the model work +run on ax. What did the tokens cost before and after that move. What is still open. Every figure +is quoted from a committed record whose path stands next to it; detail tables stay on the pages +linked from each section. + +The goal has not changed: a safety net on repositories the tool has never seen, which says +BLOCK or ADVISORY about the code a push would change and stays silent on fixes and on benign +commits. Two independent populations have been spent against that goal, with engine recall 1 of +11 and then 0 of 17. The hand edits that closed misses on the first population did not carry to +the second. So the question is no longer which rule to add next. It is how a change to detection +can be shown to work on code it was not tuned on, at a cost per candidate that can be paid for +every push. + +The answer this page records: the pre-push safety net is NO-GO against every M4 gate, and the +project has stopped trying to reach the gates by hand. Every detection change now has to win a +controlled measurement on held-out repositories, run on an agentic plane whose cost per candidate +is known. The pieces of that method exist and have been measured. None of them has produced a +BLOCK decision yet. + +## 1. Detection capability and goals: NO-GO, measured + +No M4 gate is met on an independent population. The qualification milestone for the pre-push +safety net (M4) requires, together: at least 95% actionable precision, at least 90% supported +recall, at least 95% supported-check completion, the runtime gates, and zero fixed-side or benign +false alerts. A population is frozen before any +case is scanned, scored once under a pre-registered protocol, then spent; it can inform changes +but can qualify nothing afterwards ([Evaluation](evaluation.md#independent-populations-and-the-m4-gates)). + +| Gate | Required | Latest independent result | Record | +| --- | --- | --- | --- | +| Supported recall | >= 90% | engine 0 of 17 declared sites on population v2 | `benchmarks/independent/results-v2.json` | +| Actionable precision | >= 95% | engine 0 of 17 (`precision`); the tool hunter on the same population 2 of 14 | `results-v2.json`, `results-v2-hunter.json` | +| Supported-check completion | >= 95% | 13,206 of 108,374 | `results-v2.json` | +| Fixed-side and benign alerts | zero | 0 fixed-side, which the record calls met trivially since nothing was detected; 3 benign alerts | `results-v2.json` | + +Three later measurements narrow the picture without changing the verdict. + +- The plane reproduces the script pipeline's agreement at a known cost. On the model-driven + pipeline's validation set (46 candidates, 15 cases of v2, 20 declared vulnerable sites), the + plane's verify passes a and b plus a tie-break on the 10 disputed candidates agreed on 16 of 20 + declared sites. The script reference agreed on 16 as well. The cost was $0.0205 per candidate + (`benchmarks/independent/plane-increment-2.json`). The record carries two caveats. The tie-break + was chosen after seeing the first measurement on the same set, so this compares plane with + script and qualifies nothing. And the passes flag fewer declared sites than the reference in + any pass, 17 against 19; that count is a correction in `plane-increment-1.json`, whose first + reading took agree's `declared_sites_matched` (19, coverage by candidates) for "found in either + pass". +- The learned decision engine ranks, but may not block. On six families, evaluated on + repositories the compiled program was not shown, the pooled AUC of its score runs from 0.77 + (injection) to 0.96 (access control). The stricter reading is the within-pair AUC, which asks + whether the vulnerable side of a fix scores above its own fixed side: 0.72 to 0.93. BLOCK is + offered for no family; calibration holds only for output encoding, and there no fold reaches + the required precision (`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`). + The per-family table is on [Decision engine](decision-engine.md#second-slice-six-families-2026-10-02). +- PHP coverage is in-sample. The PHP quick rules were written and tightened while looking at + the pins and fixtures they are scored on, so the record says so itself (`provenance.in_sample` in + `benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`): a development check, not + a recall estimate. On real PHP code the Joern engine is the instrument. + +What has not been touched: population v3, 15 PHP cases, frozen as the one-time final check. No +development or analysis work reads its cases, a test fails when anything in the tree names one +of its repositories, and it runs once, after a prediction is committed. + +## 2. Our approach to detection, step by step + +The path changed because hand edits did not transfer. Every gain before 2026-09-30 was a hand +edit after a miss: a rule, a taint vocabulary entry, a guard, a prompt line, a tie-break. Each was +tuned on the cases that exposed it, and the tuning measured on population v1 did not carry to v2 +(1 of 11, then 0 of 17). The conclusions drawn: hand edits do not transfer. The data that could +teach a better decision cannot be produced by hand at the scale needed, so it has to be harvested +and labelled by machinery with a boundary around it. Framework knowledge (which function is a sink in which +framework) is useful as a prior but must stay optional, or the engine learns the vocabulary +instead of the code. A reader might object that a classifier is slower to show gains than the +next rule. It is. The difference is that its gains are measured where rules were not: on +repositories it never saw. + +The chain, one job per step. The design is on [Memory and detection](memory-and-detection.md); +this is the order of operations. + +1. Instruments produce signals, never verdicts. Quick rules, the Joern engine, model sink + classification, the plane's verify passes a and b with the tie-break c, repository facts + (callers, spans) and model source/sink/sanitizer roles each write one block of a candidate's + record. None decides alone. +2. Feature records are allow-listed, and missing is not zero. A candidate is `(path, function, + family)` of one repository and pin. An instrument that did not run, had no coverage or failed + records that state; "silent" and "did not look" stay distinguishable. Input profile v1 withholds + the engine block and the function length, which a leak audit found separate the two sides of a + pair by construction (`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`), and + since 2026-10-02 the plane's `verify.flag_b` and `ms.operation`, flagged the same way by the + audit of the plane profile (`2026-10-02-decision-engine-harvest-advisory/record.json`, + `src/openultrasast/learn/schema.py`). +3. Labels come from ground truth only. A fail-closed allow-list (`src/openultrasast/learn/sources.toml`) + names the sources: the spent populations' declared sites and fix ranges, the vulnerable and + fixed sides of pairs, including the 97 real public fixes in `benchmarks/pairs/advisory-fixes/`, + benign controls and recorded adjudications. Assumed-benign commits are their own source, + weighted 0.5 and reported separately. A plane verdict is a feature, never a label: "rejected by + the plane" is not a negative, or the engine would learn to reproduce the plane's mistakes. +4. Retrieval finds similar labelled cases. A Gower distance over the signal profile, re-ranked + by cosine similarity of code embeddings (OpenAI `text-embedding-3-small` through OpenRouter, + cached by excerpt hash). The evaluation boundary is the only door into a prompt: never the + candidate's own repository group, never a group the fold evaluates. +5. A classifier compiled in house, DSPy-style. `retrieve -> classify(k)`: an instruction and + compiled demonstrations (a fixed prefix, so the provider's cache hits), the retrieved examples, + then the candidate's excerpt and signals. The model answers vulnerable, not vulnerable or + unsure with a rationale that cites a line; `k` samples give a score. One program is compiled + per profile and family on a compile split that is never evaluated. +6. Calibration on held-out repositories. Platt scaling of the score, cross-fitted by + repository group so no map is checked on the fold that fitted it. Calibration holds when the + slope interval contains 1, the intercept interval contains 0 and ECE <= 0.05. +7. Two operating points. BLOCK only where calibration holds and a fold reaches the required + precision; otherwise ADVISORY. An unsure majority can never BLOCK. +8. A/B experiments are the only way a change is adopted. Prompts, models, pass counts, + features and sampling are compared under a pre-registered manifest with frozen units, paired + arms, a repository bootstrap and two looks. exp-002 is the first to finish: a retrieval + ensemble (five samples over disjoint example sets at temperature 0) against today's + temperature sampling. It raised re-run agreement a little (0.906 against 0.875, interval + spanning zero) and lowered the within-pair AUC from 0.815 to 0.741 at 2.4 times the cost per + candidate ($0.0051 against $0.0021). Decision: REJECTED + (`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`). +9. Extrapolation with intervals. Every number carries a 95% repository-cluster bootstrap + interval; the pooled and the within-pair readings are both reported, and the stricter one is + named. +10. The one-time v3 check. When a program is adopted and a prediction is committed, population + v3 is scored once under `benchmarks/independent/protocol-v3.md`. + +## 3. AI integration: ax on the server + +All model work runs as Tasks on google/ax over Agent Substrate, in a single-node kind cluster on +the maintainer's host, each Task in its own gVisor sandbox. There is no local subprocess path. +The reconciler (`ousast plane run`) is a host process: it applies a Run's Workspaces and Tasks, +resumes each Task, sends its start signal, collects its artifacts and records state. Flow and +contract: [The plane on ax](plane.md); bring-up and the host's lessons: [ax on this host](ops/ax/README.md). + +What each Run guarantees: + +- One Model and one budget per Task. A Task binds at most one ax Model (`deepseek-flash` for + chat, `openrouter-embedding` for embeddings) and `budget: {usd, calls}`. The metered client + refuses the next call at the ceiling; the task ends unfinished and resumes on a rerun. Model-free + tasks run with a zero budget, so a stray call fails loudly. +- Credentials exist only in the start request. Agent Substrate boots each template once as a + golden actor, so the runner never starts by itself. It waits for `POST /ousast/v1/start`, which + the reconciler sends through `atenet-router` to that actor alone. The key is read from the + operator's environment or `.env`, kept in memory by the runner, passed to the command and + redacted from echoed stderr; no manifest, rendered Task or file carries it. +- Per-task egress. One EgressPolicy per actor through the `atenet-egress` gateway, deny by + default, hostnames only: plain HTTP to the artifact receiver, TLS passthrough to the Workspaces' + Git hosts and the Model's declared hosts. Deleted with the actor. +- Artifacts come back through the gateway. The runner posts its output directory as a tar to + the receiver, an HTTP server inside the reconciler process on the host, reached through a Service + with an EndpointSlice to the host. That delivery is the task's completion; ax has no Completed + phase. +- Memory on S3-compatible RustFS. `OUSAST_MEMORY=s3://sast-memory` on a RustFS server. The + store verifies the bucket at every start (versioning, lifecycle, tags, S3 Select) and refuses to + start when anything is missing; there is no local or fetch-and-filter fallback. Each row kind + has a fixed set of queryable fields, enforced at write time + ([Memory](memory.md#verify-at-startup-no-fallback), [RustFS setup](rustfs.md)). + +Lessons the host taught, each from a failed live run (the full list is on +[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run)): + +- Substrate installs no WorkerPool, and actor images must be pinned by digest. +- A release `ax` CLI skews from the server; build it from the deployed checkout. +- ax has no Completed phase, so completion is our artifact delivery. +- The golden boot means the runner must wait for a start request. +- The egress gateway dials the address the actor connected to, so the receiver is a Service + dialled by ClusterIP with its name as Host. +- ax's snapshot bucket must exist in your S3 server. When it did not, deleted golden actors piled + up in `DELETING` and filled the disk during a 600-task harvest on 2026-10-01. + +Deployment status, in the two words [Deployment](deployment.md) uses strictly. Implemented: +ax as the only executor, per-task EgressPolicy, credentials only in the start request, per-task +budgets and attribution, the S3 store verified at startup, local bring-up, doctor and smoke Run, +re-pinning generated Tasks to another registry. Planned, not done: deployment to any cluster +other than this host's kind cluster, a configurable kube context (`kind-` is hard-coded in +`doctor.py`, `egress.py`, `router.py`), the receiver as an in-cluster Deployment, the +reconciler in-cluster with Secrets injected, presigned-URL delivery to the store. + +## 4. Token economics, before and after + +Before the plane, the script pipeline cost $0.022 per candidate for two verify passes +(`benchmarks/independent/verifier-batched-check-2026-09-29.json`), and the maintainers' own +coding sessions spent about 90% of their tokens on tool input and output rather than reasoning. +After it, the per-candidate costs below are what the records show. The mechanisms (one budget +per Task, pay-once reuse, cheap-before-expensive ordering, replay at $0) are on +[Token ergonomics](token-ergonomics.md); this table is the before and after. + +| Item | Before | After | Record | +| --- | --- | --- | --- | +| Verification, two passes per candidate | $0.022 (script pipeline) | $0.0205 over 46 candidates with the recorded triage, including the tie-break pass on 10 disputed; 2,385,152 of 3,820,395 prompt tokens were cache hits in increment 1 (62%) | `verifier-batched-check-2026-09-29.json`, `plane-increment-1.json`, `plane-increment-2.json` | +| Decision per candidate (classifier, k = 5) | none existed | $0.0032 per evaluated candidate on the injection slice; $0.0027 to $0.0032 across six families | `2026-10-01-decision-engine-injection-slice/record.json`, `2026-10-02-decision-engine-slice-2/record.json` | +| An A/B arm, per candidate | none existed | $0.0021 (temperature sampling) against $0.0051 (retrieval ensemble); the cheaper arm won | `benchmarks/experiments/exp-002-retrieval-ensemble/result.json` | +| Harvest of plane signals for the engine | none existed | `verify` $2.5373 over 1,961 calls and 474 tasks; `roles` $1.4036 over 1,034 calls and 151 tasks | `2026-10-01-decision-engine-harvest/record.json` | +| Development transcript, tool I/O share | 90% | 85.5% over 2,701 tool calls (`dev_token_report`) | `plane-increment-1.json` | + +Two notes from the records. The meter prices every call at the Model's list price +(`plane/models/deepseek-flash.yaml`); the DeepSeek account moved less, about one third of the +metered amount on the injection slice ($0.30 for $0.91 metered), and budgets and gates are set +against the meter, never against the balance. And the development share barely moved: 90 to 85.5 +is a small gain, and the remedies (a read guard, a bulk reader, scripts in files) target file +contents because that is where the tokens go. + +## 5. Open items + +Five groups, each item with its record: the engine's own gates, data, measurement debt, +infrastructure, and instrument gaps. + +The engine's gates. + +- BLOCK is reachable for no family: calibration fails for five (ECE 0.071 to 0.119) and the + sixth reaches no fold with a Wilson lower bound of 0.95 on precision (slice-2 record). +- Re-run agreement is below 0.9 for three families: injection 0.77, access control 0.83, + untrusted destination 0.87. exp-002 tried the obvious variance fix and lost on the ranking. + +Data. + +- Positives-only groups: 98 (family, group) pairs in the 904-example memory hold positive examples + and no negative. 94 of them exist because the fixed side no longer has the function (the vibe-py + pairs). No advisory-fixes group is among them + (`benchmarks/measurements/2026-10-02-decision-engine-negatives-gap/record.json`). There are no + single-label negatives, so every family's pooled AUC is inflated by positives that never had to + be told apart from their own fix (slice-2 record, `paired_groups_check`). +- The model has seen public fixes. A verify pass named the advisory's CVE id for a public fix, an + observation the engine's design specification (maintainer tooling) records as visible + memorisation. Excerpts now redact advisory ids and security-worded comments + (`src/openultrasast/learn/excerpt.py`), but labels drawn from public advisories are partly in the + model's training data, and the within-pair numbers carry that. +- PHP is covered by the engine on real code and by in-sample quick rules; the population that + would measure it (v3) is reserved. + +Measurement debt. + +- exp-001 (known callers) is registered and has not run; the model-free candidate generation on + the 41 development repositories that its units need is not built + (`benchmarks/experiments/exp-001-known-callers/registration.json`). +- Leave-one-source-out, leave-one-framework-out and the memory-size curve points (0, 25, 50%) + have not run; one paired point exists (724 against 904 examples, injection, AUC 0.76 against 0.77). +- The plane-profile leak audit flagged `verify.flag_b` and `ms.operation`; they are withheld from + profile v1, and whether they track the label on unseen code has to be decided before any + plane-profile program runs (`2026-10-02-decision-engine-harvest-advisory/record.json`). +- Population v3 waits for `benchmarks/independent/prediction-v3.json`, which does not exist. It is + written only after a program is adopted. + +Infrastructure. + +- Deployment to a separate Kubernetes cluster is planned, not done; the code changes it needs are + listed with file paths on [Deployment](deployment.md#what-is-implemented-and-what-is-planned). + +Instrument gaps. + +- A missing provider key is a silent path in a scan. The plane refuses to start a Task whose Model + names an unset variable (`plane/router.py`, `StartError`), but a scan without a key asks the + model nothing, reports only the graph's entailed findings, and records no degradation for the + skipped model layer (`cli.py` writes degradations for `hunter_model_unavailable`, + `cpg_unavailable` and `learning_endpoint_unavailable` only). The rule that a scan which could + not look must say so does not yet cover this case. diff --git a/mkdocs.yml b/mkdocs.yml index add431b..36bde7a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -50,6 +50,7 @@ markdown_extensions: nav: - Overview: index.md - How it works: + - Where we stand: where-we-stand.md - Scanning: scanning.md - Architecture: architecture.md - The plane on ax: plane.md