mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
chore(p0): remove the improve lever and the plane improvement loop
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts). Deleted: src/openultrasast/improve/ (1,175 lines), the ousast improve subcommand, plane/tasks/loop.py and loop.yaml with the --loop generation, the memory-snapshot step in memory.seed, pairs.build_pair_signals, and the four improve tests plus the loop cases in other tests. Docs no longer describe the lever. Net: 3,073 lines deleted, 83 added. Targeted tests: 154 passed; 3,436 tests collect. Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Fable 5.1
Claude Fable 5.1
parent
d2010d0693
commit
94e96c6885
+2
-55
@@ -5,7 +5,7 @@ This page covers four things:
|
||||
- how a scan works;
|
||||
- how a finding earns its place;
|
||||
- how severity and the project score are decided;
|
||||
- how the ruleset improves itself.
|
||||
- how scan calibration works.
|
||||
|
||||
State as of 2026-10-02; module paths are under `src/openultrasast/`.
|
||||
|
||||
@@ -141,9 +141,7 @@ deleted. The score gate works like this:
|
||||
|
||||
The `score` stage writes `score.json` and merges it into `manifest.json`.
|
||||
|
||||
## Self-improvement
|
||||
|
||||
### Scoped calibration (every scan)
|
||||
## Scoped calibration (every scan)
|
||||
|
||||
```
|
||||
scan ─▶ verify ─▶ non-accepted findings ─▶ scoped learnings
|
||||
@@ -158,54 +156,3 @@ next scan ─▶ rank ─▶ calibrate (demote rejected scopes) ─▶ findings
|
||||
|
||||
Demotion is scoped and reversible (`tests/test_pipeline_calibration.py`).
|
||||
|
||||
### Rule-level loop: `ousast improve`
|
||||
|
||||
Each round (`improve/evolve.py`) runs these steps:
|
||||
|
||||
1. benchmark with the current ledger;
|
||||
2. per-rule signals (`rule_signals.json`);
|
||||
3. bounded proposals;
|
||||
4. validation (`improve/validator.py`);
|
||||
5. replay smoke;
|
||||
6. re-benchmark;
|
||||
7. a hard gate.
|
||||
|
||||
The levers are rule status (`enabled -> shadow` before `disabled`) and score constants. The
|
||||
loop can never edit pattern text or the authoritative CWE severity. A round is accepted only if
|
||||
all of these hold:
|
||||
|
||||
- recall >= 90%;
|
||||
- FP < 10%;
|
||||
- the project score did not regress;
|
||||
- no matched finding was lost.
|
||||
|
||||
Otherwise the ledger is not written. With `--pair-catalog`, each round is also gated per
|
||||
provenance profile on the catalog's holdout pairs. `--no-pair-gate` skips this check; do not
|
||||
use it for accepted ledgers. The same detection gate runs in CI (`python -m openultrasast.gate`).
|
||||
|
||||
### Proposals from plane memory
|
||||
|
||||
With `--memory [STORE]`, `ousast improve` also reads the rows that plane runs stored per
|
||||
repository and pin (`improve/memory.py`). Two deterministic rules turn these rows into
|
||||
rule-status proposals:
|
||||
|
||||
- a rule whose alerts are repeatedly false across repositories is shadowed;
|
||||
- a shadow rule that keeps hitting agreed, declared vulnerable sites is re-enabled.
|
||||
|
||||
Some rows are dropped before any rule sees them: rows from holdout pairs, from the gated
|
||||
manifest's own cases and from any `--qualify-population`. Each proposal records the rows it
|
||||
came from (`memory_proposals.jsonl` next to the journal). It then goes through the same
|
||||
validator and gate.
|
||||
|
||||
### The loop as a plane Run
|
||||
|
||||
`ousast plane workspaces <population> --validation-set ... --loop` appends the improvement loop
|
||||
to a generated Run. The Run holds `alerts` per case: the quick scan on the vulnerable and fixed
|
||||
pins. Then come the steps `snapshot` (task `memory-snapshot`), `measure`, `propose` and
|
||||
`improve` (`plane/tasks/loop.py`). Each step is model-free with a zero budget. The snapshot
|
||||
applies the same train-on-test guard before any row leaves the store. A recorded live run:
|
||||
`benchmarks/measurements/2026-09-30-harnessx-removal-loop/`.
|
||||
|
||||
When that run was made, quick mode covered PHP with zero rules. So for PHP, `ousast plane
|
||||
alerts-engine` produces a Run's `alerts` from the engine image on the host instead
|
||||
(`benchmarks/measurements/2026-09-30-php-engine-alerts/`).
|
||||
|
||||
+3
-4
@@ -77,8 +77,7 @@ Terms in the table:
|
||||
- **gVisor** is a sandbox runtime that isolates each actor from the node's kernel.
|
||||
- An **actor** is one running Task inside a sandbox.
|
||||
|
||||
Only the reconciler talks to the memory store. A sandboxed task cannot read it. So the reconciler seeds
|
||||
the loop's `memory-snapshot` before the Run starts ([The plane on ax](plane.md)).
|
||||
Only the reconciler talks to the memory store. A sandboxed task cannot read it.
|
||||
|
||||
## A. This host: the local setup (implemented)
|
||||
|
||||
@@ -478,7 +477,7 @@ Each Task manifest under `plane/tasks/` names the same image by digest. The pin
|
||||
`~/.cache/ousast/ax-src/runner-image`. The manifest picks its work with `spec.command`. The runner
|
||||
starts `python -m openultrasast.plane.tasks.<command[0]>` (`runner.py`, `task_module`).
|
||||
|
||||
`repo-facts`, `verify`, `agree`, `features`, `remember`, `alerts`, `loop` and `roles` are modules in
|
||||
`repo-facts`, `verify`, `agree`, `features`, `remember`, `alerts` and `roles` are modules in
|
||||
that image. They are not images of their own. The engine would be the second image (planned).
|
||||
|
||||
So the production answer to "how many containers" is:
|
||||
@@ -594,7 +593,7 @@ The worker is the unit. Every Task's `limits` fit the 1 CPU / 1.5 GiB worker of
|
||||
| `verify` | 250m / 384Mi | 1 CPU / 1Gi | 2 (`va`, `vb`), plus 1 (`vc`) on disputed candidates |
|
||||
| `agree` | 100m / 256Mi | 1 CPU / 512Mi | 1 (`agree`; `final` reuses it after `vc`) |
|
||||
| `features`, `remember` | 250m / 512Mi; 100m / 256Mi | 1 CPU / 1Gi; 1 CPU / 512Mi | optional, for the decision engine and the memory |
|
||||
| `alerts`, `loop` | 250m / 512Mi | 1 CPU / 1280Mi | not in a push Run (loop Runs only) |
|
||||
| `alerts` | 250m / 512Mi | 1 CPU / 1280Mi | not in a push Run |
|
||||
| `roles` | 250m / 384Mi | 1 CPU / 1Gi | not in a push Run (harvest Runs only) |
|
||||
| `engine` (planned) | from a measured run | above 3 GiB (the host run uses `--memory 3g`) | its own pool, not the default one |
|
||||
|
||||
|
||||
@@ -46,8 +46,7 @@ Rules live in `ruleset/<language>/rules.toml` as `[[rule]]` tables. Each has the
|
||||
- optional `framework`/`library`/`fallback_for` tags.
|
||||
|
||||
The fields are defined in `ruleset/store.py` (`ruleset/store.py:12,
|
||||
19-41`). The self-improvement loop may change only `status`, `min_evidence_level` and
|
||||
`precision_estimate`. The pattern is "human-PR-only" (`store.py:23-25`). There are 52 rules:
|
||||
19-41`). Patterns and rule policy change through reviewed commits. There are 52 rules:
|
||||
C 6, Groovy 1, Java 4, JavaScript 12, Python 11, PHP 18, of which 6 PHP rules are `shadow`.
|
||||
Shadow rules are written to `shadow_findings.json` and removed from the report
|
||||
(`cli.py:538-543`).
|
||||
|
||||
+1
-1
@@ -143,5 +143,5 @@ per-family tables are on [decision-engine.md](decision-engine.md).
|
||||
## Removal of the earlier agentic extra
|
||||
|
||||
The earlier agentic extra was removed on 2026-09-30. The removal was checked for byte-identical
|
||||
deterministic outputs: pair replay, quick benchmark, standard scan, `improve --dry-run`. Records:
|
||||
deterministic outputs: pair replay, quick benchmark, standard scan. Records:
|
||||
`benchmarks/measurements/2026-09-30-harnessx-removal-baseline/` and `-equality/`.
|
||||
|
||||
+3
-15
@@ -55,19 +55,10 @@ uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
|
||||
Each case is two isolated trees: the vuln file and the patched file, at the same relative path.
|
||||
|
||||
- `pair_correct` means the harness fired on the vuln side and stayed silent on the fix.
|
||||
- Misses become `miss` signals and fix-side leaks become `fp` signals for `ousast improve`.
|
||||
- Misses become `miss` signals and fix-side leaks become `fp` signals in the pair report.
|
||||
- The catalog and public dataset pointers live in `benchmarks/pairs/`.
|
||||
|
||||
## 6. Self-improve the ruleset from benchmark feedback
|
||||
|
||||
```bash
|
||||
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run # preview
|
||||
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml # apply
|
||||
# Accepted edits land in <target>/.openultrasast/calibration/rule_policy.json,
|
||||
# which the next scan/benchmark of that target loads automatically.
|
||||
```
|
||||
|
||||
## 7. Running the agentic plane
|
||||
## 6. Running the agentic plane
|
||||
|
||||
Model work over whole repositories runs as a Run manifest on ax. Each task gets one isolated
|
||||
actor. Bring-up is in [ops/ax/README.md](ops/ax/README.md).
|
||||
@@ -80,9 +71,6 @@ uv run ousast plane doctor # kind, Agent Substrate
|
||||
uv run ousast plane run plane/runs/validation-46.yaml # per case: repo-facts, verify a/b/c, agree, final, features, remember
|
||||
uv run ousast plane status validation-46 # per-task status and token attribution
|
||||
uv run ousast plane remember validation-46 # re-ingest the run's memory rows (OUSAST_MEMORY)
|
||||
|
||||
# Feed the stored rows back into the ruleset loop (same validator and gate)
|
||||
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
|
||||
```
|
||||
|
||||
Each task in the Run carries its own Model and `budget: {usd, calls}`.
|
||||
@@ -92,7 +80,7 @@ Each task in the Run carries its own Model and `budget: {usd, calls}`.
|
||||
- `ousast plane status` prints the token attribution table. It shows calls, prompt, cache-hit and
|
||||
output tokens, and USD per task.
|
||||
|
||||
## 8. Drive it from an MCP client (OpenCode / IDE)
|
||||
## 7. Drive it from an MCP client (OpenCode / IDE)
|
||||
|
||||
MCP (Model Context Protocol) lets an editor or agent call the tool's functions.
|
||||
|
||||
|
||||
+1
-7
@@ -28,9 +28,6 @@ flowchart LR
|
||||
scan --> report["Report: findings with evidence, SARIF, score"]
|
||||
scan -- "alerts on vulnerable and fixed pins" --> plane["Plane on ax: repo-facts, verify a/b/c, agree, features"]
|
||||
plane -- "remember" --> mem[("Memory store: rows, facts, examples, blobs")]
|
||||
mem -- "improve --memory: rule-status proposals" --> gate{"Validator and detection gate"}
|
||||
gate -- "accepted round" --> ledger["Ruleset ledger: rule status"]
|
||||
ledger --> scan
|
||||
labels["Labels from ground truth"] --> mem
|
||||
mem -- "retrieval of similar labelled cases" --> engine["Decision engine: compiled AI classifier"]
|
||||
engine -. "intended, not adopted: BLOCK / ADVISORY" .-> report
|
||||
@@ -41,10 +38,7 @@ flowchart LR
|
||||
- The **plane** runs model work per case: verify passes and roles. It also runs model-free
|
||||
bookkeeping per case: facts, agreement and feature records. Its `remember` task turns a case's
|
||||
artifacts into rows of the [memory store](memory.md).
|
||||
- Memory feeds two consumers:
|
||||
1. `ousast improve --memory` turns stored rows into rule-status proposals. They pass through the
|
||||
same validator and gate as every other edit.
|
||||
2. The decision engine retrieves labelled examples from the same store.
|
||||
- The decision engine retrieves labelled examples from the memory store.
|
||||
- The decision engine's output (dotted line) is the intended path. Nothing user-facing reads it
|
||||
today.
|
||||
|
||||
|
||||
@@ -149,37 +149,6 @@ Calibration **holds** when the slope interval contains 1, the intercept interval
|
||||
**BLOCK** is offered only when calibration holds and a fold reaches the required precision. Otherwise
|
||||
the engine offers **ADVISORY** only.
|
||||
|
||||
## The loop: memory improves decisions without retraining
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
new["New labels and adjudications"] --> labels["ousast learn labels"]
|
||||
labels --> build["ousast learn memory build, embed"]
|
||||
build --> mem[("Memory store: example rows")]
|
||||
mem --> retr["Retrieved per candidate at decision time"]
|
||||
retr --> prog["Same compiled program"]
|
||||
prog --> dec["Decision"]
|
||||
runs["Plane runs: alert, verdict, coverage rows"] --> mem2[("Memory store: run rows")]
|
||||
mem2 --> imp["ousast improve --memory: M1, M2 proposals"]
|
||||
imp --> gate{"Unchanged validator and gate"}
|
||||
gate -- "accepted round" --> ledger["Rule status in the ledger"]
|
||||
```
|
||||
|
||||
- **Decisions.**
|
||||
- The compiled part of the program is its instruction and demonstrations.
|
||||
- The examples are retrieved from memory for each candidate.
|
||||
- So a new `example` row is eligible for retrieval in the next decision without recompiling. The same
|
||||
evaluation boundary applies.
|
||||
- `ousast learn curve` measures the metric over memory size on a fixed candidate subset.
|
||||
- **Rules.** `ousast improve --memory` reads the run rows (alerts, verdicts, coverage). It proposes
|
||||
rule-status edits only:
|
||||
- shadow a rule that is repeatedly false across repositories;
|
||||
- re-enable a shadow rule that keeps hitting agreed declared sites.
|
||||
|
||||
Some rows are dropped first: rows from holdout pairs, the gated manifest's own cases and any
|
||||
`--qualify-population`. Every proposal goes through the same validator and gate as any other edit
|
||||
(`improve/memory.py`).
|
||||
|
||||
## What the first slice measured
|
||||
|
||||
Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts
|
||||
|
||||
+1
-14
@@ -25,10 +25,9 @@ replaces itself. The kinds (`KINDS`) and who writes them:
|
||||
| `facts` | `remember` task | the facts entry of a case (points at `facts/<sha256>.json`; reused by memory key) |
|
||||
| `verdict` | `remember` task | per candidate: each verify pass's answer, the final outcome, whether a tie-break ran, cost |
|
||||
| `unit_cost` | `remember` task | per verify pass and file: usd, calls and token usage |
|
||||
| `alert` | `remember` task | a quick-scan or engine alert on the vulnerable or fixed pin (loop Runs) |
|
||||
| `alert` | `remember` task | a quick-scan or engine alert on the vulnerable or fixed pin |
|
||||
| `coverage` | `remember` task | per language and pin: whether any rule could have fired there |
|
||||
| `features` | `remember` task, from the `features` task | one feature record per candidate and family |
|
||||
| `proposal_outcome` | `ousast improve --memory` (not with `--dry-run`), and the loop's `improve` step | what happened to each memory proposal in a round |
|
||||
| `example` | `ousast learn memory build` | a labelled case: label + feature record + excerpt sha (no rationale) |
|
||||
| `experiment` | `ousast learn experiment register` (task `register`) and `run` (task `run`) | a registered manifest's digests and provenance; a run's counts-only summary. Repo `experiments/<id>`, pin the registering commit |
|
||||
| `arm_outcome` | `ousast learn experiment run` | one decision per (unit, arm, replicate): score, verdict, usd, order |
|
||||
@@ -52,8 +51,6 @@ flowchart LR
|
||||
tasks["Plane tasks: facts, verify a/b/c, agree, alerts, features"] --> rem["remember task: memory.jsonl"]
|
||||
rem --> recv["Delivered to the host with the run's artifacts"]
|
||||
recv --> ingest["ingest after ousast plane run, or ousast plane remember RUN"]
|
||||
loop["Loop improve step: proposal_outcome"] --> ingest
|
||||
improve["ousast improve --memory: proposal_outcome"] --> put
|
||||
build["ousast learn memory build: labels + features + excerpts"] --> put["put_rows, put_blob"]
|
||||
embed["ousast learn memory embed"] --> put
|
||||
compile["ousast learn compile, evaluate"] --> put
|
||||
@@ -79,26 +76,16 @@ flowchart LR
|
||||
flowchart LR
|
||||
store[("Memory store")] --> seed["seed before a Run"]
|
||||
seed --> reuse["Facts reused by memory key: repo, pin, candidates, runner image"]
|
||||
seed --> snap["memory-snapshot: rows after the train-on-test guard"]
|
||||
store --> examples["learn: load example rows, excerpts, embeddings"]
|
||||
examples --> retr["Retrieval of similar labelled cases"]
|
||||
store --> responses["Response cache: replay at no cost"]
|
||||
store --> imp["ousast improve --memory: alert, verdict, coverage rows"]
|
||||
imp --> guard["Train-on-test guard"]
|
||||
guard --> rules["Rules M1, M2 to rule-status proposals"]
|
||||
rules --> gate{"Unchanged validator and gate"}
|
||||
```
|
||||
|
||||
- **Seed** (`memory.seed`).
|
||||
- A Run task gets stored facts when its `openultrasast.io/memory-key` matches them. It is then marked done.
|
||||
- A changed pin, candidate set or runner image finds nothing and recomputes.
|
||||
- The loop's `memory-snapshot` is seeded the same way. A sandboxed task cannot read the store.
|
||||
- **Retrieval** (`learn/retrieve.py`): see
|
||||
[Memory and detection](memory-and-detection.md#3-retrieval-of-similar-labelled-cases).
|
||||
- **`improve --memory`** (`improve/memory.py`). Some rows are dropped before any rule sees them:
|
||||
rows from holdout pairs, from the gated manifest's own cases, and from any `--qualify-population`. See
|
||||
[Architecture](architecture.md#proposals-from-plane-memory).
|
||||
|
||||
## Backends: FileStore and the S3 store
|
||||
|
||||
`OUSAST_MEMORY` selects the backend (`open_store`):
|
||||
|
||||
+3
-23
@@ -103,11 +103,10 @@ What each step guarantees:
|
||||
The first resume of a new image can time out while the snapshot is built. So the reconciler retries it
|
||||
for up to `OUSAST_RESUME_TIMEOUT` seconds (900).
|
||||
|
||||
## The Runs: validation and loop
|
||||
## Validation Runs
|
||||
|
||||
`ousast plane workspaces POPULATION --validation-set ...` (`generate.py`) writes one chain per case.
|
||||
`--loop` adds the `alerts` step per case and the loop's singleton steps. Every step is model-free
|
||||
except the verify passes (and `roles` in harvest Runs).
|
||||
Every step is model-free except the verify passes (and `roles` in harvest Runs).
|
||||
|
||||
### Per case
|
||||
|
||||
@@ -122,33 +121,15 @@ flowchart LR
|
||||
vb --> final
|
||||
vc --> final
|
||||
final --> features["features: one record per candidate"]
|
||||
alerts["alerts: quick scan on vulnerable and fixed pins (--loop)"] -.-> features
|
||||
features --> remember["remember: memory.jsonl"]
|
||||
alerts -.-> remember
|
||||
final --> remember
|
||||
```
|
||||
|
||||
### The loop (with `--loop`)
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
store[("Memory store")] -- "seeded before the Run, after the train-on-test guard" --> snap["memory-snapshot"]
|
||||
rem["remember of every case"] --> measure["loop-measure"]
|
||||
snap --> measure
|
||||
measure --> propose["loop-propose: rules M1 and M2, M3 advisory"]
|
||||
snap -- "index.json" --> propose
|
||||
propose --> improve["loop-improve: one evolve round, unchanged validator and gate"]
|
||||
improve --> outcome["gate.json, journal.json, and rule_policy.json only if accepted"]
|
||||
```
|
||||
|
||||
- **Before the Run**, `ousast plane run` seeds from the store (`memory.seed`).
|
||||
- Each task has an `openultrasast.io/memory-key`: repository, pin, candidates digest and runner image digest.
|
||||
- If the store has facts for that key, the task gets them and is marked done.
|
||||
- The sandboxed `memory-snapshot` task gets the store's rows after the guard.
|
||||
- **After the Run**, it ingests every delivered `memory.jsonl`. `ousast plane remember RUN` repeats
|
||||
this by hand. See [Memory](memory.md).
|
||||
- [Architecture](architecture.md#proposals-from-plane-memory) describes the loop's rules and its guard.
|
||||
Adopting an accepted ledger stays a maintainer commit (`src/openultrasast/plane/tasks/loop.py`).
|
||||
- `ousast plane harvest` writes the decision engine's harvest Runs (verify a/b with agree, and
|
||||
model `roles`).
|
||||
- `ousast plane alerts-engine` produces a Run's `alerts` from the Joern engine image on the host. It
|
||||
@@ -174,8 +155,7 @@ draws it and sizes the pre-push hook at scale.
|
||||
| `agree` | none | a and b agreed or disputed; after pass c on the disputed, 2-of-3 (the step `final`) |
|
||||
| `features` | none | one feature record per candidate for the decision engine |
|
||||
| `remember` | none | the case's artifacts as memory rows |
|
||||
| `alerts` | none | the quick scan on the vulnerable and fixed pins (loop Runs) |
|
||||
| `loop` | none | the steps `snapshot`, `measure`, `propose`, `improve` |
|
||||
| `alerts` | none | the quick scan on the vulnerable and fixed pins |
|
||||
| `roles` | `deepseek-flash` | per-repository source, sink and sanitizer roles without a vocabulary |
|
||||
|
||||
Source: [ax on this host](ops/ax/README.md#tasks).
|
||||
|
||||
@@ -14,10 +14,8 @@ It also lists the controls that keep a scan safe to run in CI and safe to share.
|
||||
- Only `deep` mode executes target-derived code. Its REGRESS stage loads promoted candidates
|
||||
inside the Docker sandbox described below. It runs only when `docker
|
||||
info` succeeds (otherwise the stage is skipped and recorded as `sandbox_unavailable`).
|
||||
- **The ruleset and CWE policy are trusted, governed data.** They change in only two ways:
|
||||
through reviewed commits, or through the bounded self-improvement loop (`ousast improve`). The
|
||||
loop can flip rule *status* and tune score constants. It can never edit pattern text or the
|
||||
authoritative 0–5 severity. See [architecture.md](architecture.md#self-improvement).
|
||||
- **The ruleset and CWE policy are trusted, governed data.** They change
|
||||
through reviewed commits.
|
||||
- **Provider credentials are the operator's.** The core install (PyYAML only) makes no network
|
||||
calls on its own.
|
||||
- A local scan calls a model only in standard or deep mode, and only when a key is available.
|
||||
|
||||
Reference in New Issue
Block a user