From 32a2cb8bcf0c6e131534d1c6c2b4f927c1fba7eb Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Fri, 2 Oct 2026 07:08:44 +0200 Subject: [PATCH] docs: describe the system as of 2026-10-02 (ax plane, AI decision engine, PHP, v3) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo --- .kiro/steering/overview.md | 35 ++- .kiro/steering/roadmap.md | 13 +- README.md | 534 ++++++++++++------------------------- RELEASE_NOTES.md | 34 +++ docs/architecture.md | 160 +++++++++++ docs/decision-engine.md | 85 ++++++ docs/evaluation.md | 84 ++++++ docs/examples.md | 39 +-- docs/threat-model.md | 49 ++-- ops/README.md | 17 +- ops/ax/README.md | 70 ++++- 11 files changed, 715 insertions(+), 405 deletions(-) create mode 100644 docs/architecture.md create mode 100644 docs/decision-engine.md create mode 100644 docs/evaluation.md diff --git a/.kiro/steering/overview.md b/.kiro/steering/overview.md index 237bd27..929758b 100644 --- a/.kiro/steering/overview.md +++ b/.kiro/steering/overview.md @@ -1,6 +1,37 @@ # Overview β€” what this is, and what is actually true of it -**Current direction, 2026-09-12:** an actionable pre-push security safety net for AI-accelerated +## Current state, 2026-10-02 + +Everything below this section is dated history; where it disagrees with this section, this section wins. +User-facing documentation: `README.md`, `docs/architecture.md`, `docs/evaluation.md`, `docs/decision-engine.md`, +`ops/ax/README.md`. + +- **Scanning.** `ousast scan --mode quick|standard|deep`. Quick rules for C/C++, Python, JavaScript/TypeScript, + Java/Groovy and PHP (PHP numbers are in-sample only: `benchmarks/measurements/2026-09-30-php-quick-rules/`). + Standard adds the MAP stage with the Joern model layer (shipped in the Docker image); deep adds sandboxed + regression snippets when Docker is available. Framework knowledge is tagged as optional priors + (`ruleset/frameworks.toml`, `priors=`). `ousast pre-push` compares head against base; the capability registry + is empty, rollout NO-GO. +- **Agentic plane.** google/ax over Agent Substrate on kind is the only executor of model work over whole + repositories (`ousast plane run|status|doctor|workspaces|remember|alerts-engine|harvest`). Credentials travel + only in the start request through `atenet-router`; egress is per task; budgets are per task; rows go to the + memory store (`OUSAST_MEMORY`: FileStore, or MinioStore on the maintainer's MinIO). +- **Removed.** HarnessX (retired 2026-09-30): its config keys fail, a `[harnessx]` section warns. The LLM judge + of the verifier and the LLM fusion panels are gone; fusion is deterministic. +- **Improvement.** `ousast improve --memory` proposes rule-status edits from plane memory through the unchanged + validator and gate; the loop also runs as a plane Run. +- **Decision engine** (`learned-decision-engine`, implementation, not adopted). First slice, injection only: + ADVISORY recall 0.90, precision 0.54, BLOCK not offered + (`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`). +- **Evaluation.** Populations v1 and v2 are spent and failed every M4 gate (`benchmarks/independent/results-v1.json`, + `results-v2.json`); v3 (PHP) is frozen and untouched for the one-time final check. The plane increment met its + gate on the second measurement with a tie-break chosen after the first (`benchmarks/independent/plane-increment-2.json`). +- **Providers.** DeepSeek for LLM calls, OpenRouter only for OpenAI embeddings; keys in `.env`, which never + overrides an exported shell variable. + +## History + +**Direction, 2026-09-12:** an actionable pre-push security safety net for AI-accelerated development across supported ecosystems. See `safety-net.md` and the approved `pre-push-safety-net` spec. The ranker remains the scope mechanism; third-party vendor code stays outside targets and graphs. Advisory must meet the same actionability bar as blocking. @@ -177,4 +208,4 @@ function-level pairs could not see any of them, and a repository showed all nine > **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax (`ai-service-plane`), > and evolutionary proposals come from the plane memory. The HarnessX specs in the table above are history. -97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests. +As of 2026-09-09: 97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 1c25741..fe5b799 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -1,6 +1,17 @@ # Roadmap -## Current release path β€” 2026-09-14 +## Status, 2026-10-02 + +The milestone table below dates from 2026-09-14. Since then (`.kiro/specs/pre-push-safety-net/release-milestones.md`): +M1a and M1b are met, M2 is measured NO-GO, and M3 is next, with M4 population preparation in parallel; none of +M1-M5 is verified and rollout remains NO-GO. Populations v1 and v2 were evaluated and failed every M4 gate +(`benchmarks/independent/results-v1.json`, `results-v2.json`); v3 (PHP) is frozen for the one-time final check. + +Work since then moved to: the ax plane as the only agentic executor (`ai-service-plane`, HarnessX retired +2026-09-30 by `harnessx-removal`), and the learned decision engine (`learned-decision-engine`, task 6 in +progress, not adopted). See `overview.md` for the current-state summary. + +## Release path as accepted 2026-09-14 Build an actionable pre-push security safety net for AI-accelerated development. The maintainer accepted the following order: **prove one useful detection, make it diff --git a/README.md b/README.md index 067b324..81d5f6e 100644 --- a/README.md +++ b/README.md @@ -1,159 +1,78 @@ # OpenUltraSAST -**v1.2.0-alpha.1:** experimental pre-push safety net. Integration and census repairs are -verified; useful hook coverage, latency and independent capability qualification remain -**NO-GO**. No real hook capability is enabled. See [release notes](RELEASE_NOTES.md) and -[spec status](.kiro/specs/pre-push-safety-net/status.md). +**v1.2.0-alpha.1 (Python package `1.2.0a1`), state as of 2026-10-02.** An experimental static +security analyser. The pre-push safety net is **NO-GO** for rollout: no hook capability is +enabled and no independent population has passed the qualification gates (see +[docs/evaluation.md](docs/evaluation.md)). See the [release notes](RELEASE_NOTES.md). -OpenUltraSAST is a command-line security analysis harness built around one -goal: eliminating false positives by turning a suspicion into an -evidence-backed finding, and learning from the claims that were rejected. A -scan combines static rules, a semantic overlay, optional model-driven hunting -and Docker-isolated proof, and every claim carries the evidence that earned its -place. +OpenUltraSAST separates what it can prove from what it merely suspects, and says which is which. +A scan combines language-scoped pattern rules, a semantic overlay, a Joern-based taint engine +and optional model calls; every finding carries the evidence that earned its place, and a scan +that could not look at something records that as a degradation instead of reporting a clean +result. -The project began as an editor-agent skill and is now a CLI first. Agent -integrations remain, but nothing requires one: `ousast` is the interface, and -the model-driven parts are optional extras that degrade visibly when absent. - -> **Maturity legend:** βœ… implemented and tested Β· 🟑 primitive exists, not yet -> auto-wired into the scan loop Β· 🧭 designed, on the roadmap. The current -> verified gate is `usable_harness_mvp`. +`ousast` is the interface. Model-driven work over whole repositories runs on a separate +agentic plane (google/ax), and a learned decision engine that weighs all the signals is in +development; neither is needed for a scan. ## Install and quickstart ```bash uv sync --group dev -# Scan a local repository (quick mode = deterministic, no model calls) +# Deterministic scan, no model calls uv run ousast scan /path/to/repo --mode quick -# Build an embedding-ready chunk index -uv run ousast index /path/to/repo - -# Score the detector against a benchmark manifest -uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick - -# Differential pair eval: fire on vuln, stay silent on the fix (local + real GitHub VFCs) -uv run ousast pairs -uv run ousast pairs --slice sast # OWASP Benchmark Java/Python + Juliet -uv run ousast pairs --slice vibe-py --pointers # nightly: also harvest non-vendored pointer pairs into ~/.cache -uv run ousast mechanisms export --slice vibe-py # seed mechanism candidates from trusted pairs (offline); `ousast improve` admits them under the holdout gate -uv run ousast pairs --slice vibe-py --loo # leave-one-out recall of the corpus as a teacher -uv run python -m openultrasast.pair_gate - -# Run the bounded self-improvement loop over the ruleset (benchmark-driven) -uv run ousast improve benchmarks/manifests/python-vulnerable.toml --dry-run +# CI: exit 1 on any evidence-verified finding +uv run ousast scan . --mode quick --fail-on verified ``` -Every run writes auditable artifacts under `.openultrasast/runs//` -(`manifest.json`, `findings.json`, `verification.json`, `report.md`, -`report.sarif`, `trace/events.jsonl`, ranking/preprocess/mapping JSON). -`--fail-on findings|verified` gives CI-friendly exit codes. +The core install depends only on PyYAML. Optional extras: `semantic` (tree-sitter grammars +for the overlay) and `minio` (the plane's MinIO memory store). -## How a scan works +For `standard` and `deep` scans, use the Docker image, which ships Joern (with `php-cli` for +the PHP frontend) next to the tool. Source the shell wrapper once and `ousast` runs in the +container with your directory mounted read-only: -The CLI is thin; the `HarnessRuntime` owns the lifecycle and emits an event -trace for every stage (`task_start … stage_start/stage_end … task_end`). - -``` -repo ──▢ preprocess (language, LOC, tags, fuzz entry points) - ──▢ static mapping (Semgrep/CodeQL SARIF ingest, normalized hints) - ──▢ entry-point + reachability mapping (routes, CLI, parsers, contracts) - ──▢ rank (surfaceΒ·0.5 + influenceΒ·0.2 + reachabilityΒ·0.3) - ──▢ findings ─ quick: language-scoped pattern rules - β”‚ β”” standard: tiered hunter pool (retrieval + skill snippets) - ──▢ independent verifier (evidence ladder enforced) - ──▢ report.md + report.sarif + manifest.json (shared finding IDs) +```bash +source /path/to/OpenUltraSAST/ops/shell/ousast.sh # PowerShell: ops/shell/ousast.ps1 +ousast scan . --mode standard ``` -- **quick** βœ…: deterministic, language-scoped pattern rules + reachability + - verification. No model calls, fully reproducible. -- **standard** βœ… scheduling / 🧭 LLM hunters: runs the tiered hunter pool - (A/B/C budgets, retrieval context, selected Trail of Bits skill snippets, - JSONL trajectories) and verification; language-aware LLM hunters are the next - upgrade (today standard reuses the deterministic findings per target). -- **deep** 🧭: sandboxed build/fuzz/dynamic reproduction (not yet implemented). +Details, including a pinned host install of Joern, are in [ops/README.md](ops/README.md). -## Detection coverage +Every run writes auditable artifacts under `/.openultrasast/runs//`: +`manifest.json`, `findings.json`, `verification.json`, `score.json`, `report.md`, +`report.sarif` and `trace/events.jsonl`, plus the stage artifacts of the mode. -Language-scoped rule families (rules only run against matching languages, which -removes cross-language false positives): +## Scan modes + +| Mode | What runs | Needs | +| --- | --- | --- | +| `quick` | Language-scoped pattern rules, entry-point reachability hints, ranking, verification, scoring. Deterministic and reproducible. | nothing | +| `standard` | `quick` plus the MAP stage: complexity map, semantic overlay, authorization obligations, and the model layer, which builds one Joern code property graph per repository and decides taint, guard-dominance and configuration questions on it. An LLM is asked only the residual question the graph cannot settle (a `suspicion`); `[models] hunter` adds the tool hunter. | Joern for the model layer (else a `cpg_unavailable` degradation); a provider key for the LLM parts (else they are skipped and recorded) | +| `deep` | The MAP stage of `standard` plus REGRESS: promoted candidates are loaded by a small snippet inside a Docker sandbox (no network, read-only source, non-root, memory/pid/time limits) and get a `triggerable` / `not_triggerable` / ... verdict. | a working `docker` (else a `sandbox_unavailable` degradation) | + +`--fail-on never|findings|verified|worth-fixing` sets the exit code: `findings` fails on any +reported finding, `verified` on any evidence-verified one, `worth-fixing` on a deep-mode +verdict that is worth fixing. `--config openultrasast.toml` loads settings. + +## Detection coverage (quick rules) + +Rules run only against files of their language. The bundled ruleset lives in +`src/openultrasast/ruleset//rules.toml`: | Language | Classes (CWE) | | --- | --- | -| C / C++ | buffer overflow (120), format string (134), command injection (78) | -| Python | command injection (78), code injection (95), deserialization (502), SQLi (89), path traversal (22), SSRF (918), weak hash (327), SSTI (94), insecure default (489) | -| JavaScript / TS | command injection (78), code injection (95), reflected/DOM XSS (79), SQLi (89), path traversal (22), SSRF (918), weak hash (327), deserialization (502) | -| Java + Groovy templates | command injection (78), SQLi via concatenation (89), weak hash (327), deserialization (502), unescaped template XSS (79) | +| C / C++ | stack buffer overflow (121), format string (134), command injection (78) | +| Python | command injection (78), code injection (95), SSTI (94), deserialization (502), SQLi (89), path traversal (22), SSRF (918), weak hash (327), insecure default (489) | +| JavaScript / TS | command injection (78), code injection (95), XSS (79), SQLi (89), path traversal (22), SSRF (918), weak hash (327), deserialization (502) | +| Java + Groovy templates | command injection (78), SQLi (89), weak hash (327), deserialization (502), unescaped template XSS (79) | +| PHP | SQLi (89), command injection (78), file inclusion (98), path (22), open redirect (601), SSRF (918); code eval (95), unserialize (502) and echo XSS (79) ship as `shadow` rules | -Cheat-sheet fixtures live under `benchmarks/fixtures/` and feed the 90/10 smoke -gate. **Efficiency** is measured separately on isolated vuln-vs-fixed pairs -(`ousast pairs`, `python -m openultrasast.pair_gate`): the harness must fire on -the vulnerable snapshot and stay silent after the patch. Local fixture -counterparts are a CI gate; vendored GitHub VFCs (SVEN, CWE-Bench-Java, NVD -patterns) are the honesty dashboard and the miss/fp signals for `ousast improve`. -Dataset pointers (CVEfixes, DiverseVul, PrimeVul, MegaVul, MoreFixes, Vul4J, -GitHub Advisory Database, OSV) are in `benchmarks/pairs/datasets.toml`. - -## Project scores and central CWE policy - -Severity used to be whatever string a rule set on itself. It is now governed centrally: one CWE, one severity, decided by policy β€” not by the rule that fired. - -**Central CWE policy** βœ… (`policy/verycode.py`): a vendored `CWE_Score.tsv` (verycode-policies, ~167 CWEs) is the single source of truth. Each CWE carries a flaw category, a `severity` (0-5), and `static` / `dynamic` scope flags. `load_policy()` reads it positionally (the upstream file ships CRLF and a trailing space in the `Flaw Severity ` header), and `resolve_severity()` resolves a finding's severity *exclusively* from policy, keyed on CWE β€” any legacy rule-local severity is discarded. CWEs that are not `static` resolve to `0` and are report-only, never scored. - -**Fail-loud startup** βœ…: `assert_rules_resolve(PATTERN_RULES, policy)` runs as the `policy_check` stage right after `policy_load`. If any *enabled* rule names a CWE the policy does not govern, it raises `PolicyError` and the scan aborts before doing work β€” you cannot ship a rule whose severity nobody decided. - -**0-100 project score** βœ… (`scoring/project_score.py`): each finding's penalty is `severity weight Γ— reachability multiplier`, and the score is an exponential decay of the total (`100 Β· e^(βˆ’total/k)`, `k=60`). - -| Severity | Weight (`SEV_WEIGHT`) | | Reachability | Multiplier (`REACH_MULT`) | -| --- | --- | --- | --- | --- | -| 5 | 50 | | `reachable` | 1.0 | -| 4 | 25 | | `inferred-file-surface` | 0.6 | -| 3 | 10 | | `unknown` | 0.4 | -| 2 / 1 / 0 | 2 / 1 / 0 | | | | - -No penalty β†’ 100; one reachable severity-5 finding β†’ ~43. The **reachability multiplier is the false-positive calibration knob**: a confirmed FP lowers a finding's effective reachability instead of deleting the rule, so the score moves without losing the detection. - -**Two-condition gate** βœ…: a finding that is both severity-5 *and* `reachable` **always** fails the gate. The `score < min_score` threshold (default `min_score=80`) only fails when `blocking` is enabled β€” so scoring is **advisory-first by default** and turns into a CI gate when you opt in. - -**Artifacts** βœ…: the `score` stage writes `score.json` (project score, `max_severity`, `penalty_total`, `by_category`, `out_of_scope_dynamic_only`, `unmapped_cwe`, gate verdict) and merges the same block into `manifest.json`. Scoring is zero-dependency (stdlib only). One mapping detail: verycode has no CWE-120 (generic buffer copy), so the C/C++ memory-unsafe rules listed under [Detection coverage](#detection-coverage) carry **CWE-121** (Stack-Based Buffer Overflow, severity 5), which the policy does govern. - -🧭 This is the first implemented slice of the self-improving ruleset work: a central, policy-governed severity model and a project score. Rules-as-data and the full self-improvement loop over rulesets are later phases. - -## Running the benchmarks - -Benchmarks wrap the same scan pipeline and score it against ground-truth -manifests in `benchmarks/manifests/`. Each manifest lists the *known* -vulnerabilities (CWE, file, line, sink, evidence) plus optional external -baselines; fixtures live in `benchmarks/fixtures/` and include safe-API files so -precision is measured honestly. - -```bash -# One language -uv run ousast benchmark benchmarks/manifests/cpp-damn-vulnerable.toml --mode quick - -# All in-scope corpora -for m in c-cpp-smoke cpp-damn-vulnerable \ - python-web-smoke python-vulnerable \ - javascript-node-web-smoke javascript-vulnerable \ - java-web-smoke java-spring-boot-vulnerable; do - uv run ousast benchmark "benchmarks/manifests/$m.toml" --mode quick -done -``` - -Each run writes `.openultrasast/benchmarks//`: -`benchmark_result.json` (recall, precision, misses, false positives, runtime), -`calibration_records.json` (every miss as a next-improvement candidate), and -`external_baseline_deltas.json` (tool-vs-tool comparison). - -### Latest results (`--mode quick`) - -The corpora are modeled on Damn Vulnerable C/C++, PyGoat/DVPWA and NodeGoat, -plus the real [`kiview/damn-vulnerable-spring-boot-app`](https://github.com/kiview/damn-vulnerable-spring-boot-app) -vendored verbatim (MIT). Three vulnerabilities are deliberately left in the -ground truth that pattern matching *cannot* reach (integer overflow, second-order -SQLi, prototype pollution) so recall is not self-fulfilling. +The detection gate (`uv run python -m openultrasast.gate`, also `tests/test_detection_benchmarks.py`) +enforces the project goal of **at least 90% recall and under 10% false positives** per language +on the bundled corpora in `benchmarks/manifests/`. Its output on this tree: | Language | Recall | False-positive rate | | --- | --- | --- | @@ -163,259 +82,152 @@ SQLi, prototype pollution) so recall is not self-fulfilling. | Java | 100% (4/4) | 0.0% | | **Overall** | **93.6% (44/47)** | **0.0%** | -The project goal is **β‰₯90% recall and <10% false positives** per language. The -gate lives in `tests/test_detection_benchmarks.py`, so a rule change that drops -recall or raises false positives fails CI. +These corpora are cheat-sheet fixtures modelled on deliberately vulnerable applications. They +show that a rule change did not break a known detection; they are not a recall estimate on +real code. Real-code results are in [docs/evaluation.md](docs/evaluation.md). -## Agent integrations, fusion, and ultra workflows +**PHP is not in the gate, and its numbers are in-sample only.** The PHP rules were written and +tightened while looking at the development corpus they were measured on +(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`): 13 of 14 expected +findings on the `php-vulnerable` fixture with no false positive, 2 false positives on +`php-benign`, and per-rule precision lower bounds from 0.03 to 1.0 on the real pins. Treat +them as development numbers, not as a holdout result. -The `ousast` CLI is the interface. An agent that can run shell commands needs -nothing else: +**Framework knowledge is optional.** Rule and taint-fact entries that know a framework or +library (WordPress, Flask, Django, Express, Spring, ...) carry a `framework` or `library` tag +naming a row of `src/openultrasast/ruleset/frameworks.toml`. The loaders take `priors=` +(`"all"`, the default and today's behaviour; `"off"`, language-level knowledge only; or a set +of ids), so their contribution can be measured and switched off. + +## Pre-push checks (experimental) + +`ousast pre-push` analyses the commits a `git push` would publish, or an explicit +`--base`/`--head` pair, and compares the head against its base so that only new or worsened +defects are candidates. `--mode advisory` (the default) never blocks; `--mode blocking` and +`--incomplete-coverage block` opt into enforcement. The artifact must be written outside the +analysed repository. ```bash -uv run ousast scan . --mode quick --fail-on verified +uv run ousast pre-push /path/to/repo --base BASE --head HEAD \ + --artifact /outside/the/repo/result.json ``` -For [OpenCode](https://opencode.ai) there are optional project skills that -describe the workflows above, so the agent drives the same CLI -(`.opencode/skills/`): +The default capability registry is empty, so no normal alert is emitted today; every +candidate stays diagnostic in the artifact. Installation into a repository's hooks +(`ops/install-pre-push`), caching and the optional witness model are documented in +[ops/README.md](ops/README.md). -| Skill | Purpose | -| --- | --- | -| `openultrasast-scan` βœ… | run/plan `ousast scan`, indexing, evidence-gated analysis | -| `openultrasast-triage` βœ… | false-positive elimination, ranking calibration, verifier adjudication | -| `openultrasast-fix-audit` βœ… | OpenUltraCode fix lifecycle and adversarial fix review | +## Model providers and keys -### MCP server (narrow surface) βœ… - -`ousast mcp` runs a zero-dependency MCP server over stdio (newline-delimited -JSON-RPC) exposing only stable, project-level operations β€” **never** arbitrary -shell, Docker, or internal hunter tools. Point an MCP client (OpenCode, Claude -Desktop, an IDE) at it: - -```jsonc -// e.g. an MCP client config entry -{ "command": "uv", "args": ["run", "ousast", "mcp"] } -``` - -Tools: `openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, -`artifacts`, `benchmark`, `explain`, `propose_patch` (degrades visibly β€” the patch -oracle is a later phase), and `export_report`. The `scan`/`benchmark` tools run the -bounded analysis pipeline; no tool accepts a free-form command. A typical flow: -`scan {path}` β†’ returns a `run_dir` β†’ `findings {run_dir}` β†’ `explain {run_dir, finding_id}`. - -**Ultra workflows (OpenUltraCode discipline):** fixes are never a one-shot patch -prompt. `openultrasast-fix-audit` runs the bounded lifecycle -`intake β†’ plan β†’ minimal patch β†’ adversarial review β†’ reconcile β†’ fresh -verification β†’ ready`, and a patch can only reach `patch_validated` after -sandboxed validation passes with no accepted blocking findings. - -**Fusion (deepening) βœ…:** when a finding needs more reasoning than the normal -ranker β†’ hunter β†’ verifier β†’ mapping loop provides (critical/high severity, -verifier disagreement, conflicting static vs semantic evidence, risky fixes, or -an explicit high-assurance request), fusion runs two independent panels that -steel-man the vulnerability and false-positive cases, vote, and a decider issues -the disposition (accepted / rejected / mitigated / deferred / blocked) with votes -and degradations disclosed. It runs automatically on triggered findings -in `--mode standard`, writing `fusion.json` and a `fusion` block in the manifest. -Fusion is deterministic (`fusion.py`); independent LLM agreement on a candidate -is the plane's `agree` task (see [The agentic plane (ax)](#the-agentic-plane-ax)): - -```toml -[fusion] -enabled = true # default; runs on triggered findings in standard mode -high_assurance = false # true forces fusion on every finding -``` - -> The `kiro-*` skills, `AGENTS.md` and the `.kiro/` directory are the -> maintainer's spec-driven development tooling. They are not part of using -> OpenUltraSAST and can be ignored by users. - -## Evidence ladder: how a false positive is eliminated - -Evidence level is a state machine, not a label a model may assign. The verifier -runs on independent context (it never sees hunter reasoning) and enforces: - -``` -suspicion ─▢ static_corroboration ─▢ crash_reproduced ─▢ root_cause_explained - ─▢ exploit_demonstrated ─▢ patch_validated -``` - -A finding is rejected or held back unless the *artifact* for the next level -actually exists: - -- below `static_corroboration` β†’ `NEEDS_EVIDENCE` (a model/heuristic suspicion is - never reported as verified); -- `static_corroboration` but **reachability unknown** β†’ `REJECTED` with a - tie-breaker demanding call-graph / route / CLI / parser / dynamic evidence - (this is what kills "pattern matched but nothing attacker-controlled reaches - it" findings); -- `static_corroboration` **and** function-level reachability β†’ `ACCEPTED`. - -When a finding is still rejected after review, `openultrasast-triage` records a -**scoped false-positive learning** (`calibration.py`): a reason from a fixed -taxonomy (`unreachable_path`, `missing_attacker_control`, `sanitizer_disproved`, -`static_rule_mismatch`, `incorrect_model_assumption`, `duplicate`, -`insufficient_impact`, `unsupported`, `contradicted`, `unverified`), evidence, -and a **scope**. Scoping keeps the learning narrow: a rejected `syscall_entry` -claim in `auth/` demotes only `auth/…`, never the whole vulnerability class. This -is covered by `tests/test_calibration.py::test_scoped_false_positive_demotion_does_not_suppress_class_globally`. -Demotion (with an audit trail) is preferred over deletion, so an analyst can -always see *why* something was downgraded. - -## How a false negative is eliminated - -Misses are first-class signals, surfaced by the benchmark layer rather than -hidden. Every expected vulnerability that no finding matched becomes a -`BenchmarkCalibrationRecord` in `calibration_records.json`, e.g. the integer -overflow the regex engine cannot see: - -```json -{ - "cwe": "CWE-190", - "vulnerability_class": "integer overflow", - "path": "src/buffer.cpp", - "failed_stage": "benchmark_ground_truth_matching", - "next_improvement_candidate": "rules_or_language_hunter", - "reason": "no OpenUltraSAST finding matched the expected benchmark vulnerability" -} -``` - -The `next_improvement_candidate` routes the gap to the stage that should close -it: a static rule, SARIF source, entry-point mapping, retrieval package, hunter -prompt, dynamic reproducer, or skill route. Once a miss is -addressed, it stays closed because the recall/precision gate runs in CI. -`external_baseline_deltas.json` additionally shows where another tool found a -vulnerability OpenUltraSAST missed (or vice-versa). - -## The agentic plane (ax) - -The core install is zero-dependency and `--mode quick` never calls a model. Model -work that spans a whole repository runs on the **agentic plane**: a Run manifest -of tasks executed on [ax](ops/ax/README.md) (Kubernetes with Agent Substrate), one -isolated actor per task. What runs there today: `repo-facts`, two independent -`verify` passes, `agree` with a 2-of-3 tie-break on disputed candidates, and -`remember`, which writes each run's per-candidate rows into the plane memory. +Keys live in `.env` in the working directory (gitignored, never committed): ```bash -uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image -uv run ousast plane run plane/runs/validation-46.yaml # a rerun skips tasks already done -uv run ousast plane status # per-task status and token attribution -uv run ousast plane remember # ingest the run's memory rows (OUSAST_MEMORY) +DEEPSEEK_API_KEY=... # every LLM call: model layer, tool hunter, plane tasks +OPENROUTER_API_KEY=... # only OpenAI embeddings (openai/text-embedding-3-small) ``` -Every task binds its own Model and its own budget (`usd`, `calls`), and a task -that reaches its ceiling stops as `unfinished` instead of overspending. Egress is -deny-by-default per task; the provider key travels only in the task's start -request and is never written to an artifact. Bring-up, the egress policy and the -host requirements are in [ops/ax/README.md](ops/ax/README.md). +The project uses DeepSeek (`deepseek-flash`) for LLM calls and OpenRouter only for embeddings. +Set `DEEPSEEK_API_KEY`: without it, chat calls fall back to OpenRouter when that key is set. -In a local `ousast scan --mode standard`, `[models] hunter` still enables the -MAP-stage tool hunter; verification and fusion are deterministic. +**`.env` never overrides a variable already exported in your shell.** A stale +`export DEEPSEEK_API_KEY=...` in your profile wins over the `.env` file silently; `unset` it +when a key change in `.env` seems to have no effect. -> The earlier optional agentic extra, HarnessX, was retired 2026-09-30, and a -> leftover `[harnessx]` section (retired 2026-09-30) is ignored with one warning. -> `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail with -> a message naming the plane replacement. See [RELEASE_NOTES.md](RELEASE_NOTES.md). +## Benchmarks and pair evaluation -## The self-improving cycle +```bash +# Score the scan against a ground-truth manifest +uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick -Each scan is a composed harness of typed processors with read/write **state -contracts**. Strict mode fails the scan on a violation; warn mode records a -degradation. The loop now runs inside the pipeline without an agent: every scan -persists verifier rejections as scoped learnings, and the next scan loads them -and demotes those scopes before findings are produced. - -``` - β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ automatic ledger (no agent in the loop) ───────────┐ - β–Ό β”‚ - scan ──▢ verify ──▢ non-accepted findings ──▢ scoped learnings ────────── - β”‚ (.openultrasast/calibration/ β”‚ loaded by - β”‚ false_positive_learnings.json) β”‚ the next - next scan ──▢ rank ──▢ calibrate (demote rejected scopes) ──▢ findings β”€β”€β”˜ scan +# Vulnerable-vs-fixed pairs: fire on the vulnerable side, stay silent on the fix +uv run ousast pairs +uv run ousast pairs --slice sast # OWASP Benchmark + Juliet +uv run ousast pairs --slice vibe-py --pointers # also fetch non-vendored pointer pairs (network) +uv run python -m openultrasast.pair_gate # CI: local pairs must all pass ``` - -What runs automatically each scan (βœ…, `calibration.py` + `cli.py`): - -1. **`calibrate` stage** (after `rank`, before findings) loads the persistent - ledger and demotes the priority of every scope that previously produced a - rejected finding (`calibrate_rankings`), writing `applied_calibrations.json`. -2. **`record_calibration` stage** (after `verify`) turns this run's non-accepted - verifier outcomes into scoped `FalsePositiveLearning` records - (`learnings_from_verifications`) and merges them into the ledger, de-duplicated - by finding ID so a repeat rejection does not compound without bound. -Demotion is **scoped and reversible**: a rejection in `lib.py` demotes only that -scope; an accepted (reachable) finding in `app.py` is never touched; and because -priority is demoted rather than the finding deleted, the audit trail -(`applied_calibrations.json`, the ledger) shows exactly why a surface lost -attention. Covered by `tests/test_pipeline_calibration.py`. +Each benchmark run writes `.openultrasast/benchmarks//` (`benchmark_result.json`, +`calibration_records.json` with every miss, `external_baseline_deltas.json`). The pair +catalogs live in `benchmarks/pairs/`. -### Rule-level loop: `ousast improve` +## Self-improvement -The ranking loop above adjusts *attention*. A second, bounded loop adjusts the -**governed ruleset itself** from benchmark feedback β€” and writes its result to the -same loop-owned ledger a scan reads, so the loop closes end-to-end: +`ousast improve` runs a bounded loop over the governed ruleset: benchmark, propose edits to +rule status and score constants only, validate, re-benchmark, and accept a round only if +recall stays at least 90%, false positives under 10%, the project score does not regress and +no matched finding is lost. Accepted rounds land in +`/.openultrasast/calibration/rule_policy.json`, which the next scan of that target loads. ```bash -# Preview what the loop would change (writes nothing) uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run - -# Apply: accepted rounds land in /.openultrasast/calibration/rule_policy.json, -# which the next `ousast scan`/`benchmark` of that target loads automatically. -uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml -``` - -Each round (`improve/evolve.py`): benchmark with the current ledger β†’ per-rule -signals (`rule_signals.json`) β†’ propose **bounded** edits β†’ validate β†’ replay -smoke β†’ re-benchmark β†’ a **hard acceptance gate** β†’ accept or revert -byte-for-byte. The loop may only pull two levers (rule status, score constants); -it can never edit a rule's pattern text or the authoritative 0–5 CWE severity, and -must stage `enabled β†’ shadow` before `disabled`. A round is accepted only if -**recall β‰₯ 90% AND FP < 10% AND the project score did not regress AND no matched -finding was lost** β€” otherwise the ledger is never written. Auto-shadowing removes -a precision-dragger's false-positive cost without deleting it (recall is -preserved); persistent misses are *nominated* through the signal file, leaving -pattern authoring to a human pull request. The project score is the optimization -reward; the recall/FP gate is a separate hard constraint that is never folded into -it β€” and the same gate runs in CI (`python -m openultrasast.gate`), so a loop -result that breached it could never merge. Covered by `tests/test_improve.py` and -`tests/test_cli_improve.py`. - -This is the deterministic substrate of the self-improvement loop. The **plane -memory** is the richer proposer that plugs into it: with `--memory`, `ousast -improve` also reads the rows that plane runs stored per repository and pin, and two -deterministic rules turn them into rule-status proposals: a rule whose alerts are -repeatedly false across repositories is shadowed, and a shadow rule that keeps -hitting agreed, declared vulnerable sites is re-enabled. Rows from holdout pairs, -from the gated manifest's own cases and from the qualifying population are dropped -before any rule sees them, so the loop never learns from the cases it is judged on. Each -proposal records the memory rows it came from (`memory_proposals.jsonl` next to the -journal), and it goes through the same validator and gate as every other edit. - -```bash uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory ``` -The whole loop (alerts, measure, propose, improve) can also run as a plane Run -(generated by `ousast plane workspaces --validation-set ... --loop`), with its own tasks, Models -and budgets; see [ops/ax/README.md](ops/ax/README.md). +With `--memory`, proposals also come from the rows plane runs stored (see below), through the +same validator and gate; rows from holdout pairs, from the gated manifest's own cases and from +any `--qualify-population` are dropped first. Details: +[docs/architecture.md](docs/architecture.md#self-improvement). -Beyond these loops, the `openultrasast-triage` skill lets an agent adjust -prompt constraints, retrieval filters, skill routing and benchmark-miss triage. -Those `RankingCalibration` fields are already produced and will be consumed -directly once the per-family detectors land. +## The agentic plane (maintainers) -## Security & hardening +Model work over whole repositories runs as a Run manifest on [google/ax](ops/ax/README.md) +over Agent Substrate (a kind cluster on the maintainer's host), one isolated actor per task. +Each task binds its own Model and its own `usd`/`calls` budget, egress is deny-by-default per +task, and the provider key travels only in the task's start request. -OpenUltraSAST treats scanned code as **untrusted input and never executes it** (quick -and standard modes). Scan artifacts are scrubbed of credentials before they are written -(`[hardening] redact_secrets`, on by default), agentic spend and output size are bounded -(per-task `usd`/`calls` budgets on the plane, `[hardening] max_findings`), provider calls -retry transient failures with backoff, and missing capabilities degrade visibly in the -manifest rather than silently. See [docs/threat-model.md](docs/threat-model.md) for the -full trust boundaries and sandbox limits, and [docs/examples.md](docs/examples.md) for -end-to-end walkthroughs. +```bash +uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image +uv run ousast plane run plane/runs/validation-46.yaml # a rerun skips tasks already done +uv run ousast plane status validation-46 # per-task status and token attribution +uv run ousast plane remember validation-46 # ingest the run's rows into the memory store +``` + +`ousast plane workspaces`, `alerts-engine` and `harvest` generate Runs and inputs. Bring-up, +the task catalogue, the memory store (`OUSAST_MEMORY`) and the checklist for moving the plane +to another Kubernetes cluster are in [ops/ax/README.md](ops/ax/README.md). + +> HarnessX, the earlier optional agentic extra, was retired 2026-09-30, and a leftover +> `[harnessx]` section (retired 2026-09-30) is ignored with one warning. `[models] verifier`, +> `[fusion] panel_model` and `[fusion] decider_model` fail with a message naming the plane +> replacement. The LLM judge and the LLM fusion panels were retired with it; fusion is +> deterministic. See [RELEASE_NOTES.md](RELEASE_NOTES.md). + +## Decision engine (in development, not adopted) + +A learned decision engine is being built to replace hand-tuned detection edits: each +instrument (quick rules, the Joern engine, model sink classification, verify passes, repository +facts) produces signals, never verdicts, and a compiled AI classifier with local memory turns +them into a calibrated probability per candidate with BLOCK and ADVISORY operating points. +No scan, pre-push or report uses it yet. Status and the first measured slice: +[docs/decision-engine.md](docs/decision-engine.md). + +## Agent integrations + +An agent that can run shell commands needs nothing but the CLI. For +[OpenCode](https://opencode.ai) there are optional project skills in `.opencode/skills/`: +`openultrasast-scan`, `openultrasast-triage` and `openultrasast-fix-audit`. + +`ousast mcp` runs a narrow MCP server over stdio (newline-delimited JSON-RPC) with ten tools: +`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`, +`benchmark`, `explain`, `propose_patch` (degrades visibly) and `export_report`. No tool runs a +shell, Docker or a free-form command. + +```jsonc +{ "command": "uv", "args": ["run", "ousast", "mcp"] } +``` + +> The `kiro-*` skills, `AGENTS.md` and `.kiro/` are the maintainer's development tooling and +> can be ignored by users. + +## Further reading + +- [docs/architecture.md](docs/architecture.md): the scan pipeline, evidence ladder, CWE policy + and project score, and the self-improving loops. +- [docs/evaluation.md](docs/evaluation.md): independent populations, the qualification gates, + pair corpora and the plane increment results. +- [docs/decision-engine.md](docs/decision-engine.md): the learned decision engine. +- [docs/threat-model.md](docs/threat-model.md): trust boundaries, sandbox and hardening. +- [docs/examples.md](docs/examples.md): end-to-end walkthroughs. ## Development @@ -424,7 +236,11 @@ uv run pytest # tests, incl. the 90/10 detection gate uv run ruff check . # lint uv run ruff format --check . # format uv run mypy src/openultrasast # types -uv run python -m openultrasast.gate # standalone detection gate -uv run python -m openultrasast.pair_gate # local vuln-vs-fix efficiency + GitHub honesty -uv run python dagger/ci.py # full containerized CI pipeline +uv run python -m openultrasast.gate # standalone detection gate +uv run python -m openultrasast.pair_gate # local vuln-vs-fix pairs +uv run python dagger/ci.py # containerized CI pipeline ``` + +Maintainer commands not covered above: `ousast repos` (pinned known-vulnerable checkouts), +`ousast model candidates` (what the candidate enumerator can reach) and `ousast learn` +(the decision engine's data; see [docs/decision-engine.md](docs/decision-engine.md)). diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 10d308c..33bc88f 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,5 +1,39 @@ # Unreleased +## PHP quick-mode rules (2026-09-30) + +`ruleset/php/rules.toml` adds 18 PHP rules (SQL injection, command injection, file inclusion, +path, open redirect, SSRF; code eval, unserialize and echo XSS as `shadow`). They were written +on the development corpus, so their numbers are in-sample only +(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`); PHP is not part of +the detection gate. `ousast plane alerts-engine` produces a Run's `alerts` for PHP from the +Joern engine image on the host. + +## Framework knowledge as optional priors (2026-09-30) + +Rule and taint-fact entries that know a framework or library carry a `framework`/`library` +tag naming a row of `ruleset/frameworks.toml`; the loaders take `priors` (`"all"` by default, +unchanged behaviour; `"off"`; or a set of ids). Scan output is unchanged. + +## Decision engine, in development (2026-10-01) + +The learned decision engine (signals from every instrument, labels from ground truth, an AI +classifier with local memory compiled in house) is in development and **not adopted**: no scan, +pre-push check or report uses it. New maintainer commands: `ousast learn labels|memory|compile| +evaluate|curve|audit-leaks` and `ousast plane harvest`; new plane tasks `features` and `roles`; +the `advisory-fixes` pair source (97 advisory fix commits). First slice (injection, 110 +out-of-repository candidates): ADVISORY recall 0.90, precision 0.54, BLOCK not offered +(`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`; +[docs/decision-engine.md](docs/decision-engine.md)). + +## Documentation (2026-10-02) + +The README is reduced to user-facing material; scan internals moved to +[docs/architecture.md](docs/architecture.md) and measured results to +[docs/evaluation.md](docs/evaluation.md). Removed references to commands that no longer exist +(`ousast mechanisms export`, `ousast pairs --loo`) and corrected the description of `deep` +mode, which runs sandboxed regression snippets when Docker is available. + ## HarnessX removed; the ax plane is the agentic path (2026-09-30) HarnessX, the optional agentic extra (`openultrasast[harnessx]`), is removed with its code, diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..399a36a --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,160 @@ +# Architecture + +How a scan works, how a finding earns its place, how severity and the project score are +decided, and how the ruleset improves itself. State as of 2026-10-02; module paths are under +`src/openultrasast/`. + +## The scan pipeline + +The CLI is thin; `HarnessRuntime` owns the lifecycle and writes an event trace for every stage +(`trace/events.jsonl`). The stages a mode requests are `MODE_STAGES` in `stages.py`: +`quick` runs STATIC, `standard` adds MAP, `deep` adds REGRESS. + +``` +policy_load ─▢ ruleset_load ─▢ policy_check (fail loud on an ungoverned CWE) + ─▢ static_mapping (SARIF ingest) ─▢ preprocess ─▢ entry_point_mapping + ─▢ rank ─▢ calibrate (demote previously rejected scopes) + ─▢ findings quick, deep: quick_findings (language-scoped pattern rules) + β”‚ standard: hunter_pool (the same rules, scheduled per ranked target) + ─▢ verify ─▢ record_calibration ─▢ fusion (standard only, deterministic) ─▢ score + ─▢ MAP (standard, deep) + β”‚ map complexity map and hotspots + β”‚ overlay semantic overlay over the findings (tree-sitter with the `semantic` extra) + β”‚ obligations absence bugs: an obligated operation with no governing guard + β”‚ model one Joern CPG per repository; taint, dominance and configuration arbiters + β”‚ tool hunter only with `[models] hunter` + ─▢ REGRESS (deep) sandboxed regression snippets ─▢ verdicts.json, worth-fixing + ─▢ report.md, report.sarif, manifest.json +``` + +**The model layer** (`model/`, `cpg/`) is the core of `standard`. It builds one code property +graph per scan with Joern (`joern-parse`, then batched `joern --script` queries from +`cpg/queries/*.sc`) and dispatches each question to the arbiter of its spec type: taint +(source to sink with no discharging sanitizer), dominance (does a guard govern the obligated +operation) and configuration (constant abstraction over a value). Taint facts ship for C, +Java, JavaScript, PHP and Python (`ruleset/semantic/*.toml`). Without Joern the layer records +`cpg_unavailable` and the scan is otherwise unchanged; the Docker image ships Joern and +`php-cli` (which the PHP frontend needs). + +The model layer reports on a rung ladder: + +``` +execution_confirmed a test or sandbox reproduced it +model_entailed the graph decided: a flow with no discharge, an obligation with no + guard, a permissive value +model_corroborated the graph found the shape; a judge agreed on the residual question +suspicion proposed, nothing corroborated it +``` + +The LLM (`model/judge.py`) is asked only the residual question the graph cannot settle, for a +flow the graph already found; it can confirm or contradict, never create a flow. Without a +provider key only the `suspicion` band goes unasked. + +**REGRESS** (`regress/`, `sandbox/`) runs only when `docker info` succeeds. Each promoted +candidate gets a small language recipe (Python, JavaScript, C) that loads the candidate's +source inside a container with no network, a read-only `/workspace`, a non-root user and the +`[sandbox]` limits. A structural deny-list rejects snippets mentioning sockets, curl, host +networking or writes under `/workspace` before anything runs. + +## Evidence ladder: how a false positive is eliminated + +Evidence level is a state machine, not a label a model may assign. The verifier +(`verification.py`) runs on independent context and enforces: + +``` +suspicion ─▢ static_corroboration ─▢ crash_reproduced ─▢ root_cause_explained + ─▢ exploit_demonstrated ─▢ patch_validated +``` + +- below `static_corroboration`: `NEEDS_EVIDENCE`; +- `static_corroboration` with reachability unknown: `REJECTED`, with a tie-breaker demanding + call-graph, route, CLI, parser or dynamic evidence; +- `static_corroboration` with function-level reachability: `ACCEPTED`. + +A rejected finding becomes a **scoped false-positive learning** (`calibration.py`) with a +reason from a fixed taxonomy (`unreachable_path`, `missing_attacker_control`, +`sanitizer_disproved`, `static_rule_mismatch`, `incorrect_model_assumption`, `duplicate`, +`insufficient_impact`, `unsupported`, `contradicted`, `unverified`) and a scope. A rejection in +`auth/` demotes only `auth/...`, never the whole class +(`tests/test_calibration.py::test_scoped_false_positive_demotion_does_not_suppress_class_globally`). + +## How a false negative is surfaced + +Every expected vulnerability that no finding matched becomes a `BenchmarkCalibrationRecord` in +`calibration_records.json`, with a `next_improvement_candidate` (a static rule, SARIF source, +entry-point mapping, retrieval package, hunter prompt, dynamic reproducer or skill route). +Once a miss is closed the detection gate keeps it closed in CI. + +## Central CWE policy and the project score + +**Policy** (`policy/verycode.py`): a vendored `CWE_Score.tsv` (verycode-policies) is the single +source of truth. Each CWE carries a flaw category, a severity (0-5) and `static`/`dynamic` scope +flags. A finding's severity comes only from policy, keyed on CWE. CWEs that are not `static` +resolve to 0 and are report-only. `assert_rules_resolve` runs as the `policy_check` stage: an +enabled rule naming an ungoverned CWE aborts the scan. Verycode has no CWE-120, so the C/C++ +buffer rules carry CWE-121. + +**Score** (`scoring/project_score.py`): each finding's penalty is +`SEV_WEIGHT[severity] x REACH_MULT[reachability]`, and the score is `100 * e^(-total/k)` with +`k = 60`. + +| Severity | Weight | | Reachability | Multiplier | +| --- | --- | --- | --- | --- | +| 5 | 50 | | `reachable` | 1.0 | +| 4 | 25 | | `inferred-file-surface` | 0.6 | +| 3 | 10 | | `unknown` | 0.4 | +| 2 / 1 / 0 | 2 / 1 / 0 | | | | + +A confirmed false positive lowers a finding's reachability multiplier instead of deleting the +rule. A severity-5 reachable finding always fails the score gate; `score < min_score` +(default 80) fails it only with `[score] blocking = true`. The `score` stage writes +`score.json` and merges it into `manifest.json`. + +## Self-improvement + +### Scoped calibration (every scan) + +``` +scan ─▢ verify ─▢ non-accepted findings ─▢ scoped learnings + (.openultrasast/calibration/false_positive_learnings.json) +next scan ─▢ rank ─▢ calibrate (demote rejected scopes) ─▢ findings +``` + +`record_calibration` (after `verify`) merges this run's non-accepted outcomes into the ledger, +de-duplicated by finding ID; `calibrate` (after `rank`) demotes the priority of every scope +that produced a rejection and writes `applied_calibrations.json`. Demotion is scoped and +reversible (`tests/test_pipeline_calibration.py`). + +### Rule-level loop: `ousast improve` + +Each round (`improve/evolve.py`): benchmark with the current ledger, per-rule signals +(`rule_signals.json`), bounded proposals, validation (`improve/validator.py`), replay smoke, +re-benchmark, and a hard gate. The levers are rule status (`enabled -> shadow` before +`disabled`) and score constants; the loop can never edit pattern text +or the authoritative CWE severity. A round is accepted only if recall >= 90%, FP < 10%, the +project score did not regress and no matched finding was lost; otherwise the ledger is not +written. With `--pair-catalog`, every round is also gated per provenance profile on the +catalog's holdout pairs (`--no-pair-gate` skips this and is not for accepted ledgers). The same +detection gate runs in CI (`python -m openultrasast.gate`). + +### Proposals from plane memory + +With `--memory [STORE]`, `ousast improve` also reads the rows plane runs stored per repository +and pin (`improve/memory.py`). Two deterministic rules turn them into rule-status proposals: a +rule whose alerts are repeatedly false across repositories is shadowed, and a shadow rule that +keeps hitting agreed, declared vulnerable sites is re-enabled. Rows from holdout pairs, from the +gated manifest's own cases and from any `--qualify-population` are dropped before any rule sees +them. Each proposal records the rows it came from (`memory_proposals.jsonl` next to the +journal) and goes through the same validator and gate. + +### The loop as a plane Run + +`ousast plane workspaces --validation-set ... --loop` appends the improvement loop +to a generated Run: `alerts` per case (the quick scan on the vulnerable and fixed pins), then +the steps `snapshot` (task `memory-snapshot`), `measure`, `propose` and `improve` (`plane/tasks/loop.py`), each model-free +with a zero budget. The snapshot applies the same train-on-test guard before any row leaves the +store. A recorded live run: `benchmarks/measurements/2026-09-30-harnessx-removal-loop/`. + +For PHP, which quick mode covered with zero rules when that run was made, `ousast plane +alerts-engine` produces a Run's `alerts` from the engine image on the host instead +(`benchmarks/measurements/2026-09-30-php-engine-alerts/`). diff --git a/docs/decision-engine.md b/docs/decision-engine.md new file mode 100644 index 0000000..a92209b --- /dev/null +++ b/docs/decision-engine.md @@ -0,0 +1,85 @@ +# The learned decision engine (in development) + +**Status 2026-10-02: in development, not adopted.** No scan, pre-push check or report uses it; +every user-facing path is today's deterministic one until a compiled program is adopted. The +design is the `learned-decision-engine` specification (maintainer tooling); the code is in +`src/openultrasast/learn/`. + +## Why + +Every detection gain so far was a hand edit made after a miss: a rule, a taint vocabulary entry, +a guard, a prompt line, a tie-break. Each was tuned on the cases that exposed it, and engine +tuning measured on population v1 did not transfer to v2 (see [evaluation.md](evaluation.md)). +The engine replaces that with a decision learned from labelled outcomes and checked out of +repository. + +## Design + +- **Signals, not verdicts.** Every instrument (quick rules, the Joern engine, model sink and + role classification, the plane's verify passes and agreement, repository facts such as + callers) contributes a feature record per candidate `(path, function, family)`, including an + explicit state when it did not run or failed. None decides alone. +- **Labels from ground truth.** `ousast learn labels` derives labels only from sources on a + fail-closed allow-list (`learn/sources.toml`): the vulnerable and fixed sides of pairs, the + declared sites and fix ranges of spent populations, benign deltas and recorded adjudications. + "Rejected by the plane" is never a negative. Reserved populations (v3) are not on the list. +- **An AI classifier with local memory, compiled in house in the DSPy style.** The program is an + instruction, a few demonstrations and up to six retrieved labelled examples, then the + candidate's code excerpt and signals; the model (`deepseek-flash`) answers + vulnerable / not vulnerable / unsure with a rationale, sampled `k` times, giving a score `s`. + Compilation bootstraps demonstrations and searches instructions on a compile split, scored by + a pre-registered balanced Brier score (`learn/compile.toml`). +- **Retrieval** from the memory store: nearest neighbours by Gower distance on the signal + profile, re-ranked by cosine similarity of code embeddings (OpenAI `text-embedding-3-small` + through OpenRouter, cached by excerpt hash), under a label/family/repository balance rule. An + evaluation-boundary filter keeps a candidate's own repository group out of what it is shown. +- **Repository-grouped evaluation.** The compile split (25% of repository groups) is never + evaluated. The rest is evaluated by outer grouped 5-fold cross-validation: the Platt + calibration map is cross-fitted on the other folds and the operating points are chosen on the + other folds (nested). One compile per profile, not one per fold. Intervals are 95% + repository-cluster bootstrap. +- **Two operating points.** BLOCK is offered only when calibration holds (slope CI contains 1, + intercept CI contains 0, ECE <= 0.05) and a fold reaches the required precision; otherwise the + engine is ADVISORY only. An `unsure` answer can never BLOCK. + +## First measured slice (2026-10-01) + +Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts +only). Family **injection** only; memory of 724 static examples from all families; input profile +v1 with the engine signals and function length withheld (a leak audit found them separating the +sides of a pair by construction: `2026-10-01-decision-engine-leak-audit/`); framework priors off. + +| Measure | Value (95% repository-cluster CI) | +| --- | --- | +| Candidates evaluated | 110 in 29 repository groups, 58 positive | +| ADVISORY recall | 0.90 (0.82 - 0.97) | +| ADVISORY precision | 0.54 (0.44 - 0.65) | +| AUC of the score `s` | 0.74 (0.65 - 0.85) | +| Majority "vulnerable" verdict | precision 0.86 (0.72 - 1.0), recall 0.43 (0.29 - 0.62) | +| BLOCK | not offered: ECE 0.064 > 0.05, and every fold reported `precision_unreachable` | +| Compiled program | balanced Brier 0.128 on the validation split vs 0.199 for the baseline instruction | +| Canary agreement on re-run | 0.83 (30 candidates) | +| Spend | $0.91 metered at list price for DeepSeek, $0.0040 for embeddings; $0.0032 per evaluated candidate | + +The other evaluable families are not evaluated yet. Leave-one-source-out, leave-one-framework-out +and learning curves, experiments, integration into scan and pre-push, adoption, and the one-time +v3 check are still ahead. + +## Commands + +`ousast learn` is a maintainer command group over the engine's data. Paid steps take +`--budget-usd`; `--replay-only` answers every call from the response cache at no cost. + +| Command | Does | +| --- | --- | +| `ousast learn labels` | labels from the allow-listed sources (host only); `--snapshot` writes the counts-only record | +| `ousast learn memory build` | label + feature record + excerpt -> `example` rows (no model) | +| `ousast learn memory embed` | embed the stored excerpts (OpenRouter, metered, cached) | +| `ousast learn compile` | compile the program on the compile split | +| `ousast learn evaluate --program ID` | outer-fold evaluation, cross-fitted calibration, nested operating points | +| `ousast learn curve --program ID` | the metric over memory size on a fixed candidate subset | +| `ousast learn audit-leaks` | how well each signal alone separates the sides of a pair (no model) | + +The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](../ops/ax/README.md)). +The harvest that produced the plane signals runs as plane Runs generated by +`ousast plane harvest` (`benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json`). diff --git a/docs/evaluation.md b/docs/evaluation.md new file mode 100644 index 0000000..945f9ec --- /dev/null +++ b/docs/evaluation.md @@ -0,0 +1,84 @@ +# Evaluation + +What has been measured on code the tool was not tuned on, and how. State as of 2026-10-02. +Every number below is quoted from a committed record; the path is given with it. + +## Instruments + +| Instrument | Measures | Where | +| --- | --- | --- | +| Detection gate | per-language recall/FP on the bundled cheat-sheet corpora (goal: >= 90% recall, < 10% FP) | `python -m openultrasast.gate`, `benchmarks/manifests/` | +| Pair corpus | fire on the vulnerable function, stay silent on its fix | `ousast pairs`, `benchmarks/pairs/` | +| Independent populations | real repositories at vulnerable, fixed and benign pins, scored under a pre-registered protocol | `benchmarks/independent/` | + +The detection gate and the local pairs are CI gates. They catch regressions; they do not +estimate recall on real code, because the rules were written against them. + +## Pair corpora + +`benchmarks/pairs/catalog.toml` and the slice directories (`local`, `github`, `sast`, `vfc`, +`vibe-py`, `vfc-js`, `agent-vfc`, `owasp`) are runnable with `ousast pairs --slice `; each +pair carries a provenance profile and a train/holdout split (`--profile`, `--split`). Pointer +pairs are not vendored: `--pointers` fetches them into `~/.cache/openultrasast/pairs/`. +`benchmarks/pairs/README.md` documents the schema and the review tiers. + +**`advisory-fixes`** (`benchmarks/pairs/advisory-fixes/`, 2026-10-01) holds 97 pointer pairs, +one per repository: a public CVE/GHSA fix commit whose advisory-named function is declared on +both sides and changed by the fix. It was assembled to give the decision engine real negatives +for families the corpus had few of (access control, deserialization, output encoding, path, +untrusted destination). It is a label source of the decision engine +(`src/openultrasast/learn/sources.toml`), not an `ousast pairs` slice; the research log and +every rejection are in `research-2026-10-01.md`. + +## Independent populations and the M4 gates + +The pre-push safety net's qualification milestone (M4) requires, together: at least 95% +actionable precision, at least 90% supported recall, at least 95% supported-check completion, +the runtime gates, and zero fixed-side or benign false alerts. A population is frozen before +any of its cases is scanned, scored once under a pre-registered protocol, and then **spent**: +it may inform changes, but it can no longer qualify anything. + +| Population | Status | Result | Record | +| --- | --- | --- | --- | +| v1 (11 cases) | spent | engine: recall 1/11, precision 1/12, 4 fixed-side and 1 benign false alert; every M4 gate failed. Eight of eleven cases completed no question at the vulnerable pin. | `benchmarks/independent/results-v1.json` | +| v2 (17 cases) | spent | engine: recall 0/17, 3 benign alerts, completion 13,206 of 108,374. Tool hunter: recall 1/17, precision 2/14, 1 fixed-side and 16 benign alerts, $8.88. No M4 gate met by either. | `results-v2.json`, `results-v2-hunter.json` | +| v2, model-driven pipeline | exploratory (v2 already spent) | stopped after 7 of 17 cases when the account emptied: recall 4/7; precision not adjudicated; no gate measurable | `results-v2-model-pipeline.json` | +| v3 (PHP, 16 cases) | frozen, untouched | none: it is the one-time final check, run once under `protocol-v3.md` after a prediction is committed | `population-v3-php.toml`, `freeze-v3-php.json` | + +v3 is reserved: development and analysis work never reads its cases, and +`tests/test_independent_population.py` fails when anything in the development tree (a recipe, +catalog, manifest or measurement) names one of its repositories. + +## The plane increment + +The first increment of the agentic plane ran the model-driven pipeline's validation set (46 +candidates in 15 cases of v2, 20 declared vulnerable sites) on google/ax and compared it with +the script pipeline on the same cases, candidates, pins and recorded triage. The pre-registered +gate: at least 16 of 20 declared sites agreed, at a cost per candidate under $0.022. + +- **Increment 1** (2026-09-29, `benchmarks/independent/plane-increment-1.json`): 14 of 20 + declared sites agreed against the reference's 16; $0.0194 per candidate with recorded triage. + **NOT MET** on recall. + *Correction (2026-09-30, in the record):* the first version read agree's + `declared_sites_matched` (coverage by candidates, 19) as "found in either pass" and concluded + detection was unchanged. Recounted from the agreed rows, the plane's passes flagged 17 of 20 + declared sites in at least one pass against the reference's 19. +- **Increment 2** (2026-09-30, `benchmarks/independent/plane-increment-2.json`): after the design + revisit, a tie-break pass c on the 10 disputed candidates with 2-of-3 agreement. 16 of 20 + declared sites agreed (reference 16) at $0.0205 per candidate over 46 ($0.02195 over the 43 + after triage, against the reference's $0.022). **MET**, with two caveats the record carries: + the tie-break was chosen after seeing the first measurement on the same validation set, so + this compares plane with script and qualifies nothing on an independent population; and the + plane's passes still flag fewer declared sites than the reference (17 vs 19). Precision is + not part of this gate and remains the open problem. + +## The decision engine + +The first measured slice of the learned decision engine (injection family, out-of-repository +evaluation) is described with its numbers in [decision-engine.md](decision-engine.md). + +## Removal of the earlier agentic extra + +The removal of the earlier agentic extra (2026-09-30) was checked for byte-identical +deterministic outputs (pair replay, quick benchmark, standard scan, `improve --dry-run`): +`benchmarks/measurements/2026-09-30-harnessx-removal-baseline/` and `-equality/`. diff --git a/docs/examples.md b/docs/examples.md index 87e6f98..926b8ed 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -13,7 +13,18 @@ jq . "$run/score.json" # 0–100 project score + gate verdict "$run/report.sarif" # SARIF for code-scanning / IDEs ``` -## 2. CI gate +## 2. Standard scan with the engine (Docker) + +```bash +source ops/shell/ousast.sh # once per shell; builds the image on first use +ousast scan . --mode standard # Joern model layer, container network off: no model calls +ousast-with-judge scan . --mode standard # network on, so the model is asked (needs DEEPSEEK_API_KEY) +``` + +Without a key the graph still decides; only the `suspicion` band goes unasked, and the manifest +records the degradation. + +## 3. CI gate ```bash # Fail the build on any evidence-verified finding: @@ -23,23 +34,21 @@ uv run ousast scan . --mode quick --fail-on verified uv run python -m openultrasast.gate ``` -## 3. Benchmark a known-vulnerable corpus +## 4. Benchmark a known-vulnerable corpus ```bash uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick # -> expected / matched / missed counts + a per-rule recommendation delta ``` -## 4. Vuln vs fixed pair eval (efficiency, not cheat-sheet recall) +## 5. Vuln vs fixed pair eval (efficiency, not cheat-sheet recall) ```bash uv run ousast pairs # local fixtures + vendored GitHub VFCs uv run ousast pairs --slice github --json # honesty dashboard as JSON uv run ousast pairs --slice sast # OWASP Benchmark + Juliet Youden (TPR-FPR) uv run ousast pairs --slice vibe-py --pointers # nightly: pointer pairs via the local cache (network) -uv run ousast mechanisms export --slice vibe-py # maintainer: seed mechanism candidates from trusted pairs (offline) -uv run ousast improve bench.toml --pair-catalog benchmarks/pairs/catalog.toml # the mechanisms lever admits candidates the holdout pairs prove; the scan then searches them -uv run ousast pairs --slice vibe-py --loo # leave-one-out: the corpus's own detection rate (reports/loo.json) +uv run ousast pairs --split holdout --profile human # filter by declared split and provenance profile uv run python -m openultrasast.pair_gate # CI: local pairs must all pass ``` @@ -48,7 +57,7 @@ Each case is two isolated trees (vuln file vs patched file, same relative path). fix. Misses and fix-side leaks become `miss`/`fp` signals for `ousast improve`. The catalog and public dataset pointers live in `benchmarks/pairs/`. -## 5. Self-improve the ruleset from benchmark feedback +## 6. Self-improve the ruleset from benchmark feedback ```bash uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run # preview @@ -57,16 +66,16 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml # which the next scan/benchmark of that target loads automatically. ``` -## 6. Running the agentic plane +## 7. Running the agentic plane Model work over whole repositories runs as a Run manifest on ax, one isolated actor per task (bring-up: [ops/ax/README.md](../ops/ax/README.md)). The provider key is -read from the environment (or `.env`) by `ousast` and sent only in each task's start -request. +read from the environment (or `.env`, which never overrides an exported variable) by +`ousast` and sent only in each task's start request. ```bash uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image -uv run ousast plane run plane/runs/validation-46.yaml # repo-facts, verify a/b(/c), agree, remember per case +uv run ousast plane run plane/runs/validation-46.yaml # per case: repo-facts, verify a/b/c, agree, final, features, remember uv run ousast plane status validation-46 # per-task status and token attribution uv run ousast plane remember validation-46 # re-ingest the run's memory rows (OUSAST_MEMORY) @@ -76,14 +85,14 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dr Each task in the Run carries its own Model and `budget: {usd, calls}`; a task that reaches its ceiling stops as `unfinished` and resumes on a rerun with a larger -budget. A local `ousast scan --mode standard` stays deterministic apart from the -optional `[models] hunter` tool hunter. +budget. `ousast plane status` prints the token attribution table: calls, prompt, +cache-hit and output tokens and USD per task. -## 7. Drive it from an MCP client (OpenCode / IDE) +## 8. Drive it from an MCP client (OpenCode / IDE) ```jsonc { "command": "uv", "args": ["run", "ousast", "mcp"] } ``` Then: `openultrasast.scan {path}` β†’ `run_dir` β†’ `openultrasast.findings {run_dir}` β†’ -`openultrasast.explain {run_dir, finding_id}`. See the README "MCP server" section. +`openultrasast.explain {run_dir, finding_id}`. See the README section "Agent integrations". diff --git a/docs/threat-model.md b/docs/threat-model.md index 008801a..5a90711 100644 --- a/docs/threat-model.md +++ b/docs/threat-model.md @@ -5,18 +5,23 @@ it does not, and the controls that keep a scan safe to run in CI and safe to sha ## Trust boundaries -- **Scanned code is untrusted input, never executed.** Quick and standard modes read - source as text (regex rules, evidence ladder, optional LLM review). They never import, - build, or run the target. Only `deep` mode would execute target-derived code, and only - inside the sandbox below β€” and deep mode is not yet implemented (it exits with a clear - message rather than running anything). +- **Scanned code is untrusted input.** Quick and standard modes never import, build or run + the target: quick reads source as text, and standard additionally parses it into a code + property graph with Joern (`joern-parse`, a parser, not a build) and may send code excerpts + to the configured model. Only `deep` mode executes target-derived code: its REGRESS stage + loads promoted candidates inside the Docker sandbox below, and it runs only when `docker + info` succeeds (otherwise the stage is skipped and recorded as `sandbox_unavailable`). - **The ruleset and CWE policy are trusted, governed data.** They change only through reviewed commits or the bounded self-improvement loop (`ousast improve`), which can flip rule *status* and tune score constants but can never edit pattern text or the - authoritative 0–5 severity. See the self-improving cycle in the README. -- **Provider credentials are the operator's.** The zero-dependency core makes no network - calls. A local scan calls a model only for the optional `[models] hunter` tool hunter - in standard mode. Agentic work runs on the plane (`ousast plane run`): `ousast` reads + authoritative 0–5 severity. See [architecture.md](architecture.md#self-improvement). +- **Provider credentials are the operator's.** The core install (PyYAML only) makes no + network calls on its own. A local scan calls a model only in standard or deep mode, and + only when a key is available: the model layer asks the residual question the graph cannot + settle, and `[models] hunter` enables the tool hunter. LLM calls go to DeepSeek + (`DEEPSEEK_API_KEY`); OpenRouter (`OPENROUTER_API_KEY`) serves embeddings. Keys come from + the environment or a gitignored `.env`, which never overrides an exported variable. The + pre-push check calls a model only with an explicit `--model-config`. Agentic work runs on the plane (`ousast plane run`): `ousast` reads the variable a task's Model names (`secretKey.key`) from the operator's environment or `.env` and sends the key only in that task's start request, through Agent Substrate's router to that task's actor. It is not baked into an image or task template, and it is @@ -54,7 +59,7 @@ LLM/embedding calls retry transient failures (HTTP 429/5xx, connection errors, t with exponential backoff; non-transient errors (e.g. 4xx, malformed JSON) fail fast. ### Visible degradation & determinism -When an optional capability is missing (the extra, a model, a provider key), the scan +When an optional capability is missing (Joern, Docker, a model, a provider key), the scan falls back to its deterministic equivalent and records a `degradations` entry in the manifest β€” it is never silently downgraded. Runs are reproducible: fixed config, artifact manifests, prompt hashes, and model identifiers are recorded. @@ -63,18 +68,20 @@ artifact manifests, prompt hashes, and model identifiers are recorded. `ousast mcp` exposes only the ten narrow project tools. No tool runs arbitrary shell, Docker, or internal hunter tools, and no tool accepts a free-form command argument. -## Sandbox limits (deep mode, when implemented) +## Sandbox (deep mode) -Dynamic analysis runs under the bounds in `[sandbox]` (`config.SandboxConfig`): +The REGRESS stage (`regress/`, `sandbox/runner.py`) runs each snippet with `docker run` +and never mounts the host Docker socket: -| Control | Default | +| Control | Setting | |---|---| -| network | disabled (`network = false`) | -| workspace | read-only (`workspace_readonly = true`) | -| memory | 2048 MB (`memory_mb`) | -| pids | 512 (`pids_limit`) | -| timeout | 300 s (`timeout_seconds`) | +| network | `--network none`, always; a host-network argument is refused | +| source | the target bind-mounted read-only at `/workspace`; root filesystem `--read-only` | +| scratch | a tmpfs at `/scratch` for the snippet | +| user | non-root `65534:65534`, `--cap-drop ALL`, `no-new-privileges` | +| memory | `[sandbox] memory_mb`, default 2048 MB | +| pids | `[sandbox] pids_limit`, default 512 | +| timeout | `[sandbox] timeout_seconds`, default 300 s | -Dynamic probes additionally declare their network scope, log commands, and capture -artifacts; a hardened runtime (e.g. gVisor) is used when available. Until deep mode -ships, no target code is executed at all. +Before anything runs, a structural deny-list (`regress/safety.py`) rejects a snippet that +mentions sockets, curl, host networking or writes under `/workspace`. diff --git a/ops/README.md b/ops/README.md index fd42e0d..f04bc8d 100644 --- a/ops/README.md +++ b/ops/README.md @@ -1,7 +1,7 @@ # ops -Infrastructure for the optional engines. Nothing here is a project dependency: the core install is -`dependencies = []` and every engine below degrades to a recorded reason when it is absent. +Infrastructure for the optional engines. Nothing here is a project dependency: the core install needs only +PyYAML, and every engine below degrades to a recorded reason when it is absent. ## Use it as a command @@ -22,7 +22,7 @@ Then, from any directory: ```bash ousast scan . ousast scan src/api -ousast-with-judge scan . # opt into the LLM judge: needs network and a key +ousast-with-judge scan . --mode standard # container network on, so the model layer may ask its question (needs DEEPSEEK_API_KEY) ``` The wrapper rewrites paths under your working directory to their mounted equivalents, so `src/api` means what @@ -48,7 +48,8 @@ TARGET=/path/to/repo docker compose run --rm ousast scan /target The target is mounted read-only; `/work` is a tmpfs so CPG scratch never outlives the run. The LLM endpoint is optional and passed through the environment, never baked into a layer β€” without it the model layer still entails and only the `suspicion` band goes unasked. `network_mode` defaults to `none`; set it to `bridge` -only when you want the judge. +(`OUSAST_NETWORK=bridge`, which `ousast-with-judge` does) only when the model layer should reach the model +endpoint. Compose passes `DEEPSEEK_API_KEY` and `OPENROUTER_API_KEY` through from the environment. **pyinfra (for a persistent host install).** Same engine, same pin, same checksum, but on the machine: @@ -146,7 +147,8 @@ No Joern server mode, no workspace/project management, no `joern-scan`, no overl ## Languages Joern ships frontends for C/C++, Java (source and bytecode), JavaScript/TypeScript, Python, Kotlin, Go, C#, -PHP, ABAP and Ghidra binaries. We currently model **python, javascript, java, c**. +PHP, ABAP and Ghidra binaries. Taint facts ship for **python, javascript, java, c and php** +(`src/openultrasast/ruleset/semantic/*.toml`). **PHP works but needs a PHP interpreter**: `php2cpg` drives PHP-Parser and shells out to `php`, so on a host without it a PHP tree fails to build with an opaque "Process exited with code 1". Set `php_frontend: True` in @@ -372,4 +374,7 @@ read under the same analysis deadline. ## The agentic service plane on ax -Model-driven pipeline stages run as tasks on google/ax over Agent Substrate (`.kiro/specs/ai-service-plane/`). Bring-up, the smoke Run, the measured footprint and the lessons from the live cluster are in [`ops/ax/README.md`](ax/README.md); `ousast plane run|status|doctor|workspaces` drives it. +Model-driven work over whole repositories runs as tasks on google/ax over Agent Substrate. Bring-up, the task +catalogue, the memory store, the measured footprint, the lessons from the live cluster and the checklist for a +separate Kubernetes cluster are in [`ops/ax/README.md`](ax/README.md); +`ousast plane run|status|doctor|workspaces|remember|alerts-engine|harvest` drives it. diff --git a/ops/ax/README.md b/ops/ax/README.md index fd65f3f..803040c 100644 --- a/ops/ax/README.md +++ b/ops/ax/README.md @@ -14,6 +14,74 @@ single-node kind cluster. ax is the only executor; `ousast plane run` submits, r Tools live in `~/go/bin` (kind, ax, ko, kubectl-ate) and `~/.local/bin` (kubectl); sources and rendered files in `~/.cache/ousast/ax-src/` (the runner digest pin is `runner-image` there). +## How a Run executes + +A Run (`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG of steps; each step names an ax Task +(`plane/tasks/`), its inputs and outputs, and its own `budget: {usd, calls}`. Every Task binds its Workspaces +(`plane/workspaces/`) and one ax Model (`plane/models/`): `deepseek-flash` for chat, `openrouter-embedding` +(OpenAI `text-embedding-3-small` through OpenRouter) for embeddings. `ousast plane run` (`plane/reconciler.py`) +applies Workspaces before the Tasks that bind them, resumes each Task, sends its start signal, collects its +artifacts and records state under `$OUSAST_RESULTS/plane//` (default `~/ousast-results/plane/`); a rerun +skips tasks already done, and `--workers` bounds the tasks in flight. + +**Runner contract** (`plane/runner.py`, PID 1 of the runner image). ax hands the Task in `AX_TASK_YAML` and the +Workspaces in `AX_WORKSPACES_YAML`; the runner serves `/healthz` and `/readyz` (503 until every workspace is +materialised). Agent Substrate boots each template once as a golden actor to snapshot it, so the command never +starts by itself: it waits for `POST /ousast/v1/start` with `{run, task, credentials}`, which the reconciler sends +through Substrate's `atenet-router` to that actor only (`plane/router.py`). **The provider key exists only in that +request**: the runner keeps it in memory, passes it to the command's environment, redacts it from echoed stderr +and never writes it. The reconciler reads the variable the Model's `secretKey.key` names from the operator's +environment or `.env` (which never overrides an exported variable). After the command exits, the runner posts +`OUSAST_OUTPUT_DIR` as a tar to the receiver; that delivery is the task's completion, and a completion marker +stops a resumed actor from repeating billed work. + +**Budgets.** The metered client refuses the next call once a task's `usd` or `calls` ceiling is reached; the task +ends `unfinished` and resumes on a rerun with a larger budget. An account or authentication error ends it +`failed`. Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly. + +**Egress** (`plane/egress.py`). One EgressPolicy per actor through the `atenet-egress` gateway, deny by default, +hostnames only: plain HTTP to the artifact receiver, TLS passthrough to the source hosts of the bound Workspaces +and to the hosts the bound Model declares (`openultrasast.io/egress-hosts`). The policy is deleted with the actor. + +**Token attribution.** `ousast plane status ` prints per task its status and `calls`, `prompt`, +`cache_hit` and `output` tokens and `usd` from the tasks' `summary.json`, and writes `/attribution.json`; +`--units` adds per-unit rows. + +## Tasks + +| Task | Model | Does | +|---|---|---| +| `repo-facts` | none | functions per product file, and each candidate's call sites in other files | +| `verify` (passes a, b, c) | `deepseek-flash` | batched tool hunt per file over the candidates, with their known callers | +| `agree` | none | a and b agreed or disputed; after pass c on the disputed, 2-of-3 (the step `final`) | +| `features` | none | one feature record per candidate for the decision engine | +| `remember` | none | the case's artifacts as memory rows (`facts`, `verdict`, `unit_cost`, `alert`, `coverage`, `features`) | +| `alerts` | none | the quick scan on the vulnerable and fixed pins (loop Runs) | +| `loop` | none | the steps `snapshot`, `measure`, `propose`, `improve`: the improvement loop as a Run | +| `roles` | `deepseek-flash` | per-repository source, sink and sanitizer roles without a vocabulary (decision-engine harvest) | + +Generators: `ousast plane workspaces --validation-set ...` writes the per-case chain +(`facts -> va, vb -> agree -> vc -> final -> features -> remember`, plus `alerts` and the loop with `--loop`); +`ousast plane harvest --labels ... --plane DIR` writes the decision engine's harvest Runs; `ousast plane +alerts-engine` produces a Run's `alerts` from the Joern engine image on the host, for PHP and other languages +quick mode does not cover. + +## Memory store + +`plane/memory.py`. One store keyed by repository and pin holds what runs learned: `index.jsonl`, `facts/` by +content hash, `repos/____/.jsonl` rows, and content-addressed blobs (excerpts, embeddings, +cached model responses, compiled programs). `ousast plane remember ` ingests a run's rows; a repeated ingest +is skipped by the index. A facts entry is reused for the same repository, pin, candidates and runner image. + +`OUSAST_MEMORY` selects the backend: + +- `file:///path` (`FileStore`), default `$OUSAST_RESULTS/plane/memory`; it refuses to write below 1 GiB free; +- `minio://[/]` (`MinioStore`, the `minio` extra), used on the maintainer's MinIO. Endpoint and + credentials come from `.env` or the environment (`MINIO_ENDPOINT`, `MINIO_ACCESS_KEY`, `MINIO_SECRET_KEY`, + `MINIO_SECURE`), never from a manifest, and are never printed. + +`ousast improve --memory [STORE]` and the `ousast learn` commands (`--memory`) read the same store. + ## Measured footprint (2026-09-29, idle, two workers) | Item | Value | @@ -62,7 +130,7 @@ are; these parts assume this laptop and must change first (checked 2026-09-30): | ax's snapshot bucket: `AX_SNAPSHOTS_BUCKET` in ax's `deploy/ax-server.yaml` points at the ax authors' GCS bucket | ax deploy manifest | your own bucket, set before deploying ax | | Egress gateway applied by hand (agentgateway variant, no Rust build) | this README | the Substrate-installed gateway; per-task EgressPolicies work unchanged | | Worker pool of 2 x 1 CPU / 1.5 GiB for the 7 GB host | `workerpool.yaml.tmpl` | sized to the cluster; `--workers` to match | -| Plane memory store and results under `~/ousast-results/` on the host | reconciler, memory layer (harnessx-removal Req 6) | a persistent volume or bucket shared by the reconciler | +| Plane memory store and results under `~/ousast-results/` on the host | `reconciler.py` (`OUSAST_RESULTS`), `memory.py` (`OUSAST_MEMORY`) | a persistent volume or bucket shared by the reconciler; the `minio://` store already works against any reachable MinIO | The provider key already travels only in the start request through `atenet-router`, which works the same through a port-forward to any cluster.