mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
steering: roadmap re-centred on generalisation to unseen repositories (G0-G5); specs updated; unseen-repo-evaluation brief and requirements (approved)
Maintainer 2026-10-04: the safety net is for unseen code and the A/B tests exist to prove the approach generalises; infrastructure work had drifted from that. Runs are designed for AX on kube-ax, kind is the fallback. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
ee422b63e0
commit
d4467f4236
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "learned-decision-engine",
|
||||
"created_at": "2026-09-30T14:00:00Z",
|
||||
"updated_at": "2026-10-01T10:20:00Z",
|
||||
"updated_at": "2026-10-04T18:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
|
||||
@@ -8,6 +8,10 @@ memory, compiled DSPy-style in-house); tasks 1-5 are unaffected, tasks 6-10 are
|
||||
against the provider balance read just before it, and none starts with a ceiling above that balance. No test makes
|
||||
a model or embedding call.
|
||||
|
||||
Direction 2026-10-04: `.kiro/steering/roadmap.md` (section of 2026-10-04, phases G0-G5) is the authority. Every
|
||||
change is judged on the unseen-repository pool (G0, spec `unseen-repo-evaluation`) by catch rate and false-alarm
|
||||
rate per push. Within-pair AUC stays a development signal. Tasks 11-14 below follow that roadmap.
|
||||
|
||||
- [x] 1. Schema and features
|
||||
- `learn/schema.py` (closed, typed allow-list; no identities; missing != zero), `learn/features.py` (host
|
||||
builder), new memory `KINDS`, `plane/tasks/features.py` and the `remember` input; tests enforcing the
|
||||
@@ -45,6 +49,11 @@ a model or embedding call.
|
||||
- _Requirements: 1.1_
|
||||
|
||||
- [ ] 6. Program, local memory and compilation (design sections 4-5)
|
||||
- Status 2026-10-04: 6.1-6.6 are done. 6.7 is partial: leave-one-source-out, the 0/25/50% memory-size points and
|
||||
the source-count curve are not run. Around this task the experiment machinery grew: rule arms
|
||||
(`precision_bound_block`, `pooled_block_gate` with a family floor) in `learn/experiments.py`, the response-cache
|
||||
fill with a hard ceiling and a dry-run estimate in `learn/fill.py`, and the in-function slice in `learn/slice.py`.
|
||||
These ran exp-003, exp-004 and exp-005 (7.5-7.7). Memory now holds 710 complete fix pairs (7.9).
|
||||
- [x] 6.1 Excerpts and the example store (no model)
|
||||
- `learn/excerpt.py` (function span at the label's pin, bounds 80/4,000 for candidates and 40/2,000 for
|
||||
examples, numbered lines, no path/repository/commit, advisory ids and security wording in comments
|
||||
@@ -151,6 +160,28 @@ a model or embedding call.
|
||||
rows' null plane-only features (1f525ff), the 788 embedding vectors absent from RustFS (copied from the file
|
||||
store), the result row's column-type collision under S3 Select (780caaa). Further program variants (`k = 5` vs
|
||||
`k = 1`, the confidence source) as new manifests.
|
||||
- [x] 7.5 exp-003 run and recorded 2026-10-03 ($0, replay only): **inconclusive**. Selective blocking (a per-family
|
||||
raw-score threshold chosen on training folds by its Wilson lower bound) offers no BLOCK on held-out paired units
|
||||
in any family; the incumbent stays. 211 of 353 frozen units evaluated (142 replay misses). Proving 0.95 needs at
|
||||
least 73 clean flags. Result: `benchmarks/experiments/exp-003-precision-bound-blocking/result.json`.
|
||||
- [x] 7.6 exp-004 run and recorded 2026-10-03: **inconclusive**. One pooled BLOCK threshold over all families with a
|
||||
family floor (Req 6.4). No outer fold found a threshold, pooled or per family; B offers no BLOCK. 501 units,
|
||||
replay at $0 after a cache fill of $1.24 (ceiling $2; the dry-run estimate over-priced 4.5x). Result and record:
|
||||
`benchmarks/experiments/exp-004-pooled-block-gate/result.json`, `record.json`.
|
||||
- [x] 7.7 exp-005 run and recorded 2026-10-03: **inconclusive**. The in-function def-use slice in the prompt of the
|
||||
six slice-2 programs. Within-pair AUC 0.762 -> 0.746 (CI [-0.063, 0.032], spans 0); unsure rate 0.29 -> 0.24
|
||||
(favours B); usd per candidate slightly higher. A $0 leak audit came first
|
||||
(`benchmarks/measurements/2026-10-03-graph-slice-leak-audit/record.json`). Result:
|
||||
`benchmarks/experiments/exp-005-graph-slice/result.json`.
|
||||
- [x] 7.8 Negatives harvest 2026-10-03 ($0.0014 embeddings, no chat-model call): 98 real fix pairs whose function
|
||||
survives the fix, from reviewed advisories, as pointer-pair corpus `advisory-fixes-2`; memory 937 -> 1133
|
||||
examples. Record: `benchmarks/measurements/2026-10-03-decision-engine-negatives-harvest/record.json`.
|
||||
- [x] 7.9 Harvest 3, 2026-10-04 ($0.003 embeddings, no chat-model call): 251 pairs as `advisory-fixes-3`; complete
|
||||
pairs in memory 459 -> 710. Record: `benchmarks/measurements/2026-10-04-decision-engine-harvest-3/record.json`.
|
||||
- [x] 7.10 Joern trace run over exp-005's 501 units and its leak audit, 2026-10-04 ($0): no leak flagged. In the 105
|
||||
pairs where both sides were asked, a path appears on both sides in 12, on the vulnerable side only in 2, on the
|
||||
fixed side only in 0, and on neither in 91. The engine cannot see what a fix changed. Record:
|
||||
`benchmarks/measurements/2026-10-04-joern-slice-leak-audit/record.json`.
|
||||
- _Requirements: 3.6, 4.1-4.4_
|
||||
|
||||
- [ ] 8. Integration
|
||||
@@ -176,3 +207,33 @@ a model or embedding call.
|
||||
`result-v3-engine.json` is committed. Nothing between the two commits may touch the program, memory snapshot,
|
||||
features, priors or thresholds. This spec's code never opens v3 files.
|
||||
- _Requirements: 5.3_
|
||||
|
||||
- [ ] 11. Fix mechanisms (plan approved 2026-10-04; see `fix-mechanisms-plan.md`)
|
||||
- Roadmap G2 arm (`.kiro/steering/roadmap.md`, section of 2026-10-04): one candidate improvement, not the spine.
|
||||
It is judged on the unseen-repository pool (G0, spec `unseen-repo-evaluation`) by catch rate and false-alarm
|
||||
rate per push. **Paused after 11.1 until the G0 pool exists.**
|
||||
- [ ] 11.1 Mine candidate discharge facts from the 710 fix diffs (sanitizer calls, guarding conditions, safe sink
|
||||
arguments, safe replacements); deterministic, $0. In progress.
|
||||
- [ ] 11.2 Admission by held-out separation per outer fold: a candidate turns "path on both sides" into "path on the
|
||||
vulnerable side only" on folds it was not mined from, removes no held-out vulnerable path, has at least 2
|
||||
proposing repositories, and is admitted per language.
|
||||
- [ ] 11.3 Engine re-run on the affected pins (kube-ax lanes plus the VM lane).
|
||||
- [ ] 11.4 Registered experiment, with the detection gates green, before `ruleset/semantic/*.toml` changes.
|
||||
- [ ] 11.5 Coverage increment: source and sink additions for pairs where neither side has a path (later).
|
||||
- [ ] 11.6 TypeScript admission in the engine (branch `engine-typescript`).
|
||||
- _Requirements: 4.1-4.4, 7.1, 8.1_
|
||||
|
||||
- [ ] 12. exp-006: Joern witness slice in the prompt **(budget go-ahead, about $2-4)**
|
||||
- Planned, roadmap G1/G2. Against the current prompt, after a $0 leak audit, on re-frozen units. Judged on the
|
||||
unseen-repository pool, not on within-pair AUC alone.
|
||||
- _Requirements: 4.1-4.4, 7.1_
|
||||
|
||||
- [ ] 13. exp-007: the combiner ($0 replay)
|
||||
- Planned, roadmap G1/G2. Three or four weights (score, engine entailment, neighbour agreement) over cached
|
||||
responses. Judged on the unseen-repository pool.
|
||||
- _Requirements: 4.1-4.4, 7.1_
|
||||
|
||||
- [ ] 14. Re-run the pooled block gate on re-frozen units
|
||||
- Planned, roadmap G3. The exp-004 design with the family floor (Req 6.4), together with the pool's false-alarm
|
||||
rate per family.
|
||||
- _Requirements: 6.4, 7.3_
|
||||
|
||||
@@ -1,5 +1,20 @@
|
||||
# Design Document
|
||||
|
||||
> **Status 2026-10-04.** Groups 1-2 are done on kind (records `benchmarks/measurements/2026-10-02-k8s-01-profile/`
|
||||
> and `2026-10-02-k8s-02-store-delivery/`). Task 2.6 is paused. The production target changed. The operator runs a
|
||||
> kube-ax cluster (k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes). There we may only submit AX Tasks: no k8s
|
||||
> object creation, no secrets, no access to `ate-system`. The engine now runs there as AX Tasks through
|
||||
> `benchmarks/learn/engine_trace_ax.py`. The engine-task image (`plane/Dockerfile.engine-task`) is built FROM
|
||||
> `ghcr.io/norandom/ax-task-runner:v0.3.1`. Tasks read and write through presigned HTTPS links, are named
|
||||
> `ousast-engine-*`, and run a 1.7 GB heap with `ActiveProcessorCount=2` and one resident Joern session per pin. A
|
||||
> pin whose sandbox dies is retried once under a new name, then handed to the VM lane (2.5 GB heap). Egress is
|
||||
> operator-managed: HTTPS 443 to public hosts is open; LAN and cluster-internal addresses are blocked. Archive
|
||||
> names (`.tar`, `.gz`, `.zip`, `.bin`) failed through the store host's Cloudflare cache until the operator added a
|
||||
> cache-bypass rule; presigned objects are staged as `.dat` as a safeguard. Groups 3-8 are **to re-plan** against
|
||||
> that cluster under roadmap G4 (`.kiro/steering/roadmap.md`, section of 2026-10-04). Their text below is kept
|
||||
> as written. G4 (detection on push for any repository: `ousast plane scan --base --head`, engine pins on kube-ax,
|
||||
> model calls on the VM or CI) is the new target for group 7, sized by what G1-G3 keep.
|
||||
|
||||
## Overview
|
||||
|
||||
The plane keeps its shape: `ousast plane run` is the reconciler, ax is the only executor, every task is one
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "plane-on-kubernetes",
|
||||
"created_at": "2026-10-02T20:00:00Z",
|
||||
"updated_at": "2026-10-02T22:00:00Z",
|
||||
"updated_at": "2026-10-04T18:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
|
||||
@@ -1,5 +1,20 @@
|
||||
# Implementation Plan
|
||||
|
||||
> **Status 2026-10-04.** Groups 1-2 are done on kind (records `benchmarks/measurements/2026-10-02-k8s-01-profile/`
|
||||
> and `2026-10-02-k8s-02-store-delivery/`). Task 2.6 is paused. The production target changed. The operator runs a
|
||||
> kube-ax cluster (k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes). There we may only submit AX Tasks: no k8s
|
||||
> object creation, no secrets, no access to `ate-system`. The engine now runs there as AX Tasks through
|
||||
> `benchmarks/learn/engine_trace_ax.py`. The engine-task image (`plane/Dockerfile.engine-task`) is built FROM
|
||||
> `ghcr.io/norandom/ax-task-runner:v0.3.1`. Tasks read and write through presigned HTTPS links, are named
|
||||
> `ousast-engine-*`, and run a 1.7 GB heap with `ActiveProcessorCount=2` and one resident Joern session per pin. A
|
||||
> pin whose sandbox dies is retried once under a new name, then handed to the VM lane (2.5 GB heap). Egress is
|
||||
> operator-managed: HTTPS 443 to public hosts is open; LAN and cluster-internal addresses are blocked. Archive
|
||||
> names (`.tar`, `.gz`, `.zip`, `.bin`) failed through the store host's Cloudflare cache until the operator added a
|
||||
> cache-bypass rule; presigned objects are staged as `.dat` as a safeguard. Groups 3-8 are **to re-plan** against
|
||||
> that cluster under roadmap G4 (`.kiro/steering/roadmap.md`, section of 2026-10-04). Their text below is kept
|
||||
> as written. G4 (detection on push for any repository: `ousast plane scan --base --head`, engine pins on kube-ax,
|
||||
> model calls on the VM or CI) is the new target for group 7, sized by what G1-G3 keep.
|
||||
|
||||
One task group per increment of `design.md` section 7, in that order; a later group never starts before the
|
||||
earlier group's kind proof is recorded. Every task ships with its tests; a task is done when its tests pass in
|
||||
the host suite, ruff and mypy are clean, and nothing in `src/`, `plane/`, `ops/` (minus `ops/ax/up.sh` and
|
||||
@@ -9,7 +24,7 @@ its kind proof and the record it writes under `benchmarks/measurements/<date>-k8
|
||||
(commit, images, the instrument checks, the numbers). Steps that call a model are marked **paid: needs a budget
|
||||
go-ahead** with a ceiling; steps that need a hand on a cluster that does not exist yet are marked **manual**.
|
||||
|
||||
- [ ] 1. Profile and literals (design section 1; open-question checks 2, 3, 7)
|
||||
- [x] 1. Profile and literals (design section 1; open-question checks 2, 3, 7)
|
||||
- [x] 1.1 `PlaneProfile`
|
||||
- `src/openultrasast/plane/profile.py`: frozen dataclass with the fields of design section 1; `load_profile()`
|
||||
reads `ops/k8s/profiles/<OUSAST_PLANE_PROFILE>.toml` (a name or a path; default `kind`), then applies
|
||||
@@ -107,7 +122,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
prints the table. Instrument: the tar's byte size from the store equals the extracted files' total.
|
||||
- `benchmarks/measurements/<date>-k8s-02-store-delivery/record.json`: bytes, PUT-to-poll seconds, rules.
|
||||
- _Requirements: 3.1, 3.2, 7.1_
|
||||
- [ ] 2.6 Remove the receiver (last; after 2.5 is recorded)
|
||||
- [ ] 2.6 Remove the receiver (last; after 2.5 is recorded) -- paused 2026-10-04
|
||||
- Delete `Receiver`, `_Handler`, `receiver_address`, `receiver_cluster_ip`, `RECEIVER_HOST`, `RECEIVER_PORT`
|
||||
from `egress.py`; `OUSAST_ARTIFACT_URL/DIAL/HOST/PORT` from `runner.py` and `reconciler.py`; the
|
||||
`OUSAST_DELIVERY` switch; `ops/ax/receiver-service.yaml.tmpl` and the `up.sh` receiver step; the EndpointSlice
|
||||
@@ -117,7 +132,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
`receiver_removed: <commit>`.
|
||||
- _Requirements: 3.1, 1.3_
|
||||
|
||||
- [ ] 3. `--exec remote` from a CI-shaped principal (design section 2; open-question check 1)
|
||||
- [ ] 3. `--exec remote` from a CI-shaped principal (design section 2; open-question check 1) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 3.1 Execution mode and RBAC manifests
|
||||
- `cli.py`: `ousast plane run --exec local|remote` (default `profile.exec`); `reconciler.Ax` passes the
|
||||
profile's context to `ax` (its context flag, read from `ax --help` and pinned in a test); `remote` refuses a
|
||||
@@ -150,7 +165,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
status, usd spent.
|
||||
- _Requirements: 2.1, 2.2, 2.3, 2.4, 7.1_
|
||||
|
||||
- [ ] 4. Images on GHCR (design section 8)
|
||||
- [ ] 4. Images on GHCR (design section 8) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 4.1 `images.yml`
|
||||
- `.github/workflows/images.yml` on tags `v*`: `docker/build-push-action` for `plane/Dockerfile.runner` ->
|
||||
`ghcr.io/norandom/ousast-runner:<tag>` and the root `Dockerfile` -> `ghcr.io/norandom/ousast-engine:<tag>`;
|
||||
@@ -178,7 +193,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
- `benchmarks/measurements/<date>-k8s-04-ghcr-images/record.json`: image sizes, pull seconds, visibility.
|
||||
- _Requirements: 1.1, 4.2, 7.1_
|
||||
|
||||
- [ ] 5. The engine as a plane task (design section 4)
|
||||
- [ ] 5. The engine as a plane task (design section 4) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 5.1 `cpg/dump.py` and `tasks/engine.py`
|
||||
- Lift `benchmarks/push/finding_dump.py`'s scan into `src/openultrasast/cpg/dump.py` (the script becomes a
|
||||
thin caller; `ops/README.md:78` command unchanged). `src/openultrasast/plane/tasks/engine.py`: env
|
||||
@@ -208,7 +223,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
placement, image size and pull seconds.
|
||||
- _Requirements: 4.1, 4.2, 7.1_
|
||||
|
||||
- [ ] 6. Pools, autoscaling, node checklist (design section 5)
|
||||
- [ ] 6. Pools, autoscaling, node checklist (design section 5) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 6.1 `plane/k8s.py` and the manifests
|
||||
- `src/openultrasast/plane/k8s.py`: `render(profile, out)` writes `workerpool-default.yaml`,
|
||||
`workerpool-engine.yaml`, `ax-server-snapshots.yaml` (patch with `profile.snapshots_bucket`), `rbac-ci.yaml`;
|
||||
@@ -232,7 +247,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
- `benchmarks/measurements/<date>-k8s-06-pools/record.json`: footprint table, securityContext, dry-run result.
|
||||
- _Requirements: 5.1, 5.2, 7.1_
|
||||
|
||||
- [ ] 7. `ousast plane scan` (design section 6)
|
||||
- [ ] 7. `ousast plane scan` (design section 6) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 7.1 Workspaces and candidates from URL + commits
|
||||
- `generate.scan_cases(url, base, head, cache, *, all=False)`: shallow clone into the case cache, two pinned
|
||||
Workspaces, head-side hunks via `fix_ranges(side="new")`, functions via `repo_facts.enclosing` over
|
||||
@@ -265,7 +280,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
seconds from submit to verdict, whether the known CVE site is among the agreed.
|
||||
- _Requirements: 6.1, 6.2, 6.3, 7.1_
|
||||
|
||||
- [ ] 8. Runbook (design section 7, Req 7)
|
||||
- [ ] 8. Runbook (design section 7, Req 7) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 8.1 `docs/deployment.md` as the runbook
|
||||
- Sections: prerequisites for k3s (Substrate, ax with the snapshots patch, GHCR and the pull secret, buckets,
|
||||
the CI SA and its grant from 3.2, the node checklist of 6.2), the profile switch, `--exec remote` from a laptop
|
||||
|
||||
@@ -623,6 +623,7 @@ Preserve all completed task evidence below.
|
||||
- [x] 17. Operational readiness (2026-10-02)
|
||||
- **Named by the 2026-10-02 operational-readiness audit**, maintainer-approved as the first increment. The maintainer's definition: a user installs the hook on ANY repository and gets honest, advisory-only results; nothing ever blocks a push. The audit found the plumbing sound (snapshots, resolution, deadline accounting, advisory exit 0, the JSON artifact) and five things in the way: the hook can block a push, quick rules are not part of pre-push, the engine cannot finish a 26-file repository inside 30 s, the terminal hides every skip reason, and there is no supported install path without the source checkout. Before numbers for every item are the audit's live runs on records (Python), serve-static (JavaScript) and unrolled/secure (Go), each with one planted commit.
|
||||
- This group is plumbing and honesty. It enables no capability, edits no rule for a planted sample, and leaves graph reuse (the engine speed work decided in 11.6) as a later task.
|
||||
- Record: `benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json` (see 17.7).
|
||||
- _Done 2026-10-02:_ all seven subtasks done. The engine still does not fit the 30 s hook deadline (graph reuse, the 11.6 direction, stays a later task), and plane verdicts after the push were not in this increment.
|
||||
- _Requirements: 9.1–9.6_
|
||||
- [x] 17.1 Never block a push the user did not choose to block
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# Brief: does the safety net work on repositories it has never seen?
|
||||
|
||||
Maintainer, 2026-10-04: "we want to spread the safety net across repos in general, unseen code. that was the purpose
|
||||
of the AB tests to see if our approach is generalizeable." And: "we should run our tests / runs in kube. now we can
|
||||
speed it up and design the workloads for ax directly. kind is only a fallback."
|
||||
|
||||
## Why this spec exists
|
||||
|
||||
The A/B experiments so far (exp-003, exp-004, exp-005) score the decision engine on repository-held-out folds of known
|
||||
fix pairs. That measures whether the vulnerable side of a pair scores above its fixed side. It does not measure what a
|
||||
safety net on an arbitrary repository lives on:
|
||||
|
||||
- on an ordinary push, does it stay quiet? (false-alarm rate)
|
||||
- on a push that introduces a real vulnerability, does it say so? (catch rate)
|
||||
|
||||
Populations v1 and v2 measured the engine scan on fresh cases once (1 of 11, 0 of 17) and are spent. Nothing since has
|
||||
measured push-level behaviour on unseen repositories.
|
||||
|
||||
## What this spec delivers
|
||||
|
||||
1. A frozen pool of repositories never used anywhere in development, with two kinds of change each: real
|
||||
vulnerability-introducing commits and ordinary commits.
|
||||
2. A push-level metric: each change replayed through the hook exactly as a user would run it, and through other
|
||||
arms (decision engine, engine evidence) where an experiment asks.
|
||||
3. Execution designed for the kube-ax cluster from the start: one push replay is one AX Task; kind and the VM are
|
||||
fallbacks.
|
||||
4. A baseline for today's hook, then a table of arms scored on the same pool.
|
||||
|
||||
## Not in scope
|
||||
|
||||
Changing detection. This spec only builds the measurement. Improvements are separate, registered experiments judged
|
||||
by it.
|
||||
@@ -0,0 +1,103 @@
|
||||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
The safety net is meant for repositories it has never seen. This spec builds the instrument that measures that: a
|
||||
frozen pool of unseen repositories, a push-level metric (catch rate and false-alarm rate as the hook reports them),
|
||||
and an execution path that runs the replays as AX Tasks on the kube-ax cluster. Context: `brief.md`, the roadmap's
|
||||
G0-G1 (`.kiro/steering/roadmap.md`).
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: pool selection and freezing; change extraction (vulnerability-introducing and ordinary commits);
|
||||
replay of each change through `ousast pre-push` and through registered experiment arms; the metric and its
|
||||
confidence intervals; AX Task execution with kind/VM fallback; the baseline for today's hook; rotation of pool
|
||||
slices.
|
||||
- **Out of scope**: any detection change; population v3 (stays reserved and unread); model calls inside the cluster.
|
||||
- **Adjacent expectations**: `learned-decision-engine` experiments use this pool as their primary unseen-repo score;
|
||||
`pre-push-safety-net` owns the hook being replayed; `plane-on-kubernetes` and the AX executor
|
||||
(`benchmarks/learn/engine_trace_ax.py`) supply the execution pattern; the reserved-repository guard covers every
|
||||
population and this pool.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: A pool no development step has touched
|
||||
|
||||
**User Story:** As the maintainer, I want the evaluation repositories to be ones nothing in development has seen, so a
|
||||
good number means the approach generalises.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. A repository is eligible only if it appears in no pair corpus, no population (v1, v2, v3), no harvest record, no
|
||||
memory example, no experiment unit and no measurement record. Eligibility is checked by code against every source
|
||||
before the pool is frozen; v3 is checked through the existing guard without being opened.
|
||||
2. The pool is frozen (a committed manifest with repository URLs, commit SHAs, licences and the freeze digest) before
|
||||
any change in it is replayed. A frozen pool is never edited; a correction is a new pool.
|
||||
3. Pool repositories are referenced only inside the pool's own directory (like `benchmarks/independent/`), and the
|
||||
reserved-repository guard is extended to them.
|
||||
4. Only permissively licensed repositories are published by name; others may be used with their names kept out of
|
||||
the repository (pointers in a local, gitignored manifest), recorded as such.
|
||||
|
||||
### Requirement 2: Two kinds of change per repository
|
||||
|
||||
**User Story:** As the maintainer, I want both vulnerable and ordinary pushes from the same repositories, so the
|
||||
false-alarm rate is measured on the code the catch rate is measured on.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Vulnerability-introducing changes: from public advisories with a linked fix, the commit range that introduced the
|
||||
vulnerable code where it can be determined, else the fix's parent compared with an earlier base; each labelled
|
||||
with family and the vulnerable function. Post-model-cutoff advisories are preferred and flagged.
|
||||
2. Ordinary changes: commits from the same repositories' histories with no security fix, revert or advisory link,
|
||||
sampled across time and size; each recorded with its changed files and lines.
|
||||
3. Per repository, at least one vulnerability-introducing change and several ordinary changes; the pool's totals are
|
||||
sized so the false-alarm rate's 95% interval is no wider than +/- 3 points and the catch rate's no wider than
|
||||
+/- 10 points (sizes computed in the design).
|
||||
|
||||
### Requirement 3: The push-level metric
|
||||
|
||||
**User Story:** As the maintainer, I want the score to be what a user would see, so the number means something.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Each change is replayed as `ousast pre-push --base <base> --head <head>` with the hook's default settings, in a
|
||||
clean checkout, with the record of every skipped check kept.
|
||||
2. Catch: a vulnerability-introducing change counts as caught when the hook (or the arm) reports a finding of the
|
||||
right family on the vulnerable function or its changed lines. False alarm: an ordinary change counts as a false
|
||||
alarm when the hook reports any advisory finding.
|
||||
3. Reported per family and overall: catch rate, false-alarm rate, and both with 95% Wilson intervals; plus coverage
|
||||
(changes where the hook could analyse the changed language) so a silent zero is visible.
|
||||
4. Arms (decision engine, combiner, engine evidence, fix mechanisms) are scored on the same replays: catch rate at
|
||||
fixed false-alarm budgets (1%, 5%, 10%), so arms are compared at equal noise.
|
||||
|
||||
### Requirement 4: Execution on kube-ax, designed for AX
|
||||
|
||||
**User Story:** As the maintainer, I want the replays to run on the cluster, so measurements are fast and the VM is
|
||||
free.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. One replayed change is one AX Task, named with the `ousast-engine-` prefix the operator's egress watcher matches,
|
||||
using an image built `FROM ghcr.io/norandom/ax-task-runner:v0.3.1` with this package, Joern and php-cli, pinned by
|
||||
digest.
|
||||
2. The task fetches the repository at the needed commits itself over HTTPS (public hosts are reachable from actors)
|
||||
and writes its record through a presigned link; no API keys or store credentials enter the cluster.
|
||||
3. A generic AX batch dispatcher serves both this replay and the existing engine passes: items to Tasks, at most the
|
||||
cluster's safe concurrency (two heavy tasks per worker), retry once on sandbox death, then a VM or kind fallback
|
||||
lane; progress and resume as in the engine runner.
|
||||
4. Arms that need model calls take the cluster's records and make their calls on the VM, within registered budgets.
|
||||
5. kind runs the same tasks unchanged as the local fallback.
|
||||
|
||||
### Requirement 5: Baseline, rotation and use
|
||||
|
||||
**User Story:** As the maintainer, I want a baseline now and a fresh slice for each decision, so the pool is not
|
||||
worn out by tuning.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The first use is a baseline of today's hook on the first slice; its record is committed (counts and digests only,
|
||||
no repository names outside the pool directory).
|
||||
2. A slice that informed a change can no longer qualify that change; each adoption decision draws a new, untouched
|
||||
slice, recorded with its freeze digest.
|
||||
3. Every learned-decision-engine experiment from now on reports its unseen-repo score from this pool next to its
|
||||
held-out-fold score.
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"feature_name": "unseen-repo-evaluation",
|
||||
"created_at": "2026-10-04T18:00:00Z",
|
||||
"updated_at": "2026-10-04T18:30:00Z",
|
||||
"language": "en",
|
||||
"phase": "requirements-approved",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false
|
||||
}
|
||||
@@ -1,8 +1,23 @@
|
||||
# Overview — what this is, and what is actually true of it
|
||||
|
||||
## Current state, 2026-10-02
|
||||
## Current state, 2026-10-04
|
||||
|
||||
Everything below this section is dated history; where it disagrees with this section, this section wins. The
|
||||
direction lives in `roadmap.md`, section of 2026-10-04 (phases G0-G5).
|
||||
|
||||
- **Goal.** A pre-push safety net that works on repositories it has never seen. Two numbers per push decide it:
|
||||
catch rate on real vulnerability-introducing changes, and false-alarm rate on ordinary changes. Neither has been
|
||||
measured yet; the unseen-repository pool (G0, spec `unseen-repo-evaluation`) comes first.
|
||||
- **Correction, 2026-10-04.** Infrastructure and engine work had drifted from that goal. It is now done only when
|
||||
it removes a bottleneck for those measurements.
|
||||
- **Engine.** Joern runs on the operator's kube-ax cluster as AX Tasks, with a VM fallback lane; kind stays the
|
||||
development profile. It is almost blind to fixes: paths differ between sides in 2 of 105 both-asked pairs.
|
||||
- **Decision engine.** Ranks but cannot block (exp-003 to exp-005 inconclusive; 710 fix pairs in memory). Fix
|
||||
mechanisms is one G2 arm, paused after mining until the pool exists.
|
||||
- **Hook.** `ousast pre-push` is operational and advisory on any repository; rollout stays NO-GO.
|
||||
|
||||
## State, 2026-10-02 (superseded by the section above)
|
||||
|
||||
Everything below this section is dated history; where it disagrees with this section, this section wins.
|
||||
User-facing documentation: `README.md`, `docs/architecture.md`, `docs/evaluation.md`, `docs/decision-engine.md`,
|
||||
`ops/ax/README.md`.
|
||||
|
||||
|
||||
@@ -1,5 +1,100 @@
|
||||
# Roadmap
|
||||
|
||||
## Roadmap to detection with the decision engine on kube-ax (2026-10-04)
|
||||
|
||||
This section supersedes the ordering below it. The older sections stay as history.
|
||||
|
||||
### Destination
|
||||
|
||||
On every push, from a hook or CI, OpenUltraSAST answers for the changed code. The answer is advisory for every
|
||||
family, and BLOCK only for families whose precision has been proven on repositories the system never saw. The work
|
||||
is split by what each place is good at:
|
||||
|
||||
- **kube-ax cluster** (operator k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes): the Joern engine, as AX Tasks.
|
||||
No API keys and no store credentials in the cluster; tasks read and write through short-lived presigned links.
|
||||
- **The VM or CI runner**: candidate selection, the decision engine's model calls (DeepSeek), the memory store,
|
||||
and the fallback engine lane (2.5 GB heap) for pins that do not fit the cluster's 1.7 GB heap.
|
||||
- **kind** stays the local development profile, with the same code and manifests.
|
||||
|
||||
### The question everything answers
|
||||
|
||||
Does the safety net work on repositories it has never seen? That is the purpose of the A/B experiments, and every
|
||||
phase below is judged by it. On an unseen repository almost every push is an ordinary change, so two numbers decide
|
||||
whether the safety net is useful, measured on the hook's advisory output exactly as a user would see it:
|
||||
|
||||
1. **Catch rate:** on real vulnerability-introducing changes in unseen repositories, how often it flags the change.
|
||||
2. **False-alarm rate:** on ordinary changes in the same unseen repositories, how often it flags anything.
|
||||
|
||||
Within-pair AUC on known fix pairs answers neither. It stays a development signal, not the goal.
|
||||
|
||||
### Where we are (2026-10-04)
|
||||
|
||||
| Area | State | Evidence |
|
||||
| --- | --- | --- |
|
||||
| Generalisation measured so far | The classifier ranks above chance on repository-held-out folds (within-pair AUC about 0.76) but cannot reach the block bar; the engine-based scan failed fresh populations (v1 1/11, v2 0/17); push-level catch and false-alarm rates on unseen repositories have never been measured | exp-003/004/005, `results-v1.json`, `results-v2.json` |
|
||||
| Pre-push hook | Operational and advisory on any repository | `2026-10-02-pre-push-operational-1/record.json` |
|
||||
| Labelled data | 710 complete real fix pairs | `2026-10-04-decision-engine-harvest-3/record.json` |
|
||||
| Engine | Almost blind to fixes (paths differ between sides in 2 of 105 both-asked pairs) | `2026-10-04-joern-slice-leak-audit/record.json` |
|
||||
| Infrastructure | Engine runs on kube-ax as AX Tasks with a VM fallback; kind for development | memory `kube-ax-cluster` |
|
||||
|
||||
Correction recorded 2026-10-04: the previous two days went mostly into infrastructure and engine internals that did
|
||||
not move a generalisation number. From here, infrastructure and engine work is done only when it removes a
|
||||
bottleneck for the measurements below.
|
||||
|
||||
### Phases
|
||||
|
||||
**G0. The unseen-repository pool and the push-level metric** (new spec `unseen-repo-evaluation`)
|
||||
|
||||
1. A frozen pool of repositories never used for training, tuning, mining or harvesting, and not in any population.
|
||||
Two kinds of change per repository: real vulnerability-introducing commits (the parent of a fix, post-cutoff where
|
||||
possible) and ordinary commits from the same history.
|
||||
2. The metric: replay each change through `ousast pre-push` (and the decision engine where an arm uses it) and
|
||||
record catch rate and false-alarm rate, with confidence intervals, per family and overall.
|
||||
3. Rotation: the pool is spent per decision, like the populations; a new slice is drawn for the next one.
|
||||
|
||||
Exit: the pool is frozen and the current production hook has a measured baseline on it.
|
||||
|
||||
**G1. Re-score the arms we have on the pool**
|
||||
|
||||
Current hook (quick rules + engine), decision-engine advisory (current programs), combiner, engine evidence. Each is
|
||||
an arm with a registered decision rule, scored by catch rate at a fixed false-alarm budget.
|
||||
|
||||
Exit: one measured table of arms on unseen code; keep only what generalises.
|
||||
|
||||
**G2. Improve what moves the unseen-repo numbers**
|
||||
|
||||
Candidates, each admitted only if it improves the pool metric on a fresh slice: fix mechanisms in the engine
|
||||
(approved plan, now one arm, not the spine), engine coverage and TypeScript, decision-engine prompt and retrieval
|
||||
changes, more labelled pairs from more distinct repositories.
|
||||
|
||||
**G3. Decide BLOCK per family** on the pool's false-alarm rate and the pooled precision bound (Req 6.4).
|
||||
|
||||
**G4. Detection on push for any repository** (`plane-on-kubernetes` P4 work: `ousast plane scan --base --head`,
|
||||
engine pins on kube-ax, model calls on the VM or CI), sized by what G1-G3 kept.
|
||||
|
||||
**G5. Qualification and rollout:** adopt, commit `prediction-v3.json`, spend population v3 once, advisory rollout,
|
||||
BLOCK only for families that passed.
|
||||
|
||||
### Cross-cutting rules
|
||||
|
||||
- No hand-tuning per miss. Every detection change is a registered experiment or an admission by held-out
|
||||
separation.
|
||||
- Records carry counts, digests and pattern names, never repository names: the reserved-repository guard covers
|
||||
every population.
|
||||
- Instrument first: a zero from an unread input is a failed run (AGENTS.md).
|
||||
- Cluster stability: sandbox deaths are retried once, then sent to the VM lane; the operator owns workers, egress
|
||||
and Substrate.
|
||||
|
||||
### Spec map
|
||||
|
||||
| Spec | Role in this roadmap | State |
|
||||
| --- | --- | --- |
|
||||
| `unseen-repo-evaluation` (new) | G0, G1: the pool and the push-level metric | To specify |
|
||||
| `learned-decision-engine` | G1-G3, G5; fix-mechanisms plan is one G2 arm | Implementation; experiments 3-5 recorded |
|
||||
| `plane-on-kubernetes` | G4; the engine executors for kube-ax | Groups 1-2 done (kind); production path re-scoped to the operator's kube-ax (AX Tasks), groups 3-8 to re-plan |
|
||||
| `pre-push-safety-net` | the hook replayed in G0-G1; M4-M5 in G5 | Operational increment 1 done; M4-M5 open |
|
||||
| `ai-service-plane` | The plane the above runs on | Complete |
|
||||
|
||||
## Status, 2026-10-02
|
||||
|
||||
The milestone table below dates from 2026-09-14. Since then (`.kiro/specs/pre-push-safety-net/release-milestones.md`):
|
||||
|
||||
Reference in New Issue
Block a user