mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
docs: correct G0 status (record 07 done on docker) + ax-lane is a G1 target
Roadmap "Where we are" table said Baseline (G0.2) = "Not yet, run on kube" -- stale.
Record 07 (G0 exit) was completed right after on the docker lane (e901640, 1300/1300,
catch 3.3% / FP 3.1%, regex/quick-rule floor, engine_covered=0). G0 is exited; the kube
`ax` lane was never needed for the baseline. Reframed the readiness plan: steps 1-4 done,
the `ax` lane moves to G1 (engine arm at scale, gated by spec tasks 4.4/5.4).
Also fixed the ax binary in the coordinator commands: use $HOME/go/bin/ax, NOT
$HOME/.local/ax-v0.3.1/ax (the latter hits the AX-lane dispatch blocker recorded in the
unseen spec tasks.md). README.md + benchmarks/unseen/README.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
9a3250f6c5
commit
6c789eac40
+34
-13
@@ -43,8 +43,11 @@ bottleneck for the measurements below.
|
||||
|
||||
### Where we are (2026-10-11)
|
||||
|
||||
The G0 measurement is now runnable: the unseen pool is **frozen** and the whole pipeline is hardened and
|
||||
**bucket-first**. The baseline replay is the one remaining step, to run on kube.
|
||||
The G0 measurement is **done**: the unseen pool is **frozen**, the pipeline is hardened and **bucket-first**,
|
||||
and the slice-1 baseline (record 07) ran on the **docker lane** — the sanctioned interim while the kube `ax`
|
||||
lane is restored. G0 is exited; the kube `ax` lane is now a **G1** concern (the engine arm at scale), not a
|
||||
baseline blocker. (Correction 2026-10-12: an earlier revision of this section said the baseline was "the one
|
||||
remaining step, to run on kube" — that was stale; record 07 was completed on docker right after, see below.)
|
||||
|
||||
| Area | State | Evidence |
|
||||
| --- | --- | --- |
|
||||
@@ -53,23 +56,41 @@ The G0 measurement is now runnable: the unseen pool is **frozen** and the whole
|
||||
| Harvest | Rebuilt to converge. Pure GitHub-API harvesting is a dead end for this benchmark (it needs the SZZ introducing commit + 12 stratified history negatives, not one commit). The fix: a full-objects clone (drop `--filter=blob:none`, which caused ~600 per-blob lazy fetches) + bucket-first durability + a get_repo cache that paid the 4,499-name used-set canonicalization once. Yield ~2.2%; **GitHub-rate-limit bound, not CPU**. | 95bed70, 857263d, 21aebe8, 7b711a6 |
|
||||
| Persistence | **Bucket-first, not local disk.** `ousast-corpus` is the source of truth: enumeration, resolution, get_repo, and accept/reject decisions all stream to and resume from it. The host disk (~9-13 GB free) is the scarce resource; RustFS/S3 has the room, so run artifacts and **replay outputs go to RustFS too**, not `~/ousast-results` on local disk. | `storage-streaming.md`, `bank.py` |
|
||||
| Replay image | Provenance pinned. CI `engine-image.yml` builds `ousast-replay-task` (harness with `/app/benchmarks/unseen/replay_entry.py`) + `ousast-engine-task` (Joern hook), publicly pullable by digest. The old `ousast-runner:dev` (248 MB ax-runner, no `/app/benchmarks`, no Joern) is the **wrong** image for replay. | CI run 38116281676 |
|
||||
| Baseline (G0.2) | **Not yet.** Replay reads the frozen pool cleanly (dry-run = 1,300 changes, `freeze_digest` verified). Remaining: mirror CI's passing `replay` job wiring, and stand up the kube lane (`ax` binary + a public-HTTPS presign-capable store endpoint; GHCR pull already public). | CI `replay` job (passed) |
|
||||
| Baseline (G0.2) | **DONE — record 07 (`e901640`), docker lane.** 1300/1300 changes; catch 3/92 = **3.3%** (Wilson95 1.1–9.2%), false-alarm **3.1%** (cluster95 1.6–5.4%), instrument_failure 54 (4.2% < 5% guard). `engine_covered=0` and no model calls → this is a **regex/quick-rule floor** (the engine never ran under the ~200s pre-push wall; the LLM tier was disabled). Baseline row in `usage-p1.jsonl` (freeze_digest `3371c67d`) = **G0 exit**. Ran on docker (not kube `ax`) per the sanctioned interim. | `benchmarks/measurements/2026-10-11-unseen-07-slice1-baseline/` |
|
||||
|
||||
Lesson, recorded so it is not re-paid: sourcing and engine-image work cost real time. The durable takeaway now lives
|
||||
in the pipeline and these docs — bucket-first persistence, pinned image provenance, a measured yield — so the next
|
||||
pool or arm is cheap, not a rediscovery.
|
||||
|
||||
### Readiness plan (to G0 exit = record 07)
|
||||
### Readiness plan (G0 exited; next is G1)
|
||||
|
||||
1. **Kube replay lane (infra, operator-owned):** install the `ax` binary; provide the store endpoint the AX lane
|
||||
needs. The only thing settled in-repo is a code fact: `benchmarks/ax/batch.py` currently expects a public-HTTPS
|
||||
DNS, presign-capable S3 endpoint. How that endpoint and the cluster's access to the store are provided is an
|
||||
operator/AX decision, not an assumption to make here. GHCR pull of the images is verified public.
|
||||
2. **Unblock the replay** by mirroring CI's passing `replay` job (replay-task + engine-task wired, store + network).
|
||||
3. **Run slice-1 baseline on kube:** `--lane ax --arm default`, 1,300 changes, no model cost, resumable; outputs to
|
||||
RustFS.
|
||||
4. **Adjudicate (7.3) -> record 07 (7.4)** = the measured machine-label unseen baseline = **G0 exit**.
|
||||
5. Then **G1 arms** (DeepSeek decision engine) on the frozen pool.
|
||||
**G0 exit = record 07 is DONE** (docker lane, regex/quick-rule floor; see the Baseline row above). Steps 1–4
|
||||
below are complete. The kube `ax` lane was NOT needed for the baseline and is now a **G1** target.
|
||||
|
||||
1. ~~Kube replay lane~~ — **not needed for G0**; the baseline ran on the docker lane (the sanctioned interim).
|
||||
The `ax` lane moves to G1 (see below).
|
||||
2. ~~Unblock the replay~~ — done; replay ran over the frozen slice.
|
||||
3. ~~Run slice-1 baseline~~ — done: `--lane docker --arm default`, 1,300 changes, no model cost, outputs to RustFS.
|
||||
4. ~~Adjudicate (7.3) → record 07 (7.4)~~ — done = **G0 exit** (`usage-p1.jsonl` freeze_digest `3371c67d`).
|
||||
5. **Next — G1 arms on the frozen pool.** Re-score the arms that the regex floor cannot be (engine evidence,
|
||||
DeepSeek decision engine, combiner) to see what beats catch 3.3% at a fixed false-alarm budget. This is where
|
||||
the kube `ax` lane is needed, because the engine arm cannot run under the ~200s pre-push wall on a single host.
|
||||
|
||||
### Restoring the kube `ax` lane (for G1, operator-owned)
|
||||
|
||||
The AX executor is **built and smoke-tested** (`benchmarks/ax/batch.py`, `engine_trace.py` `ax`/`mixed-ax`
|
||||
lanes, OOM→docker fallback; an engine smoke test dispatched to the 2-worker cluster end-to-end on 2026-10-04).
|
||||
What remains before the G1 engine-arm replay:
|
||||
|
||||
- **Use the working `ax` binary: `~/go/bin/ax`** — NOT `~/.local/ax-v0.3.1/ax`, which hits the AX-lane blocker
|
||||
(recorded in the unseen spec `tasks.md`, 2026-10-11). The coordinator commands in the READMEs are being
|
||||
corrected to this binary.
|
||||
- **Store endpoint:** `benchmarks/ax/batch.py` expects a public-HTTPS DNS, presign-capable S3 endpoint reachable
|
||||
from the cluster; the cluster carries no model/store credentials (it reads/writes via short-lived presigned
|
||||
links). This and the cluster's store access are operator/AX-owned.
|
||||
- **Run the open spec tasks** that gate G1: **4.4** (engine smoke on kube-ax → record 04) and **5.4** (replay
|
||||
proof on kind, then kube-ax → record 05), both marked `(coordinator, network)`. Then the full engine-arm
|
||||
replay (`--lane ax`).
|
||||
|
||||
### Phases
|
||||
|
||||
|
||||
@@ -439,12 +439,13 @@ uncommitted changes without publishing a git commit.
|
||||
|
||||
With `KUBECONFIG` pointing to the operator's kube-ax config and store settings in
|
||||
`.env`, run a one-pin smoke using the default experiment units and local example
|
||||
memory (or add `--examples /path/to/frozen-examples.jsonl`):
|
||||
memory (or add `--examples /path/to/frozen-examples.jsonl`). Use the `ax` binary at
|
||||
`$HOME/go/bin/ax` — `$HOME/.local/ax-v0.3.1/ax` hits the AX-lane dispatch blocker:
|
||||
|
||||
```bash
|
||||
: "${KUBECONFIG:?Set KUBECONFIG to the operator kube-ax configuration}"
|
||||
python benchmarks/learn/engine_trace.py --executor ax \
|
||||
--ax-bin "$HOME/.local/ax-v0.3.1/ax" --kubeconfig "$KUBECONFIG" \
|
||||
--ax-bin "$HOME/go/bin/ax" --kubeconfig "$KUBECONFIG" \
|
||||
--atespace default --image "$TASK_IMAGE" --docker-image openultrasast:engine-ax \
|
||||
--parallel 1 --limit 1 --deadline 900 \
|
||||
--out "$HOME/ousast-results/plane/engine-ax-smoke"
|
||||
@@ -454,7 +455,7 @@ Then run the mixed lanes (repeat to resume; defaults re-run failed/timeout pins)
|
||||
|
||||
```bash
|
||||
python benchmarks/learn/engine_trace.py --executor mixed-ax \
|
||||
--ax-bin "$HOME/.local/ax-v0.3.1/ax" --kubeconfig "$KUBECONFIG" \
|
||||
--ax-bin "$HOME/go/bin/ax" --kubeconfig "$KUBECONFIG" \
|
||||
--atespace default --image "$TASK_IMAGE" --docker-image openultrasast:engine-ax \
|
||||
--parallel 2 --deadline 900 --rerun-status failed,timeout \
|
||||
--out "$HOME/ousast-results/plane/engine-ax-mixed"
|
||||
|
||||
@@ -243,6 +243,11 @@ to the operator's kube-ax kubeconfig. Host store credentials stay on the VM; use
|
||||
HTTPS `S3_ENDPOINT=https://files.because-security.com` and the operator's `OUSAST_MEMORY`.
|
||||
Use a fresh output directory, so the smoke executes instead of resuming prior records.
|
||||
|
||||
**`ax` binary:** use `$HOME/go/bin/ax` (passed via `--ax-bin` below). `$HOME/.local/ax-v0.3.1/ax`
|
||||
hits the AX-lane dispatch blocker recorded in the spec `tasks.md` (2026-10-11). These tasks (4.4,
|
||||
5.4) gate **G1** (the engine arm at scale); the G0 baseline (record 07) already ran on the docker
|
||||
lane and did not need this lane.
|
||||
|
||||
```sh
|
||||
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
|
||||
export PYTHONPATH=$PWD/src
|
||||
@@ -250,6 +255,7 @@ export S3_ENDPOINT=https://files.because-security.com
|
||||
python benchmarks/learn/engine_trace.py \
|
||||
--units "$ENGINE_UNITS" --examples "$ENGINE_EXAMPLES" \
|
||||
--executor ax --parallel 2 --limit 2 --image "$ENGINE_IMAGE" \
|
||||
--ax-bin "$HOME/go/bin/ax" \
|
||||
--kubeconfig "$AX_KUBECONFIG" --out "$HOME/ousast-results/unseen/proof-04"
|
||||
python -m benchmarks.unseen.proof engine --image "$ENGINE_IMAGE" \
|
||||
--records "$HOME/ousast-results/unseen/proof-04" --baseline "$ENGINE_BASELINE" \
|
||||
@@ -275,7 +281,7 @@ python -m benchmarks.unseen.replay --manifest "$HOME/ousast-results/unseen/proof
|
||||
--image "$REPLAY_IMAGE" --lane kind --kubeconfig "$KIND_KUBECONFIG" \
|
||||
--out "$HOME/ousast-results/unseen/proof-05-kind"
|
||||
python -m benchmarks.unseen.replay --manifest "$HOME/ousast-results/unseen/proof-05-inputs.json" \
|
||||
--image "$REPLAY_IMAGE" --lane ax --kubeconfig "$AX_KUBECONFIG" \
|
||||
--image "$REPLAY_IMAGE" --lane ax --ax-bin "$HOME/go/bin/ax" --kubeconfig "$AX_KUBECONFIG" \
|
||||
--out "$HOME/ousast-results/unseen/proof-05-ax"
|
||||
python -m benchmarks.unseen.replay --manifest "$HOME/ousast-results/unseen/proof-05-inputs.json" \
|
||||
--image "$REPLAY_IMAGE" --lane docker \
|
||||
|
||||
Reference in New Issue
Block a user