Author SHA1 Message Date
Marius CiepluchandClaude Opus 4.8 f36165a4ed wip(g2): park change-scoped attempt + full diagnosis (NOT the fix)
Scope+bound (change-first ordering, adaptive budget-fitted portioning,
_context_pass scoped to changed files, evidence_pass_bounded degradation) is
preserved here but does NOT unblock analysis: query_seconds stayed 0 across every
live re-verify. Uncontended measurement proved why -- the evidence pass is ~150s
fixed (first-query graph-load + census) + ~93ms/pair, so ~200s fixed overhead
(with the 48s build) precedes any analysis and a 300s budget leaves ~100s.
Pair-count scoping only trims the ~86s per-pair tail. The warm-session env
(OUSAST_ENGINE_SESSION=1) times out at startup under contention and, even working,
spares only subsequent batches -- not the first evidence query's load.

Parked per decision to finish the G0 baseline first; resume on a clean host.
Branch unmerged. 95 model tests pass; no push-suite regression.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-11 13:31:54 +02:00
Marius CiepluchandClaude Opus 4.8 7558951493 feat(g2): change-scoped, budget-bounded evidence pass (unblock push analysis)
Measured: on an unseen push the evidence-tier pass is the whole cost -- 238s of a
291s bedrock run, 99.9% of the non-build/non-query time -- because it tiers every
region of the repo to rank them, and a cold large CPG answers each pair ~50-100x
slower than the design's assumed few ms. The budget is gone before one flow query
runs: selected=0, deferred=1365 all deadline_exhausted, query_seconds=0.

When the push carries a change context AND an execution budget, the evidence pass
is now spent CHANGE-FIRST and BOUNDED:
- evidence candidates are ordered so the changed regions tier before the rest;
- the pass stops once it has taken EVIDENCE_BUDGET_FRACTION (0.5) of the remaining
  budget, in BOUNDED_EVIDENCE_PORTION (256) chunks so the stop is granular;
- the CPG stays whole-repo, so a flow INTO a changed region from unchanged code is
  still traced -- only the question population's long unchanged tail may go untiered.

Honesty: _portioned_evidence returns requests ATTEMPTED, so _evidence_pass splits a
genuine query failure (engine asked, no answer) from a budget skip; the skipped tail
is recorded as evidence_pass_bounded, never read as a clean repository.

Preserves the tested invariants: with no execution budget (the unit tests) the path
is inert, so change context still does not alter the ranker's selection and a changed
dependency still reaches its unchanged ranked question.

Tests: _portioned_evidence bounds at the deadline keeping the first portion; the scan
sets the sub-deadline + bounded portion + change-first order; the bounded tail is
recorded and is not a query failure. 128 model/change-context tests pass; no push-suite
regression (Joern-gated engine_trace/cpg_budget failures pre-exist on HEAD).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-11 12:28:35 +02:00
Marius CiepluchandClaude Opus 4.8 15992d3090 docs(g2): increment 2 measured -- the 236s is 99.9% the evidence-tier pass
Saved-artifact run: total=291s build=52s query=0s arbitrate=238.75s, and
query_seconds_by_kind[evidence]=238.54s. The evidence-tier pass is the whole
middle; affected_context + _scope_work are 0.21s. It tiered 1015 pairs at
~235ms/pair (contended) and exhausted the budget -> selected=0, deferred=1365
all deadline_exhausted. Secondary anomaly: ~50-100x the design's assumed
2.5ms/pair -- the evidence query is pathologically slow on a cold large CPG.

Increment 2's single target is the evidence pass (scan.py:338-359). Fork
recorded: (i) scope+bound the pass change-first, (ii) two-phase neighbourhood
pass, (iii) per-pair speedup. Must preserve reach-into-unchanged.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-11 12:11:57 +02:00
Marius CiepluchandClaude Opus 4.8 a0637f2f9d docs(g2): correct increment 2 -- preserve reach-into-unchanged, measure the 236s split first
A first scoping attempt (restrict analyzed regions to changed files) was reverted:
it breaks a tested design invariant. The engine deliberately selects an unchanged
region's question when a changed region reaches it
(test_changed_dependency_reaches_unchanged_ranked_question;
test_change_does_not_replace_ranker_selection). _changed_files feeds _evidence_pass
only as contextFilter/contextPaths (itemise vs summarise), not to shrink the tier
population. Any scope must keep changed regions PLUS the regions they reach.

Also: increment 1 over-claimed. Increment 0 measured build=67s/query=0s but NOT
the split of the ~236s middle (evidence-tier pass vs affected_context/context
resolution vs _scope_work enumeration). Increment 2 must capture per_kind /
other_seconds / affected_context timing and cut the dominant phase while
preserving reach-into-unchanged -- instrument before fixing.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-11 11:50:41 +02:00
Marius CiepluchandClaude Opus 4.8 9580ad1e30 docs(g2): increment 1 located -- regions are discovered whole-repo, never change-scoped
The 4095 candidates are not an in-scan artifact: push/runner.py:345 discovers
head_regions across the whole head tree, then runner.py:397 passes them wholesale
to scan_repository with change_context threaded in but unused for scoping. The
scan enumerates all regions x families (scan.py:167/364) and, with
order_by_evidence=True, evidence-tiers every pair before analysis -> the ~236s
from increment 0 -> deadline at query_seconds=0.

Fix locus (increment 2): scope the analyzed region set to context.changed_paths +
dataflow neighbourhood (regions.py:279 primitive exists; compare_targeted_base is
precedent). Keep the CPG whole-repo so taint can trace into the change from
unchanged code; only the question set is change-scoped.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-11 11:24:43 +02:00
Marius CiepluchandClaude Opus 4.8 7a1e47e0cc docs(g2): increment 0 confirmed -- CPG build cheap, candidate enumeration is the overspend
Timed live bedrock replay: build_seconds=66.69, query_seconds=0.0, total 303s,
4095 candidate questions all analysis_not_started=deadline_exhausted, selected=[],
regions_scanned=0. ~67s builds the CPG; ~236s enumerates/ranks/prepares 4095
repo-wide candidates for a 3-file change, then the deadline hits before any query
runs. The CPG build is not the bottleneck; the over-broad candidate set is ->
change-scoping is the confirmed lever (and compounds with the correspondence fix,
since per-candidate prep includes correspondence work).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-11 11:21:33 +02:00
Marius CiepluchandClaude Opus 4.8 dc23151022 Merge engine-change-scoped-workstream: open the G2 change-scoped-analysis workstream
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-11 11:10:26 +02:00
Marius CiepluchandClaude Opus 4.8 9f312177ce docs(g2): open the change-scoped engine-analysis workstream (brief + roadmap)
Track the engine-detection lever found while driving G0. Live diagnosis on the
frozen pool: the engine's "no finding" on unseen changes is usually
analysis-never-runs -- it ranks repo-wide candidates and defers them all
(deadline_exhausted) before analyzing the pushed change (bedrock injection,
303s, selected=[], ranked sites unrelated to the change). Lever: seed analysis
from the change's declarations + dataflow neighbourhood (rank.py/mapping.py) so
it runs within budget.

- .kiro/specs/engine-change-scoped-analysis/brief.md: problem (measured),
  hypothesis, anchors, and checkable increments (0 confirm build-vs-analysis
  cost split -> 1 locate selection/budget -> 2 seed from change -> 3 soundness
  + tests -> 4 live re-test -> 5 measure delta on the engine arm).
- roadmap.md G2: active workstreams (change-scoped analysis; the done-but-
  necessary-not-sufficient line-correspondence fix), for progress tracking.

Honest scope: G2, gated on the pool metric; not part of the G0 quick-rule
record-07. Coverage also needs the capability registry (Task 8.3); arm-scale
measurement needs the kube lane.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-11 11:10:26 +02:00
Marius CiepluchandClaude Opus 4.8 c39fbb8457 Merge push-correspondence-run-aware: run-aware base/head line-correspondence
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-11 10:12:03 +02:00
Marius CiepluchandClaude Opus 4.8 abb2058a54 push(snapshot): run-aware base/head line-correspondence (stop discarding intra-run anchors)
The correspondence mapped an equal-region base<->head line pair only when the
line's content was GLOBALLY unique in the file, so common lines (`}`, blanks,
imports) tripped `ambiguous_line_correspondence` and the finding's anchor was
discarded -> the operation came back `unmapped` -> novelty `unknown` ->
`comparison_unknown` -> the engine finding was suppressed (a direct cause of
"near-blind to fixes").

Replace the global-uniqueness gate with run-aware positional anchoring: segment
the equal-region pairs into contiguous runs (strictly +1/+1, as _equal_line_pairs
emits), and map a pair when its content occupies exactly one old run and one new
run. Duplicates WITHIN a contiguous run are positionally forced and now map;
duplicates ACROSS runs split by an edit are aligned arbitrarily by git and stay
withheld + flagged (soundness preserved -- the moved-repeated-operation guard
still passes, so no false `new`/identity claims).

Tests: update test_change_context_unknown_base_binary_and_ambiguous_lines_are_
unresolved (its two in-run sinks now anchor); keep the moved-operation soundness
test; add intra-run-maps, cross-run-withheld, and distinctive-line-anchors cases.
163 push tests pass, ruff clean.

Honest scope (design.md amendment): this removes one real cause but is
necessary-not-sufficient -- live-testing a messy 14-file Java change showed
coverage stays `incomplete` due to other, dominant blockers (capability_unavailable
until Task 8.3; the engine produced no actionable defect; eligibility/projection
gaps). Engine coverage on unseen diffs needs those too, not just correspondence.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-11 10:12:03 +02:00
Marius CiepluchandClaude Opus 4.8 c657c2219b Merge doc-g0-readiness-2026-10-11: G0 state, readiness plan, bucket-first docs
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-11 08:17:06 +02:00
Marius CiepluchandClaude Opus 4.8 d03590148f docs(g0): record the 2026-10-11 state and readiness plan; bucket-first + image provenance
Fold the night's learnings into the roadmap, steering and the
unseen-repo-evaluation spec so the process is reproducible and the
project is "ready now":

- roadmap.md: a 2026-10-11 state table (pool p1 frozen; harvest hardened
  + bucket-first; replay-image provenance pinned; baseline-on-kube next)
  and a 5-step readiness plan to record 07.
- storage-streaming.md: RustFS-over-local as the default (local disk is
  the scarce resource); the implemented ousast-corpus layout; replay
  outputs belong on RustFS; the AX store endpoint is operator-owned.
- design.md: an "Amendments (2026-10-11)" section (bucket-first
  persistence, ~2.2% rate-bound yield, machine-label record 07, pinned
  replay image provenance).
- tasks.md: a folded-in amendment block (pool p1 frozen + freeze recipe;
  bucket-first; CI replay image; machine-label baseline; kube lane +
  ax/store as operator-owned infra).

Facts stated are verified (measured yield, freeze digest, GHCR anon pull,
batch.py's endpoint check); infra capability/policy is left to the
operator, not assumed. The frozen pool artifacts themselves are not
committed here (a separate samples/licensing decision).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-11 08:17:06 +02:00
Marius CiepluchandClaude Opus 4.8 7b711a6150 Merge unseen-getrepo-cache: bucket-cache get_repo (one-time used-set canonicalization)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-10 17:10:19 +02:00
Marius CiepluchandClaude Opus 4.8 a3a38f1b9f unseen: bucket-cache get_repo (kill the repeated used-set canonicalization tax)
The real bottleneck was not enumeration or extraction: canonicalize_used
resolves the entire exclusion set (measured: 4,499 names) via serial
GitHub get_repo on every launch, before any harvesting -- ~30 min of
network-latency-bound work that also burns ~4,499 of GitHub's 5,000/hr
budget before real work starts (hence the low CPU and the "never
finishes"). The earlier resolution-cache only banked via
BankedGitHub.resolve, but canonicalize_used calls eligibility.resolve ->
get_repo directly, so it was never cached and re-ran every time.

Cache get_repo at the real choke point. CorpusBank gains cache_repo /
get_cached_repo storing one object per normalized name at
<prefix>/repos/<sha256(name)>.json -- NOT seed-scoped, since repo
metadata is seed-independent, so every seed/run shares it. 404s are
cached as None too (a found-but-None hit) so known-missing repos are not
re-fetched. BankedGitHub.get_repo now checks the local journal, then the
bucket, and only then GitHub (banking the result). canonicalize_used,
advisory_index and candidate resolution all funnel through get_repo, so
all of them are covered.

Effect: the first run pays the lookups once and banks them; every later
launch reads them from the bucket (no GitHub, no rate-limit spend), so
the rate budget goes to actual candidate harvesting and restarts are
cheap. Proven live against ousast-corpus (repos/ fills 1:1 with
resolutions). 206 unseen tests + new repo-cache test pass, ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-10 17:10:19 +02:00
Marius CiepluchandClaude Opus 4.8 21aebe8538 Merge unseen-resolution-cache: cache enumeration + resolution to ousast-corpus
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-10 16:33:29 +02:00
Marius CiepluchandClaude Opus 4.8 06b8a40c46 unseen: cache enumeration + resolution to ousast-corpus (restarts skip the ~26-min prefix)
The decision bank made extraction durable, but every restart still
re-paid the ~26-minute advisory enumeration + per-advisory resolution,
since those lived only in the local journal. Mirror them into the bucket
so a restart reads them back and goes essentially straight to any
remaining extraction.

bank.py: bank_resolution/resolved at <prefix>/<seed>/resolved/<sha256(ghsa)>.json
(streamed one-by-one as each advisory is decided), cache_advisories/
cached_advisories at <prefix>/<seed>/advisories/list.json, and
bank_repository/repositories (caches the advisory_index repo-resolution
pass, preserving renames and 404s). All gated by RESOLUTION_CONTRACT="v1"
(extract.py) -- bumped only when enumeration/eligibility/source.candidate
output changes; recent fetch/parse/pool changes do not bump it.

draft.py run(): with --bank, advisories load from the cache (else
enumerate then cache); eligibility decisions load from the bucket
(contract-matched) to skip re-resolution; new resolutions stream to the
bucket before the local checkpoint write. Repository lookups reuse the
repositories/ cache. No --bank => local-journal behavior unchanged.

Net: advisories + resolutions + extraction all resume from ousast-corpus;
restarts no longer re-enumerate or re-resolve. Fresh and resumed runs
produce byte-identical manifests at 1 and 4 workers. Bank methods proven
live against ousast-corpus. 356 tests pass, ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-10 16:33:29 +02:00
Marius CiepluchandClaude Opus 4.8 857263d975 Merge unseen-corpus-bank: durable decision bank in ousast-corpus (stream + resume)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-10 16:09:47 +02:00
Marius CiepluchandClaude Opus 4.8 4aa612253a unseen: durable decision bank in ousast-corpus (stream + resume, retires the local journal)
The draw lost all progress on any restart because the local journal
invalidates on a code change and is the only cross-run state. Make the
bucket the source of truth: stream every per-repo accept/reject to
ousast-corpus as it is decided, keyed by sha256(repository+"\n"+fix) so
writes are idempotent (content dedup at rest), and resume by listing the
bucket and skipping already-decided repos. No supervisor/watchdog needed
-- a crash or a code change just re-runs and resumes.

New benchmarks/unseen/bank.py: corpus_settings (CORPUS_S3_* from .env,
missing key named, value never printed), CorpusObjects (lazy boto3,
path-style + SigV4 verified live against ousast-corpus @ 10.0.0.54:9001),
CorpusBank.bank/decided with one-retry put/get, split_decided. Decisions
live at <prefix>/<seed>/decisions/<key>.json with repository, fix,
accept/reject, reason, entry, exclusions, counts, contract, seed, ts.

draft.py: --bank [prefix] opens the bank on the coordinator (never shipped
to fork workers); run() resumes from bank.decided() before extracting,
reusing accepted entries across contracts (deterministic from repo+commit)
and rejections only when contract == EXTRACTION_CONTRACT ("v1", in
extract.py); collect() streams each decision one-by-one. Seed-order
acceptance, fork/used-set dedup, and duplicate-advisory reuse preserved.
Without --bank, local-journal behavior is unchanged.

CorpusBank proven live end-to-end (bank accept+reject, idempotent re-bank,
decided/split roundtrip). 325 tests pass offline, ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-10 16:09:47 +02:00
Marius CiepluchandClaude Opus 4.8 95bed7005c Merge unseen-fetch-bulk: full-objects clone, no per-blob lazy fetch
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-10 15:53:05 +02:00
Marius CiepluchandClaude Opus 4.8 00a1fdb30f unseen: full-objects clone instead of blob:none promisor (bulk pack, no lazy per-blob fetch)
The harvester cloned --filter=blob:none --no-checkout, then blame/log/diff
triggered the promisor to fetch each needed blob ON DEMAND -- hundreds of
round-trips per repo, the dominant cost that made the draw unable to
converge. The blobless clone was a premature optimization that backfired.

Drop --filter=blob:none from the clone and the two commit fetches. A
full-objects --no-checkout clone brings the whole commit graph AND every
blob in one pack, so blame/log/diff read locally; --no-checkout still
skips the working tree so disk stays bounded (plus the existing 500 MiB
clone_size_cap and the disk guard). With no promisor remote, a genuinely
missing object now errors loudly instead of silently round-tripping --
the instrument guarantee that the bulk fetch was complete.

Extraction semantics are unchanged (same SZZ introducing-commit positive,
same stratified ordinary negatives, byte-identical entries); only the
object-transfer strategy changes. Test inverted to assert no blob:none
filter on the clone; README updated. 145 unseen tests pass, ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-10 15:53:05 +02:00
Marius CiepluchandClaude Opus 4.8 f712b2cffa Merge unseen-parse-deadline: wall-clock deadline for the Python parse phase
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-10 14:46:18 +02:00
Marius CiepluchandClaude Opus 4.8 a49b15b720 unseen: bound the per-repo Python parse phase (--parse-deadline)
Git subprocess calls were deadline-bounded (Git.command_timeout uses the
remaining per-repo budget), but a pure-Python parse between git calls
(repository_changes -> history/blame/diff on a giant file) had no
deadline: when it hung, no further git call fired the git deadline, so
the worker stalled indefinitely (observed >720s with zero git
processes). An external watchdog was papering over it at ~20 min/stall.

Add extract.process_deadline(seconds): a SIGALRM/setitimer wall-clock
alarm installed on the worker's main thread that raises
ExtractionTimeout("wallclock_timeout"), with no-op guards (None/<=0, off
the main thread, or no SIGALRM support) and strict timer-cancel +
handler-restore in finally. Wrap the whole extract_repository body in it
so the timeout propagates to the existing `except source.Rejected` and
journals as a clean rejection, not a worker failure.

Budget = --extract-deadline (git, 240s) + --parse-deadline (new, 120s)
= 360s total ceiling, so git-bound repos keep their full budget (yield
unchanged) while a hung parse is rejected well before any external
watchdog. Record parse_deadline_seconds in the report.

295 unseen tests pass (incl. timeout + handler-restore + no-op cases),
ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-10 14:46:18 +02:00
Marius CiepluchandClaude Opus 4.8 c4c08d1ca1 Merge unseen-pool-size: configurable pool size / slices
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-10 07:54:49 +02:00
Marius CiepluchandClaude Opus 4.8 5ea28ca9f8 unseen: configurable pool size and slice count (--pool-size / --slices)
The draw hardcoded 300 repos / 3 slices. At the measured lazy-fetch rate
the full 300 projects ~2.5 days; a 100-repo single slice (slice 1, all
the G0 baseline needs) is ~20h. Add --pool-size (default 300) and
--slices (default 3), validated (>=1, slices<=pool_size, evenly
divisible). Generalize assign_slices' stratified allocation from a
literal 3 to `slices`, the run() break / insufficient check / report
(changes = pool_size*13) to pool_size, and record both in the journal
inputs. Remove bisect's three-slice assumption; freeze reads the size
from the draft. Defaults (300/3) reproduce prior output exactly.

With --pool-size 100 --slices 1 the pool is the first 100 accepted repos
in seed order, all in slice 1 (deterministic, byte-identical across
worker counts). 280 unseen tests pass, ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-10 07:54:49 +02:00
Marius CiepluchandClaude Opus 4.8 f0bcbe08b5 Merge unseen-revert-materialize: lazy blob fetch, drop broken prefetch
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-10 05:06:24 +02:00
Marius CiepluchandClaude Opus 4.8 0023f61ff8 unseen: revert the batched-blob materialize (it failed git fetch on ~half of repos)
Git.materialize enumerated every object reachable from HEAD and tried to
`git fetch --no-filter` them by SHA. Git rejects that for real repos
("did not send all necessary objects" / "bad revision"), exiting 1 ->
InstrumentFailure(git_exit_1) that failed the whole repository. Measured
in a live draw: 326/670 attempts, ~49% of all failures, which froze the
accepted pool at 44/300. Reproduced on pallets/flask (26221 reachable,
9287 missing, fetch exit 1).

Git cannot batch-fetch arbitrary tree/blob objects by SHA, so remove
materialize and restore the original lazy path: the blob:none clone keeps
the commit graph and the introducing/blame/ordinary-history reads fetch
their blobs on demand through Git's promisor. The per-fix tolerant fetch
loop, the process-pool parallelism, the 25-with-zero-entries guard, the
per-repo deadline and resume are unchanged. Extraction content is
byte-identical (only the prefetch is gone).

217 unseen tests pass, ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-10 05:06:24 +02:00
Marius CiepluchandClaude Opus 4.8 40bfdb8e18 Merge unseen-guard-and-gitexit1: guard + tolerant fix-fetch
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-09 23:54:08 +02:00
Marius CiepluchandClaude Opus 4.8 44311ff2fa unseen: fix systemic-guard miscalibration + the batched-fetch yield regression
Two defects stalled the draw at 40/300:

1. The systemic guard used an absolute `instrument_failures > 200` cap.
   At ~10% extraction yield, reaching 300 entries legitimately produces
   far more than 200 per-candidate failures, and the counter is rebuilt
   from the journal on resume -- so once past 200 the guard tripped
   immediately on every restart (permanent stall; the supervisor's
   no-progress guard then stopped). Drop the absolute cap; keep only
   `>=25 failures with zero entries`, which still catches a genuinely
   broken instrument (it yields no entries) without aborting a working
   low-yield draw. Bounded anyway by the population / max-hours.

2. repository_changes fetched all fixes in one `git fetch origin *fixes`,
   so a single unfetchable secondary fix ref failed the whole repo
   (extract_git_exit_1 was 166 of ~200 extract_ failures -- the dominant
   loss). Fetch the extra advisory fixes per-fix, best-effort, skipping a
   git_exit_ failure; candidate.fix stays strict (already fetched in
   extract_repository). Only resolved fixes are materialized and
   fix-site-indexed.

218 unseen tests pass (incl. 1000 failures after a success not tripping
the guard, and fetch-error regressions), ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-09 23:54:08 +02:00
Marius CiepluchandClaude Opus 4.8 a9be42b5f8 Merge unseen-extract-ascompleted: saturate the extraction pool
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-09 20:15:54 +02:00
Marius CiepluchandClaude Opus 4.8 2cf4caa959 unseen: keep the extraction pool saturated (as-completed, not oldest-first)
The ProcessPool spawned its workers but ordered_extractions blocked the
main thread on the OLDEST future, so while one slow repo finished the
other workers sat idle: live profiling showed 1 of 4 cores busy, ~67%
idle, ~1 attempt/min.

Separate the seed-order acceptance cursor (`pending`: row+decision+key,
no futures) from the worker set (`in_flight`: future->key). Keep up to
`workers` jobs running; journal each result as it COMPLETES via
wait(FIRST_COMPLETED) in any order and refill freed slots, never waiting
on the oldest. A candidate is accepted/rejected only when its result is
in the journal, advanced strictly in seed order -- so the pool runs
out of order while the accepted pool stays byte-identical to a 1-worker
run. Fork-network dedup and the systemic guard still apply at the cursor.

New real-process test proves later extractions are journaled before an
earlier slow one completes (completion order != seed order) while
accepted entries stay in seed order; the old loop fails it. 210 unseen
tests pass, ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-09 20:15:54 +02:00
Marius CiepluchandClaude Opus 4.8 192b2b7e7c steering: stream run artifacts to RustFS, one bucket per purpose
Local disk is the scarce resource (host near full; the draw stops at
<2G free). Record the direction to stream durable run artifacts to RustFS
instead of hoarding on local disk, split across purpose-specific buckets
(sast-memory durable, sast-pools durable archive, sast-runs transient
with aggressive lifecycle), each with a lifecycle tuned to its artifact
-- the lesson from the engine-queue versioning bloat. Extends the
existing S3Store pattern; new buckets are an operator admin action.
Planned as a follow-up spec, not retrofitted into an in-flight run.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-09 19:43:26 +02:00
Marius CiepluchandClaude Opus 4.8 1b0b2bdf31 Merge unseen-freeze: freeze p1 (bisect guard, freeze writer, protocol)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-09 19:23:26 +02:00
Marius CiepluchandClaude Opus 4.8 197fe8a19c unseen: implement freeze p1 (group 3: bisect guard, freeze writer, protocol)
Group 3 was unimplemented, so there was no way to turn the draft into a
frozen pool. Add:

- bisect_guard.py: stages the cache draft and bisects it against the
  reserved-repository guard, run as a black box
  (pytest -k referenced, stdout/stderr discarded, exit code only; an exit
  other than 0/1 aborts as an instrument failure). Drops guard-tripping
  candidates and reports the dropped COUNT only -- never a name or URL.
  Guard runner is injectable for offline tests.
- freeze.py: writes canonical pool-p1.toml (+ private counterpart,
  reusing draft's public/private split and private_digest) and
  freeze-p1.json (used-set digest, per-source counts, guard result,
  dropped count, label-check counts, freeze digest = sha256 of the
  canonical public manifest). A matching retry is a no-op; any changed
  frozen artifact requires a new pool id (never an overwrite).
- protocol-p1.md: the frozen scoring/statistics protocol from design
  sections 3 and 4.

Offline only; the guard subprocess is the sole subprocess and is mocked
in tests. 42 new tests pass (incl the group-1.3 guard digest check on a
fixture pool); neighbouring unseen suites green; ruff clean. New files
only -- does not touch the journal-pinned draft/extract modules, so a
running draw is unaffected.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-09 19:23:26 +02:00
Marius CiepluchandClaude Opus 4.8 23ede07608 Merge unseen-extract-processes: process-pool extraction for true multicore
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-09 19:08:01 +02:00
Marius CiepluchandClaude Opus 4.8 79f04400f6 unseen: extract in processes, not threads (GIL was the ceiling)
Extraction is CPU-bound Python (git-output parsing, HUNK regex,
ast.parse of hundreds of source files per repo). A ThreadPoolExecutor
cannot parallelize it: live profiling showed one thread pegging a core
at 99.7% while three cores sat idle and the others waited on the GIL, so
six "workers" barely beat serial (~0.25 entries/min).

Switch ordered_extractions to a ProcessPoolExecutor with a fork context
(copy-on-write memory; lets the existing in-memory test fakes cross into
children without pickling). Effective workers are clamped to the core
count. The parent still owns the journal, REST (finish_license), decide,
counters, the fork-network dedup, strict seed-order acceptance, the
systemic guard, and the graceful drain. Children run extract_repository
with github=None on picklable args and return their own count deltas
(threads had shared a Counter; processes return counts in the result).
A broken pool is replaced; a collateral worker_crash is a skip outside
the guard, a task-specific extract_worker_failure counts toward it.

Determinism is unchanged and tested with REAL processes: workers=6 ==
workers=1 byte-identical (fork collisions, out-of-order completion,
resume). 208 unseen tests pass, ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-09 19:08:01 +02:00
Marius CiepluchandClaude Opus 4.8 9f078fe7d0 Merge unseen-extract-parallel: concurrent extraction + batched blobs
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-09 17:37:05 +02:00
Marius CiepluchandClaude Opus 4.8 65714405eb unseen: parallelize extraction + batch per-repo blob fetch
Serial extraction at ~147s/repo made a 300-repo pool take days, and
~626 lazy per-file blob faults per clone pushed slow repos past the
240s deadline (a draw reached 21 entries from 347 attempts, 160 of them
extraction_timeout). Both fixed.

Parallel: a bounded ThreadPoolExecutor (--extract-workers, default 6)
runs clones concurrently, each into its own clone with a private counter.
The main thread alone owns the journal, REST (license via finish_license),
counters, and acceptance. Entries are accepted in STRICT SEED ORDER with
the fork-network dedup applied at acceptance, so the pool is byte-
identical to the serial draw (tested: workers=6 == workers=1, including
fork collisions, out-of-order completion, and resume). The systemic guard
and resilient skips are unchanged; a graceful stop drains started work to
the journal for resume.

Batched blobs: repository_changes fetches the fixes together and
Git.materialize fetches all missing blobs for the read revisions in one
`fetch --no-filter --stdin` (rev-list --missing=print), instead of per-
file promisor faults. Content is byte-identical (tested; >=12 fetches
collapsed to 1).

202 unseen tests pass, ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-09 17:37:05 +02:00
Marius CiepluchandClaude Opus 4.8 4f7eedddf1 Merge search-executor-command-once: upload command once per task
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-09 14:03:30 +02:00
Marius CiepluchandClaude Opus 4.8 b9f7958414 search: upload the executor command once, not on every poll
ObjectStoreExecutor.submit re-PUT the identical command.json on every
0.2s result poll, so a task running minutes wrote ~1,500 versions of one
engine-queue/search-exec/<id>/command.json. On the versioned memory
bucket that was the dominant growth: 109,715 versions, 10.3 GB (91% of
the bucket). retry_mailbox already retries a lost PUT acknowledgement
internally, so one upload per task is the correct delivery guarantee.

Guard the PUT with an `uploaded` flag: upload once (still retried on
transient failure), then poll only the result. One command.json version
per task instead of ~1,500. All submit semantics preserved.

59 executor tests pass, ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-09 14:03:30 +02:00
Marius CiepluchandClaude Opus 4.8 6773c63634 steering: operator Discord notifications for OpenUltraSAST process status
Record the policy: OpenUltraSAST notifies the system/AI operator on
Discord about its own processes and results, and especially when a
process is stuck or impaired (stall, insufficient result, abort,
instrument_failure, timeout, budget/resource limit). Operator-facing,
not developer debugging, and explicitly not k3s/ax infrastructure.

Credentials by name in gitignored .env (DISCORD_WEBHOOK_URL,
DISCORD_OPERATOR_ID), listed blank in .env.example; off when unset.
Messages carry status/counts/digests only, under the hardening and
measurement-record redaction (no payloads, secrets, or repo content).
Notifier implementation to be wired/specced; this states the behavior.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-09 11:58:10 +02:00
Marius CiepluchandClaude Opus 4.8 aae586de69 unseen: decide phase skips a per-candidate HTTP failure too
The resilient draw only covered extraction. A per-candidate transport
failure in the decide/eligibility phase -- get_commit on an unresolvable
fix ref raising ValueError("http_failure") (observed: ljharb/shell-quote
commit 1518179) -- still aborted the whole draft, because decide only
caught source.Rejected and eligibility.RepositoryNotFound.

decide now catches that http_failure and returns a "decide_http_failure"
skip, and the run loop feeds decide + extract per-candidate instrument
failures into one systemic guard (25 with zero successes, or >200), so a
single bad commit lookup skips while a broken instrument (e.g. a network
outage failing every lookup) still aborts loudly.

Full unseen draft suite passes (48), ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-08 18:54:46 +02:00
Marius CiepluchandClaude Opus 4.8 1f89f04fcf Merge unseen-resilient-draw: skip bad repos, abort on systemic failure
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-08 17:19:36 +02:00
Marius CiepluchandClaude Opus 4.8 b59dd651d8 unseen: resilient draw -- a bad repo skips, a broken instrument aborts
Individual repositories in the advisory population raise per-candidate
extract.InstrumentFailure during extraction (git_exit_128 on an
unfetchable fix commit, clone_size_cap, and others), and each one
aborted the whole 300-repo draft. The draw is healthy; one bad repo
should not reset it.

extract_repository now also catches extract.InstrumentFailure and
OSError and records them as "extract_<reason>" rejections, so the draw
skips that repo and continues (the reason, e.g. extract_git_exit_128,
shows in the rejections counter). ExtractionTimeout keeps its existing
extraction_timeout skip; any other exception still propagates as a real
abort, so genuine code bugs are not masked.

A systemic guard preserves the instrument rule: run() raises
InstrumentFailure("systemic_extraction_failure") if 25 extraction
attempts fail with zero successes (e.g. git missing, auth broken) or if
extraction failures exceed 200 overall. A handful of bad repos with many
successes never trips it.

190 unseen tests pass, ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-08 17:19:36 +02:00
Marius CiepluchandClaude Opus 4.8 ec7bb64378 Merge unseen-extract-deadline: per-repo extraction time bound
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-08 13:22:23 +02:00
Marius CiepluchandClaude Opus 4.8 b074cf7f89 unseen: bound per-repo extraction time; a slow repo skips, never aborts
A pathologically slow repository (observed: tensorzero, 183s to clone,
then `git blame -C` past the 180s command limit) raised
subprocess.TimeoutExpired, which Git.run mislabeled as a fatal
InstrumentFailure("git_launch_failed") and aborted the whole 300-repo
draft. The same slowness stalled throughput (a 54-minute no-progress
gap).

Give each clone a per-repo wall-clock budget (--extract-deadline,
default 240s, plumbed Clones -> Git, counting the initial clone). A
timeout -- a command exceeding the remaining budget, or the budget
elapsing -- raises ExtractionTimeout(Rejected), which the existing
rejection handler turns into an "extraction_timeout" skip so the draw
moves to the next candidate. Genuine launch failures and non-zero git
exits stay fatal; timeouts and launch failures are now distinct. The
deadline is recorded on the report only, not the journal inputs, so
changing it does not refuse resume.

181 unseen tests pass, ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-08 13:22:22 +02:00
Marius CiepluchandClaude Opus 4.8 23c8644a37 docs: README reflects the dynamic proof arm and unseen-repo release gate
Update the front door and Status to the current state: static
abstract-interpretation blocking is NO-GO by hand, so the work is a
dynamic proof arm that builds and runs the app and demonstrates a
finding by a differential (triggers on the vulnerable revision, not the
fixed one), earning a block by execution rather than calibration. It is
pilot, not shipped. The release gate is a measured catch/false-alarm
number on unseen repositories, now being produced. Bump v2.0.0 -> v2.0.1,
link the developer guides.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-08 12:51:35 +02:00
Marius CiepluchandClaude Opus 4.8 f4c7143525 unseen: make clone disk-size guard tolerate git's mid-walk churn
disk_bytes() summed the clone tree with rglob + stat() as a size cap
before every git command. On a blob:none clone, `git show` triggers an
on-demand blob fetch that writes and deletes transient
.git/objects/pack/*.rev files, so a path rglob had just listed could be
gone by the stat() call -> FileNotFoundError aborted the whole draft as
an instrument_failure. The race can hit any candidate, so it was a
non-deterministic draft-killer (observed extracting a real npm repo).

Skip a file that raises OSError mid-walk; the size cap is unaffected.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-08 11:16:13 +02:00
Marius CiepluchandClaude Opus 4.8 0a637e8989 Merge unseen-cursor-pagination: follow Link rel=next cursors
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-08 10:26:11 +02:00
Marius CiepluchandClaude Opus 4.8 5620165540 unseen: cursor pagination for advisory enumeration
The GitHub /advisories endpoint ignores the `page` query parameter: it
paginates with the Link header's rel="next" cursor. The page-number
pager re-fetched the same first 100 rows forever and never terminated,
so enumeration never completed and no draft converged (a run fetched
5137 "pip pages" holding only 100 unique advisories).

Surface the successful response headers from JSONTransport, and walk the
rel="next" cursor per ecosystem until it is absent. The ecosystem /
published>=2019 / bare-number cwes filters are preserved on the first
request. Journal keys stay ecosystem:n (now cursor-walk positions) with
the saved next URL for resume; journal_version 4 refuses older journals.

Live smoke via the production pager: 300 unique advisories pulled in 3
API calls, advancing past the first page. Offline: 170 unseen tests
pass, ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-08 10:26:11 +02:00
Marius CiepluchandClaude Opus 4.8 2d8f4f9e22 Merge unseen-sourcing-filter: server-side advisory filtering
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-08 08:19:30 +02:00
Marius CiepluchandClaude Opus 4.8 90e2400e6f unseen: filter advisory enumeration server-side (ecosystem/date/cwes)
enumerate_advisories paged the entire global GitHub reviewed-advisory
database (every ecosystem, every year, every CWE) before extraction, so
a 300-repo draft never converged -- a live run reached 21,962 advisories
in 4.5h with extraction never started.

Push the existing candidate() rejections into the query: page per
ecosystem, published>=2019-01-01, and cwes restricted to the seven
families. The cwes filter uses bare CWE numbers (89), which is what the
GitHub advisories API matches -- "CWE-89" silently returns zero. The
whole cwes filter is omitted if the taxonomy ever holds a non-exact
token, so a selectable advisory is never dropped. Journal version 3
(ecosystem:page keys) refuses incompatible older journals.

Live page-1 smoke (pip, npm): 100/100 rows carry a family CWE, none
published before 2019. Offline: 166 unseen tests pass, ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-08 08:19:30 +02:00
Marius CiepluchandClaude Opus 4.8 d7bf0ba750 unseen: sourcing robustness -- retry transient HTTP/connection errors, shrink the advisory journal
A live G0 sourcing run failed on both: a transient connection/decode error aborted
the whole enumeration (the bare except raised http_failure on the first blip), and
the advisory journal stored full raw advisory JSON and reached 1.77 GB on a near-full
host. Fix: the JSON transport now retries URLError/timeout/ConnectionError/IncompleteRead/
JSONDecodeError through the same attempt-4 backoff as HTTP 5xx/429 (fatal only after the
cap or on a genuinely non-retryable error); enumerate_advisories journals only the fields
resolution/extraction consume via source.reduce_advisory (ghsa id, published/withdrawn,
type, CWE ids, package ecosystem/name for OSV, fix-commit links), with journal_version
bumped to 2 so an old full-format journal is rejected and resume-skip stays intact.

Offline: test_unseen_source/draft/extract 133 passed; ruff + format clean; mypy unchanged
(pre-existing verify.py lambda only).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 18:23:40 +02:00
Marius CiepluchandClaude Opus 4.8 56cde9f535 spec(proof-arm-opencode): OpenCode MCP is the endpoint for PoC AND fix [skip ci]
Maintainer direction: the same agentic endpoint authors the PoC, writes a candidate
fix against the repair direction + failing PoC, and hands it to the differential
verifier, which re-runs the PoC and confirms absence. Closes the no-auto-patch gap:
the proof is the acceptance test for the fix (a wrong fix is caught), the fix stays a
developer-reviewed proposal, and the green proof is the objective signal. Specced
alongside proof authoring, after qualification.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 17:34:52 +02:00
Marius CiepluchandClaude Opus 4.8 7f8a492857 docs: use-case reference -- catch, prove, fix, re-verify absence (+ GitHub Actions example) [skip ci]
The loop: catch a regression in the diff, prove it with a reproducible check,
the dev/agent fixes it (tool gives a repair direction + the failing proof, not an
auto-patch), then the same proof re-runs and no longer triggers = absence
demonstrated by execution; the proof becomes a committed regression test. Example
GitHub Actions workflow uses the real 'ousast pre-push . --base --head --deadline
--mode --artifact' entry; PR-comment formatter marked as a placeholder. Honest
status separates what exists (re-verify by construction, --base/--head) from
delivery work (PR formatter, auto-landing the regression test) and the open
large-repo latency.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 17:34:06 +02:00
Marius CiepluchandClaude Opus 4.8 dfc0956086 docs: frame the proof as prioritization evidence for busy devs [skip ci]
Revise the developer guide for engineers with full backlogs and no spare schedule:
a vague warning dies in the backlog, so the value of the demonstrated proof is the
prioritization conversation -- reproducible evidence (input, behavior, the one-line
fix difference) a dev takes to a product owner or the business to win capacity,
dropped into the ticket system as a location, repair direction, and reproduction.
mcp-shift-left-agent brief: the agent's return should be ticket-ready evidence.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 17:30:14 +02:00
Marius CiepluchandClaude Opus 4.8 c49f334651 docs: developer-facing pre-push security guide + mcp-shift-left-agent brief [skip ci]
docs/pre-push-for-developers.md: COBESY-structured guide for engineers who are
strong at dev but not security. Leads with the conclusion (zero-setup, no
annotations, diff-scoped, proof-backed), frames the agent-era case (a pre-push
check is how you oversee coding agents), states control (advisory vs blocking,
budget, lane, models), and is honest about limits with an explicit disproof
condition (coverage, large-repo latency, generalization being measured by G0).

.kiro/specs/mcp-shift-left-agent/: follow-up brief for an MCP agent + skills that
give devs and coding agents early shift-left feedback and a required pre-push
security step for agent oversight; builds on the existing ousast mcp server;
sequenced after the safety net is qualified.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 17:28:49 +02:00
Marius CiepluchandClaude Opus 4.8 7bfe6fff7e merge(search-with-proof): verification-architecture doc + proof-arm-opencode brief [skip ci]
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 16:06:56 +02:00
Marius CiepluchandClaude Opus 4.8 60617a92ba docs(search-with-proof): verification architecture + proof-arm-opencode follow-up brief
verification-architecture.md documents the existing two-stage design: the Cairn-inspired
search (brain + model calls on the VM) and the DAST proof (gVisor AX tasks in kube-ax),
the trust boundary (secrets VM-only, cluster tasks secret-free with restricted egress),
and the three verify lanes (ax=cluster default, vm=local Bubblewrap for $0 proofs,
docker=replay). Clarifies the proof is dynamic, not abstract interpretation: AI triage
stays the static backbone, proof earns the block by execution.

proof-arm-opencode/ is a follow-up brief: drive proof-authoring with OpenCode MCP, an
abliterated/GLM proof model (DeepSeek stays the search model), DSPy-optimized from
OpenObserve traces, to beat the safety-classifier refusals and nondeterministic wiring.
Flags the trust-domain decision that must come first (A: scoped key + egress inside the
cluster; B: brain on VM, only DAST in gVisor — recommended to start). Credentials live in
.env by name only (ABLITERATION_*, OPENOBSERVE_*); no secret value is in any tracked file.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 16:06:56 +02:00
Marius CiepluchandClaude Opus 4.8 b1a050e16b merge(search-with-proof): reproducible npm builds via --before commit date [skip ci]
Lockless old npm projects resolve era-appropriate deps (fixes glance filed->mime@4 crash);
npm ci untouched; git-failure falls back to plain install.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 13:37:23 +02:00
Marius CiepluchandClaude Opus 4.8 c30dd9126a search-with-proof: reproducible npm builds for old lockless projects (--before commit date)
A lockless npm project (e.g. glance, 2015) resolved an unpinned transitive range to the
latest major ([email protected] -> mime@4, which dropped .lookup()), so the built app crashed
serving files -- a build-reproducibility defect, not an app or agent fault. The executor
now, for the npm-install (no-lock) case only and only when .git is present, resolves
dependencies as of the checkout's HEAD commit date via `npm install --before <date>`,
giving era-appropriate, compatible versions. npm ci (lockfile present) is untouched, and
any git/date failure falls back to plain install. demo.py also recognizes
npm-shrinkwrap.json as a lock for the ci-vs-install choice.

Tests: executor + smoke 75 passed, demo 107 passed; ruff clean; mypy clean on executor.py.
Executor-side change -> image rebuild required.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 13:37:23 +02:00
Marius CiepluchandClaude Opus 4.8 8de0c2732b spec+docs(search-with-proof): record first cluster demonstration + fixed-side negative [skip ci]
Feed the milestone back into the spec and steering. HANDOFF: the open "first demonstrated"
question is RESOLVED -- xiaomusic (path, prefix-check-bypass) demonstrated on the vulnerable
side (3/3 affected, 0/3 safe) and the blind fixed-side search produced 0 false proofs; pair
total $1.22. Records the six-fix chain that enabled it (parity, verify-time-only, fair-share
budget, served-root guidance, oracle prefix-bypass coverage, control-signal feedback) plus the
keystone. tasks.md: implementation note; pilot (6.2/6.3) in progress. spec.json: pilot status.
safety-net steering: proof-based BLOCK mechanism reached first execution-backed demonstration
(feasibility-pilot scope, not release readiness; pilot exit + 73-negative qualification open;
classifier stays triage). search README: status note. Payload-free; demos/boards stay gitignored.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 13:25:52 +02:00
Marius CiepluchandClaude Opus 4.8 ecd1372568 measurements: FIRST cluster demonstration (xiaomusic prefix-bypass path traversal) [skip ci]
search-with-proof produced its first demonstrated finding on a real CVE fix-pair end to
end on kube-ax: vuln observes, fixed does not, 0 false proofs, $0.37, 55 calls. Enabled
by the full session fix chain (parity, verify-time-only, fair-share budget, served-root
guidance, oracle prefix-bypass coverage, control-signal feedback). Payload-free record.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 12:41:01 +02:00
Marius CiepluchandClaude Opus 4.8 1ec4ee4c55 merge(search-with-proof): served-root control-signal feedback to the agent [skip ci]
Non-blocking advisory hint when the control shows the app didn't serve ${served_root},
so the agent self-corrects its served-directory wiring. Host-side; verdicts unchanged.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 12:03:45 +02:00
Marius CiepluchandClaude Opus 4.8 20c5742d84 search-with-proof: feed the control signal back to the agent (served-root hint)
When a path demo does not serve ${served_root}, the keystone control fails
(SideRecord.control_observed False) but the search only reported a generic
inconclusive, so the agent kept making the same wiring mistake. The pilot now
surfaces the affected side's control_observed in the verify result, and the
coordinator appends one advisory, deduplicated "served-root" hint (visible in the
board snapshot the agent reads) telling it to set the served-directory config to the
${served_root} placeholder. Advisory only: verdicts, _end, the oracle, the differential
and the demo validator are unchanged; the hint fires only when control_observed is False.

Tests: coordinator + pilot_run suites 78 passed; ruff clean; mypy unchanged (pre-existing
benchmarks errors only). Host-side change, no image rebuild.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 12:03:45 +02:00
Marius CiepluchandClaude Opus 4.8 d1fe78987b measurements: xiaomusic vuln search -- oracle fix landed (agent targets prefix-sibling) but served-dir wiring missed this run [skip ci]
Oracle broadening works in the agent: demo now targets the prefix-sibling vector. Still
inconclusive (0 false proofs, $0.29): the agent set the served-dir env var to its own
value, not ${served_root}, so the sibling canary was outside the served tree. Served-root
wiring is nondeterministic (a prior run wired it). Last-mile agent-reliability gap.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 11:52:19 +02:00
Marius CiepluchandClaude Opus 4.8 84509fbdc0 merge(search-with-proof): path oracle covers prefix-check-bypass class [skip ci]
Plant a sibling canary so prefix-bypass path traversals (startswith without separator)
are demonstrable; parent canary + differential unchanged; glance still demonstrates.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 11:24:31 +02:00
Marius CiepluchandClaude Opus 4.8 6c7d6bb1d9 search-with-proof: path oracle covers the prefix-check-bypass class (sibling canary)
The path oracle only modelled classic parent traversal (a `canary` file in the parent of
${served_root}). A common real path-traversal class is the prefix-check bypass:
startswith(base) without a trailing separator, which the fix closes with base + os.sep;
it reaches a SIBLING whose path string-extends the served directory, not the parent.
PathOracle now also plants a sibling canary `public_leak` beside the served `public`
(same nonce, 0600), so a prefix-bypass demo leaks it on the vulnerable side and the fix
blocks it. ORACLE_GUIDE names the second target and tells the agent to pick the vector
its code shows. One shared nonce stays correct: a canary leaks only when the demo's own
request reaches it, and each demo targets one vector whose fix blocks it on the fixed side.

observe()/capture()/the differential/SideRecord are UNCHANGED. Local glance ceiling proof
still demonstrates (parent canary, 3/3 vuln, 0/3 fixed, control on both sides). Gate:
worker+control+verify suites 106 passed / 4 socket-skips; ruff clean; mypy clean on oracles.py.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 11:24:31 +02:00
Marius CiepluchandClaude Opus 4.8 4c356caa21 measurements: py re-validation 2 -- served-root fix works (demo now wires ${served_root}); still inconclusive, budget-bound [skip ci]
Served-root guidance validated: authored demo now references ${served_root}. Still
inconclusive (0 false proofs); agent spread budget across vectors and hit run_ceiling
(164 calls, $1.47/$1.50) without converging, and the committed traversal did not
differential. Remaining = per-app exploit-authoring + convergence (the layer the $0 local
glance proof did by hand).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 11:05:07 +02:00
Marius CiepluchandClaude Opus 4.8 ec09a39deb merge(search-with-proof): path demo must serve ${served_root} [skip ci]
Agent guidance: point the app's served directory at the ${served_root} placeholder via
arg/env/config so the planted observable is reachable. Prompt strings only; 37 worker tests pass.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 10:29:40 +02:00
Marius CiepluchandClaude Opus 4.8 00d1c26f5f search-with-proof: require the path demo to serve ${served_root} (via arg/env/config)
Path searches authored and verified many demos but all came back with no differential:
the demo never pointed the app's served directory at ${served_root}, so the observable
(which lives relative to ${served_root}) was outside the served tree and unreachable. The
agent had set an app env var to a value of its own; the guide only showed the --dir form.

ORACLE_GUIDE now states the agent MUST set the app's served/root to the ${served_root}
placeholder via whichever mechanism the app exposes (command-line argument, environment
variable, or config written to start.files), not only --dir, and that serving elsewhere
makes the observable unreachable. Verify-time-only clarification, observable names, and all
prohibitions are kept. Prompt strings only.

Tests: tests/test_search_worker.py 37 passed; ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 10:29:40 +02:00
Marius CiepluchandClaude Opus 4.8 e387ca16a2 measurements: py re-validation -- budget fix works (18 verify tasks, 0 before); demo didn't wire served_root [skip ci]
Fair-share budget validated: the Python agent completed the full loop and verified 18
demos within $0.74 of $1.50 (0 verify tasks before). Still inconclusive, 0 false proofs.
Root cause of no-differential: the authored demo never references ${served_root}; the
agent pointed the app's served dir at its own value, so the observable was outside the
served tree. Next: guidance to wire the served dir to ${served_root} for the path oracle.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 09:23:33 +02:00
Marius CiepluchandClaude Opus 4.8 f3ce5be8c4 merge(search-with-proof): per-search fair-share pilot budget [skip ci]
Each pilot search gets a fair share of --ceiling-usd with settled rollover, so one
expensive search can no longer starve the others; total stays within the ceiling.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 08:35:24 +02:00
Marius CiepluchandClaude Opus 4.8 2c0c45fd1f search-with-proof: per-search fair-share pilot budget (stop one search starving the rest)
--ceiling-usd was one shared SpendBudget passed to every search, so an expensive
search drained the pool and starved later ones: in the last 2-pair/$2 run the first
search spent $1.60 and the second hit the ceiling at $0.34 after finishing its analysis.
The driver now allocates a dynamic fair share per search (remaining_ceiling /
searches_not_yet_started), rolls unused settled budget forward, caps each search at its
share, and keeps total settled <= ceiling. A search now gets room to author and verify
after it finds the path.

Driver-only change (benchmarks/search/pilot_run.py) plus tests. tests/test_search_pilot_run.py
39 passed; ruff clean; mypy unchanged (pre-existing benchmarks errors only).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 08:35:24 +02:00
Marius CiepluchandClaude Opus 4.8 e01fb513f2 measurements: validation of explore-trap fix (ts now authors a demo; py analysis complete but out of budget) [skip ci]
2 path searches (ts/py), vulnerable, cluster, $1.95 of $2. The trap fix is validated
behaviourally: ts pair 9 now authors a demo (verify_tasks 0->2; fails could_not_build on
entrypoint), py pair 13 drops 118->36 calls and its board shows a complete route-to-sink
+ entrypoint trace but hits the per-run budget ceiling before authoring/verifying. Still
0 demonstrations, 0 false proofs. Payload-free aggregate; raw stays gitignored. Next
levers: per-run budget/step allocation for demo-write+verify after analysis; demo
entrypoint/build (start.path) for compiled/served apps.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 08:28:37 +02:00
Marius CiepluchandClaude Opus 4.8 e8e6637833 merge(search-with-proof): verify-time-only observable guidance (fix explore-time reachability trap) [skip ci]
Agent guidance so path searches stop trying to confirm the oracle observable during
exploration (where it is absent) and instead author a demo for the differential verifier.
Prompt strings only; validators untouched; 37 worker tests pass.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 07:43:29 +02:00
Marius CiepluchandClaude Opus 4.8 563d006c77 search-with-proof: tell the agent the oracle observable is verify-time-only (fix the explore-time reachability trap)
The parity fix named each oracle's observable, but path searches then spent their
whole budget without authoring a demo: the agent tried to confirm reachability during
exploration, where the oracle fixture does not exist (it is injected only in the
verifier's sandbox at verify time), concluded "unreachable", and never committed a demo.

ORACLE_GUIDE now states the fixture/observable is injected at VERIFY time only and is
absent during exploration; its explore-time absence never means unreachable or justifies
ending the search. The agent confirms the mechanism (sink reachability, access to data the
app should not expose) from app code and its own sandbox test data, targets the observable's
known verify-time location in the demo, and relies on differential verification as the only
proof. All prior prohibitions (no out-of-band/off-disk access, no input capture, no revision
sniffing) and the parity observable names are kept. Adjacent wording shortened to fit.

Tests: tests/test_search_worker.py 37 passed; ruff clean; mypy unchanged (pre-existing
verify.py lambda note only). Prompt strings only; validators untouched.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-07 07:43:13 +02:00
Marius CiepluchandClaude Opus 4.8 7c220307d4 measurements: path-family pilot after keystone + target-parity (0 demos, parity validated) [skip ci]
3 path searches (php/ts/py), vulnerable side, cluster verify lane, $2.95 of $3.
Payload-free aggregate; raw boards/demos stay gitignored. Finding: the parity fix
produced directed, target-aware search (the agent reasons about reaching the named
observable and inspects the app's serving guard), but no demonstration landed -- all
three spent their per-pair allowance before a differential, and two wrote zero verify
demos (budget consumed in reason/explore). Next levers: allowance/budget split toward
demo-writing, entrypoint discovery for php, and the explore-vs-verify path layout the
py agent flagged.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-06 19:35:42 +02:00
Marius CiepluchandClaude Opus 4.8 dc307c7d2e search-with-proof: path-oracle agent guidance to parity with command/ssrf (name the observable)
The path family's agent guidance named no target, while command (${marker_dir}/marker)
and ssrf (${callback_url}) disclose theirs, so path searches could not author a demo
that surfaces the observable through the app. ORACLE_GUIDE now names it (canary in the
parent of ${served_root}, matching PathOracle.prepare). The reason/explore constraints
are refined to forbid only out-of-band access (direct disk reads, non-app channels),
input capture, and revision sniffing, while allowing the app to disclose it through the
vulnerability under test. Prompt strings only; the demo validator and its _PRIVATE guard
are untouched, so direct references to verifier paths remain blocked.

Tests: tests/test_search_worker.py 37 passed; ruff clean; mypy unchanged (pre-existing
verify.py lambda note only).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-06 18:06:00 +02:00
Marius CiepluchandClaude Fable 5.1 70553cb7c6 measurements: first search-with-proof demonstrated (glance ceiling proof, $0 local) [skip ci]
Outcome-only record of the first end-to-end demonstrated verdict and the two
masking defects (regenerated npm lock pinning an incompatible mime; agent demo
targeting the wrong file). Payloads stay in the gitignored private area.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 17:54:47 +02:00
Marius CiepluchandClaude Fable 5.1 de2771997a merge(search-with-proof): positive control for path oracle [skip ci]
Honest serve signal for the HTTP verifier: SideRecord.control_observed records
whether a benign verifier-planted in-root marker came back through the app's
interface, as its own guarded app start that never changes the verdict. A
2xx-based block was rejected (any HTTP status counts as alive). Gate green;
ceiling re-proof on glance still demonstrated with control_observed on both
sides. The search task image needs a rebuild before the next cluster pilot.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 17:54:47 +02:00
Marius CiepluchandClaude Opus 4.8 328406ad41 search-with-proof: positive control for path oracle — honest serve signal, no false negatives
The HTTP verifier could not distinguish a mis-wired or non-serving app from a
genuine no-effect: both yielded `inconclusive`. This adds a positive control for
the path oracle.

- PathOracle plants a benign in-root marker file (random token from os.urandom,
  distinct from the secret canary) and exposes it via a new Oracle.control()
  hook (default None; only the path oracle implements it today).
- On the first run of an HTTP side, _side runs a benign GET for that marker as
  its own fully-guarded app start. The marker returning proves the interface
  serves the fixture; its absence — or any control failure — is recorded in the
  new SideRecord.control_observed field but never changes the verdict, since a
  valid app may answer only its own routes. A crash is still caught by the
  existing could_not_run path.

A 2xx-based "nothing served" block was rejected: the verifier deliberately
treats any HTTP status as the app being alive (a 503-only app is a valid
observed side), so status-based blocking would false-fail valid apps. Soundly
converting a mis-wired-but-responding app into could_not_run requires a
demo-declared control, which is a separate follow-up.

Tests: new tests/test_search_control.py covers marker observed, absent marker
not blocking, control failure not blocking a working attack, and crash still
could_not_run. Full search-verify/probe/worker suites pass; ruff clean; mypy
unchanged (only the pre-existing file_bindings lambda note).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-06 17:07:24 +02:00
Marius CiepluchandClaude Opus 4.8 d17ceb0e0c spec(search-with-proof): handoff -- plumbing cleared, open question is demo attack step vs oracle capture [skip ci]
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-06 15:14:45 +02:00
Marius CiepluchandClaude Fable 5.1 918b1f5eec tests: a timed-out AX attempt may carry its reason instead of None [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 14:51:07 +02:00
Marius CiepluchandClaude Fable 5.1 af6a68691c merge(search-with-proof): demo replay tool [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 14:50:14 +02:00
Marius CiepluchandClaude Fable 5.1 eb05ed18da search-with-proof: no-model demo replay (executor + verifier on a stored demo)
Replays a stored demo through the executor task and the verifier lane for one
side or both, with per-phase timings, so verifier changes are checked on the
exact image before any paid search.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 14:50:14 +02:00
Marius CiepluchandClaude Fable 5.1 cc2d0b3d1f merge(search-with-proof): one interpreter, sys.path root, node_modules path [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 14:40:15 +02:00
Marius CiepluchandClaude Fable 5.1 630603ee28 search-with-proof: one interpreter for pip and apps; checkout root on sys.path; real node_modules path
The pip recipe ran under the system Python 3.13 while apps started under the
venv 3.12, so compiled extensions never loaded. Both now use sys.executable.
Package-style starts see the checkout root; NODE_PATH lists the project
node_modules under the checkout before the products mount.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 14:37:58 +02:00
Marius CiepluchandClaude Fable 5.1 b227e74da8 merge(search-with-proof): oracle placeholders and venv interpreter [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 12:43:11 +02:00
Marius CiepluchandClaude Fable 5.1 7a91cc1482 search-with-proof: oracle placeholders for demos; Python apps run under the venv interpreter
Demos may reference ${served_root}, ${database_path}, ${marker_dir} or
${callback_url}; the verifier substitutes them per side, so apps that take a
directory or database argument can be wired to the oracle. Python apps start
under the verifier interpreter (the one pip built the products for): the image
carries a second system Python whose ABI tag broke compiled extensions.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 12:43:11 +02:00
Marius CiepluchandClaude Fable 5.1 dcbaec0043 merge(search-with-proof): archive parts, isolated pip, verifier module paths [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 11:15:09 +02:00
Marius CiepluchandClaude Fable 5.1 0d3e6df31c search-with-proof: split compressed verifier archives, isolated pip installs, verifier module paths
Archives are gzipped and uploaded in parts of at most 90 MB with per-part
digests (edge request limit); pip installs ignore the task venv so transitive
dependencies land in products; the verifier sets NODE_PATH and the Java
classpath from products when starting the app.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 11:15:09 +02:00
Marius Ciepluch c9bd1ec152 merge(search-with-proof): readiness probe, longer tool turns, 40-step tasks [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>

# Conflicts:
#	benchmarks/ax/search_verify.py
#	benchmarks/search/pilot_run.py
2026-10-06 09:41:57 +02:00
Marius CiepluchandClaude Fable 5.1 da804472b9 search-with-proof: readiness from the budget with a probe, 4096-token tool turns, 40-step tasks
HTTP start waits up to 90 s (CLI 30 s) with TCP then HTTP probing; app stdout
and stderr tails are recorded; tool-call turns get 4096 output tokens with one
retry on truncation; tasks get 40 steps with reminders.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 09:37:47 +02:00
Marius CiepluchandClaude Fable 5.1 a4ec526039 tests: verify task identities are plain ids; the prefix is the workload kind [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 08:21:37 +02:00
Marius CiepluchandClaude Fable 5.1 b1024b8daa ax: verify and probe tasks carry the search-verify/search-probe name prefixes [skip ci]
The kind default "pin" had produced ousast-engine-pin-search-verify-* names,
which the operator per-host egress rules do not match.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 08:20:40 +02:00
Marius CiepluchandClaude Fable 5.1 72ab4c8925 search-with-proof: 1 GiB spooled verifier archive, tolerant text replies, transport retries
Built projects spool to /workspace without package caches (cap 1 GiB, both
sides); a plain-text model reply is fed back once; mailbox PUT/GET retry for up
to 90 s with idempotent re-send by seq; executor tasks are deleted on every exit
path.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 08:18:34 +02:00
Marius CiepluchandClaude Fable 5.1 aebc834667 merge(search-with-proof): archive, text replies, transport retries [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 08:18:34 +02:00
Marius CiepluchandClaude Fable 5.1 dae05a7c1a merge(search-with-proof): demo startup configuration and loop fixes [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 00:25:57 +02:00
Marius Ciepluch 30cbc1ac76 Merge branch 'main' into search-with-proof 2026-10-06 00:24:26 +02:00
Marius CiepluchandClaude Fable 5.1 ceeaa66717 search-with-proof: startup environment and data files, dead-end dedupe, task_spend continues
Demos may set a bounded environment (loader variables denied) and up to 8 small
data files under .demo/; the reason step sees conclusions and may not re-propose
them; a task spend limit concludes its intent, not the search; finish wins over
stray tool calls.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 00:24:26 +02:00
Marius CiepluchandClaude Fable 5.1 263be30979 search-with-proof 3.2: oracle probe passes on kube-ax in task-boundary mode [skip ci]
All five oracles demonstrated, gaming demos inconclusive, 38 s, ~500 MB RSS.
The probe dispatcher accepts a skipped isolation check in task-boundary mode.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-06 00:14:51 +02:00
Marius CiepluchandClaude Fable 5.1 81cb02cd35 merge(search-with-proof): build recipes and stderr-bearing failures [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 21:39:39 +02:00
Marius Ciepluch 67eb629c16 Merge branch 'main' into search-with-proof 2026-10-05 21:37:56 +02:00
Marius CiepluchandClaude Fable 5.1 3866e2771b search-with-proof: ecosystem-aware build recipes and stderr in build failures
The agent passes a manifest file or project directory; pip uses -r for
requirements files and the directory for pyproject/setup, npm ci/install runs in
the directory, composer/maven/gradle likewise. Build and verify failures carry
phase, exit code and a sanitised stderr tail.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 21:37:56 +02:00
Marius CiepluchandClaude Fable 5.1 13a20008e7 merge(probe-task-boundary): probe runs oracle self-tests in task-boundary mode [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 21:17:00 +02:00
Marius Ciepluch 007415c151 Merge branch 'main' into probe-task-boundary
# Conflicts:
#	src/openultrasast/search/probe.py
2026-10-05 21:14:55 +02:00
Marius CiepluchandClaude Fable 5.1 2d3c94e33b search probe: task-boundary mode runs the oracle self-tests in-process
On kube-ax the probe detected task-boundary and then failed the bwrap preflight it
no longer needs; now it skips the preflight, runs every oracle pair through the
verifier path a task uses, and reports ok only if every real demo is
demonstrated and no gaming demo is. Fixes a double fixture-path translation.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 21:11:26 +02:00
Marius CiepluchandClaude Fable 5.1 a40122d1b5 merge(search-with-proof): oracle mapping, build memory, tolerant tool calls [skip ci]
Image rebuild dispatched by hand (code in the last layer, ~10 MB).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 21:09:08 +02:00
Marius CiepluchandClaude Fable 5.1 bf243e6680 search-with-proof: family-to-oracle mapping, build memory, tolerant tool calls
Demos declare their oracle within the family allowed set (injection: sql or
command; output_encoding: xss; untrusted_destination: ssrf; path: path);
builds and app starts use the search memory budget instead of the 256 MB tool
limit (npm died with SIGTRAP); malformed tool-call JSON is fed back once.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 21:07:58 +02:00
Marius CiepluchandClaude Fable 5.1 0750c6fdd7 merge(search-with-proof): named exhaustion limits and partial evidence [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 21:01:31 +02:00
Marius Ciepluch c190d34f60 Merge branch 'main' into search-with-proof
# Conflicts:
#	tests/test_search_task_storage.py
2026-10-05 20:58:10 +02:00
Marius CiepluchandClaude Fable 5.1 c33b053b15 search-with-proof: name the exhausted limit, keep partial evidence, tighter reservation
budget_spent carries end_detail (spend, task_spend, reason_rounds, ...); a task
whose allowance cannot cover the next call finishes with its facts; per-task
metrics in pilot records; input reservation bytes/2 + 1024 per message (measured
3-4x over-reservation before).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 20:57:56 +02:00
Marius CiepluchandClaude Fable 5.1 35f942f4c8 images: pin task base b696c08 (base and browser) by digest
Triggers the one intended rebuild of engine, replay and search on the shared base.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 20:50:16 +02:00
Marius Ciepluch b696c08446 merge(image-layering): shared task base and code-last layering [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>

# Conflicts:
#	tests/test_search_task_storage.py
2026-10-05 20:44:02 +02:00
Marius CiepluchandClaude Fable 5.1 94d6a7dd11 search executor: NPROC bound counts threads, as the kernel does [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 20:44:02 +02:00
Marius CiepluchandClaude Fable 5.1 1b9d1ca85c images: shared task base, code-last layers, no per-push rebuilds [skip ci]
One base (Temurin 21 JDK, node/npm/tsc, php/composer, maven, sqlite, bwrap)
built only when its Dockerfile or dependencies change, plus a browser variant;
engine/search/VM images FROM base@sha256 with the code in the last layer; the
image workflow runs on dispatch, tags, Dockerfile or pin changes, with a build
cache. Executor NPROC bound relative to the current process count.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 20:40:13 +02:00
Marius CiepluchandClaude Fable 5.1 509c6562fe search budget: recalibrated caps after the first measured searches [skip ci]
Settled ~$0.003 per model call; worst-case reservations exhausted the $0.05 task
allowance after 2-4 calls. Now $0.40 per task, $2 per search (maintainer decision).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 20:35:27 +02:00
Marius CiepluchandClaude Fable 5.1 b44fb56d8d ax batch: workload-kind prefixes in task names (pass-, replay-) [skip ci]
The operator keys per-host egress rules on the task-name prefix; engine passes and
replays had none. Names stay within 49 bytes.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 20:33:35 +02:00
Marius Ciepluch daf95b16a1 merge(search-with-proof): cluster-unavailable handling [skip ci]
No image rebuild: VM-side changes only (operator asked to stop per-merge rebuilds).

Co-Authored-By: Claude Fable 5.1 <[email protected]>

# Conflicts:
#	benchmarks/ax/batch.py
2026-10-05 20:27:01 +02:00
Marius CiepluchandClaude Fable 5.1 4be2063d40 search-with-proof: cluster-unavailable detection, kept messages, run-level abort
Zero running workers behind "no free workers available" now raises
ClusterUnavailable instead of waiting out the capacity deadline; CapacityDeadline
carries its message; cleanup failures no longer mask the original exception; the
pilot driver aborts the run on infrastructure failure before any model call.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 20:26:37 +02:00
Marius CiepluchandClaude Opus 5.5 7fb061fb69 merge(search-with-proof): search contracts
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 19:18:12 +02:00
Marius CiepluchandClaude Opus 5.5 e39a4882b0 search-with-proof: bootstrap intent, real reason/explore contracts, visible refusals
First live run: the reason step got ~190 tokens and returned {} -> silent
no_evidence. Now: a bootstrap intent reads the candidate and its entry point; the
reason prompt states the goal, family oracles, demo schema and must propose
intents when none are open; empty replies retry once then end as reason_refused;
failure records keep messages.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 19:18:12 +02:00
Marius CiepluchandClaude Opus 5.5 0fb098c478 merge(search-with-proof): pilot driver
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 18:58:24 +02:00
Marius CiepluchandClaude Opus 5.5 87b82564b4 search-with-proof: pilot driver (6.2)
One side per run over selected pilot pairs: board, brain on the VM with the
project DeepSeek client, executor tasks on kube-ax, verification on the VM lane
(AX verify lane switchable). Hard run ceiling with reservation before every call,
per-search and step caps; fixed side never sees the vulnerable revision; records
carry digests only. Dry run with the stub model reaches demonstrated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 18:58:24 +02:00
Marius CiepluchandClaude Opus 5.5 89516d4000 ax batch: fast capacity retries in the first minute (golden-snapshot window)
Operator 2026-10-05: Substrate boots a ~1 s golden-snapshot actor per new task on the
other worker; retry apply after 2-5 s for the first minute, then 15-30 s.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 18:07:47 +02:00
Marius CiepluchandClaude Opus 5.5 941fbb5fb0 search-with-proof: diagnosable failures, pip install race, no duplicate tasks
Reasons carry type plus a sanitised message tail; command failures carry phase,
exit code and bounded stderr. Scratch walk tolerates directories pip removes
mid-walk; writable /workspace/home and runtime paths. Capacity retries delete a
possibly created task before resubmitting. Docker smoke checks the image first.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 17:58:27 +02:00
Marius CiepluchandClaude Opus 5.5 7c1af9f518 merge(search-with-proof): diagnosable failures and dispatch fixes
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 17:58:27 +02:00
Marius CiepluchandClaude Opus 5.5 cf10f38589 search-task image: clear the base image GPG_KEY fingerprint
The official Python base sets GPG_KEY (public CPython signing fingerprint); the
executor rightly refused it as a credential-like name on kube-ax.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 17:42:03 +02:00
Marius CiepluchandClaude Opus 5.5 75f2134eb4 search executor: name the credential-like variables it refuses (names only)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 17:36:12 +02:00
Marius CiepluchandClaude Opus 5.5 1d67412e2b search-with-proof: executor smoke driver and pilot selection (6.1)
20 development pairs, one per repository, 4 per language, 5 per family; identities
in a gitignored manifest, counts and digest committed. Executor smoke drives one
executor task end to end without a model and fails loudly on an unread clone.
kube-ax operator constraints as steering.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 17:26:06 +02:00
Marius CiepluchandClaude Opus 5.5 1bfb729422 merge(search-with-proof): executor smoke and pilot selection
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 17:26:06 +02:00
Marius CiepluchandClaude Opus 5.5 85f4f0b3d4 search-with-proof: declarative demos, brain/executor split, kube-ax constraints
Demos are data the verifier executes (fixed build recipes, runtime+path start,
HTTP/CLI steps with captures; the canary is never referable). The agent loop with
the key never runs repository code: executor tasks without a key exchange commands
and results through presigned objects. AX capacity errors are back-pressure with at
most 2 tasks in flight; caches and builds under /workspace with a 2 GiB guard;
verifiers take one presigned archive per side. Launcher fix: private /dev in bwrap.
190 tests on the host; probe 5/5 demonstrated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 17:12:46 +02:00
Marius CiepluchandClaude Opus 5.5 ccd64757c8 merge(search-with-proof): trust-domain split and operator constraints
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 17:12:46 +02:00
Marius CiepluchandClaude Opus 5.5 721c4b0997 spec(search-with-proof): separate trust domains (brain with key, executor with repo code)
Replaces the accepted key-exposure exception: the key never shares a sandbox with
repository code; executor AX tasks exchange commands and results via presigned
objects. Firecracker executors only if the pilot build-failure rate requires it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 16:26:19 +02:00
Marius CiepluchandClaude Opus 5.5 5e03bdf07f spec(search-with-proof): gVisor is the cluster boundary; declarative demos; capped rotated key
kube-ax probe: AX tasks have no capabilities (no namespaces, no setuid), so no
in-task isolation. Demos become data the verifier executes; the explore key risk is
accepted and bounded by cap and rotation.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 16:20:26 +02:00
Marius CiepluchandClaude Opus 5.5 e5ce630edc merge(search-with-proof): gVisor isolation mode
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 15:00:25 +02:00
Marius CiepluchandClaude Opus 5.5 03857c9bec search-with-proof: root-no-userns isolation mode for gVisor
The kube-ax probe showed bwrap cannot map uids in gVisor as an unprivileged user.
As root (gVisor is the outer boundary) the sandbox uses pid/ipc/uts/net/mount
namespaces without a user namespace and drops the artefact to its own uid via
setpriv; the mode is recorded. The probe reports each namespace kind separately.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 15:00:25 +02:00
Marius CiepluchandClaude Opus 5.5 e6871fef9a merge(search-with-proof): probe harness and agent worker
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 14:45:48 +02:00
Marius CiepluchandClaude Opus 5.5 0f74767db9 search-with-proof: probe harness (3.2) and agent worker (group 5)
Probe runs each oracle toy pair with a real demo and two gaming demos, records
image facts, RSS and disk, fails loudly if isolation is unavailable; dispatchable
as an AX task. Worker: reason step without tools, explore tool loop whose run tool
goes through the sandbox with a scrubbed env (model key absent), spend reserved
before each call. Host probe: 5/5 real demos demonstrated, 10/10 gaming demos
inconclusive. 89 tests.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 14:45:48 +02:00
Marius CiepluchandClaude Opus 5.5 a17a6afaf8 merge(search-with-proof): group 4 and the search-task image
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 14:26:43 +02:00
Marius CiepluchandClaude Opus 5.5 4b38239f68 search-with-proof group 4 + image: board, spend reservation, coordinator, search-task image
Board as immutable versioned blobs (coordinator sole writer); budget reserve/settle
before model calls; per-task and run caps; coordinator loop over an injected
executor; batch.py accepts a fixed allow-list of search variables; XSS browser
oracle; ousast-search-task image (Python, Node/TypeScript, PHP, Java 17, sqlite,
bwrap, chromium) built by CI. 103 tests on the host.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 14:26:43 +02:00
Marius CiepluchandClaude Opus 5.5 6394ee473d merge(search-with-proof): group 2 verifier and oracles
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 14:09:48 +02:00
Marius CiepluchandClaude Opus 5.5 742469d671 search-with-proof group 2: verifier with owned oracles and differential
The verifier, not the artefact, observes the effect: SQL, path, command and SSRF
oracles owned by the verifier (XSS needs a browser executor, else no_oracle);
artefact exit code and output ignored; immutable checkout and dependencies;
3 runs per side; one-sided build or readiness failure is inconclusive. An
isolation failure raises IsolationUnavailable instead of reading as
could_not_build (the NPROC limit now applies inside the user namespace).
35 tests, including gaming attempts and 300 host processes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 14:09:48 +02:00
Marius CiepluchandClaude Opus 5.5 c74b5eaeb0 spec(learned-decision-engine): engine becomes triage where a demonstration oracle exists
Req 6.5 (maintainer decision 2026-10-05, with search-with-proof): BLOCK comes from
verified demonstrations; the engine is judged by top-K recall per push and advisory
false alarms; the pooled block gate governs only families without an oracle.
Task 17 adds the metric and re-scores existing arms at no cost.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 13:45:10 +02:00
Marius CiepluchandClaude Opus 5.5 9d70ef9257 spec(search-with-proof): requirements and design approved; tasks for probe and pilot
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 13:35:55 +02:00
Marius CiepluchandClaude Opus 5.5 c3820e184c spec(search-with-proof): requirements amended after two Codex reviews; design v2
False alarms count every visible finding; the verifier owns per-family oracles;
inconclusive on one-sided build failure; feasibility pilot separated from a
73-negative BLOCK qualification; coverage ceilings; spend reserved before calls;
key isolated from executed code; cluster model calls supersede the pool VM-only
rule for this arm only. Requirements approval reopened.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 13:35:18 +02:00
Marius CiepluchandClaude Opus 5.5 8bbea86cf8 spec(search-with-proof): requirements approved
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 12:01:39 +02:00
Marius CiepluchandClaude Opus 5.5 45e641f25c spec(search-with-proof): brief and requirements for review
After reviewing Cairn: detection so far classified excerpts and hit a precision
wall; this adds directed search with an objective goal (a re-verified
demonstration), a fact/intent board, and reason/explore steps as AX Tasks on
kube-ax. Design only learns from Cairn (AGPL); no code copied.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 12:01:07 +02:00
Marius CiepluchandClaude Opus 5.5 6d27c9bfcf unseen-eval: calibrated replay deadline + native agreement check, docker-lane diagnostics, proof 4.4 record; engine runner cleans up fetched sources
gVisor replays exhausted the hook's 30 s deadline in the snapshot step on a
1,725-file repository; cluster replays now use 30 s x a measured slowdown
factor, verified on a native sample. The engine runner deletes each pin's fetched
source (1.8 GB had accumulated), caps the cache at 500 MB and refuses below
1.5 GB free. Proof 4.4: two engine pins via the generic dispatcher match the VM
baseline exactly. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 08:14:57 +02:00
Marius CiepluchandClaude Opus 5.5 5dc86c2f3f merge(unseen-eval): groups 1-2 and 4-6 -- used set, sourcing with resume, AX batch dispatcher, replay task, scorer
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 20:28:15 +02:00
Marius CiepluchandClaude Opus 5.5 c6e87a8517 unseen-eval groups 4-6: generic AX batch dispatcher (engine_trace on it), replay task and image stage, scorer with adoption gate
Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 20:19:14 +02:00
Marius CiepluchandClaude Opus 5.5 48ae918dec unseen-eval group 2 (2.1-2.4): advisory sourcing, vulnerability-introducing ranges, ordinary-commit draw, draft manifest and slices; journal with resume
Implemented with Codex. Sourcing run (2.5) next, by the coordinator.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 19:51:30 +02:00
Marius CiepluchandClaude Opus 5.5 8213d3baad unseen-eval group 1: used set (1,029 repositories, every source read), extended guard, usage ledger; record 01
Implemented with Codex; task 1.5 run by the coordinator against the real
sources (S3 memory read-only).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 19:17:35 +02:00
Marius CiepluchandClaude Opus 5.5 60512fff88 spec(unseen-repo-evaluation): design and tasks approved; run through the baseline
Decisions: long-deadline arm (300 s), adoption needs no unseen regression
(Req 5.4), permissive repositories by name only, two AX lanes. Also adds the
trained-classifier and joint-optimisation arms to learned-decision-engine.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 18:48:13 +02:00
Marius CiepluchandClaude Opus 5.5 d4467f4236 steering: roadmap re-centred on generalisation to unseen repositories (G0-G5); specs updated; unseen-repo-evaluation brief and requirements (approved)
Maintainer 2026-10-04: the safety net is for unseen code and the A/B tests exist
to prove the approach generalises; infrastructure work had drifted from that.
Runs are designed for AX on kube-ax, kind is the fallback.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 17:34:57 +02:00
Marius CiepluchandClaude Opus 5.5 ee422b63e0 spec(learned-decision-engine): fix-mechanisms plan approved -- discharge facts first, >=2 proposing repos, per language
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 17:25:35 +02:00
Marius CiepluchandClaude Opus 5.5 02b2cbf99c engine: retry a pin once when its AX sandbox dies, then hand it to the VM lane
Two engine actors died at gVisor level during a 12-pin run (AX phase Failed, no
result). Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 17:14:20 +02:00
Marius CiepluchandClaude Opus 5.5 eaf436d3db merge(engine): one resident Joern session per pin; ActiveProcessorCount=2 on the cluster
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 16:50:21 +02:00
Marius CiepluchandClaude Opus 5.5 187e3c14c5 engine: one resident Joern session per pin (frontend + one query JVM instead of four), ActiveProcessorCount=2 on the cluster
Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 16:41:22 +02:00
Marius CiepluchandClaude Opus 5.5 b715ff7d0e engine: stage presigned objects as .dat; task entry reports the failing step and HTTP status
Through Cloudflare, signed requests for keys ending .tar/.gz/.zip/.bin reached
RustFS altered and failed with 403 (cf-cache-status BYPASS); the same signed
request straight to RustFS on the LAN returned 200. A Cloudflare cache-bypass
rule on files.because-security.com fixes it (all extensions 200, DYNAMIC); the
engine executors keep .dat as a safeguard, the plane keeps output.tar.
Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 16:18:01 +02:00
Marius CiepluchandClaude Opus 5.5 035710493c merge(engine): AX-task executor for the kube-ax cluster
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 15:30:26 +02:00
Marius CiepluchandClaude Opus 5.5 abca40b072 engine: AX-task executor for kube-ax (engine-task image FROM ax-task-runner v0.3.1, presigned HTTPS I/O, ousast-engine- names, 60 s egress-ready retry, 1.7 GB cluster heap, VM OOM fallback)
Implemented with Codex. Live smoke pending the operator egress watcher.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 15:22:44 +02:00
Marius CiepluchandClaude Opus 5.5 afff080874 merge(engine): Kubernetes engine workers (Deployment from the image, S3 queue); leak-audit records carry digests, not repository names
The Joern leak-audit record named spent population v1/v2 repositories, which
the reserved-repository guard forbids outside benchmarks/independent/ (no v3
repository was involved: 0 matches). Records now carry counts and SHA-256
digests. With Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 14:39:36 +02:00
Marius CiepluchandClaude Opus 5.5 e174c06ee7 engine: worker Deployment from the image (S3 queue with leases, dispatcher --executor queue, engine-image workflow, scoped RustFS policy); merge main
Implemented with Codex. Cluster deploy pending the k3s kubeconfig and the scoped account.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 14:23:04 +02:00
Marius CiepluchandClaude Opus 5.5 8d6921d2df spec(learned-decision-engine): fix-mechanisms plan for review -- learn discharge facts from 710 fix diffs, admitted by held-out pair separation
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 14:11:41 +02:00
Marius CiepluchandClaude Opus 5.5 298bd17780 learn: Joern-slice leak audit on the completed trace run -- no leak; paths differ between pair sides in 2 of 105 both-asked pairs
Coverage 72 path, 231 asked-nothing, 9 failed, 189 unsupported. The engine
almost never distinguishes a vulnerable function from its fix, so a Joern-slice
prompt experiment is not funded; the lever is the engine's knowledge of fix
mechanisms.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 14:08:26 +02:00
Marius CiepluchandClaude Opus 5.5 6a2138fbef merge(ops): lean kind profile -- observability off by default on the server VM
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 12:13:32 +02:00
Marius CiepluchandClaude Opus 5.5 35ded61c85 merge(learn): harvest 3 -- 251 real fix pairs (advisory-fixes-3); complete pairs in memory 459 -> 710
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 12:06:16 +02:00
Marius CiepluchandClaude Opus 5.5 0d78d33fd2 ops(kind): skip Substrate's observability stack by default (930 MiB of 2.3 GB); opt in with OUSAST_KIND_OBSERVABILITY=1
A lean overlay omits otel-collector, prometheus and jaeger and disables the
exporters; on an existing cluster up.sh scales them to 0. Verified live: render
has no otel-system resources, up.sh idempotent (54 s second run), smoke Run
done, kind node 2.66 -> 1.82 GiB. Docs say server VM. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 12:03:19 +02:00
Marius CiepluchandClaude Opus 5.5 cbc252553a learn: 251 more advisory-fix pairs whose function survives the fix (advisory-fixes-3)
Batch 3 of real (vulnerable function, fixed function) pointer pairs, mined at
zero model cost from reviewed GHSA advisories (pip, npm, maven, composer,
published since 2023, newest first). Same rules as batch 2, widened three
ways: a merged PR's single commit when the advisory links no commit, commits
up to 6 files / 240 lines, and two-family advisories with the family read
from the diff. Every candidate's function diff was read: 412 reviewed, 255
kept, 3 one-per-repository, 2 withheld by the reserved-population guard
(bisected with output discarded). Permissive licences only; no repository
named elsewhere under benchmarks/; no WordPress plugin.

Kept 251: injection 56, untrusted_destination 26, path 71, output_encoding
69, access_control 20, deserialization 9; python 83, typescript 48, php 46,
java 37, javascript 37. Only 12 post-cutoff: batch 2 had used up the
post-2026-06-01 pool. Memory 1133 -> 1635 static examples; complete pairs in
the whole memory 459 -> 710. Embeddings: 15 calls, $0.0030. No chat-model
call, no Docker, no Joern. Record:
benchmarks/measurements/2026-10-04-decision-engine-harvest-3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 11:57:34 +02:00
Marius Ciepluch e555ba68dc Merge branch 'joern-leak-audit' 2026-10-04 10:53:23 +02:00
Marius CiepluchandClaude Opus 5.5 231dca1f04 merge: taint-query null-guard fix and trace-enabled engine runner
The query fix stops a class-level call from aborting the whole taint batch on
real PHP code; gate, map_gate and pair_gate outputs are unchanged. Trace parsing
lives with the runner (benchmarks/learn/engine_trace_parse.py).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 10:53:23 +02:00
Marius CiepluchandClaude Opus 5.5 2e6ba93aee learn: Kubernetes executor for the engine trace runner (Jobs, presigned source/result via the S3 store, mixed lanes, --k8s-check)
Implemented with Codex; live cluster check pending.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 10:35:59 +02:00
Marius CiepluchandClaude Opus 5.5 431ab1e738 learn: engine_trace materialises catalog pairs without commits (dedented parseable excerpts, in-function scope), protects labelled files in exports, scopes large repos, records frontend diagnostics, retries the jssrc2cpg engine bug, writes progress.json
Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 08:19:46 +02:00
Marius CiepluchandClaude Opus 5.5 94ec4cef16 learn: Joern-slice leak audit (has-path, steps, sanitizer, guard, sink, asked; both-asked pairs; per-corpus AUC)
Implemented with Codex. The record is written after the trace re-run completes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 23:03:52 +02:00
Marius CiepluchandClaude Opus 5.5 e492ebd16e fix(cpg): null-guard method access in the query scripts -- a class-level call aborted the whole taint batch
On real PHP source (CPS-IT/mailqueue) a call outside any method has a null
.method; inScope/nestedInLabeled dereferenced it (NullPointerException) and the
script exited 1, losing every taint answer for the repository. 16 sites in
taint/dominance/config/vocabulary.sc now treat a null method as out of scope;
non-null behaviour is unchanged (same outputs on the repro graph for the other
scripts). Static regression test added. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 22:53:28 +02:00
Marius CiepluchandClaude Opus 5.5 41f56170bb learn: engine_trace worker renders request params through the production _render; records query stderr
The worker sent tuple params as JSON arrays; taint.sc expects comma-joined
strings (ujson InvalidData, exit 1). Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 22:41:54 +02:00
Marius CiepluchandClaude Opus 5.5 fe2a424018 learn: engine_trace fetches real source at the pinned commit, 2.5 GB JVM heap, one CPG per language
Root causes of the 50 failed pins in trace run 1: advisory pairs were fed as
excerpts (invalid Java/JS fragments), the JVM defaulted to ~768 MB heap in a
3 GB container, and mixed-language pins were refused. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 22:36:23 +02:00
Marius CiepluchandClaude Opus 5.5 36a3355b81 learn: engine_trace guard judges frontends by their CPG, query steps by time and question markers
php2cpg legitimately finishes in ~2 s on small files. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 19:22:37 +02:00
Marius CiepluchandClaude Opus 5.5 34d00391f1 learn: engine_trace container worker imports only the engine; dry-run writes nothing; resume keyed on selection
The 2026-09-14 engine image has no PyYAML or boto3; the worker no longer imports
learn.examples. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 19:19:22 +02:00
Marius CiepluchandClaude Opus 5.5 19c936343b learn: trace-enabled Joern run over labelled units (opt-in trace in taint.sc, trace parsing, resumable runner)
Implemented with Codex. Default scan queries do not request a trace.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 19:10:47 +02:00
Marius CiepluchandClaude Opus 5.5 20777d8863 chore(audit): module audit gains learn.slice
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 17:04:33 +02:00
Marius Ciepluch 77e47ac7d3 Merge branch 'graph-slice' 2026-10-03 16:58:37 +02:00
Marius CiepluchandClaude Opus 5.5 094f6d226c learn: exp-005 graph slice -- inconclusive (within-pair AUC 0.762 -> 0.746, CI [-0.063, 0.032]); unsure rate 0.29 -> 0.24; fill \$1.59
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 16:58:37 +02:00
Marius CiepluchandClaude Opus 5.5 0acbc7ec41 learn: exp-005 graph slice -- slice in the prompt (opt-in), program-arm fill, manifest and units
Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 15:06:08 +02:00
Marius CiepluchandClaude Opus 5.5 1b2c7993a7 learn: in-function def-use slice from the tree-sitter layer + leak audit (\$0)
slice.py renders source -> step -> sink lines with fixed wording for flow,
no-flow and not-analysable; the prompt option is default-off (byte-identical
default prompt). semantic IR: JS/C declarations count as assignments, augmented
assignment keeps its target (gate, map_gate, pair_gate outputs unchanged).
Audit over exp-004's 501 units: 59 flow, 161 no-flow, 281 not analysable;
has-flow within-pair accuracy 0.52, pooled AUC 0.55 -- no leak. Fixes a
numbering bug that rejected every excerpt with a blank line. With Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 14:56:52 +02:00
Marius CiepluchandClaude Opus 5.5 463fe1503a learn: build_features reads v2 _a/_b scans and TypeScript repository-scan witnesses; --dry-run
Population v2 scans are named <case>--<side>_a/_b.json; the script looked only
for <case>--<side>.json and recorded those examples as engine not-run. The two
repeats combine symmetrically for both sides. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 14:41:55 +02:00
Marius CiepluchandClaude Opus 5.5 2836b78819 merge(learn): exp-004 -- inconclusive, pooled top-30 precision 0.90; experiment fill command; registration fix
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 14:29:22 +02:00
Marius CiepluchandClaude Opus 5.5 6d9850578e learn: dry-run estimate from measured cached-response cost; docs: exp-004 result; tidy instrument check
The estimator priced all input at the full rate (\$5.58 estimated vs \$1.24
spent); it now uses the family mean billed cost of cached responses when usage
is recorded. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 14:23:36 +02:00
Marius CiepluchandClaude Opus 5.5 b50f7afccb learn: exp-004 run -- inconclusive, no pooled t_block; fill \$1.24 of a \$2 ceiling, replay \$0; instrument check recorded
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 14:18:17 +02:00
Marius CiepluchandClaude Opus 5.5 90a2642019 learn: registered() returns the frozen registration, not the first row
A pending-then-frozen registration left two rows and rows[0] could be the stale
pending one, so fill and run refused a properly frozen experiment. Implemented
with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 12:42:39 +02:00
Marius CiepluchandClaude Opus 5.5 8aeca972c0 learn: freeze exp-004 units (501 units, 390 paired, 220 groups) after the negatives harvest; audit gains learn.fill
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 12:36:02 +02:00
Marius CiepluchandClaude Opus 5.5 8d14b0c13f learn: experiment fill -- fill a rule experiment's response cache with a hard ceiling, dry-run estimate
Implemented with Codex; review fix: the bounded client keeps the configured
thinking setting so a fill answers as the run would.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 12:35:27 +02:00
Marius CiepluchandClaude Opus 5.5 212a391872 merge(learn): cache fill and exp-004 registration -- pooled block gate with family floor, Req 6.4
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 09:53:10 +02:00
Marius CiepluchandClaude Opus 5.5 da91ef0dd9 exp-003: supplementary all-units replay after the cache fill; exp-004 registered
Supplementary analysis, not a new decision: exp-003's verdict stands.
The 142 units excluded for an uncached response were filled with the same
six incumbent programs, parameters and request keys, on exp-003's
937-example memory (the harvest's 196 rows and its 2026-10-03 embeddings
left out; a zero-cost replay first reproduced exp-003 exactly: 211
decided, 1056 replayed, $0.582876). Fill: MeteredClient, ceiling $0.99
under the approved $1.00, 697 client calls, $0.331759, no provider error.

Replay on all 353 units (244 paired, 146 groups): 0 client calls, $0.0,
1768 responses replayed; no arm flags any unit, the registered rule reads
inconclusive. Calibration holds only for untrusted destination (ECE
0.0454) and arm A still finds no BLOCK point there.
benchmarks/experiments/exp-003-precision-bound-blocking/result-complete-cache.json

exp-004 registered in s3://sast-memory (manifest 87fceb65..., pin
2ad9c1d, units pending): benchmarks/experiments/exp-004-pooled-block-gate/
registration.json. docs/decision-engine.md: both, briefly.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 09:47:20 +02:00
Marius CiepluchandClaude Opus 5.5 2ad9c1db7a learn(exp-004): Req 6.4 pooled BLOCK gate with a family floor; manifest before units
Maintainer decision 2026-10-03: BLOCK precision is shown pooled across
families. Req 6.4 (amended; 7.4 annotated): one t_block on a per-family
Platt-normalised score (calibrate.fit_map, training folds, inner
cross-fit), Wilson 95% LB >= 0.95 over all families' training flags; a
family blocks only with >= F = 10 held-out flags and zero errors, else
advisory; t_quiet per family keeps >= 90% of training paired positives.

evaluate: normalised, pooled_threshold, quiet_threshold,
pooled_block_gate, family_floor, apply_floor. experiments: rule
pooled_block_gate (normalise: platt, floor, recall), decide_pooled on the
pre-floor pooled held-out LB, floor table, quiet_recall and after-floor
totals in analyse_rule; exp-003's analysis is unchanged.

plane/experiments/exp-004-pooled-block-gate.yaml: procedure and decision
rule registered now; units frozen at run time from the post-harvest
memory (units_status pending until then).

Tests (tests/test_learn_rules.py): pooled threshold where no family alone
reaches the bound, no held-out label moves a threshold or the map, a
family with one held-out error cannot block, t_quiet and no blind
silencing, the pooled verdict, the exp-004 manifest.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 09:40:58 +02:00
Marius CiepluchandClaude Opus 5.5 5b2bd9bb91 merge(learn): negatives harvest -- 98 real fix pairs with surviving functions (79 post-cutoff), paired groups 151 -> 249
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:29:32 +02:00
Marius CiepluchandClaude Opus 5.5 50c93edfb8 learn: 98 advisory-fix pairs whose function survives the fix (advisory-fixes-2)
Real (vulnerable function, fixed function) pointer pairs for the six compiled
families, mined at zero model cost from reviewed GHSA advisories (pip, npm,
maven, composer), newest first: 79 of 98 published after 2026-06-01
(post_cutoff = true). Each pair: one linked one-parent fix commit, <= 4 source
files / 120 lines, the function holding the changed code lines declared once on
both sides with a changed body; every function diff read and kept only when the
named function holds the fix (145 reviewed, 100 kept); 2 dropped by the
reserved-population test; permissive licences only, no repository named
elsewhere under benchmarks/.

New label source advisory-fixes-2; the advisory-fix tests run over both
batches. Memory 937 -> 1133 static examples (98 paired groups added);
positives-only groups stay 99 (vibe-py/dev-php/v2 data gaps), their share of
positive groups in the six families 0.40 -> 0.29. Embeddings: 7 calls,
$0.0014. No chat-model call. Record:
benchmarks/measurements/2026-10-03-decision-engine-negatives-harvest.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:23:42 +02:00
Marius CiepluchandClaude Opus 5.5 6ecf0a304e merge(learn): exp-003 precision-bound blocking -- inconclusive; the Wilson bound needs 73 clean flags, injection has 38 paired positives
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:21:27 +02:00
Marius CiepluchandClaude Opus 5.5 b31c85678f exp-003: result -- inconclusive, no BLOCK rule flags a held-out unit
Zero-cost replay of the six slice-2 programs: meter 0 client calls, $0.0;
211 of 353 units scored, 142 excluded for an uncached response (path lost
all 39). Arm A (calibrated) fails calibration (ECE 0.077 access control,
0.1067 injection); arm B (Wilson LB >= 0.95, nested) chooses no threshold on
any fold, as the registration's power note predicted (73 clean flags needed,
38 paired injection positives); secondary arm C flags nothing either. The
instrument check shows real scores; the zero is the rule's.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:15:22 +02:00
Marius CiepluchandClaude Opus 5.5 7bb97ad0bc exp-003: registration, before the run
Registered in s3://sast-memory at dd7ebc1: manifest 5917f9a1, units 338de998
(353 units, 146 groups). Primary metric, decision rule and the threshold
selection are fixed in the manifest; the power note says arm B cannot reach
a Wilson LB of 0.95 on this data (73 clean flags needed, 38 paired
positives at most).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:03:22 +02:00
Marius CiepluchandClaude Opus 5.5 dd7ebc119b learn(exp-003): decision-rule experiments and the precision-bound BLOCK arm
kind: rule experiments score once from the response cache behind a
zero-ceiling meter (a miss is excluded and counted, never asked) and apply
each arm's BLOCK rule to the same held-out scores. Arm B chooses a per-family
raw-score threshold on the other outer folds only (lowest t with Wilson 95%
LB >= 0.95); a test fails if a fold's own labels move its threshold.
Manifest and frozen units (353 units, 146 groups, 244 paired) for exp-003.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 08:02:51 +02:00
Marius CiepluchandClaude Opus 5.5 b24eeb15f3 merge(plane-on-kubernetes): groups 1-2 -- cluster profile, no host literals, delivery through the S3 store
Kind proofs: records 01 and 02. Task 2.6 (receiver removal) is paused while the
safety net and the decision engine come first. deployment.md conflicts resolved
in the plain-language style; corrects the RuntimeClass claim (Substrate runs
runsc inside its worker pod).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 07:57:13 +02:00
Marius CiepluchandClaude Opus 5.5 c8b418c34c plane(2.5): kind proof and record 02 -- artifacts through the store, one TLS rule
Live on kind with the runner image that carries 2.2
(ousast-runner@sha256:d2a55293...; templates and validation-46 Tasks repinned,
128 lines; kind-images.json updated): the smoke Run on store delivery finished
in 26 s and 24 s; the tar was PUT to runs/ax-e2e-smoke/facts/output.tar
(10240 B) and collected 2 s later; read back from the store its members
(facts.json 451 B, summary.json 238 B, units.jsonl 291 B) equal the extracted
files; facts.json, summary.json and done.json (149 B) re-put, state.json
(350 B) mirrored; a rerun from an empty results root seeded state.json from the
store and finished in 2 s without resubmitting. The actor's egress policy read
back live: one tlsPassthrough rule on 443 naming the store host, nothing else.
`ousast plane status ax-e2e-smoke` printed the table each time. Record:
benchmarks/measurements/2026-10-02-k8s-02-store-delivery/record.json with the
smoke outputs, store read-back and policy beside it. Task 2.6 (receiver
removal) is paused by the maintainer's reprioritisation.

Gates: pytest 2031 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0. (The first run, at 0.97 GiB free disk, failed 11 tests and errored 5
on the FileStore's 1 GiB free-space refusal; rerun after the disk was freed.)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 07:45:19 +02:00
Marius CiepluchandClaude Opus 5.5 432fd2947e merge(pre-push): operational readiness increment 1 -- never block, never hide, quick tier, new files, background engine, install
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-03 07:29:17 +02:00
Marius CiepluchandClaude Opus 5.5 2ec8071c26 pre-push(17.7): prove it on the audit's three repositories
Rerun records, serve-static and unrolled/secure (same planted commits) with
this build installed from its wheel: 6 replays and 10 hook pushes, all exit 0
in 0.69-1.21 s against the 30 s deadline, input read proven per run. The
planted JS eval in a new file surfaces in the quick tier on the host and, in
one Docker engine run (190.9 s, deadline 300), as two advisory engine findings
classified new/file_added_in_head. The planted Python %-formatted SQL is still
missed (no quick rule, no host engine): recorded as a coverage gap, no rule
edited. Go is named as not covered. The run caught a wrong install hint in
the hook (the package is not on PyPI); corrected here.
Record: benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 23:02:32 +02:00
Marius CiepluchandClaude Opus 5.5 591818d5fc pre-push(17.6): install, Docker wrapper and first-run docs
`ousast pre-push install|uninstall [REPO] [--force] [--docker]` ships in the
wheel with the hook as package data; it honours core.hooksPath, refuses an
existing hook unless --force (which chains it as pre-push.before-ousast), and
uninstall restores the chained hook. ops/ousast-docker is a hook-callable
script: the repository read-only and the artifact directory at their own
paths, stdin passed through, no network, --memory 3g, the GHCR 2.0.1 image by
default, advisory exit unless blocking. Help text for every pre-push flag; a
first-run guide, settings, skip-reason glossary, CI recipe and full --help in
docs/scanning.md; README tiers 1 and 2 pull the published image. The image
was not rebuilt: 1.1 GB free, below the 4 GB threshold. Requirements 7.4, 9.6.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 22:51:22 +02:00
Marius CiepluchandClaude Opus 5.5 66357ffea1 pre-push(17.5): say when the engine cannot finish, and offer it in the background
--engine inline|background|off (hook: OUSAST_PUSH_ENGINE). background returns
after the quick tier and starts one detached explicit replay of the same base
and head (push/background.py), one at a time per artifact directory; its
artifact lands under <artifact dir>/engine/<repo key>/<head>.json and the next
run prints "Engine result for <head> from the previous push" once. A deadline
miss says so in one line naming the deadline and the background option. No
engine speed work. Requirement 9.5.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 22:31:14 +02:00
Marius CiepluchandClaude Opus 5.5 511de4b75f pre-push(17.4): a file new in head is new, not comparison unknown
ChangeContext records the paths Git reports as added. An entailed,
undischarged operation in an added file, with complete head evidence and no
identical operation in any base answer, is classified new
(file_added_in_head) instead of base_incomplete; admission accepts that reason
as change evidence. Unpaired add/delete pushes and every other unknown
comparison keep their vetoes. An engine finding whose novelty is established
and whose only vetoes are capability vetoes is shown as an advisory engine
finding, not an alert. Requirement 9.4.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 22:24:24 +02:00
Marius CiepluchandClaude Opus 5.5 15999a09e1 pre-push(17.3): quick rules as the fast first tier
Before the engine and inside the same deadline, the shipped quick rules run
over the head side of the changed files (push/quick.py) and keep matches on
changed lines or in changed functions. They print as advisory pattern matches,
labeled apart from admitted alerts, are kept in the artifact's quick_tier with
the files and bytes read, and never admit or block. A new branch with no
remote base runs the quick tier over its last commit and says so. The planted
eval in serve-static's new debug.js now surfaces; the planted Python
%-formatted SQL in records does not (a quick-rule coverage gap, no rule
edited). Requirement 9.3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 22:18:14 +02:00
Marius CiepluchandClaude Opus 5.5 f555dee640 pre-push(17.2): name every skipped check in plain words
One terminal line per distinct coverage reason (push/explain.py): hex-encoded
paths decoded, per-question sha256 tokens dropped and counted, unknown reasons
still printed; the JSON artifact keeps the raw values. A changed language with
no engine frontend and no quick rules is recorded and printed as "language not
covered: <lang> (...)", in pre-push and in `ousast scan`. The scan report
renders the eight degradations cli.py recorded but never showed, and any
reason without a sentence. The hook forwards OUSAST_COMPARISON_BASE.
Requirement 9.2.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 22:10:33 +02:00
Marius CiepluchandClaude Opus 5.5 bf07cd4d1d pre-push(17.1): never block a push the user did not choose to block
The installed hook exits 0 with one line when ousast (or OUSAST_COMMAND) is
not on PATH, still chaining a prior hook. In Git hook mode without an explicit
--mode blocking, argument/usage errors (an invalid OUSAST_PUSH_DEADLINE, a cache
directory inside the repository, a refused experimental flag) and unexpected
exceptions print one "check did not run" line and exit 0. Blocking mode keeps
its usage errors. Requirement 9.1.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:57:42 +02:00
Marius CiepluchandClaude Fable 5.1 605b7af5ba plane(2.4): reconciler on Delivery -- the start body carries the URLs, the store completes the task
_execute builds the task's `delivery {put, inputs}` from Delivery, passes it to
router.starter, grants the store's host in the egress policy, awaits
Delivery.delivered (the tar's exact key in the store), then collect()s:
download, extract with the member checks, re-put the declared artifacts, write
done.json; the outcome is the collected summary.json. Three findings from the
tests: a rerun of an unfinished task completed at once on the previous
attempt's tar, so Delivery.reset(task) deletes output.tar and done.json before
every attempt; a worker thread that raised left the run blocked on its queue,
so the worker now turns any exception into a failed task; a consumer may
declare a producer's summary.json without the producer listing it (the loop
Run does), so declared(run) publishes outputs[] plus every artifact a consumer
names. The receiver thread survives behind OUSAST_DELIVERY=receiver for this
group only (no presigned URLs, the http rule, OUSAST_ARTIFACT_URL in the env);
the default path renders no receiver address into any Task. The fake ax PUTs
to the start body's URL and GETs its inputs from theirs; the reconciler tests
run through the S3-shaped fake store and assert the objects, done.json, the
seeded rerun and the receiver mode. reconciler.py: 549 lines (bound 550).

Gates: pytest 2031 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:49:58 +02:00
Marius CiepluchandClaude Opus 5.5 a656fda2f4 pre-push(17): operational-readiness task group and Requirement 9
Append task group 17 "Operational readiness (2026-10-02)" with one task per
maintainer-approved item (never block, name every skip, quick tier, new-file
classification, engine honesty and background mode, install/Docker/docs, proof),
add Requirement 9 (9.1-9.6, marked added 2026-10-02, maintainer-approved), and
check 13.4 and the parent of 14 as the audit found them done.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:46:25 +02:00
Marius CiepluchandClaude Fable 5.1 c49eae4f30 plane(2.3): egress -- one TLS rule over store host, Git hosts and Model hosts; no http rule
policy_for(task, workspaces, model, store_host, receiver_host=None) emits one
tlsPassthrough rule on 443 over the sorted union of the memory store's host,
the bound Workspaces' https Git hosts and the bound Model's egress hosts. There
is no http rule: an http Git repo is refused by name (it could not be reached),
a store given as an address is refused naming S3_ENDPOINT (the gateway allows
hostnames only). receiver_host adds the old receiver's http rule for the path
group 2 keeps until 2.6; the reconciler passes it by keyword (passing the
receiver's address positionally would have raised outside the worker's try and
hung the run on the queue -- 2.4 hardens the worker). Tests: the store-only
policy, the Model-bound policy, the two refusals, the receiver exception, and
every committed template under plane/tasks/ rendering exactly the store plus,
for verify and roles, the provider host.

Gates: pytest 2027 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:29:32 +02:00
Marius CiepluchandClaude Opus 5.5 49e11e0252 docs(readme): one paragraph for the published site and local preview
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:17:44 +02:00
Marius CiepluchandClaude Opus 5.5 f0fcb6686f docs: publish the MkDocs site to GitHub Pages
A Docs workflow builds the site strictly with the docs extra and deploys it to
https://norandom.github.io/OpenUltraSAST/ on pushes to main that touch the docs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:16:49 +02:00
Marius CiepluchandClaude Fable 5.1 9506a18b56 plane(2.2): runner -- delivery {put, inputs} in the start body, PUT to the store
StartGate.offer validates an optional `delivery` beside the credentials: https
URLs to a hostname (never an address, no userinfo; AX_RUNNER_INSECURE_DELIVERY=1
admits http in unit tests only), names ^[A-Z][A-Z0-9_]*$, a 400 that names the
field and echoes no URL. The runner PUTs its output tar to delivery.put and
GETs each declared input from delivery.inputs[NAME] (deliver() gains a method;
log lines and DeliveryError carry the URL without its query, so no presigned
signature reaches a log, a marker or a summary). OUSAST_ARTIFACT_URL/DIAL stay
honoured until task 2.6 so both paths run in the same build; the completion
marker and the retry-only boot are unchanged, and a re-start brings a fresh
URL for the retry. Router.start_task carries `delivery` in the body and
redacts its URLs like credential values. Tests: the gate's acceptance and
refusals, an end-to-end PUT/GET against the S3-shaped fake with no URL in any
log or file, the failed-PUT-then-retry-only boot, deliver() with PUT.

Gates: pytest 2024 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:12:19 +02:00
Marius CiepluchandClaude Fable 5.1 35d8032c0a plane(2.1): Delivery -- presigned store URLs, collection, state mirrored to the store
src/openultrasast/plane/delivery.py: Delivery(store, run, base) mints a presigned
PUT for runs/<run>/<task>/output.tar and a GET per declared input
(runs/<run>/<producer>/<artifact>), expiry OUSAST_TASK_TIMEOUT + 600 s;
delivered() asks the store for the exact key; collect() downloads, extracts
under the run directory with the receiver's member checks (no symlink, hard
link or escape; nothing written on a rejection), re-puts the declared outputs
as single objects and writes done.json; put_input() for inputs no task
produces; save_state()/load_state() mirror state.json. The reconciler opens the
store first (no fallback; the CLI reports a store that cannot open and submits
nothing), mirrors state.json after every change and seeds a rerun with no local
state from the store (the 2.1 bullet's state mirroring is what gives the new
module its importer). The reconciler crossed 500 lines here (517) rather than at
2.4 as 1.6 projected: the bound moves to 550 with the reason in the assertion,
as 1.6 decided. tests/plane_fake_store.py is the S3-shaped fake (objects in a
dict, presigned URLs on a local server); tests/test_plane_delivery.py covers the
body, expiry, polling, rejection, declared-only re-put, state seeding and that
no URL reaches a log. Module-audit manifest regenerated (148/5, no orphan).

Gates: pytest 2020 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:05:48 +02:00
Marius Ciepluch 9833571e94 Merge branch 'worktree-agent-af2e4a098d650d0a2' 2026-10-02 21:03:41 +02:00
Marius Ciepluch c217d1d1d3 Merge branch 'worktree-agent-aed844222b4750ed7' 2026-10-02 21:03:41 +02:00
Marius CiepluchandClaude Opus 5.5 352f8c07aa docs: plain-language pass -- detection techniques, architecture, evaluation, decision engine
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:03:24 +02:00
Marius CiepluchandClaude Opus 5.5 552671ca6a docs: plain-language pass -- README, index, where we stand, scanning, examples, threat model
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:01:10 +02:00
Marius CiepluchandClaude Opus 5.5 7e23379e96 docs: plain-language pass -- deployment, plane, memory, RustFS, token economics
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 21:00:37 +02:00
Marius CiepluchandClaude Fable 5.1 79296bb4a4 plane(1.8): kind proof and record 01 -- profile, literals, doctor, probe, bound
benchmarks/measurements/2026-10-02-k8s-01-profile/record.json with the live
evidence of group 1: `ousast plane doctor --profile kind` passes every line
(exit 0, 1.4 s; the store check reaches the bucket: verify 0.62 s, probe object
24 B); `--profile k3s` fails on the unreachable context and on the images file
the release has not published yet (two expected lines; the namespaces are not
checked behind an unreachable context); the literal scan has 0 offenders (23
before 1.3) and fails naming workspaces.py:37 when a literal is reinserted;
up.sh reran idempotently in 38 s and the smoke Run finished in 24 s on the
rebuilt image (facts.json 451 B, summary.json 238 B); the placement probe says
`any`; the reconciler bound numbers of 1.6. Doctor, literal, smoke and up.sh
outputs beside it.

Gates: pytest full suite on this tree's src/ and tests/ content: 2010 passed,
42 skipped, exit 0 (the 1.5 run; the 1.8 run reported two module-audit
failures caused by the uncommitted 2.1 module present in the working tree);
ruff check 0; ruff format 0; mypy 0; mkdocs build --strict 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:55:57 +02:00
Marius CiepluchandClaude Fable 5.1 2cca03c1c0 plane(1.7): gvisor tarball reachability documented as planned (manual, k3s)
docs/deployment.md B.1: Substrate's ateom-gvisor worker runs runsc from the
release tarball named in manifests/ate-install/sandboxconfig-gvisor.yaml
(gs://gvisor/releases/nightly/2026-09-02/<arch>/gvisor.tar.zstd, sha256-pinned
per architecture), fetched by the atelet on the node; the node must reach it or
a copied SandboxConfig must name a mirror with the same sha256. The check is the
atelet's fetch line in the worker pod's logs, run by hand on the day k3s exists;
labelled planned until then (design open question 3 stays open).

Gates: docs only; mkdocs build --strict passed with this paragraph present in
the 1.4 gate; no file a test or linter reads changed since the 1.5 gate.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:50:14 +02:00
Marius CiepluchandClaude Fable 5.1 2afa20e2ea plane(1.6): reconciler bound recorded -- 489 lines, bound moves to 550 in 2.4
reconciler.py: 477 lines at 6bd44ca, 489 after 1.2 (the profile parameter of
run() and its docstring). Task 2.4 adds the Delivery wiring (about +25) while
the receiver path is kept for group 2, and 2.6 removes the receiver wiring
(about -20): projected 514 during group 2, 494 after 2.6. The estimate crosses
500 while both paths coexist, so 2.4 moves tests/test_plane_reconciler.py's
bound to 550 with the reason in the assertion message; 2.6 records the final
count. Numbers in benchmarks/measurements/2026-10-02-k8s-01-profile/record.json
(record 01, drafted here, completed by 1.8).

Gates: no file a test or linter reads changed since the 1.5 gate (pytest 2010
passed, 42 skipped, exit 0; ruff 0; format 0; mypy 0).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:49:45 +02:00
Marius CiepluchandClaude Fable 5.1 86ec237483 plane(1.5): placement probe -- Substrate places by any free worker, not by fit
ops/ax/probe-placement.sh scales ousast-pool to one worker, applies probe-pool
(1 x 2 CPU / 3Gi, the already-built ateom-gvisor image, no ko build), runs one
model-free repo-facts Task requesting 2560Mi (limit 3Gi) and one requesting
256Mi, records `kubectl ate get workers -o json` after each resume, and removes
everything it created. Live on kind: both actors landed on the 1536Mi
ousast-pool worker (status.allocated.actors=1 there, 0 on probe-pool) ->
placement: any. design.md section 4 records the outcome and the fallback group
5 must take (an engine atespace with engine-pool as its only pool if Substrate
scopes pools by atespace; else a plain Kubernetes Job on k3s). Evidence under
benchmarks/measurements/2026-10-02-k8s-01-profile/probe/.

Gates (tree identical to 1.4 in src/ and tests/; ops/ gains the probe script,
which the literal scan covers): pytest 2010 passed, 42 skipped, exit 0; ruff
check 0; ruff format 0; mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:49:03 +02:00
Marius CiepluchandClaude Fable 5.1 8099b29deb plane(1.4): up.sh by profile -- context, registry, pool and images file from kind.toml
ops/ax/up.sh reads kube_context, registry, pools.default and the images path
with `python -m openultrasast.plane.profile --print`, writes the runner digest to
ops/k8s/profiles/kind-images.json instead of ~/.cache/.../runner-image, renders
ops/ax/workerpool.yaml.tmpl from the profile's pool (POOL_NAME/REPLICAS/CPU/
MEMORY, WORKER_IMAGE), and keeps an existing pool and an ax CLI newer than its
checkout instead of rebuilding them (ko and go install would refill several GB
of Go cache on this disk). smoke-run.sh takes the runner from `--print
images.runner` and runs the checkout's code with OUSAST_PYTHON. repin_templates
reads the JSON images file (runner key; the one-line pin still reads). `--print
images` returns the file path (the first live rerun fell over this). Templates
and the generated validation-46 Tasks re-pinned to the rebuilt image (128
lines, one digest).

Live: up.sh reran idempotently on kind in 38 s (every step kept except the
runner build); smoke-run.sh done in 24 s on the new image, facts.json 451 B,
summary.json 238 B delivered (record: benchmarks/measurements/2026-10-02-k8s-01-profile/).

Gates: pytest 2010 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0; mkdocs build --strict 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:43:40 +02:00
Marius CiepluchandClaude Fable 5.1 36f5927999 plane(1.3): host-literal scan test over src, plane, ops and docs
tests/test_plane_literals.py scans src/, plane/, ops/ (minus ops/ax/up.sh and
ops/k8s/profiles/) and docs/ for the kind context, the kind registry, the kind
network's gateway, a loopback address with a port and port-forward; router.py
keeps the last two, a line tagged ax-tunnel or loopback is allowed, and a
committed template's digest-pinned image is accepted only while it names the
kind profile's registry (checked by a third test). A self-check reinserts each
literal into a fixture and sees it caught. The runner's health URL and the Joern
session's query URL are tagged loopback; docs/deployment.md and ops/ax/README.md
stop naming this host and describe the kube context and the registry as the
profile's (what 1.2 made configurable).

Gates: pytest 2009 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0; mkdocs build --strict 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:32:51 +02:00
Marius CiepluchandClaude Fable 5.1 77a03bad61 plane(1.2): literals out of doctor, egress, router, generate -- the profile is handed in
doctor(profile) checks the profile's kube context, the two namespaces behind a
reachable context only, the images file's runner reference in the profile's
registry (a local registry over its v2 API, a remote one via docker manifest
inspect) and opens the memory store; its first line reports every configured
address (Req 1.2). Egress(executable, context), open_router(context, url=...),
receiver_cluster_ip(context) and Receiver(..., context) take the profile's
context and build none; generate.increment/render and harvest take the profile's
atespace. `ousast plane doctor|run|workspaces|harvest --profile`. The test fakes
(kubectl-ate, kubectl, ax) record the context they received and the tests assert
it is the profile's. grep "kind-" src/openultrasast/plane: no match.

Gates: pytest 2006 passed, 42 skipped, exit 0; ruff check 0; ruff format 0; mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:26:46 +02:00
Marius CiepluchandClaude Opus 5.5 50f4e69321 release: v2.0.1 -- Apache-2.0 licence, GHCR images, CI green, front-door README
v2.0.0 was tagged but its release job stopped at the format check, so no release
or package was published; v2.0.1 is the first published 2.x release.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:18:39 +02:00
Marius CiepluchandClaude Fable 5.1 8eac1b2c69 plane(1.1): PlaneProfile -- one TOML per cluster, env-overridable, no secrets
ops/k8s/profiles/kind.toml and k3s.toml hold the values that differ per cluster
(exec, kube_context, registry, images, router_url, ax_server, memory, atespace,
snapshots_bucket, image_pull_secret, pools); kind-images.json carries the digest
pin ops/ax/up.sh wrote. load_profile() reads OUSAST_PLANE_PROFILE (name or path,
default kind), applies OUSAST_<FIELD> and OUSAST_POOL_<NAME>_<FIELD>, and refuses
an unknown key, a credential-shaped key or value, an exec outside local|remote
and an images file without digest pins. `python -m openultrasast.plane.profile
--print FIELD` serves shell scripts. The module-audit manifest is regenerated
for the new module (counts 147/5, no orphan).

Gates: pytest full suite 2002 passed, 1 failed (the module-audit manifest, fixed
by regenerating it; tests/test_module_audit.py re-run exit 0), ruff check 0,
ruff format 0, mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:16:23 +02:00
Marius Ciepluch b1acc865e8 Merge branch 'worktree-agent-a94cfc2aafc87f382' 2026-10-02 20:09:34 +02:00
Marius CiepluchandClaude Opus 5.5 d1aabf0f66 ci: green again -- format and lint the whole tree; push the scan and runner images to GHCR on tags
CI and the release job check `ruff format --check .` and `ruff check .` over the
whole repository, while the local gate checked only src and tests, so eight
benchmark scripts drifted and the v2.0.0 release job stopped before building.
The fixes are formatting, two noqa comments and one renamed loop variable; no
script computes anything differently. The local gate now checks the whole tree.

The release workflow gains an images job: Dockerfile -> ghcr.io/<owner>/openultrasast,
plane/Dockerfile.runner -> ghcr.io/<owner>/openultrasast-runner, tagged with the
version, and the digests written to the job summary.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:09:28 +02:00
Marius CiepluchandClaude Fable 5.1 bdabb978f8 docs(readme): front door -- current commands, status, and links to every docs page
Rewrite README.md as the front door of the docs site: a short "What it is", the
Status (NO-GO against M4, figures quoted from docs/where-we-stand.md), a tiered
install (quick scan without Docker; standard/deep with the openultrasast:dev
image; the plane on ax for maintainers; k3s planned), then one section per
docs page, each linked by relative path so it works on GitHub and on the site.

Stale claims removed: `ousast index` (deleted), `OPENROUTER_EMBEDDING_MODEL`
(no longer read), the duplicated scan-mode, rule and gate tables (now on
docs/scanning.md and docs/evaluation.md), the HarnessX-era wording. Adds a
Licence section stating that no licence file exists yet.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:06:21 +02:00
Marius CiepluchandClaude Fable 5.1 c978de9ff8 merge(cleanup): second legacy inventory -- vector index, equality test, unread config keys, status notes
Also drops the removed ousast index line from the scan skill.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 20:02:53 +02:00
Marius CiepluchandClaude Fable 5.1 61ff5f3c33 docs(kiro): status notes -- ai-service-plane complete, contributor-scan paused, finding-feedback-loop design unapproved; stale lines
Legacy inventory 2, the six STATUS NOTE items (the archive README row landed with the R2 commit).
ai-service-plane: 11 of 11 tasks checked, spec.json phase set to implementation-complete as 5b96199
did for four other specs. contributor-scan: dated note in requirements.md -- paused since 2026-09-23,
27 of 51 tasks, v1.2.0-alpha.1 NO-GO, remaining tasks wait on pre-push M2/M3. finding-feedback-loop:
dated note -- dormant, design unapproved since 2026-09-09, depends on the paused spec. The steering
overview's spec table carries the same three states, its leftover 2026-09-09 module-count line is
removed and the collected-test count is the measured 2,023. docs/memory-and-detection.md names the
archive path of model_sinks, the one stale script name left in docs after 067adb9.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:57:04 +02:00
Marius CiepluchandClaude Fable 5.1 515f94e3c2 docs(where-we-stand): a scan without a provider key records learning_endpoint_unavailable
cli.py appends the degradation when resolve_chat_endpoint returns nothing; the
gap is that report.md has no line for it, not that nothing is recorded.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgMh3zqD1QMwEo
2026-10-02 19:55:22 +02:00
Marius CiepluchandClaude Fable 5.1 6bd44caa50 spec(plane-on-kubernetes): tasks approved; groups 1-2 first
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:54:25 +02:00
Marius Ciepluch 4c1e24ceae Merge branch 'worktree-agent-a630761c8cf5ded05' 2026-10-02 19:54:24 +02:00
Marius Ciepluch ed769e2334 Merge branch 'worktree-agent-a3e8fa755c00b3e20' 2026-10-02 19:53:52 +02:00
Marius CiepluchandClaude Fable 5.1 03c9f745a4 merge(skills): apply the 2026-10-02 review -- current CLI and evidence, merges, removals, steering names
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:53:52 +02:00
Marius CiepluchandClaude Fable 5.1 a097240702 spec(plane-on-kubernetes): tasks for review
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:53:27 +02:00
Marius CiepluchandClaude Fable 5.1 58e788cecb spec(plane-on-kubernetes): design for review
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:48:48 +02:00
Marius CiepluchandClaude Fable 5.1 d0ac8b5f3d docs: detection techniques -- representations, taint, obligations, ranking, the model's role
New page docs/detection-techniques.md: the three representations (regex over the
file, tree-sitter IR, Joern CPG) and what each can see; Joern as a section of its
own (frontends, one graph per language partition, census as the instrument check,
batching against the measured cost curve, deadlines, degradations, where the engine
sits in the plane, cost and misses with records); taint from fact tables to query to
verdict with each refinement traced to its record or commit; obligations, dominance,
constant abstraction, census; regions, ranking ties and budgets; the residual
question and MAX_JUDGED_CANDIDATES; REGRESS verdicts and the unassigned
execution_confirmed rung; an honest table of what each technique misses.

Nav entry under "How it works" after Scanning; one link from the index; one
cross-link line each in scanning.md and architecture.md.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:48:32 +02:00
Marius CiepluchandClaude Fable 5.1 16d9ef7110 chore(skills): apply the 2026-10-02 review -- current CLI and evidence, merges, removals, steering names
openultrasast-scan: evidence ladder as verification.py defines it, with the levels nothing produces today; --mode prerequisites, --fail-on, recorded degradations, run artifacts, ousast mcp; retired [evidence] threshold language removed. openultrasast-triage: reasons verified against calibration.py; deterministic fusion; adjudications as labels, plane verdicts as features, rule changes through improve or a registered experiment; fix-audit merged as a fix-and-re-scan section. kiro: spec-quick and spec-batch removed; validate-gap and validate-design merged into spec-design (--gap, --review; the review gate stays); steering-custom merged into steering; core steering names overview.md, roadmap.md, safety-net.md; instrument rule in debug and verify-completion; repo gates named in validate-impl. AGENTS.md, README skills paragraph and tests/test_agent_skills.py (.agents/skills, two product skills, drift tests) updated.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:48:01 +02:00
Marius Ciepluch 9f58525f0e Merge branch 'worktree-agent-ad1b3451052ec3033' 2026-10-02 19:46:46 +02:00
Marius CiepluchandClaude Fable 5.1 94d2bfcef4 spec(plane-on-kubernetes): remove the last in-cluster reconciler wording
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:46:46 +02:00
Marius CiepluchandClaude Fable 5.1 91a35a4406 spec(plane-on-kubernetes): Req 2/3 -- the reconciler stays the CLI with a remote mode; artifacts land in the store
Maintainer: trigger the checks in kube from the CLI and feed the results back;
execution is remote instead of local, not a separate service.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:46:22 +02:00
Marius CiepluchandClaude Fable 5.1 6db5a35f07 chore(config): retire unread keys [sandbox] network, [sandbox] workspace_readonly, [complexity] top_k
Legacy inventory 2, item R3. The three keys loaded into ResolvedConfig and nothing read them: no
`.network`, `.workspace_readonly` or `.top_k` access exists in src outside config.py. The sandbox
runner always passes `--network none` and `--read-only` and rejects host networking by argv
inspection, so the two sandbox keys were knobs that turned nothing; no stage read top_k, and
`[complexity] max_hunter_hotspots` is the one complexity key the scan reads. They join RETIRED_KEYS
with the same 2026-10-02 mechanism as the other unread keys (RetiredConfigError naming the reason).
The fields and their loaders are removed; memory_mb, timeout_seconds, pids_limit (read by
regress/recipes.py and sandbox/runner.py) and max_hunter_hotspots stay. test_config.py covers the three
retirements and the override test now exercises max_hunter_hotspots. RELEASE_NOTES Unreleased carries
the line.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:45:37 +02:00
Marius CiepluchandClaude Fable 5.1 297f6a8af2 docs(deployment): production topology -- k3s and GHCR target, kind as a first-class profile, the reconciler stays the CLI
Reworks the production topology section after the maintainer's three
decisions of 2026-10-02: the target cluster is k3s (containerd, no Docker
socket, runsc + RuntimeClass for gVisor workers, Traefik) with digest-pinned
images published to GHCR by GitHub Actions; kind stays a first-class local
execution mode (two profiles of one code base, every change proven on kind
first); and the reconciler stays the CLI, with a planned remote mode in which
the same `ousast plane run` submits to ax-server in k3s, tasks deliver their
artifacts to the S3 memory store through presigned URLs, and the CLI polls the
store for the same status and attribution table. The in-cluster reconciler
and the in-cluster receiver become a later option. The diagram is redrawn
without an inbound path; the pre-push section says the hook or CI calls the
CLI in remote mode and either waits with a deadline or reads a later `plane
status`. Every item that is not built stays labelled planned.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:45:27 +02:00
Marius CiepluchandClaude Fable 5.1 ac9c30da0a docs(deployment): production topology on Kubernetes and the pre-push hook at scale
Answers the maintainer's questions on deployment.md: ax is a Kubernetes
application (ax-server Deployment + Redis in ax-system) that schedules every
Task as a gVisor actor on Agent Substrate (ate-system, WorkerPool CRD); there
is one runner image and concurrency equals warm workers, one actor per worker.
Names the three host-bound components (reconciler, receiver, Joern engine) with
their planned in-cluster replacements, the production shape of the pre-push
hook (local deterministic check in the deadline, delta Run on the plane
asynchronously), the estimated per-push cost from the two measured
per-candidate rates, what is not built (no `ousast plane scan`, no diff
candidates task, population-only Workspace generation, no in-cluster
reconciler), and a sizing table from plane/tasks/*.yaml. plane.md gets a
two-line pointer to the section.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:42:36 +02:00
Marius CiepluchandClaude Fable 5.1 adb70b6b98 chore(tests): drop the HarnessX equality test that guards an archived one-off
Legacy inventory 2, item R2. tests/test_harnessx_removal_equality.py loaded
benchmarks/archive/harnessx_removal_equality.py by path and checked that its comparison sees a
difference. The spec is implementation-complete and the proof is the committed pair of records
(2026-09-30-harnessx-removal-{baseline,equality}); the archive README says these scripts are
reference only and a record is re-run at the commit it names, not with today's package. A unit test
that keeps the archived copy importable contradicted that and would break the next time cli or
learn.cli move. The archive README row now says the test was dropped. test_removed_plane_references.py
(the Req 2.4 name search) is a different test and stays.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:39:27 +02:00
Marius CiepluchandClaude Fable 5.1 17ac7ad69d spec(plane-on-kubernetes): kind stays a first-class local execution mode
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:38:03 +02:00
Marius CiepluchandClaude Fable 5.1 85107bfbcc spec(plane-on-kubernetes): requirements approved; target is k3s with images on GitHub (GHCR)
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:37:48 +02:00
Marius CiepluchandClaude Fable 5.1 6187214be3 spec(plane-on-kubernetes): brief and requirements for review
Maintainer: the whole system moves to a Kubernetes cluster soon. Seven host-bound
assumptions become cluster resources or configuration; the engine becomes a
plane task on a larger pool; a delta Run command serves the pre-push use; every
step is proven on kind first.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:36:59 +02:00
Marius CiepluchandClaude Fable 5.1 e5c338070d merge(docs): where we stand -- capability, approach, ax integration, token economics, open items
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:36:23 +02:00
Marius CiepluchandClaude Fable 5.1 e1db7e1920 chore(index): remove the unused vector index, the ousast index subcommand and the [embeddings] section
Legacy inventory 2, item R1. The vector half of index.py (VectorIndex, VectorRecord, RetrievalHit,
RetrievalPackage, the EmbeddingClient protocol, index_reuse_key, build/write/read/query_vector_index,
build_retrieval_package and their helpers) had no importer in src; `ousast index` wrote index/chunks.json
that nothing read, and was the only reader of `[embeddings]`. index.py keeps CodeChunk and
chunk_text_namespace, which skills.py imports; chunk_file/build_code_chunks go with their only caller.

`[embeddings] model` and `[embeddings] store` join RETIRED_KEYS (RetiredConfigError naming the reason),
the same way the [variants] keys were retired; OPENROUTER_EMBEDDING_MODEL is no longer read, so the
README and .env.example drop it (`ousast learn memory embed --model` names the model). test_index.py
now tests the chunker only; test_config.py covers the two retired keys and no longer asserts the
removed field. The module audit record is regenerated (counts unchanged: 151 modules, 0 orphaned;
the index module's importer list shrinks to skills). RELEASE_NOTES Unreleased carries the line.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:32:46 +02:00
Marius CiepluchandClaude Fable 5.1 3344b1fc45 chore(skills): repoint the remaining references to .agents/skills
The previous commit moved the directory but a failed git add left these seven
reference edits unstaged.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:30:48 +02:00
Marius CiepluchandClaude Fable 5.1 5010b15ffc chore(skills): .agents/skills -- the plural directory OpenCode searches by default
Verified against opencode.ai/docs/skills: project-local search paths are
.opencode/skills, .claude/skills and .agents/skills, not configurable.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:24:33 +02:00
Marius CiepluchandClaude Fable 5.1 6b12546531 docs: where we stand -- capability, approach, ax integration, token economics, open items
One page, docs/where-we-stand.md, composed with the COBESY composition method (governing idea,
SCQA opener, MECE sections): the M4 gates and the state against them, the detection approach step
by step, ax on the server, token economics before and after, and the open items. Every figure is
quoted from a committed record whose path stands next to it.

decision-engine.md, evaluation.md and token-ergonomics.md link to the page where they repeated its
content (detail tables kept); decision-engine.md's Experiments section now records exp-002's
result instead of saying it has not run. Linked from the nav, the index and the README.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:18:54 +02:00
Marius CiepluchandClaude Fable 5.1 69b5e0efcc chore(skills): move the project skills from .opencode/skills to .agent/skills
Maintainer decision 2026-10-02 (no symlink). AGENTS.md, README, the two spec
docs, the kiro-spec-batch skill and the provenance ignore list are repointed;
the OpenCode plugin package.json stays under .opencode.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:17:19 +02:00
Marius CiepluchandClaude Fable 5.1 48510fd5c2 bench(learn): exp-002 result -- retrieval ensemble vs temperature sampling
Reject B. 160 paired units (injection 119, access_control 41; 56 repositories),
both arms, two replicates each, no early stop. Canary agreement A 0.875 -> B
0.906 (CI [-0.03, +0.10], spans; B meets 0.90). Within-pair AUC A 0.815 -> B
0.741 (CI [-0.13, -0.02], against B). AUC -0.02 (against), ADVISORY recall
-0.06 and precision +0.01 (span), usd per candidate A $0.0021 -> B $0.0051
(against). Priced usage A $0.68 / B $1.63; balance $16.72 -> $16.04 at the
run's end (lags). Counts only; task 7.4 marked done.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 19:05:34 +02:00
Marius CiepluchandClaude Fable 5.1 780caaa558 fix(learn): the experiment_result row keeps one JSON type per column of its object
An experiment's register, run, result and arm_outcome rows share one object,
and RustFS's S3 Select infers one schema per object: exp-002's result row
carried `units` as an object beside the run row's integer (and `looks` as
objects beside the register row's fractions), and after `analyse --record` no
row of the experiment could be read (JSONParsingError). The result row now
writes `unit_counts` and `looks_taken`; a test asserts one JSON shape per
column across the four row kinds. The live object was repaired by dropping
the one row (drop_rows) and recording the analysis again.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 18:57:56 +02:00
Marius CiepluchandClaude Fable 5.1 8d45483d10 bench(learn): exp-002 registration recorded with the frozen units
The experiment row in s3://sast-memory now carries units_digest 0de66bfa... at
pin 9213e5f (manifest digest unchanged, f5004fb8...). Counts per family and the
two repairs the freeze needed are in the record; no model.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 17:28:55 +02:00
Marius CiepluchandClaude Fable 5.1 9213e5ffd9 bench(learn): exp-002 units frozen and registered
160 units (injection 119 in 34 repositories, 76 paired as 38 candidate pairs;
access_control 41 in 24 repositories, 32 paired as 16 pairs), 56 repositories
in all, under the incumbent folds' seed on the 937-example memory (slice 2
evaluated 110 + 37 on the 904-example memory; the memory grew by the recovered
negatives on 2026-10-02). No model. sha256 0de66bfa3ff844825e06f4f4227a14210d1b77f1cc08c0441c6d23e23fa5d91f.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 17:27:26 +02:00
Marius CiepluchandClaude Fable 5.1 1f525ff540 fix(learn): example rows drop another profile's null features when written and read
Static-profile feature records carry the nine plane-only features (agree.*,
ms.*, verify.*) as null in x; after cb85542 the 937 live example rows loaded,
but the renderer's validate_x refused every one of them ("features outside the
static allow-list"), so no program could prepare a prompt (found while probing
exp-002 model-free). profile_x drops a null feature outside the row's profile
and keeps a valued one, so a real leak still fails. Test added.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 17:24:49 +02:00
Marius CiepluchandClaude Fable 5.1 cb85542578 fix(learn): example rows skip null instrument entries when written and read
Static-profile feature records carry the plane-only instruments as null; every
one of the 937 live example rows was unreadable (TypeError), which blocked
ousast learn evaluate and freezing exp-002 units.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:54:24 +02:00
Marius CiepluchandClaude Fable 5.1 ef1f4daa99 merge(learn): recovered negatives; excerpt matcher covers more declaration forms
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:48:09 +02:00
Marius CiepluchandClaude Fable 5.1 4c8c3ce1c4 merge(learn): A/B experiment runner; exp-001 and exp-002 registered (decision-engine task 7)
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:37:57 +02:00
Marius CiepluchandClaude Fable 5.1 ef6945ec53 bench(learn): labels, features and memory rebuilt with the recovered negatives
Labels regenerated (identical sha: the builder's fixed-side rule already
accepted these pairs), feature records built for the 9 corpus pairs the
matcher now resolves (36 rows; the store's ingest index had remembered them
from a build before the declares() gate and skipped them as a repeat after a
later build dropped them -- build_features now checks presence after an index
hit and restores the rows), static memory rebuilt on RustFS: 904 -> 937
examples (+16 fixed-side negatives, +17 positives: 7 advisory Ruby pairs, 9
corpus pairs, 1 population positive), 9 wrong excerpts corrected (Ruby blocks,
a TS default-value brace), no embedding call (33 new + 9 changed excerpts
without a vector, ~$0.0003 to embed). Positives-only groups 98 -> 99: all
data gaps (94 vibe-py same-snapshot fixed sides, 3 dev-php, 2 population
fixes that did not change the function). Leak audit: pairs 376 -> 385; static
no new flag; plane two new injection flags (agree.final, verify.votes at
0.2292 over 48 pairs), v1 profile unchanged, reported.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:31:30 +02:00
Marius CiepluchandClaude Fable 5.1 9d74839128 fix(learn): excerpt matcher covers more declaration forms
The brace matcher now finds a declaration head in more forms -- name(...) with
any return type, qualifiers, generics or a parameter list spread over many
lines (a paren-aware scan replaces the 8-line lookahead; a prototype's ';' is
not a body, a ')' closing an outer parenthesis marks a call), JS object-literal
methods and property/chained assignments of function expressions and arrows,
$ as an identifier character, Perl 'sub name {', C# expression bodies -- and a
file given as 'other'/'unknown' is sniffed for Perl and Ruby so their def/sub
forms are matched by the right pattern instead of a {} block inside them. The
memory build hands the matcher the label's language where the record's closed
vocabulary language ('other' for Go, Ruby, C#) resolves nothing. One test per
form. Fixed-side negatives still require the fixed side to declare the
function; the 94 vibe-py positives-only groups stay counted.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:12:08 +02:00
Marius CiepluchandClaude Fable 5.1 3b91a52788 bench(learn): exp-001 and exp-002 registered in the memory store, units pending (decision-engine task 7.2)
Registration rows at pin fbc3220 in s3://sast-memory; in-repo records under
benchmarks/experiments/<id>/registration.json. exp-002's units could not be
frozen: every example row in the store carries null plane-only instruments
and learn.examples.load_examples raises on them (unchanged code since
a90e570) -- a store-side change to diagnose before any paid run.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:08:36 +02:00
Marius CiepluchandClaude Fable 5.1 fbc3220bbe feat(learn): A/B experiment runner -- registered manifests, paired arms, bootstrap and McNemar (decision-engine task 7)
learn/experiments.py: YAML manifests under plane/experiments/<id>.yaml,
digest-checked at register, run and analyse time; `experiment`,
`arm_outcome` and `experiment_result` rows; paired arms in a seeded
repository order with a per-unit coin; arm A replay-only at $0; canary
replicates under a fresh cache salt; repository-cluster paired bootstrap,
exact McNemar, two O'Brien-Fleming looks, adopt/reject/equivalent/
inconclusive. `ousast learn experiment register|units|run|analyse`.
Program: `ProgramSpec.sampling = retrieval_ensemble` (k samples at
temperature 0 over k disjoint retrieved example sets; default unchanged);
`Caller.salt`; `retrieve(exclude=)`. verify: OUSAST_VERIFY_CALLERS=off.
Manifests exp-001-known-callers (plane arms, units pending) and
exp-002-retrieval-ensemble (slice-2 incumbents, ceiling $4) -- not run.
18 scripted tests; module audit regenerated (151 modules).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:04:33 +02:00
Marius CiepluchandClaude Fable 5.1 5aac7eac36 docs: audit against v2.0.0; deployment guide (local ax and a separate Kubernetes cluster)
Every README and docs page checked against ousast --help, the code and the committed records. New docs/deployment.md (in the nav): what runs where today, this host's setup (up.sh, doctor, smoke Run, footprint), step-by-step guidance for a separate Kubernetes cluster with implemented vs planned stated per item (no cluster deployment has been made; kube context and doctor's registry check are hard-coded in doctor.py, egress.py, router.py), and what the plane/ manifests need changed. Decision-engine pages state slice 2 (six families, pooled and within-pair AUC, BLOCK offered nowhere, canary below 0.9 for three families, positives-only inflation) and that the engine is not adopted; exp-002 registered, not run, no result quoted. Fixed: model/judge.py reference (now model/pipeline.py), missing-key behaviour, .env reference (OPENROUTER_EMBEDDING_MODEL, store variables), docs extra, plane memory-normalise and ousast index listed, ax README's cluster table made precise. RELEASE_NOTES gets an Unreleased entry.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:03:46 +02:00
Marius CiepluchandClaude Fable 5.1 090265b247 bench(learn): where the positives-only groups come from
Counts-only record of every repository group of the static memory that holds
positives and no negatives (98), traced label -> feature record -> excerpt:
94 are vibe-py pairs whose fixed side is another function of the same
snapshot (data gap, counted), 3 dev-php recipes without a fixed pin, 1
population-v2 fix that did not change the function. Separately, 16 pairs are
absent from the memory on both sides for tooling reasons: 7 advisory Ruby
pairs matched under the feature vocabulary's 'other' language, 9 corpus pairs
whose declaration form the excerpt matcher misses (JS object-literal methods,
property-assignment arrows, a $ identifier, a C parameter list past the
8-line lookahead, Perl sub).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 16:00:37 +02:00
Marius CiepluchandClaude Fable 5.1 a90e570f48 fix(learn): v1 withholds verify.flag_b and ms.operation (plane-profile leak audit)
Maintainer decision 2026-10-02. Not pushed with the release; main is otherwise
at v2.0.0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 15:25:21 +02:00
404 changed files with 100486 additions and 3117 deletions
@@ -55,6 +55,14 @@ Extract:
- The command that produced the failure
- Whether the failure is deterministic or intermittent
### 1b. Verify the Instrument Before You Believe the Measurement
Every wrong diagnosis recorded in this project had the same shape: the input was unreadable and the tool reported a plausible zero (`file_exists()` false, the parser wrote nothing, the frontend exited 0), and "no errors, 0 findings" read exactly like a clean result. Before trusting any measurement in the failure report:
- **Prove the tool read its input.** Make it open one file and print the size. A zero from an unread tree and a zero from clean code are indistinguishable afterwards.
- **Distrust a number that is too good.** A Joern script "finishing" in 0.5 s when a JVM takes 15 s to start is a failed launch. A harness must fail loudly with the exit code.
- **Prefer the shipped environment** (the Docker image has a native `php`); local wrappers that proxy stdio are not valid instruments for `php2cpg`.
- **When a result contradicts a working baseline, suspect the instrument first.**
A plausible zero is a failed instrument until proven otherwise; classify it as `CONFIG_GAP` or `RUNTIME_MISMATCH`, not as `LOGIC_ERROR` in the code under test.
### 2. Inspect Local Runtime and Repository State
Inspect the repository for local evidence:
- `package.json`, `pyproject.toml`, `go.mod`, `Makefile`, `README*`
@@ -134,6 +142,7 @@ If the issue is fixable by repo changes inside the current task plan, do not esc
| “Let’s try a few fixes” | Multi-fix guessing hides root cause. |
| “The spec is probably wrong, I’ll adapt it” | Spec conflicts must be surfaced explicitly. |
| “The docs search is optional” | For runtime/dependency issues, docs and version issues often contain the shortest path to root cause. |
| “It ran clean: no errors, zero findings” | A tool that never read its input reports the same. Prove the read (one file, its size) before believing the zero. |
## Output Format
@@ -20,7 +20,7 @@ description: Entry point for new work. Determines the best action path or work d
Gather **only metadata** to determine the action path. Do NOT read full file contents yet.
- **Specs inventory**: Scan `.kiro/specs/*/spec.json` for `name`, `phase` fields and `approvals` status. Note feature names and their current status.
- **Steering existence**: Check which files exist in `.kiro/steering/` (product.md, tech.md, structure.md, roadmap.md). Do NOT read their contents yet.
- **Steering existence**: Check which files exist in `.kiro/steering/` (core: overview.md, roadmap.md, safety-net.md; custom files may exist too). Do NOT read their contents yet.
- **Roadmap check**: If `.kiro/steering/roadmap.md` exists, read it. This contains project-level context (approach, scope, constraints, spec list) from a previous discovery session. Use it to restore project context.
- **Top-level structure**: List the project root directory to note key directories and files. Do NOT recurse into subdirectories.
@@ -60,7 +60,7 @@ For Path A/B, recommend the next action and stop.
**Only for Path C, D, and E.** Now load the context needed for discovery.
**In main context** (essential for dialogue with user):
- **Steering documents**: Read product.md and tech.md (if they exist) for project goals, constraints, and tech stack
- **Steering documents**: Read overview.md and safety-net.md (if they exist) for project goals, constraints, and the current state of the stack
- **Relevant specs**: If the request is adjacent to an existing spec, read that spec's requirements.md to understand boundaries and avoid overlap
**Delegate to sub-agent** (keeps exploration out of main context):
@@ -198,7 +198,7 @@ Use this roadmap structure:
- [ ] feature-c -- [one-line description]. Dependencies: feature-a, feature-b
```
Then write `.kiro/specs/<feature>/brief.md` for **every** feature listed under `## Specs (dependency order)` using the Path C brief format. This enables parallel spec creation via `/kiro-spec-batch`.
Then write `.kiro/specs/<feature>/brief.md` for **every** feature listed under `## Specs (dependency order)` using the Path C brief format. Each spec is then created one at a time with `/kiro-spec-init`, in dependency order.
**For Path E (mixed decomposition)**:
@@ -219,7 +219,7 @@ Use the same roadmap structure as Path D, plus these additional sections:
```
Path E rules:
- Keep `## Specs (dependency order)` reserved for **new specs only** so `/kiro-spec-batch` can still parse it unchanged
- Keep `## Specs (dependency order)` reserved for **new specs only** so the list stays a plain, parseable creation order
- Record existing-spec extensions under `## Existing Spec Updates`
- Record true no-spec work under `## Direct Implementation Candidates`
- Write `brief.md` only for the **new specs** listed under `## Specs (dependency order)`
@@ -235,15 +235,12 @@ Suggest the next command and stop. Do NOT automatically run downstream spec gene
- Path A: `/kiro-spec-requirements {feature}` to update the existing spec
- Path B: Recommend direct implementation without creating a spec
- Path C: Default to `/kiro-spec-init <feature-name>`
- Optional fast path: `/kiro-spec-quick <feature-name>` when the user explicitly wants to continue immediately
- Path D: Default to `/kiro-spec-batch` (creates all specs in parallel based on roadmap.md dependency order)
- Optional cautious path: `/kiro-spec-init <first-feature-name>` when the user wants to validate the first slice before batching the rest
- Path C: `/kiro-spec-init <feature-name>`, then the step-by-step phases (requirements, design, tasks), each human-reviewed
- Path D: `/kiro-spec-init <first-feature-name>` for the first spec in roadmap.md dependency order; validate that slice before the next one
- Path E: Choose the next command based on the new-spec portion of the decomposition
- If there is exactly one new spec: `/kiro-spec-init <new-feature-name>`
- If there are multiple new specs: `/kiro-spec-batch`
- `/kiro-spec-init <new-feature-name>` for the first (or only) new spec, the rest in dependency order afterwards
- Also note which existing specs should be revisited with `/kiro-spec-requirements <feature>`
- Re-entry: `/kiro-spec-init <next-feature-name>` or `/kiro-spec-batch` if multiple specs remain
- Re-entry: `/kiro-spec-init <next-feature-name>` for the next spec that remains
If the decomposition contains only existing-spec updates plus direct implementation candidates, do NOT use Path E. Prefer Path A when one existing spec is the clear home, or recommend the existing-spec update plus direct implementation work without creating roadmap entries.
@@ -26,7 +26,7 @@ You operate in two modes:
If steering/spec context is already available from conversation, skip redundant file reads.
Otherwise, load all necessary context:
- `.kiro/specs/{feature}/spec.json`, `requirements.md`, `design.md`, `tasks.md`
- Core steering context: `product.md`, `tech.md`, `structure.md`
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to the selected task's boundary, runtime prerequisites, integrations, domain rules, security/performance constraints, or team conventions that affect implementation or validation
- Relevant local agent skills or playbooks only when they clearly match the task's host environment or use case; read the specific artifact(s) you need, not entire directories
@@ -1,13 +1,18 @@
---
name: kiro-spec-design
description: Create comprehensive technical design for a specification
description: Create comprehensive technical design for a specification; optionally run the implementation gap analysis first (--gap) or an interactive design review of an existing design (--review)
metadata:
shared-rules: "design-principles.md, design-discovery-full.md, design-discovery-light.md, design-synthesis.md, design-review-gate.md"
shared-rules: "design-principles.md, design-discovery-full.md, design-discovery-light.md, design-synthesis.md, design-review-gate.md, gap-analysis.md, design-review.md"
---
# Technical Design Generator
**Arguments**: `$1` is the feature. `$2` may be `-y` (auto-approve requirements), `--gap` (run the
optional gap analysis of Step 1b before designing) or `--review` (run only the optional interactive
design review of Step 7 on an existing `design.md`). The design review gate of Step 5 always runs
when a design is generated; the optional steps never replace it.
<background_information>
- **Success Criteria**:
- All requirements mapped to technical components with clear interfaces
@@ -24,8 +29,8 @@ metadata:
**Read all necessary context**:
- `.kiro/specs/$1/spec.json`, `requirements.md`, `design.md` (if exists)
- `.kiro/specs/$1/research.md` (if exists, contains gap analysis from `/kiro-validate-gap`)
- Core steering context: `product.md`, `tech.md`, `structure.md`
- `.kiro/specs/$1/research.md` (if exists, contains the gap analysis from Step 1b or an earlier run)
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (this repository's core steering; custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to requirement coverage, architecture boundaries, integrations, runtime prerequisites, security/performance constraints, or team conventions that affect implementation readiness
- `.kiro/settings/templates/specs/design.md` for document structure
- Read `rules/design-principles.md` from this skill's directory for design principles
@@ -35,6 +40,16 @@ metadata:
- If `-y` flag provided ($2 == "-y"): Auto-approve requirements in spec.json
- Otherwise: Verify approval status (stop if unapproved, see Safety & Fallback)
### Step 1b (optional, `--gap`): Implementation Gap Analysis
Recommended for brownfield features; skip for greenfield. Analyze the gap between the approved requirements and the existing codebase to inform the design, following `rules/gap-analysis.md` from this skill's directory.
1. Read `rules/gap-analysis.md` for the analysis framework.
2. Research in parallel where possible (sub-agents if available, otherwise sequentially): existing implementations, architecture patterns, integration points and extension possibilities; external dependency compatibility and version constraints when needed; the requirements, core steering, task-relevant extra steering and relevant local agent skills or playbooks.
3. Execute the gap analysis: Grep and Read the codebase, use WebSearch/WebFetch for external dependencies if needed, evaluate multiple implementation approaches (extend / new / hybrid) with trade-offs, flag areas needing further research. Provide information and options, not final implementation choices.
4. Write the analysis to `.kiro/specs/$1/research.md` (append after a `---` rule if the file exists; never overwrite earlier research) and read it back to verify the write. Use the language specified in spec.json.
5. Unapproved requirements are a warning here, not a stop: gap analysis can inform requirement revisions. Then continue with Step 2, or stop and report if the user asked for the gap analysis alone.
### Step 2: Discovery & Analysis
**Critical: This phase ensures design is based on complete, accurate information.**
@@ -137,6 +152,15 @@ After all findings return, synthesize in main context before proceeding.
- Set `approvals.requirements.approved: true`
- Update `updated_at` timestamp
### Step 7 (optional, `--review`): Interactive Design Review
A human-paced quality review of an existing `.kiro/specs/$1/design.md`, in addition to the automatic gate of Step 5. Follow `rules/design-review.md` from this skill's directory.
1. Gather context: spec.json, requirements.md, design.md, core steering, review-relevant extra steering and local playbooks; survey existing codebase patterns, naming conventions and component structure for reference (parallel sub-agents if available).
2. Review: Analysis -> Critical Issues -> Strengths -> GO/NO-GO. Limit to the 3 most important concerns; recognize 1-2 strengths; engage interactively (clarifying questions, alternatives); leverage conversation history for requirements context and user intent. Quality assurance, not perfection seeking; accept acceptable risk; every suggestion must be implementable.
3. Decide: a clear GO/NO-GO with rationale. GO: `/kiro-spec-tasks $1` (or `-y`). NO-GO: address the critical issues, re-run `/kiro-spec-design $1`, then `/kiro-spec-design $1 --review` again.
4. If design.md does not exist, stop: "Run `/kiro-spec-design $1` first to generate design document". If the design phase is not marked generated in spec.json, warn but proceed.
## Critical Constraints
- **Type Safety**:
- Enforce strong typing aligned with the project's technology stack.
@@ -199,7 +223,7 @@ Provide brief summary in the language specified in spec.json:
**If Design Approved**:
- Review generated design at `.kiro/specs/$1/design.md`
- **Optional**: Run `/kiro-validate-design $1` for interactive quality review
- **Optional**: Run `/kiro-spec-design $1 --review` for the interactive quality review (Step 7)
- Then `/kiro-spec-tasks $1 -y` to generate implementation tasks
**If Modifications Needed**:
@@ -24,7 +24,7 @@ metadata:
- Read `.kiro/specs/$1/spec.json` for language and metadata
- Read `.kiro/specs/$1/brief.md` if it exists (discovery context: problem, approach, scope decisions, boundary candidates)
- Read `.kiro/specs/$1/requirements.md` for project description
- Core steering context: `product.md`, `tech.md`, `structure.md`
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to feature scope, user personas, business/domain rules, compliance/security constraints, operational constraints, or existing product boundaries
- Relevant local agent skills or playbooks only when they clearly match the feature's host environment or use case and contain domain terminology or workflow rules that shape user-observable requirements
@@ -40,7 +40,7 @@ The following research areas are independent. Decide the optimal decomposition b
**In main context** (essential for requirements generation):
- Spec files: spec.json, brief.md, requirements.md (project description)
- EARS format rules, requirements review gate, requirements template
- Core steering: product.md, tech.md (directly inform scope and constraints)
- Core steering: overview.md, safety-net.md (directly inform scope and constraints)
**Delegate to sub-agent** (keeps exploration out of main context):
- **Codebase hints** (brownfield projects): Spawn a sub-agent to explore existing implementations that inform requirement scope. Ask it to summarize: (1) what already exists, (2) relevant interfaces/APIs, (3) patterns that new requirements should align with. Return a summary under 150 lines.
@@ -133,7 +133,7 @@ Provide output in the language specified in spec.json with:
**If Requirements Approved**:
- Review generated requirements at `.kiro/specs/$1/requirements.md`
- **Optional Gap Analysis** (for existing codebases):
- Run `/kiro-validate-gap $1` to analyze implementation gap with current code
- Run `/kiro-spec-design $1 --gap` to analyze the implementation gap with the current code before designing
- Identifies existing components, integration points, and implementation strategy
- Recommended for brownfield projects; skip for greenfield
- Then `/kiro-spec-design $1 -y` to proceed to design phase
@@ -25,7 +25,7 @@ metadata:
**Read all necessary context**:
- `.kiro/specs/$1/spec.json`, `requirements.md`, `design.md`
- `.kiro/specs/$1/tasks.md` (if exists, for merge mode)
- Core steering context: `product.md`, `tech.md`, `structure.md`
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to requirements coverage, design boundaries, runtime prerequisites, or team conventions that affect task executability
**Validate approvals**:
@@ -1,6 +1,6 @@
---
name: kiro-steering
description: Manage .kiro/steering/ as persistent project knowledge
description: Manage .kiro/steering/ as persistent project knowledge, including custom steering documents for specialized contexts
metadata:
shared-rules: "steering-principles.md"
---
@@ -11,9 +11,12 @@ metadata:
<background_information>
**Role**: Maintain `.kiro/steering/` as persistent project memory.
**Core steering in this repository**: `overview.md` (what the project is and what is actually true of it, dated current state), `roadmap.md` (milestones and status), `safety-net.md` (product direction). Custom files may exist beside them (for example a domain or standards file); all `.kiro/steering/*.md` are loaded as project memory. The generic bootstrap templates under `.kiro/settings/templates/steering/` are named `product.md`, `tech.md`, `structure.md`; they are starting points for a fresh project, not this repository's file names.
**Mission**:
- Bootstrap: Generate core steering from codebase (first-time)
- Sync: Keep steering and codebase aligned (maintenance)
- Custom: Create specialized steering documents beyond the core files (optional, see below)
- Preserve: User customizations are sacred, updates are additive
**Success Criteria**:
@@ -27,8 +30,9 @@ metadata:
Check `.kiro/steering/` status:
**Bootstrap Mode**: Empty OR missing core files (product.md, tech.md, structure.md)
**Sync Mode**: All core files exist
**Bootstrap Mode**: Empty OR missing core files (here `overview.md`, `roadmap.md`, `safety-net.md`; in a fresh project the template names `product.md`, `tech.md`, `structure.md`)
**Sync Mode**: All core files exist
**Custom Mode**: The user asks for a specialized steering document (see Custom Steering below)
---
@@ -75,6 +79,20 @@ After all parallel research completes, synthesize patterns for steering files.
---
## Custom Steering (optional)
Create a specialized steering document for one domain beyond the core files.
1. **Ask the user** for the domain/topic (e.g. "API standards", "testing approach") and the specific patterns to document
2. **Check for a template** in `.kiro/settings/templates/steering-custom/`: `api-standards.md`, `testing.md`, `security.md`, `database.md`, `error-handling.md`, `authentication.md`, `deployment.md`. Load the matching one as a starting point; without a template, generate from domain knowledge
3. **Analyze the codebase** (JIT) for the domain's patterns with Glob/Grep/Read (template and principles in parallel with the domain analysis when multi-agent is available)
4. **Generate** the document: follow the template structure if any, apply `rules/steering-principles.md`, patterns not exhaustive lists, one domain per file, 100-200 lines (a 2-3 minute read), never secrets
5. **Create** `.kiro/steering/{name}.md`; ensure it does not duplicate core steering content
Custom files are loaded as project memory like the core files and carry equal weight.
---
## Granularity Principle
From `rules/steering-principles.md` (in this skill's directory):
@@ -106,9 +124,9 @@ Chat summary only (files updated directly).
✅ Steering Created
## Generated:
- product.md: [Brief description]
- tech.md: [Key stack]
- structure.md: [Organization]
- overview.md: [What it is, current state]
- roadmap.md: [Milestones, status]
- safety-net.md: [Product direction]
Review and approve as Source of Truth.
```
@@ -118,8 +136,8 @@ Review and approve as Source of Truth.
✅ Steering Updated
## Changes:
- tech.md: React 18 → 19
- structure.md: Added API pattern
- overview.md: Current state dated, retired component removed
- roadmap.md: Milestone status updated
## Code Drift:
- Components not following import conventions
@@ -128,6 +146,20 @@ Review and approve as Source of Truth.
- Consider api-standards.md
```
### Custom:
```
✅ Custom Steering Created
## Created:
- .kiro/steering/api-standards.md
## Based On:
- Template: api-standards.md
- Analyzed: src/api/ directory patterns
Review and customize as needed.
```
## Examples
### Bootstrap
@@ -136,7 +168,12 @@ Review and approve as Source of Truth.
### Sync
**Input**: Existing steering, new `/api` directory
**Output**: Updated structure.md, flagged non-compliant files, suggested api-standards.md
**Output**: Updated overview.md, flagged non-compliant files, suggested api-standards.md
### Custom
**Input**: "Create API standards steering"
**Action**: Load template, analyze src/api/, extract patterns
**Output**: api-standards.md with project-specific REST conventions
## Safety & Fallback
@@ -84,7 +84,11 @@ Never include:
## File-Specific Focus
- **product.md**: Purpose, value, business context (not exhaustive features)
- **tech.md**: Key frameworks, standards, conventions (not all dependencies)
- **structure.md**: Organization patterns, naming rules (not directory trees)
- **Custom files**: Specialized patterns (API, testing, security, etc.)
This repository's core steering files:
- **overview.md**: What the project is and what is actually true of it; a dated current-state section wins over older history
- **roadmap.md**: Milestones, their measured status, and the accepted order of work (not a task list)
- **safety-net.md**: Product direction and acceptance criteria (not feature catalogs)
- **Custom files**: Specialized patterns (API, testing, security, etc.); they may exist beside the core files and carry equal weight
(The generic bootstrap templates `product.md`, `tech.md`, `structure.md` under `.kiro/settings/templates/steering/` cover purpose, key frameworks and organization patterns for a fresh project.)
@@ -68,10 +68,16 @@ For each detected feature:
- Read `.kiro/specs/<feature>/requirements.md` for requirements
- Read `.kiro/specs/<feature>/design.md` for design structure
- Read `.kiro/specs/<feature>/tasks.md` for task list and Implementation Notes
- Core steering context: `product.md`, `tech.md`, `structure.md`
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
- Additional steering files only when directly relevant to the validated boundaries, runtime prerequisites, integrations, domain rules, security/performance constraints, or team conventions that affect the GO/NO-GO call
**Discover canonical validation commands**:
**This repository's gates** (`.github/workflows/ci.yml`; run them as written, judge each by its own exit code):
- `TEST_COMMANDS`: `pytest -q -p no:cacheprovider < /dev/null; echo "exit=$?"` -- judge ONLY by the printed exit code, never by a summary line; `ruff check src tests`; `ruff format --check src tests`; `mypy src/openultrasast`
- Detection gates: `python -m openultrasast.gate` (>=90% recall, <10% FP per language; hard, score-independent), `python -m openultrasast.map_gate`, and the pair gate `python -m openultrasast.pair_gate` (local vuln vs fix). A ruleset change that fails a gate is NO-GO.
- Docs: `mkdocs build --strict` when `docs/` or `mkdocs.yml` changed
- `SMOKE_COMMANDS`: `ousast --help`, and for a scan change `ousast scan <fixture> --mode quick` whose `manifest.json` names the stages that ran and the files preprocess kept. A plausible zero (0 findings, 0 degradations) from a tool that never read its input is a failed instrument, not a pass: prove the tool read its input first.
**Discover canonical validation commands** (for other repositories, or anything the list above does not cover):
- Inspect repository-local sources of truth in this order: project scripts/manifests (`package.json`, `pyproject.toml`, `go.mod`, `Cargo.toml`, app manifests), task runners (`Makefile`, `justfile`), CI/workflow files, existing e2e/integration configs, then `README*`
- Derive a feature-level validation set for this repo: `TEST_COMMANDS`, `BUILD_COMMANDS`, and `SMOKE_COMMANDS`
- Prefer commands already used by repo automation over ad hoc shell pipelines
@@ -61,7 +61,8 @@ Use the language specified in `spec.json`.
4. Check exit code, failure count, skipped scope, and missing coverage.
5. Reject claims that are broader than the evidence.
6. If mandatory validation cannot be completed, return `MANUAL_VERIFY_REQUIRED`.
7. Only then allow the claim.
7. Verify the instrument before you believe the measurement: prove the tool read its input (it opened one file and printed the size; the stage list names what ran). A plausible zero -- "0 findings, 0 errors, exit 0" from a tree the tool never read, or a Joern script that "finished" before a JVM could start -- is a failed instrument, not evidence. For pytest, judge only by the printed exit code (`pytest -q -p no:cacheprovider < /dev/null; echo "exit=$?"`), never by a summary line.
8. Only then allow the claim.
## Claim-Specific Rules
@@ -116,6 +117,7 @@ Return `NOT_VERIFIED` when:
| “Build should be fine because lint passed” | Lint does not prove build success. |
| “Tests passed and build succeeded, so it must run” | Type erasure, module loading, native ABI, and boot-time config issues can still fail at runtime. |
| “The feature is done because all tasks are checked off” | `FEATURE_GO` also requires coverage, integration, and design alignment. |
| “The scan found nothing, so the code is clean” | An unread input and clean code both report zero. Prove the tool read its input first. |
## Output Format
+123
View File
@@ -0,0 +1,123 @@
---
name: openultrasast-scan
description: OpenUltraSAST scan workflows - `ousast scan` in quick, standard or deep mode, the evidence ladder, `--fail-on` exit codes, recorded degradations, the run directory and `ousast mcp`. Use when running or planning a scan, reading a run directory, or wiring OpenUltraSAST into an agent.
---
# OpenUltraSAST Scan
Use this skill to run `ousast scan PATH --mode quick|standard|deep` from an agent and to read what
it wrote.
## Workflow
1. Confirm the target is a local checkout (a path, not a URL).
2. Run through the project entry point: `uv run ousast ...` inside the repository, or `ousast` from an
environment where it is installed. Never activate a virtualenv by hand.
3. Start with `--mode quick` unless the task needs the Joern model layer (`standard`) or sandboxed
regression snippets (`deep`). Check the prerequisites below first; a mode whose engine is missing
still exits 0 and records a degradation.
4. Read the printed `scan_id=` and `run_dir=` lines; the run directory is `.openultrasast/runs/<scan-id>/`.
5. Report the finding count, how many the verifier accepted, and **every** recorded degradation
(`manifest.json`, key `degradations`). No findings plus a degradation is not a clean scan: the scan
answered a smaller question than it was asked.
6. Never call a finding verified from its title or severity. `verification.json` holds the status.
## Commands
```bash
uv run ousast scan <repo> --mode quick
uv run ousast scan <repo> --mode standard --fail-on verified
uv run ousast scan <repo> --mode deep --fail-on worth-fixing
uv run ousast scan <repo> --mode quick --config openultrasast.toml
uv run ousast mcp # the MCP server over stdio, for agent hosts
```
## Modes and prerequisites
| Mode | Adds | Needs | Without it |
| --- | --- | --- | --- |
| `quick` | language-scoped pattern rules, entry-point hints, ranking, verification, score; deterministic | nothing | - |
| `standard` | the MAP stage: complexity map, semantic overlay, authorization obligations, the Joern model layer (one code property graph per repository; taint, guard-dominance and configuration questions), deterministic fusion | Joern (shipped in the Docker image, with `php-cli` for PHP); a provider key for the residual LLM question | `cpg_unavailable`; without a key the LLM question is not asked and only the graph's entailed findings are reported |
| `deep` | REGRESS: promoted candidates run as snippets in a Docker sandbox (no network, read-only source, non-root) and get a `triggerable` / `not_triggerable` / ... verdict in `verdicts.json` | a working `docker` | `sandbox_unavailable` |
Model stages read `DEEPSEEK_API_KEY` (every LLM call); with only `OPENROUTER_API_KEY` set, chat
calls fall back to OpenRouter. An exported shell variable wins over `.env` silently. `[models]
hunter` in the config enables the tool hunter in `standard`; it is off by default.
## Exit code: `--fail-on`
| Policy | Exit 1 when |
| --- | --- |
| `never` (default) | never |
| `findings` | any finding was reported |
| `verified` | any verification result is `verified` (accepted at `static_corroboration` or above) |
| `worth-fixing` | a deep-mode sandbox verdict is marked `worth_fixing` |
A degradation never changes the exit code. Read `manifest.json` for it.
## Recorded degradations
Each entry in `manifest.json` `degradations` is `{"stage": ..., "reason": ...}`. The ones to expect:
- `cpg_unavailable` (stage `model`): no Joern on the host; the model layer ran nothing.
- `sandbox_unavailable` (stage `regress`): no working `docker`; deep mode fell back to standard.
- `learning_endpoint_unavailable` (stage `model`): no provider key resolved. The graph's entailed
findings are still reported; the residual question is skipped. This is recorded in `manifest.json`
but has no line in `report.md`'s "What could not be analysed" section.
- `hunter_model_unavailable` (stage `map`): `[models] hunter` is unset, so the tool hunter did not run.
Recorded on every standard scan without that setting; it is expected, not an error.
- coverage facts from the graph, rendered in `report.md` under "What could not be analysed":
`cpg_empty`, `cpg_build_failed`, `cpg_sharded`, `files_unparsed`, `query_failed`,
`query_budget_reserved`, `query_too_expensive`, `budget_exhausted`, `regions_truncated`,
`region_failed`; plus `max_findings_exceeded` and `facts_unavailable` (overlay, obligations).
Prove the instrument read its input before believing a zero: `manifest.json` names the stages that
ran and the files the preprocess step kept (`preprocess/file_targets.json`).
## Evidence ladder
`src/openultrasast/verification.py` defines the levels in this order. A level moves one rung at a
time and never backward.
| Level | Meaning | Produced today by |
| --- | --- | --- |
| `suspicion` | a rule, model or weak signal raised a hypothesis | the Joern model layer (its rung, `model_entailed` or `suspicion`, travels in the finding's tags), the tool hunter, authorization-obligation checks |
| `static_corroboration` | deterministic code evidence at the line | quick rules, the semantic overlay |
| `crash_reproduced` | an artifact-backed reproducer | nothing: there is no crash reproducer. Deep-mode sandbox verdicts (`verdicts.json`, `worth_fixing`) do not raise the level |
| `root_cause_explained` | independent verifier reasoning | nothing: the LLM judge was retired in v2.0.0 |
| `exploit_demonstrated` | a demonstrated exploit | nothing |
| `patch_validated` | relevant checks re-run after a patch | nothing: there is no patch stage |
The verifier's status per finding (`verification.json`): below `static_corroboration` is
`needs_evidence`; `static_corroboration` with reachability `unknown` is `rejected` (the tie-breaker
asks for call-graph, route, CLI, parser or dynamic evidence); `static_corroboration` with
function-level reachability is `accepted`. `verified` means accepted at `static_corroboration` or
above, and that is what `--fail-on verified` counts. There is no configurable report threshold: the
`[evidence]` config section was retired in v2.0.0 and is ignored.
## Run artifacts
Under `.openultrasast/runs/<scan-id>/`:
- `report.md`, `report.sarif` (SARIF 2.1.0 for code hosts and IDEs), `manifest.json` (stages, degradations, model payload)
- `findings.json`, `verification.json`, `score.json`, `shadow_findings.json` (rules shipped as `shadow`)
- `harness.json`: the resolved harness configuration the run used
- standard: `fusion.json`, `complexity_map.json`, `overlay.json`; deep: `verdicts.json`
- `calibration/false_positive_learnings.json`, `rank/ranking.json`, `mapping/`, `preprocess/`, `trace/events.jsonl`
## Changed code only
`ousast pre-push --base <rev> --head <rev> --artifact <file> <repo>` replays a revision pair and
reports only new or worsened defects (experimental; `--mode advisory` is the default and never
blocks). The capability registry is empty, so every candidate is diagnostic.
## MCP
`ousast mcp` serves a narrow MCP server over stdio (newline-delimited JSON-RPC) with ten tools:
`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`,
`explain`, `propose_patch` (degrades visibly) and `export_report`. No tool runs a shell, Docker or a
free-form command.
```jsonc
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
```
@@ -0,0 +1,75 @@
---
name: openultrasast-triage
description: OpenUltraSAST finding triage - false-positive adjudication with the fixed reason taxonomy, the evidence ladder, what a verdict may feed (labels, never rule edits), and fixing then re-scanning. Use when reviewing findings, reducing false positives, deciding whether evidence is enough, or auditing a fix for a finding.
---
# OpenUltraSAST Triage
Use this skill when triaging OpenUltraSAST findings or auditing a fix for one.
## Triage Loop
1. Identify the finding: id, path, rule, CWE and policy severity, confidence, evidence level and
verification status (`findings.json`, `verification.json` in the run directory).
2. Separate deterministic evidence (rule match, graph entailment, attached reachability) from model
assumptions.
3. Check attacker control, reachability, sanitizer behaviour, duplicates, impact, and contradictory
evidence.
4. If rejected or unresolved, record the false-positive reason and its scope.
5. Prefer scoped demotion over deleting a vulnerability class globally: a rejection in `auth/`
demotes `auth/...`, never the class (`calibration.py`).
6. Record the adjudication. Do not edit a rule because of one finding (see below).
## False-Positive Reasons
The taxonomy is `FalsePositiveReason` in `src/openultrasast/calibration.py`; use these values exactly:
- `unreachable_path`
- `missing_attacker_control`
- `sanitizer_disproved`
- `static_rule_mismatch`
- `incorrect_model_assumption`
- `duplicate`
- `insufficient_impact`
- `unsupported`
- `contradicted`
- `unverified`
## Decision Standard
If evidence is uncertain, mark the finding `needs_evidence` and name the next evidence instead of
escalating severity. Fusion (`fusion.json`, standard mode) is deterministic: it adjudicates between
the quick-rule and MAP panels by fixed triggers and is not a second opinion from a model. Independent
model agreement exists only in the plane's verify passes a and b, with pass c on disputes, and the
`agree` step; those verdicts arrive as features of the decision engine, never as a verification
status.
## What an Adjudication Feeds
- Recorded adjudications (a per-finding verdict with a reason) become decision-engine labels through
the fail-closed source list `src/openultrasast/learn/sources.toml` (`[[source]] id =
"adjudications"`). Aggregate counts label nothing; only per-finding verdicts do.
- Plane verdicts (`agreed`, `rejected`, `disputed`) are features, never labels. "Rejected by the
plane" is not a negative.
- Rule changes go through the ledger: `ousast improve` (the validator and the detection gate, holdout
pairs per round) or a registered experiment (`ousast learn experiment register|units|run|analyse`).
Never a hand edit of `rules.toml` per miss or per false positive.
## Fix and Re-scan
1. Intake: restate the finding id, evidence level, affected path, and the acceptance criterion.
2. Plan: the smallest defensive change that addresses the root cause.
3. Implement: modify only the files the finding requires.
4. Adversarial review: look for bypasses, regressions, false assumptions, and unrelated edits. Use a
differential review for changes in auth, crypto, config, parsers, and sandbox behaviour.
5. Reconcile: fix each review finding or document why it does not apply.
6. Verify: a re-scan of the checkout in the same mode, or a replay of the change with
`ousast pre-push --base <before> --head <after> --artifact <file> <repo>`, shows the finding gone
and no new findings. That re-scan is the evidence. Nothing in the tool records `patch_validated`;
do not claim it.
Guardrails:
- Do not rewrite modules when a local fix is enough.
- Do not change public APIs unless the vulnerability requires it.
- Do not call a finding fixed from model reasoning alone; the re-scan or replay decides.
+8 -2
View File
@@ -6,9 +6,8 @@
DEEPSEEK_API_KEY=
# DEEPSEEK_BASE_URL=https://api.deepseek.com
# Embeddings only: OpenAI text-embedding-3-small through OpenRouter.
# Embeddings only (`ousast learn memory embed`, default openai/text-embedding-3-small through OpenRouter).
OPENROUTER_API_KEY=
OPENROUTER_EMBEDDING_MODEL=openai/text-embedding-3-small
# Plane memory store. Default is a local FileStore under ~/ousast-results/plane/memory.
# For an S3-compatible server (RustFS is the tested one): OUSAST_MEMORY=s3://<bucket>[/<prefix>], or
@@ -22,3 +21,10 @@ OPENROUTER_EMBEDDING_MODEL=openai/text-embedding-3-small
# AWS_SESSION_TOKEN=
# The bucket must have versioning enabled, a lifecycle rule expiring runs/, S3 Select and readable object tags
# (see docs/rustfs.md); the store verifies them at startup and refuses to run otherwise.
# Operator notifications (Discord). Status of OpenUltraSAST processes and their results for the
# system/AI operator -- not developer debugging, and not k3s/ax infra health. See
# .kiro/steering/operator-notifications.md. Webhook posts to the operator channel; the operator id
# is the account to @mention on a stuck/impaired alert. Leave blank to disable notifications.
DISCORD_WEBHOOK_URL=
DISCORD_OPERATOR_ID=
+43
View File
@@ -0,0 +1,43 @@
name: Docs
# Builds the MkDocs site and publishes it to GitHub Pages on every push to main that touches it.
on:
push:
branches: [main]
paths: ["docs/**", "mkdocs.yml", "README.md", ".github/workflows/docs.yml"]
workflow_dispatch:
permissions:
contents: read
pages: write
id-token: write
concurrency:
group: pages
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: astral-sh/setup-uv@v7
with:
enable-cache: true
- uses: actions/setup-python@v6
with:
python-version-file: .python-version
- run: uv sync --extra docs
- run: uv run mkdocs build --strict
- uses: actions/upload-pages-artifact@v3
with:
path: site
deploy:
needs: build
runs-on: ubuntu-latest
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- id: deployment
uses: actions/deploy-pages@v4
+199
View File
@@ -0,0 +1,199 @@
name: Engine image
on:
workflow_dispatch:
push:
branches: [main]
tags: ['v*']
paths:
- 'plane/Dockerfile.*'
- 'plane/task-base.digest'
- 'Dockerfile'
permissions:
contents: read
packages: write
concurrency:
group: engine-image-${{ github.ref }}
cancel-in-progress: false
jobs:
pins:
runs-on: ubuntu-latest
outputs:
base_tag: ${{ steps.pins.outputs.base_tag }}
base_digest: ${{ steps.pins.outputs.base_digest }}
browser_tag: ${{ steps.pins.outputs.browser_tag }}
browser_digest: ${{ steps.pins.outputs.browser_digest }}
steps:
- uses: actions/checkout@v5
- uses: docker/setup-buildx-action@v3
# No login: every pinned base must be anonymously pullable.
- id: pins
run: |
python3 - <<'PYTHON'
import json, os, re, subprocess
from pathlib import Path
pins = json.loads(Path("plane/task-base.digest").read_text())
manifests = {}
for key, output in (("base", "base"), ("base-browser", "browser")):
ref = pins.get(key)
pattern = r"ghcr.io/norandom/ousast-task-base:(base-[a-z0-9-]+)@(sha256:[0-9a-f]{64})"
match = re.fullmatch(pattern, ref or "")
if not match:
raise SystemExit(f"{key}: unpublished/invalid pin; run task-base-image and update plane/task-base.digest")
raw = subprocess.check_output(["docker", "buildx", "imagetools", "inspect", "--raw", ref])
manifests[key] = json.loads(raw)["layers"]
with open(os.environ["GITHUB_OUTPUT"], "a") as out:
out.write(f"{output}_tag={match[1]}\n{output}_digest={match[2]}\n")
base = manifests["base"]
if not base or manifests["base-browser"][:len(base)] != base:
raise SystemExit("Browser pin does not share the pinned base's layers")
PYTHON
image:
needs: pins
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- package: openultrasast
dockerfile: Dockerfile
- package: ousast-search-task
dockerfile: plane/Dockerfile.search-task
steps:
- uses: actions/checkout@v5
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: names
env:
OWNER: ${{ github.repository_owner }}
PACKAGE: ${{ matrix.package }}
run: |
echo "image=ghcr.io/${OWNER,,}/$PACKAGE" >> "$GITHUB_OUTPUT"
echo "sha=sha-${GITHUB_SHA:0:7}" >> "$GITHUB_OUTPUT"
- id: build
uses: docker/build-push-action@v6
with:
context: .
file: ${{ matrix.dockerfile }}
platforms: linux/amd64
push: true
provenance: false
build-args: |
TASK_BASE_TAG=${{ needs.pins.outputs.base_tag }}
TASK_BASE_DIGEST=${{ needs.pins.outputs.base_digest }}
TASK_BROWSER_TAG=${{ needs.pins.outputs.browser_tag }}
TASK_BROWSER_DIGEST=${{ needs.pins.outputs.browser_digest }}
cache-from: |
type=gha,scope=${{ matrix.package }}
type=gha,scope=ousast-engine-task
cache-to: type=gha,mode=max,scope=${{ matrix.package }}
tags: |
${{ steps.names.outputs.image }}:${{ steps.names.outputs.sha }}
${{ steps.names.outputs.image }}:main
- env:
IMAGE: ${{ steps.names.outputs.image }}
DIGEST: ${{ steps.build.outputs.digest }}
run: |
echo "$IMAGE@$DIGEST"
echo "$IMAGE@$DIGEST" >> "$GITHUB_STEP_SUMMARY"
docker logout ghcr.io
docker buildx imagetools inspect "$IMAGE@$DIGEST"
engine:
needs: pins
runs-on: ubuntu-latest
outputs:
image: ${{ steps.names.outputs.image }}
digest: ${{ steps.build.outputs.digest }}
steps:
- uses: actions/checkout@v5
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: names
env:
OWNER: ${{ github.repository_owner }}
PACKAGE: ousast-engine-task
run: |
echo "image=ghcr.io/${OWNER,,}/$PACKAGE" >> "$GITHUB_OUTPUT"
echo "sha=sha-${GITHUB_SHA:0:7}" >> "$GITHUB_OUTPUT"
- id: build
uses: docker/build-push-action@v6
with:
context: .
file: plane/Dockerfile.engine-task
target: engine
platforms: linux/amd64
push: true
provenance: false
build-args: |
TASK_BASE_TAG=${{ needs.pins.outputs.base_tag }}
TASK_BASE_DIGEST=${{ needs.pins.outputs.base_digest }}
TASK_BROWSER_TAG=${{ needs.pins.outputs.browser_tag }}
TASK_BROWSER_DIGEST=${{ needs.pins.outputs.browser_digest }}
cache-from: type=gha,scope=ousast-engine-task
cache-to: type=gha,mode=max,scope=ousast-engine-task
tags: |
${{ steps.names.outputs.image }}:${{ steps.names.outputs.sha }}
${{ steps.names.outputs.image }}:main
- env:
IMAGE: ${{ steps.names.outputs.image }}
DIGEST: ${{ steps.build.outputs.digest }}
run: |
echo "$IMAGE@$DIGEST"
echo "$IMAGE@$DIGEST" >> "$GITHUB_STEP_SUMMARY"
docker logout ghcr.io
docker buildx imagetools inspect "$IMAGE@$DIGEST"
replay:
needs: [pins, engine]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: names
env:
OWNER: ${{ github.repository_owner }}
PACKAGE: ousast-replay-task
run: |
echo "image=ghcr.io/${OWNER,,}/$PACKAGE" >> "$GITHUB_OUTPUT"
echo "sha=sha-${GITHUB_SHA:0:7}" >> "$GITHUB_OUTPUT"
- id: build
uses: docker/build-push-action@v6
with:
context: .
file: plane/Dockerfile.engine-task
target: replay
platforms: linux/amd64
push: true
provenance: false
build-args: |
ENGINE_REF=${{ needs.engine.outputs.image }}@${{ needs.engine.outputs.digest }}
TASK_BASE_TAG=${{ needs.pins.outputs.base_tag }}
TASK_BASE_DIGEST=${{ needs.pins.outputs.base_digest }}
TASK_BROWSER_TAG=${{ needs.pins.outputs.browser_tag }}
TASK_BROWSER_DIGEST=${{ needs.pins.outputs.browser_digest }}
cache-from: |
type=gha,scope=ousast-replay-task
type=gha,scope=ousast-engine-task
cache-to: type=gha,mode=max,scope=ousast-replay-task
tags: |
${{ steps.names.outputs.image }}:${{ steps.names.outputs.sha }}
${{ steps.names.outputs.image }}:main
- env:
IMAGE: ${{ steps.names.outputs.image }}
DIGEST: ${{ steps.build.outputs.digest }}
run: |
echo "$IMAGE@$DIGEST"
echo "$IMAGE@$DIGEST" >> "$GITHUB_STEP_SUMMARY"
docker logout ghcr.io
docker buildx imagetools inspect "$IMAGE@$DIGEST"
+42
View File
@@ -34,3 +34,45 @@ jobs:
generate_release_notes: true
prerelease: ${{ contains(github.ref_name, '-') }}
make_latest: ${{ !contains(github.ref_name, '-') }}
images:
# The scan image (Joern + php-cli, `docker run ... scan`) and the plane's task runner, pushed to GHCR under the
# tag's version. Pin them by the digests this job prints, not by tag: Agent Substrate rejects tags.
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
strategy:
fail-fast: false
matrix:
include:
- name: openultrasast
dockerfile: Dockerfile
- name: openultrasast-runner
dockerfile: plane/Dockerfile.runner
steps:
- uses: actions/checkout@v5
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository_owner }}/${{ matrix.name }}
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}},enable=${{ !contains(github.ref_name, '-') }}
type=raw,value=latest,enable=${{ !contains(github.ref_name, '-') }}
- id: build
uses: docker/build-push-action@v6
with:
context: .
file: ${{ matrix.dockerfile }}
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
- run: |
echo "ghcr.io/${{ github.repository_owner }}/${{ matrix.name }}@${{ steps.build.outputs.digest }}" >> "$GITHUB_STEP_SUMMARY"
+78
View File
@@ -0,0 +1,78 @@
name: Task base image
on:
workflow_dispatch:
push:
branches: [main]
paths:
- 'plane/Dockerfile.base'
- 'pyproject.toml'
- 'uv.lock'
permissions:
contents: read
packages: write
concurrency:
group: task-base-image
cancel-in-progress: false
jobs:
base:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: names
run: |
echo "tag=base-${GITHUB_SHA:0:7}" >> "$GITHUB_OUTPUT"
content=$(cat plane/Dockerfile.base pyproject.toml uv.lock | sha256sum | cut -d' ' -f1)
echo "content=base-content-$content" >> "$GITHUB_OUTPUT"
- id: base
uses: docker/build-push-action@v6
with:
context: .
file: plane/Dockerfile.base
target: base
platforms: linux/amd64
push: true
provenance: false
sbom: false
tags: |
ghcr.io/norandom/ousast-task-base:${{ steps.names.outputs.tag }}
ghcr.io/norandom/ousast-task-base:${{ steps.names.outputs.content }}
cache-from: type=gha,scope=task-base
cache-to: type=gha,mode=max,scope=task-base
- id: browser
uses: docker/build-push-action@v6
with:
context: .
file: plane/Dockerfile.base
target: base-browser
platforms: linux/amd64
push: true
provenance: false
sbom: false
build-args: |
TASK_BASE_REF=ghcr.io/norandom/ousast-task-base:${{ steps.names.outputs.tag }}@${{ steps.base.outputs.digest }}
tags: |
ghcr.io/norandom/ousast-task-base:${{ steps.names.outputs.tag }}-browser
ghcr.io/norandom/ousast-task-base:${{ steps.names.outputs.content }}-browser
cache-from: type=gha,scope=task-base-browser
cache-to: type=gha,mode=max,scope=task-base-browser
- name: Print manual pin update and verify public access
env:
BASE: ghcr.io/norandom/ousast-task-base:${{ steps.names.outputs.tag }}@${{ steps.base.outputs.digest }}
BROWSER: ghcr.io/norandom/ousast-task-base:${{ steps.names.outputs.tag }}-browser@${{ steps.browser.outputs.digest }}
run: |
python3 - <<'PYTHON' >> "$GITHUB_STEP_SUMMARY"
import json, os
print("Replace plane/task-base.digest with this JSON after both public checks pass:")
print("```json")
print(json.dumps({"base": os.environ["BASE"], "base-browser": os.environ["BROWSER"]}, indent=2))
print("```")
PYTHON
docker logout ghcr.io
docker buildx imagetools inspect "$BASE"
docker buildx imagetools inspect "$BROWSER"
+6
View File
@@ -36,3 +36,9 @@ workspace/
# MkDocs build output (mkdocs build)
site/
# Unseen evaluation pointers for non-permissive repositories.
benchmarks/unseen/private/
# Feasibility pilot identities remain local; only aggregate records are tracked.
benchmarks/search/private/
+2 -2
View File
@@ -1,9 +1,9 @@
{
"feature_name": "ai-service-plane",
"created_at": "2026-09-29T15:40:00Z",
"updated_at": "2026-09-29T17:55:00Z",
"updated_at": "2026-10-02T00:00:00Z",
"language": "en",
"phase": "implementation",
"phase": "implementation-complete",
"approvals": {
"requirements": {
"generated": true,
@@ -1,5 +1,7 @@
# Requirements Document: contributor-scan
> **Status 2026-10-02:** paused since 2026-09-23 (its last change). 27 of 51 tasks done: the MAP integration was delivered and `v1.2.0-alpha.1` rolled out NO-GO (`census-repair-verification.md`); the remaining tasks wait on `pre-push-safety-net` M2/M3 and nothing here moved after the ax plane landed. Not active.
## Introduction
`model-grounded-detection` built an arbiter and measured it: 41.3% of the corpus reaches `model_entailed`
@@ -0,0 +1,140 @@
# Engine change-scoped analysis (G2 workstream)
Status: **proposed / in progress** (opened 2026-10-11). Owner: engine. Tracks a concrete lever for the
engine-detection gap found while driving the G0 baseline. This is G2 ("improve what moves the unseen-repo
numbers"), gated on the pool metric; it is NOT part of the G0 record-07 baseline (that runs on the quick-rule tier).
## Problem (measured, not assumed)
On real unseen vulnerability changes the engine produces **no finding** — but the diagnosis (live, 300 s long-deadline
replays) shows it is usually **not a detection-capability gap**:
- **Clean Python injection change** (`aws/bedrock-agentcore-sdk-python`, `install_packages`, introducing, 3-file
diff): `wall=303 s`, hit the deadline. The ranker found candidates (`static_rank=1.0`) but **`selected: []`** and
every candidate `deferred` with `reason: deadline_exhausted`, `analysis_not_started=deadline_exhausted`,
`change_context_incomplete`. **Zero analyses ran.** And the ranked candidates were **repo-wide and unrelated to
the push** (`auth.py` access_control/config_secrets), not the changed `install_packages`. The engine is trying to
re-audit the whole repo and runs out of budget before analyzing the change.
- **Noisy last_touch config_secrets change** (`jenkinsci/metrics-plugin`, 14-file diff): finished in 206 s but
`comparison_unknown` + `capability_unavailable` + `eligibility_provenance_mismatch` +
`contributor-scan ... projection_unavailable`. Multi-causal; correspondence was one part (fixed separately,
commit c39fbb8), capability registry is Task 8.3.
## Hypothesis
A pre-push hook should vet **the code this push changed**, not re-scan the whole repository. Seeding/scoping the
engine's analysis to the **changed declarations and their dataflow neighborhood** (the `change_sites` are already in
the replay spec) should let analysis **actually run within budget** instead of being deferred — which is where
detection is being lost, not in Joern's capability.
## Anchors (confirm in increment 1)
- `src/openultrasast/rank.py` — the static-evidence ranker (`ranking_mode="evidence"`, `existing-static-evidence-v1`).
- `src/openultrasast/mapping.py` — builds the `selected`/`deferred` scope + the budget allocation that defers on
`deadline_exhausted`.
- `src/openultrasast/push/policy.py`, `push/runner.py` — consume the scope; change_sites arrive via the push input.
- `benchmarks/unseen/replay_entry.py` — carries `changed_paths` / `declaration_paths` in the spec.
## Increments (progress tracking)
- [x] **0. Cost split confirmed (2026-10-11, timed live replay).** `build_seconds=66.69` (CPG build is cheap),
`query_seconds=0.0` (no query ever executed), total 303 s (hit the deadline), **4,095** candidate questions all
`analysis_not_started=deadline_exhausted`, `selected: []`, `regions_scanned=0`. So of the 303 s, ~67 s built the
CPG and ~236 s went to enumerating/ranking/preparing 4,095 repo-wide candidate questions for a **3-file** change —
the budget was spent before a single query ran. The CPG build is NOT the bottleneck; the over-broad candidate set
is. Change-scoping is confirmed as the right lever, and because per-candidate prep includes the line-correspondence
work, fewer candidates also cuts that cost (the two fixes compound).
- [x] **1. Located (2026-10-11).** The over-enumeration is upstream of the scan, not inside it:
- `push/runner.py:345-348` — `head_regions = discovery(tip.root, …)` discovers regions across the **whole head
tree** (the ~4,095), independent of the change.
- `push/runner.py:397-407` — those regions are passed wholesale to `scan_repository(..., change_context=context)`;
`change_context` is threaded in but **never scopes the region set**.
- `model/scan.py:167` and `:364` — `[_identity(unit, r, f) for r in regions for f in r.families]` enumerates all
regions × families; with `order_by_evidence=True` (`runner.py:288/403`) the scan evidence-tiers **every** pair
before analysis — that pre-work is the ~236 s measured in increment 0, so the deadline hits at `query_seconds=0`.
- Primitive already present: `model/regions.py:279` computes the changed-path set from the context;
`compare_targeted_base` (`runner.py:411`) is a precedent for change-targeted work.
- **Fix locus (increment 2):** restrict the *analyzed* region set to the change's declarations + dataflow
neighbourhood (filter `head_regions` by `context.changed_paths` before the scan, or scope the enumeration inside
`scan_repository`). Keep the **CPG whole-repo** so taint queries can still trace sources from unchanged code into
the changed regions — only the question set is change-scoped.
**Correction (2026-10-11), after a reverted first attempt — two things the naive filter got wrong:**
- **The engine already reaches into unchanged code by design, and it is tested.** A changed `source` that reaches
an unchanged `sink` must still select the unchanged sink's question
(`tests/test_model_change_context.py::test_changed_dependency_reaches_unchanged_ranked_question`;
`test_change_does_not_replace_ranker_selection` pins that change context must NOT alter the ranker's selection).
`_changed_files` feeds `_evidence_pass` only as `contextFilter`/`contextPaths` — which locations to itemise vs
summarise — NOT to shrink the tier population. A "changed files only" scope deletes the reach-into-unchanged
behaviour. Any scope must keep changed regions **plus the regions they reach** (the affected/`relationships`
neighbourhood), not changed files alone.
- **The 236 s is now split (2026-10-11, saved artifacts).** `total=291s build=52s query=0s arbitrate=238.75s`,
and `query_seconds_by_kind["evidence"]=238.54s` — the evidence-tier pass is **99.9 %** of the arbitrate phase;
`affected_context` + `_scope_work` are **0.21 s**. The pass tiered 1,015 pairs (`tier_counts 0:1009, 1:6`) at
**~235 ms/pair** (contended; ~120 ms uncontended) and that alone blew the budget — `selected=0`,
`deferred=1365` all `deadline_exhausted`. Secondary anomaly: the design assumes ~2.5 ms/pair
("17,700 in 44 s"); here it is 50–100× that, i.e. the evidence query is pathologically slow on a cold large
CPG — a separate optimisation from cutting the pair count.
- **So increment 2's single target is the evidence pass** (`scan.py:338-359`, `_evidence_pass`). The fork is HOW
to cut it without deleting reach-into-unchanged:
- (i) **Scope + bound:** order the evidence candidates change-first and bound the pass to a fraction of the
budget so analysis queries still run; reached unchanged regions are tiered only while budget lasts.
- (ii) **Neighbourhood pass:** tier the changed regions, read the reached regions from the context rows the
pass already returns, tier those in a second small pass — exact reach-into-unchanged, more code.
- (iii) **Per-pair speedup:** attack the ~235 ms/pair (warm server / batching) rather than the pair count —
helps every scan but is a deeper engine change and does not by itself bound a huge repo.
**CORRECTION (2026-10-11, after implementing (i) + four live re-verifies — the cost model was wrong).**
Scope + bound was built and tested (change-first ordering, adaptive budget-fitted portioning, `_context_pass`
scoped to changed files, honest `evidence_pass_bounded` degradation; branch `engine-change-scoped-evidence-pass`).
It correctly cut the tiered pair count — but `query_seconds` stayed **0** every time. The live numbers show why:
| pairs tiered | run_batch calls | evidence s | build s |
|---|---|---|---|
| 1015 | 1 | 238 | 52 |
| 512 | 1 | 219 | 68 |
| 64 | 2 | 220 | 68 |
| 4 (probe) | 1 | 160 | 127 |
Evidence time is **flat regardless of pair count** → the cost is **per-*call* fixed** (~150–220 s: loading the CPG
into the query engine + the census the evidenceOnly batch computes), ~85 ms/pair on top. Pair-count scoping
therefore cannot free budget, and portioning re-pays the fixed cost per call (so (i) can be counterproductive).
Build is also large and swings 52→127 s — two engines thrashing the 7 GB host (the record-07 baseline shares it).
The real levers are now (iii)-shaped, not (i): **warm/persistent query engine or a cached census** to kill the
~150 s per-call fixed cost, and/or a **bigger engine-arm budget** than 300 s; plus a **clean (uncontended)
measurement**, because contention alone may be inflating build + fixed cost 2–3×. Scope+bound is NOT merged.
**UNCONTENDED measurement (2026-10-11, baseline paused via SIGSTOP + container kill, then resumed).** Shipped
engine, one evidence call over 1,015 pairs: `build=48 s`, `evidence=243.79 s`, `query=0`. The evidence pass is
**the same ~244 s uncontended as ~238 s contended** → the cost is **INTRINSIC, not contention**; contention only
inflated `build` (48 → 127 s) and added run-to-run noise. The model is `evidence ≈ 150 s fixed (CPG load into the
query engine + the census the evidenceOnly batch computes) + ~93 ms/pair` (244 = 150 + 1015×0.093). Consequences:
- **~200 s of fixed overhead** (`build` + evidence-load) precedes any analysis, so a 300 s budget leaves only
~100 s for everything else. This is why `query=0` even on a clean host.
- **Change-scoping still helps, but only by cutting the per-pair tail** (~1,015×93 ms ≈ 95 s): one evidence call
over ~30 changed pairs is ~150 s + 3 s instead of 244 s, which would free ~90 s — possibly enough for analysis,
but marginal.
- **It must be ONE call, not portioned.** The ~150 s is a first-call load, so the adaptive probe attributed it to
its 4 probe pairs (per-pair≈37 s), concluded the budget fit ~0 more, and tiered only 4 — the implemented
scope+bound is doubly wrong for this cost model. The correct shape: change-first candidates, count-capped, ONE
evidence call, `deadline=None`.
- **Pre-push viability:** the ~200 s fixed floor means the engine arm cannot be a fast (≤30 s) pre-push check on a
repo this size without a warm/persistent session or a cached census. A larger budget (e.g. 600 s) makes the
benchmark arm run but is not a realistic pre-push deadline. The lever that matters for the product is killing
the ~150 s fixed cost, not scoping pairs.
- [ ] **2. Seed analysis from the change.** Prioritize (or restrict first-pass to) the changed declarations plus
their call/data neighborhood over repo-wide static rank, within budget.
- [ ] **3. Soundness.** Keep cross-file dataflow INTO the changed function (don't scope to literal changed lines);
add tests; keep existing push tests green.
- [ ] **4. Live re-test.** Re-run the bedrock change: does analysis now RUN and produce a finding on
`install_packages` (vs `deferred`)? Re-run metrics-plugin.
- [ ] **5. Measure the delta.** Engine-arm sample on the pool (catch/false-alarm + engine-coverage rate) — needs the
kube lane (blocked on the AX completion path) for scale.
## Dependencies / notes
- Coverage `complete_within_scope` also needs the capability registry (Task 8.3) regardless of this work; the win
here is findings moving from `comparison_unknown`/deferred to a decided novelty → advisory engine finding.
- Implement directly and verify on the LIVE instrument (a fake-client/unit pass is not evidence the live engine
changed); the kube-lane and store endpoints are operator-owned infra, not assumptions to make here.
</content>
@@ -1,5 +1,7 @@
# Requirements Document: finding-feedback-loop
> **Status 2026-10-02:** dormant. Requirements approved and design generated on 2026-09-09; the design was never approved and no tasks exist. It depends on `contributor-scan`, which is paused; reopen it with that spec.
## Introduction
Nine engine defects were found in one session against two repositories. **Three were the same bug in three
@@ -0,0 +1,80 @@
# Plan: learn fix mechanisms into the engine (draft for review, 2026-10-04)
Status: approved 2026-10-04 by the maintainer. Increment 1 = discharge facts only; coverage additions (Req 7) come later.
Decisions: minimum 2 proposing repositories before a candidate is tried; admission per language (not pooled by callee name).
## Why
The Joern trace run over exp-005's 501 units shows the engine cannot tell a vulnerable function from its
fix. In the 105 pairs where both sides were analysed, a taint path appears on both sides in 12, on the
vulnerable side only in 2, on the fixed side only in 0, and on neither in 91
(`benchmarks/measurements/2026-10-04-joern-slice-leak-audit/record.json`).
So the fix is invisible to the engine. Either the flow survives and the engine does not know the fix's
mechanism (an `allowed_classes` argument, a path normalisation check, an escaping call), or the engine's
sources and sinks miss the function on both sides. A classifier fed this evidence cannot improve on it.
The corpus now holds 710 complete fix pairs (`2026-10-04-decision-engine-harvest-3/record.json`). Each fix
diff shows what was added. That is the material the engine lacks.
## Goal
Grow the engine's discharge knowledge (sanitizers, guards, safe-argument forms) and, second, its source and
sink coverage, from the fix diffs. Admit a fact only when it improves held-out separation. Spend no model
money.
## Requirements (proposed)
1. **Mining.** From each fix pair, extract what the fixed function adds on the vulnerable flow: new calls,
new conditions guarding the sink, new or changed sink arguments. Record each as a candidate fact with its
language, family, form (sanitizer call, guard predicate, safe argument, sink restriction) and the pairs
that propose it. Mining is deterministic (tree-sitter diff of the two function bodies). No model calls.
2. **Generalisation is explicit.** A candidate is a pattern, not a repository name: a callee name, an
argument key, a predicate shape. Candidates naming project-specific identifiers are rejected (Req 8: no
application-specific trees).
3. **Admission by held-out separation.** Folds are by repository, as in the decision engine. A candidate is
admitted for a fold only if, on that fold's held-out pairs, adding it to the fact tables:
- increases pairs where the vulnerable side keeps a path and the fixed side loses it or becomes
sanitized;
- and does not remove any held-out vulnerable-side path (no recall loss).
A candidate proposed by fewer than 2 distinct repositories in the training folds is not tried.
4. **No train-on-test.** Mining and admission never read held-out pairs; the closed-loop leak recorded in
memory (`closed-loop-train-on-test-leak`) is the failure this rule exists to prevent. Population v3 is
never read.
5. **Measurement by the trace runner.** Each admission round re-runs the engine on the affected pins only
(the runner's resume and selection already support this; on k3s the worker Deployment runs them in
parallel). The outcome is the pair-asymmetry table per family, before and after, on held-out folds.
6. **Gate before production.** An admitted fact enters `src/openultrasast/ruleset/semantic/*.toml` only after
the detection gates (`gate`, `map_gate`, `pair_gate`) show no regression, and only as a registered
experiment whose decision rule was fixed before the run.
7. **Coverage second.** After discharge facts, the same loop proposes source and sink additions from pairs
where neither side has a path (91 of 105 today), admitted under the same held-out rule.
## How it relates to what exists
- `corpus-seeded-mechanisms` already has a closed loop that admits and retracts mechanisms by leave-one-out.
This plan reuses its lever and store, and points it at discharge facts mined from real fix diffs instead
of variant search.
- `benchmarks/pairs/mechanisms.toml` (23 mechanisms) labels what kind of bug a pair is. The new candidates
describe what a fix does. They live in the fact tables, not in the mechanism vocabulary.
## Measures of success
- Pair asymmetry on held-out folds: vulnerable-only paths rise from 2 of 105 toward a substantial share,
with zero fixed-only paths.
- No held-out recall loss on vulnerable sides.
- Then the decision engine is re-run with the new engine evidence (exp-005 design), and the pooled block gate
(exp-004 design) is re-measured.
## Cost and time
- Model cost: $0 (mining and admission are deterministic; the classifier re-run afterwards is a separate,
approved spend).
- Engine time: one pass over affected pins per admission round; about 2 hours per full pass on k3s plus the
VM, about 6 hours on the VM alone.
## Open questions for the maintainer
1. Should coverage additions (Req 7) be in the same increment or a later one?
2. Minimum proposing repositories: 2 is the proposal. Higher is safer, lower finds more.
3. Admission per family and language, or pooled across languages for the same callee name?
@@ -119,6 +119,26 @@ rule hits.
2. Without a trained model available the tools fall back to today's behaviour and say so in the report.
3. Hand-written rules remain as signal producers; changing their enabled/shadow status no longer changes what is
reported unless the engine's decision changes.
4. *(Amended 2026-10-03, maintainer decision; refines the BLOCK rule of 7.3 and 7.4. Tested by exp-004.)* BLOCK
precision is demonstrated pooled across families, not per family:
- **Normalisation**, declared before any run: each family's raw score is put on one scale by a per-family Platt
map (`learn.calibrate.fit_map` with its floors) fitted on training folds only. A family the training folds
cannot map does not block on that fold.
- **t_block**: one threshold on the normalised score for all families, chosen on training folds as the lowest
one whose Wilson 95% lower bound on precision, over the training flags of all families together, is >= 0.95.
It is reported by the Wilson 95% lower bound over all held-out flags of all families, before the floor.
- **Family floor**: a family may block only if its own held-out flags have zero errors and number at least F
(F = 10, reasoned in the exp-004 registration); otherwise that family stays ADVISORY.
- **t_quiet**: per family, the highest threshold keeping >= 90% of the paired positives at or above it (chosen
on training folds, its held-out recall reported); findings below it are silent.
- No held-out label moves a threshold or a normalisation map; candidates whose verdict is `unsure` never block.
5. *(Amended 2026-10-05, maintainer decision, with `search-with-proof`.)* Where a family has an owned demonstration
oracle, the engine is **triage, not the gate**: BLOCK comes only from a verified demonstration (`search-with-proof`
Req 3, 6.3), and the engine's job is to put the true vulnerable function among the top K candidates of a push.
For those families the engine is judged by (a) top-K recall per push (K = 3 to start) on the unseen-repo pool and
(b) its advisory false-alarm rate there; the pooled block gate of 6.4 then governs only families without an
oracle (access control, deserialisation, configuration and secrets), where the engine remains the only signal.
Every arm reports top-K recall next to its catch and false-alarm rates.
### Requirement 7: Built for unseen repositories -- a repo safety net
@@ -136,7 +156,8 @@ rule hits.
ADVISORY, trading precision for recall; both reported with out-of-repository intervals. The decision unit is a
candidate in changed code (`pre-push` delta), with whole-repository scans as a secondary mode.
4. Calibration is checked on held-out repositories; where it does not hold, only ADVISORY is offered for that
family.
family. *(Amended 2026-10-03: once exp-004 adopts the pooled gate of 6.4, BLOCK precision is shown by its held-out
bound and family floor, and calibration is reported, not required.)*
5. Any adaptation to a user's own repository (facts reuse, their dismissals) is opt-in, separate from the
generalisation model, and never contributes to its reported numbers.
@@ -1,7 +1,7 @@
{
"feature_name": "learned-decision-engine",
"created_at": "2026-09-30T14:00:00Z",
"updated_at": "2026-10-01T10:20:00Z",
"updated_at": "2026-10-04T18:00:00Z",
"language": "en",
"phase": "implementation",
"approvals": {
+123 -6
View File
@@ -8,6 +8,10 @@ memory, compiled DSPy-style in-house); tasks 1-5 are unaffected, tasks 6-10 are
against the provider balance read just before it, and none starts with a ceiling above that balance. No test makes
a model or embedding call.
Direction 2026-10-04: `.kiro/steering/roadmap.md` (section of 2026-10-04, phases G0-G5) is the authority. Every
change is judged on the unseen-repository pool (G0, spec `unseen-repo-evaluation`) by catch rate and false-alarm
rate per push. Within-pair AUC stays a development signal. Tasks 11-14 below follow that roadmap.
- [x] 1. Schema and features
- `learn/schema.py` (closed, typed allow-list; no identities; missing != zero), `learn/features.py` (host
builder), new memory `KINDS`, `plane/tasks/features.py` and the `remember` input; tests enforcing the
@@ -45,6 +49,11 @@ a model or embedding call.
- _Requirements: 1.1_
- [ ] 6. Program, local memory and compilation (design sections 4-5)
- Status 2026-10-04: 6.1-6.6 are done. 6.7 is partial: leave-one-source-out, the 0/25/50% memory-size points and
the source-count curve are not run. Around this task the experiment machinery grew: rule arms
(`precision_bound_block`, `pooled_block_gate` with a family floor) in `learn/experiments.py`, the response-cache
fill with a hard ceiling and a dry-run estimate in `learn/fill.py`, and the in-function slice in `learn/slice.py`.
These ran exp-003, exp-004 and exp-005 (7.5-7.7). Memory now holds 710 complete fix pairs (7.9).
- [x] 6.1 Excerpts and the example store (no model)
- `learn/excerpt.py` (function span at the label's pin, bounds 80/4,000 for candidates and 40/2,000 for
examples, numbered lines, no path/repository/commit, advisory ids and security wording in comments
@@ -116,12 +125,63 @@ a model or embedding call.
- _Requirements: 5.1, 5.2, 8.3_
- [ ] 7. Experiments (exp-001 and program variants need budget go-ahead)
- `learn/experiments.py`, `register`, `experiment-outcome`, Run annotation checks, `OUSAST_VERIFY_CALLERS`;
exp-001 (known-callers line on/off, ~$7.7 expected, ceiling $15) registered, run and recorded -- an
inconclusive result is pre-registered as likely.
- Program-variant experiments (paired on candidates, repository cluster, arm A replayed from cache at $0): the
machinery and its test; the first variant (`k = 5` vs `k = 1`, or the confidence source) **(budget go-ahead:
~$0.9-1.35 per arm-B pass of 300 candidates, ceiling $2 each)**.
- [x] 7.1 Machinery (no model in tests): `learn/experiments.py` -- YAML manifests under `plane/experiments/<id>.yaml`
(hypothesis, arms as program-spec overrides or plane `task_env`, unit, pairing, repository cluster, metrics with
`[label=]` selectors and targets, MDE, two looks with O'Brien-Fleming boundaries, budget ceiling, families, folds,
frozen units file), `register` (`experiment` row with both digests; refuses an uncommitted, modified or re-declared
manifest), `units` (freeze the families' evaluation candidates, paired by candidate across pins; no model), `run`
(paired arms in a seeded repository order, a seeded coin per unit for which arm goes first, `arm_outcome` rows,
resumable, arm A replay-only at $0, canary replicates under a fresh cache salt, the first look during the run),
`analyse` (paired repository bootstrap, exact McNemar, the looks, adopt / reject / equivalent / inconclusive,
`experiment_result` row + `benchmarks/experiments/<id>/result.json`), `ousast learn experiment
register|units|run|analyse`; `OUSAST_VERIFY_CALLERS=off` in `plane/tasks/verify.py`; the retrieval-ensemble
sampling variant (`ProgramSpec.sampling = retrieval_ensemble`: `k` samples at temperature 0 over `k` disjoint
retrieved example sets; default unchanged). Tests: `test_learn_experiments.py` (18, scripted clients, FileStore).
- [x] 7.2 Registered, not run: `exp-001-known-callers` (plane arms, `flag_rate[label=1]`, MDE 0.15, ceiling $15;
units `pending` -- model-free candidate generation on the 41 development repositories is not built, and plane Run
annotation checks / the `experiment-outcome` task are not written) and `exp-002-retrieval-ensemble` (arm A today's
k = 5 temperature sampling, arm B the retrieval ensemble; primary canary agreement (target 0.90) and within-pair
AUC, secondary usd per candidate, AUC, ADVISORY recall/precision; families injection and access_control on the
slice-2 candidates; ceiling $4: A $1, B $3). Both registered in `s3://sast-memory` at fbc3220
(`benchmarks/experiments/<id>/registration.json`). exp-002's units are `pending` too: on 2026-10-02 every example
row in the store carried null plane-only instruments (`agree`, `model_sinks`, `verify`) and `load_examples` raised
on them, so `ousast learn experiment units` could not freeze the candidates.
- [ ] 7.3 exp-001 run and recorded (~$7.7 expected, ceiling $15) -- an inconclusive result is pre-registered as
likely; needs the frozen units file first.
- [x] 7.4 exp-002 run and recorded 2026-10-02 (budget go-ahead ~$2 expected, ceiling $4): **reject B**. 160 paired
units (injection 119, access_control 41; 56 repositories; 108 units in 54 candidate pairs), both arms, two
replicates each, no early stop. Primary: canary agreement A 0.875 -> B 0.906 (diff +0.03, CI [-0.03, +0.10],
spans; B meets the 0.90 target), within-pair AUC A 0.815 -> B 0.741 (diff -0.07, CI [-0.13, -0.02], **against B**).
Secondary: AUC -0.02 (CI [-0.04, -0.003], against), ADVISORY recall -0.06 (spans; McNemar 7/2, p = 0.18),
precision +0.01 (spans), usd per candidate A $0.0021 -> B $0.0051 (CI [+0.0027, +0.0032], against: the five
disjoint example sets share only the prompt prefix in DeepSeek's cache). Priced usage A $0.68 (1206 paid calls,
395 replayed), B $1.63 (1350 paid, 250 replayed). Result: `benchmarks/experiments/exp-002-retrieval-ensemble/result.json`,
`benchmarks/measurements/2026-10-02-exp-002/record.json`. Three repairs on the way (each its own commit): example
rows' null plane-only features (1f525ff), the 788 embedding vectors absent from RustFS (copied from the file
store), the result row's column-type collision under S3 Select (780caaa). Further program variants (`k = 5` vs
`k = 1`, the confidence source) as new manifests.
- [x] 7.5 exp-003 run and recorded 2026-10-03 ($0, replay only): **inconclusive**. Selective blocking (a per-family
raw-score threshold chosen on training folds by its Wilson lower bound) offers no BLOCK on held-out paired units
in any family; the incumbent stays. 211 of 353 frozen units evaluated (142 replay misses). Proving 0.95 needs at
least 73 clean flags. Result: `benchmarks/experiments/exp-003-precision-bound-blocking/result.json`.
- [x] 7.6 exp-004 run and recorded 2026-10-03: **inconclusive**. One pooled BLOCK threshold over all families with a
family floor (Req 6.4). No outer fold found a threshold, pooled or per family; B offers no BLOCK. 501 units,
replay at $0 after a cache fill of $1.24 (ceiling $2; the dry-run estimate over-priced 4.5x). Result and record:
`benchmarks/experiments/exp-004-pooled-block-gate/result.json`, `record.json`.
- [x] 7.7 exp-005 run and recorded 2026-10-03: **inconclusive**. The in-function def-use slice in the prompt of the
six slice-2 programs. Within-pair AUC 0.762 -> 0.746 (CI [-0.063, 0.032], spans 0); unsure rate 0.29 -> 0.24
(favours B); usd per candidate slightly higher. A $0 leak audit came first
(`benchmarks/measurements/2026-10-03-graph-slice-leak-audit/record.json`). Result:
`benchmarks/experiments/exp-005-graph-slice/result.json`.
- [x] 7.8 Negatives harvest 2026-10-03 ($0.0014 embeddings, no chat-model call): 98 real fix pairs whose function
survives the fix, from reviewed advisories, as pointer-pair corpus `advisory-fixes-2`; memory 937 -> 1133
examples. Record: `benchmarks/measurements/2026-10-03-decision-engine-negatives-harvest/record.json`.
- [x] 7.9 Harvest 3, 2026-10-04 ($0.003 embeddings, no chat-model call): 251 pairs as `advisory-fixes-3`; complete
pairs in memory 459 -> 710. Record: `benchmarks/measurements/2026-10-04-decision-engine-harvest-3/record.json`.
- [x] 7.10 Joern trace run over exp-005's 501 units and its leak audit, 2026-10-04 ($0): no leak flagged. In the 105
pairs where both sides were asked, a path appears on both sides in 12, on the vulnerable side only in 2, on the
fixed side only in 0, and on neither in 91. The engine cannot see what a fix changed. Record:
`benchmarks/measurements/2026-10-04-joern-slice-leak-audit/record.json`.
- _Requirements: 3.6, 4.1-4.4_
- [ ] 8. Integration
@@ -147,3 +207,60 @@ a model or embedding call.
`result-v3-engine.json` is committed. Nothing between the two commits may touch the program, memory snapshot,
features, priors or thresholds. This spec's code never opens v3 files.
- _Requirements: 5.3_
- [ ] 11. Fix mechanisms (plan approved 2026-10-04; see `fix-mechanisms-plan.md`)
- Roadmap G2 arm (`.kiro/steering/roadmap.md`, section of 2026-10-04): one candidate improvement, not the spine.
It is judged on the unseen-repository pool (G0, spec `unseen-repo-evaluation`) by catch rate and false-alarm
rate per push. **Paused after 11.1 until the G0 pool exists.**
- [ ] 11.1 Mine candidate discharge facts from the 710 fix diffs (sanitizer calls, guarding conditions, safe sink
arguments, safe replacements); deterministic, $0. In progress.
- [ ] 11.2 Admission by held-out separation per outer fold: a candidate turns "path on both sides" into "path on the
vulnerable side only" on folds it was not mined from, removes no held-out vulnerable path, has at least 2
proposing repositories, and is admitted per language.
- [ ] 11.3 Engine re-run on the affected pins (kube-ax lanes plus the VM lane).
- [ ] 11.4 Registered experiment, with the detection gates green, before `ruleset/semantic/*.toml` changes.
- [ ] 11.5 Coverage increment: source and sink additions for pairs where neither side has a path (later).
- [ ] 11.6 TypeScript admission in the engine (branch `engine-typescript`).
- _Requirements: 4.1-4.4, 7.1, 8.1_
- [ ] 12. exp-006: Joern witness slice in the prompt **(budget go-ahead, about $2-4)**
- Planned, roadmap G1/G2. Against the current prompt, after a $0 leak audit, on re-frozen units. Judged on the
unseen-repository pool, not on within-pair AUC alone.
- _Requirements: 4.1-4.4, 7.1_
- [ ] 13. exp-007: the combiner ($0 replay)
- Planned, roadmap G1/G2. Three or four weights (score, engine entailment, neighbour agreement) over cached
responses. Judged on the unseen-repository pool.
- _Requirements: 4.1-4.4, 7.1_
- [ ] 14. Re-run the pooled block gate on re-frozen units
- Planned, roadmap G3. The exp-004 design with the family floor (Req 6.4), together with the pool's false-alarm
rate per family.
- _Requirements: 6.4, 7.3_
- [ ] 15. Trained classifier arm (maintainer decision 2026-10-04: train a model, not hand-set profiles; judged on the
unseen-repo pool, roadmap G1)
- [ ] 15.1 Features: code embedding of the excerpt (stored vectors) + every allow-listed signal; no hand-picked
profile. Automated screening: a signal that separates pair sides by itself on training folds above the leak
threshold is dropped for that fold (the leak audit as a filter, fitted per fold).
- [ ] 15.2 Models: regularised logistic regression and gradient-boosted trees, fitted per repository-held-out outer
fold with inner-fold hyper-parameter search; per-family and pooled variants.
- [ ] 15.3 Scores on held-out folds: within-pair AUC, pooled AUC, top-k precision, Wilson bound at the pooled
threshold; compared with the current LLM programs on the same units.
- [ ] 15.4 Scored on the unseen-repo pool when G0 exists: catch rate at fixed false-alarm budgets.
- _Requirements: 3 (revised 2026-10-04 by maintainer decision), 6, 7_
- [ ] 16. Jointly optimised LLM program arm (judged on the unseen-repo pool)
- Search input fields, retrieval parameters, k, instruction and demonstrations together (DSPy MIPRO-style) on
training folds; no hand-set input profile. Budget go-ahead before paid search.
- _Requirements: 3, 6, 7_
- [ ] 17. Triage objective (Req 6.5, amended 2026-10-05 with `search-with-proof`)
- [ ] 17.1 Metric: top-K recall per push (K = 1, 3, 5) of the true vulnerable function among changed functions,
on held-out folds and on the unseen-repo pool; added to the pool scorer next to catch and false alarm.
- [ ] 17.2 Score the existing arms on it ($0 replay): trained logistic (cheapest per candidate, likely default),
LLM programs (exp-005 A), engine-evidence ordering, and an averaged ensemble.
- [ ] 17.3 Re-scope 14 (pooled block gate) to families without a demonstration oracle; 15.4 and 16 report top-K
recall as their primary unseen-pool number for families with an oracle.
- _Requirements: 6.5, 7_
+67
View File
@@ -0,0 +1,67 @@
# Brief: MCP shift-left agent + skills
Status: brief / not yet specced. Proposed 2026-10-07. Delivery surface for the
qualified safety net; builds on the existing `ousast mcp` server and the product
skills. Related: `proof-arm-opencode` (the proof model), `pre-push-safety-net`
(the gate), `unseen-repo-evaluation` (the qualification).
## Motivation
The pre-push hook is the gate. Developers and their coding agents want the same
security feedback **earlier**, while the code is still being written, not only at
push. And in an agent-driven workflow, a mandatory security check is how a human
oversees agents: the agent's output has to clear it before it ships.
Proposal: expose the safety net through an MCP agent and a small set of skills so
an editor or a coding agent can ask for a security read of the changed code on
demand, shift-left, and so a coding agent can be required to pass it before it
proposes a push.
## Shape
- **Transport:** extend the existing narrow MCP server (`ousast mcp`, stdio) that
OpenCode and other agents already speak. No new network surface; same
no-secret, bounded contract.
- **What the agent offers:**
1. Scan the current diff with the static arm (fast, no model, no cluster) and
return findings scoped to the change, with location and a concrete repair
direction, not a confidence label.
2. Triage a finding: explain it in plain terms and whether it is reachable.
3. Prove on demand: hand the top candidate to the proof arm when a runnable
harness exists, and return the demonstrated proof-of-concept.
- **Skills:** the existing `openultrasast-scan` and `openultrasast-triage`
product skills, plus shift-left additions (scan-the-diff, explain-a-finding,
suggest-a-fix-direction) that a coding agent invokes mid-task.
- **Oversight mode:** a coding agent can be configured to call the scan as a
required step before it finalizes a change, so security review is automatic
rather than remembered.
## Control (same model as the hook)
Advisory by default; blocking is opt-in. Budget and a push/call deadline bound
the work. The inline path runs the fast static arm only; heavy proof is explicit
and lane-selected. Models and spend caps stay in `.env`.
## Open questions for the spec
1. Inline latency budget: in-editor feedback must return in seconds, so the
static arm must be fast on a single diff. What is the time contract, and does
the large-repo latency problem block it?
2. Return contract: exactly what a coding agent gets back (findings, locations,
repair direction, optional proof) and how it decides pass/fail in oversight
mode. The return should be ticket-ready evidence a busy developer can paste
into the issue tracker to win capacity in a prioritization discussion with a
product owner, since a reproducible proof is what moves a security fix out of
the backlog.
3. Does oversight mode need a signed/attestable result so a reviewer can trust
that the agent actually ran the check?
4. Reuse: the `ousast mcp` tool set already exists; which tools are enough and
what is genuinely new.
## Scope and sequencing
This is a delivery and adoption surface, correctly **after** the safety net is
qualified on unseen repositories (G0 through G5). It is not a detection change;
it is how developers and agents consume the detection early. The proof model
improvement lives in `proof-arm-opencode`; this brief is the dev- and
agent-facing feedback channel.
@@ -0,0 +1,10 @@
{
"feature_name": "mcp-shift-left-agent",
"created_at": "2026-10-07T15:10:00Z",
"updated_at": "2026-10-07T15:10:00Z",
"language": "en",
"phase": "brief",
"approvals": {"requirements": {"generated": false, "approved": false}, "design": {"generated": false, "approved": false}, "tasks": {"generated": false, "approved": false}},
"ready_for_implementation": false,
"summary": "MCP agent + skills so developers and coding agents get the safety net's feedback early (shift-left, in-editor) and so a coding agent can be required to pass a security check before it ships. Delivery surface after G0-G5 qualification; builds on the existing ousast mcp server."
}
+33
View File
@@ -0,0 +1,33 @@
# Brief: the plane on a Kubernetes cluster
Maintainer, 2026-10-02: "the plan is to move the whole to a kube system soon. what is in the way?" and earlier
"later i will deploy this in a separate kube env."
## Where we are
The plane already runs on google/ax over Agent Substrate, on a single-node kind cluster on the maintainer's
laptop (`ops/ax/`). Its manifests are ax's own kinds, every task is one digest-pinned runner image, the memory
store is external (S3-compatible RustFS), and provider credentials travel only in the start request. What still
assumes the laptop is small and listed in `docs/deployment.md` and `ops/ax/README.md`:
1. the reconciler is a host CLI process (`ousast plane run`);
2. the artifact receiver is a thread of that process, reached through a Service + EndpointSlice to the kind
gateway; the router is reached through a kubectl port-forward;
3. the kube context `kind-ousast` and the registry `localhost:5001` are hard-coded in `doctor.py`, `egress.py`,
`router.py`;
4. images come from the kind registry;
5. ax's deploy manifest points `AX_SNAPSHOTS_BUCKET` at the ax authors' bucket;
6. the Joern engine runs as a host Docker container (2.8 GB image, 3 GB memory), not as an ax task;
7. the worker pool is two 1 CPU / 1.5 GiB workers with no autoscaling.
## Direction
Make the plane deployable to any Kubernetes cluster that runs Agent Substrate and ax, prove every change on the
kind cluster first (it already behaves like the target), and leave nothing that only works on this host. The
production use (a pre-push safety net fed by CI) needs, in addition, a delta Run command and Workspaces from a
URL and commit; those are in scope as the last increment because they define what the cluster is for.
## Not in scope
Provisioning the maintainer's cluster, Substrate's and ax's own deployment procedures (followed, not changed),
and any detection change.
+426
View File
@@ -0,0 +1,426 @@
# Design Document
> **Status 2026-10-04.** Groups 1-2 are done on kind (records `benchmarks/measurements/2026-10-02-k8s-01-profile/`
> and `2026-10-02-k8s-02-store-delivery/`). Task 2.6 is paused. The production target changed. The operator runs a
> kube-ax cluster (k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes). There we may only submit AX Tasks: no k8s
> object creation, no secrets, no access to `ate-system`. The engine now runs there as AX Tasks through
> `benchmarks/learn/engine_trace_ax.py`. The engine-task image (`plane/Dockerfile.engine-task`) is built FROM
> `ghcr.io/norandom/ax-task-runner:v0.3.1`. Tasks read and write through presigned HTTPS links, are named
> `ousast-engine-*`, and run a 1.7 GB heap with `ActiveProcessorCount=2` and one resident Joern session per pin. A
> pin whose sandbox dies is retried once under a new name, then handed to the VM lane (2.5 GB heap). Egress is
> operator-managed: HTTPS 443 to public hosts is open; LAN and cluster-internal addresses are blocked. Archive
> names (`.tar`, `.gz`, `.zip`, `.bin`) failed through the store host's Cloudflare cache until the operator added a
> cache-bypass rule; presigned objects are staged as `.dat` as a safeguard. Groups 3-8 are **to re-plan** against
> that cluster under roadmap G4 (`.kiro/steering/roadmap.md`, section of 2026-10-04). Their text below is kept
> as written. G4 (detection on push for any repository: `ousast plane scan --base --head`, engine pins on kube-ax,
> model calls on the VM or CI) is the new target for group 7, sized by what G1-G3 keep.
## Overview
The plane keeps its shape: `ousast plane run` is the reconciler, ax is the only executor, every task is one
digest-pinned image under the runner contract, memory is the S3 store. What changes is where the reconciler's
peers live and how they are named. Three facts from the code and the upstream checkouts decide the design:
- **Every host assumption is an address or a process in four places.** `kind-` + `KIND_CLUSTER_NAME` is built in
`doctor.py:39`, `egress.py:115`/`:148` and `router.py:145`; `localhost:5001` in `doctor.py:45-47` and
`up.sh:10`; the receiver is a thread of the CLI (`reconciler.py:350-352`) reached by an EndpointSlice to the kind
gateway (`receiver-service.yaml.tmpl`, `up.sh:55-57`); the router and ax-server are reached by `kubectl
port-forward` (`router.py:146-148`; ax's own CLI does the same, `ax/internal/tunnel/tunnel.go:180-238`).
- **The store is already the one thing every profile shares.** `S3Store` is verified at every open
(`memory.py:608-640`), its `runs/` prefix expires after 30 days (`RUNS_RULE_ID`, `memory.py:589-591`), and no
task reaches it today. A presigned URL is a credential, and the plane already has a channel for credentials that
never touches a manifest or the golden boot: the start request (`router.py`, `start_task`, `runner.py:219-275`).
- **ax-server authenticates nobody; Kubernetes does.** `deploy/ax-server.yaml` runs `--addr=:8080` with no auth
flag; the `ax` CLI tunnels to `svc/ax-server` through kubectl; `kubectl-ate` port-forwards to `ate-api-server`
and sends a Kubernetes ServiceAccount token (`cmd/kubectl-ate/internal/cmd/root.go:59-60`). So the only
authenticated door into either control plane is the Kubernetes API with a kubeconfig.
Hence (maintainer, 2026-10-02): the reconciler stays the CLI and gains an execution mode. `local` is the kind
profile as today; `remote` is the same `ousast plane run` (later `ousast plane scan`) from a laptop or a CI job
against ax in k3s, with task artifacts delivered to the S3 store through presigned URLs minted by the CLI and
carried in the start request, and the CLI polling the store for completion. No in-cluster reconciler is needed
for the move; the receiver thread goes in both profiles once the store path passes its kind proof.
### Goals
- Two profiles of one code base, `kind` and `k3s`; every change proven on kind first (Req 2.4, 7.1).
- No module names a host (`kind-ousast`, `localhost:5001`, `172.19.0.1`); a test enforces it (Req 1.3).
- One delivery path (the store) in both profiles; one credential channel (the start request).
- The Joern engine as a plane task on its own pool (Req 4); a delta Run for a push (Req 6).
### Non-Goals
Provisioning k3s; changing Substrate or ax (their manifests are applied with our values, not edited in place);
any detection, scoring or population change; an unattended in-cluster reconciler (deferred, section 2.4).
## Architecture
```mermaid
flowchart LR
subgraph laptop ["kind profile: laptop"]
cli1["ousast plane run --exec local"]
env1[".env: provider + S3 keys"]
end
subgraph kind ["kind cluster (same host)"]
ax1["ax-server (ax-system)"]
sub1["Substrate: atenet-router, atenet-egress, ate-api"]
pool1["ousast-pool: 2 gVisor workers"]
epool1["engine-pool: 1 worker, 4 GiB"]
reg1["kind registry: images by digest"]
end
s3[("S3 store (RustFS, external): runs/, repos/, facts/")]
cli1 -- "ax CLI tunnel; kubectl-ate; router port-forward" --> ax1
cli1 --> sub1
sub1 --> pool1
sub1 --> epool1
pool1 -- "presigned PUT/GET via egress (TLS 443)" --> s3
epool1 --> s3
cli1 -- "mint URLs, poll, ingest" --> s3
reg1 -.-> pool1
```
```mermaid
flowchart LR
subgraph ci ["k3s profile: laptop or CI job"]
cli2["ousast plane run|scan --exec remote"]
kc["kubeconfig: SA ousast-ci token"]
sec["CI secrets: provider + S3 keys"]
end
subgraph k3s ["k3s cluster"]
api["Kubernetes API (RBAC: portforward in ax-system, ate-system)"]
ax2["ax-server"]
sub2["Substrate router / egress / ate-api"]
pool2["ousast-pool + HPA"]
epool2["engine-pool + HPA"]
end
ghcr["GHCR ghcr.io/norandom: runner + engine images by digest"]
s3b[("S3 store (RustFS)")]
cli2 -- "port-forward tunnels only" --> api
api --> ax2
api --> sub2
sub2 --> pool2
sub2 --> epool2
pool2 -- "presigned PUT/GET" --> s3b
epool2 --> s3b
cli2 --> s3b
ghcr -. "imagePullSecret if private" .-> pool2
```
The two diagrams differ in the kubeconfig principal, the registry, the pool sizes and the HPA. Nothing else.
New code: `plane/profile.py` (the profile), `plane/delivery.py` (presign, poll, extract), `plane/k8s.py`
(manifest rendering per profile), `plane/tasks/engine.py`, `plane/scan.py` (Req 6). `reconciler.py` loses the
receiver wiring and gains a `Delivery` dependency; it stays under the 500 lines `test_plane_reconciler.py:803`
enforces because every new concern is a sibling module, as `router.py` and `egress.py` already are.
## Components and Interfaces
### 1. Configuration surface: `PlaneProfile` (Req 1)
**Decision.** One frozen dataclass `PlaneProfile` in `plane/profile.py`, loaded by `load_profile()` from a TOML
file named by `OUSAST_PLANE_PROFILE` (a name resolved under `ops/k8s/profiles/<name>.toml`, or a path; default
`kind`), with every field overridable by an environment variable of the same upper-cased name
(`OUSAST_KUBE_CONTEXT`, `OUSAST_REGISTRY`, ...). Profiles hold no secret: keys stay in the environment or
`.env`, as today.
| Field | `kind.toml` | `k3s.toml` | Replaces |
| --- | --- | --- | --- |
| `exec` | `local` | `remote` | the implicit host mode |
| `kube_context` | `kind-ousast` | `ousast-k3s` (the maintainer's context name) | `"kind-" + KIND_CLUSTER_NAME` in three modules |
| `registry` | `localhost:5001` | `ghcr.io/norandom` | `doctor.py:45`, `up.sh:10` |
| `images` | `kind-images.json` | `images.json` (from the release, section 8) | the `image:` literals in `plane/tasks/*.yaml` |
| `router_url` | `""` (port-forward) | `""` (port-forward) | `OUSAST_ROUTER_URL` stays as the test override |
| `ax_server` | `""` (ax CLI tunnel) | `""` | `$AX_SERVER` stays as ax's own override |
| `memory` | `s3://sast-memory` | `s3://sast-memory` | `OUSAST_MEMORY` default `file://` is removed for the plane (no fallback) |
| `atespace` | `default` | `default` | `generate.ATESPACE` |
| `snapshots_bucket` | `s3://ax-snapshots` | `s3://ax-snapshots` | ax's `gs://dberkov-gke-dev3/ate-env/` (rendered into our ax-server patch) |
| `pools.default`, `pools.engine` | `{replicas: 2, cpu: 1, memory: 1536Mi}`, `{1, 2, 4Gi}` | documented production values | `workerpool.yaml.tmpl` |
| `image_pull_secret` | `""` | `ghcr-pull` | none today |
**Why a file plus env, not env only.** Eleven values change together per cluster; env-only has no reviewable
record of what "k3s" is (the `ops/ax/README.md` table at lines 193-204 already struggles to list them), and a
CI job would carry eleven exports. The env override keeps the existing tests (`OUSAST_ROUTER_URL`,
`OUSAST_ARTIFACT_HOST`) and one-off experiments working. **Rejected:** a `[plane]` section in the scan config
TOML (the scan config is a user's tool configuration; the plane is a maintainer surface and must not leak into
`ousast scan`); a Kubernetes ConfigMap (there is no in-cluster reconciler to read it).
`doctor()` takes the profile: it checks `kube_context` reachability, the two namespaces, that the `images` file's
runner reference resolves (`docker manifest inspect` for GHCR, the `/v2/` catalog for a local registry), that
`memory` opens (`open_store` verifies the bucket), and prints each configured address (Req 1.2). The literal scan
(section 9) covers `src/`, `ops/`, `plane/` and `docs/` except the profile files themselves and
`ops/ax/up.sh`, which may name kind because it creates it.
### 2. Execution mode: the CLI with `--exec local|remote` (Req 2)
**Decision.** `ousast plane run RUN.yaml [--exec local|remote]` (default from the profile). Both modes run the
same `reconciler.run`: render, `ax apply`, egress policy, resume, start, await, delete. What differs is
addressing and principal, and none of it is new client code:
| Call | Through | Principal in `remote` |
| --- | --- | --- |
| `ax apply/resume/get/delete` (`reconciler.Ax`) | ax's own tunnel: `kubectl port-forward -n ax-system svc/ax-server` (`tunnel.go:203-238`), context from the profile | kubeconfig for SA `ousast-ci` |
| `kubectl ate create/get/update egress-policy` (`egress.Egress`) | kubectl-ate's own port-forward to `ate-api-server`, bearer = the SA token (`root.go:59-60`) | same SA; Substrate's authorization of that token for `egress-policy` is an open question (below) |
| `POST /ousast/v1/start` (`router.open_router`) | `kubectl port-forward svc/atenet-router` in `ate-system`, as today | same SA |
| artifacts, inputs, state | the S3 store (section 3) | the store's agent key from the CI secrets |
**Auth: kubeconfig with a ServiceAccount token, not an ingress.** ax-server has no authentication
(`deploy/ax-server.yaml:71-79`: `--addr=:8080`, nothing else) and Substrate's router routes on a header. Exposing
either through Traefik would publish an unauthenticated control plane behind whatever middleware we bolt on;
tunnelling through the Kubernetes API keeps one authenticated, audited door and uses the clients as they ship.
**Rejected:** Traefik IngressRoute for ax-server and the router (unauthenticated upstreams, a second auth layer to
design, and ax's CLI would still tunnel); a bespoke ax HTTP client (Req boundary: no replacement of the `ax` CLI
beyond what addressing needs, and the tunnel already handles addressing).
**RBAC (`ops/k8s/base/rbac-ci.yaml`).** ServiceAccount `ousast-ci` in `ax-system`; Role in `ax-system` and in
`ate-system`: `pods` get/list, `services` get, `pods/portforward` create; nothing cluster-wide, no verbs on
Substrate's or ax's CRDs (both are reached through their servers, not through the Kubernetes API). The token is a
bound TokenRequest (`kubectl create token ousast-ci --duration=...`) stored as the CI secret `KUBECONFIG`. Secrets
for the provider and the store stay where the reconciler runs: `.env` on a laptop, repository secrets in CI,
Kubernetes Secrets only on the day an in-cluster reconciler exists (2.4). The start-request path is unchanged
(Req 2.3).
**Run state of record moves to the store.** `state.json` is written locally under `$OUSAST_RESULTS/plane/<run>/`
as today and mirrored to `runs/<run>/state.json` on the store after each change; a rerun from another machine
seeds its local copy from the store. This is what lets CI job N+1 resume CI job N's Run (Runs are long,
resumable and artifact-driven: `reconciler.py:272-312`).
**Kind keeps `local` (Req 2.4).** Same command, same manifests, local registry, laptop as reconciler; the only
differences are the profile's values. The kind proof of each increment is run in `local`, and increment 3 runs
`remote` against kind from a container that holds only the kubeconfig and the secrets (section 7).
#### 2.4 Deferred: an in-cluster reconciler
Not needed for the move. It would add: survival of a laptop sleeping or a CI job timing out mid-Run, scheduled
Runs, and Secrets injected by Kubernetes. It would cost: a workload (Job per Run is the natural shape, keeping
the CLI path identical), a PVC or store-only state, a submit path, and RBAC for ax's and Substrate's Services from
inside the cluster. Revisit when unattended Runs are wanted; the store-backed state above is its prerequisite.
### 3. Delivery: presigned URLs in the start request, both profiles (Req 3)
**Decision.** The runner PUTs `output.tar` to a presigned S3 URL and GETs each declared input from a presigned
URL; the CLI mints them and polls the store. The URLs travel in the start body, never in env:
```json
{"run": "...", "task": "...", "credentials": {"DEEPSEEK_API_KEY": "..."},
"delivery": {"put": "<presigned PUT runs/<run>/<task>/output.tar>",
"inputs": {"CANDIDATES": "<presigned GET runs/<run>/<producer>/<artifact>>"}}}
```
- **Why the start body.** A presigned URL is a bearer capability, so the credentials-only-in-the-start-request
rule applies; env would bake it into the ActorTemplate and hand it to the golden boot; and the body has no
32 KB ceiling (Substrate rejects env values over 32768 characters, `reconciler.py:141`). `StartGate.offer`
(`runner.py:250-275`) validates `delivery` the way it validates `credentials`: `https` URLs only, hostnames,
no echo. Expiry = `OUSAST_TASK_TIMEOUT` (7200 s) + 600 s; a re-start after a re-resume re-mints.
- **Why presigned rather than a scoped delivery credential.** A per-task S3 key needs a per-task policy on RustFS,
gives the sandbox `ListBucket` and reads beyond its inputs, and would travel in the same body. Presigned URLs
name exactly the objects a task may write and read, which is what the receiver enforces today
(`Receiver.input_file`, `egress.py:230-247`).
- **Why not an in-cluster receiver Service.** It is a server to run, holds state in a process, and is reachable
only from a pod; that is the gap remote mode has. **Rejected.**
- **Objects.** `runs/<run>/<task>/output.tar` (the runner's PUT); after the CLI extracts it, the declared
`outputs[]` are re-put as `runs/<run>/<task>/<artifact>` so a consumer's presigned GET names one artifact, and
`runs/<run>/<task>/done.json` carries the summary's status. The 30-day `runs/` expiry already applies; memory
rows are ingested from the local extraction by `remember` as today (f: annotations and `memory_key` seeding
unchanged, `generate.py:_chain_inputs`, `memory.seed`).
- **Completion.** `_await`'s `delivered()` becomes `Delivery.delivered(task)`: a HEAD on `output.tar`, polled at
`OUSAST_POLL_SECONDS` (3 s). The runner's completion marker logic (`runner.py`, `_run_once`) is unchanged.
- **Inputs no task produces** (`candidates.json`, `functions.json`, `case.json`) may also be store objects
declared as inputs, which lifts the ~20 KB inline-files limit (`ops/ax/README.md:117`); Workspace `files`
stay for the smoke Task and small cases.
**One path for both profiles (d).** The store is required and verified in both; keeping the receiver for kind
would keep `receiver_address`, `OUSAST_ARTIFACT_DIAL`, the `Host`-dial trick and the EndpointSlice alive for one
profile and leave the kind proof proving a path k3s never runs. The receiver thread, `receiver-service.yaml.tmpl`,
`receiver_cluster_ip` and the `up.sh` step go after increment 2's proof (the `harnessx-removal` sequencing rule:
nothing deleted before its replacement passes). Tests that used `OUSAST_ARTIFACT_HOST` move to a local HTTP PUT
target that accepts presigned-shaped URLs.
**Egress per task (c).** `policy_for` drops the `http` rule and emits one `tls_passthrough` rule on 443 for the
sorted union of the store endpoint's hostname (`urlsplit(S3_ENDPOINT).hostname`, IPs rejected as today), the bound
Workspaces' Git hosts (`git_hosts`) and the bound Model's `openultrasast.io/egress-hosts`. A model-free task's
policy names the store and its Git hosts only. The gateway already passes TLS to the Model host by hostname, so
the store host is the same mechanism (`egress.py:7-16`). Open question: the store endpoint must be a DNS name
the gateway resolves; `files.because-security.com` is; an in-cluster RustFS would need a resolvable Service name.
### 4. The engine as a plane task (Req 4)
**Decision.** `plane/tasks/engine.py` runs in the Joern image under the runner contract, on a second WorkerPool
`engine-pool`. The root `Dockerfile` gains the two lines `Dockerfile.runner:25-29` has (the `ax-task-runner`
shim and `/workspace`), so one *contract* in two images: `ousast-runner` (slim, PyYAML) and `ousast-engine`
(JRE + Joern + php-cli, ~2.8 GB). **Rejected:** Joern inside the single runner image (every task would pull
2.8 GB and every golden snapshot carry it; the engine runs for a minority of cases); Docker-in-pod (no socket on
containerd; `Dockerfile:1-7` already argues against it).
- **Task.** Template `plane/tasks/engine.yaml`: `command: ["engine"]`, env `OUSAST_WORKSPACE_DIR`,
`OUSAST_FIXED_DIR`, `OUSAST_INPUT_CASE`, pins and `OUSAST_FAMILIES`, bound to the same two Workspaces the
`alerts` task binds (`generate.py:342-353`). The module lifts `benchmarks/push/finding_dump.py`'s scan into
`openultrasast.cpg.dump` (the image ships `src/`, not `benchmarks/`) and reuses `engine_alerts.engine_rows`,
`read_record` and the `alerts` helpers (`scan`, `coverage`, `write_rows`, `write_summary`). It emits the same
`alerts.jsonl` rows (`rule_id: engine:<family>`, `source: "engine"`) and `summary.json` (coverage with the
engine's languages, files and bytes read, questions, seconds, degradations per pin) as `alerts-engine`, with
`host: false`. The instrument rule stays: no `read N files` line or `questions == 0` fails the task loudly.
- **Sizing from the record** (`benchmarks/measurements/2026-09-30-php-engine-alerts/record.json`): container
peak 2.129 GiB under a 3 GiB limit, 789 s for both pins of froxlor, deadline 1800 s per pin. Task resources:
requests 3 Gi / 1 CPU, limits 4 Gi / 2 CPU; `engine-pool` workers 4 Gi / 2 CPU, replicas 1 on kind (the 7 GB
host holds it only with `ousast-pool` at one worker), production replicas in `k3s.toml`. Task timeout: two pins
times 1800 s plus grace fits `OUSAST_TASK_TIMEOUT` 7200.
- **Placement is the open question of this increment.** ax's templates set no worker selector
(`workerpool.yaml.tmpl:1-3`), so Substrate picks any free worker of the sandbox class. The kind proof must show
a 3 Gi actor lands only on `engine-pool`. If Substrate does not place by resource fit, the fallback is a
separate atespace with its own pool if Substrate scopes pools that way; failing that, the engine stays
`alerts-engine` on a Docker host and the k3s engine becomes a plain Kubernetes Job outside ax, recorded as such.
**Probe outcome (task 1.5, 2026-10-02, `benchmarks/measurements/2026-10-02-k8s-01-profile/`): `placement: any`.**
With `ousast-pool` at one 1536Mi worker and a `probe-pool` of one 3Gi worker both ACTIVE, a `repo-facts` actor
requesting 2560Mi (limit 3Gi) was placed on the 1536Mi `ousast-pool` worker (`kubectl ate get workers`: ACTORS
1/1000 there, 0/1000 on `probe-pool`), and so was the 256Mi actor. Substrate does not place by resource fit, so
group 5 takes the fallback: the engine pool must be the only pool its actors can land on. Before 5.2, check whether
Substrate scopes pools by atespace (an `engine` atespace with `engine-pool` as its only pool); if it does not, the
k3s engine runs as a plain Kubernetes Job outside ax and `alerts-engine` stays the kind path.
- **`alerts-engine` stays** for development on a host with Docker (kind profile), its help text and
`docs/plane.md` row marked "development only; the plane task is `engine`" (Req 4.3). Generated Runs choose
`engine` when the profile's `images` file has an engine image, else leave `alerts` for `alerts-engine`.
### 5. `ops/k8s/` layout (Req 5)
```
ops/k8s/
profiles/kind.toml, k3s.toml # section 1; k3s values documented, not secrets
base/
workerpool-default.yaml.tmpl # replicas, limits, nodeSelector from the profile (render: plane/k8s.py)
workerpool-engine.yaml.tmpl # engine-pool, same template, engine values
ax-server-snapshots.yaml # a strategic-merge patch setting AX_SNAPSHOTS_BUCKET (applied over ax's deploy)
rbac-ci.yaml # SA ousast-ci; Roles + RoleBindings in ax-system and ate-system (section 2)
secrets.example.env # the variable names only: DEEPSEEK_API_KEY, OPENROUTER_API_KEY, S3_*, AWS_*
k3s/
imagepullsecret.yaml.tmpl # ghcr-pull from a GitHub PAT, referenced by the pools' pod template
hpa-default.yaml, hpa-engine.yaml # External metric ate_workerpool_workers{state=at_capacity}, averageValue 0.7
prometheus-adapter.yaml # from substrate/demos/autoscaled-workerpool, namespace ax-system
render.sh # ousast plane manifests --profile <p> --out DIR; kubectl apply -f DIR
```
`ops/ax/up.sh` keeps creating kind and calls `render.sh kind` for the pools; its receiver step is deleted in
increment 2. **gVisor on k3s is a correction to the requirements text:** Substrate does not use a Kubernetes
`RuntimeClass`. The `ateom-gvisor` worker (`workerImage: ko://.../cmd/ateom-gvisor`) fetches the gvisor release
tarball named by `manifests/ate-install/sandboxconfig-gvisor.yaml` (`gs://gvisor/releases/nightly/...`) and runs
`runsc` itself inside the worker pod (`cmd/ateom-gvisor/actorpath.go:35-45`). The node checklist is therefore:
PodSecurity in `ax-system` admits the worker pod's security context (recorded from kind in increment 6), the
node reaches the tarball URL or a mirror named in a copied SandboxConfig, and `/var/lib/ate` sits on local-path
storage with room for golden actors (24 MB each, `ops/ax/README.md:112`). Autoscaling is k3s-only: the HPA
needs Prometheus and the adapter (the Substrate demo README says kind-only for the demo; we take its manifests),
and the 7 GB host cannot scale anyway. The queueing consequence of one actor per worker is recorded in the
runbook: a Run with `--workers N` beyond the pool's replicas queues in ax as Suspended until a worker frees.
### 6. `ousast plane scan <url> --base <c> --head <c>` (Req 6)
`plane/scan.py`, reusing `generate.py`'s render path with a new case source:
1. **Workspace generation** (`generate.scan_cases(url, base, head, cache)`): a shallow clone into the case cache
(`runner.find_cached_checkout` layout), two pinned Workspaces `<repo_dir>-<head7>` and `<repo_dir>-<base7>`
(`workspaces.py` shape, pins in `openultrasast.io/git-commits`), repository key by `memory.repo_key`.
2. **Candidates from changed functions**: `generate.fix_ranges(repo, base, head, side="new")` gives the head-side
hunks (the `push/` delta primitive `_diff_records` is the same `git diff -U0`), `repo_facts.enclosing` names
the function per hunk line, filtered to `repo_facts.product_files`. Quick rules on the changed files
(`tasks.alerts.scan`) add alerted functions; `roles` runs once per repository when memory has no `roles` rows
for it; `repo-facts` supplies callers. `candidates.json` and `functions.json` are store inputs (section 3).
3. **Run composition**: per changed-function set `facts -> va, vb -> agree -> vc -> final -> features ->
remember`, plus `engine` when a changed file's language is engine-covered and not quick-covered, or PHP
(`ENGINE_ALONGSIDE_QUICK`). Budgets from `generate.verify_budget` (0.022 USD per candidate, headroom 1.5);
`--ceiling` caps the sum. `--all` (whole repository, secondary) takes every product function.
4. **Results** land in memory under `repos/<host>__<owner>__<name>/<head>.jsonl` through `remember` and
`ousast plane remember` (rows already carry `repo` and `pin`, `memory.py:77`); `runs/<run>/` holds the raw
artifacts for 30 days. The report prints the attribution table and the plane's agreement (`final`); the
decision engine's verdict only when a model is adopted (learned-decision-engine Req 6 unchanged).
5. **Latency split (e).** `ousast pre-push` stays the blocking hook: quick rules and the delta engine inside
`OUSAST_PUSH_DEADLINE` (30 s, `ops/README.md:322`). `ousast plane scan --exec remote --no-wait` submits and
prints the Run name in the hook's report; `--wait SECONDS` blocks for CI. `ousast plane status <run>` reports
later. The wrapper `ops/pre-push` gains an opt-in `OUSAST_PLANE_SCAN=1`; it never blocks on the plane.
### 7. Increment order, kind proof and measurement
| # | Increment | Kind proof (`local` unless stated) | Recorded under `benchmarks/measurements/` |
| --- | --- | --- | --- |
| 1 | `PlaneProfile`, literals replaced, `doctor` by profile | `ousast plane doctor` passes with `kind.toml`; literal scan 0 | `<date>-k8s-01-profile/record.json`: doctor lines per profile, scan count |
| 2 | Store delivery + receiver-less egress; receiver removed after | `ops/ax/smoke-run.sh` delivers `facts.json` via the store; policy has one TLS rule | `-02-store-delivery/`: bytes, PUT-to-poll seconds, rendered policies |
| 3 | `--exec remote` from a CI-shaped container (kubeconfig + secrets only) against kind | smoke Run completes; audit log lists only the Role's verbs | `-03-remote-exec/`: kube audit verbs vs Role, run status |
| 4 | Images on GHCR, digests, repin (section 8) | kind pulls `ghcr.io/norandom/ousast-runner@sha256` (pull secret if private); smoke Run | `-04-ghcr-images/`: sizes, pull seconds per image |
| 5 | `engine` task + `engine-pool` | froxlor case as `engine`: rows equal the 2026-09-30 host record (6/6), placement on engine-pool shown | `-05-engine-task/`: rows diff, seconds, peak memory, placement |
| 6 | Pools per profile, HPA manifests, node checklist | kind pools rendered and applied; worker securityContext recorded; HPA renders (k3s-only live) | `-06-pools/`: footprint table, securityContext |
| 7 | `plane scan` on a real pair (a PHP CVE pair from the dev corpus) | Run completes; rows under `repos/<repo>/<head>` | `-07-plane-scan/`: candidates, cost, agreement, rows |
| 8 | Runbook (`docs/deployment.md`), checklist mapped to R1-R7 | n/a (documentation; every "planned" label removed only with a recorded run) | the checklist links the records above |
### 8. GitHub Actions and re-pinning
`.github/workflows/images.yml` on tags `v*` (separate from `release.yml`, which builds the wheel): build
`plane/Dockerfile.runner` as `ghcr.io/norandom/ousast-runner` and the root `Dockerfile` as
`ghcr.io/norandom/ousast-engine` with `docker/build-push-action`, tag by version, and write the two digests to a
release asset `images.json` (`{"runner": "ghcr.io/...@sha256:...", "engine": "..."}`) and to job outputs.
`generate.repin_templates` (`generate.py:484-497`, one image for all templates) becomes `repin(plane, images)`:
`engine.yaml` gets `images["engine"]`, every other template `images["runner"]`; `ousast plane repin images.json`
exposes it and `--runner-image FILE` keeps working for a single-image file. Private packages: the pools' pod
template names `imagePullSecret` from the profile.
### 9. Tests
- `tests/test_plane_literals.py`: scans `src/`, `plane/`, `ops/` (minus `ops/ax/up.sh`, `ops/k8s/profiles/`) and
`docs/` for `kind-ousast`, `localhost:5001`, `172.19.`, `127.0.0.1:` and `port-forward` outside `router.py`
and the ax tunnel note; built at runtime like `test_removed_plane_references.py`.
- `tests/test_plane_profile.py`: defaults, file + env precedence, unknown key rejected, no secret field, both
committed profiles load.
- `tests/test_plane_k8s.py`: render both profiles; kind has no HPA and no pull secret; k3s has both; engine pool
limits >= the engine task's limits; every image reference is digest-pinned.
- `tests/test_plane_rbac.py`: the rendered Role's verbs equal the set of Kubernetes API calls the CLI makes
(fake `kubectl`/`ax`/`kubectl-ate` record their argv; the test maps them to `pods/portforward`, `pods`,
`services`) and nothing more.
- `tests/test_plane_egress.py`: for every committed Task template, `policy_for` yields one TLS rule whose hosts are
exactly the store host, the bound Git hosts and the Model hosts; no `http` rule; an IP endpoint is refused.
- `tests/test_plane_delivery.py`: start body with `delivery` accepted; URLs never logged; expiry; PUT and GET
against a local server; `delivered()` polling; declared-only inputs.
- `tests/test_plane_engine_task.py`: fixture record -> rows and summary equal `engine_alerts.produce`'s shape;
the instrument failures.
- `tests/test_plane_scan.py`: candidates from a two-commit fixture repository; Run composition; budgets.
- **Only live on kind:** placement on `engine-pool`; Substrate accepting the SA token for `egress-policy`;
the gateway passing TLS to the store host; GHCR pulls; the HPA (k3s only). These are the kind proofs above.
## Data Models
Store layout additions: `runs/<run>/state.json`, `runs/<run>/<task>/output.tar`, `runs/<run>/<task>/<artifact>`,
`runs/<run>/<task>/done.json`, `runs/<run>/inputs/<name>`; all under the existing 30-day `runs/` rule. Start body:
`credentials` as today plus optional `delivery {put, inputs{name: url}}`. `PlaneProfile` as in section 1.
`images.json` as in section 8.
## Error Handling
A profile naming an unreadable context, a bucket that fails `verify_bucket`, or an images file without a digest
fails before `ax apply` with the field named. A presigned PUT that fails three times ends the task `failed` with
the HTTP status (`runner.deliver` semantics kept); an expired URL after a re-resume is re-minted by the next
start. A store HEAD error during polling is retried with the resume backoff and reported after ten consecutive
failures, like `ax get` (`reconciler.py`, `_await`). A `delivery` URL that is not `https` to a hostname is a 400
from `StartGate.offer`. The engine task fails loudly on an unread input, as `read_record` does.
## Requirement Traceability
| Req | Components | Tests | Kind proof / record |
| --- | --- | --- | --- |
| R1 nothing assumes this host | `profile.py`, `doctor.py`, `egress.py`, `router.py`, `ops/k8s/profiles/` | literals, profile | increment 1 |
| R2 reconciler in the cluster -> CLI with `--exec remote`; kind first-class | `reconciler.py` (`Delivery`), `Ax` with context, `rbac-ci.yaml`, store-backed state | rbac, profile | increments 3, 1 |
| R3 receiver-less artifacts and egress | `delivery.py`, `runner.py` start body and PUT/GET, `egress.policy_for` | delivery, egress | increment 2 |
| R4 engine as a task | `tasks/engine.py`, `cpg/dump.py`, `Dockerfile` shim, `engine.yaml`, `engine-pool` | engine task, k8s | increment 5 |
| R5 sizing and autoscaling | `k8s.py`, `ops/k8s/base`, `ops/k8s/k3s` | k8s | increment 6 (HPA live only on k3s) |
| R6 delta Run | `scan.py`, `generate.scan_cases`, `ops/pre-push` opt-in | scan | increment 7 |
| R7 proven on kind, documented | every increment's record; `docs/deployment.md` | n/a | increment 8 |
## Open Questions
1. **Substrate authorization for the CI token.** `kubectl-ate` sends a ServiceAccount token; which audience and
which Substrate-side grant let `ousast-ci` create `egress-policy` in `default` is not in the checkouts read
here. Proven or refuted in increment 3.
2. **Engine placement.** Whether Substrate places a 3 Gi actor only on workers that fit it (section 4). The
fallback is named.
3. **gVisor tarball reachability on k3s** (`gs://gvisor/...` in the SandboxConfig): mirror or egress needed.
4. **Store endpoint hostname** for any future in-cluster RustFS; today's external endpoint is fine.
5. **Two images under one contract** versus the maintainer's "one runner image for all tasks": the design
reads it as one contract; confirm.
6. **GHCR visibility** (public packages need no pull secret) and whether Substrate's and ax's `ko` images go to
the same registry namespace.
7. **Reconciler size**: remote mode and `Delivery` must stay under the 500-line test; if they do not, the test's
bound moves with a reason, not silently.
@@ -0,0 +1,135 @@
# Requirements Document
## Introduction
The plane moves from this laptop's kind cluster to a Kubernetes cluster of the maintainer's choosing without
changing what runs: the same ax manifests, runner image, memory store and credential path. Every host-bound
piece becomes a cluster resource or a configuration value, each change is proven on kind first, and the
production entry point (a delta Run for a push) is added. Context: `brief.md`, `docs/deployment.md`.
## Boundary Context
- **In scope**: two profiles of one code base, `kind` (local, kept) and `k3s` (production); a remote execution mode of the CLI; artifacts delivered to the store; configurable
cluster context, registry and image pins; our own snapshot and memory buckets; the Joern engine as a plane task
on a larger worker pool; worker pool sizing and autoscaling; a delta Run command and Workspaces from URL +
commit; a deployment runbook with a proof on kind and a checklist for the target cluster.
- **Out of scope**: provisioning or operating the target cluster; modifying Agent Substrate or ax; any change to
detection, scoring, populations or the decision engine; replacing the `ax` CLI with a bespoke client beyond what
remote submission needs.
- **Adjacent expectations**: `ai-service-plane` requirements (one Model + budget per task, attribution,
credentials only in the start request, egress policies, verify-at-startup memory store) keep holding;
`harnessx-removal` Req 6 (memory on the S3 store) keeps holding; nothing is deleted before its replacement
passes its test (the `harnessx-removal` sequencing rule).
## Target Environment (maintainer, 2026-10-02)
- **Cluster: k3s.** Consequences: containerd (no Docker socket; the engine's host-Docker path does not exist there),
gVisor needs `runsc` and a `RuntimeClass` installed on the nodes Substrate's workers run on, k3s ships Traefik
and local-path storage, and single-node k3s is close enough to kind that the kind proof transfers.
- **Images and source: GitHub.** Images are published to GitHub Container Registry (`ghcr.io/norandom/...`),
built by GitHub Actions from tagged commits, digest-pinned in the manifests; private packages need an
`imagePullSecret` on the worker ServiceAccounts and the CI principal. Substrate's and ax's own images are built with
`ko` from their checkouts and pushed to the same registry.
## Requirements
### Requirement 1: Nothing assumes this host
**User Story:** As the maintainer, I want every host-bound assumption replaced by a cluster resource or a
configuration value, so the plane runs the same on kind and on the target cluster.
#### Acceptance Criteria
1. The kube context, the registry (GHCR in production), the ax-server address, the router address, the receiver address and the
snapshot and memory buckets are configuration (environment or one config file), with the kind values as the
documented defaults for development; no module names `kind-ousast`, `localhost:5001` or `172.19.0.1`.
2. `ousast plane doctor` checks the configured cluster, not kind by name, and reports each configured address.
3. A test enumerates the source tree for host-only literals and fails on any.
### Requirement 2: The reconciler stays the CLI, with a remote execution mode
**User Story:** As the maintainer, I want to trigger the checks in the cluster from the CLI and get the results
fed back, so that remote execution is a mode of the same tool, not a separate service.
#### Acceptance Criteria
1. `ousast plane run` (and the later `ousast plane scan`) takes an execution mode, `local` or `remote`, from the
profile or a flag; `remote` submits the Run to ax-server in the target cluster (over the cluster's ingress or a
kubeconfig), `local` is today's kind path. The manifests, the memory-key seeding, the budgets and the
attribution table are identical in both modes.
2. In remote mode the CLI needs no inbound path: it polls the memory store for task completion and artifacts and
prints the same status, attribution table and verdicts; `ousast plane status <run>` reports a Run started
earlier from any machine with the store credentials.
3. Provider and store credentials come from the caller's environment (a laptop `.env`, a CI secret), travel only
in the start request, and never appear in a manifest, an ActorTemplate or a log.
4. The principal the CLI uses against the cluster has the least privilege the ax API needs and nothing against
Agent Substrate directly; a test compares the rendered role with the calls the CLI makes.
5. kind stays a first-class local execution mode, not a stepping stone: `ousast plane run` from a laptop
against the kind cluster keeps working with the development profile (same code, same manifests, local
registry), and every release is proven on it before anything is applied to k3s.
6. An unattended in-cluster reconciler (a Deployment or Job that runs without a caller) is a later option, not
part of this spec.
### Requirement 3: Artifacts land in the store, not on a host
**User Story:** As the maintainer, I want task artifacts delivered to the memory store so that no host has to
be reachable from the cluster.
#### Acceptance Criteria
1. Tasks deliver their artifacts to the S3 memory store (presigned URLs minted by the CLI and carried in the
start request, or a scoped delivery credential; the design chooses one and states why) and the CLI reads
completion from the store. The receiver thread and the kind gateway EndpointSlice remain only if the design
shows the local profile needs them; one delivery path for both profiles is preferred.
2. Egress policies grant each task exactly the hosts it needs (the store's endpoint, git hosts, the bound
Model's host) and nothing else; a test renders the policy for every committed Task.
### Requirement 4: The engine is a plane task
**User Story:** As the maintainer, I want Joern to run where the other tasks run, on a worker sized for it.
#### Acceptance Criteria
1. A second WorkerPool sized for the engine (memory and CPU from a measured run) exists alongside the default
pool; an `engine` task module wraps today's host engine run and emits the same `alerts.jsonl` rows and
coverage as `alerts-engine`.
2. The engine image is digest-pinned in the registry like the runner image; its size and pull time are recorded.
3. `alerts-engine` on the host remains available for development and is marked as such.
### Requirement 5: Sizing and autoscaling
**User Story:** As the maintainer, I want the pool to follow the load of pushes without manual resizing.
#### Acceptance Criteria
1. Worker pool replicas, limits and the autoscaling rule (Substrate's assigned-worker signal) are manifests under
`ops/k8s/` with the kind values as a small profile and a documented production profile.
2. The runbook records the measured footprint per task type and the queueing consequence of one actor per
worker.
### Requirement 6: A delta Run for a push
**User Story:** As a user with a repository the project has never seen, I want a push checked by the plane.
#### Acceptance Criteria
1. `ousast plane scan <repo-url> --base <commit> --head <commit>` generates Workspaces from the URL and commits,
derives candidates from the changed functions (quick rules, inferred roles, repository facts), and submits a
Run of facts, verify a/b, tie-break and agree with per-task budgets; a whole-repository mode is secondary.
2. Results are written to the memory store under the repository and head commit and reported with the
attribution table; the decision engine's verdict is reported only when a model is adopted, else the plane's
agreement (ai-service-plane Req 6 and learned-decision-engine Req 6 unchanged).
3. The latency split is explicit: a blocking pre-push carries quick rules and the delta engine within its
deadline; plane verdicts arrive asynchronously.
### Requirement 7: Proven on kind, then documented for the target
**User Story:** As the maintainer, I want each step shown working on kind before I touch the real cluster.
#### Acceptance Criteria
1. Every requirement above is exercised end to end on the kind cluster (remote mode included, with the kind cluster as the remote target), with
a recorded run under `benchmarks/measurements/`.
2. `docs/deployment.md` becomes the runbook: prerequisites for the target cluster (Substrate, ax, registry,
buckets, Secrets, node requirements for gVisor workers), the profile switch, and a checklist whose every item
maps to a requirement here; planned items are labelled planned until their run is recorded.
+22
View File
@@ -0,0 +1,22 @@
{
"feature_name": "plane-on-kubernetes",
"created_at": "2026-10-02T20:00:00Z",
"updated_at": "2026-10-04T18:00:00Z",
"language": "en",
"phase": "implementation",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": true
}
+295
View File
@@ -0,0 +1,295 @@
# Implementation Plan
> **Status 2026-10-04.** Groups 1-2 are done on kind (records `benchmarks/measurements/2026-10-02-k8s-01-profile/`
> and `2026-10-02-k8s-02-store-delivery/`). Task 2.6 is paused. The production target changed. The operator runs a
> kube-ax cluster (k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes). There we may only submit AX Tasks: no k8s
> object creation, no secrets, no access to `ate-system`. The engine now runs there as AX Tasks through
> `benchmarks/learn/engine_trace_ax.py`. The engine-task image (`plane/Dockerfile.engine-task`) is built FROM
> `ghcr.io/norandom/ax-task-runner:v0.3.1`. Tasks read and write through presigned HTTPS links, are named
> `ousast-engine-*`, and run a 1.7 GB heap with `ActiveProcessorCount=2` and one resident Joern session per pin. A
> pin whose sandbox dies is retried once under a new name, then handed to the VM lane (2.5 GB heap). Egress is
> operator-managed: HTTPS 443 to public hosts is open; LAN and cluster-internal addresses are blocked. Archive
> names (`.tar`, `.gz`, `.zip`, `.bin`) failed through the store host's Cloudflare cache until the operator added a
> cache-bypass rule; presigned objects are staged as `.dat` as a safeguard. Groups 3-8 are **to re-plan** against
> that cluster under roadmap G4 (`.kiro/steering/roadmap.md`, section of 2026-10-04). Their text below is kept
> as written. G4 (detection on push for any repository: `ousast plane scan --base --head`, engine pins on kube-ax,
> model calls on the VM or CI) is the new target for group 7, sized by what G1-G3 keep.
One task group per increment of `design.md` section 7, in that order; a later group never starts before the
earlier group's kind proof is recorded. Every task ships with its tests; a task is done when its tests pass in
the host suite, ruff and mypy are clean, and nothing in `src/`, `plane/`, `ops/` (minus `ops/ax/up.sh` and
`ops/k8s/profiles/`) or `docs/` names `kind-ousast`, `localhost:5001`, `172.19.` or a port-forward address
(task 1.3 makes that a test). Nothing is deleted before its replacement passes its proof. Each group ends with
its kind proof and the record it writes under `benchmarks/measurements/<date>-k8s-NN-<slug>/record.json`
(commit, images, the instrument checks, the numbers). Steps that call a model are marked **paid: needs a budget
go-ahead** with a ceiling; steps that need a hand on a cluster that does not exist yet are marked **manual**.
- [x] 1. Profile and literals (design section 1; open-question checks 2, 3, 7)
- [x] 1.1 `PlaneProfile`
- `src/openultrasast/plane/profile.py`: frozen dataclass with the fields of design section 1; `load_profile()`
reads `ops/k8s/profiles/<OUSAST_PLANE_PROFILE>.toml` (a name or a path; default `kind`), then applies
`OUSAST_<FIELD>` overrides; rejects an unknown key, a secret-looking key (`*_KEY`, `*_SECRET`), an `exec`
outside `local|remote`, an `images` file without digest pins. `ops/k8s/profiles/kind.toml` and `k3s.toml`
committed with the design's values; `ops/k8s/profiles/kind-images.json` from `~/.cache/ousast/ax-src/runner-image`.
- `tests/test_plane_profile.py`: defaults; file then env precedence; each rejection names the field; both
committed profiles load; no field holds a value matching a credential pattern.
- _Requirements: 1.1_
- [x] 1.2 Literals out of `doctor.py`, `egress.py`, `router.py`, `generate.py`
- `doctor(profile)`: context from `profile.kube_context`; the registry check resolves the `images` file's runner
reference (`docker manifest inspect` for a remote registry, `/v2/_catalog` for a local one); `open_store`
verifies `profile.memory`; prints every configured address (Req 1.2). `Egress(context=...)`,
`open_router(context=...)` and `receiver_cluster_ip(context)` take the profile's context; `generate.ATESPACE`
becomes `profile.atespace`. `cli.py`: `ousast plane doctor|run|workspaces|harvest` accept `--profile`.
- `tests/test_plane_reconciler.py`, `test_plane_router.py`, `test_plane_egress.py`: the fakes receive the
profile's context, never a built one.
- Evidence: `grep -rn "kind-" src/openultrasast/plane` matches only comments citing the kind profile.
- _Requirements: 1.1, 1.2_
- [x] 1.3 Host-literal scan test
- `tests/test_plane_literals.py`: pattern built at runtime (as `tests/test_removed_plane_references.py` does)
over `src/`, `plane/`, `ops/` minus `ops/ax/up.sh` and `ops/k8s/profiles/`, and `docs/`, for `kind-ousast`,
`localhost:5001`, `172.19.`, `127.0.0.1:` and `port-forward` outside `router.py` and lines tagged
`ax-tunnel`; fails listing `path:line`.
- Evidence: the test passes on this commit and fails when a literal is reinserted (shown once in the record).
- _Requirements: 1.3_
- [x] 1.4 `up.sh` by profile
- `ops/ax/up.sh`: `KO_DOCKER_REPO` and the context from `kind.toml` (read with `python -m
openultrasast.plane.profile --print registry|kube_context`); writes `kind-images.json` instead of
`runner-image`; `generate.repin_templates` accepts that file (runner key only until task 4.3).
- Evidence: `ops/ax/up.sh` reruns idempotently on kind; `ops/ax/smoke-run.sh` passes.
- _Requirements: 1.1, 2.4_
- [x] 1.5 Placement probe (open question 2, decided before the engine work) -- result: `placement: any`
- `ops/ax/probe-placement.sh`: applies a second WorkerPool `probe-pool` (1 replica, 3 Gi) from the design's
engine template, then one model-free `repo-facts` Task with `resources.requests.memory: 2560Mi`, and records
with `kubectl ate get workers` which worker took the actor; then the same Task with 256Mi. Removed after.
- Evidence: `record.json` says whether Substrate places by resource fit (`placement: fit|any`). `any` selects
the design's fallback for group 5 and is written into `design.md` section 4 before group 5 starts.
- _Requirements: 4.1_
- [x] 1.6 Reconciler bound (open question 7) -- 489 lines after 1.2; projected 514 during group 2: the bound moves to 550 in 2.4
- Record `wc -l src/openultrasast/plane/reconciler.py` (477 at 17ac7ad) and the lines task 2.4 may add
(receiver wiring removed: `-12`, `Delivery` calls: `+8` estimated). If the estimate crosses 500, the bound
in `tests/test_plane_reconciler.py:803` moves to 550 in task 2.4 with the reason in the assertion message.
- Evidence: the numbers in `record.json`.
- _Requirements: 2.1_
- [x] 1.7 gvisor tarball reachability (open question 3) **manual** -- documented as planned in `docs/deployment.md` B.1; the live check waits for k3s
- `docs/deployment.md` (B.1, planned): the k3s node must fetch
`gs://gvisor/releases/nightly/2026-09-02/<arch>/gvisor.tar.zstd` (`substrate/manifests/ate-install/sandboxconfig-gvisor.yaml`)
or a mirrored copy named in a copied SandboxConfig; the check is `kubectl -n ate-system logs` of the worker
pod's atelet fetch line, run by hand on the day k3s exists. Labelled planned until then.
- _Requirements: 7.2_
- [x] 1.8 Kind proof and record 01
- `ousast plane doctor --profile kind` passes every line; `ousast plane doctor --profile k3s` fails only on the
unreachable context (the expected line); `test_plane_literals` 0 offenders; the probe of 1.5; the bound of 1.6.
- `benchmarks/measurements/<date>-k8s-01-profile/record.json`.
- _Requirements: 1.1, 1.2, 1.3, 7.1_
- [ ] 2. Store delivery, receiver-less egress (design section 3)
- [x] 2.1 Presigned URLs and the store run directory
- `src/openultrasast/plane/delivery.py`: `Delivery(store, run)` mints presigned PUT for
`runs/<run>/<task>/output.tar` and GET for `runs/<run>/<producer>/<artifact>` (boto3 `generate_presigned_url`
on `S3Client.sdk`, expiry `OUSAST_TASK_TIMEOUT + 600`); `delivered(task)` HEADs the tar; `collect(task)`
downloads, extracts under the run dir with the member checks `Receiver.accept` has, re-puts declared
`outputs[]` as single objects, writes `done.json`; `put_input(name, path)` for inputs no task produces;
`state.json` mirrored to `runs/<run>/state.json` and seeded from it on a rerun.
- `tests/test_plane_delivery.py`: against a local S3-shaped fake (PUT/GET/HEAD by key); URLs never appear in
logs; expiry; member rejection; declared-only re-put; state seeding.
- _Requirements: 3.1, 2.1_
- [x] 2.2 Runner: `delivery` in the start body
- `src/openultrasast/plane/runner.py`: `StartGate.offer` validates an optional `delivery {put, inputs}` (https,
hostname, no echo); `_deliver_output` PUTs the tar to `delivery.put` (same retry and `DeliveryError`
semantics); `fetch_inputs` GETs from `delivery.inputs[name]`; `OUSAST_ARTIFACT_URL` and `OUSAST_ARTIFACT_DIAL`
stay honoured until task 2.6 so both paths run in the same build.
- `tests/test_plane_runner.py`: accepted body with `delivery`; 400 on an `http` or IP URL; PUT to a local
server; inputs fetched from presigned-shaped URLs; the marker and retry-only boot unchanged.
- _Requirements: 3.1_
- [x] 2.3 Egress without the receiver
- `src/openultrasast/plane/egress.py`: `policy_for(task, workspaces, model, store_host)` emits one
`tlsPassthrough` rule on 443 over the sorted union of store host, Git hosts, Model hosts; no `http` rule; an
IP store host raises naming `S3_ENDPOINT`.
- `tests/test_plane_egress.py`: for every committed template under `plane/tasks/` the rendered policy is the
expected set; the IP refusal.
- _Requirements: 3.2_
- [x] 2.4 Reconciler on `Delivery`
- `src/openultrasast/plane/reconciler.py`: `run()` opens the store first (no fallback), builds `Delivery`,
passes `delivery` to `router.starter`/`start_task` (body extended in `router.py`), `_await` on
`Delivery.delivered`, outcome from the collected `summary.json`; the receiver thread is kept behind
`OUSAST_DELIVERY=receiver` for this group only. Line bound per task 1.6.
- `tests/test_plane_reconciler.py`: the ax fake run completes through the delivery fake; a rerun from an empty
results root resumes from the store's `state.json`.
- _Requirements: 2.1, 3.1_
- [x] 2.5 Kind proof and record 02
- `ops/ax/smoke-run.sh` with `OUSAST_DELIVERY=store`: `facts.json` and `summary.json` arrive through the store;
the actor's policy read back has one TLS rule naming the store host; `ousast plane status ax-e2e-smoke`
prints the table. Instrument: the tar's byte size from the store equals the extracted files' total.
- `benchmarks/measurements/<date>-k8s-02-store-delivery/record.json`: bytes, PUT-to-poll seconds, rules.
- _Requirements: 3.1, 3.2, 7.1_
- [ ] 2.6 Remove the receiver (last; after 2.5 is recorded) -- paused 2026-10-04
- Delete `Receiver`, `_Handler`, `receiver_address`, `receiver_cluster_ip`, `RECEIVER_HOST`, `RECEIVER_PORT`
from `egress.py`; `OUSAST_ARTIFACT_URL/DIAL/HOST/PORT` from `runner.py` and `reconciler.py`; the
`OUSAST_DELIVERY` switch; `ops/ax/receiver-service.yaml.tmpl` and the `up.sh` receiver step; the EndpointSlice
from kind (`kubectl delete`). Docs: `docs/plane.md` sequence diagram, `docs/deployment.md`, `ops/ax/README.md`.
- Tests using `OUSAST_ARTIFACT_HOST` move to the delivery fake; `test_plane_literals` covers `172.19.`.
- Evidence: `ops/ax/smoke-run.sh` passes again on the build without the receiver; recorded in 02's record as
`receiver_removed: <commit>`.
- _Requirements: 3.1, 1.3_
- [ ] 3. `--exec remote` from a CI-shaped principal (design section 2; open-question check 1) -- to re-plan (kube-ax, roadmap G4)
- [ ] 3.1 Execution mode and RBAC manifests
- `cli.py`: `ousast plane run --exec local|remote` (default `profile.exec`); `reconciler.Ax` passes the
profile's context to `ax` (its context flag, read from `ax --help` and pinned in a test); `remote` refuses a
`file://` store. `ops/k8s/base/rbac-ci.yaml`: SA `ousast-ci` in `ax-system`; Roles and RoleBindings in
`ax-system` and `ate-system` with `pods` get/list, `services` get, `pods/portforward` create, nothing else.
- `tests/test_plane_rbac.py`: fake `kubectl`, `ax`, `kubectl-ate` record argv over a full fake Run; the
verbs implied (`port-forward` -> `pods/portforward` create + `pods` get + `services` get) equal the rendered
Role's rules exactly.
- _Requirements: 2.1, 2.2_
- [ ] 3.2 Substrate's authorization of the SA token (open question 1)
- `ops/ax/probe-ci-token.sh`: `kubectl create token ousast-ci --duration=2h` into a kubeconfig; `kubectl ate
--token-file - get workers` and `create egress-policy` on a throwaway actor with that kubeconfig. If refused,
read `substrate/docs/api-guide.md` for the grant (atespace role or token audience) and add the manifest to
`ops/k8s/base/rbac-ci.yaml`; if no grant exists for a non-admin token, record it and the design's remote mode
uses the maintainer's admin kubeconfig for the proof, with the gap listed in the runbook.
- Evidence: `record.json` names the grant used (`grant: <kind/name>` or `admin-kubeconfig`).
- _Requirements: 2.1_
- [ ] 3.3 CI-shaped container
- `ops/k8s/ci-shell.Dockerfile`: `python:3.12-slim` + this package + `kubectl`, `kubectl-ate`, `ax` (built from
the checkouts); no `.env`; runs with `-e KUBECONFIG=/run/kc -e DEEPSEEK_API_KEY -e S3_*` only.
`.github/workflows/plane-remote.yml` (dispatch only, no schedule): the same image running `ousast plane run
--exec remote --profile k3s` with repository secrets; disabled until k3s exists (labelled in the file).
- _Requirements: 2.2, 2.3_
- [ ] 3.4 Kind proof and record 03 **paid: needs a budget go-ahead, ceiling 0.50 USD**
- From the container of 3.3 against kind (`kind.toml` with `exec = "remote"`): the smoke Run (model-free), then
one `verify` pass over one case of `plane/runs/validation-46.yaml` (budget `{usd: 0.5, calls: 60}`) so the
credential path is exercised remotely. Kube audit log (kind apiserver `--audit-policy` of the SA's verbs)
lists only the Role's verbs.
- `benchmarks/measurements/<date>-k8s-03-remote-exec/record.json`: audit verbs vs Role, grant of 3.2, run
status, usd spent.
- _Requirements: 2.1, 2.2, 2.3, 2.4, 7.1_
- [ ] 4. Images on GHCR (design section 8) -- to re-plan (kube-ax, roadmap G4)
- [ ] 4.1 `images.yml`
- `.github/workflows/images.yml` on tags `v*`: `docker/build-push-action` for `plane/Dockerfile.runner` ->
`ghcr.io/norandom/ousast-runner:<tag>` and the root `Dockerfile` -> `ghcr.io/norandom/ousast-engine:<tag>`;
digests as job outputs and as the release asset `images.json` (`{"runner": "...@sha256:...", "engine": ...}`);
`GITHUB_TOKEN` with `packages: write`.
- Evidence: a dry run on a `v*-rc` tag publishes both and the asset (the engine build is ~2.8 GB: record the
build minutes).
- _Requirements: 1.1, 4.2_
- [ ] 4.2 Engine image under the runner contract
- Root `Dockerfile`: the `ax-task-runner` shim and `/workspace` as `plane/Dockerfile.runner:25-29`; the `s3`
extra is not needed in either image (the runner PUTs by URL).
- `tests/test_plane_images.py`: both Dockerfiles declare the shim and entrypoint (text check).
- _Requirements: 4.2_
- [ ] 4.3 `repin` for two images
- `generate.repin(plane, images)`: `engine.yaml` (added in 5.1; until then absent is fine) takes
`images["engine"]`, every other template `images["runner"]`; `ousast plane repin images.json`;
`--runner-image FILE` keeps the single-image file working. `ops/k8s/profiles/k3s.toml` names `images.json`
downloaded from the release.
- `tests/test_plane_increment.py`: repin with a two-key file; a template with two image lines is refused.
- _Requirements: 1.1, 4.2_
- [ ] 4.4 Kind proof and record 04
- Pools' pod template takes `imagePullSecret` from the profile (`ops/k8s/base/workerpool-*.tmpl`, task 6.1
brings the full template; here the field only); kind pulls `ghcr.io/norandom/ousast-runner@sha256:...` with
`ghcr-pull` if the package is private; the smoke Run passes on the GHCR image.
- `benchmarks/measurements/<date>-k8s-04-ghcr-images/record.json`: image sizes, pull seconds, visibility.
- _Requirements: 1.1, 4.2, 7.1_
- [ ] 5. The engine as a plane task (design section 4) -- to re-plan (kube-ax, roadmap G4)
- [ ] 5.1 `cpg/dump.py` and `tasks/engine.py`
- Lift `benchmarks/push/finding_dump.py`'s scan into `src/openultrasast/cpg/dump.py` (the script becomes a
thin caller; `ops/README.md:78` command unchanged). `src/openultrasast/plane/tasks/engine.py`: env
`OUSAST_WORKSPACE_DIR`, `OUSAST_FIXED_DIR`, `OUSAST_INPUT_CASE`, pins, `OUSAST_FAMILIES`, `OUSAST_DEADLINE`;
per pin the quick scan (`tasks.alerts.scan`) and the engine in-process; rows via `engine_alerts.engine_rows`,
the instrument checks via `read_record`'s rules; `summary.json` as `engine_alerts.produce` writes it with
`host: false`. `plane/tasks/engine.yaml`: `command: ["engine"]`, requests 3Gi/1, limits 4Gi/2, image from
`images["engine"]`. `generate.TEMPLATES` gains `engine`; `--loop` emits `engine` instead of `alerts` for a
case needing the engine when the images file has an engine key.
- `tests/test_plane_engine_task.py`: a fixture record -> rows and summary equal to `produce`'s shape; the
unread-input and zero-question failures; the generator's choice.
- _Requirements: 4.1_
- [ ] 5.2 `engine-pool`
- `ops/k8s/base/workerpool-engine.yaml.tmpl` rendered by `plane/k8s.py` (task 6.1 generalises; here the engine
values): kind replicas 1, 4Gi/2; if 1.5 recorded `any`, apply the design's fallback and write it here.
- Evidence: `kubectl -n ax-system get workerpool engine-pool` ready on kind with `ousast-pool` at 1 replica.
- _Requirements: 4.1, 5.1_
- [ ] 5.3 `alerts-engine` marked development-only
- `cli.py` help, `docs/plane.md` task table, `ops/README.md`: "development only (needs Docker on the host);
the plane task is `engine`". No code removed.
- _Requirements: 4.3_
- [ ] 5.4 Kind proof and record 05
- The froxlor case of `validation-46` as an `engine` task on kind (model-free, deadline 1800 s per pin):
rows equal `benchmarks/measurements/2026-09-30-php-engine-alerts/` (6/6 per pin), placement on `engine-pool`
shown by `kubectl ate get workers`, peak memory from `kubectl top`.
- `benchmarks/measurements/<date>-k8s-05-engine-task/record.json`: rows diff, seconds per pin, peak memory,
placement, image size and pull seconds.
- _Requirements: 4.1, 4.2, 7.1_
- [ ] 6. Pools, autoscaling, node checklist (design section 5) -- to re-plan (kube-ax, roadmap G4)
- [ ] 6.1 `plane/k8s.py` and the manifests
- `src/openultrasast/plane/k8s.py`: `render(profile, out)` writes `workerpool-default.yaml`,
`workerpool-engine.yaml`, `ax-server-snapshots.yaml` (patch with `profile.snapshots_bucket`), `rbac-ci.yaml`;
for `k3s` also `imagepullsecret.yaml`, `hpa-default.yaml`, `hpa-engine.yaml`, `prometheus-adapter.yaml`
(copied from `substrate/demos/autoscaled-workerpool`, namespace `ax-system`, metric
`ate_workerpool_workers{state=at_capacity}`, `averageValue 0.7`). `ousast plane manifests --profile P --out
DIR`; `ops/k8s/render.sh`. `ops/ax/workerpool.yaml.tmpl` deleted once `up.sh` uses the renderer.
- `tests/test_plane_k8s.py`: both profiles render; kind has no HPA and no pull secret; k3s has both; engine
pool limits >= `engine.yaml` limits; every image digest-pinned; no host literal in the output.
- _Requirements: 5.1_
- [ ] 6.2 Node checklist and footprint
- `docs/deployment.md`: the gVisor correction (no RuntimeClass; `ateom-gvisor` runs `runsc` from the
SandboxConfig tarball), PodSecurity for the worker pod's securityContext (recorded from kind), local-path for
`/var/lib/ate`, the measured footprint per task type (`verify`, `repo-facts`, `engine` from 05) and the
queueing consequence of one actor per worker with `--workers`.
- _Requirements: 5.2, 7.2_
- [ ] 6.3 Kind proof and record 06 (HPA: render only; live is k3s-only, **manual** later)
- `render.sh kind` applied on kind replaces the hand template with identical pools (`kubectl diff` empty but
for labels); worker pod `securityContext` captured; `render.sh k3s` validates with `kubectl apply
--dry-run=client` against kind's API (CRDs present).
- `benchmarks/measurements/<date>-k8s-06-pools/record.json`: footprint table, securityContext, dry-run result.
- _Requirements: 5.1, 5.2, 7.1_
- [ ] 7. `ousast plane scan` (design section 6) -- to re-plan (kube-ax, roadmap G4)
- [ ] 7.1 Workspaces and candidates from URL + commits
- `generate.scan_cases(url, base, head, cache, *, all=False)`: shallow clone into the case cache, two pinned
Workspaces, head-side hunks via `fix_ranges(side="new")`, functions via `repo_facts.enclosing` over
`product_files`, quick alerts via `tasks.alerts.scan` on the changed files, `candidates.json` and
`functions.json` as store inputs (`Delivery.put_input`); `--all` takes every product function.
- `tests/test_plane_scan.py`: a two-commit fixture repository yields exactly the changed functions; a rename
and a deleted function are handled; `--all` count.
- _Requirements: 6.1_
- [ ] 7.2 Run composition and the command
- `src/openultrasast/plane/scan.py` + `cli.py`: `ousast plane scan URL --base C --head C [--all] [--ceiling USD]
[--exec] [--wait S | --no-wait]`: per set `facts -> va, vb -> agree -> vc -> final -> features -> remember`,
`engine` when a changed file's language needs it, `roles` once per repository when memory has no `roles`
rows for `repo_key(url)`; budgets from `verify_budget`; Run annotated `openultrasast.io/population: scan`,
`split: <repo_dir>`; after the Run, `memory.ingest`, then the attribution table and the plane's agreement
(`final`), the decision engine's verdict only when adopted. `--no-wait` prints the Run name and exits 0.
- `tests/test_plane_scan.py`: the rendered Run for the fixture; budgets sum under the ceiling; `roles`
included only when the store lacks rows; the report's verdict line.
- _Requirements: 6.1, 6.2_
- [ ] 7.3 Hook opt-in
- `ops/pre-push`: with `OUSAST_PLANE_SCAN=1`, after the blocking check, `ousast plane scan <remote url>
--base <old> --head <new> --exec remote --no-wait` and the Run name in the report; the hook's exit code never
depends on it. `ops/README.md` and `docs/deployment.md`: the latency split.
- `tests/test_push_hook.py`: the wrapper calls the scan only with the variable set and ignores its failure.
- _Requirements: 6.3_
- [ ] 7.4 Kind proof and record 07 **paid: needs a budget go-ahead, ceiling 3.00 USD**
- A PHP pair from the development corpus (pmpro's CVE commit and its parent, never a v2 or v3 case) scanned on
kind: candidates, the Run through `final`, `engine` for the PHP files, rows under
`repos/<host>__<owner>__<name>/<head>.jsonl` on the store, the report; then `ousast plane status`.
- `benchmarks/measurements/<date>-k8s-07-plane-scan/record.json`: candidates, usd, calls, agreement, rows,
seconds from submit to verdict, whether the known CVE site is among the agreed.
- _Requirements: 6.1, 6.2, 6.3, 7.1_
- [ ] 8. Runbook (design section 7, Req 7) -- to re-plan (kube-ax, roadmap G4)
- [ ] 8.1 `docs/deployment.md` as the runbook
- Sections: prerequisites for k3s (Substrate, ax with the snapshots patch, GHCR and the pull secret, buckets,
the CI SA and its grant from 3.2, the node checklist of 6.2), the profile switch, `--exec remote` from a laptop
and from CI, and the checklist whose every item names its requirement and its record 01-07; items without a
recorded run (HPA live, 1.7's fetch) labelled planned. `ops/ax/README.md` and `ops/README.md` point to it;
the "moving to a separate cluster" tables are replaced by the checklist.
- _Requirements: 7.2_
- [ ] 8.2 Spec closure
- `design.md` open questions updated with the outcomes of 1.5, 1.6, 3.2 (1.7 stays open until k3s); the
`kind-tooling` memory and `.kiro/steering/overview.md` current-state line updated; `/kiro-spec-status`.
- Evidence: every record 01-07 linked from the runbook's checklist.
- _Requirements: 7.1, 7.2_
@@ -67,7 +67,7 @@ real capability; task 8.3 still owns actual eligibility. Task 4.2 brings progres
## Task 4.3 verification
The agent service rejected the task 4.3 dispatch with `agent thread limit reached`.
The [kiro-impl skill](../../../.opencode/skills/kiro-impl/SKILL.md) explicitly provides:
The [kiro-impl skill](../../../.agents/skills/kiro-impl/SKILL.md) explicitly provides:
“If multi-agent is not available, fall back to manual mode execution for all tasks.”
Implementation, review and completion checks therefore ran in the main context.
This is a manual review record, not an independent-review claim.
@@ -7,7 +7,7 @@ Real capability eligibility and rollout readiness remain unproven.
## Execution and review mode
Fresh implementer dispatch for 5.1 returned `agent thread limit reached`.
The [Kiro implementation skill](../../../.opencode/skills/kiro-impl/SKILL.md) explicitly
The [Kiro implementation skill](../../../.agents/skills/kiro-impl/SKILL.md) explicitly
states: “If multi-agent is not available, fall back to manual mode execution for all tasks.”
Its manual review rule permits review in the main context when a fresh reviewer is unavailable.
This run therefore uses manual implementation and adversarial review, not independent review.
@@ -147,6 +147,43 @@ with Express-style routes and middleware, including vulnerable, fixed and benign
frontend/framework adaptations and new abstract capabilities, and shall report C/C++ envelope or
unsupported-property results separately from demonstrated vulnerability detection.
## Requirement 9: Operate as an advisory safety net on any repository
Added 2026-10-02, maintainer-approved, from the operational-readiness audit. The maintainer's
definition: a user installs the hook on ANY repository and gets honest, advisory-only results;
nothing ever blocks a push unless blocking was explicitly chosen.
9.1 (added 2026-10-02, maintainer-approved) When the installed hook cannot run the check, because the
`ousast` executable is missing or an argument or setting is invalid, the hook shall allow the push and
print one line saying the check did not run and how to fix it; only an explicit `--mode blocking`
opt-in may turn an analysis result into a rejected push.
9.2 (added 2026-10-02, maintainer-approved) When a check is skipped or incomplete, the terminal output
shall name each distinct reason in plain words, one line per reason, with repository paths decoded and
no opaque hash tokens; a language with no engine frontend and no quick rules shall be named with its
file count; a first push of a new branch shall say there was no base to compare against and what was
checked instead. The JSON artifact keeps the raw reason values.
9.3 (added 2026-10-02, maintainer-approved) Before the engine, within the same deadline, the Safety Net
shall run the shipped quick pattern rules over the head side of the changed lines and show their
findings as advisory pattern matches, labeled separately from admitted alerts. This amends 3.2 for this
tier only: quick-rule findings are shown, never admitted, and never block.
9.4 (added 2026-10-02, maintainer-approved) When a finding's operation lies in a file that is new in the
head revision, the comparison shall classify it as new rather than unknown; an engine finding whose
novelty is established but whose capability is not yet qualified shall be shown as an advisory engine
finding, not an admitted alert. Genuinely unknown comparisons keep their vetoes.
9.5 (added 2026-10-02, maintainer-approved) When the engine cannot finish within the deadline, one line
shall say so; on explicit request (`--engine background`) the Safety Net shall start the engine
detached after the push, record its result in the artifact directory, and show it on the next run as
the engine result for that head from the previous push.
9.6 (added 2026-10-02, maintainer-approved) The hook installer and remover shall ship in the wheel
(`ousast pre-push install|uninstall`), refuse to overwrite an existing hook unless forced, and a
hook-callable container wrapper shall pass the push input and artifact path through; the documentation
shall give a first-run guide, a glossary of skip reasons, a CI recipe and help text for every flag.
## Accepted experimental acceptance profile
These values are the initial implementation/evaluation profile accepted for task generation, not
+52 -2
View File
@@ -475,7 +475,7 @@ Preserve all completed task evidence below.
- _Depends: 13.1_
- _Requirements: 6.2, 6.3, 8.1, 8.2_
- [ ] 14. Ask every declared family whether it establishes anything (precondition for M4)
- [x] 14. Ask every declared family whether it establishes anything (precondition for M4)
- Seven families are declared and two were proven end to end. A declared family that has never been asked whether it finds anything is advertised coverage, and M4 cannot qualify a capability that establishes nothing.
- [x] 14.1 Census the seven declared families on NodeGoat and VAmPI
- Run production discovery and scan over each whole checkout and report, per family, the regions that offered it, the questions asked, the questions completed, and what was found with the witness and rung. No comparison, no novelty, no admission: detection only.
@@ -575,7 +575,7 @@ Preserve all completed task evidence below.
- _Boundary: Evaluation harness, full-run integration_
- _Depends: 13.2, contributor-scan 7.3, contributor-scan 7.8_
- _Requirements: 6.2, 6.3, 8.1, 8.2_
- [ ] 13.4 Let a question inside one partition complete, whatever else the repository holds
- [x] 13.4 Let a question inside one partition complete, whatever else the repository holds
- **Diagnosed 2026-09-21, and PHP was never the variable.** A repository holding TWO languages can never complete a question, so it can never admit a finding. `partitions.py` raises `cross_partition_semantics_unresolved` whenever more than one language partition exists; that reason is a member of `GRAPH_INTEGRITY_GAPS`; and a graph-integrity gap demotes every completed outcome in the scan.
- The evidence is one column wide: NodeGoat is JavaScript alone and completes 299 of 299, VAmPI is Python alone and completes 35 of 35, PMPro is PHP with 36 regions of bundled JavaScript and completes 0 of 127 and 0 of 520 while establishing 19 and 43 findings. NodeGoat's own `vendor_semantics_unresolved` is a DECLARED exclusion rather than an integrity gap, which is why it does not demote.
- **This is larger than the PHP runtime limit and it lands on the declared v0.1 target.** A WordPress plugin is PHP plus its admin JavaScript, so every one of them is multi-language, so on the target this tool was built for it can detect defects and never admit one. Neither subject that currently passes could reveal it.
@@ -587,6 +587,7 @@ Preserve all completed task evidence below.
- _Three causes, not one, and the third was hidden by the record rather than by the code._ The same census gap is raised in TWO passes, arbitration and evidence, and carrying the partition in one but not the other left an unscoped copy that kept demoting everything, so the fix looked applied and was not. It was found by making the census print each outcome's reason instead of only its status, after which three separate causes stopped looking identical.
- _Admission is scoped too._ `_blocking_degradations` already exempted declared exclusions; it now also ignores a gap another language partition raised, so a JavaScript build artefact cannot block a PHP finding. A gap naming no partition still reaches everything, and so does every gap when the claim's own language is unknown, because unknown scope is not evidence of independence. Evidence: `benchmarks/measurements/2026-09-21-partition-gap-blast-radius.json`.
- _Boundary: contributor-scan query/context evidence, Ranker Scope Contract_
- _Checked 2026-10-02 per the operational-readiness audit:_ implemented and verified on PMPro (120 of 127, above). 13.3 stays open on its own blocker; this box was left unchecked only because it waits on it.
- _Depends: 13.3_
- _Requirements: 2.1, 2.2, 6.3, 8.1_
@@ -619,6 +620,55 @@ Preserve all completed task evidence below.
- _Requirements: 8.1, 8.2_
- [x] 17. Operational readiness (2026-10-02)
- **Named by the 2026-10-02 operational-readiness audit**, maintainer-approved as the first increment. The maintainer's definition: a user installs the hook on ANY repository and gets honest, advisory-only results; nothing ever blocks a push. The audit found the plumbing sound (snapshots, resolution, deadline accounting, advisory exit 0, the JSON artifact) and five things in the way: the hook can block a push, quick rules are not part of pre-push, the engine cannot finish a 26-file repository inside 30 s, the terminal hides every skip reason, and there is no supported install path without the source checkout. Before numbers for every item are the audit's live runs on records (Python), serve-static (JavaScript) and unrolled/secure (Go), each with one planted commit.
- This group is plumbing and honesty. It enables no capability, edits no rule for a planted sample, and leaves graph reuse (the engine speed work decided in 11.6) as a later task.
- Record: `benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json` (see 17.7).
- _Done 2026-10-02:_ all seven subtasks done. The engine still does not fit the 30 s hook deadline (graph reuse, the 11.6 direction, stays a later task), and plane verdicts after the push were not in this increment.
- _Requirements: 9.1–9.6_
- [x] 17.1 Never block a push the user did not choose to block
- The installed hook exits 0 with one line when `ousast` is missing (the check did not run, and how to install). In hook (stdin) mode, an argument or usage error (an invalid `OUSAST_PUSH_DEADLINE`, a cache directory inside the repository) is advisory: one line and exit 0. `--mode blocking` stays the explicit opt-in that can reject a push, and keeps its usage errors. The prior-hook input-retention failure keeps exit 1, because it is the chained hook's verdict that cannot be obtained, not ours.
- Files: `ops/pre-push`, `src/openultrasast/cli.py`. Tests: `tests/test_push_hook.py` (a push with `ousast` absent from PATH succeeds and prints the line; a bad deadline in hook mode exits 0; blocking mode keeps exit 2).
- Evidence: the audit's `hook.sh` push 1 (rejected before) and `fail.sh` (rc=2 before) rerun with exit 0.
- _Done 2026-10-02:_ the hook prints one line and exits 0 (chaining a prior hook if present) when `ousast` (or `OUSAST_COMMAND`) is missing; in hook mode without `--mode blocking`, argparse/usage errors and unexpected exceptions print one line and exit 0. Tests: `test_missing_ousast_never_blocks_the_push`, `test_hook_mode_usage_errors_are_advisory_unless_blocking` (blocking keeps exit 2), and `test_cli_refuses_the_flag_outside_explicit_replay` updated (the experimental flag is still refused in hook mode, now advisory). Gates: 1987 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.1, 5.3, 5.4_
- [x] 17.2 Name every skipped check in plain words
- One terminal line per distinct coverage reason: hex-encoded paths decoded, sha256-suffixed reasons grouped with a count, JSON unchanged. "language not covered: <lang> (<n> files)" for a language with no engine frontend and no quick rules, in both `pre-push` and `scan`. A first push of a new branch says there is no base and what was checked instead. `reports.py` gains lines for the eight scan degradations `cli.py` records and the report did not render (`learning_endpoint_unavailable`, `cpg_unavailable`, `facts_unavailable`, `policy_invalid`, `model_withheld_execution_budget`, `max_findings_exceeded`, `hunter_model_unavailable`, `sandbox_unavailable`).
- Files: `src/openultrasast/push/report.py`, `src/openultrasast/push/explain.py` (new), `src/openultrasast/reports.py`, `src/openultrasast/cli.py`. Tests: `tests/test_push_report.py`, `tests/test_reports.py`.
- Evidence: the three repositories' terminal output names engine-missing, Go not covered and the new branch.
- _Done 2026-10-02:_ new `push/explain.py` turns every raw coverage reason into one plain `Skipped:` line per distinct kind (hex paths decoded, sha256 tokens dropped and counted, unknown reasons still printed); the existing one-line `Notice:` keeps its next action and gains an engine-missing variant. `language_not_covered:<lang>:<changed>:<total>` is recorded for a changed language with no engine frontend and no quick rules; `ousast scan` prints `notice: language not covered: go (N files)` and its report says so. `reports.py` renders the eight previously dropped degradations and prints any reason it has no sentence for. The hook passes `OUSAST_COMPARISON_BASE`. The new-branch line names the base it lacks; what was checked instead lands with the quick tier (17.3). Tests: `tests/test_push_explain.py` (6), two in `tests/test_reports.py`; module-audit manifest regenerated for the new module. Gates: 1995 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.2, 5.2, 8.2_
- [x] 17.3 Run the quick rules as the fast first tier
- Inside the deadline and before the engine, the shipped quick rules run over the head side of the changed files and keep the findings on changed lines or in changed functions. They print as advisory pattern matches, labeled apart from admitted alerts, recorded in the artifact's `quick_tier`, and never block.
- Files: `src/openultrasast/push/quick.py` (new), `src/openultrasast/push/runner.py`, `src/openultrasast/push/report.py`. Tests: `tests/test_push_quick.py` (new).
- Evidence: serve-static's planted `eval` in the new `debug.js` is shown; whether records' planted `%`-formatted SQL is shown is recorded as is, with no rule edits for it.
- _Done 2026-10-02:_ new `push/quick.py` runs the shipped quick rules (no shadow rules, no model) over the changed head files inside `_prepare` under the shared deadline, before the engine, keeping matches on a changed line or inside a changed function (entry-point ranges); it records `files`/`bytes_read` as proof of input. The terminal shows them under "Quick rules: N pattern match(es) on changed code ... Advisory: not verified by the engine."; the artifact keeps `quick_tier`; admission and push disposition are untouched. A new branch runs the quick tier over its last commit's changes and says so. `_analyze` gained `engine="off"`. On the audit's serve-static planted commit the `eval` in the new `debug.js` is now shown (js-eval, js-xss-response at debug.js:2, ~1.1 s on the host); the planted `%`-formatted SQL in records is NOT matched by any quick rule (coverage gap, no rule edited). Tests: `tests/test_push_quick.py` (3). Gates: 1998 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.3_
- [x] 17.4 Classify a file new in head as new, not unknown
- `ChangeContext` records the paths Git reports as added. An operation in an added file, when no add/delete pairing is unresolved and its head evidence is complete, is `new` with reason `file_added_in_head`, and admission accepts that reason as change evidence. An engine finding with established novelty whose only vetoes are capability vetoes renders as an advisory engine finding, not an alert. Every other unknown comparison keeps its veto.
- Files: `src/openultrasast/model/contracts.py`, `src/openultrasast/push/snapshot.py`, `src/openultrasast/push/policy.py`, `src/openultrasast/push/report.py`. Tests: `tests/test_push_policy.py`, `tests/test_push_snapshot.py`, `tests/test_push_report.py`.
- Evidence: unit tests on the comparison and admission; a live engine run if the host allows one (free memory >= 3 GB).
- _Done 2026-10-02:_ `ChangeContext.added_paths` records Git status-A paths (rename targets excluded). In `_compare_evidence`, after every existing context/head/deadline check, an operation in an added file is `new`/`file_added_in_head` when the claim is entailed, undischarged and has change evidence, and no base answer holds the same operation (else `operation_moved_within_change`); an unpaired add/delete still blocks as `change_context_unresolved`. Admission accepts `(new, file_added_in_head)`; with the empty registry such a candidate's only veto is `capability_unavailable`, and the terminal now shows it under "Engine: N finding(s) this push introduced. Advisory ... not an alert." Any comparison, evidence or location veto keeps a candidate out of that line. Tests: `tests/test_push_new_file.py` (9). Live engine evidence depends on host memory (17.7). Gates: 2007 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.4, 3.4_
- [x] 17.5 Say when the engine cannot finish, and offer it in the background
- One line when the engine did not finish (deadline, engine not installed). `--engine background` starts the engine detached after the quick tier, writes its result under the artifact directory and, on the next run, prints "engine result for <head> from the previous push". One background engine at a time; `--engine off` skips it with a stated line. No speed work.
- Files: `src/openultrasast/push/background.py` (new), `src/openultrasast/push/runner.py`, `src/openultrasast/cli.py`, `ops/pre-push`. Tests: `tests/test_push_background.py` (new).
- Evidence: a hook push with `OUSAST_PUSH_ENGINE=background` followed by a second push showing the previous result.
- _Done 2026-10-02:_ `--engine inline|background|off` (hook: `OUSAST_PUSH_ENGINE`) and `--background-deadline` (default 900 s). `off` and `background` run the quick tier and record `engine_off`/`engine_background` with a plain skip line; the deadline line names the deadline and the background option. New `push/background.py` starts one detached explicit replay (`start_new_session`, stdin closed, output to a log) per artifact directory and repository (lock file with a live-pid check: a second start reports busy), writes its artifact under `<artifact dir>/engine/<repo key>/<head>.json`, and the next run prints "Engine result for <head> from the previous push: ..." once, then marks it shown. Not usable through the Docker wrapper (the container exits with the hook); documented in 17.6. Tests: `tests/test_push_background.py` (4, one launching a real detached run). Gates: 2011 passed, 42 skipped; ruff, format, mypy clean.
- _Requirements: 9.5, 4.1, 4.2_
- [x] 17.6 Install, Docker and documentation for a first-time user
- `ousast pre-push install|uninstall [--force] [--docker] [REPO]` ships the hook as package data, honours `core.hooksPath`, refuses an existing hook unless `--force` (which chains it as `pre-push.before-ousast`), and works from a pip install. `ops/ousast-docker` is a hook-callable script (not a shell function) that passes stdin and the artifact path through. The `openultrasast:dev` image is rebuilt only if `df -h /` shows at least 4 GB free; otherwise the rebuild command is documented and the record says it was not rebuilt. `docs/scanning.md` gains a first-run guide, a glossary of skip reasons, a GitHub Actions recipe and full `--help` text; README tiers 1 and 2 updated, pulling `ghcr.io/norandom/openultrasast:2.0.1` where the wrapper allows.
- Files: `src/openultrasast/push/hook/pre-push`, `src/openultrasast/push/install.py` (new), `src/openultrasast/cli.py`, `ops/ousast-docker`, `docs/scanning.md`, `README.md`, `ops/README.md`. Tests: `tests/test_push_install.py` (new), `mkdocs build --strict`.
- Evidence: install from the package into a scratch clone, push through it, uninstall restores the prior hook.
- _Done 2026-10-02:_ `ousast pre-push install|uninstall [REPO] [--force] [--docker]` (`push/install.py`) installs the hook shipped as package data (`openultrasast/push/hook/pre-push`, byte-identical to `ops/pre-push`, verified in the built wheel), honours `core.hooksPath` without editing it, refuses an existing hook unless `--force` (ours is replaced; a foreign hook moves to `pre-push.before-ousast` and is chained), and uninstall removes only a hook it wrote and restores the chained one. `ops/ousast-docker` (also package data; `--docker` installs it beside the hook, which then uses it) is a script that mounts the repository read-only and the artifact directory at their own absolute paths, passes stdin with `-i`, runs as the user with no network and `--memory 3g`, defaults to `ghcr.io/norandom/openultrasast:2.0.1`, never pulls inside a hook, and exits 0 unless `OUSAST_PUSH_MODE=blocking`. The hook passes `--engine`/`--comparison-base` only when set, so an older ousast or image still runs. Every pre-push flag has help text; `docs/scanning.md` has the first-run guide, settings table, new-branch and background notes, the artifact keys, a skip-reason glossary (a test keeps it in sync with `push/explain.py`), a GitHub Actions recipe and the full `--help` text; README tiers 1 and 2 and `ops/README.md` updated (pull and tag the GHCR image instead of building). **Image not rebuilt:** `df -h /` showed 1.1 GB free (< 4 GB); rebuild with `docker build -t openultrasast:dev .`. Tests: `tests/test_push_install.py` (11) and the glossary test. Gates: 2022 passed, 42 skipped; ruff, format, mypy clean; `mkdocs build --strict` exit 0.
- _Requirements: 9.6, 7.4_
- [x] 17.7 Prove it on the audit's three repositories
- Rerun records, serve-static and unrolled/secure (same planted commits) with the new build in quick mode and with the hook installed, plus one Docker engine run on serve-static if free memory >= 3 GB and free disk >= 1 GB. Prove each run read its input (files, bytes).
- Files: `benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json`. Evidence: before (the audit) and after: exit codes, wall time against the deadline, findings shown, every skip line printed. A planted sample still missed is a coverage gap, recorded, never a rule edit.
- _Done 2026-10-02:_ `benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json` (before = the audit, after = this build installed from its wheel into a scratch venv). 10 hook pushes and 6 replays, all exit 0, 0.69-1.21 s against 30 s on the host; the quick tier read exactly the changed file and every head snapshot read the full tree (records 43,577 B, serve-static 61,525 B, secure 103,023 B). The planted JS `eval` in the new `debug.js` surfaces: quick tier on the host, and as two advisory engine findings classified `new`/`file_added_in_head` in one Docker engine run (190.9 s against a 300 s deadline; MemAvailable 3.28 GB, 1.16 GB disk). **The planted Python `%`-formatted SQL is still missed**: no quick rule matches it and the host has no engine. Recorded as a coverage gap, no rule edited. Go is named as not covered. The proof caught a wrong install hint (`pip install openultrasast`, not on PyPI), corrected in the hook in this commit.
- _Requirements: 9.1–9.6, 6.3_
## Implementation Notes
- Task5.2 measured PMPro timing out after both snapshots were materialized but before change context/graph analysis. Group6 must account for compatible snapshot/discovery reuse as well as graph reuse; identical-tip hits cannot substitute for the later representative changed-code latency gate. No new detector, scope heuristic or incremental CPG mutation is implied.
+104
View File
@@ -0,0 +1,104 @@
# Brief: OpenCode-MCP proof arm (follow-up to search-with-proof)
Status: brief / not yet specced. Proposed 2026-10-07 by the maintainer. Builds on
`search-with-proof` (see `../search-with-proof/verification-architecture.md`).
## Motivation
The search arm (directed, Cairn-inspired) works. The **proof-authoring** last mile is the drag:
1. **Safety-classifier refusals.** Authoring exploit demos repeatedly tripped a safety classifier
this session when exploit payloads or step-by-step attack prose passed through the model or a
tool prompt. The mitigation so far (keep payloads on disk, treat demos as opaque) is a workaround,
not a fix, and it limits how directly the proof model can reason about the exploit.
2. **Nondeterministic wiring.** The agent authors a demo but intermittently mis-wires the app to
the oracle fixture (served directory for path, database URL for SQL), so a demonstrable pair
comes back inconclusive until a retry. Each miss has cost a paid run.
Proposal: drive the proof with an **agentic coding tool (OpenCode MCP)** running **inside gVisor**,
using a model chosen for exploit authoring rather than one that refuses it, and **learn** the
proof-authoring policy with DSPy from **traces logged in OpenObserve**.
## Shape
- **Search model:** DeepSeek (current). Good at the directed fact/intent search.
- **Proof model:** GLM 5.3 (Anthropic-recommended) and/or an abliterated model via
`ABLITERATION_BASE_URL` (`abliterated large v2`). The abliterated model authors exploit **demos**
without the refusals that block a safety-trained model on this defensive task.
- **Proof driver:** OpenCode MCP (github.com/AlaeddineMessadi/opencode-mcp) as the agentic
demo-author, running inside the gVisor verification task so the exploit-authoring is sandboxed.
- **Learning:** DSPy compiles/optimizes the proof-authoring program (demo schema, wiring, steps)
from past traces, instead of the per-family hand-guidance we keep adding (which is now hitting
the prompt character budget — a signal the hand-rule approach is near its limit). Matches the
maintainer's "learned decisions, not hand-tuning" direction.
- **Tracing:** proof runs (reason/explore/author/verify, timings, outcomes, error classes) log to
**OpenObserve AI** at `OPENOBSERVE_URL` (org `OPENOBSERVE_ORG`, stream `OPENOBSERVE_TRACES_STREAM`).
DSPy consumes these traces as its training signal.
- **Credentials:** stored in `.env` like the existing keys, referenced by name only:
`ABLITERATION_API_KEY`, `ABLITERATION_BASE_URL`, `ABLITERATION_MODEL`, `OPENOBSERVE_URL`,
`OPENOBSERVE_ORG`, `OPENOBSERVE_TRACES_STREAM`, `OPENOBSERVE_TOKEN`. No secret value ever enters a
tracked file, a prompt, a log stream, or a commit.
## The trust-domain decision (must be resolved first)
The current design keeps the **model key on the VM only**; cluster tasks carry no secrets and have
restricted egress (verify tasks reach the files host only). Running the proof **model** inside gVisor
(as OpenCode MCP would) breaks both: the task would hold a model key and need egress to the model
API host. Two options:
- **A. Scoped key + egress in the cluster (operator-managed).** Inject a spend-limited proof-model
key into the proof task (like the existing `gemini-api-secret` slot) and add a per-host egress
allowlist for the proof-model API. Keeps the fully-sandboxed agentic authoring the maintainer
wants; requires an operator change and widens the cluster's secret/egress surface.
- **B. Brain on the VM, execution in gVisor (current split preserved).** OpenCode MCP and the proof
model run on the VM; only the DAST execution stays in gVisor. No new cluster secret or egress;
but the agentic authoring loop is not itself sandboxed.
Recommendation: start with **B** (no trust-domain change, fastest to try the abliterated proof model
and DSPy loop), and move to **A** only if sandboxing the authoring loop itself proves necessary.
## Security scope (why an abliterated model is in-scope here)
This is authorized **defensive** work: authoring proof-of-concept demos for **known CVE fix-pairs**
in a **gVisor-isolated** corpus, to demonstrate a regression before push. Guardrails to keep:
- Demos are **data**, not agent code; they stay in the gitignored private area, never echoed.
- No real-world or third-party live targets; only the frozen fix-pair corpus and owned oracles.
- gVisor is the execution boundary; the abliterated model never runs code itself, it only writes a
declarative demo the trusted verifier executes.
- The abliterated model is confined to proof-authoring; the search and all reporting stay on the
normal models.
## Open questions for the spec
1. A or B for the trust domain (above).
2. Does the abliterated proof model actually raise the demonstrated rate and remove the wiring
misses, measured on the pairs already shown demonstrable (xiaomusic, glance)? Qualify before scaling.
3. DSPy program shape: what is the proof-authoring signature (inputs: candidate + oracle contract;
output: a valid, wired demo), and what metric does it optimize (demonstrated rate at zero false
proofs) from OpenObserve traces?
4. Does OpenCode MCP fit the existing executor/verifier task contract (no-secret, presigned I/O,
scratch/egress limits), or does it need a new task kind?
5. Cost envelope per proof vs the current DeepSeek authoring.
## OpenCode MCP as the endpoint for both PoC and fix
Maintainer note (2026-10-07): OpenCode MCP is a good server endpoint for the PoC **and** the fix,
not only proof authoring. The same agentic endpoint can:
1. Author the PoC (the demonstrated proof) for a caught finding.
2. Write a candidate fix against the finding's repair direction and the failing PoC.
3. Hand the fix back to the differential verifier, which re-runs the PoC and confirms absence
(the proof flips from failing to passing).
This turns the "no auto-patch" gap in `../../../docs/use-case-fix-and-reverify.md` into "the proof
endpoint also proposes the fix, and the fix is accepted only when the proof it generated goes green."
The PoC is the acceptance test for the fix, so a wrong fix is caught, not trusted. The fix stays a
proposal for the developer to review; the verifier's green proof is the objective signal, not the
model's say-so. This fix capability is specced alongside the proof authoring when this arm is built.
## Not in scope
Changing the search arm, the oracle differential, or the abstract-interpretation triage. This brief
is about the **proof-authoring and fix** driver and its learning loop, planned as a full spec after
the safety net is qualified.
+10
View File
@@ -0,0 +1,10 @@
{
"feature_name": "proof-arm-opencode",
"created_at": "2026-10-07T14:40:00Z",
"updated_at": "2026-10-07T14:40:00Z",
"language": "en",
"phase": "brief",
"approvals": {"requirements": {"generated": false, "approved": false}, "design": {"generated": false, "approved": false}, "tasks": {"generated": false, "approved": false}},
"ready_for_implementation": false,
"summary": "OpenCode-MCP proof-authoring arm for search-with-proof: abliterated/GLM proof model, DSPy-optimized from OpenObserve traces; trust-domain decision (A scoped-key-in-cluster vs B brain-on-VM) must be resolved first. Creds in .env by name."
}
+103
View File
@@ -0,0 +1,103 @@
# Search-with-proof — handoff (2026-10-06)
State at handoff, for resuming on the search problem and abstract interpretation.
## RESOLVED 2026-10-07 — the one open question is CLOSED: first cluster demonstration achieved
The "first `demonstrated`" question below is answered. search-with-proof produced its first autonomous
`demonstrated` finding end to end on kube-ax, and a blind negative confirmed no false proof.
- **Pair:** hanxi/xiaomusic (pilot pair 13), family `path`, vulnerability class **prefix-check bypass**
(`startswith(root)` without a trailing separator; the fix adds `os.sep`). Image `e2c8b460`.
- **Vulnerable side:** `demonstrated` — owned effect in 3/3 affected runs, 0/3 safe runs. $0.37, 55 calls.
- **Fixed side (blind negative, Req 6.3):** `inconclusive`, **0 false proofs**. $0.56, 85 calls.
- **Pair total:** $1.22. Record: `benchmarks/measurements/2026-10-07-search-first-cluster-demonstration/`.
**The original pair-6 both-sides-false was a TWO-part diagnosis, both now fixed:** the demo did not reach the
observable (demo/agent), and — the decisive one — the path oracle only modelled **classic parent traversal** while
many real path CVEs (xiaomusic included) are **prefix-bypass**, which reaches a sibling that string-extends the served
dir, not the parent. The oracle now also plants a prefix-sibling canary.
**Fix chain that enabled the demonstration (each merged to main and validated by a paid run):**
1. Parity — agent is told the observable's location (path oracle reached parity with command/ssrf).
2. Verify-time-only — the observable is injected at verify time and absent during exploration; stop the explore-time
reachability trap that burned budget without authoring a demo.
3. Fair-share pilot budget — each search gets a fair share of `--ceiling-usd` with rollover, so one expensive search
no longer starves the rest.
4. Served-root guidance — the demo MUST set the app's served directory to the `${served_root}` placeholder.
5. Oracle prefix-bypass coverage — plant a sibling canary for the `startswith`-without-separator class.
6. Control-signal feedback — the keystone control (does the app serve `${served_root}`?) is fed back to the agent as a
`served-root` hint so it self-corrects its wiring. This was the decisive last step.
**Keystone (honest measurement), merged earlier:** the verifier runs a benign positive control; a broken or mis-wired
app yields `could_not_run`/annotation, never a silent clean-looking `inconclusive`.
**Next toward the pilot exit (Req 6.2):** fan out the path family to ≥5 demonstrations with 0 false proofs, then the
≥73-negative BLOCK qualification (Req 6.3). The glance pair (classic traversal) also demonstrates once its npm
dependency build-break (`filed`→`mime@4`) is pinned.
---
## The one open question (HISTORICAL — resolved above)
The full chain runs end to end on kube-ax. Every execution layer is cleared. The **first `demonstrated` has not
happened yet**, and the remaining gap is narrow and reproducible at **$0 model cost** via `demo_replay`.
No-model replay of pilot pair 6 (path / Node / npm build), both sides, image `aa54880`:
- Both sides **build, start, and run** (ready ~18–20 s, "fresh task run completed", no `could_not_run`).
- Oracle observed **no effect on either side** → `inconclusive`, "no consistent differential".
Both-sides-false means it is **not the harness** any more. It is one of:
1. The demo's attack step as written does not trigger the effect even on the vulnerable revision, or
2. The oracle does not capture the app's output where the effect would show.
## How to close it (no model spend)
- Control: the **probe path app** (`src/openultrasast/search/probe_apps/`) is known-good and *does* observe through the
same verifier code path. Diff its behaviour against the real app: how the served directory is wired
(`${served_root}` substitution in `start.arguments`), and where the oracle reads the nonce from (app stdout/response
capture in `oracles.py` `PathOracle.observe`).
- Re-run with `python -m benchmarks.search.demo_replay --both --demo <stored demo.json> --family path --lane ax
--image <digest> ...`. Stored demos live under `benchmarks/search/private/runs/*/<pair>/demo/demo.json` (gitignored).
- **Classifier note:** keep exploit payloads on disk. Hand Codex file-path tasks that treat demo contents as opaque;
print outcomes / phase timings / error *classes*, never request strings, env values, or step bodies. See memory
`search-with-proof-direction` and the classifier false-positive note. Code changes never tripped it; echoing
payloads or step-by-step attack prose did.
## What is done and merged (main)
- Trust-domain split: model key on the VM only; executor and verifier tasks carry no secrets; exchange via presigned
objects. Declarative demos (data, not agent code). gVisor is the cluster boundary.
- Verifier + 5 oracles pass inside a gVisor task (probe: 5/5 demonstrated, gaming demos inconclusive).
- Shared task base image, code-last layers; `plane/task-base.digest` pins it; `engine-image.yml` builds only on
dispatch / tags / Dockerfile / pin changes. Measured unpacked: search 2.07 GB, engine 3.61 GB.
- Pilot driver (`benchmarks/search/pilot_run.py`), budgets with reservation, named exhaustion limits, `demo_replay`.
- Per-host egress live: exec → any 443; verify/probe/pass → files host; replay → files + github. Task names carry
the right prefixes.
## Plumbing defects fixed along the way (each cost one run)
step allowance → family-vs-oracle mapping → build memory (256 MB SIGTRAP) → manifest path appended positionally →
startup env + data files → dead-end intent dedupe → archive size (edge ~100 MB, now split+gzip) → two-interpreter
image (pip 3.13 vs venv 3.12 → use `sys.executable`) → checkout root on sys.path → `node_modules` path.
## Next, once a demo demonstrates
1. Fan out the pilot across all 20 pairs, both sides (feasibility: demonstrated rate per family, zero false proofs on
fixed sides). Exit: ≥1 family with ≥5 demonstrations and zero false proofs (Req 6.2).
2. BLOCK qualification per family: 0 false proofs over ≥73 negatives from ≥73 repos (Req 6.3).
3. G1 arm on the unseen pool vs the triage/decision arms at equal false-alarm budgets (Req 6.5).
## Cluster / spend facts
- kube-ax: **2 tasks in flight, one per worker** (operator `--max-actors=1`). Dispatcher caps AX concurrency at 2;
raise it only when the operator raises worker count. "no free workers available" = retry, not failure.
- `/workspace` ~3 GiB free per node until the volume move; scratch guard 2 GiB/task.
- Pilot spend to date ~$11 across 9 paid runs; the decisive replay was $0.
## Decision model vs search — current read
- Decision/classification: measured, **promising as triage** (within-pair AUC 0.762; trained logistic ties at 0.721),
**capped as a blocker** (cannot prove 95% precision). Reframed as triage (`learned-decision-engine` Req 6.5).
- Search-with-proof: **not yet demonstrated**, but no observation so far argues against the idea — every failure was
our execution, now cleared. The pair-6 demo/oracle question is the first verdict on the method itself.
+40
View File
@@ -0,0 +1,40 @@
# Brief: search with proof
Maintainer, 2026-10-05, after reviewing Cairn (github.com/oritera/Cairn): "we are still not progressing through the
state space search / not finding what we need and the progress is sort of slow ... i think something is missing."
And: "I'd move to primarily executing in the cluster."
## What was missing
Every detection step so far classifies: it scores one function excerpt and then tries to make the score precise
enough to block. Calibration, Wilson bounds, pooled block gates, prompt variants and trained models all hit the same
wall: ranking is decent (within-pair AUC about 0.76), but precision at the top cannot be proven to 95% on our data.
Cairn treats a security problem as a directed search with an objectively checkable goal. A shared board holds
confirmed facts, open intents and human hints. A reasoner proposes independent intents; explorers with real tools
each confirm one fact; the loop ends when the goal is demonstrably met.
Three things follow for OpenUltraSAST:
1. **An objective success condition.** A finding is demonstrated when a test or proof of concept fails on the
vulnerable code and passes on the fixed code (or, for a push, fails on head and passes on base where applicable).
A demonstrated finding is right by construction; it needs no calibration and no training data, so it generalises
to unseen repositories by construction.
2. **Exploration with tools.** Agents that read other files, follow callers, build and run the code, instead of a
single-shot judgement of one excerpt.
3. **Search state per repository and push.** Facts and intents accumulate on a board instead of every question
starting from zero.
## Shape
- The cheap layers (quick rules, engine, classifier) become triage: they rank candidates.
- Agentic search runs only on the top-ranked candidates of a push, under a per-push budget.
- BLOCK only on a demonstrated finding. Advisory for plausible, unproven ones. "Could not demonstrate" is recorded as
such, never as "not vulnerable".
- Execution is primarily in the kube-ax cluster: reason and explore steps are AX Tasks; the VM coordinates.
- Model access: a scoped, budget-limited key in the cluster, managed by the operator.
## Not in scope
Copying Cairn code (AGPL-3.0; this repository is Apache-2.0). Offensive use against systems we do not own: all
execution is against code under analysis, inside gVisor sandboxes.
+25
View File
@@ -0,0 +1,25 @@
# Design: search with proof (v2, after two Codex reviews; the Claude Doc is the readable version)
## Why: from scoring excerpts to proving findings
BLOCK will mean a vulnerability was demonstrated and re-run by the system, not that a score passed a threshold. Every detection step so far classified one function excerpt, then tried to make the score precise enough to block. Same wall in every experiment: LLM programs (exp-005 A) within-pair AUC 0.762, pooled gate needs >=73 clean flags; trained logistic 0.721 (tie, CI spans zero); gradient boosting 0.597 (worse). Cairn (AGPL, design only) treats a security problem as directed search with a checkable goal. Three things follow: (1) objective goal: test/PoC fails on vulnerable, passes on fixed; needs no calibration or training data, so it carries over to unseen repos; (2) exploration with tools (read other files, follow callers, build, run); (3) search state: facts and intents accumulate on a board per search.
## Architecture
VM coordinates and is the only writer to the board (memory store); every model call and execution runs as an AX Task on kube-ax. Triage (quick rules, engine, classifier) -> top K candidates -> first facts on the board. Reason task (reads snapshot, returns complete or <=3 intents) <-> board; Explore tasks (one per intent; read, grep, run, write demo/; return one fact) <-> board; Verify task (fresh sandbox, planted canary, vulnerable vs fixed / head vs base) -> outcome: BLOCK only if demonstrated, else advisory or silent. Scheduler and budgets on the VM check caps before every dispatch. Tasks read/write via presigned links; store credentials stay on the VM. VM is fallback lane, kind local fallback, same image.
## The demonstration contract (v2)
The verifier, not the agent, observes the effect, on one side and not the other, consistently, in fresh sandboxes. Artefact: demo/ with build.sh and attack.sh (or a request file) touching only the documented external interface; its exit code and output are ignored. Oracles owned by the verifier per family: injection SQL = canary row seen in verifier's query log/response capture; path = canary file outside root read, seen by file-access watch; XSS = script executes in verifier's headless browser; SSRF = request arrives at verifier's in-sandbox listener; injection (command) = canary side effect seen by process watch. Deserialisation, access control, config/secrets: no owned oracle yet -> plausible_unproven at most. Attacker boundary excludes source tree, dependencies, runtime, fixtures; checkout and dependencies read-only during run; untracked files, dependency swaps, runtime patching refused. Differential: fix pair vulnerable-vs-fixed; push head-vs-base. Both sides must build and pass readiness independently; each side 3 runs from fresh state; inconsistent runs rejected; a side that fails to build or start => inconclusive, never a pass. Outcomes: demonstrated (BLOCK once the family is qualified), plausible_unproven (advisory, a visible finding), could_not_build/could_not_run/inconclusive (advisory if facts support, else silent; never "not vulnerable"), no_oracle (as plausible_unproven), no_evidence (silent). REGRESS's nonzero-exit-is-triggerable rule (regress/verdict.py) is not reused for proof; its sandbox plumbing is.
## Agent worker and sandbox tools
Own small tool loop in src/openultrasast/search/ via the existing OpenAI-compatible client (provider swappable, spend metered, Apache-2.0). Image ousast-search-task from the AX task runner, toolchains for Python, Node and TypeScript, PHP, Java; repo fetched over HTTPS. Tools: list_files, read_file (400 lines/call), grep (200 matches), run (per-call timeout, truncated output), write_demo (only write, under demo/), finish (one fact or recorded failure). Engine evidence (taint path, sinks, missing guards) is in the explore prompt. Reason step uses no tools.
## Triage, budgets and model access (v2)
Top K per push (K=3); candidate carries family hint + engine evidence as first facts. Caps (ceilings, not estimates): reason rounds 4/search, intents 3/round, explore 15 min/intent, $0.50/search, $1.50/push, 3 demos x 2 sides x 3 runs. Spend reserved before each model call at max cost and settled after (plane/budget.py gets a reservation step); per-task memory/disk/cleanup limits sized to workers (~3 vCPU, 5-5.75 GiB, two heavy lanes); total run wall-time and retry cap. Worst case 120 task-hours for 40 searches (~60 h on two lanes); a 1,300-change slice at K=3 could reach $1,950. First 3 pilot searches measure real cost; pilot ceiling set from that and asked for. Model key: operator scoped secret (AX injects only GEMINI_API_KEY; operator decides mechanism); never in manifest/log/board/record; repository code cannot read it: model client runs in a separate process as its own user.
## Pilot, qualification, then the unseen pool (v2)
1. Feasibility pilot: 20 dev-corpus pairs (never v3/pool), the four pool families with an owned oracle (injection, path, output encoding, untrusted destination), across the pool's five languages (Python, JS, TS, PHP, Java). Candidate = labelled function (measures proof, not triage); advantages recorded; fixed-side search never sees the vulnerable revision or its demo. Reported: demonstrated rate per family/language, false proofs, could_not_build/run/inconclusive/no_oracle rates, spend/tasks/wall. Exit: >=1 family with >=5 verified demonstrations and zero false proofs, else stop.
2. BLOCK qualification per family: 0/20 still allows 16.1% (Wilson 95%); a family BLOCKs only after >=73 negative-side searches (fixed revisions + ordinary changes) with zero demonstrated (upper bound 5%, same bar as the pooled gate). Until then demonstrations show as advisories.
3. G1 arm on the unseen pool: triage top K on every replayed change in a fresh slice; scored with the pool's rules on every visible finding incl. advisories, family + location match; 1/5/10% FA budgets beside classifier arms; adopted only under the no-regression rule. No evaluation label, fixing revision or evaluation board is ever an input to unseen search.
Every run reports the coverage ceiling (true candidate x owned oracle x both sides runnable x genuine differential, build failures in the denominator) next to catch. Delayed results go via push/background.py and are scored separately from push-time results.
## Decisions, risks and open questions
Largest risk: environment failure caps the demonstrated rate. Risks: builds fail; agent games oracle (mitigated by re-run, canary, differential, tracked-file check); repo code misuses egress (gVisor; ask operator for no-egress verify class); cost (hard caps); minutes per search too slow for synchronous hook (decide product path after pilot). Open: operator key mechanism + no-egress class; own tool loop vs CLI agent; pilot go-ahead $20; how a minutes-long search reaches the pre-push user.
@@ -0,0 +1,80 @@
# Declarative verification and separate execution domains
Implemented on `search-with-proof`, 2026-10-05, under the maintainer's explicit instructions.
`git merge -q main` failed because the shared Git metadata is read-only. The amended requirements
were read from the main checkout, including Requirements 3.1 and 5.2. No commits or network operations.
## Changes
- Strict, bounded JSON demonstration schema, also supplied to the model's `write_demo` tool.
Verifier-owned recipes cover pip, npm, composer, Maven, Gradle and no build. Runtime entrypoints,
CLI/HTTP sequences and earlier-response captures are validated; private fixtures, environment
substitutions, canary captures and script artefacts are rejected. Injection/traversal strings remain
legitimate request payload data, never executable verifier instructions.
- The verifier owns build, readiness and request execution. Local runs retain Bubblewrap (`userns`
on this host). HTTP requests require an acknowledged readiness gate; incomplete observations fail
closed. Build-generated internal symlinks are supported without accepting external or cyclic links.
- A supplied AX side dispatcher selects `task-boundary` and creates three separate fresh tasks per
side. Failure of local isolation without a dispatcher fails closed.
- The model-holding worker delegates list/read/grep/run to an executor interface. Production uses
presigned command/result objects; the in-process adapter is for tests. Commands have sequence
numbers, identical-retry caching, deadlines, stop and bounded results. Uncertain outcomes prevent
later commands from reusing that executor session.
- AX executor tasks use the `ousast-engine-search-exec-` prefix. Task entrypoints reject inherited
credentials before reading checkout input, then use an explicit environment allowlist.
## Changed files
Production:
- `src/openultrasast/search/demo.py`
- `src/openultrasast/search/executor.py`
- `src/openultrasast/search/worker.py`
- `src/openultrasast/search/verify.py`
- `src/openultrasast/search/verify_task.py`
- `src/openultrasast/search/_sandbox.py`
- `src/openultrasast/search/oracles.py`
- `src/openultrasast/search/probe.py`
- `src/openultrasast/search/probe_apps/app.py.txt`
- `benchmarks/ax/batch.py`
- `benchmarks/ax/search_verify.py`
- `plane/Dockerfile.search-task`
Tests and audit:
- `tests/test_search_demo.py`
- `tests/test_search_executor.py`
- `tests/test_search_verify_transport.py`
- `tests/test_search_verify.py`
- `tests/test_search_worker.py`
- `tests/test_search_isolation.py`
- `tests/test_search_probe.py`
- `tests/test_ax_batch.py`
- `benchmarks/measurements/2026-09-08-module-audit.json`
This report is the additional documentation file.
## Fresh validation
`/home/mc/Source/OpenUltraSAST/.venv/bin/pytest -q tests/test_search_*.py tests/test_ax_batch.py tests/test_plane_literals.py tests/test_module_audit.py -rs`
Result: **171 passed, 3 skipped, exit 0**, in 35.46 seconds.
The three skips are Chromium isolation/network setup, SSRF localhost sockets and HTTP localhost sockets,
all prohibited by this host's policy. Host verification mode was explicitly checked: `userns`.
Ruff passes for search sources, AX changes and affected tests. Targeted strict mypy passes for the seven
changed Python search implementation modules checked. Independent review approved the implementation,
including follow-up timeout, readiness, capture, credential and generated-link regressions.
Module audit: **171 modules**, **165 load-bearing**, **6 standalone capabilities**, **0 orphaned**.
The audit read the worktree source explicitly with `PYTHONPATH=src`.
## Remaining validation gaps
- No live AX/gVisor/kind dispatch, image build/pull/publish, or paid model calls were performed.
- Actual ecosystem dependency installations remain unqualified. Python dependency loading was tested
with a controlled offline build output and real isolated execution.
- The three socket/browser integrations need a host or task where those operations are permitted.
- Hard aggregate disk isolation in `task-boundary` mode requires an outer task storage quota. The
executor records this requirement and additionally bounds individual files and monitors checkout
growth; it cannot enforce a quota across every writable task path without that outer resource limit.
- AX tasks configured to inject a model credential will fail closed; executor and verifier task classes
must be provisioned without that injection.
@@ -0,0 +1,101 @@
# Host launcher repair and kube-ax operator constraints
2026-10-05, branch `search-with-proof`. Implemented under the maintainer's explicit
instructions. No commits or network operations. Existing uncommitted work was preserved.
This supersedes the build/transport and disk-bound statements in
`implementation-trust-domains.md` where they differ.
## Root cause and repair
The HTTP driver calls `subprocess.Popen(..., stdin=DEVNULL, stdout=DEVNULL,
stderr=DEVNULL)` inside bubblewrap. Its filesystem had no `/dev/null`; Python failed
in `_get_handles` before starting the app, binding port 18765, or applying captures.
Other probe apps do not use this nested HTTP launcher path.
A socket-free before/after reproduction read a 19-byte instrument input. Removing
`--dev /dev` reproduced exit 1 and the reported `subprocess.py:992 → _get_handles`
stack, ending with `FileNotFoundError: [Errno 2] No such file or directory: '/dev/null'`.
With the private device mount, the identical launcher exited 0 with empty stderr.
The original HTTP capture test's fixed port, token exchange, and differential
assertion remain unchanged. Reasons now retain the last 2,000 stderr characters;
executor result truncation retains stderr tails too.
## Operator contract
- Shared two-slot AX reservations cover workload lane instances and long-lived
search executors. Capacity rejection retries the identical manifest with 15–30
seconds of jitter until the deadline. Exhaustion is `capacity_timeout`, never
`sandbox_failure`, and does not trigger another execution attempt or VM fallback.
Uncertain task cleanup retains the reservation and halts dispatch.
- Task entrypoints configure temporary files and the specified package caches under
`/workspace`. The aggregate guard defaults to 2 GiB, configurable through
`OUSAST_SCRATCH_BYTES`. It checks workspace files, caches, and unlinked output logs
during child execution and after exit. Exceeding it returns `could_not_build` with
`scratch limit`; unreadable scratch is not interpreted as zero usage.
- The default AX side dispatcher first builds/packages each side in an executor,
retrieves its digest-checked archive, and deletes the executor before verification.
One archive contains the checkout, products, and spec. Three fresh verifier tasks
consume that archive through `VERIFY_INPUT_URL`; results use `RESULT_URL`.
Verifiers reject dependency builds. Git metadata is omitted from source archives;
safe internal generated links are materialized before export.
- Search tasks receive source archives. Executor tool guidance and project/operator
documentation require HTTPS GitHub git clone or codeload, never `api.github.com`.
First-download retries tolerate delayed egress activation for up to 60 seconds.
## Files changed in this follow-up
Production:
- `src/openultrasast/search/_sandbox.py`
- `src/openultrasast/search/verify.py`
- `src/openultrasast/search/executor.py`
- `src/openultrasast/search/verify_task.py`
- `src/openultrasast/search/task_storage.py` (new)
- `src/openultrasast/search/worker.py`
- `benchmarks/ax/batch.py`
- `benchmarks/ax/search_verify.py`
Tests:
- `tests/test_search_isolation.py`
- `tests/test_search_verify.py`
- `tests/test_search_verify_transport.py`
- `tests/test_search_task_storage.py` (new)
- `tests/test_ax_batch.py`
Documentation/evidence:
- `.kiro/steering/kube-ax.md` (new)
- `ops/ax/README.md`
- `benchmarks/measurements/2026-09-08-module-audit.json`
- This report (new).
Other files already modified/untracked at session start remain uncommitted.
## Fresh verification
- Requested suite, with `-rs` for skip reasons:
`/home/mc/Source/OpenUltraSAST/.venv/bin/pytest -q tests/test_search_*.py tests/test_ax_batch.py tests/test_plane_literals.py -rs`
— **181 passed, 3 skipped, exit 0**, 24.94 seconds.
- Final executor/storage regression and module audit tests after tightening stderr
truncation: **34 passed, exit 0**, including all **6 module audit tests**.
- Shared dispatcher consumer `tests/test_unseen_calibrate.py`: **10 passed, exit 0**.
- Module audit regenerated from the worktree using `PYTHONPATH=src`, with an explicit
source read/byte count: **172 modules, 166 load-bearing, 6 standalone capabilities,
0 orphaned**.
- Ruff passes across search, AX, and changed tests; strict mypy passes for the six
changed typed implementation modules. `git diff --check` passes.
## Gaps
- Host policy prohibits Chromium network/isolation setup, SSRF localhost sockets,
and the full HTTP capture test. These are the three skips. The exact missing-device
failure is reproduced and repaired without sockets; full HTTP host confirmation
remains necessary.
- No live AX tasks, image builds/pulls, registry installs, or external downloads were
run. Mocked transport plus local real child processes establish code contracts,
not deployed cluster readiness. Verifier egress to
`files.because-security.com:443` is requested from the operator and still pending.
- Scratch enforcement is a polling guard, not an instantaneous filesystem quota.
Programs that hardcode RAM-backed paths can still consume the worker memory limit.
Archive bounds remain 64 MiB compressed / 256 MiB file contents / 20,000 entries;
larger built projects fail explicitly and require a separately qualified envelope.
- Multi-ecosystem dependency build/runtime qualification remains open. The offline
preparation tests cover no-build projects and controlled Python build products.
@@ -0,0 +1,146 @@
# Requirements Document
## Introduction
Add a search-with-proof stage: for the top-ranked candidates of a push or a labelled pair, sandboxed tool-using agents
search for a demonstration of the vulnerability, coordinated through a fact/intent board, executing primarily in the
kube-ax cluster. BLOCK becomes "demonstrated", not "scored above a threshold". Context: `brief.md`, roadmap G1-G2,
`unseen-repo-evaluation` (the pool and metric this arm is judged by).
## Boundary Context
- **In scope**: the board (facts, intents, hints) in the memory store; reason and explore steps as AX Tasks; the
demonstration contract and its verification; triage-to-search budgeting; a pilot on known fix pairs; a G1 arm on the
unseen pool; model access through an operator-managed scoped key.
- **Out of scope**: copying Cairn code (AGPL-3.0); network actions against anything but the code under analysis;
changing the existing quick rules, engine or classifier (they are inputs here).
- **Adjacent expectations**: `unseen-repo-evaluation` scores this arm; the AX batch dispatcher (`benchmarks/ax/batch.py`)
and the kube-ax constraints (2 heavy lanes, `ousast-engine-` names, egress to public HTTPS) apply;
`learned-decision-engine` Req 6.4 BLOCK rule is superseded for this arm by "demonstrated".
## Requirements
### Requirement 1: The board
**User Story:** As the maintainer, I want each search to keep its state, so exploration builds on what is already
known instead of restarting.
#### Acceptance Criteria
1. Per search (one candidate or one push), the board holds facts (confirmed, objective, with the evidence that
confirmed them), intents (declared directions, open or concluded, with the facts they start from) and hints (human
input, absorbed on the next read).
2. The board lives in the memory store, versioned, and every fact and intent carries the step and the task that wrote
it.
3. A search ends when a fact meets the goal (Requirement 3), when its budget is spent, or when the reasoner proposes no
new intent and none is open; the end reason is recorded.
### Requirement 2: Reason and explore steps
**User Story:** As the maintainer, I want the search driven by the board's state, not by fixed roles.
#### Acceptance Criteria
1. A reason step reads the board and returns: the goal is met (citing facts), or at most N new, independent,
non-overlapping intents, or no new intent.
2. An explore step claims one intent, works it with tools inside the sandbox (read files, search the repository,
follow callers, build, run tests and scripts against the code under analysis), and returns exactly one new fact or
a recorded failure.
3. Both run as AX Tasks in the kube-ax cluster by default; the VM coordinates (board, dispatch, budgets) and is a
fallback lane only.
4. Parallel explores of one search never race on the board (claims as in the batch dispatcher).
### Requirement 3: The demonstration contract
**User Story:** As the maintainer, I want BLOCK to mean "shown", so precision does not depend on calibration.
#### Acceptance Criteria
1. *(Amended 2026-10-05, maintainer decision after the kube-ax probe: AX tasks have no capabilities, so gVisor and one
fresh task per side are the only isolation.)* A demonstration is **declarative data, never agent-written code**: a
build recipe chosen from the verifier's fixed per-ecosystem recipes, a start command of the form runtime + path +
arguments (no shell), and a request sequence (HTTP requests or CLI invocations, with values captured from earlier
responses) that the verifier itself sends. The system, not the agent, runs it in fresh sandboxes. The **verifier observes the effect itself** through a per-family oracle it owns (for
example a request log, a planted resource it watches, a browser execution check for XSS); the artefact's own exit
code or output is never the oracle.
2. Each family's oracle states the attacker boundary (what the artefact may touch: only the documented external
inputs of the code under analysis, never the verifier's fixtures, the source tree, dependencies or the runtime).
Inputs are immutable during a run; untracked files, dependency replacement and runtime patching are refused.
3. Differential: for a labelled pair the effect is observed on the vulnerable revision and not on the fixed one; for a
push, on head and not on base. Both sides must build and pass a readiness check independently; a side that fails
to build or start makes the result **inconclusive**, never a pass. Each side is run at least three times from fresh
state; inconsistent runs are rejected.
4. Only a verified demonstration counts as `demonstrated`. Outcomes: `demonstrated`, `plausible_unproven`,
`could_not_build`, `could_not_run`, `inconclusive`, `no_oracle` (family without an owned oracle), `no_evidence`;
none of the failure outcomes is ever reported as "not vulnerable".
5. Families without an owned oracle (for now: deserialisation, access control, configuration and secrets) can reach
`plausible_unproven` at most until an oracle is specified and reviewed.
### Requirement 4: Triage and budget
**User Story:** As the maintainer, I want search spent where it matters, at a known cost.
#### Acceptance Criteria
1. Candidates come from the existing layers (quick rules, engine, classifier scores); only the top K per push or per
pair side are searched; K and the per-search and per-push budgets (model spend, task count, wall time) are
configuration, enforced by the coordinator.
2. Every search records its model spend, task count and wall time. Spend is **reserved before each model call** (the
call's maximum cost) and released after; wall time, memory, disk and retries are capped per task and in total; a
budget is never exceeded, including by calls in flight.
3. Paid runs need the maintainer's budget go-ahead with a ceiling, as all paid steps do.
### Requirement 5: Model access in the cluster
**User Story:** As the maintainer, I want agents in the cluster to reach a model without spreading keys.
#### Acceptance Criteria
1. The agents' model key is a scoped, budget-limited key provided by the operator inside the cluster (AX v0.3.1
injects only `gemini-api-secret` as `GEMINI_API_KEY`; the design names the exact mechanism agreed with the
operator). Store credentials stay out of the cluster (presigned links, as today). For this arm, model calls inside
the cluster supersede `unseen-repo-evaluation` Req 4.4 (model calls on the VM); that boundary is unchanged for all
other arms. *(maintainer decision 2026-10-05: scoped key in the cluster)*
2. The key never appears in a manifest, a log, a board entry or a record, and never shares a sandbox with repository
code. *(Amended 2026-10-05, maintainer decision: trust domains are separated instead of isolated inside one task.)*
The agent loop (the "brain", holding the key) runs on the VM or in its own AX task and executes no repository code;
every tool call that touches the repository runs in a separate executor AX task without the key, exchanging
commands and results through presigned store objects. A Firecracker executor runtime is considered only if the
pilot's `could_not_build` rate shows gVisor cannot build the projects.
### Requirement 6: Feasibility pilot, false-proof qualification, then the unseen pool
**User Story:** As the maintainer, I want to know quickly whether search with proof works, and to admit BLOCK only on
evidence that meets the 95% precision bar.
#### Acceptance Criteria
1. **Feasibility pilot** (amended 2026-10-05 after review): about 20 known fix pairs from development corpora. It
measures feasibility only and is reported as such: demonstrated rate on vulnerable sides, false proofs on fixed
sides, `could_not_build` / `could_not_run` / `inconclusive` / `no_oracle` rates, spend, tasks and wall time. Its
advantages (known location, labelled family, runnable-project bias) are stated in the record. Fixed-side searches
never receive the vulnerable revision or its demo.
2. Before any paid search, an infrastructure probe without model calls proves each owned oracle works inside a gVisor
AX Task (database fixture for SQL, in-sandbox listener for SSRF, headless browser for XSS, file and process
watches), that the artefact cannot reach the observers directly, and records image size, RAM and the full
verification cost (up to 3 demos x 2 sides x 3 runs per search). Pilot exit: at least one family with at least 5 verified demonstrations on vulnerable sides and zero false proofs;
otherwise the result is recorded and the arm is not scaled.
3. **BLOCK admission per family** (deferred until the pilot passes; not part of the first tasks) requires a false-proof
qualification. A **negative** is a search on code that should be safe: a fixed revision searched on its own (the
search never sees the vulnerable revision), or an ordinary change. A **false proof** is the verifier observing the
effect on that safe code, confirmed by adjudication (an ordinary-change alarm may be a real new bug and is
adjudicated before it counts either way). Qualification needs zero false proofs over at least 73 negatives drawn
from at least 73 distinct repositories, so the Wilson 95% upper bound on the false-proof rate per negative is at
most 5%. This bounds the false-proof rate, not precision among flags; precision is reported separately from the
pool. Repeated runs and repeated searches of one repository are one negative.
4. **Coverage ceilings**: every run reports, separately, (a) the demonstrated ceiling: the share of vulnerable changes
with a true candidate selected, an owned oracle, both sides runnable and a genuine differential, build and run
failures in the denominator; and (b) the advisory ceiling: the share with a true candidate selected. Families
without an owned oracle (access control, deserialisation, configuration and secrets) have a demonstrated and BLOCK
ceiling of zero today; their advisory ceiling need not be zero.
5. **G1 arm on the unseen pool**: scored with the pool's rules on **every visible finding** (BLOCK and advisory), with
its family and location match; catch at the 1%, 5% and 10% false-alarm budgets; compared with the classifier arms;
adopted only under the pool's no-regression rule on a fresh slice. No evaluation label, fixing revision or board
from evaluation searches is ever an input to unseen search.
6. Delayed results (a search finishing after the push) reach the user through the existing background delivery and
are scored separately from results available at push time.
+29
View File
@@ -0,0 +1,29 @@
{
"feature_name": "search-with-proof",
"created_at": "2026-10-05T12:00:00Z",
"updated_at": "2026-10-07T13:30:00Z",
"language": "en",
"phase": "tasks-generated",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": false
}
},
"ready_for_implementation": false,
"pilot": {
"status": "in_progress",
"first_cluster_demonstration": "2026-10-07",
"pair": "xiaomusic (path, prefix-bypass)",
"false_proofs": 0,
"exit_remaining": "path family >=5 demonstrations / 0 false proofs (Req 6.2), then >=73-negative BLOCK qualification (Req 6.3)"
}
}
+130
View File
@@ -0,0 +1,130 @@
# Implementation Plan
Scope: infrastructure probe and feasibility pilot only (Req 6.1-6.2). BLOCK qualification (6.3), the unseen-pool arm
(6.5) and delivery (6.6) are deferred until the pilot passes. Implementation via Codex in a worktree; Claude reviews,
gates, runs every networked or paid step.
- [ ] 1. Operator prerequisites (Claude, no code)
- [ ] 1.1 Ask the operator: scoped spend-limited model key mechanism (reuse `gemini-api-secret` slot or a new
injection), and whether a no-egress task class exists for verify runs. _Requirements: 5.1, 3.2_
- [ ] 1.2 Record the answers in `kube-ax-cluster` memory and design.md; no key value anywhere. _Requirements: 5.2_
- [x] 2. Verifier and oracles (no model calls)
- [x] 2.1 `src/openultrasast/search/verify.py`: fresh-sandbox runner per side; build, readiness check, read-only
checkout and dependencies, untracked-file and runtime-patch refusal, 3 runs per side, consistency check,
outcomes `demonstrated|inconclusive|could_not_build|could_not_run|no_oracle`. Artefact exit code ignored.
_Requirements: 3.1-3.4_
- [x] 2.2 Oracles owned by the verifier: SQL (database fixture + query log), path (file-access watch on a planted
canary), XSS (headless browser execution check), SSRF (in-sandbox listener), command injection (process watch).
Observers unreachable from the artefact. _Requirements: 3.1, 3.2, 3.5_
- [x] 2.3 Unit tests with hand-written vulnerable/fixed toy apps per oracle, including gaming attempts (reading the
canary, revision sniffing, patching a dependency, contacting the listener directly): each must not yield
`demonstrated`. _Requirements: 3.2, 3.3_
- [ ] 3. Image and infrastructure probe (no model calls)
- [ ] 3.1 `ousast-search-task` image from the AX task runner: Python, Node/TypeScript, PHP, Java toolchains, the
oracle fixtures, headless browser; built by CI to GHCR, pinned by digest. _Requirements: 2.3_
- [ ] 3.2 Probe: the toy apps from 2.3 verified as AX Tasks on kube-ax (gVisor) and on kind. Record image size,
RAM peak, disk, wall time per verification and per 18-run search, oracle success per family, and that observers
are unreachable. Record `benchmarks/measurements/2026-10-xx-search-probe/record.json`. _Requirements: 6.2_
- [ ] 3.3 Gate: every oracle works under gVisor, or the family is marked `no_oracle` for the pilot.
- [x] 4. Board, coordinator, budgets
- [x] 4.1 Board in the memory store as immutable versioned blobs (`plane/memory.py`); facts, intents, hints with
writer step and task; coordinator is the only writer; tasks read and write through presigned links.
_Requirements: 1.1-1.3_
- [x] 4.2 Spend reservation before each model call in `plane/budget.py`, settled after; per-task memory, disk,
cleanup caps; total wall-time and retry caps; tests that in-flight calls cannot overrun. _Requirements: 4.1, 4.2_
- [x] 4.3 Dispatch reason, explore and verify as AX Tasks through `benchmarks/ax/batch.py` (extend its environment
contract beyond numeric deadlines), VM lane as fallback, resume from checkpoints. _Requirements: 2.3, 2.4_
- [ ] 5. Agent worker
- [x] 5.1 Reason step (no tools): board snapshot in, `complete` / up to 3 intents / nothing out. _Requirements: 2.1_
- [x] 5.2 Explore step: tool loop (`list_files`, `read_file`, `grep`, `run`, `write_demo`, `finish`) via the
existing OpenAI-compatible client; model client in a separate process and user from executed code; one fact
or a recorded failure. _Requirements: 2.2, 5.2_
- [ ] 5.3 Smoke test with a stub model (no spend) end to end on one toy pair on kube-ax.
- [ ] 6. Feasibility pilot (paid, after go-ahead)
- [x] 6.1 Select 20 development-corpus pairs (never v3 or the unseen pool) in the four pool families with an owned
oracle, across the five languages; record selection advantages. _Requirements: 6.1_
- [ ] 6.2 Run 3 searches, measure real cost per search, ask the maintainer for the pilot ceiling. _Requirements: 4.3_
- [ ] 6.3 Run the pilot: both sides per pair, fixed side blind to the vulnerable revision; record demonstrated
rate, false proofs, failure-outcome rates, both coverage ceilings, spend, tasks, wall time.
_Requirements: 6.1, 6.4_
- [ ] 6.4 Exit decision against 6.2 and report to the maintainer; on pass, spec tasks for qualification and the
unseen-pool arm. _Requirements: 6.2_
## Implementation Notes
- Group 2 (2026-10-05): independently reviewed; `tests/test_search_verify.py` has 32 passing tests and one
skipped SSRF integration test because this host forbids even localhost sockets. The namespace handoff and
listener request parsing are tested separately; the full SSRF effect still needs the task 3.2 probe before
claiming that oracle operational. The module audit and neighbouring sandbox/regression tests pass.
- The local lane requires Linux Bubblewrap and util-linux, reuses `SandboxJob`/`SandboxResult`, and fails closed
without isolation. Artefacts receive a batch CLI `TARGET` bridge and private scratch; HTTP apps need a trusted
CLI adapter. Only trusted `Side.build_command` outputs can enter the app runtime (read-only `/build`);
artefact `build.sh` writes scratch only. XSS requires a trusted browser executor, otherwise `no_oracle`.
- Tasks 3.1 (files only) and group 4 (2026-10-05): independent review approved; final offline validation:
107 passed, 2 skipped (Chromium namespace/socket restrictions and the existing SSRF localhost restriction).
Ruff and changed-source mypy pass; module audit: 160 load-bearing, 6 standalone, zero orphaned. No network
actions, model calls, image builds, pushes or commits were performed. Task 3.1 remains unchecked until the
image is built/published and a real digest is available; CI now includes `plane/Dockerfile.search-task`.
- The coordinator accepts an injected executor. `BatchExecutor` uses the repository's `benchmarks.ax.batch`
transport with one attempt per admission; coordinator owns the global retry budget and VM fallback. It requires
a repository checkout import path and a trusted worker command; reason/explore workers remain group 5.
Per-task memory, disk and cleanup ceilings are passed in the worker contract; worker enforcement and gVisor
compatibility still require group 5 and the 3.2 probe. Only trusted verification results can meet the goal.
- Board heads are immutable memory-store hashes; retain the returned head with the host run to resume. Unknown
interrupted model calls conservatively consume their full reservation; settled failures with zero spend release
it. Cleanup exceptions stop dispatch. Chromium uses a fresh profile in a private network namespace; injected
script must call `alert("ousast-xss")`, while escaped/reflected payloads cannot produce the verifier DOM marker.
- Tasks 3.2 and 5.1–5.3 offline implementation (2026-10-05, explicitly authorized): packaged five-family
toy pairs and real/two-gaming demos in `search/probe_apps/`; `python -m openultrasast.search.probe` reports
per-side stage times, image versions, child/self peak RSS, retained scratch usage and sampled verification
scratch peak. Generic isolation and later oracle namespace failures are instrument failures, never misses.
Optional RESULT_URL uses an object-scoped urllib PUT. `benchmarks.ax.search_probe` reuses AX/Docker lanes
(kind via its kubeconfig), with `ousast-engine-search-probe-` task names.
- Worker reason/explore uses the existing OpenAI-compatible client with retries disabled and SpendBudget
reservation before each call. Repository commands run only in the scrubbed sandbox with bounded tmpfs;
the key-absence, timeout, output limit and disk limit tests execute real isolated processes. The scripted
model/fake coordinator executor smoke reaches `demonstrated` through the real path oracle verifier.
- Independent review APPROVED the requested offline scope. Fresh requested search/AX/plane tests plus module
audit: 87 passed, 2 skipped, exit 0; focused new tests: 15 passed; Ruff and changed-source mypy pass.
Audit: 162 load-bearing, 6 standalone, zero orphaned. Tasks 3.2 and 5.3 remain unchecked for their original
live-cluster criteria; their code/local smoke is implemented. No network, paid calls, image builds, pushes
or commits. Offline wheel build could not resolve the uncached uv-build backend, so installed-wheel and
real AX/gVisor/kind/Docker image validation remain gaps.
- Task 6.1 and executor smoke (2026-10-05, explicitly authorized): added
`benchmarks/search/executor_smoke.py` and `pilot_select.py`, with offline tests and usage documentation.
The smoke drives one managed, key-free executor mailbox and records per-command timing/output measurements
and sampled workspace peak; AX and Docker lifecycle cleanup is covered by fake-lane tests. No live task ran.
Pilot selection read 10 development catalogs (873 rows, 904,050 bytes), retained 192 eligible pairs and selected
20 distinct repositories: four per language and five per family. Identities live only in the gitignored local
manifest; the aggregate record is `benchmarks/measurements/2026-10-05-search-pilot-selection/record.json`.
All 20 selections have unknown manifest/test-suite evidence; no matching local cache or frozen unseen-pool
metadata was present. These are feasibility inputs, not proof of buildability or qualification.
Both task-local independent reviews approved after a reserved-cache-symlink regression was fixed.
Fresh requested search/AX/plane tests, module audit, independent/reserved and unseen guards, and label guards:
287 passed, four skipped, exit 0. Skips: Chromium isolation, two localhost-socket cases, and no frozen pool.
Ruff and changed-source mypy pass; audit: 166 load-bearing, six standalone, zero orphaned.
No network or commits. The requested main merge could not update read-only Git metadata; HEAD and main
had identical file trees. Live AX/Docker executor smoke and oracle probes remain unmeasured.
- **FIRST CLUSTER DEMONSTRATION (2026-10-07) — pilot is live (6.2/6.3 in progress).** search-with-proof produced its
first autonomous `demonstrated` finding end to end on kube-ax: pair 13 (hanxi/xiaomusic, path, prefix-check-bypass
class), vulnerable side `demonstrated` (3/3 affected, 0/3 safe, $0.37), fixed side blind negative `inconclusive`
with 0 false proofs ($0.56); pair total $1.22, image e2c8b460. Record:
`benchmarks/measurements/2026-10-07-search-first-cluster-demonstration/`. This closes the handoff's open question.
The infra probe (3.2) passed earlier; the pilot (6.x) has run many times. Enabling fix chain, each merged + paid-run
validated: (1) parity (name the observable to the agent), (2) verify-time-only (observable injected at verify time,
absent in exploration — ends the explore reachability trap), (3) fair-share pilot budget (no starvation across
searches), (4) served-root guidance (demo MUST serve `${served_root}`), (5) oracle prefix-bypass coverage (sibling
canary for `startswith`-without-separator path traversal), (6) control-signal feedback (served-root hint so the
agent self-corrects wiring). Plus the keystone (positive control → `could_not_run`, never a silent clean-looking
`inconclusive`). NEXT: fan out the path family to ≥5 demonstrations / 0 false proofs (6.2 exit), then the
≥73-negative BLOCK qualification (6.3). glance (classic traversal) also demonstrates once its npm `filed`→`mime@4`
build-break is pinned.
@@ -0,0 +1,52 @@
# Search-with-proof: where each stage runs (verification architecture)
Recorded 2026-10-07. search-with-proof has two stages across a trust boundary: a Cairn-inspired
directed state-space **search**, then a DAST-style **proof (verification)**.
## The two stages
1. **Search (Cairn-inspired).** A blackboard of facts, intents and hints. A reasoner proposes
independent, non-overlapping intents; tool-using explorers each confirm one fact against the
code. The goal is a declarative demo (a candidate PoC), never proof by itself.
2. **Proof (DAST verification).** The verifier builds and starts the app, drives the demo's steps
through the app's real interface, and watches a verifier-owned canary for a **differential**
between the vulnerable and fixed revisions: the owned effect appears on the vulnerable side and
not on the fixed side (3/3 vs 0/3). Only this differential earns `demonstrated`.
## Trust boundary — what runs where
The brain runs on the VM; the agent's tool executions and the DAST proof run as isolated tasks in
the kube-ax cluster. Secrets never enter the cluster.
| Component | Location | Notes |
|-----------|----------|-------|
| Coordinator, board, budgets | VM (host) | Orchestrates the search; owns the global budget. |
| Reasoner + explorer **model calls** | VM (host) | The model key lives only on the VM. This is the brain. |
| Explorer **tool runs** (`list_files`, `read_file`, `grep`, `run`, build) | kube-ax AX Task (gVisor) | Carry no secrets; inputs/results via presigned object links. |
| **DAST proof / verification** | kube-ax AX Task (gVisor) | Builds, starts, attacks and observes the app. One task per revision. |
- **gVisor is the isolation boundary.** The proof runs untrusted exploit code against a running
app, so it is sandboxed per-task in the cluster.
- **Secrets stay on the VM.** Executor and verifier tasks carry no credentials; they exchange
checkout archives, demos and results through presigned object-store links. Per-host egress is
restricted (verify tasks reach the files host only).
## Verification lanes (the `--verify-lane` choice)
- **`ax` (default for paid pilot runs):** verification runs **inside kube-ax** as gVisor AX Tasks,
one per side. This is how every paid pilot run this session executed the proof.
- **`vm` (local):** verification runs in **Bubblewrap on the host**, no cluster. Used for the
zero-cost ceiling proofs (a hand-authored demo verified locally with no model and no cluster),
which is how glance and xiaomusic were confirmed demonstrable before any spend.
- **`docker` (replay):** `benchmarks.search.demo_replay` runs a stored demo with no model calls in
a local Docker lane.
## Why the proof is dynamic, not abstract interpretation
Abstract interpretation (the static engine and the decision model) is the cheap, generalizable
**triage/ranker**: it selects and orders candidate regions across any repo without executing. It
plateaued as a **blocking** gate because it could not prove the required precision. search-with-proof
is the **dynamic** confirmation: a demonstrated finding is ground truth by execution, so it earns a
block without calibration. The per-app harness cost (building and running the app, wiring its served
directory or database to the oracle fixture) is the price of the dynamic proof and is why proof pays
off where a runnable harness exists, with abstract interpretation remaining the static backbone.
@@ -0,0 +1,32 @@
# Brief: does the safety net work on repositories it has never seen?
Maintainer, 2026-10-04: "we want to spread the safety net across repos in general, unseen code. that was the purpose
of the AB tests to see if our approach is generalizeable." And: "we should run our tests / runs in kube. now we can
speed it up and design the workloads for ax directly. kind is only a fallback."
## Why this spec exists
The A/B experiments so far (exp-003, exp-004, exp-005) score the decision engine on repository-held-out folds of known
fix pairs. That measures whether the vulnerable side of a pair scores above its fixed side. It does not measure what a
safety net on an arbitrary repository lives on:
- on an ordinary push, does it stay quiet? (false-alarm rate)
- on a push that introduces a real vulnerability, does it say so? (catch rate)
Populations v1 and v2 measured the engine scan on fresh cases once (1 of 11, 0 of 17) and are spent. Nothing since has
measured push-level behaviour on unseen repositories.
## What this spec delivers
1. A frozen pool of repositories never used anywhere in development, with two kinds of change each: real
vulnerability-introducing commits and ordinary commits.
2. A push-level metric: each change replayed through the hook exactly as a user would run it, and through other
arms (decision engine, engine evidence) where an experiment asks.
3. Execution designed for the kube-ax cluster from the start: one push replay is one AX Task; kind and the VM are
fallbacks.
4. A baseline for today's hook, then a table of arms scored on the same pool.
## Not in scope
Changing detection. This spec only builds the measurement. Improvements are separate, registered experiments judged
by it.
@@ -0,0 +1,518 @@
# Design Document
## Overview
This spec builds the instrument for roadmap G0: a frozen pool of repositories that no development step has seen,
and a push-level score on it. The score is what a user of the hook sees. On a vulnerability-introducing push, did
the hook say so (catch rate)? On an ordinary push, did it say anything (false-alarm rate)? Replays run as AX Tasks
on kube-ax. kind and the VM run the same tasks as fallbacks.
Four facts from the code shape the design:
- **Populations already have a freeze-and-guard pattern.** `benchmarks/independent/freeze.py` verifies cases by
reading git only, `evaluate.py` scores with fixed rules and Wilson intervals, and
`tests/test_independent_population.py` fails when a reserved repository is named anywhere in `benchmarks/`,
`src/` or `tests/` outside its directory. Harvest batch 3 used that guard as a black box: candidates written,
guard run with output discarded, subsets bisected on red. That is how v3 stays unread.
- **The hook can already replay a change.** `ousast pre-push PATH --base B --head H --artifact F` resolves two
local commits (`snapshot.py`, `resolve_replay`: no fetch, no merge-base), runs quick rules and the engine inside
one deadline (default 30 s, engine inline) and writes a JSON artifact. The terminal text has one `Skipped:` line
per skipped check. The artifact holds three finding streams: admitted defects, advisory engine findings
(new or worsened, vetoed only by capability) and quick-rule matches with their CWE.
- **The AX pattern works but lives in one script.** `engine_trace_ax.py` (manifest, egress checks, presigned
objects, retry once under a new name, cleanup) and the lane scheduler in `engine_trace.py` (`run_plan`: fixed
lanes, fallback queue, `progress.json`, `STOP`, resume by input digest) are engine-specific.
- **Placement on kube-ax is random.** One worker per node, shared by all actors. Two heavy tasks fit per worker,
but only two concurrent tasks in all guarantee at most two on one worker.
### Goals
- A pool frozen by code against every used source (Req 1), with two kinds of change per repository (Req 2).
- A scorer whose numbers mean what the hook shows, with silent zeros visible (Req 3).
- One replay is one AX Task via a dispatcher shared with the engine (Req 4); a baseline and fresh slices (Req 5).
### Non-Goals
Detection changes, population v3, model calls in the cluster, languages the hook does not analyse.
## Architecture
```mermaid
flowchart LR
M[pool manifest slice] --> D[batch dispatcher on VM]
D --> P[presign spec GET and record PUT]
P --> T[AX Task ousast-engine-replay]
T --> G[shallow fetch base and head from GitHub]
G --> H[ousast pre-push --base --head]
H --> R[record tar: artifact, text, instrument]
R --> S[(store via presigned PUT)]
D -->|sandbox died twice or OOM| V[VM docker lane]
D -->|cluster down| K[kind lane, same tasks]
V --> S
K --> S
S --> C[scorer: catch, false alarm, coverage]
S --> A[model arms on VM, budgeted]
A --> C
C --> O[record: counts and digests]
```
## Components and Interfaces
### 1. Pool sourcing and eligibility (Req 1)
**Decision 1.1: sources and ecosystems.** Candidates come from GitHub reviewed advisories (`gh api /advisories`,
the enumeration harvest-3 used) for pip, npm, maven and composer, published 2019 or later, whose CWE maps to one of
the seven families through the project's family CWE table. Each needs a linked fix commit with one parent in one
repository. OSV records for the same advisory give the first affected version. That version is a cross-check for
the introducing commit (section 2). Reason: these four ecosystems are the five languages the hook analyses
(Python, JavaScript, TypeScript, PHP, Java). GHSA links commits and CWEs; OSV adds version brackets. Rejected:
drawing from every ecosystem (Go, Ruby, C#). That would mostly measure missing frontends, which is already known.
Coverage still reports uncovered files inside the drawn repositories. Rejected: NVD and published fix datasets.
NVD links commits weakly, and the datasets are mined corpora that development or model training has likely seen.
**Decision 1.2: eligibility by code against every used source.** `benchmarks/unseen/eligibility.py` builds the
used set before the draw. It never prints names; it records counts per source and a digest of the set.
| Source (Req 1.1) | Where it is read |
| --- | --- |
| Pair corpora | every catalog, recipe and dataset file under `benchmarks/pairs/`, plus the local pointer-pair manifests |
| Populations v1, v2 | `population-v1.toml`, `population-v2.toml`, read directly |
| Population v3 | never opened; the existing guard test only (below) |
| Harvest records | the batch catalogs under `benchmarks/pairs/advisory-fixes*` (the records hold counts only) |
| Memory examples | `example` rows of the local FileStore and the S3 store (`repo` field) |
| Experiment units | `plane/experiments/*.units.jsonl` and registered units files |
| Measurement records | everything under `benchmarks/measurements/` and `benchmarks/experiments/` |
| Anything else | a text sweep for `github.com/<owner>/<name>` over `benchmarks/`, `src/`, `tests/`, `plane/`, `.kiro/` |
Names are normalised (lower case, no `.git`, no trailing slash). Each candidate is resolved through the GitHub API
to its canonical `full_name`, its former names (redirects) and its fork network (`parent`, `source`). A candidate
is out if any of these is in the used set. A fork shares code with what it was forked from.
v3 is checked like harvest-3 did. The draft manifest is written into `benchmarks/unseen/`. Then
`pytest -q tests/test_independent_population.py -k referenced` runs with its output discarded. The guard already
sweeps `benchmarks/` outside the population directory. On red, the candidate list is bisected by rewriting the
draft until green, and the dropped count is recorded. No v3 name is ever printed or read by a person.
**Decision 1.3: post-cutoff preference.** An advisory published after 2026-06-01 (harvest-3's cutoff for the
decision engine's model) is flagged `post_cutoff`. Within each ecosystem stratum post-cutoff advisories are drawn
first. Results are reported for both groups. Reason: a post-cutoff advisory cannot be in the model's training
data as a known vulnerability. Rejected: post-cutoff only. Harvest-3 kept 12 post-cutoff pairs out of 251, and
three slices of 100 repositories cannot be filled from that window.
**Decision 1.4: licences.** The licence is read at the pinned head commit (LICENSE file, then the API's SPDX id).
Only permissive licences are published by name: MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, 0BSD, Unlicense,
Zlib, PostgreSQL, BSL-1.0. Everything else (copyleft, source-available, none) goes to the gitignored
`benchmarks/unseen/private/pool-p1.toml`. The committed manifest then holds an opaque id and the sha256 of
(URL, commits) for that entry, so the freeze digest still covers it. Reason: Req 1.4 read literally. Rejected:
"OSI-approved" as the bar, as population v2 used. It is wider than the requirement says (open question 1,
resolved 2026-10-04: copyleft stays private).
Also required: anonymous HTTPS, at most 500 MB, at least 40 eligible ordinary commits, one per fork network.
### 2. Change extraction (Req 2.1, 2.2)
Extraction runs on the VM over a full `--filter=blob:none` clone in a scratch cache. It uses git only, like
`freeze.py`. No scanner, engine or model touches a pool repository before the freeze.
**Decision 2.1: the vulnerability-introducing range.** Let F be the fix and P its parent. The vulnerable lines are
the old-side lines of F's hunks in the vulnerable function. For a pure-addition fix (an absence bug such as a
missing guard), they are the vulnerable function's own lines.
1. **`introducing` (preferred).** `git blame -w -M -C --porcelain P -L <lines> -- <file>`. The candidate V is the
commit that last wrote the sink line (for absence bugs, the function's declaration). V is accepted if all hold:
at V's first parent the sink line's normalised text is absent from the file's vulnerable function; V is not a
merge or root commit; V changes at most 50 files and 2,000 lines; when OSV names a first affected version, V is
reachable from that tag and not from the previous release tag. If the text is present at V's parent, blame
walks back from there, at most five steps. The replay is `base = V^`, `head = V`.
2. **`last_touch` (fallback).** When no V passes, `git log -L :<function>:<file> P` gives the newest commit L
before F that changed the vulnerable function. The replay is `base = L^`, `head = P`. The vulnerable code is in
head, and the range contains a change to the vulnerable function.
Each change records its method, family, vulnerable function (resolved in head), head-side changed lines,
`post_cutoff` and the advisory id.
Bias of each, stated in the record. `introducing` inherits SZZ noise: blame finds the last writer, not the author
of the flaw. A refactor, a move the heuristics miss or a reformat can stand in for the real introduction. The
absence check, the walk-back and the version bracket cut this down, but do not remove it. The bias runs both
ways: a wrong V may introduce nothing (a miss is not the hook's fault) or may move a flaw (the delta-based hook is
right to call it existing). `last_touch` is pessimistic for a delta hook. The flaw is often already in base, so
the hook's novelty rule correctly stays quiet and the change counts as a miss. Catch is therefore reported per
method. The headline is `introducing`; `last_touch` is a lower bound. Rejected: fix-parent against a fixed old
base (for example 50 commits back). The range often misses the vulnerable function entirely and grows without
bound on busy repositories.
Label check: every third `introducing` change is read by an agent before the freeze, with no model call, as
harvest-3 reviewed diffs. It is marked `confirmed`, `refactor` or `unclear`. The confirmed share is the label
noise estimate. The scorer also reports catch on the confirmed subset.
**Decision 2.2: ordinary commits.** The candidates are non-merge commits on the first-parent line of the default
branch that change at least one source file of an analysed language. Excluded:
- explicit security signals in the message: `secur|cve|ghsa|vuln|inject|xss|csrf|ssrf|exploit|sanitiz|escap`;
- reverts (`^Revert`, `This reverts commit`);
- dependency bumps (bot authors such as dependabot and renovate, `bump|upgrade .* from`, or only manifest and lock
files changed);
- commits whose source changes are all generated or vendored (`vendor/`, `node_modules/`, `dist/`, `*.min.js`,
lock files, `linguist-generated` in `.gitattributes`, "generated" headers);
- any fix commit of any advisory of the repository, and any commit touching a known vulnerable function of it;
- commits over 1,000 changed source lines (counted, reported).
Twelve are drawn per repository with a seeded random draw. Strata are three time thirds of the repository's
history crossed with four size buckets of changed source lines (1-10, 11-50, 51-200, 201-1,000). Allocation is
proportional to the repository's own commits per cell, with at least one per time third. Each change records base
(its first parent), head, changed files and line ranges. Reason: proportional allocation keeps the false-alarm
rate a per-push rate as a user meets it. Rejected: `freeze.py`'s broad `SECURITY_WORDS`. It also drops commits
saying "query", "token", "session" or "valid", which removes the ordinary commits where false alarms are most
likely, and the rate would look better than it is. Rejected: docs-only commits. They are trivially quiet and
would dilute the rate.
Label noise: an ordinary commit may itself introduce an unknown vulnerability, so a flag counted as a false alarm
may be true. The false-alarm rate stays as Req 3.2 defines it. Flagged ordinary changes are adjudicated beside it
(all if 30 or fewer, else every third) against protocol-v2's standard, and the adjudicated share is reported.
### 3. Sizing and slices (Req 2.3, 5.2)
Wilson 95% interval, z = 1.96: centre (p + z²/2n)/(1 + z²/n), half width z·sqrt(p(1-p)/n + z²/4n²)/(1 + z²/n).
Taking the larger distance from p as the width, the smallest n that meets the bar, computed exactly:
| Rate | Assumed p | Bar | n needed |
| --- | --- | --- | --- |
| false alarm | 0.05 | +/- 3 points | 315 |
| false alarm | 0.10 | +/- 3 points | 483 |
| false alarm | 0.20 | +/- 3 points | 756 |
| catch | 0.10 | +/- 10 points | 62 |
| catch | 0.25 | +/- 10 points | 88 |
| catch | 0.40 | +/- 10 points | 97 |
Catch: 100 vulnerability-introducing changes per slice, one per repository. At p = 0.4, Wilson gives
[0.309, 0.498], within 10 points. One per repository keeps them independent.
False alarm: ordinary changes of one repository are correlated (noisy code flags often). Wilson assumes
independence, so the count is inflated by a design effect 1 + (m - 1)·ICC. With m = 12 per repository and an
assumed ICC of 0.05, that is 1.55. 100 repositories × 12 = 1,200 changes, effective n 774 ≥ 756. The plain Wilson
interval at p = 0.2, n = 1,200 is about +/- 2.3 points. A repository-cluster bootstrap interval is reported beside
it, and the ICC measured on slice 1 is recorded. If it exceeds 0.05, the next pool sizes up. A frozen slice is
never changed.
**Decision 3.1: slices.** Pool p1 has three slices of 100 repositories each (100 + 1,200 changes per slice; 300
repositories and 3,900 changes in all). Repositories are assigned by a seeded shuffle, stratified by ecosystem and
`post_cutoff`. Slice 1 serves the baseline and the G1 arms table. Slices 2 and 3 serve the first two adoption
decisions. When they are spent, pool p2 is drawn by the same code, and it excludes p1. Reason: each slice alone
meets the Req 2.3 widths, so any decision can stand on one slice. Rejected: one large pool scored whole. Every
decision would then see every repository, which Req 5.2 forbids. Rejected: ten small slices. None would meet the
widths. Per family, 100 changes split over seven families give intervals of about +/- 20 to 25 points. They are
reported, not sized (open question 3).
Use ledger: `benchmarks/unseen/usage-p1.jsonl` is append-only, one row per use (slice, decision or experiment id,
date, result digest). A slice with a row whose purpose is `informed` cannot appear again as `qualifies` for that
change. A test checks this.
### 4. Scoring rules (Req 3.1, 3.2, 3.3)
Each replay uses the hook's defaults: `--deadline 30`, `--engine inline`, `--max-regions 500`, `--mode advisory`.
That is what a user's push gets. The text output is kept with every `Skipped:` line, and so is the artifact.
**Decision 4.0: a registered long-deadline arm** (maintainer decision 2026-10-04, open question 5). Beside the
default replay, slice 1 is replayed once more with `--deadline 300` and every other flag unchanged. The arm is
registered (manifest committed, `unseen: {pool: p1, slice: 1}`) before its first replay. G1 then reports both
what the user sees (30 s) and what the engine could catch given time (300 s), so a miss caused by the deadline is
told apart from a miss by the engine. It is a measurement arm, not a proposed default: adopting a longer hook
deadline would be a separate decision under 7.2.
**Decision 4.0a: calibrated cluster deadline and native check** (maintainer, 2026-10-05).
The gVisor proof exhausted 30 seconds in snapshot preparation on a 1,725-file repository.
Cluster replays therefore scale the 30/300-second arms by the median paired cluster/native
time through the quick tier, rounded to integer seconds. Native runc replays retain 30/300 seconds.
A deterministic sample (default 20, or all of a smaller proof manifest) runs both lanes at
600 seconds for calibration. Report per-change stage times and median slowdown with bootstrap
95% intervals for time through quick, engine build and engine queries separately. Incomplete
timings or degraded scans invalidate calibration. Records contain counts, times and opaque digests.
A deterministic native sample accompanies the cluster slice. Compare whether each push flags
anything, with a Wilson 95% interval. If its lower bound is below 95%, or any sample pair is
missing/failed, flag the run and exclude cluster results from the baseline. Disagreements name
change IDs only. The scorer reports applied deadlines and factors. Docker pulls explicitly after
checking its data-root free space against an unpacked image size upper bound plus 1 GiB; failures
retain sanitized stderr tails and exit codes. See the unseen README for coordinator commands.
**Decision 4.1: catch, per family.** A finding counts if its family is the change's family. Engine streams carry a
family; a quick-rule match maps its CWE through the family CWE table. Its head location must also be in the
vulnerable function or within 10 lines of the change's head-side hunks (protocol v1's window).
| Family | Location rule |
| --- | --- |
| injection, path, deserialization, output_encoding, untrusted_destination | sink line in the vulnerable function, or the changed lines +/- 10 |
| access_control | the vulnerable function itself (the handler that lacks the check), or the changed lines +/- 10 |
| config_secrets | the changed file and lines +/- 10 (file scope; function `<global>`) |
All three streams count: admitted defects, advisory engine findings and quick-rule matches. Each catch also names
the stream that made it. Reason: the hook shows all three to the user. Rejected: admitted defects only. That
scores the blocking path, which G3 decides separately, and would hide what users read.
**Decision 4.2: false alarm.** An ordinary change is a false alarm when any of the three streams holds at least one
finding, of any family, anywhere. The alerts-only rate (admitted defects) is reported beside it.
**Decision 4.3: coverage, so zeros cannot hide.** Every change gets exactly one state:
- `instrument_failure`: no record, a failed fetch, the head commit not verified, or zero bytes of the changed files
read. It is re-run once, then reported. It never counts as quiet or missed.
- `unanalysable`: no changed file is in a language the hook covers (the hook's `language_not_covered` notice).
- `quick_only`: the quick tier read the changed files, but no engine question completed on them.
- `engine_covered`: at least one engine question completed on a changed file.
The task entry proves its input as AGENTS.md requires. It records files and bytes in the checkout, bytes of each
changed file, the JVM's wall time and the hook's exit code. A replay faster than a JVM can start is flagged
`suspect_fast` and goes to the failure path. Catch and false alarm are reported over all changes and over each
coverage state.
### 5. The AX workload (Req 4)
**Decision 5.1: the image.** `plane/Dockerfile.engine-task` gains a second stage, `FROM engine AS replay`, built
with `--target replay` and pinned by digest. It adds `git`, the package with its runtime dependencies and
`benchmarks/unseen/replay_entry.py`. Joern, php-cli, Java 21 and the frontend-retention patch come from the engine
stage, so both lanes run the same Joern. The import smoke runs `ousast pre-push --help`. The root image stays
`ghcr.io/norandom/ax-task-runner:v0.3.1`. Rejected: a separate Dockerfile. It would duplicate the Joern install,
and two Joern copies are two instruments that drift. Rejected: one stage for both. The engine image's isolated
venv without yaml and boto3 is proven, and replay dependencies would change its digest on every bump.
**Decision 5.2: the task entry.** `replay_entry.py` is standard library only, like `engine_task_entry.py`, and
reuses its `transfer` (egress readiness wait, three attempts, no URL text in errors).
1. GET `SPEC_URL` (presigned, `.json`): repository URL, base, head, change id, hook flags.
2. `git init`, then `git fetch --depth 1 origin <head>` and `git fetch --depth 1 origin <base>` over HTTPS
(GitHub serves commits by SHA). Verify both with `git cat-file -e`. The fetch is bounded at 180 s; the 500 MB
repository cap bounds its size.
3. Run `ousast pre-push <checkout> --base <base> --head <head> --artifact /tmp/out/push.json` with defaults, the
backend heap at 1700 MB and the cluster JVM flags of `engine_trace_worker.py`. Capture stdout.
4. Pack `push.json`, `push.txt`, `engine.json` (the head and base scan records and witness rows from the artifact,
keyed by repository and revision, for engine-evidence arms) and `instrument.json` (section 4.3, image digest,
step timings). PUT to `RESULT_URL` as `.dat`. The PUT is completion.
On any error it PUTs a failure record naming the step and HTTP status, as the engine entry does. Task names are
`ousast-engine-replay-<slug>-<digest>`, the prefix the operator's egress watcher matches. No key or store
credential enters the manifest. Only the two presigned URLs do, and the spec object keeps repository names out of
AX status and cluster telemetry.
**Decision 5.3: the generic dispatcher.** `benchmarks/ax/batch.py` takes over from `engine_trace_ax.AX` and
`engine_trace.run_plan`:
- `Item`: id, input objects (name to bytes), the result object name, command, extra env, input digest.
- `Workload`: image digest, allowed URL env names, deadline, how to validate a received result.
- Lanes: `ax` (manifest, apply, poll the result object and AX phase, delete, clean objects), `kind` (the same
manifest against the kind AX), `docker` (the VM image, one container at a time).
- Scheduler: two AX lanes plus one VM lane by default. A sandbox death is retried once under a new name; a second
death or a JVM OOM goes to the VM fallback queue, which beats overflow on the VM lane. `progress.json` is written
only by the scheduler. A `STOP` file halts new launches. Resume skips items whose stored record has the same
input digest. Cleanup failure stops dispatching.
The engine pass becomes a workload over it, with the same tests. Reason: the egress, secrecy and cleanup logic took
several fixes and must exist once. Rejected: a replay-specific copy of `engine_trace_ax.py`. Rejected: putting it in
`src/` now. G4's `ousast plane scan` may need it there, and it moves when G4 is planned.
Concurrency is two AX tasks. Placement is random, and only two guarantees two per worker at most. Four would need
the operator to add placement (open question 2, resolved 2026-10-04: stays at two, no placement change
requested). Estimated wall time per slice: about 75 s per replay (fetch, 30 s hook, setup), 1,300 replays over
three lanes, about 9 hours. The long-deadline arm (4.0) is bounded at about 345 s per replay; at that bound 1,300
replays over three lanes take about 42 hours, less where the hook finishes early. It runs with `STOP` and resume
across sessions.
**Decision 5.4: model arms on the VM.** Arms that need DeepSeek (the decision engine, the combiner) read the stored
records on the VM. They build candidates from the head side's changed functions, score them, and take the change
score as the maximum. Each runs under its experiment manifest's `budget_usd` through the metered client, dry run
first, with responses cached. Nothing in the cluster calls a model.
**kind (Req 4.5)** runs the identical manifest and image through the `kind` lane. Every new task is proven there
before kube-ax, as `plane-on-kubernetes` requires.
### 6. Records and the guard (Req 1.2, 1.3, 5.1)
`benchmarks/unseen/` holds `pool-p1.toml` (repositories, licences, slices, changes with base, head, method,
family, function, lines), `freeze-p1.json` (used-set digests and counts per source, guard result, label-check
counts, freeze digest = sha256 of the canonical manifest), `protocol-p1.md` (the scoring rules of section 4,
committed before the first replay) and `usage-p1.jsonl`. The `private/` directory is gitignored.
The guard grows by one parametrised reservation. The existing sweep becomes a list of (reservation directory,
names): `benchmarks/independent/` for populations and `benchmarks/unseen/` for pools, including the private
manifest when present. It also sweeps `plane/`. A second test asserts that pool and population names are disjoint,
and prints only a count on failure. A third asserts that a frozen pool's digest matches `freeze-p1.json`, so the
pool is never edited. Raw per-change records stay outside the repository (`~/ousast-results/unseen/` and the
store). Committed result records under `benchmarks/measurements/` carry counts, rates, intervals and digests,
never names.
### 7. Metrics and reporting (Req 3.3, 3.4, 5.3)
`benchmarks/unseen/score.py` reports per family and overall: catch and false-alarm rate with Wilson 95%
intervals, the cluster bootstrap interval for false alarms, coverage counts per state, catch per method,
post-cutoff and per stream, and the alerts-only rate.
**Decision 7.1: arms at fixed budgets.** An arm with a score is thresholded per budget (1%, 5%, 10%). The
threshold is the lowest score whose false-alarm rate on the slice's ordinary changes stays within the budget.
Catch at that threshold is reported with its Wilson interval and the realised false-alarm rate. That in-slice
threshold is labelled exploratory. For a qualification, the threshold is fixed beforehand on an earlier slice or
on held-out folds, and it is applied unchanged. A binary arm (today's hook) has one operating point, reported with
the budgets it meets. Rejected: AUC over changes (it averages operating points no user runs).
**Decision 7.2: next to held-out folds.** Experiment manifests registered after this spec lands carry
`unseen: {pool, slice}`. `analyse` writes an `unseen` block into `result.json` next to the fold estimates: catch at
each budget for each arm, and the paired difference B - A with an exact McNemar test on the 100 changes. The
fold rule decides as today. The slice is recorded in the usage ledger.
**Decision 7.3: adoption gate on the unseen slice** (amended 2026-10-04, maintainer decision; Req 5.4; open
question 4 resolved). Reporting is not enough. An arm B is adopted over the production arm A only if, on the
untouched slice that qualifies it, both hold:
- catch at the 5% budget, with B's threshold fixed beforehand (7.1), is not significantly lower than A's: exact
two-sided McNemar on the paired vulnerability-introducing changes, alpha 0.05;
- for an arm with one operating point (a binary arm such as a hook setting), its false-alarm rate is not
significantly higher than A's: exact two-sided McNemar on the paired ordinary changes, alpha 0.05, with the
repository-cluster bootstrap interval of the difference reported beside it, since those changes are clustered.
`analyse` writes `unseen.gate: pass | regression | not_run` into `result.json`. `not_run` (no slice, slice
already `informed` for this change, or more than 5% `instrument_failure`) blocks adoption like `regression`. The
fold rule and this gate must both pass.
### 8. Increments
1. **Used set and guard.** `eligibility.py`, the extended guard and the usage-ledger test. Exit: guard green on an
empty pool; a planted used repository, a renamed one and a fork are each rejected in tests.
2. **Sourcing and extraction.** Candidates, both change kinds and the label check, on the VM. Exit: a draft for 300
repositories with method counts, the confirmed share and exclusion counts.
3. **Freeze p1.** Guard bisection for v3, then manifest, freeze record and protocol committed. Exit: freeze digest
committed and guard green.
4. **Dispatcher.** `benchmarks/ax/batch.py`, with the engine pass moved onto it. Exit: existing tests green and a
two-pin engine smoke on kube-ax matching its stored records.
5. **Replay task.** Image stage and entry, proven on kind, then kube-ax on five changes from this repository's own
history (never pool changes). Exit: instrument fields show bytes read and realistic JVM time, and quick-tier
output matches the VM lane.
6. **Scorer.** `score.py` on synthetic records, including every coverage state and an instrument failure. Exit:
unit tests green.
7. **Baseline.** Today's hook, from a frozen export of HEAD, on slice 1 (1,300 replays). Exit: the record committed
(counts and digests only) and a `baseline` row in the usage ledger. That is G0's exit.
8. **Arms on slice 1 (G1)** (added 2026-10-04 with decisions 4.0 and 7.3). The registered long-deadline arm,
the engine-evidence arm from the stored records, the model arms on the VM (5.4) under their budgets, and the
`unseen` block with the adoption gate in `analyse`. Exit: one committed table of arms on slice 1 at the 1/5/10%
budgets (counts, rates, intervals, digests), and `exploratory` rows in the usage ledger.
## Error Handling
A failed fetch, a missing commit or a hook crash gives a failure record, never an empty success. A dead sandbox is
retried once, then the VM runs it. A store or AX cleanup error stops the dispatcher. A slice with more than 5%
`instrument_failure` after re-runs is not scored; the cause is fixed first.
## Testing Strategy
Unit tests: used-set normalisation (case, `.git`, renames, forks), blame acceptance on a synthetic repository,
ordinary-commit exclusions, Wilson values against section 3, budget thresholds, coverage states, the task manifest
and dispatcher resume and fallback with a fake lane. Guard tests run in CI; cluster runs stay manual.
## Requirement Traceability
| Requirement | Design |
| --- | --- |
| 1.1 eligibility against every source, v3 via the guard | 1.2 |
| 1.2 frozen manifest, never edited | 6, increment 3 |
| 1.3 pool directory, guard extended | 6 |
| 1.4 licences, private manifest | 1.4 |
| 2.1 introducing changes, family, function, post-cutoff | 1.1, 1.3, 2.1 |
| 2.2 ordinary changes across time and size | 2.2 |
| 2.3 sizes for interval widths | 3 |
| 3.1 default replay, skipped checks kept | 4, 5.2 |
| 3.2 catch and false-alarm rules | 4.1, 4.2 |
| 3.3 per family, Wilson, coverage | 4.3, 7 |
| 3.4 arms at 1/5/10% budgets | 4.0, 7.1, increment 8 |
| 4.1 one change one Task, prefix, image by digest | 5.1, 5.2 |
| 4.2 HTTPS fetch, presigned record, no keys | 5.2 |
| 4.3 generic dispatcher, concurrency, retry, fallback, resume | 5.3 |
| 4.4 model arms on the VM within budgets | 5.4 |
| 4.5 kind runs the same tasks | 5.3, 5 (kind paragraph) |
| 5.1 baseline on slice 1, counts and digests | 6, increment 7 |
| 5.2 fresh slice per decision | 3.1, usage ledger |
| 5.3 experiments report unseen next to folds | 7.2 |
| 5.4 adoption needs no significant unseen regression | 7.3 |
## Amendments (2026-10-11)
Folded in after the G0 pool was frozen; the original sections above stay as the design of record, these refine it
where reality differed.
- **Persistence is bucket-first, not local disk.** The draw streams every stage to the `ousast-corpus` RustFS
bucket and resumes from it, so a restart (even after a code change) is cheap. Layout: `unseen/<seed>/decisions/`
(accept/reject per repo, key `sha256(repo+fix)`, idempotent), `unseen/<seed>/resolved/` (per-advisory
resolution), `unseen/<seed>/advisories/list.json`, `unseen/repos/` (get_repo cache, **not** seed-scoped, shared
across seeds), and `unseen/<seed>/pool/` (the frozen pool + provenance — the "don't redo sourcing" archive). The
local journal is a within-process cursor only. Implementation: `benchmarks/unseen/bank.py` (`CorpusBank`). The
VM's local disk is the scarce resource; run artifacts and replay outputs belong on RustFS (see steering
`storage-streaming.md`). Credentials: `.env` `CORPUS_S3_*`; boto3 needs path-style + SigV4.
- **Harvest cost is GitHub-rate-limit bound, measured yield ~2.2%.** Through the strict change criteria (1 vuln +
12 stratified ordinary, >=40 eligible commits) ~2.2% of candidates are accepted, so "100 repos" is ~4,500
candidate attempts at ~4 GitHub calls each — the bottleneck is the 5,000/hr API limit, not CPU or cores. Pure
GitHub-API harvesting cannot replace the clone: the positive is the SZZ introducing commit and the negatives are
stratified over full history, both needing repo history. The convergence fix was a full-objects clone (dropping
`--filter=blob:none`, which caused ~600 per-blob lazy fetches) plus the bucket caches above.
- **Record 07 is a machine-label baseline.** The pool's 100 vulnerability labels are SZZ-derived and unchecked
(`freeze-p1.json` `label_check_counts`: 100 unchecked). The G0 baseline is reported as a machine-label number
with that caveat; a human label pass is a later quality refinement, required before any published precision
claim, not before record 07.
- **Replay image provenance is pinned.** CI `engine-image.yml` builds `ghcr.io/norandom/ousast-replay-task` (the
replay harness, which `COPY`s `/app/benchmarks/unseen/replay_entry.py`) and `ousast-engine-task` (the Joern
hook), publicly pullable by digest. `ousast-runner` (the 248 MB ax-task-runner, no `/app/benchmarks`, no Joern)
is **not** the replay image. The AX lane's store check (`benchmarks/ax/batch.py`) requires a public-HTTPS DNS
presign-capable endpoint; how that and the cluster's store access are provided is operator-owned infra, not
assumed here.
- **AX lane blocker (2026-10-11) — kube is the target lane, so this is tracked, not dismissed.** The replay
*dispatches* correctly but tasks do not *complete*: `pins_done` stayed 0 for both the 5-change proof and the
1,300-change slice 1. Sub-issues resolved along the way: (a) the replay image is `ousast-replay-task` (public);
(b) the ax CLI must be `~/go/bin/ax` — the `~/.local/ax-v0.3.1/ax` build calls an `UpdateTask` RPC the deployed
ax-server answers `Unimplemented` (client/server version mismatch); (c) the store endpoint
`files.because-security.com` / `sast-memory` passes the egress check. **Remaining blocker (AX/cluster-internal,
operator-owned):** dispatched tasks reach the `ousast-pool` workers and the replay actors execute and checkpoint,
but the result never returns to the coordinator and no completion is recorded. `ousast-pool` logs show a gVisor
cleanup failure: `runsc delete ... connecting to control server at PID NNN: connection refused` and `failed to
terminate workload after checkpoint ... runsc delete: exit status 128`. Until the completion / checkpoint-cleanup
path is fixed, the ax lane cannot produce results; the G0 baseline (7.2) was taken on the **docker lane**
instead. The ax lane must be restored before it is the baseline/G1 lane (the stated target).
- **Engine coverage gap is comparison, not deadline (2026-10-11, preliminary n=1).** The cold-cache default arm
(30s) times out the Joern CPG build → `quick_only`. But a long-deadline sample (`--arm long_deadline`, 300s) on a
vulnerability change ran **206s with no timeout and still did not cover** (`coverage_status: incomplete`,
`quick_only`): the reasons were `ambiguous_line_correspondence`, `unmatched_added_deleted_path_correspondence`,
`eligibility_provenance_mismatch`, `witness_unresolved`, `comparison_unknown` — i.e. the engine could not
correspond the replayed diff's lines between base and head. So **calibration / a longer deadline alone will not
give engine coverage**; the blocker is the engine's base↔head comparison on real replayed diffs (consistent with
the earlier "engine near-blind to fixes" slice-leak audit). This is G2 engine-fix territory (comparison /
line-correspondence), not a G0 deadline knob. To confirm beyond n=1, the long-deadline arm must run at scale
(needs the kube lane). Implication: the record-07 baseline is measured on the **quick-rule tier** (the default
arm), and the engine arm's value is gated on fixing the comparison path, not on calibration.
- **Line-correspondence fix made, but it is necessary-not-sufficient (2026-10-11).** Root cause confirmed in
`src/openultrasast/push/snapshot.py`: equal-region base↔head line pairs were mapped only when the line content
was *globally unique in the file*, so common lines (`}`, blanks, imports) tripped `ambiguous_line_correspondence`
and the finding's anchor was discarded → `comparison_unknown` → finding suppressed. Fixed to **run-aware
positional anchoring**: a pair maps when its content occupies exactly one contiguous equal run per side (intra-run
duplicates are forced/safe; cross-run duplicates stay withheld — soundness preserved, `test_push_snapshot.py`
moved-operation guard still passes). 163 push tests pass. BUT live-testing the fix (mounted over the installed
package, 300s) on a real messy change (a 14-file Java diff) did **not** flip coverage: it stayed `incomplete`
because the binding constraints are elsewhere — `capability_unavailable` (empty capability registry always blocks
`coverage_status=complete` until Task 8.3), the engine produced **no actionable defect at all**
(`actionable_defect_ids: []`, a detection gap, not correspondence), plus `eligibility_provenance_mismatch` and
`contributor-scan ... projection_unavailable`. So "near-blind to fixes" is **multi-causal**; the correspondence
fix removes one real cause but engine coverage on unseen diffs also needs the capability registry (Task 8.3), the
engine's actual vuln detection, and the eligibility/projection paths. Validating the correspondence fix's benefit
needs a change where correspondence is the sole blocker (a small single-file diff), or the engine arm at scale.
## Open Questions
Maintainer decisions of 2026-10-04 resolve questions 1, 2, 4 and 5; question 3 stays open for G3.
1. **Resolved 2026-10-04.** Licences: only permissively licensed repositories are published by name. Copyleft and
all other licences go to the gitignored private manifest, covered in the committed manifest by an opaque id and
a digest (Req 1.4 as written; decision 1.4). Population v2's OSI-approved bar is not used.
2. **Resolved 2026-10-04.** Concurrency stays at two AX lanes plus the VM lane. No operator placement change is
requested (5.3).
3. Per-family BLOCK (G3) needs about 100 changes per family. Pool slices together, or draw a family-enriched pool
for G3?
4. **Resolved 2026-10-04.** An adoption requires no significant regression on the unseen slice, not only a report
(Req 5.4, amended; decision 7.3).
5. **Resolved 2026-10-04.** G1 adds a registered long-deadline arm (`--deadline 300`) next to the default 30 s, so
the table shows both what the user sees and what the engine could catch (decision 4.0, increment 8).
@@ -0,0 +1,107 @@
# Requirements Document
## Introduction
The safety net is meant for repositories it has never seen. This spec builds the instrument that measures that: a
frozen pool of unseen repositories, a push-level metric (catch rate and false-alarm rate as the hook reports them),
and an execution path that runs the replays as AX Tasks on the kube-ax cluster. Context: `brief.md`, the roadmap's
G0-G1 (`.kiro/steering/roadmap.md`).
## Boundary Context
- **In scope**: pool selection and freezing; change extraction (vulnerability-introducing and ordinary commits);
replay of each change through `ousast pre-push` and through registered experiment arms; the metric and its
confidence intervals; AX Task execution with kind/VM fallback; the baseline for today's hook; rotation of pool
slices.
- **Out of scope**: any detection change; population v3 (stays reserved and unread); model calls inside the cluster.
- **Adjacent expectations**: `learned-decision-engine` experiments use this pool as their primary unseen-repo score;
`pre-push-safety-net` owns the hook being replayed; `plane-on-kubernetes` and the AX executor
(`benchmarks/learn/engine_trace_ax.py`) supply the execution pattern; the reserved-repository guard covers every
population and this pool.
## Requirements
### Requirement 1: A pool no development step has touched
**User Story:** As the maintainer, I want the evaluation repositories to be ones nothing in development has seen, so a
good number means the approach generalises.
#### Acceptance Criteria
1. A repository is eligible only if it appears in no pair corpus, no population (v1, v2, v3), no harvest record, no
memory example, no experiment unit and no measurement record. Eligibility is checked by code against every source
before the pool is frozen; v3 is checked through the existing guard without being opened.
2. The pool is frozen (a committed manifest with repository URLs, commit SHAs, licences and the freeze digest) before
any change in it is replayed. A frozen pool is never edited; a correction is a new pool.
3. Pool repositories are referenced only inside the pool's own directory (like `benchmarks/independent/`), and the
reserved-repository guard is extended to them.
4. Only permissively licensed repositories are published by name; others may be used with their names kept out of
the repository (pointers in a local, gitignored manifest), recorded as such.
### Requirement 2: Two kinds of change per repository
**User Story:** As the maintainer, I want both vulnerable and ordinary pushes from the same repositories, so the
false-alarm rate is measured on the code the catch rate is measured on.
#### Acceptance Criteria
1. Vulnerability-introducing changes: from public advisories with a linked fix, the commit range that introduced the
vulnerable code where it can be determined, else the fix's parent compared with an earlier base; each labelled
with family and the vulnerable function. Post-model-cutoff advisories are preferred and flagged.
2. Ordinary changes: commits from the same repositories' histories with no security fix, revert or advisory link,
sampled across time and size; each recorded with its changed files and lines.
3. Per repository, at least one vulnerability-introducing change and several ordinary changes; the pool's totals are
sized so the false-alarm rate's 95% interval is no wider than +/- 3 points and the catch rate's no wider than
+/- 10 points (sizes computed in the design).
### Requirement 3: The push-level metric
**User Story:** As the maintainer, I want the score to be what a user would see, so the number means something.
#### Acceptance Criteria
1. Each change is replayed as `ousast pre-push --base <base> --head <head>` with the hook's default settings, in a
clean checkout, with the record of every skipped check kept.
2. Catch: a vulnerability-introducing change counts as caught when the hook (or the arm) reports a finding of the
right family on the vulnerable function or its changed lines. False alarm: an ordinary change counts as a false
alarm when the hook reports any advisory finding.
3. Reported per family and overall: catch rate, false-alarm rate, and both with 95% Wilson intervals; plus coverage
(changes where the hook could analyse the changed language) so a silent zero is visible.
4. Arms (decision engine, combiner, engine evidence, fix mechanisms) are scored on the same replays: catch rate at
fixed false-alarm budgets (1%, 5%, 10%), so arms are compared at equal noise.
### Requirement 4: Execution on kube-ax, designed for AX
**User Story:** As the maintainer, I want the replays to run on the cluster, so measurements are fast and the VM is
free.
#### Acceptance Criteria
1. One replayed change is one AX Task, named with the `ousast-engine-` prefix the operator's egress watcher matches,
using an image built `FROM ghcr.io/norandom/ax-task-runner:v0.3.1` with this package, Joern and php-cli, pinned by
digest.
2. The task fetches the repository at the needed commits itself over HTTPS (public hosts are reachable from actors)
and writes its record through a presigned link; no API keys or store credentials enter the cluster.
3. A generic AX batch dispatcher serves both this replay and the existing engine passes: items to Tasks, at most the
cluster's safe concurrency (two heavy tasks per worker), retry once on sandbox death, then a VM or kind fallback
lane; progress and resume as in the engine runner.
4. Arms that need model calls take the cluster's records and make their calls on the VM, within registered budgets.
5. kind runs the same tasks unchanged as the local fallback.
### Requirement 5: Baseline, rotation and use
**User Story:** As the maintainer, I want a baseline now and a fresh slice for each decision, so the pool is not
worn out by tuning.
#### Acceptance Criteria
1. The first use is a baseline of today's hook on the first slice; its record is committed (counts and digests only,
no repository names outside the pool directory).
2. A slice that informed a change can no longer qualify that change; each adoption decision draws a new, untouched
slice, recorded with its freeze digest.
3. Every learned-decision-engine experiment from now on reports its unseen-repo score from this pool next to its
held-out-fold score.
4. An arm is adopted only if, on the untouched slice that qualifies it, it shows no significant regression against
the current production arm: catch at the 5% false-alarm budget not significantly lower, and for an arm with one
operating point the false-alarm rate not significantly higher (exact paired tests, two-sided, alpha 0.05).
Reporting the unseen score alone is not enough to adopt. *(amended 2026-10-04, maintainer decision)*
@@ -0,0 +1,22 @@
{
"feature_name": "unseen-repo-evaluation",
"created_at": "2026-10-04T18:00:00Z",
"updated_at": "2026-10-04T20:00:00Z",
"language": "en",
"phase": "implementation",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": true
}
+402
View File
@@ -0,0 +1,402 @@
# Implementation Plan
One task group per increment of `design.md` section 8, in that order. A group starts only after the previous
group's exit is recorded. Maintainer decisions of 2026-10-04 are folded in: a registered long-deadline arm
(`--deadline 300`) beside the hook's default 30 s (design 4.0, group 8); adoption needs no significant regression
on the unseen slice (Req 5.4 as amended, design 7.3, tasks 6.4 and 8.2); only permissively licensed repositories
are published by name, everything else goes to the gitignored private manifest under a digest (Req 1.4, design
1.4); concurrency stays at two AX lanes plus the VM lane (design 5.3).
**2026-10-11 amendments folded in** (see design.md "Amendments (2026-10-11)"):
- **Pool p1 is frozen.** `benchmarks/unseen/pool-p1.toml` = 100 repos, slice 1; `freeze-p1.json` guard=0,
dropped=0, deterministic (manifest `6bd2fa5d`); archived to `ousast-corpus/unseen/20260601/pool/`. Groups 1-3 are
effectively done (the draw, draft and freeze ran end to end); formal records 02/03 were not written as separate
`record.json` files — the freeze was run directly (`bisect_guard` then `freeze`). Freeze recipe: `bisect_guard
--root $PWD --draft $CACHE/journal/draft-p1.toml` then `freeze --root $PWD --used-report <FILE>` where the file
is the draw report's `used_set` object (not the whole report).
- **Persistence is bucket-first** (`ousast-corpus`, `benchmarks/unseen/bank.py`); the local journal is a
within-process cursor only. Raw per-change replay records go to **RustFS**, not `~/ousast-results/` on local disk
— amend tasks 5.3/7.2's `--out` accordingly.
- **Group 5 image note:** the replay image is the CI-built `ghcr.io/norandom/ousast-replay-task` (+ engine-task),
publicly pullable by digest — not a local export-HEAD build and not `ousast-runner`. Task 7.1's "build and push
the replay image" is satisfied by the CI `engine-image.yml` run; use that digest.
- **Record 07 is a machine-label baseline** (labels SZZ-derived, 100 unchecked); state the caveat in the record.
- **Baseline lane (7.2):** run on kube (`--lane ax`). The `ax` binary and the AX-lane store endpoint are
operator-owned infra to stand up before 7.2; do not assume the cluster/network state.
Execution rules:
- **Who does what.** Codex implements every coding task offline: no network, no model, no store, no cluster. The
coordinator runs every networked step (GitHub and OSV API calls, git fetches and clones, the S3 store, image
pushes, kind and kube-ax). Those tasks are marked **(coordinator, network)**. Steps that need the coordinator but
no network (agent label reads, adjudication, the v3 guard bisection, commits) are marked **(coordinator)**.
Steps that spend model money are marked **paid: needs a budget go-ahead**. Each starts only after the maintainer
confirms a ceiling against the provider balance read just before it.
- **Done means.** A coding task is done when its tests pass in the host suite, `ruff check`, `ruff format --check`
and `mypy` are clean, and the reserved-repository guard is green. No test makes a network, model or cluster call.
- **Names stay in the pool directory.** Pool repository names and URLs appear only under `benchmarks/unseen/`
(copyleft and other non-permissive ones only under the gitignored `benchmarks/unseen/private/`). Raw per-change
records live in `~/ousast-results/unseen/` and the store, never in the repository.
- **Population v3 is never opened.** It is checked only through the guard test, run as a black box with its
output discarded.
- **Every group ends with a measured exit** and a record at
`benchmarks/measurements/<date>-unseen-NN-<slug>/record.json`: commit, image digests where used, the instrument
checks (bytes read, JVM time, exit codes), counts and digests. No repository names. The guard sweeps these
records.
- **Instrument first** (AGENTS.md). A zero, an empty draft or a fast run is treated as an instrument failure until
bytes read and wall time prove otherwise.
- [x] 1. Used set and guard (design 1.2, 6; increment 1)
- [x] 1.1 Used-set builder
- `benchmarks/unseen/eligibility.py`: normalises names (lower case, no `.git`, no trailing slash) and builds the
used set from every source of design table 1.2: pair catalogs, recipes and datasets under `benchmarks/pairs/`
plus the local pointer-pair manifests as `benchmarks/pairs/catalog_gen.py` reads them; `population-v1.toml`
and `population-v2.toml`; the advisory-fix batch catalogs; memory `example` rows (`repo` field) through an
injected `MemoryStore` (`FileStore` locally, the S3 store when the coordinator passes one); experiment units
under `plane/experiments/`; `benchmarks/measurements/` and `benchmarks/experiments/`; a text sweep for
`github.com/<owner>/<name>` over `benchmarks/`, `src/`, `tests/`, `plane/` and `.kiro/`. It never reads
population v3 and never prints a name. Output: counts per source and the sha256 of the sorted set.
- `tests/test_unseen_eligibility.py`: a synthetic tree with one repository per source, each found; case, `.git`
and trailing-slash variants collapse; stdout and the output JSON contain no name; a v3-shaped file planted in
the synthetic tree is not opened (open is patched to fail on it).
- _Requirements: 1.1_
- [x] 1.2 Canonical resolution, renames and forks
- `eligibility.py`: `resolve(candidate, api)` with an injected GitHub client returns the canonical `full_name`,
former names (redirects) and the fork network (`parent`, `source`). A candidate is ineligible when any of them
is in the used set; one candidate per fork network is kept.
- Tests with a fake client: a planted used repository, a renamed one and a fork of a used one are each
rejected; two forks of one network keep one; the rejection reason is a source label, not a name.
- _Requirements: 1.1_
- [x] 1.3 Guard extended to pools
- `tests/test_independent_population.py`: the sweep becomes a list of (reservation directory, names):
`benchmarks/independent/` for populations, `benchmarks/unseen/` for pools (public manifests, plus
`benchmarks/unseen/private/*.toml` when present). The sweep also covers `plane/`. New tests: pool and
population names are disjoint (failure prints a count only); a frozen pool's manifest digest equals its
`freeze-pN.json` (skipped while no pool is frozen). The existing v3 handling is unchanged.
- Each reservation is exempt only inside its own directory: population names (v3 included) are still swept in
`benchmarks/unseen/`, the gitignored `private/` files included. Task 3.1 depends on this.
- `.gitignore`: `benchmarks/unseen/private/`.
- Evidence: the guard is green with an empty `benchmarks/unseen/`, and red when a test fixture names a pool
repository in `benchmarks/measurements/` (shown once in the record).
- _Requirements: 1.2, 1.3, 1.4_
- [x] 1.4 Usage ledger
- `benchmarks/unseen/ledger.py`: append-only `usage-pN.jsonl`, one row per use (slice, decision or experiment
id, purpose `baseline | exploratory | informed | qualifies`, date, result digest, freeze digest). It refuses a
`qualifies` row for a change whose slice already has an `informed` row for it, and refuses to rewrite rows.
- `tests/test_unseen_ledger.py`: append, the refusal, a rewritten row detected by a digest chain.
- _Requirements: 5.2_
- [x] 1.5 Used-set measurement and record 01 **(coordinator, network)**
- Run `eligibility.py` against the real sources, the S3 memory store included (read only).
- Instrument: each source reports files read and bytes; a source with zero rows fails the run unless it is
known empty.
- `benchmarks/measurements/<date>-unseen-01-used-set/record.json`: counts per source, used-set digest, guard
result (green on the empty pool), test counts of 1.1-1.4.
- _Requirements: 1.1, 1.3_
- [ ] 2. Sourcing and extraction (design 1.1, 1.3, 1.4, 2, 3; increment 2)
- [x] 2.1 Advisory candidates and repository filters
- `benchmarks/unseen/source.py`: from GHSA advisory JSON (pip, npm, maven, composer; published 2019 or later)
keep those whose CWE maps to a family through `src/openultrasast/model/taxonomy.py` and that link one fix
commit with one parent in one repository; attach the OSV first affected version; flag `post_cutoff` when
published after 2026-06-01. Repository filters: anonymous HTTPS, at most 500 MB, licence at the pinned head
(LICENSE file, then the API's SPDX id) classed `permissive` (the design 1.4 list) or `private`.
- `tests/test_unseen_source.py` on recorded fixture JSON (no network): CWE mapping, multi-parent and
multi-repository fixes dropped, cutoff flag, licence classing (GPL, AGPL, LGPL, none and source-available go
private).
- _Requirements: 1.4, 2.1_
- [x] 2.2 Vulnerability-introducing range
- `benchmarks/unseen/extract.py`: the `introducing` method of design 2.1 (blame of the sink line, or the
declaration for an absence bug; absence at V's first parent; no merge or root; at most 50 files and 2,000
lines; OSV version bracket when present; walk-back of at most five steps) and the `last_touch` fallback. Each
change records method, family, vulnerable function resolved in head, head-side lines, `post_cutoff`, advisory
id. Git only; no scanner or engine.
- `tests/test_unseen_extract.py` on a synthetic repository built in a temp dir: accepted V; a reformat walked
back; a merge rejected; size cap; bracket rejection; fallback chosen when no V passes.
- _Requirements: 2.1_
- [x] 2.3 Ordinary-commit draw
- `extract.py`: first-parent, non-merge commits touching an analysed language; the exclusions of design 2.2
(security words of the narrow pattern, reverts, bot and bump commits, manifest- or lock-only, generated or
vendored, advisory fix commits, commits touching a known vulnerable function, more than 1,000 source lines,
counted). Seeded draw of 12 per repository over three time thirds by four size buckets, proportional, at least
one per third; a repository with fewer than 40 eligible commits is dropped.
- Tests: each exclusion; the narrow pattern keeps "query", "token", "session"; allocation and the seed
reproduce the same draw; the 40-commit floor.
- _Requirements: 2.2, 2.3_
- [x] 2.4 Draft manifest and slices
- `benchmarks/unseen/draft.py`: writes `benchmarks/unseen/draft-p1.toml` (permissive entries by name) and
`benchmarks/unseen/private/draft-p1.toml` (the rest); the public file holds an opaque id and
sha256(URL, commits) per private entry. Seeded shuffle into three slices of 100 repositories, stratified by
ecosystem and `post_cutoff`; each repository has one introducing change and 12 ordinary changes.
- Tests: the digest covers private entries; no private URL in the public file; the same seed gives the same
slices; per-slice strata within one repository of proportional.
- _Requirements: 1.2, 1.4, 2.3, 5.2_
- [ ] 2.5 Sourcing run **(coordinator, network)**
- `gh api /advisories` enumeration, OSV lookups, canonical resolution against the used set (1.2), blobless
clones into a scratch cache, extraction (2.2, 2.3), draft (2.4).
- Instrument: clones counted with bytes on disk; a repository whose clone reads zero source files is an
instrument failure, not a drop. Fewer than 300 eligible repositories stops the run with counts; the
eligibility bars are not relaxed to fill it.
- Observable: a draft of 300 repositories (3,900 changes) on disk.
- _Requirements: 1.1, 1.4, 2.1, 2.2, 2.3_
- [ ] 2.6 Label check and record 02 **(coordinator)**
- Every third `introducing` change read by an agent, no model call, marked `confirmed | refactor | unclear`
in the draft.
- `benchmarks/measurements/<date>-unseen-02-draft/record.json`: candidates per ecosystem, rejections per used
source, licence split (public, private), method counts, post-cutoff counts, confirmed share, ordinary-commit
exclusion counts. No names.
- _Requirements: 2.1, 2.2_
- [ ] 3. Freeze p1 (design 1.2, 6; increment 3)
- [x] 3.1 Guard bisection tool
- `benchmarks/unseen/bisect_guard.py`: rewrites the draft with a candidate subset, runs
`.venv/bin/pytest -q -p no:cacheprovider tests/test_independent_population.py -k referenced` with stdout and
stderr discarded, reads only the exit code, and bisects until green. Reports the dropped count only.
- `tests/test_unseen_bisect.py` with an injected fake guard (a hidden set): finds and drops exactly the hidden
members; never logs a candidate; an exit code other than 0 or 1 aborts (instrument failure).
- _Requirements: 1.1_
- [x] 3.2 Freeze writer
- `benchmarks/unseen/freeze.py`: canonical `pool-p1.toml` (and the private counterpart) from the bisected draft;
`freeze-p1.json` with used-set digest and per-source counts, guard result, dropped count, label-check counts
and the freeze digest (sha256 of the canonical manifest). Refuses to write over a frozen pool; a correction
is `p2`.
- Tests: digest stable under key order; overwrite refused; the guard's digest test (1.3) passes on a fixture
pool.
- _Requirements: 1.2_
- [x] 3.3 Protocol p1
- `benchmarks/unseen/protocol-p1.md`: the scoring rules of design section 4 (default flags, decision 4.0's
long-deadline arm, catch and location rules, false alarm, coverage states), the statistics of section 3 and
7, and the adoption gate of 7.3, stated before any replay.
- Observable: the file exists and its sha256 is listed in `freeze-p1.json`.
- _Requirements: 3.1, 3.2, 3.3, 3.4, 5.4_
- [ ] 3.4 Freeze run and record 03 **(coordinator)**
- Run the bisection (3.1), then the freeze (3.2). Guard green, digest test active.
- `benchmarks/measurements/<date>-unseen-03-freeze/record.json`: freeze digest, protocol digest, dropped
count, final counts per slice and kind.
- _Requirements: 1.1, 1.2, 1.3_
- [ ] 4. Dispatcher (design 5.3; increment 4)
- [x] 4.1 Items, workloads and the AX lane
- `benchmarks/ax/batch.py`: `Item` (id, input objects, result object name, command, extra env, input digest),
`Workload` (image digest, allowed URL env names, deadline, result validator) and the `ax` lane, moved from
`benchmarks/learn/engine_trace_ax.py`: manifest render, egress readiness, presigned GET and PUT as `.dat`,
apply, poll result object and AX phase, delete, object cleanup. The `kind` lane is the same lane with the
kind context.
- `tests/test_ax_batch.py` with a fake `ax` CLI and a fake store: manifest holds only the allowed URL env
names and no credential; names carry the `ousast-engine-` prefix; cleanup on success and failure.
- _Requirements: 4.1, 4.2, 4.5_
- [x] 4.2 Scheduler, fallback and resume
- `batch.py`: two AX lanes and one VM `docker` lane by default (one container at a time); sandbox death retried
once under a new name, a second death or a JVM OOM goes to the VM fallback queue ahead of overflow;
`progress.json` written by the scheduler only; a `STOP` file halts launches; resume skips items whose stored
record has the same input digest; a cleanup failure stops dispatching.
- Tests with fake lanes: retry then fallback; OOM straight to fallback; resume skips; `STOP`; cleanup failure
halts; never more than two AX tasks in flight.
- _Requirements: 4.3_
- [x] 4.3 Engine pass on the dispatcher
- `benchmarks/learn/engine_trace_ax.py` and `engine_trace.run_plan` become an engine `Workload` over `batch.py`;
no behaviour change.
- Observable: `tests/test_ax_executor.py` and `tests/test_engine_trace.py` pass with their assertions unchanged
(moved imports only).
- _Requirements: 4.3_
- [ ] 4.4 Engine smoke on kube-ax and record 04 **(coordinator, network)**
- Two pins through the new dispatcher on kube-ax, compared with their stored records.
- `benchmarks/measurements/<date>-unseen-04-dispatcher/record.json`: image digest, per-pin wall time and JVM
time, record digests equal to the stored ones (or the diff counted), cleanup result.
- _Requirements: 4.3_
- [ ] 5. Replay task (design 5.1, 5.2; increment 5)
- [x] 5.1 Image stage
- `plane/Dockerfile.engine-task`: second stage `FROM engine AS replay` adding `git`, the package with its
runtime dependencies and `benchmarks/unseen/replay_entry.py`; smoke `ousast pre-push --help`. The root image
stays `ghcr.io/norandom/ax-task-runner:v0.3.1`; the engine stage is unchanged.
- `tests/test_unseen_image.py`: parses the Dockerfile; the `replay` stage builds from `engine`; the root image
pin; the engine stage's text is byte-identical to before.
- _Requirements: 4.1_
- [x] 5.2 Task entry
- `benchmarks/unseen/replay_entry.py`, standard library only, reusing `engine_task_entry.transfer`: GET the spec;
`git init` and depth-1 fetches of head and base over HTTPS (180 s bound), verified with `git cat-file -e`; run
`ousast pre-push` with the spec's hook flags (defaults, or `--deadline 300` for the long-deadline arm), heap
1700 MB and the cluster JVM flags; pack `push.json`, `push.txt`, `engine.json`, `instrument.json` (files and
bytes in the checkout, bytes of each changed file, JVM wall time, hook exit code, step timings, image
digest); PUT as `.dat`. Any error PUTs a failure record naming the step and status, never URL text.
- `tests/test_unseen_replay_entry.py`: a local bare repository and a localhost presign stub; success record
fields; a missing commit, a failed fetch and a hook crash each give a failure record; zero bytes read and a
replay faster than JVM start are flagged `suspect_fast`.
- _Requirements: 3.1, 4.1, 4.2_
- [x] 5.3 Replay workload
- `benchmarks/unseen/replay.py`: builds `Item`s from a frozen slice and an arm (`default` or `long_deadline`);
the spec object carries the repository URL so names stay out of AX status and telemetry; task names
`ousast-engine-replay-<slug>-<digest>` with an opaque slug; the validator rejects a record without
`instrument.json`; records go to `~/ousast-results/unseen/<pool>/<slice>/<arm>/` and the store.
- Tests: no repository name in any task name or manifest; only the two presigned URLs in the env; input
digest changes with the arm's flags; `instrument_failure` items queued for one re-run.
- _Requirements: 3.1, 4.1, 4.2, 4.3_
- [ ] 5.4 Proof on kind, then kube-ax, and record 05 **(coordinator, network)**
- Build and push the `replay` target, pinned by digest. Five changes from this repository's own history (never
pool changes) on kind, then the same five on kube-ax, then on the VM lane.
- Exit: instrument fields show bytes read and realistic JVM time on every lane; quick-tier findings equal
across lanes.
- `benchmarks/measurements/<date>-unseen-05-replay-task/record.json`: image digest, per-lane timings, bytes,
exit codes, the quick-tier comparison.
- _Requirements: 4.1, 4.2, 4.5_
- [x] 6. Scorer (design 3, 4, 7.1, 7.3; increment 6)
- [x] 6.1 Coverage states
- `benchmarks/unseen/score.py`: each record gets one state (`instrument_failure`, `unanalysable`, `quick_only`,
`engine_covered`) from `instrument.json` and the artifact; `suspect_fast` goes to failure; a slice above 5%
`instrument_failure` after re-runs is refused.
- `tests/test_unseen_score.py` on synthetic records: every state; the refusal; a failure never counts as quiet
or missed.
- _Requirements: 3.3_
- [x] 6.2 Catch and false-alarm rules
- `score.py`: family match (engine family, quick-rule CWE through `taxonomy.py`), the location table of design
4.1, all three streams with the stream named per catch; false alarm on any finding of any stream; the
alerts-only rate.
- Tests: right family wrong place misses; `access_control` on the handler; `config_secrets` file scope; a
quick-rule-only catch; one advisory finding on an ordinary change is a false alarm.
- _Requirements: 3.2_
- [x] 6.3 Rates, intervals and breakdowns
- `score.py`: Wilson 95% intervals; repository-cluster bootstrap for false alarms; ICC; breakdowns per family,
method, `post_cutoff`, stream, coverage state and the confirmed subset.
- Tests: the n values of design section 3 (315, 483, 756, 62, 88, 97) reproduced; bootstrap seeded; ICC on a
synthetic clustered set.
- _Requirements: 2.3, 3.3_
- [x] 6.4 Budgets, paired tests, the gate and record 06
- `score.py`: per-budget thresholds (1%, 5%, 10%), labelled `exploratory` in-slice and `fixed` when supplied;
a binary arm's single operating point with the budgets it meets; exact two-sided McNemar for catch and for
false alarm; `gate(A, B)` returning `pass | regression | not_run` per design 7.3.
- Tests: thresholds on a synthetic score set; McNemar exact values; a binary arm with a significant
false-alarm increase gives `regression`; missing slice or excess failures give `not_run`.
- `benchmarks/measurements/<date>-unseen-06-scorer/record.json`: test counts and the digest of the scorer's
output on the committed synthetic fixture.
- _Requirements: 3.4, 5.4_
- [ ] 7. Baseline (design 4, 6; increment 7; G0 exit)
- [ ] 7.1 Frozen source and dry run **(coordinator, network)**
- Export HEAD to a frozen tree; build and push the `replay` image from it, pinned by digest.
- `benchmarks/push/python_overhead.py` run with the stub engine to size per-replay Python cost; dispatcher dry
run on slice 1 lists 1,300 items without launching.
- _Requirements: 5.1, 4.1_
- [ ] 7.2 Baseline replay of slice 1 **(coordinator, network)**
- Default arm, 1,300 replays over two AX lanes and the VM lane (about 9 hours); `instrument_failure` items
re-run once; the run stops for a cause above 5% failures.
- Observable: 1,300 records in `~/ousast-results/unseen/p1/1/default/` and the store, each with
`instrument.json`.
- _Requirements: 3.1, 4.3, 5.1_
- [ ] 7.3 Adjudication **(coordinator)**
- Flagged ordinary changes adjudicated against protocol-v2's standard (all if 30 or fewer, else every third);
the adjudicated share kept beside the false-alarm rate. ICC measured; above 0.05 noted as "next pool sizes
up".
- _Requirements: 3.2, 2.3_
- [ ] 7.4 Record 07 and the ledger **(coordinator)**
- `benchmarks/measurements/<date>-unseen-07-baseline/record.json`: `score.py` output (counts, rates, Wilson and
bootstrap intervals, coverage per state, per family, method, stream), freeze digest, image digest, commit.
`usage-p1.jsonl` row `baseline`.
- Observable: guard green over the record; G0's exit met.
- _Requirements: 5.1, 5.2, 3.3_
- [ ] 8. Arms on slice 1 (design 4.0, 5.4, 7.1-7.3; increment 8; G1)
- [ ] 8.1 Registered long-deadline arm
- `plane/experiments/` manifest for the long-deadline arm: hook flags with `--deadline 300`, budget 0,
`unseen: {pool: p1, slice: 1}`, committed before its first replay. `src/openultrasast/learn/experiments.py`
accepts `unseen` and a hook-flag arm.
- `tests/test_learn_experiments.py`: the manifest loads; an `unseen` block without pool or slice is refused.
- _Requirements: 3.4, 5.3_
- [ ] 8.2 `unseen` block and adoption gate in `analyse`
- `experiments.py` `analyse`: writes `unseen` into `result.json` (catch at each budget per arm, paired B - A,
McNemar, `gate`) from `score.py`; checks the usage ledger; adoption requires the fold rule and
`gate == pass`.
- Tests: `regression` and `not_run` each block adoption; an `informed` slice cannot qualify; result JSON shape.
- _Requirements: 5.2, 5.3, 5.4_
- [ ] 8.3 Engine-evidence arm
- `benchmarks/unseen/arms.py`: a score per change from `engine.json` (witness rows on head-side changed
functions), no model; plugs into the budget thresholds of 6.4.
- Tests on synthetic records: score monotone in witness strength; a change without engine coverage gets no
score and is reported, not counted quiet.
- _Requirements: 3.4_
- [ ] 8.4 Model-arm adapter
- `arms.py`: candidates from head-side changed functions of stored records, scored by the current decision
engine program (and the combiner when its program exists; else its row reads `not_built`); change score is the
maximum; metered client, dry run first, response cache, the manifest's `budget_usd` as the hard ceiling.
- Tests with a scripted client: dry-run estimate, ceiling stop, cache replay; no model call.
- _Requirements: 4.4, 3.4_
- [ ] 8.5 Long-deadline replay of slice 1 **(coordinator, network)**
- 1,300 replays of the `long_deadline` arm, bounded at about 345 s each (up to about 42 hours over three
lanes), with `STOP` and resume across sessions; failures re-run once.
- Observable: 1,300 records under `~/ousast-results/unseen/p1/1/long_deadline/`, JVM time in each.
- _Requirements: 3.1, 3.4, 4.3_
- [ ] 8.6 Model arms on slice 1 **(coordinator, network; paid: needs a budget go-ahead)**
- Dry run gives the candidate count; estimate at the injection slice's measured cost (about $0.008 per
candidate at k = 5); the maintainer sets the ceiling against the balance read just before. Then the decision
engine arm (and the combiner if built) on the VM.
- _Requirements: 4.4, 3.4_
- [ ] 8.7 Record 08: the arms table **(coordinator)**
- `benchmarks/measurements/<date>-unseen-08-arms/record.json`: per arm (hook 30 s, hook 300 s, engine evidence,
decision engine, combiner or `not_built`) catch at 1/5/10% budgets with Wilson intervals and realised
false-alarm rates, coverage per state, spend; `exploratory` rows in `usage-p1.jsonl`.
- Observable: one committed table of arms on slice 1; G1's table exists.
- _Requirements: 3.4, 5.2, 5.3_
## Implementation Notes
- 2026-10-04: Tasks 1.1–1.4 implemented offline and independently reviewed. Task 1.5 remains for the
coordinator; no networked measurement or commit was performed. Command and API contracts:
`benchmarks/unseen/README.md`. Offline evidence:
`benchmarks/measurements/2026-10-04-unseen-01-offline/record.json`.
- Population guard scope is preserved under the maintainer's hard rule: benchmarks/src/tests for
populations, plus plane for pools. Each directory exemption is reservation-specific, including private
pool manifests. Extending population scope to plane exposed 30 existing references; no historical
records were changed or exempted inside the existing population scope.
- S3 inventory opens through `open_store(read_only=True)` because ordinary store construction writes a
Select capability probe. Default store behavior is unchanged; inventory uses LIST/GET only.
- The freeze writer must match the canonical TOML digest encoding documented in the pool README and
exercised by `test_independent_population.py`; final pool discovery does not trust mutable status.
### Group 2 implementation evidence (2026-10-04)
Tasks 2.1–2.4 are implemented offline and independently reviewed. Task 2.5 is
`python -m benchmarks.unseen.draft --root "$PWD" --s3-store s3:// --seed 20260601`
with the documented PATH/PYTHONPATH and GH_TOKEN environment. No networked draw,
label check, group-2 measurement record, guard bisection, or commit was performed.
The group header remains pending its coordinator exits 2.5/2.6.
The command rebuilds the group-1 used set, resolves renamed/fork repositories,
uses disposable external blobless clones, re-reads licences at extracted heads,
and emits counts/digests only. The README documents injected clients, method bias,
the hard 300-repository floor, and an explicitly unmeasured resource estimate.
Independent review caught and fixed ambiguous parent-function absence and stale
candidate-404 handling; regression tests cover both.
Validation: requested pytest selection 103 passed, 1 skipped, exit 0 (population-test
output discarded); `ruff check .` exit 0; `ruff format --check .` exit 0 (411 files);
`mypy src/openultrasast` exit 0 (160 source files). The CLI `--help` smoke exited 0.
Source-read instrumentation printed 20,101 bytes for the extraction implementation.
### Groups 4–6 offline implementation (2026-10-04)
Tasks 4.1–4.3, 5.1–5.3 and 6.1–6.4 are implemented and independently reviewed.
Tasks 4.4 and 5.4 remain coordinator-owned network proofs; group headers 4 and 5 stay
pending those exits. No commit, network request, Docker, AX or kubectl command was run.
The sourcing job's draft.py, source.py, extract.py and journal.py are unchanged.
The shared dispatcher retains the engine adapter's historical questions.json object
name to preserve its unchanged assertions; replay uses spec.dat and result.dat.
The engine Dockerfile body is unchanged except its required AS engine alias; CI builds
engine and replay separately. S3Store's read_only class default preserves existing
constructor-bypassing transport fixtures; initialized read-only stores still refuse writes.
Exact coordinator commands and input contracts are in benchmarks/unseen/README.md,
including frozen TOML/private pointer verification, scoring joins, own-history proof
preparation and anonymous measurement-record generation. JVM wall time is the hook's
aggregate scan build/query timing and is labelled as such. Unsupported-only partitions
with explicit language_not_covered evidence do not pretend to have launched a JVM.
Record 06 contains the committed synthetic fixture and scorer-output digests. The
reserved-repository guard was run as a black box with both output streams discarded.
Independent review: APPROVED; all reported integration and coverage defects resolved.
Final verification: requested pytest command 279 passed, 8 skipped, exit 0; separate
`tests/test_ax_batch.py` 5 passed, exit 0; `ruff check .`, `ruff format --check .`,
`mypy src/openultrasast`, and the output-discarded reserved-repository guard all exit 0.
The own-history proof preparer and replay dry run read five items and exit 0.
+32
View File
@@ -0,0 +1,32 @@
# kube-ax operator constraints
Verified by the operator on 2026-10-05. These constraints apply across all workloads.
- Two workers, one task per worker, at most two tasks in flight cluster-wide. AX has no
queue: a third apply returns `no free workers available`. Treat this as back-pressure,
retry the same task/lane with 15–30 seconds of jitter until its deadline, and never
count it as a sandbox failure or trigger VM fallback. The host dispatcher shares a
two-slot reservation across workload instances and long-lived search executors.
- `/` and `/tmp` are RAM-backed and consume the worker's 3 GiB. `/workspace` is the only
disk-backed path, with about 2 GiB usable per task. Executor and verifier entrypoints
use `/workspace/tmp`, `/workspace/.npm`, `/workspace/.m2`, `/workspace/.composer`,
`/workspace/.pip`, and `/workspace/.gradle` through the respective tool environment
variables. Builds and extracted checkouts live under `/workspace` too. The default
aggregate workspace scratch guard is 2 GiB; `OUSAST_SCRATCH_BYTES` configures it.
Exceeding it returns `could_not_build`, reason `scratch limit`. The guard polls
during child execution and checks after exit; it is not an instantaneous filesystem
quota or a replacement for the worker memory limit.
- `ousast-engine-search-verify-*` tasks have no egress. The requested exception for
`files.because-security.com:443` is **pending**, not operationally verified. They
receive a presigned manifest for a gzip level-6 archive containing `checkout/`,
`products/`, and `spec.json`, built/exported by an executor. Compressed archives
retain the 1 GiB cap and travel in parts of at most 90 MB, each digest-checked
before assembly and extraction under `/workspace`. Only `VERIFY_INPUT_URL` and `RESULT_URL` cross the
verifier boundary. Verifiers reject build recipes and do not install dependencies
or fetch repositories. Executors finish and release their slots before verifier
repetitions start. Both task types must be provisioned without model credentials.
- All tasks share one public IP. Repository acquisition uses HTTPS git clone from
`github.com` or archives from `codeload.github.com`, never `api.github.com`. Host-side
metadata queries are a separate concern. Search tasks receive checkout archives;
executor tool guidance carries the same acquisition rule. Retain the 60-second
first-download retry for delayed task egress policy activation.
+60
View File
@@ -0,0 +1,60 @@
# Operator notifications (Discord)
Recorded 2026-10-09 from the maintainer's direction. OpenUltraSAST notifies the operator on
Discord about the status and results of its own processes, and especially when a process is
**stuck or otherwise impaired**. This is an operator channel, not a developer or infrastructure
one.
## Audience
The **system operator** and the **AI operator** who run and oversee the tool. Not developers
debugging the code, and not an infrastructure on-call. The messages answer "is the work moving,
did it finish, what did it produce, is it stuck" for someone supervising a run, not "why does this
function throw".
## Scope: OpenUltraSAST processes and their results
Notify on the actual OpenUltraSAST workloads and what they produce:
- scans and pre-push checks (a completed check and its verdict/degradations),
- unseen-repo sourcing / draft, freeze, and replay / baseline runs,
- learning and evaluation runs (compile, evaluate, experiments) and their recorded results,
- fusion / adjudication and any long-running job.
**Not** k3s or ax cluster infrastructure. Cluster, node, and ax-task health is a separate
infrastructure concern monitored elsewhere; this channel does not carry it. The line is: a message
here is about an OpenUltraSAST process or result, never about the plane's plumbing.
## When to notify
- **Completion:** a process finished, with a one-line result summary (counts, status, digest).
- **Stuck or impaired (the important case):** a stall, a clean-but-short result
(`insufficient_*`), an abort or `instrument_failure`, a timeout or deadline hit, budget
exhaustion, or a resource limit (disk, memory). These @mention the operator so a supervised run
does not sit silently broken.
Advisory and low-volume by design: a start/finish and the impaired cases, not per-step chatter.
## Transport and secrets
A Discord webhook, with credentials referenced by name only and stored in gitignored `.env`, like
every other credential (`.env.example` lists the names, blank):
- `DISCORD_WEBHOOK_URL` — the operator channel webhook.
- `DISCORD_OPERATOR_ID` — the account to @mention on a stuck/impaired alert.
The webhook URL is a secret (anyone holding it can post to the channel): it never enters a tracked
file, a prompt, a log, or a commit, and it is rotated if exposed. Notifications are **off** when the
webhook is unset, so the tool runs the same with or without it.
## Content discipline
Messages carry status, counts, and digests only, under the same redaction as the hardening and
measurement-record conventions: no exploit payloads or request strings, no secrets, no customer
content, no repository contents. A notification is safe to read in a shared channel.
## Status
Policy recorded; the notifier (reading the env-named webhook, quiet when unset, honoring the
redaction above) is to be wired into the process entry points and may be specced separately. Until
then this steering states the intended behavior for any process that adds notifications.
+22 -8
View File
@@ -1,8 +1,23 @@
# Overview — what this is, and what is actually true of it
## Current state, 2026-10-02
## Current state, 2026-10-04
Everything below this section is dated history; where it disagrees with this section, this section wins. The
direction lives in `roadmap.md`, section of 2026-10-04 (phases G0-G5).
- **Goal.** A pre-push safety net that works on repositories it has never seen. Two numbers per push decide it:
catch rate on real vulnerability-introducing changes, and false-alarm rate on ordinary changes. Neither has been
measured yet; the unseen-repository pool (G0, spec `unseen-repo-evaluation`) comes first.
- **Correction, 2026-10-04.** Infrastructure and engine work had drifted from that goal. It is now done only when
it removes a bottleneck for those measurements.
- **Engine.** Joern runs on the operator's kube-ax cluster as AX Tasks, with a VM fallback lane; kind stays the
development profile. It is almost blind to fixes: paths differ between sides in 2 of 105 both-asked pairs.
- **Decision engine.** Ranks but cannot block (exp-003 to exp-005 inconclusive; 710 fix pairs in memory). Fix
mechanisms is one G2 arm, paused after mining until the pool exists.
- **Hook.** `ousast pre-push` is operational and advisory on any repository; rollout stays NO-GO.
## State, 2026-10-02 (superseded by the section above)
Everything below this section is dated history; where it disagrees with this section, this section wins.
User-facing documentation: `README.md`, `docs/architecture.md`, `docs/evaluation.md`, `docs/decision-engine.md`,
`ops/ax/README.md`.
@@ -200,10 +215,10 @@ Counted 2026-10-02 from each spec's `tasks.md` checkboxes and `spec.json`.
|---|---|---|
| `learned-decision-engine` | implementation | 11 of 17 tasks done |
| `pre-push-safety-net` | implementation | 67 of 79 tasks done |
| `contributor-scan` | implementation | 27 of 51 tasks done |
| `ai-service-plane` | implementation | 11 of 11 tasks done; the ax plane is the agentic path |
| `contributor-scan` | paused | 27 of 51 tasks done; paused since 2026-09-23 (`v1.2.0-alpha.1` NO-GO), waits on pre-push M2/M3 |
| `ai-service-plane` | complete | 11 of 11; the ax plane is the agentic path |
| `harnessx-removal` | complete | 8 of 8; HarnessX removed 2026-09-30 |
| `finding-feedback-loop` | design | requirements approved, design awaiting approval |
| `finding-feedback-loop` | dormant | requirements approved, design unapproved since 2026-09-09; depends on `contributor-scan` |
| `model-grounded-detection` | complete | 16 of 16; built the arbiter this all rests on |
| `authorization-obligations` | dormant | tasks approved, 0 of 24 started; superseded in practice by the dominance arbiter |
| `reachability-flow-model` | dormant | 23 tasks generated, nothing approved; largely delivered by adopting Joern |
@@ -214,6 +229,5 @@ Counted 2026-10-02 from each spec's `tasks.md` checkboxes and `spec.json`.
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax (`ai-service-plane`),
> and evolutionary proposals come from the plane memory. The HarnessX specs in the table above are history.
As of 2026-09-09: 97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests.
150 source modules, 145 load-bearing and 5 standalone capabilities, 0 orphaned
(`benchmarks/measurements/2026-09-08-module-audit.json`, regenerated 2026-10-02); 1,964 tests collected.
151 source modules, 146 load-bearing and 5 standalone capabilities, 0 orphaned
(`benchmarks/measurements/2026-09-08-module-audit.json`, regenerated 2026-10-02); 2,023 tests collected.
+138
View File
@@ -1,5 +1,143 @@
# Roadmap
## Roadmap to detection with the decision engine on kube-ax (2026-10-04)
This section supersedes the ordering below it. The older sections stay as history.
### Destination
On every push, from a hook or CI, OpenUltraSAST answers for the changed code. The answer is advisory for every
family, and BLOCK only for families whose precision has been proven on repositories the system never saw. The work
is split by what each place is good at:
- **kube-ax cluster** (operator k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes): the Joern engine, as AX Tasks.
No API keys and no store credentials in the cluster; tasks read and write through short-lived presigned links.
- **The VM or CI runner**: candidate selection, the decision engine's model calls (DeepSeek), the memory store,
and the fallback engine lane (2.5 GB heap) for pins that do not fit the cluster's 1.7 GB heap.
- **kind** stays the local development profile, with the same code and manifests.
### The question everything answers
Does the safety net work on repositories it has never seen? That is the purpose of the A/B experiments, and every
phase below is judged by it. On an unseen repository almost every push is an ordinary change, so two numbers decide
whether the safety net is useful, measured on the hook's advisory output exactly as a user would see it:
1. **Catch rate:** on real vulnerability-introducing changes in unseen repositories, how often it flags the change.
2. **False-alarm rate:** on ordinary changes in the same unseen repositories, how often it flags anything.
Within-pair AUC on known fix pairs answers neither. It stays a development signal, not the goal.
### Where we are (2026-10-04)
| Area | State | Evidence |
| --- | --- | --- |
| Generalisation measured so far | The classifier ranks above chance on repository-held-out folds (within-pair AUC about 0.76) but cannot reach the block bar; the engine-based scan failed fresh populations (v1 1/11, v2 0/17); push-level catch and false-alarm rates on unseen repositories have never been measured | exp-003/004/005, `results-v1.json`, `results-v2.json` |
| Pre-push hook | Operational and advisory on any repository | `2026-10-02-pre-push-operational-1/record.json` |
| Labelled data | 710 complete real fix pairs | `2026-10-04-decision-engine-harvest-3/record.json` |
| Engine | Almost blind to fixes (paths differ between sides in 2 of 105 both-asked pairs) | `2026-10-04-joern-slice-leak-audit/record.json` |
| Infrastructure | Engine runs on kube-ax as AX Tasks with a VM fallback; kind for development | memory `kube-ax-cluster` |
Correction recorded 2026-10-04: the previous two days went mostly into infrastructure and engine internals that did
not move a generalisation number. From here, infrastructure and engine work is done only when it removes a
bottleneck for the measurements below.
### Where we are (2026-10-11)
The G0 measurement is now runnable: the unseen pool is **frozen** and the whole pipeline is hardened and
**bucket-first**. The baseline replay is the one remaining step, to run on kube.
| Area | State | Evidence |
| --- | --- | --- |
| Unseen pool (G0.1) | **Frozen.** `pool-p1.toml` = 100 repos (slice 1); `freeze-p1.json` guard=0, dropped=0; deterministic (manifest `6bd2fa5d`). License 49 permissive / 51 private; methods 81 introducing / 19 last_touch. | `benchmarks/unseen/pool-p1.toml`; archived `ousast-corpus/unseen/20260601/pool/` |
| Labels | 100 machine-derived (SZZ), **unchecked**. Record 07 is a **machine-label baseline** (caveat stated); a human label pass is a later refinement, before any published claim. | `freeze-p1.json` label_check_counts |
| Harvest | Rebuilt to converge. Pure GitHub-API harvesting is a dead end for this benchmark (it needs the SZZ introducing commit + 12 stratified history negatives, not one commit). The fix: a full-objects clone (drop `--filter=blob:none`, which caused ~600 per-blob lazy fetches) + bucket-first durability + a get_repo cache that paid the 4,499-name used-set canonicalization once. Yield ~2.2%; **GitHub-rate-limit bound, not CPU**. | 95bed70, 857263d, 21aebe8, 7b711a6 |
| Persistence | **Bucket-first, not local disk.** `ousast-corpus` is the source of truth: enumeration, resolution, get_repo, and accept/reject decisions all stream to and resume from it. The host disk (~9-13 GB free) is the scarce resource; RustFS/S3 has the room, so run artifacts and **replay outputs go to RustFS too**, not `~/ousast-results` on local disk. | `storage-streaming.md`, `bank.py` |
| Replay image | Provenance pinned. CI `engine-image.yml` builds `ousast-replay-task` (harness with `/app/benchmarks/unseen/replay_entry.py`) + `ousast-engine-task` (Joern hook), publicly pullable by digest. The old `ousast-runner:dev` (248 MB ax-runner, no `/app/benchmarks`, no Joern) is the **wrong** image for replay. | CI run 38116281676 |
| Baseline (G0.2) | **Not yet.** Replay reads the frozen pool cleanly (dry-run = 1,300 changes, `freeze_digest` verified). Remaining: mirror CI's passing `replay` job wiring, and stand up the kube lane (`ax` binary + a public-HTTPS presign-capable store endpoint; GHCR pull already public). | CI `replay` job (passed) |
Lesson, recorded so it is not re-paid: sourcing and engine-image work cost real time. The durable takeaway now lives
in the pipeline and these docs — bucket-first persistence, pinned image provenance, a measured yield — so the next
pool or arm is cheap, not a rediscovery.
### Readiness plan (to G0 exit = record 07)
1. **Kube replay lane (infra, operator-owned):** install the `ax` binary; provide the store endpoint the AX lane
needs. The only thing settled in-repo is a code fact: `benchmarks/ax/batch.py` currently expects a public-HTTPS
DNS, presign-capable S3 endpoint. How that endpoint and the cluster's access to the store are provided is an
operator/AX decision, not an assumption to make here. GHCR pull of the images is verified public.
2. **Unblock the replay** by mirroring CI's passing `replay` job (replay-task + engine-task wired, store + network).
3. **Run slice-1 baseline on kube:** `--lane ax --arm default`, 1,300 changes, no model cost, resumable; outputs to
RustFS.
4. **Adjudicate (7.3) -> record 07 (7.4)** = the measured machine-label unseen baseline = **G0 exit**.
5. Then **G1 arms** (DeepSeek decision engine) on the frozen pool.
### Phases
**G0. The unseen-repository pool and the push-level metric** (new spec `unseen-repo-evaluation`)
1. A frozen pool of repositories never used for training, tuning, mining or harvesting, and not in any population.
Two kinds of change per repository: real vulnerability-introducing commits (the parent of a fix, post-cutoff where
possible) and ordinary commits from the same history.
2. The metric: replay each change through `ousast pre-push` (and the decision engine where an arm uses it) and
record catch rate and false-alarm rate, with confidence intervals, per family and overall.
3. Rotation: the pool is spent per decision, like the populations; a new slice is drawn for the next one.
Exit: the pool is frozen and the current production hook has a measured baseline on it.
**G1. Re-score the arms we have on the pool**
Current hook (quick rules + engine), decision-engine advisory (current programs), combiner, engine evidence. Each is
an arm with a registered decision rule, scored by catch rate at a fixed false-alarm budget.
Exit: one measured table of arms on unseen code; keep only what generalises.
**G2. Improve what moves the unseen-repo numbers**
Candidates, each admitted only if it improves the pool metric on a fresh slice: fix mechanisms in the engine
(approved plan, now one arm, not the spine), engine coverage and TypeScript, decision-engine prompt and retrieval
changes, more labelled pairs from more distinct repositories.
Active G2 workstreams (2026-10-11, opened while driving G0; progress tracked in their briefs):
- **Change-scoped engine analysis** (`.kiro/specs/engine-change-scoped-analysis/brief.md`). Live diagnosis on the
frozen pool showed the engine's "no finding" on unseen changes is usually *analysis-never-runs*: the engine ranks
repo-wide candidates and `deferred: deadline_exhausted` before analyzing anything (even at 300 s), and the ranked
sites are unrelated to the pushed change. Lever: seed analysis from the change's declarations + dataflow
neighbourhood (`rank.py`/`mapping.py`) so it runs within budget. Increment 0 (confirm CPG-build-vs-analysis cost
split) gates the rest. Scale measurement needs the kube lane.
- **Line-correspondence (done, necessary-not-sufficient).** `snapshot.py` discarded base↔head anchors on
non-globally-unique lines → `comparison_unknown` → suppressed findings; fixed to run-aware anchoring
(commit c39fbb8, 163 tests). Removes one cause; coverage still needs change-scoped analysis + the capability
registry (Task 8.3).
**G3. Decide BLOCK per family** on the pool's false-alarm rate and the pooled precision bound (Req 6.4).
**G4. Detection on push for any repository** (`plane-on-kubernetes` P4 work: `ousast plane scan --base --head`,
engine pins on kube-ax, model calls on the VM or CI), sized by what G1-G3 kept.
**G5. Qualification and rollout:** adopt, commit `prediction-v3.json`, spend population v3 once, advisory rollout,
BLOCK only for families that passed.
### Cross-cutting rules
- No hand-tuning per miss. Every detection change is a registered experiment or an admission by held-out
separation.
- Records carry counts, digests and pattern names, never repository names: the reserved-repository guard covers
every population.
- Instrument first: a zero from an unread input is a failed run (AGENTS.md).
- Cluster stability: sandbox deaths are retried once, then sent to the VM lane; the operator owns workers, egress
and Substrate.
### Spec map
| Spec | Role in this roadmap | State |
| --- | --- | --- |
| `unseen-repo-evaluation` (new) | G0, G1: the pool and the push-level metric | To specify |
| `learned-decision-engine` | G1-G3, G5; fix-mechanisms plan is one G2 arm | Implementation; experiments 3-5 recorded |
| `plane-on-kubernetes` | G4; the engine executors for kube-ax | Groups 1-2 done (kind); production path re-scoped to the operator's kube-ax (AX Tasks), groups 3-8 to re-plan |
| `pre-push-safety-net` | the hook replayed in G0-G1; M4-M5 in G5 | Operational increment 1 done; M4-M5 open |
| `ai-service-plane` | The plane the above runs on | Complete |
## Status, 2026-10-02
The milestone table below dates from 2026-09-14. Since then (`.kiro/specs/pre-push-safety-net/release-milestones.md`):
+16
View File
@@ -142,3 +142,19 @@ promising but does not qualify security queries or hook latency. The contributor
vendor/correspondence completeness vetoes, and query-to-operation identity as the next
shared-contract work. Resolving a boundary requires evidence of its relevance and scope;
do not simply drop completeness checks to increase admissions. Rollout remains NO-GO.
On 2026-10-07, the **search-with-proof** arm reached its first milestone: an autonomous,
execution-backed `demonstrated` finding end to end on the kube-ax cluster. On a real CVE
fix-pair (hanxi/xiaomusic, path traversal of the prefix-check-bypass class), the agent
found the vulnerability, authored a declarative demo, and the differential verifier
confirmed the owned effect on the vulnerable revision and its absence on the fixed one;
a blind search of the fixed revision produced zero false proofs. This is the proof-based
BLOCK mechanism the maintainer directed after classification hit a precision wall: a BLOCK
rests on a demonstrated finding with a direct execution trace, not a confidence score.
Scope note: this is feasibility-pilot evidence on a development-corpus pair (known location,
labelled family, runnable-project selection bias), not release readiness. The pilot exit
(≥5 demonstrations in one family with 0 false proofs, Req 6.2) and the ≥73-negative BLOCK
qualification (Req 6.3) remain open, and no capability is enabled. The decision/classification
engine is retained as a triage ranker (it could not prove 95% precision as a gatekeeper);
proof is the BLOCK gate. See `search-with-proof/HANDOFF.md` and
`benchmarks/measurements/2026-10-07-search-first-cluster-demonstration/`.
+80
View File
@@ -0,0 +1,80 @@
# Stream run artifacts to RustFS, one bucket per purpose
Recorded 2026-10-09 from the maintainer's direction. The host's local disk is the scarce
resource (it runs near full, and the draw stops itself at under 2 GB free). Runs should stream
their durable artifacts to RustFS instead of accumulating them on local disk, and should spread
them across **purpose-specific buckets** rather than one shared bucket. This relieves the disk and
simplifies runs: local state becomes mostly a scratch cache that can be rebuilt or resumed from the
object store.
## Why per-purpose buckets
One shared bucket (`sast-memory`) already bit us: an append-rewritten key on a versioned bucket
with no noncurrent-version expiry ballooned to >10 GB (see the `engine-queue` command-upload fix and
the lifecycle rules added 2026-10-09). Separate buckets let each kind of artifact carry the
lifecycle it actually needs, and keep a transient firehose from polluting durable data.
Proposed split (names illustrative; the admin creates them and grants the store key):
- `sast-memory` — the plane memory store (existing): facts, excerpts, embeddings, index. Durable,
versioned, with the noncurrent-version lifecycle now in place.
- `sast-pools` — frozen unseen pools and their provenance: `pool-pN.toml`, `freeze-pN.json`, and the
draw journal needed to resume/reproduce. Durable, versioned. This is also the "don't redo the
sourcing" archive.
- `sast-runs` (or `sast-journals`) — ephemeral run and queue state (engine-queue, replay results,
per-run journals). Aggressive lifecycle: short current expiry + 1-day noncurrent expiry, since it
is a transient firehose.
- optionally `sast-traces` — engine/replay traces, if kept.
## Principles
- **Stream, don't hoard.** Write a run's durable outputs to its bucket as they are produced; keep
only a working scratch (e.g. the live git clone, which cannot be streamed) on local disk, and let
the disk guard stay as the backstop.
- **Lifecycle per bucket**, tuned to the artifact: durable+versioned for pools/memory;
short+noncurrent-expiry for transient run/queue state. Never a versioned bucket without a
noncurrent-version rule.
- **Resume from the object store**, so a wiped or small local disk does not lose a long run.
- **Least privilege per bucket.** A run gets a key scoped to the bucket it writes; the store never
creates or configures a bucket (admin does that once), consistent with the current S3Store.
- Content discipline unchanged: no secrets, no exploit payloads; pool manifests keep the existing
public/private (licence) split.
## Foundation and dependency
`src/openultrasast/plane/memory.py` already speaks boto3 to RustFS (S3Store) with startup
verification of versioning/lifecycle/Select/tags; this direction extends that pattern to more
buckets rather than inventing a new transport. New buckets and their grants are an **operator
admin** action — the store credentials cannot create buckets or set lifecycle (confirmed by the
`AccessDenied` on the 2026-10-09 lifecycle apply).
## Default: RustFS over local disk
Local disk on the VM is the scarce resource (it runs near full; the draw stops itself under 2 GB free).
RustFS/S3 has far more room. So the default for any run's durable state and large artifacts is the bucket,
not local disk. Local files are a rebuildable scratch cache only (the live git clone, a within-process
journal). A long run must be resumable purely from the bucket, and its outputs — including replay result
records — are written to RustFS, not `~/ousast-results` on local disk.
## Status (implemented 2026-10-10/11)
Bucket-first persistence is live, not a future spec. The unseen draw streams every stage to the
`ousast-corpus` bucket (`http://10.0.0.54:9001`, creds in `.env` as `CORPUS_S3_*`; boto3 needs path-style +
SigV4) and resumes from it:
- `unseen/<seed>/decisions/<sha256(repo+fix)>.json` — accept/reject per repository (idempotent; content dedup).
- `unseen/<seed>/resolved/<sha256(ghsa)>.json` — per-advisory resolution.
- `unseen/<seed>/advisories/list.json` — the enumerated advisory list.
- `unseen/repos/<sha256(name)>.json` — get_repo cache, **not** seed-scoped (shared across seeds/runs), so the
thousands of used-set canonicalization lookups are paid once ever.
- `unseen/<seed>/pool/` — the frozen pool and its provenance (`pool-p1.toml`, `private-pool-p1.toml`,
`freeze-p1.json`, `bisect-p1.json`, `protocol-p1.md`): the durable "don't redo the sourcing" archive.
Implementation: `benchmarks/unseen/bank.py` (`CorpusBank`, `corpus_settings`). The old local journal is a
within-process cursor only; the bucket is the cross-run source of truth, so a restart (even after a code
change) resumes cheaply. Still to do and consistent with the default above: replay result records
(`replay.py --out`) currently default to `~/ousast-results/unseen/...` on local disk — point them at RustFS
for the slice-1 baseline and after. The AX (kube) lane already uses presigned object transfer, which is the
same principle. One in-repo code fact: `benchmarks/ax/batch.py` currently expects a public-HTTPS DNS,
presign-capable endpoint. How the store endpoint and the cluster's access to it are provided is operator/AX-owned
infra, not something this doc should presume.
-165
View File
@@ -1,165 +0,0 @@
---
name: kiro-spec-batch
description: Create complete specs (requirements, design, tasks) for all features in roadmap.md using parallel sub-agent dispatch by dependency wave.
---
# Spec Batch
<background_information>
- **Success Criteria**:
- All features have complete spec files (spec.json, requirements.md, design.md, tasks.md)
- Dependency ordering respected (upstream specs complete before downstream)
- Independent features processed in parallel via sub-agent dispatch
- Cross-spec consistency verified (data models, interfaces, naming)
- Mixed roadmap context understood without breaking `## Specs (dependency order)` parsing
- Controller context stays lightweight (sub-agents do the heavy work)
</background_information>
<instructions>
## Step 1: Read Roadmap and Validate
1. Read `.kiro/steering/roadmap.md`
2. Parse the `## Specs (dependency order)` section to extract:
- Feature names
- One-line descriptions
- Dependencies for each feature
- Completion status (`[x]` = done, `[ ]` = pending)
3. If present, also read for context:
- `## Existing Spec Updates`
- `## Direct Implementation Candidates`
Do not include these in dependency-wave execution; they are awareness-only inputs for sequencing and consistency review.
4. For each pending feature in `## Specs (dependency order)`, verify `.kiro/specs/<feature>/brief.md` exists
5. If any brief.md is missing, stop and report: "Missing brief.md for: [list]. Run `/kiro-discovery` to generate briefs first."
## Step 2: Build Dependency Waves
Group pending features into waves based on dependencies:
- **Wave 1**: Features with no dependencies (or all dependencies already completed `[x]`)
- **Wave 2**: Features whose dependencies are all in Wave 1 or already completed
- **Wave N**: Features whose dependencies are all in earlier waves or already completed
Display the execution plan:
```
Spec Batch Plan:
Wave 1 (parallel): app-foundation
Wave 2 (parallel): block-editor, page-management
Wave 3 (parallel): sidebar-navigation, database-views
Wave 4 (parallel): cli-integration
Total: 6 specs across 4 waves
```
If roadmap contains `## Existing Spec Updates` or `## Direct Implementation Candidates`, mention them separately as non-batch items so the user can see the whole decomposition.
## Step 3: Execute Waves
For each wave, dispatch all features in the wave as **parallel sub-agents**.
**For each feature in the wave**, spawn a sub-agent with this task:
```
Create a complete specification for feature "{feature-name}".
1. Read the brief at .kiro/specs/{feature-name}/brief.md for feature context
2. Read the roadmap at .kiro/steering/roadmap.md for project context
3. Execute the full spec pipeline. For each phase, read the corresponding skill's SKILL.md for complete instructions (templates, rules, review gates):
a. Initialize: Read .opencode/skills/kiro-spec-init/SKILL.md, then create spec.json and requirements.md
b. Generate requirements: Read .opencode/skills/kiro-spec-requirements/SKILL.md, then follow its steps
c. Generate design: Read .opencode/skills/kiro-spec-design/SKILL.md, then follow its steps
d. Generate tasks: Read .opencode/skills/kiro-spec-tasks/SKILL.md, then follow its steps
4. Set all approvals to true in spec.json (auto-approve mode, equivalent of -y flag)
5. Report completion with file list and task count
```
If multi-agent is not available, execute features in the wave sequentially.
**After all sub-agents in the wave complete**:
1. Verify each feature has: spec.json, requirements.md, design.md, tasks.md
2. If any feature failed, report the error and continue with features that succeeded
3. Display wave completion: "Wave N complete: [features]. Files verified."
4. Proceed to next wave
## Step 4: Cross-Spec Review
After all waves complete, spawn a **single sub-agent** for cross-spec consistency review. This is the highest-value quality gate -- it catches issues that per-spec review gates cannot.
**Sub-agent task**:
Read ALL generated specs and check for consistency across the entire project:
- `.kiro/specs/*/design.md` (primary: contains interfaces, data models, architecture)
- `.kiro/specs/*/requirements.md` (for scope and acceptance criteria)
- `.kiro/specs/*/tasks.md` (for boundary annotations only -- read _Boundary:_ lines, skip task descriptions)
- `.kiro/steering/roadmap.md`
Reading priority: Focus on design.md files (they contain interfaces, data models, architecture). For requirements.md, focus on section headings and acceptance criteria. For tasks.md, focus on _Boundary:_ annotations.
Check:
1. **Data model consistency**: Same entities defined consistently across specs (field names, types, relationships)
2. **Interface alignment**: Where spec A outputs what spec B consumes, do contracts match exactly?
3. **No duplicate functionality**: Any capability specified in more than one spec?
4. **Dependency completeness**: Every design.md references correct upstream specs? Implicit dependencies not in roadmap?
5. **Naming conventions**: Component names, file paths, API routes, table names consistent across specs?
6. **Shared infrastructure**: Shared concerns (auth, error handling, logging) handled in one spec and correctly referenced?
7. **Task boundary alignment**: Task _Boundary:_ annotations partition codebase cleanly? No files claimed by multiple specs?
8. **Roadmap boundary continuity**: If roadmap includes `Existing Spec Updates` or `Direct Implementation Candidates`, do the generated new specs avoid absorbing that work by accident?
9. **Architecture boundary integrity**: Do the specs preserve clean responsibility seams, avoid shared ownership, keep dependency direction coherent, and include enough revalidation triggers to catch downstream impact?
10. **Change-friendly decomposition**: Has any spec absorbed multiple independent seams that should probably be split instead of kept together?
Output: CONSISTENT areas + ISSUES with (which specs, what's inconsistent, suggested fix).
**After the review sub-agent returns**:
- **Critical/important issues found**: Dispatch fix sub-agents for each affected spec to apply the suggested fixes. If the issue is really a decomposition problem (for example boundary overlap or one spec carrying multiple independent seams), stop and return to roadmap/discovery instead of papering over it locally. Re-run cross-spec review after fixes (max 3 remediation rounds).
- **Minor issues only**: Report them for user awareness, proceed to Step 5.
- **No issues**: Proceed to Step 5.
## Step 5: Finalize
1. Scan `.kiro/specs/*/tasks.md` to verify all specs exist
2. For each completed spec, read spec.json to confirm phase and approvals
3. Update roadmap.md: mark completed specs as `[x]`
4. If roadmap.md includes `Existing Spec Updates` or `Direct Implementation Candidates`, leave them untouched and mention them as remaining follow-up items unless already explicitly completed elsewhere
Display final summary:
```
Spec Batch Complete:
✓ app-foundation: X requirements, Y design components, Z tasks
✓ block-editor: ...
✓ page-management: ...
...
Total: N specs created, M tasks generated
Cross-spec review: PASSED / N issues found (M fixed)
Existing spec updates pending: <count or none>
Direct implementation candidates pending: <count or none>
Next: Review generated specs, then start implementation with /kiro-impl <feature>
```
</instructions>
## Critical Constraints
- **Controller stays lightweight**: Only read roadmap.md and brief.md existence checks in main context. All spec generation happens in sub-agents.
- **Wave ordering is strict**: Never start a wave until all features in previous waves are complete.
- **Parallel within waves**: All features in the same wave should be dispatched in parallel if multi-agent is available.
- **No partial waves**: If a feature in a wave fails, still complete the other features in that wave before reporting.
- **Skip completed specs**: Features with `[x]` in roadmap.md or existing tasks.md are skipped.
- **`## Specs (dependency order)` remains authoritative for batch execution**: Other roadmap sections are context, not wave inputs.
## Safety & Fallback
**Sub-agent failure**:
- Log the error, skip the failed feature
- Continue with remaining features in the wave
- Report failed features in the summary
- Suggest: "Run `/kiro-spec-quick <feature> --auto` manually for failed features."
**Circular dependencies**:
- If dependency graph has cycles, report the cycle and stop
- Suggest: "Fix dependency ordering in roadmap.md"
**Roadmap not found**:
- Stop and report: "No roadmap.md found. Run `/kiro-discovery` first."
**All specs already complete**:
- Report: "All specs in roadmap.md are already complete. Nothing to do."
-256
View File
@@ -1,256 +0,0 @@
---
name: kiro-spec-quick
description: Quick spec generation with interactive or automatic mode
---
# Quick Spec Generator
<instructions>
## CRITICAL: Automatic Mode Execution Rules
**If `--auto` flag is present in `$ARGUMENTS`, you are in AUTOMATIC MODE.**
In Automatic Mode:
- Execute ALL 4 phases in a continuous loop without stopping
- Display progress after each phase (e.g., "Phase 1/4 complete: spec initialized")
- IGNORE any "Next Step" messages from Phase 2-4 (they are for standalone usage)
- After Phase 4, run the final sanity review before exiting
- Stop ONLY after the sanity review completes or if error occurs
---
## Core Task
Execute 4 spec phases sequentially. In automatic mode, execute all phases without stopping. In interactive mode, prompt user for approval between phases.
Before claiming quick generation is complete, run one lightweight sanity review over the generated requirements, design, and tasks. If the host supports fresh sub-agents, use one. Otherwise run the sanity review inline.
## Execution Steps
### Step 1: Parse Arguments and Initialize
Parse `$ARGUMENTS`:
- If contains `--auto`: **Automatic Mode** (execute all 4 phases)
- Otherwise: **Interactive Mode** (prompt at each phase)
- Extract description (remove `--auto` flag if present)
Example:
```
"User profile with avatar upload --auto" → mode=automatic, description="User profile with avatar upload"
"User profile feature" → mode=interactive, description="User profile feature"
```
Display mode banner and proceed to Step 2.
### Step 2: Execute Phase Loop
Execute these 4 phases in order:
---
#### Phase 1: Initialize Spec (Direct Implementation)
**Core Logic**:
1. **Check for Brief**:
- If `.kiro/specs/{feature-name}/brief.md` exists (created by `/kiro-discovery`), read it for discovery context (problem, approach, scope, constraints)
- Use brief content as the project description instead of `$ARGUMENTS`
2. **Generate Feature Name**:
- Convert description to kebab-case
- Example: "User profile with avatar upload" → "user-profile-avatar-upload"
- Keep name concise (2-4 words ideally)
3. **Check Uniqueness**:
- Use Glob to check `.kiro/specs/*/`
- If directory exists with only `brief.md` (no `spec.json`), use that directory (discovery created it)
- Otherwise if feature name exists, append `-2`, `-3`, etc.
4. **Create Directory**:
- Use Bash: `mkdir -p .kiro/specs/{feature-name}` (skip if already exists from discovery)
5. **Initialize Files from Templates**:
a. Read templates:
```
- .kiro/settings/templates/specs/init.json
- .kiro/settings/templates/specs/requirements-init.md
```
b. Replace placeholders:
```
{{FEATURE_NAME}} → feature-name
{{TIMESTAMP}} → current ISO 8601 timestamp (use `date -u +"%Y-%m-%dT%H:%M:%SZ"`)
{{PROJECT_DESCRIPTION}} → description
en → language code (detect from user's input language, default to `en`)
```
c. Write files using Write tool:
```
- .kiro/specs/{feature-name}/spec.json
- .kiro/specs/{feature-name}/requirements.md
```
6. **Output Progress**: "Phase 1/4 complete: Spec initialized at .kiro/specs/{feature-name}/"
**Automatic Mode**: IMMEDIATELY continue to Phase 2.
**Interactive Mode**: Prompt "Continue to requirements generation? (yes/no)"
- If "no": Stop, show current state
- If "yes": Continue to Phase 2
---
#### Phase 2: Generate Requirements
Invoke `/kiro-spec-requirements {feature-name}`.
Wait for completion. IGNORE any "Next Step" message (it is for standalone usage).
**Output Progress**: "Phase 2/4 complete: Requirements generated"
**Automatic Mode**: IMMEDIATELY continue to Phase 3.
**Interactive Mode**: Prompt "Continue to design generation? (yes/no)"
- If "no": Stop, show current state
- If "yes": Continue to Phase 3
---
#### Phase 3: Generate Design
Invoke `/kiro-spec-design {feature-name} -y`. The `-y` flag auto-approves requirements.
Wait for completion. IGNORE any "Next Step" message.
**Output Progress**: "Phase 3/4 complete: Design generated"
**Automatic Mode**: IMMEDIATELY continue to Phase 4.
**Interactive Mode**: Prompt "Continue to tasks generation? (yes/no)"
- If "no": Stop, show current state
- If "yes": Continue to Phase 4
---
#### Phase 4: Generate Tasks
Invoke `/kiro-spec-tasks {feature-name} -y`. The `-y` flag auto-approves requirements, design, and tasks.
Wait for completion.
**Output Progress**: "Phase 4/4 complete: Tasks generated"
#### Final Sanity Review
After Phase 4, run a lightweight sanity review before claiming completion.
- Review `requirements.md`, `design.md`, and `tasks.md` directly from disk. If `brief.md` exists, use it only as supporting context.
- Prefer a fresh review sub-agent when the host supports it. Pass only file paths and the review objective; the reviewer should read the generated files itself.
- Review focus:
- Do requirements, design, and tasks tell a coherent story?
- Are there obvious contradictions, missing prerequisites, or missing task coverage for required design work?
- Are `_Depends:_`, `_Boundary:_`, and `(P)` markers plausible for implementation?
- If the review finds only task-plan-local issues, repair or update the generated `tasks.md` once, then re-run the sanity review.
- If the review finds a real requirements/design gap or contradiction, stop and report follow-up instead of claiming the quick spec is implementation-ready.
**All 4 phases plus sanity review complete.**
Output final completion summary (see Output Description section) and exit.
---
## Important Constraints
### Error Handling
- Any phase failure stops the workflow
- Display error and current state
- Suggest manual recovery command
</instructions>
## Output Description
### Mode Banners
**Interactive Mode**:
```
Quick Spec Generation (Interactive Mode)
You will be prompted at each phase.
Note: Skips gap analysis and design validation.
```
**Automatic Mode**:
```
Quick Spec Generation (Automatic Mode)
All phases execute automatically without prompts.
Note: Skips optional validations (gap analysis, design review) and user approval prompts. Internal review gates still run.
Final sanity review still runs.
```
### Intermediate Output
After each phase, show brief progress:
```
Spec initialized at .kiro/specs/{feature}/
Requirements generated → Continuing to design...
Design generated → Continuing to tasks...
```
### Final Completion Summary
Provide output in the language specified in `spec.json`:
```
Quick Spec Generation Complete!
## Generated Files:
- .kiro/specs/{feature}/spec.json
- .kiro/specs/{feature}/requirements.md ({X} requirements)
- .kiro/specs/{feature}/design.md ({Y} components, {Z} endpoints)
- .kiro/specs/{feature}/tasks.md ({N} tasks)
Quick generation skipped:
- `/kiro-validate-gap` - Gap analysis (integration check)
- `/kiro-validate-design` - Design review (architecture validation)
Sanity review: PASSED | FOLLOW-UP REQUIRED
## Next Steps:
1. Review generated specs (especially design.md)
2. Optional validation:
- `/kiro-validate-gap {feature}` - Check integration with existing codebase
- `/kiro-validate-design {feature}` - Verify architecture quality
3. Start implementation: `/kiro-impl {feature}`
```
## Safety & Fallback
### Error Scenarios
**Template Missing**:
- Check `.kiro/settings/templates/specs/` exists
- Report specific missing file
- Exit with error
**Directory Creation Failed**:
- Check permissions
- Report error with path
- Exit with error
**Phase Execution Failed** (Phase 2-4):
- Stop workflow
- Show current state and completed phases
- Suggest: "Continue manually from `/kiro-spec-{next-phase} {feature}`"
**Sanity Review Failed**:
- Stop workflow
- Report the exact contradiction, missing prerequisite, or task-plan issue
- Suggest targeted follow-up with `/kiro-spec-design {feature}`, `/kiro-spec-tasks {feature}`, or manual edits depending on the finding
**User Cancellation** (Interactive Mode):
- Stop gracefully
- Show completed phases
- Suggest manual continuation
@@ -1,145 +0,0 @@
---
name: kiro-steering-custom
description: Create custom steering documents for specialized project contexts
metadata:
shared-rules: "steering-principles.md"
---
# Kiro Custom Steering Creation
<background_information>
**Role**: Create specialized steering documents beyond core files (product, tech, structure).
**Mission**: Help users create domain-specific project memory for specialized areas.
**Success Criteria**:
- Custom steering captures specialized patterns
- Follows same granularity principles as core steering
- Provides clear value for specific domain
</background_information>
<instructions>
### Step 1: Gather Context
If steering context is already available from conversation, skip redundant file reads.
Otherwise:
- Check `.kiro/settings/templates/steering-custom/` for available templates
- Read `rules/steering-principles.md` from this skill's directory for steering principles
## Workflow
1. **Ask user** for custom steering needs:
- Domain/topic (e.g., "API standards", "testing approach")
- Specific requirements or patterns to document
2. **Check if template exists**:
- Load from `.kiro/settings/templates/steering-custom/{name}.md` if available
- Use as starting point, customize based on project
3. **Analyze codebase** (JIT) for relevant patterns:
#### Parallel Research
The following research areas are independent and can be executed in parallel:
1. **Template & principles**: Load matching template and steering-principles.md
2. **Domain patterns**: Analyze codebase for domain-specific patterns using Glob/Grep/Read
If multi-agent is enabled, spawn sub-agents for each area above. Otherwise execute sequentially.
After all parallel research completes, synthesize findings for steering document.
4. **Generate custom steering**:
- Follow template structure if available
- Apply principles from `rules/steering-principles.md` from this skill's directory
- Focus on patterns, not exhaustive lists
- Keep to 100-200 lines (2-3 minute read)
5. **Create file** in `.kiro/steering/{name}.md`
## Available Templates
Templates available in `.kiro/settings/templates/steering-custom/`:
1. **api-standards.md** - REST/GraphQL conventions, error handling
2. **testing.md** - Test organization, mocking, coverage
3. **security.md** - Auth patterns, input validation, secrets
4. **database.md** - Schema design, migrations, query patterns
5. **error-handling.md** - Error types, logging, retry strategies
6. **authentication.md** - Auth flows, permissions, session management
7. **deployment.md** - CI/CD, environments, rollback procedures
Load template when needed, customize for project.
## Steering Principles
From `rules/steering-principles.md` (in this skill's directory):
- **Patterns over lists**: Document patterns, not every file/component
- **Single domain**: One topic per file
- **Concrete examples**: Show patterns with code
- **Maintainable size**: 100-200 lines typical
- **Security first**: Never include secrets or sensitive data
</instructions>
## Tool guidance
- **Read**: Load template, analyze existing code
- **Glob**: Find related files for pattern analysis
- **Grep**: Search for specific patterns
- **Bash** with `ls`: Understand relevant structure
**JIT Strategy**: Load template only when creating that type of steering.
## Output description
Chat summary with file location (file created directly).
```
✅ Custom Steering Created
## Created:
- .kiro/steering/api-standards.md
## Based On:
- Template: api-standards.md
- Analyzed: src/api/ directory patterns
- Extracted: REST conventions, error format
## Content:
- Endpoint naming patterns
- Request/response format
- Error handling conventions
- Authentication approach
Review and customize as needed.
```
## Examples
### Success: API Standards
**Input**: "Create API standards steering"
**Action**: Load template, analyze src/api/, extract patterns
**Output**: api-standards.md with project-specific REST conventions
### Success: Testing Strategy
**Input**: "Document our testing approach"
**Action**: Load template, analyze test files, extract patterns
**Output**: testing.md with test organization and mocking strategies
## Safety & Fallback
- **No template**: Generate from scratch based on domain knowledge
- **Security**: Never include secrets (load principles)
- **Validation**: Ensure doesn't duplicate core steering content
## Notes
- Templates are starting points, customize for project
- Follow same granularity principles as core steering
- All steering files loaded as project memory
- Custom files equally important as core files
- Avoid documenting agent-specific tooling directories (e.g. `.cursor/`, `.gemini/`, `.claude/`)
- Light references to `.kiro/specs/` and `.kiro/steering/` are acceptable; avoid other `.kiro/` directories
@@ -1,90 +0,0 @@
# Steering Principles
Steering files are **project memory**, not exhaustive specifications.
---
## Content Granularity
### Golden Rule
> "If new code follows existing patterns, steering shouldn't need updating."
### ✅ Document
- Organizational patterns (feature-first, layered)
- Naming conventions (PascalCase rules)
- Import strategies (absolute vs relative)
- Architectural decisions (state management)
- Technology standards (key frameworks)
### ❌ Avoid
- Complete file listings
- Every component description
- All dependencies
- Implementation details
- Agent-specific tooling directories (e.g. `.cursor/`, `.gemini/`, `.claude/`)
- Detailed documentation of `.kiro/` metadata directories (settings, automation)
### Example Comparison
**Bad** (Specification-like):
```markdown
- /components/Button.tsx - Primary button with variants
- /components/Input.tsx - Text input with validation
- /components/Modal.tsx - Modal dialog
... (50+ files)
```
**Good** (Project Memory):
```markdown
## UI Components (`/components/ui/`)
Reusable, design-system aligned primitives
- Named by function (Button, Input, Modal)
- Export component + TypeScript interface
- No business logic
```
---
## Security
Never include:
- API keys, passwords, credentials
- Database URLs, internal IPs
- Secrets or sensitive data
---
## Quality Standards
- **Single domain**: One topic per file
- **Concrete examples**: Show patterns with code
- **Explain rationale**: Why decisions were made
- **Maintainable size**: 100-200 lines typical
---
## Preservation (when updating)
- Preserve user sections and custom examples
- Additive by default (add, don't replace)
- Add `updated_at` timestamp
- Note why changes were made
---
## Notes
- Templates are starting points, customize as needed
- Follow same granularity principles as core steering
- All steering files loaded as project memory
- Light references to `.kiro/specs/` and `.kiro/steering/` are acceptable; avoid other `.kiro/` directories
- Custom files equally important as core files
---
## File-Specific Focus
- **product.md**: Purpose, value, business context (not exhaustive features)
- **tech.md**: Key frameworks, standards, conventions (not all dependencies)
- **structure.md**: Organization patterns, naming rules (not directory trees)
- **Custom files**: Specialized patterns (API, testing, security, etc.)
@@ -1,103 +0,0 @@
---
name: kiro-validate-design
description: Interactive technical design quality review and validation
metadata:
shared-rules: "design-review.md"
---
# Technical Design Validation
<background_information>
- **Mission**: Conduct interactive quality review of technical design to ensure readiness for implementation
- **Success Criteria**:
- Critical issues identified (maximum 3 most important concerns)
- Balanced assessment with strengths recognized
- Clear GO/NO-GO decision with rationale
- Actionable feedback for improvements if needed
</background_information>
<instructions>
## Core Task
Interactive design quality review for feature **$1** based on approved requirements and design document.
## Execution Steps
1. **Gather Context**:
- Read `.kiro/specs/$1/spec.json` for language and metadata
- Read `.kiro/specs/$1/requirements.md` for requirements
- Read `.kiro/specs/$1/design.md` for design document
- Core steering context: `product.md`, `tech.md`, `structure.md`
- Additional steering files only when directly relevant to architecture boundaries, integrations, runtime prerequisites, domain rules, security/performance constraints, or team conventions that affect implementation readiness
- Relevant local agent skills or playbooks only when they clearly match the feature's host environment or use case and provide review-relevant context
#### Parallel Research
The following research areas are independent and can be executed in parallel:
1. **Context & rules loading**: Spec documents, core steering, task-relevant extra steering, relevant local agent skills/playbooks, and `rules/design-review.md` from this skill's directory for review criteria
2. **Codebase pattern survey**: Gather existing architecture patterns, naming conventions, and component structure from the codebase to use as reference during review
If multi-agent is enabled, spawn sub-agents for each area above. Otherwise execute sequentially.
After all parallel research completes, synthesize findings for review.
2. **Execute Design Review**:
- Reference conversation history when available: leverage prior requirements discussion and user's stated design intent
- Follow design-review.md process: Analysis → Critical Issues → Strengths → GO/NO-GO
- Limit to 3 most important concerns
- Engage interactively with user — ask clarifying questions, propose alternatives
- Use language specified in spec.json for output
3. **Decision and Next Steps**:
- Clear GO/NO-GO decision with rationale
- Provide specific actionable next steps (see Next Phase below)
## Important Constraints
- **Quality assurance, not perfection seeking**: Accept acceptable risk
- **Critical focus only**: Maximum 3 issues, only those significantly impacting success
- **Conversation-aware**: Leverage discussion history for requirements context and user intent when available
- **Interactive approach**: Engage in dialogue, ask clarifying questions, propose alternatives
- **Balanced assessment**: Recognize both strengths and weaknesses
- **Actionable feedback**: All suggestions must be implementable
- **Context Discipline**: Start with core steering and expand only with review-relevant steering or use-case-aligned local agent skills/playbooks
</instructions>
## Tool Guidance
- **Read first**: Load spec, core steering, relevant local playbooks/agent skills, and rules before review
- **Grep if needed**: Search codebase for pattern validation or integration checks
- **Interactive**: Engage with user throughout the review process
## Output Description
Provide output in the language specified in spec.json with:
1. **Review Summary**: Brief overview (2-3 sentences) of design quality and readiness
2. **Critical Issues**: Maximum 3, following design-review.md format
3. **Design Strengths**: 1-2 positive aspects
4. **Final Assessment**: GO/NO-GO decision with rationale and next steps
**Format Requirements**:
- Use Markdown headings for clarity
- Follow design-review.md output format
- Keep summary concise
## Safety & Fallback
### Error Scenarios
- **Missing Design**: If design.md doesn't exist, stop with message: "Run `/kiro-spec-design $1` first to generate design document"
- **Design Not Generated**: If design phase not marked as generated in spec.json, warn but proceed with review
- **Empty Steering Directory**: Warn user that project context is missing and may affect review quality
- **Language Undefined**: Default to English (`en`) if spec.json doesn't specify language
### Next Phase: Task Generation
**If Design Passes Validation (GO Decision)**:
- Review feedback and apply changes if needed
- Run `/kiro-spec-tasks $1` to generate implementation tasks
- Or `/kiro-spec-tasks $1 -y` to auto-approve and proceed directly
**If Design Needs Revision (NO-GO Decision)**:
- Address critical issues identified
- Re-run `/kiro-spec-design $1` with improvements
- Re-validate with `/kiro-validate-design $1`
**Note**: Design validation is recommended but optional. Quality review helps catch issues early.
-109
View File
@@ -1,109 +0,0 @@
---
name: kiro-validate-gap
description: Analyze implementation gap between requirements and existing codebase
metadata:
shared-rules: "gap-analysis.md"
---
# Implementation Gap Validation
<background_information>
- **Mission**: Analyze the gap between requirements and existing codebase to inform implementation strategy
- **Success Criteria**:
- Comprehensive understanding of existing codebase patterns and components
- Clear identification of missing capabilities and integration challenges
- Multiple viable implementation approaches evaluated
- Technical research needs identified for design phase
</background_information>
<instructions>
## Core Task
Analyze implementation gap for feature **$1** based on approved requirements and existing codebase.
## Execution Steps
1. **Load Context**:
- Read `.kiro/specs/$1/spec.json` for language and metadata
- Read `.kiro/specs/$1/requirements.md` for requirements
- Core steering context: `product.md`, `tech.md`, `structure.md`
- Additional steering files only when directly relevant to the feature's domain rules, integrations, runtime prerequisites, compliance/security constraints, or existing product boundaries
- Relevant local agent skills or playbooks only when they clearly match the feature's host environment or use case and provide analysis-relevant context
2. **Read Analysis Guidelines**:
- Read `rules/gap-analysis.md` from this skill's directory for comprehensive analysis framework
#### Parallel Research
The following research areas are independent and can be executed in parallel:
1. **Codebase analysis**: Existing implementations, architecture patterns, integration points, extension possibilities
2. **External dependency research**: Dependency compatibility, version constraints, known integration challenges (when needed)
3. **Context loading**: Requirements, core steering, task-relevant extra steering, relevant local agent skills/playbooks, and gap-analysis rules
If multi-agent is enabled, spawn sub-agents for each area above. Otherwise execute sequentially.
After all parallel research completes, synthesize findings for gap analysis.
3. **Execute Gap Analysis**:
- Follow gap-analysis.md framework for thorough investigation
- Analyze existing codebase using Grep and Read tools
- Use WebSearch/WebFetch for external dependency research if needed
- Evaluate multiple implementation approaches (extend/new/hybrid)
- Use language specified in spec.json for output
4. **Generate Analysis Document**:
- Create comprehensive gap analysis following the output guidelines in gap-analysis.md
- Present multiple viable options with trade-offs
- Flag areas requiring further research
5. **Write Gap Analysis to Disk**:
**Write the gap analysis to disk so it survives session boundaries and can be referenced during design phase.**
- Save the gap analysis to `.kiro/specs/$1/research.md`
- If the file already exists, append the new analysis (separated by a horizontal rule `---`) rather than overwriting previous research
- Verify the file was written by reading it back
## Important Constraints
- **Information over Decisions**: Provide analysis and options, not final implementation choices
- **Multiple Options**: Present viable alternatives when applicable
- **Thorough Investigation**: Use tools to deeply understand existing codebase
- **Explicit Gaps**: Clearly flag areas needing research or investigation
- **Context Discipline**: Start with core steering and expand only with analysis-relevant steering or use-case-aligned local agent skills/playbooks
</instructions>
## Tool Guidance
- **Read first**: Load spec, core steering, relevant local playbooks/agent skills, and rules before analysis
- **Grep extensively**: Search codebase for patterns, conventions, and integration points
- **WebSearch/WebFetch**: Research external dependencies and best practices when needed
- **Write last**: Generate analysis only after complete investigation
## Output Description
Provide output in the language specified in spec.json with:
1. **Analysis Summary**: Brief overview (3-5 bullets) of scope, challenges, and recommendations
2. **Document Status**: Confirm analysis approach used
3. **Next Steps**: Guide user on proceeding to design phase
**Format Requirements**:
- Use Markdown headings for clarity
- Keep summary concise (under 300 words)
- Detailed analysis follows gap-analysis.md output guidelines
## Safety & Fallback
### Error Scenarios
- **Missing Requirements**: If requirements.md doesn't exist, stop with message: "Run `/kiro-spec-requirements $1` first to generate requirements"
- **Requirements Not Approved**: If requirements not approved, warn user but proceed (gap analysis can inform requirement revisions)
- **Empty Steering Directory**: Warn user that project context is missing and may affect analysis quality
- **Complex Integration Unclear**: Flag for comprehensive research in design phase rather than blocking
- **Language Undefined**: Default to English (`en`) if spec.json doesn't specify language
### Next Phase: Design Generation
**If Gap Analysis Complete**:
- Review gap analysis insights
- Run `/kiro-spec-design $1` to create technical design document
- Or `/kiro-spec-design $1 -y` to auto-approve requirements and proceed directly
**Note**: Gap analysis is optional but recommended for brownfield projects to inform design decisions.
@@ -1,26 +0,0 @@
---
name: openultrasast-fix-audit
description: OpenUltraSAST patch, fix audit, differential review, adversarial review, and patch validation workflow. Use when proposing or auditing fixes for OpenUltraSAST findings from opencode.
---
# OpenUltraSAST Fix Audit
Use this skill to steer fixes for OpenUltraSAST findings.
## Fix Lifecycle
1. Intake: restate the finding ID, evidence level, affected path, and acceptance criteria.
2. Plan: choose the smallest defensive change that addresses the root cause.
3. Implement: modify only files required by the finding.
4. Adversarial review: look for bypasses, regressions, false assumptions, and unrelated edits.
5. Reconcile: either fix the review finding or document why it is not applicable.
6. Verify: rerun the reproducer or affected checks.
7. Ready: only then consider advancing to `patch_validated`.
## Guardrails
- Do not rewrite modules when a local fix is enough.
- Do not change public APIs unless the vulnerability requires it.
- Do not mark a patch validated from model reasoning alone.
- Use differential review for risky changes in auth, crypto, config, parsers, and sandbox behavior.
- Run Semgrep or CodeQL delta checks when those analyzers supplied the original evidence.
@@ -1,35 +0,0 @@
---
name: openultrasast-scan
description: OpenUltraSAST scan, index, quick, standard, deep, SARIF, and report workflows. Use when running or planning `ousast scan`, repository indexing, or evidence-gated SAST analysis from opencode.
---
# OpenUltraSAST Scan
Use this skill to steer OpenUltraSAST scan workflows from opencode.
## Workflow
1. Confirm the target is a local repository path.
2. Run commands through uv, never by activating a virtualenv manually.
3. Start with quick mode unless the task explicitly requires standard or deep analysis.
4. Treat quick findings as `static_corroboration`, not verified vulnerabilities.
5. Preserve run artifacts under `.openultrasast/runs/<scan-id>/`.
6. Report `scan_id`, `run_dir`, finding count, and the strongest available evidence level.
## Commands
```bash
uv run ousast scan <repo> --mode quick
uv run ousast index <repo>
uv run pytest
```
## Evidence Rules
- `suspicion` means a model or weak signal raised a hypothesis.
- `static_corroboration` means deterministic code evidence exists.
- `crash_reproduced` requires an artifact-backed reproducer.
- `root_cause_explained` requires independent verifier reasoning.
- `patch_validated` requires rerunning relevant checks after the patch.
Do not describe a finding as verified unless its evidence level meets the configured report threshold.
@@ -1,34 +0,0 @@
---
name: openultrasast-triage
description: OpenUltraSAST false-positive elimination, ranking calibration, evidence ladder, verifier triage, and finding adjudication. Use when reviewing findings, reducing false positives, or deciding whether evidence is enough.
---
# OpenUltraSAST Triage
Use this skill when triaging OpenUltraSAST findings from opencode.
## Triage Loop
1. Identify the finding, path, rule, severity, confidence, and evidence level.
2. Separate deterministic evidence from model assumptions.
3. Check attacker control, reachability, sanitizer behavior, duplicates, impact, and contradictory evidence.
4. If rejected or unresolved, record the false-positive reason and scope.
5. Prefer scoped demotion over deleting a vulnerability class globally.
6. Feed the outcome into ranking calibration and prompt constraints.
## False-Positive Reasons
- `unreachable_path`
- `missing_attacker_control`
- `sanitizer_disproved`
- `static_rule_mismatch`
- `incorrect_model_assumption`
- `duplicate`
- `insufficient_impact`
- `unsupported`
- `contradicted`
- `unverified`
## Decision Standard
If evidence is uncertain, mark the finding as requiring next evidence instead of escalating severity. Difficult or high-impact disagreements should trigger fusion/deeper reasoning when available.
+15 -15
View File
@@ -28,17 +28,14 @@ Project memory keeps persistent guidance (steering, specs notes, component docs)
- Think in English, generate responses in English. All Markdown content written to project files (e.g., requirements.md, design.md, tasks.md, research.md, validation reports) MUST be written in the target language configured for this specification (see spec.json.language).
## Minimal Workflow
- Phase 0 (optional): `/kiro-steering`, `/kiro-steering-custom`
- Phase 0 (optional): `/kiro-steering` (bootstrap, sync, or a custom steering document)
- Discovery: `/kiro-discovery "idea"` — determines action path, writes brief.md + roadmap.md for multi-spec projects
- Phase 1 (Specification):
- Single spec: `/kiro-spec-quick {feature} [--auto]` or step by step:
- `/kiro-spec-init "description"`
- `/kiro-spec-requirements {feature}`
- `/kiro-validate-gap {feature}` (optional: for existing codebase)
- `/kiro-spec-design {feature} [-y]`
- `/kiro-validate-design {feature}` (optional: design review)
- `/kiro-spec-tasks {feature} [-y]`
- Multi-spec: `/kiro-spec-batch` — creates all specs from roadmap.md in parallel by dependency wave
- Phase 1 (Specification), one spec at a time, each phase human-reviewed:
- `/kiro-spec-init "description"`
- `/kiro-spec-requirements {feature}`
- `/kiro-spec-design {feature} [-y | --gap | --review]` (`--gap`: optional gap analysis against the existing code first; `--review`: optional interactive review of an existing design; the design-review gate always runs)
- `/kiro-spec-tasks {feature} [-y]`
- Multi-spec projects: the specs of roadmap.md in dependency order, with the same steps each
- Phase 2 (Implementation): `/kiro-impl {feature} [tasks]`
- Without task numbers: autonomous mode (subagent per task + independent review + final validation)
- With task numbers: manual mode (selected tasks in main context, still reviewer-gated before completion)
@@ -46,14 +43,17 @@ Project memory keeps persistent guidance (steering, specs notes, component docs)
- Progress check: `/kiro-spec-status {feature}` (use anytime)
## Skills Structure
Skills are located in `.opencode/skills/kiro-*/SKILL.md`
Skills are located in `.agents/skills/*/SKILL.md` (the plural directory OpenCode searches by default; moved from `.opencode/skills` on 2026-10-02)
- Each skill is a directory with a `SKILL.md` file
- Use `/skills` to inspect currently available skills
- Invoke a skill directly with `/kiro-<skill-name>`
- **If there is even a 1% chance a skill applies to the current task, invoke it.** Do not skip skills because the task seems simple.
- Workflow skills: `kiro-steering`, `kiro-discovery`, `kiro-spec-init`, `kiro-spec-requirements`, `kiro-spec-design`, `kiro-spec-tasks`, `kiro-impl`, `kiro-validate-impl`, `kiro-spec-status`
- `kiro-review` — task-local adversarial review protocol used by reviewer subagents
- `kiro-debug` — root-cause-first debug protocol used by debugger subagents
- `kiro-verify-completion` — fresh-evidence gate before success or completion claims
- `kiro-debug` — root-cause-first debug protocol used by debugger subagents; carries the instrument rule below
- `kiro-verify-completion` — fresh-evidence gate before success or completion claims; carries the instrument rule below
- Product skills for users of the tool (the only ones user documentation names): `openultrasast-scan`, `openultrasast-triage`
- Reviewed 2026-10-02: `kiro-spec-quick`, `kiro-spec-batch` removed (no fast-track or parallel spec creation); `kiro-validate-gap` and `kiro-validate-design` merged into `kiro-spec-design` (`--gap`, `--review`); `kiro-steering-custom` merged into `kiro-steering`; `openultrasast-fix-audit` merged into `openultrasast-triage`
## Development Rules
- 3-phase approval workflow: Requirements → Design → Tasks → Implementation
@@ -81,5 +81,5 @@ So, before reporting any measurement:
## Steering Configuration
- Load entire `.kiro/steering/` as project memory
- Default files: `product.md`, `tech.md`, `structure.md`
- Custom files are supported (managed via `/kiro-steering-custom`)
- Core files in this repository: `overview.md` (current state wins over dated history), `roadmap.md`, `safety-net.md`
- Custom files may exist beside them and carry equal weight (created via `/kiro-steering`, templates under `.kiro/settings/templates/steering-custom/`)
+19 -40
View File
@@ -1,24 +1,14 @@
# OpenUltraSAST with its CPG engine, in one image.
#
# The model layer's arbiter is Joern, and `openultrasast.cpg.backend` reaches it by subprocess. Shipping both
# in one image is what makes that simple: Joern is on PATH inside, so there is no container-to-container call,
# no docker socket, no path translation between host and container, and no second code path in the backend to
# get wrong. The alternative -- tool and engine as sibling services -- needs the tool to exec into the other
# container, which means mounting the docker socket, and that is a large amount of privilege for a linter.
#
# The image is big (a JRE plus ~2 GB of Joern). That is the honest cost of bundling an engine, and it buys a
# contributor a working tool from `docker compose run` with nothing installed on their machine.
FROM eclipse-temurin:21-jre-noble
# syntax=docker/dockerfile:1
# Supply digests from plane/task-base.digest; deliberately no floating fallback.
ARG TASK_BASE_TAG=base-unpublished
ARG TASK_BASE_DIGEST
FROM ghcr.io/norandom/ousast-task-base:${TASK_BASE_TAG}@${TASK_BASE_DIGEST} AS joern-build
ENV PATH="/opt/joern-cli:${PATH}"
ARG JOERN_VERSION=v4.0.625
ARG ARCHIVE=joern-cli-linux-x86_64.zip
# php-cli is required by php2cpg, which drives PHP-Parser and shells out to a real interpreter. Without it a
# PHP tree fails with an opaque "Process exited with code 1" -- measured on this project, not assumed.
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
python3 python3-venv python3-pip unzip curl ca-certificates git php-cli \
&& apt-get install -y --no-install-recommends curl unzip \
&& rm -rf /var/lib/apt/lists/*
RUN set -eux; \
@@ -30,39 +20,28 @@ RUN set -eux; \
unzip -q "${ARCHIVE}" -d /opt; \
rm -f "${ARCHIVE}" "${ARCHIVE}.sha512" checksum.sha512; \
/opt/joern-cli/joern-parse --help > /dev/null
ENV PATH="/opt/joern-cli:${PATH}"
# This release ships the JS launcher with a .sh suffix; the backend's direct fallback uses
# the same extensionless command convention as the other frontends.
RUN if [ ! -e /opt/joern-cli/jssrc2cpg ] && [ -x /opt/joern-cli/jssrc2cpg.sh ]; then \
ln -s jssrc2cpg.sh /opt/joern-cli/jssrc2cpg; \
fi
COPY ops/frontend-retention/install.py /tmp/frontend-retention-install.py
RUN python3 /tmp/frontend-retention-install.py /opt/joern-cli \
&& rm /tmp/frontend-retention-install.py
WORKDIR /app
COPY pyproject.toml README.md ./
COPY src ./src
COPY benchmarks ./benchmarks
# The core install is zero-dependency by design; `semantic` adds the tree-sitter grammars the candidate
# enumerator uses. The LLM endpoint stays optional and is configured by environment, never baked in.
RUN python3 -m venv /venv \
&& /venv/bin/pip install --no-cache-dir -e ".[semantic]"
ENV PATH="/venv/bin:${PATH}"
# Analysis never needs root.
# Noble may already provide the ubuntu account at UID 1000. Reuse that identity rather than
# failing installation or creating a second account with the same UID.
FROM ghcr.io/norandom/ousast-task-base:${TASK_BASE_TAG}@${TASK_BASE_DIGEST} AS engine
ENV PATH="/opt/joern-cli:${PATH}"
COPY --from=joern-build /opt/joern-cli /opt/joern-cli
# Create runtime identity before the application layers.
RUN if getent passwd 1000 > /dev/null; then \
usermod --login ousast --home /home/ousast --move-home "$(getent passwd 1000 | cut -d: -f1)"; \
else \
useradd --create-home --uid 1000 ousast; \
fi \
&& chown -R ousast /app
else useradd --create-home --uid 1000 ousast; fi
WORKDIR /app
COPY pyproject.toml README.md LICENSE ./
COPY src ./src
RUN --mount=type=cache,target=/root/.cache/pip \
/venv/bin/pip install --no-deps --no-cache-dir .
COPY --chown=1000 benchmarks ./benchmarks
RUN chown -R ousast /app
USER ousast
ENTRYPOINT ["python", "-m", "openultrasast.cli"]
CMD ["--help"]
+202
View File
@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
+394 -183
View File
@@ -1,239 +1,344 @@
# OpenUltraSAST
**v2.0.0 (Python package `2.0.0`), state as of 2026-10-02.** An experimental static
security analyser. The pre-push safety net is **NO-GO** for rollout: no hook capability is
enabled and no independent population has passed the qualification gates (see
[docs/evaluation.md](docs/evaluation.md)). See the [release notes](RELEASE_NOTES.md).
OpenUltraSAST is an experimental static security analyser. It is meant to be a safety net on
repositories it has never seen. It scans a checkout, or the commits a `git push` would publish. It
keeps what it can prove apart from what it only suspects.
OpenUltraSAST separates what it can prove from what it merely suspects, and says which is which.
A scan combines language-scoped pattern rules, a semantic overlay, a Joern-based taint engine
and optional model calls; every finding carries the evidence that earned its place, and a scan
that could not look at something records that as a degradation instead of reporting a clean
result.
A scan combines up to four parts:
`ousast` is the interface. Model-driven work over whole repositories runs on a separate
agentic plane (google/ax), and a learned decision engine that weighs all the signals is in
development; neither is needed for a scan.
- language-scoped pattern rules;
- a tree-sitter overlay (a syntax tree of each file, built by the tree-sitter parser);
- a Joern code property graph (a graph of the program's syntax, control flow and data flow);
- optionally, a model. It is asked only the question the graph cannot settle.
## Install and quickstart
Every finding carries the evidence that earned its place. When a scan could not look at
something, it records a degradation. It does not report a clean result.
Where a repository's application can be built and run, an experimental proof arm goes further. It
demonstrates a finding by executing it: the issue triggers on the vulnerable revision and not on
the fixed one, so a demonstrated finding is a reproducible fact, not a score. This arm is in pilot
and being qualified on repositories the tool has never seen; see [Status](#status).
`ousast` is the interface. Model work over whole repositories runs on a separate agentic plane
(google/ax). A learned decision engine is in development. A scan needs neither of them.
This file is the front door. Each section links to its page of the documentation site.
The site is published at <https://norandom.github.io/OpenUltraSAST/>. Its sources are in `docs/` and
`mkdocs.yml`; to preview it locally, run `uv sync --extra docs && uv run mkdocs serve`.
## Status
**v2.0.1, state as of 2026-10-08.** The pre-push safety net is **not yet released**. Blocking on
the static analyser by hand is **NO-GO** against every M4 gate, and the project no longer tries to
reach them that way.
- No blocking hook capability is enabled; the pre-push hook is advisory only.
- Two independent populations (case sets frozen before any scan) have been spent on the static
engine: recall was 1 of 11, then 0 of 17. The learned decision engine ranks but blocks for no
family.
- Why blocking is hard: static abstract interpretation could not prove the precision a block
needs. So the current work is a **dynamic proof arm**. It builds and runs a repository's app and
demonstrates a finding by a differential: the issue triggers on the vulnerable revision and not
on the fixed one. A demonstrated finding is ground truth by execution, so it can earn a block
without calibration. Pilot demonstrations exist, including one executed on the cluster; the arm
is being qualified, not shipped.
- The release gate is a **measured** catch-rate and false-alarm number on repositories the tool
has never seen (the unseen-repo evaluation), produced rather than asserted. Until that number
exists, treat all output as advisory.
Every detection change has to win a controlled measurement on held-out repositories. One page
states where things stand, with the record behind each figure:
[docs/where-we-stand.md](docs/where-we-stand.md). For developers, the pre-push check and its
honest limits: [docs/pre-push-for-developers.md](docs/pre-push-for-developers.md); the
catch–prove–fix–re-verify loop: [docs/use-case-fix-and-reverify.md](docs/use-case-fix-and-reverify.md).
The release history is in [RELEASE_NOTES.md](RELEASE_NOTES.md).
## Install and first scan
Pick the tier you need:
1. Tier 1: a deterministic scan of a checkout, or a CI gate.
2. Tier 2: the Joern engine (`standard`, `deep`).
3. Tier 3, for maintainers: model work over whole repositories on the plane.
4. Tier 4 does not exist yet.
### Tier 1: quick scan, no Docker
The core install depends only on PyYAML. From a checkout, `uv sync` (or `pip install .`) installs
the `ousast` entry point. `uv sync --group dev` adds the test and lint tools.
```bash
uv sync --group dev
# Deterministic scan, no model calls
uv sync
uv run ousast scan /path/to/repo --mode quick
# CI: exit 1 on any evidence-verified finding
uv run ousast scan . --mode quick --fail-on verified
uv run ousast scan . --mode quick --fail-on verified # CI: exit 1 on any evidence-verified finding
```
The core install depends only on PyYAML. Optional extras: `semantic` (tree-sitter grammars
for the overlay) and `s3` (the plane's memory store on an S3-compatible bucket, boto3).
This runs the language-scoped pattern rules, entry-point reachability hints, ranking,
verification and scoring. It is deterministic and reproducible. It needs no network and no keys.
For `standard` and `deep` scans, use the Docker image, which ships Joern (with `php-cli` for
the PHP frontend) next to the tool. Source the shell wrapper once and `ousast` runs in the
container with your directory mounted read-only:
The same install gives you the advisory pre-push hook for any repository. It runs the quick rules
on the lines a push changes, names every check it could not run, and never blocks a push:
```bash
cd /path/to/your/repository && ousast pre-push install # remove: ousast pre-push uninstall
```
What it cannot do:
- There is no engine. Nothing follows a value across statements or files.
- The PHP quick rules are a low-precision, in-sample development set. Per-rule precision lower
bounds run from 0.03 to 1.0 on the real pins
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`).
The optional `semantic` extra (`uv sync --extra semantic`) installs the tree-sitter grammars for
the overlay.
### Tier 2: standard and deep scans (Docker)
`standard` and `deep` need Joern. The Docker image ships three things in one image: the tool,
Joern v4.0.625 and `php-cli` (the PHP frontend needs it). Each release publishes it to GHCR. It is
big: a JRE plus about 2 GB of Joern. Pull it instead of building, and tag it with the name the
shell wrapper uses:
```bash
docker pull ghcr.io/norandom/openultrasast:2.0.1
docker tag ghcr.io/norandom/openultrasast:2.0.1 openultrasast:dev # or build: docker compose build
```
The shell wrapper runs `ousast` in the container. It builds the image only if `openultrasast:dev`
is missing. Your directory is mounted read-only and the container network is off. For the pre-push
hook, use `ousast pre-push install --docker`: Git cannot call a shell function, so it installs the
script `ousast-docker` beside the hook.
```bash
source /path/to/OpenUltraSAST/ops/shell/ousast.sh # PowerShell: ops/shell/ousast.ps1
ousast scan . --mode standard
ousast scan . --mode standard # the graph decides; no model is asked
ousast-with-judge scan . --mode standard # network on, the model is asked (DEEPSEEK_API_KEY)
```
Details, including a pinned host install of Joern, are in [ops/README.md](ops/README.md).
`docker compose build` and `TARGET=/path/to/repo docker compose run --rm ousast scan /target` do
the same without the wrapper.
Every run writes auditable artifacts under `<target>/.openultrasast/runs/<scan-id>/`:
`manifest.json`, `findings.json`, `verification.json`, `score.json`, `report.md`,
`report.sarif` and `trace/events.jsonl`, plus the stage artifacts of the mode.
`deep` adds REGRESS. REGRESS runs each reproduction snippet with `docker run` from wherever the
scan runs. The engine container has no Docker socket. So run `deep` in one of two ways:
- from a host install with Joern on the PATH (`pyinfra @local ops/joern.py`, pinned and checksum-
verified) and a working `docker`;
- or accept the recorded `sandbox_unavailable` degradation.
Details, including the host install, are in [ops/README.md](ops/README.md).
Keys and store settings live in `.env` in the working directory. The file is gitignored;
`.env.example` lists the names.
- `DEEPSEEK_API_KEY` serves every LLM call.
- `OPENROUTER_API_KEY` is used for embeddings only (`ousast learn memory embed`).
- `.env` never overrides a variable already exported in your shell. If a key change seems to have
no effect, `unset` the stale export.
- The plane's memory store defaults to a local file store under `~/ousast-results/plane/memory`.
- An S3-compatible bucket is optional: `uv sync --extra s3`, then `OUSAST_MEMORY=s3://<bucket>`
with `S3_ENDPOINT` and the AWS credentials in `.env`. Nothing in a scan needs it.
### Tier 3: the plane on ax (maintainers)
Tier 1 and 2 need none of this. Today's profile is a single-node kind cluster on the maintainer's
host.
- `ops/ax/up.sh` needs `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` on the PATH. It refuses to
start when one is missing.
- It builds Agent Substrate and ax from their checkouts under `~/.cache/ousast/ax-src/`.
- Measured idle footprint with two workers: the kind node takes 1.7 GiB of memory (27% of the
7.7 GiB host) and 4.9 GB of disk for images. The build uses several GB more for Go caches.
- The worker pool is sized for that 7 GB host.
- The verify and roles tasks need `DEEPSEEK_API_KEY`.
- Memory lives in the local file store or the S3 store. RustFS is the tested server.
Bring-up, doctor and the smoke Run: [ops/ax/README.md](ops/ax/README.md).
### Tier 4: Kubernetes (k3s), planned
Nothing has been deployed to any cluster other than that kind cluster. The target is two profiles
of one code base: `kind` locally and `k3s` in production. The same `ousast plane run` would
submit a Run in a remote execution mode. It needs:
- a kubeconfig context or the cluster's ingress (and an `imagePullSecret` on the worker
ServiceAccount for a private package);
- runner and engine images published to GHCR and pinned by digest;
- the S3 store as the artifact path, with no receiver on a laptop.
The plan, and the code changes it needs with file paths, are in
[docs/deployment.md](docs/deployment.md#production-topology-ax-on-a-kubernetes-cluster).
## Scan modes
| Mode | What runs | Needs |
| --- | --- | --- |
| `quick` | Language-scoped pattern rules, entry-point reachability hints, ranking, verification, scoring. Deterministic and reproducible. | nothing |
| `standard` | `quick` plus the MAP stage: complexity map, semantic overlay, authorization obligations, and the model layer, which builds one Joern code property graph per repository and decides taint, guard-dominance and configuration questions on it. An LLM is asked only the residual question the graph cannot settle (a `suspicion`); `[models] hunter` adds the tool hunter. | Joern for the model layer (else a `cpg_unavailable` degradation); a provider key for the LLM parts (else they are skipped and recorded) |
| `deep` | The MAP stage of `standard` plus REGRESS: promoted candidates are loaded by a small snippet inside a Docker sandbox (no network, read-only source, non-root, memory/pid/time limits) and get a `triggerable` / `not_triggerable` / ... verdict. | a working `docker` (else a `sandbox_unavailable` degradation) |
There are three modes. Each adds to the one before.
`--fail-on never|findings|verified|worth-fixing` sets the exit code: `findings` fails on any
reported finding, `verified` on any evidence-verified one, `worth-fixing` on a deep-mode
verdict that is worth fixing. `--config openultrasast.toml` loads settings.
- `quick` runs the pattern rules, hints, ranking, verification and score. It needs nothing.
- `standard` adds the MAP stage: complexity map, semantic overlay, authorization obligations and
the Joern model layer. The model layer builds one code property graph per repository. It decides
taint, guard-dominance and configuration questions on that graph. A model is asked only the
residual question. Without a key, that question is skipped and only the graph's entailed
findings are reported.
- `deep` adds REGRESS. Promoted candidates are loaded by a small snippet in a Docker sandbox. Each
gets a `triggerable` or `not_triggerable` verdict.
## Detection coverage (quick rules)
A missing engine is a recorded degradation (`cpg_unavailable`, `sandbox_unavailable`), never a
clean result.
Rules run only against files of their language. The bundled ruleset lives in
`src/openultrasast/ruleset/<language>/rules.toml`:
`--fail-on never|findings|verified|worth-fixing` sets the exit code. `--config openultrasast.toml`
loads settings. Every run writes these files under `<target>/.openultrasast/runs/<scan-id>/`:
`manifest.json`, `findings.json`, `verification.json`, `score.json`, `report.md`, `report.sarif`
and `trace/events.jsonl`.
| Language | Classes (CWE) |
| --- | --- |
| C / C++ | stack buffer overflow (121), format string (134), command injection (78) |
| Python | command injection (78), code injection (95), SSTI (94), deserialization (502), SQLi (89), path traversal (22), SSRF (918), weak hash (327), insecure default (489) |
| JavaScript / TS | command injection (78), code injection (95), XSS (79), SQLi (89), path traversal (22), SSRF (918), weak hash (327), deserialization (502) |
| Java + Groovy templates | command injection (78), SQLi (89), weak hash (327), deserialization (502), unescaped template XSS (79) |
| PHP | SQLi (89), command injection (78), file inclusion (98), path (22), open redirect (601), SSRF (918); code eval (95), unserialize (502) and echo XSS (79) ship as `shadow` rules |
`ousast pre-push` (experimental) analyses the commits a push would publish against their base.
First, the quick rules run on the changed lines. Then the engine compares head with base, and only
new or worsened defects become candidates. It is advisory by default, and it prints one line per
check it could not run. Its capability registry is empty, so it emits no normal alert today. An
engine finding the push introduced is shown as advisory. Install, settings, skip reasons and a CI
recipe: [docs/scanning.md](docs/scanning.md#first-run-install-the-hook-on-any-repository).
The detection gate (`uv run python -m openultrasast.gate`, also `tests/test_detection_benchmarks.py`)
enforces the project goal of **at least 90% recall and under 10% false positives** per language
on the bundled corpora in `benchmarks/manifests/`. Its output on this tree:
Flows and the per-language rule table: [docs/scanning.md](docs/scanning.md).
| Language | Recall | False-positive rate |
| --- | --- | --- |
| C / C++ | 93.3% (14/15) | 0.0% |
| Python | 93.3% (14/15) | 0.0% |
| JavaScript | 92.3% (12/13) | 0.0% |
| Java | 100% (4/4) | 0.0% |
| **Overall** | **93.6% (44/47)** | **0.0%** |
## How detection works
These corpora are cheat-sheet fixtures modelled on deliberately vulnerable applications. They
show that a rule change did not break a known detection; they are not a recall estimate on
real code. Real-code results are in [docs/evaluation.md](docs/evaluation.md).
The tool uses three representations of code, from cheap to expensive:
**PHP is not in the gate, and its numbers are in-sample only.** The PHP rules were written and
tightened while looking at the development corpus they were measured on
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`): 13 of 14 expected
findings on the `php-vulnerable` fixture with no false positive, 2 false positives on
`php-benign`, and per-rule precision lower bounds from 0.03 to 1.0 on the real pins. Treat
them as development numbers, not as a holdout result.
1. Text: regex quick rules, one statement at a time.
2. Tree: the tree-sitter CST (concrete syntax tree) and an IR (intermediate representation) per
function. The overlay and the obligations use it.
3. Graph: the Joern CPG. It is the only one that follows a value across statements, calls and
files.
**Framework knowledge is optional.** Rule and taint-fact entries that know a framework or
library (WordPress, Flask, Django, Express, Spring, ...) carry a `framework` or `library` tag
naming a row of `src/openultrasast/ruleset/frameworks.toml`. The loaders take `priors=`
(`"all"`, the default and today's behaviour; `"off"`, language-level knowledge only; or a set
of ids), so their contribution can be measured and switched off.
Taint runs as fact tables, queries and verdicts on the graph. Absence bugs are operations that
should have been guarded. They are found as obligations, dominance and configuration census. A
model answers only the residual question.
## Pre-push checks (experimental)
- [docs/detection-techniques.md](docs/detection-techniques.md) explains each technique, with the
code it comes from and what it misses.
- [docs/architecture.md](docs/architecture.md) gives the stage order and the evidence ladder (a
state machine, not a label a model may assign). It also covers the central CWE policy, the
project score and the self-improving loops (`ousast improve`, with `--memory` for proposals from
plane rows).
`ousast pre-push` analyses the commits a `git push` would publish, or an explicit
`--base`/`--head` pair, and compares the head against its base so that only new or worsened
defects are candidates. `--mode advisory` (the default) never blocks; `--mode blocking` and
`--incomplete-coverage block` opt into enforcement. The artifact must be written outside the
analysed repository.
Quick rules cover C/C++, Python, JavaScript/TypeScript, Java with Groovy templates, and PHP. Each
rule runs only against files of its language.
```bash
uv run ousast pre-push /path/to/repo --base BASE --head HEAD \
--artifact /outside/the/repo/result.json
```
The detection gate (`uv run python -m openultrasast.gate`) enforces at least 90% recall and under
10% false positives per language on the bundled cheat-sheet corpora. On this tree it reports
93.62% recall (44/47) and 0.0% false positives. Those corpora show that a rule change did not
break a known detection. They are not a recall estimate on real code, and PHP is not in the gate.
The default capability registry is empty, so no normal alert is emitted today; every
candidate stays diagnostic in the artifact. Installation into a repository's hooks
(`ops/install-pre-push`), caching and the optional witness model are documented in
[ops/README.md](ops/README.md).
Framework knowledge (WordPress, Flask, Django, Express, Spring) is tagged as an optional prior in
`src/openultrasast/ruleset/frameworks.toml`. So its contribution can be measured and switched off.
## Model providers and keys
## The plane on ax
Keys live in `.env` in the working directory (gitignored, never committed):
Model-driven work over whole repositories runs as a Run manifest. A Run is a DAG of ax Tasks. Each
Task runs in its own gVisor actor (an isolated sandbox) on Agent Substrate. ax is the only agentic
executor; there is no local subprocess path.
```bash
DEEPSEEK_API_KEY=... # every LLM call: model layer, tool hunter, plane tasks
OPENROUTER_API_KEY=... # only OpenAI embeddings (openai/text-embedding-3-small)
```
- Each Task binds at most one Model and its own `usd`/`calls` budget.
- Egress is deny-by-default per Task, by hostname only.
- The provider key travels only in the Task's start request, never in a manifest or an image.
The project uses DeepSeek (`deepseek-flash`) for LLM calls and OpenRouter only for embeddings.
Set `DEEPSEEK_API_KEY`: without it, chat calls fall back to OpenRouter when that key is set.
**`.env` never overrides a variable already exported in your shell.** A stale
`export DEEPSEEK_API_KEY=...` in your profile wins over the `.env` file silently; `unset` it
when a key change in `.env` seems to have no effect.
## Benchmarks and pair evaluation
```bash
# Score the scan against a ground-truth manifest
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
# Vulnerable-vs-fixed pairs: fire on the vulnerable side, stay silent on the fix
uv run ousast pairs
uv run ousast pairs --slice sast # OWASP Benchmark + Juliet
uv run ousast pairs --slice vibe-py --pointers # also fetch non-vendored pointer pairs (network)
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
```
Each benchmark run writes `.openultrasast/benchmarks/<id>/` (`benchmark_result.json`,
`calibration_records.json` with every miss, `external_baseline_deltas.json`). The pair
catalogs live in `benchmarks/pairs/`.
## Self-improvement
`ousast improve` runs a bounded loop over the governed ruleset: benchmark, propose edits to
rule status and score constants only, validate, re-benchmark, and accept a round only if
recall stays at least 90%, false positives under 10%, the project score does not regress and
no matched finding is lost. Accepted rounds land in
`<target>/.openultrasast/calibration/rule_policy.json`, which the next scan of that target loads.
```bash
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
```
With `--memory`, proposals also come from the rows plane runs stored (see below), through the
same validator and gate; rows from holdout pairs, from the gated manifest's own cases and from
any `--qualify-population` are dropped first. Details:
[docs/architecture.md](docs/architecture.md#self-improvement).
## The agentic plane (maintainers)
Model work over whole repositories runs as a Run manifest on [google/ax](ops/ax/README.md)
over Agent Substrate (a kind cluster on the maintainer's host), one isolated actor per task.
Each task binds its own Model and its own `usd`/`calls` budget, egress is deny-by-default per
task, and the provider key travels only in the task's start request.
Measured on the first increment: the plane agreed on 16 of 20 declared sites at $0.0205 per
candidate.
```bash
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
uv run ousast plane run plane/runs/validation-46.yaml # a rerun skips tasks already done
uv run ousast plane status validation-46 # per-task status and token attribution
uv run ousast plane run plane/runs/validation-46.yaml # a rerun skips tasks already done; --workers bounds tasks in flight
uv run ousast plane status validation-46 # per-task status and the token attribution table
uv run ousast plane remember validation-46 # ingest the run's rows into the memory store
```
`ousast plane workspaces`, `alerts-engine` and `harvest` generate Runs and inputs. Bring-up,
the task catalogue, the memory store (`OUSAST_MEMORY`) and the checklist for moving the plane
to another Kubernetes cluster are in [ops/ax/README.md](ops/ax/README.md).
`workspaces`, `harvest`, `alerts-engine` and `memory-normalise` generate Runs, their inputs and
the store's rewrites. The flow and the task catalogue: [docs/plane.md](docs/plane.md). What runs
where today, and what a separate cluster would take: [docs/deployment.md](docs/deployment.md).
> HarnessX, the earlier optional agentic extra, was retired 2026-09-30, and a leftover
> `[harnessx]` section (retired 2026-09-30) is ignored with one warning. `[models] verifier`,
> `[fusion] panel_model` and `[fusion] decider_model` fail with a message naming the plane
> replacement. The LLM judge and the LLM fusion panels were retired with it; fusion is
> deterministic. See [RELEASE_NOTES.md](RELEASE_NOTES.md).
HarnessX, the earlier agentic extra, was retired 2026-09-30. A `[harnessx]` section is ignored
with one warning. `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail and
name the plane replacement. Config keys that nothing ever read fail the same way. Fusion is
deterministic.
## Decision engine (in development, not adopted)
## Memory
A learned decision engine is being built to replace hand-tuned detection edits: each
instrument (quick rules, the Joern engine, model sink classification, verify passes, repository
facts) produces signals, never verdicts, and a compiled AI classifier with local memory turns
them into a calibrated probability per candidate with BLOCK and ADVISORY operating points.
No scan, pre-push or report uses it yet. Status and the first measured slice:
[docs/decision-engine.md](docs/decision-engine.md).
One store holds what plane runs and the decision engine learned. It is keyed by repository and pin.
It holds two kinds of data:
## Agent integrations
- rows: facts, verdicts, unit costs, alerts, coverage and feature records;
- content-addressed blobs: excerpts, embeddings, cached responses and compiled programs.
An agent that can run shell commands needs nothing but the CLI. For
[OpenCode](https://opencode.ai) there are optional project skills in `.opencode/skills/`:
`openultrasast-scan`, `openultrasast-triage` and `openultrasast-fix-audit`.
`OUSAST_MEMORY` selects the backend. It is either a local file store or `s3://<bucket>` on any
S3-compatible server. The S3 server must support versioning, lifecycle rules, object tags and S3
Select. RustFS is the tested server.
`ousast mcp` runs a narrow MCP server over stdio (newline-delimited JSON-RPC) with ten tools:
`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`,
`benchmark`, `explain`, `propose_patch` (degrades visibly) and `export_report`. No tool runs a
shell, Docker or a free-form command.
An admin sets the bucket up once. The store verifies it at every start. If anything is missing, it
refuses to run and names each missing piece. It does not fall back.
```jsonc
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
- Layout and the read and write paths: [docs/memory.md](docs/memory.md).
- The bucket policy and admin steps: [docs/rustfs.md](docs/rustfs.md).
- How stored rows are meant to turn into better decisions:
[docs/memory-and-detection.md](docs/memory-and-detection.md).
## Decision engine and evaluation
A learned decision engine is being built. It is meant to replace hand-tuned detection edits. It
follows three rules:
- Every instrument produces signals, never verdicts.
- Labels come from ground truth only.
- A compiled AI classifier with local memory turns retrieved similar cases and the candidate's
signals into a calibrated probability. It has BLOCK and ADVISORY operating points.
It is **in development, not adopted**. No scan, pre-push check or report uses it.
It was measured on six families, on repositories the compiled program was not shown:
- The pooled AUC (area under the ROC curve) of its score runs from 0.77 (injection) to 0.96
(access control).
- The stricter within-pair AUC is 0.72 to 0.93 across the same families.
- BLOCK is offered for no family
(`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`).
Maintainer surface: `ousast learn labels|memory|compile|evaluate|curve|experiment|audit-leaks`.
Design, slices and the A/B protocol: [docs/decision-engine.md](docs/decision-engine.md).
[docs/evaluation.md](docs/evaluation.md) explains how anything is measured on code the tool was not
tuned on. It covers the pair corpora, the independent populations and the M4 gates. It also covers
the fold discipline that keeps learned decisions out of their own training data. Population v3
(PHP, 15 cases) is frozen as the one-time final check. Nothing in the tree may read it.
```bash
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
uv run ousast pairs --slice sast # fire on the vulnerable side, stay silent on the fix
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
```
> The `kiro-*` skills, `AGENTS.md` and `.kiro/` are the maintainer's development tooling and
> can be ignored by users.
## Token economics
## Further reading
Model tokens are the one running cost. At runtime the plane routes them:
Documentation site with flow diagrams (`docs/`, `mkdocs.yml`): `uv sync --extra docs && uv run mkdocs serve`.
- one Model and one budget per Task;
- a metered client that refuses the next call at the ceiling;
- repository facts computed without a model and reused;
- blobs replayed at $0, with no new model call;
- triage before the hunt;
- a tie-break only on disputed candidates.
- [docs/architecture.md](docs/architecture.md): the scan pipeline, evidence ladder, CWE policy
and project score, and the self-improving loops.
- [docs/evaluation.md](docs/evaluation.md): independent populations, the qualification gates,
pair corpora and the plane increment results.
- [docs/decision-engine.md](docs/decision-engine.md): the learned decision engine.
- [docs/threat-model.md](docs/threat-model.md): trust boundaries, sandbox and hardening.
- [docs/token-ergonomics.md](docs/token-ergonomics.md): model spend at runtime (budgets, reuse,
measured costs) and in coding sessions.
- [docs/examples.md](docs/examples.md): end-to-end walkthroughs.
In development, 85.5% of the maintainers' coding-session tokens were tool input and output rather
than reasoning. That is down from 90%. The remedies in `benchmarks/dev/` target that share.
Mechanisms, the measured costs and the before-and-after table:
[docs/token-ergonomics.md](docs/token-ergonomics.md).
## Development
## Examples, threat model
- End-to-end walkthroughs, from a quick scan to an MCP client: [docs/examples.md](docs/examples.md).
- What the tool trusts and what it does not: [docs/threat-model.md](docs/threat-model.md). This
covers scanned code as untrusted input, secret redaction, per-task egress, budgets and the
`deep` sandbox.
## Contributing and maintainer tooling
```bash
uv run pytest # tests, incl. the 90/10 detection gate
@@ -245,6 +350,112 @@ uv run python -m openultrasast.pair_gate # local vuln-vs-fix pairs
uv run python dagger/ci.py # containerized CI pipeline
```
Maintainer commands not covered above: `ousast repos` (pinned known-vulnerable checkouts),
`ousast model candidates` (what the candidate enumerator can reach) and `ousast learn`
(the decision engine's data; see [docs/decision-engine.md](docs/decision-engine.md)).
An agent that can run shell commands needs nothing but the CLI. For [OpenCode](https://opencode.ai)
there are two project skills in `.agents/skills/`:
- `openultrasast-scan`: modes and their prerequisites, `--fail-on`, the evidence ladder, recorded
degradations and run artifacts.
- `openultrasast-triage`: false-positive reasons, what an adjudication may feed, and fixing and
re-scanning.
`ousast mcp` runs a narrow MCP (Model Context Protocol) server over stdio. It has ten tools:
`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`,
`explain`, `propose_patch` and `export_report`. No tool runs a shell, Docker or a free-form
command.
```jsonc
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
```
The `kiro-*` skills, `AGENTS.md` and the Kiro-style specifications under `.kiro/` are the
maintainer's development tooling. Users can ignore them. Other maintainer commands:
- `ousast repos`: pinned known-vulnerable checkouts.
- `ousast model candidates`: what the candidate enumerator can reach.
## Licence
OpenUltraSAST is licensed under the [Apache License 2.0](LICENSE). Third-party material keeps its
own terms:
- Joern (Apache-2.0) is downloaded at image build time.
- The tree-sitter wheels of the `semantic` extra are MIT.
- The published benchmark cases were licence-checked one by one. v1's OpenCVE case is BUSL-1.1, see [RELEASE_NOTES.md](RELEASE_NOTES.md).
## Engine traces on kube-ax
`benchmarks/learn/engine_trace.py --executor ax` submits AX v0.3.1 Tasks with
`ax apply -f -` and removes them with `ax delete task NAME`. The operator supplies
kubeconfig and AX access; the dispatcher creates no Kubernetes resources or secrets.
The `ousast-engine-task` GHCR package **must be made Public once** in its package
settings so the actors can pull it without registry credentials.
The task image contains the current engine and worker code. Rebuild it after code
changes and pass its digest with `--image`; no analyzer download is needed. The host
materializes each pin as usual and uploads source and prepared questions under
`engine-queue/<run>/<pin>/`. Only object-scoped presigned URLs and deadline parameters
enter Task env. Credentials stay on the coordinator. Configure `OUSAST_MEMORY=s3://…`
and the store credentials in the coordinator's `.env` or environment, with
`S3_ENDPOINT=https://files.because-security.com`. Presigned links expire after the
pin deadline plus ten minutes. IP and HTTP endpoints are rejected: the operator's
actor EgressPolicy must allow **files.because-security.com:443**.
Completion is the result object, even while AX reports Running. Failed phases and
host deadlines produce explicit failure/timeout rows. Task and staging-object cleanup
runs on success and failure; cleanup failure stops further dispatch. Per-pin files,
`--rerun-status`, resume and `progress.json` retain the existing trace-runner behavior.
AX results include `executor: "ax"`, `worker_ip`, and the image digest. Placement is
random; `--parallel` defaults to two AX tasks and cannot exceed two.
AX workers use `--heap-profile cluster` with
`-Xms1700m -Xmx1700m -XX:MaxMetaspaceSize=256m -XX:ReservedCodeCacheSize=128m -XX:MaxDirectMemorySize=256m -Xss512k -XX:ActiveProcessorCount=1 -XX:+UseG1GC`
on every frontend/query JVM; instruments record the flags. A `JVM out of memory`
result is automatically re-queued to one local Docker lane with `--heap-profile vm`
(2560m), recording `fallback: "vm-oom"` and the AX attempt. `mixed-ax` also uses this
one Docker lane for overflow, prioritizing pending OOM fallbacks. Thus a mixed run has
up to two AX tasks plus one Docker container. Use `--docker-image` for the locally
available VM image; Docker never pulls it implicitly. A STOP file prevents new
launches; interrupted fallback queues are recovered on resume.
Coordinator commands, from this worktree (the AX binary and kubeconfig remain
configurable). Build/push the actual uncommitted tree and read its digest:
```bash
cd /home/mc/Source/OpenUltraSAST/.claude/worktrees/engine-ax
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
export PYTHONPATH="$PWD/src"
TASK_TAG="ghcr.io/norandom/ousast-engine-task:sha-$(git rev-parse --short=7 HEAD)"
docker buildx build --platform linux/amd64 -f plane/Dockerfile.engine-task \
-t "$TASK_TAG" -t ghcr.io/norandom/ousast-engine-task:main \
--push --metadata-file /tmp/ousast-engine-task-build.json .
export TASK_IMAGE="ghcr.io/norandom/ousast-engine-task@$(python -c 'import json; print(json.load(open("/tmp/ousast-engine-task-build.json"))["containerimage.digest"])')"
printf '%s\n' "$TASK_IMAGE"
docker build -f Dockerfile -t openultrasast:engine-ax .
```
The Engine image workflow also builds/pushes both images and prints each digest;
CI needs the changes published before dispatch. The local build above includes
uncommitted changes without publishing a git commit.
With `KUBECONFIG` pointing to the operator's kube-ax config and store settings in
`.env`, run a one-pin smoke using the default experiment units and local example
memory (or add `--examples /path/to/frozen-examples.jsonl`):
```bash
: "${KUBECONFIG:?Set KUBECONFIG to the operator kube-ax configuration}"
python benchmarks/learn/engine_trace.py --executor ax \
--ax-bin "$HOME/.local/ax-v0.3.1/ax" --kubeconfig "$KUBECONFIG" \
--atespace default --image "$TASK_IMAGE" --docker-image openultrasast:engine-ax \
--parallel 1 --limit 1 --deadline 900 \
--out "$HOME/ousast-results/plane/engine-ax-smoke"
```
Then run the mixed lanes (repeat to resume; defaults re-run failed/timeout pins):
```bash
python benchmarks/learn/engine_trace.py --executor mixed-ax \
--ax-bin "$HOME/.local/ax-v0.3.1/ax" --kubeconfig "$KUBECONFIG" \
--atespace default --image "$TASK_IMAGE" --docker-image openultrasast:engine-ax \
--parallel 2 --deadline 900 --rerun-status failed,timeout \
--out "$HOME/ousast-results/plane/engine-ax-mixed"
```
+38
View File
@@ -1,3 +1,41 @@
# v2.0.1 (2026-10-02)
v2.0.0 was tagged but never published: its release job stopped at the format check (see below). v2.0.1 is
the first published 2.x release and carries everything listed under v2.0.0 plus the following.
- **Licence: Apache-2.0.** `LICENSE` added; `pyproject.toml` declares `license = "Apache-2.0"`; both images
carry the `org.opencontainers.image.licenses` label.
- **Container images on GHCR.** Each version tag pushes `ghcr.io/norandom/openultrasast` (the scan image:
Joern and php-cli) and `ghcr.io/norandom/openultrasast-runner` (the plane's task runner), tagged with the
version and, for full releases, `latest`. The digests are in the release job's summary; pin by digest
(Agent Substrate rejects tags).
- **CI green again.** CI and the release job check formatting and lint over the whole repository; eight
benchmark scripts had drifted. They were reformatted (two `noqa` comments, one renamed loop variable; no
script computes anything differently), and the local gate now runs the same whole-tree commands.
- **Front-door README.** Install in four tiers (quick scan without Docker; standard and deep with the scan
image; the plane on kind; Kubernetes on k3s, planned), every claim checked against the CLI, and links to
every documentation page.
## Also in this release (previously listed as Unreleased)
- **Documentation audited against v2.0.0** (no behaviour change). New `docs/deployment.md`: where the plane runs
today, this host's setup, and step-by-step guidance for a separate Kubernetes cluster with what is implemented
and what is planned; no cluster deployment has been made, and the kube context (`kind-$KIND_CLUSTER_NAME`) and
`doctor`'s registry check (`localhost:5001`) are still hard-coded. The decision-engine pages state slice 2
(six families, `benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`: pooled AUC 0.77-0.96,
within-pair 0.72-0.93, BLOCK offered nowhere, canary below 0.9 for three families) and that the engine is not
adopted. The model layer's residual question is documented at `model/pipeline.py` (`model/judge.py` was
deleted); a missing provider key is described as it behaves (no question asked, entailed findings reported),
not as a recorded degradation. The `.env` reference lists the memory-store variables; `ousast plane memory-normalise` is listed.
- **Removed: the unused vector index, `ousast index` and `[embeddings]`** (legacy cleanup, 2026-10-02). `index.py`
keeps only the text chunker the skill router uses; the `VectorIndex` half had no caller, and nothing read the
`chunks.json` the subcommand wrote. `[embeddings] model`/`store` fail to load with the reason, like the other
retired keys; `OPENROUTER_EMBEDDING_MODEL` is no longer read (`ousast learn memory embed --model` names the model).
- **Retired three config keys nothing read** (2026-10-02): `[sandbox] network`, `[sandbox] workspace_readonly` and
`[complexity] top_k` now fail to load naming why. The sandbox always ran with `--network none` and a read-only
workspace regardless of the first two; no stage read `top_k`. `memory_mb`, `timeout_seconds`, `pids_limit` and
`max_hunter_hotspots` are read and stay.
# v2.0.0 (2026-10-02)
A new major release: the agentic plane runs on google/ax, HarnessX is gone, detection decisions move
+1 -1
View File
@@ -16,7 +16,7 @@ maps each old path to the new one. The records themselves are not rewritten.
| `model_sinks.py` | Stage-1 source-only model classification of sinks. Moved into the package as `plane/tasks/roles.py`. | `independent/selection-v2-model-sinks.json`, `measurements/2026-10-01-decision-engine-harvest/record.json`, both `2026-10-01-decision-engine-leak-audit` records | `05d5699` (named in `selection-v2-model-sinks.json`) |
| `sink_candidates.py` | Diagnostic: vocabulary-declared sink candidates per case. Its declaration patterns live on in `plane/tasks/repo_facts.py`. | `independent/results-v2-model-pipeline.json` ("vocabulary sink-first: 2/17") | no commit recorded |
| `hunt.py` | The v2 tool-hunter run over the population (protocol `hunter-protocol-v2.md`, including its `FAMILY_WORDS` table). Replaced by the plane's `verify` / `agree` tasks. | `independent/hunter-protocol-v2.md`, `independent/results-v2-hunter.json` (a label source in `learn/sources.toml`) | `6bfb05f`, the commit that introduced the protocol, which pins the runner |
| `harnessx_removal_equality.py` | One-off equality proof for `harnessx-removal` (record and compare deterministic outputs). The spec is closed. `tests/test_harnessx_removal_equality.py` still checks that the comparison detects differences. | `measurements/2026-09-30-harnessx-removal-baseline/`, `measurements/2026-09-30-harnessx-removal-equality/` | baseline `ca5caf7`, candidate `b9afb66` |
| `harnessx_removal_equality.py` | One-off equality proof for `harnessx-removal` (record and compare deterministic outputs). The spec is closed; its unit test was dropped on 2026-10-02 so the archived copy is reference only like the rest. | `measurements/2026-09-30-harnessx-removal-baseline/`, `measurements/2026-09-30-harnessx-removal-equality/` | baseline `ca5caf7`, candidate `b9afb66` |
The six `independent` scripts still read the population, `evaluate.py` and the results from
`benchmarks/independent/`. Their `HERE` points there, so outputs land where they always did.
+38 -3
View File
@@ -40,8 +40,34 @@ from openultrasast.preprocess import build_file_target, enumerate_source_files
# The model names a family in its own words; the protocol scores three. Fixed before any v2 pin is hunted.
FAMILY_WORDS = {
"untrusted_destination": ("ssrf", "server-side request", "open redirect", "redirect", "url fetch", "request forgery"),
"config_secrets": ("cors", "debug", "secret", "hardcoded", "hard-coded", "credential", "misconfig", "configuration", "tls", "certificate"),
"injection": ("inject", "sql", "sqli", "nosql", "command", "rce", "code execution", "eval", "template", "ssti", "xss", "deserializ", "ldap", "xpath"),
"config_secrets": (
"cors",
"debug",
"secret",
"hardcoded",
"hard-coded",
"credential",
"misconfig",
"configuration",
"tls",
"certificate",
),
"injection": (
"inject",
"sql",
"sqli",
"nosql",
"command",
"rce",
"code execution",
"eval",
"template",
"ssti",
"xss",
"deserializ",
"ldap",
"xpath",
),
}
@@ -182,7 +208,16 @@ def main() -> int:
result["root"] = "/case"
out.write_text(json.dumps(result, indent=2) + "\n")
print(
json.dumps({"case": case["id"], "pin": label, "findings": len(result["findings"]), "hunts": f"{result['completed']}/{result['questions']}", "usd": result["cost_usd"], "total_usd": round(sum(spent), 4)}),
json.dumps(
{
"case": case["id"],
"pin": label,
"findings": len(result["findings"]),
"hunts": f"{result['completed']}/{result['questions']}",
"usd": result["cost_usd"],
"total_usd": round(sum(spent), 4),
}
),
flush=True,
)
return 0
+13 -5
View File
@@ -37,7 +37,9 @@ from openultrasast.config import load_config # noqa: E402
from openultrasast.model.endpoint import DEFAULT_DETECTOR_MODEL # noqa: E402
from openultrasast.preprocess import detect_language, enumerate_source_files # noqa: E402
NOT_PRODUCT = re.compile(r"(^|/)(tests?|__tests__|spec|specs|docs?|examples?|fixtures?|e2e|cypress|storybook)/|\.(test|spec|stories)\.[jt]sx?$|_test\.py$|(^|/)test_[^/]*\.py$|\.min\.js$|(^|/)dist/|(^|/)build/")
NOT_PRODUCT = re.compile(
r"(^|/)(tests?|__tests__|spec|specs|docs?|examples?|fixtures?|e2e|cypress|storybook)/|\.(test|spec|stories)\.[jt]sx?$|_test\.py$|(^|/)test_[^/]*\.py$|\.min\.js$|(^|/)dist/|(^|/)build/"
)
OPERATIONS = {
"injection": (
"executes a database query (SQL, NoSQL, JCR/graph query), an operating-system command or shell, code or "
@@ -69,7 +71,9 @@ def product_files(root: Path, languages: set[str]) -> list[Path]:
def chunks(text: str, size: int) -> list[str]:
lines = text.splitlines()
return ["\n".join(f"{n + 1:5d}| {line}" for n, line in enumerate(lines[i : i + size], start=i)) for i in range(0, max(len(lines), 1), size)]
return [
"\n".join(f"{n + 1:5d}| {line}" for n, line in enumerate(lines[i : i + size], start=i)) for i in range(0, max(len(lines), 1), size)
]
class Meter:
@@ -85,7 +89,9 @@ class Meter:
return sum(float(c.cost_usd()) for c in self.clients)
def classify(root: Path, path: Path, family: str, args: argparse.Namespace, meter: Meter, local: threading.local) -> tuple[list[tuple[str, str, int]], int, int]:
def classify(
root: Path, path: Path, family: str, args: argparse.Namespace, meter: Meter, local: threading.local
) -> tuple[list[tuple[str, str, int]], int, int]:
from openultrasast import tool_hunter
if not hasattr(local, "client"):
@@ -99,7 +105,9 @@ def classify(root: Path, path: Path, family: str, args: argparse.Namespace, mete
asked += 1
prompt = PROMPT.format(operation=OPERATIONS[family], path=relative, code=code)
try:
response = local.client.complete(model=args.model, messages=[{"role": "user", "content": prompt}], tools=[], timeout_seconds=90, json_object=True)
response = local.client.complete(
model=args.model, messages=[{"role": "user", "content": prompt}], tools=[], timeout_seconds=90, json_object=True
)
items = json.loads(response.content or "{}").get("functions", [])
except Exception: # noqa: BLE001 -- counted as unclassified, never as "no sinks"
failed += 1
@@ -138,7 +146,7 @@ def main() -> int:
files = product_files(checkout, {case["language"], *case.get("also", [])})
found, failed, asked = [], 0, 0
with concurrent.futures.ThreadPoolExecutor(max_workers=args.workers) as pool:
for f, fail, ask in pool.map(lambda p: classify(checkout, p, case["family"], args, meter, local), files):
for f, fail, ask in pool.map(lambda p, c=checkout, fam=case["family"]: classify(c, p, fam, args, meter, local), files):
found += f
failed += fail
asked += ask
+2 -1
View File
@@ -66,7 +66,8 @@ def main() -> int:
"agreed_findings": len(vulnerable["findings"]), "disputed_findings": len(vulnerable.get("disputed", [])),
"triaged_out": vulnerable.get("triaged_out"), "candidates": vulnerable.get("questions"),
"fixed_side_alerts": fixed_alerts, "fixed_agreed_total": len(fixed["findings"]) if fixed else None,
"removed_by_fix": fixed.get("removed_by_fix") if fixed else None, "usd": vulnerable.get("usd"), "usage": vulnerable.get("usage"),
"removed_by_fix": fixed.get("removed_by_fix") if fixed else None,
"usd": vulnerable.get("usd"), "usage": vulnerable.get("usage"),
}
) # fmt: skip
pool += [{"case": case["id"], **f} for f in vulnerable["findings"]]
+3 -1
View File
@@ -36,7 +36,9 @@ from openultrasast.preprocess import detect_language, enumerate_source_files #
DECLARATION = {
"python": re.compile(r"^\s*(?:async\s+)?def\s+(\w+)"),
"php": re.compile(r"^\s*(?:(?:public|private|protected|static|final|abstract)\s+)*function\s+&?(\w+)"),
"javascript": re.compile(r"^\s*(?:export\s+)?(?:async\s+)?function\s*\*?\s*(\w+)|^\s*(?:(?:public|private|protected|static|async)\s+)*(\w+)\s*\([^;]*\)\s*(?::[^{=]+)?\{\s*$|^\s*(?:export\s+)?(?:const|let|var)\s+(\w+)\s*=\s*(?:async\s*)?(?:function|\()"),
"javascript": re.compile(
r"^\s*(?:export\s+)?(?:async\s+)?function\s*\*?\s*(\w+)|^\s*(?:(?:public|private|protected|static|async)\s+)*(\w+)\s*\([^;]*\)\s*(?::[^{=]+)?\{\s*$|^\s*(?:export\s+)?(?:const|let|var)\s+(\w+)\s*=\s*(?:async\s*)?(?:function|\()"
),
}
DECLARATION["typescript"] = DECLARATION["javascript"]
KEYWORDS = {"if", "for", "while", "switch", "catch", "return", "function", "else"}
+87 -14
View File
@@ -101,13 +101,22 @@ def triage(root: Path, path: str, candidates: list[list], family: str, model: st
except OSError:
return {}
names = sorted({fn for _, fn, _ in candidates})
prompt = TRIAGE.format(operation=OPERATIONS[family], functions=json.dumps(names), path=path, code=_relevant_code(text, [ln for _, _, ln in candidates], chunk_lines))
prompt = TRIAGE.format(
operation=OPERATIONS[family],
functions=json.dumps(names),
path=path,
code=_relevant_code(text, [ln for _, _, ln in candidates], chunk_lines),
)
response = client.complete(model=model, messages=[{"role": "user", "content": prompt}], tools=[], timeout_seconds=90, json_object=True)
try:
items = json.loads(response.content or "{}").get("functions", [])
except (json.JSONDecodeError, AttributeError):
return {}
return {str(i.get("function")): str(i.get("data_origin", "")).strip().lower() for i in items if isinstance(i, dict) and i.get("function") in names}
return {
str(i.get("function")): str(i.get("data_origin", "")).strip().lower()
for i in items
if isinstance(i, dict) and i.get("function") in names
}
def hunt(root: Path, path: str, group: list[list], family: str, model: str, steps: int) -> tuple[list[dict], dict]:
@@ -120,8 +129,30 @@ def hunt(root: Path, path: str, group: list[list], family: str, model: str, step
f"Operations to judge, all in `{path}`, each of which {OPERATIONS[family]}:\n{listing}\n"
"The file is above. For each, trace where the operation's data comes from and whether it is guarded."
)
spots = [Hotspot(path=path, function_name=fn, score=1.0, band="candidate", signals={}, rationale="model-classified sink", test_hint=None, inventory_finding_ids=()) for _, fn, _ in group]
found = tool_hunter.run_tool_hunter(root, spots, client=recording, model=model, max_steps=steps, system_prompt=SYSTEM, user_prompt=prompt, context_files=[path], tags=("sink-verifier",))
spots = [
Hotspot(
path=path,
function_name=fn,
score=1.0,
band="candidate",
signals={},
rationale="model-classified sink",
test_hint=None,
inventory_finding_ids=(),
)
for _, fn, _ in group
]
found = tool_hunter.run_tool_hunter(
root,
spots,
client=recording,
model=model,
max_steps=steps,
system_prompt=SYSTEM,
user_prompt=prompt,
context_files=[path],
tags=("sink-verifier",),
)
names = {fn for _, fn, _ in group}
rows = []
for f in found:
@@ -131,7 +162,16 @@ def hunt(root: Path, path: str, group: list[list], family: str, model: str, step
# The verdict is about the candidate operation, so the site is the candidate's file and function; the
# model's own location is kept beside it (it often names the file where the input enters instead).
line = f.line if f.path == path else next(c[2] for c in group if c[1] == fn)
rows.append({"candidate": f"{path}::{fn}", "site": f"{path}:{line or 0}:{fn}", "reported_at": f"{f.path}:{f.line or 0}", "family": family, "title": f.title, "witness": f.rationale[:600]})
rows.append(
{
"candidate": f"{path}::{fn}",
"site": f"{path}:{line or 0}:{fn}",
"reported_at": f"{f.path}:{f.line or 0}",
"family": family,
"title": f.title,
"witness": f.rationale[:600],
}
)
return rows, recording.usage()
@@ -143,7 +183,12 @@ def verify_file(root: str, path: str, candidates: list[list], family: str, args_
try:
if not args_["no_triage"]:
record["triage"] = triage(Path(root), path, candidates, family, args_["model"], args_["chunk_lines"])
kept = [c for c in candidates if record["triage"].get(c[1], "unclassified") in VERIFY | {"unclassified"} or record["triage"].get(c[1]) not in {"internal", "constant"}]
kept = [
c
for c in candidates
if record["triage"].get(c[1], "unclassified") in VERIFY | {"unclassified"}
or record["triage"].get(c[1]) not in {"internal", "constant"}
]
record["kept"] = [c[1] for c in kept]
usage: dict[str, int] = {}
for _pass in range(args_["passes"]):
@@ -169,14 +214,16 @@ def agreement(record: dict) -> tuple[list[dict], list[dict]]:
some = set.union(*(set(b) for b in by_pass))
first = by_pass[0]
pick = lambda c: next(b[c] for b in by_pass if c in b) # noqa: E731
return [dict(first[c], passes=len(by_pass)) for c in sorted(every)], [dict(pick(c), passes=sum(c in b for b in by_pass)) for c in sorted(some - every)]
return [dict(first[c], passes=len(by_pass)) for c in sorted(every)], [
dict(pick(c), passes=sum(c in b for b in by_pass)) for c in sorted(some - every)
]
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--population", type=Path, default=HERE / "population-v2.toml")
parser.add_argument("--only", default="")
parser.add_argument("--subset", type=Path, help='JSON {case_id: [[path, function], ...]}: verify only these candidates')
parser.add_argument("--subset", type=Path, help="JSON {case_id: [[path, function], ...]}: verify only these candidates")
parser.add_argument("--model", default=DEFAULT_DETECTOR_MODEL)
parser.add_argument("--max-steps", type=int, default=6)
parser.add_argument("--per-hunt", type=int, default=6)
@@ -206,7 +253,11 @@ def main() -> int:
if out.is_file():
continue
log = out_dir / f"{case['id']}--{label}.jsonl"
done = {r["path"]: r for r in (json.loads(row) for row in log.read_text().splitlines()) if "unverified" not in r} if log.is_file() else {}
done = (
{r["path"]: r for r in (json.loads(row) for row in log.read_text().splitlines()) if "unverified" not in r}
if log.is_file()
else {}
)
spent += sum(float(r.get("usd", 0.0)) for r in done.values())
record = json.loads((candidates_dir / f"{case['id']}.json").read_text())
unique = list({(p, fn): [p, fn, ln] for p, fn, ln in sorted(record["candidates"])}.values())
@@ -265,19 +316,41 @@ def main() -> int:
agreed += a
disputed += d
triaged_out = sum(len(r["candidates"]) - len(r.get("kept", [])) for r in done.values())
usage = {k: sum(int(r.get("usage", {}).get(k, 0)) for r in done.values()) for k in ("prompt_tokens", "prompt_cache_hit_tokens", "completion_tokens", "calls")}
usage = {
k: sum(int(r.get("usage", {}).get(k, 0)) for r in done.values())
for k in ("prompt_tokens", "prompt_cache_hit_tokens", "completion_tokens", "calls")
}
complete = len(done) >= len(by_file) and not account_error
result = {
"root": "/case", "families": [case["family"]], "questions": len(unique), "completed": sum(len(r["candidates"]) for r in done.values()),
"root": "/case", "families": [case["family"]],
"questions": len(unique), "completed": sum(len(r["candidates"]) for r in done.values()),
"completed_regions": sorted(f"{c[0]}:{c[1]}" for r in done.values() for c in r["candidates"]),
"triaged_out": triaged_out, "passes": args.passes, "usage": usage, "usd": round(sum(float(r.get("usd", 0.0)) for r in done.values()), 4),
"seconds": round(time.monotonic() - started, 1), "findings": sorted(agreed, key=lambda f: f["site"]), "disputed": sorted(disputed, key=lambda f: f["site"]),
"triaged_out": triaged_out, "passes": args.passes, "usage": usage,
"usd": round(sum(float(r.get("usd", 0.0)) for r in done.values()), 4),
"seconds": round(time.monotonic() - started, 1),
"findings": sorted(agreed, key=lambda f: f["site"]),
"disputed": sorted(disputed, key=lambda f: f["site"]),
} # fmt: skip
if args.fixed:
result["removed_by_fix"] = removed
if complete:
out.write_text(json.dumps(result, indent=1) + "\n")
print(json.dumps({"case": case["id"], "files": f"{len(done)}/{len(by_file)}", "candidates": len(unique), "triaged_out": triaged_out, "agreed": len(agreed), "disputed": len(disputed), "usd": result["usd"], "total_usd": round(spent, 4), "complete": complete}), flush=True)
print(
json.dumps(
{
"case": case["id"],
"files": f"{len(done)}/{len(by_file)}",
"candidates": len(unique),
"triaged_out": triaged_out,
"agreed": len(agreed),
"disputed": len(disputed),
"usd": result["usd"],
"total_usd": round(spent, 4),
"complete": complete,
}
),
flush=True,
)
if account_error:
print(json.dumps({"stopped": "account", "error": account_error}), flush=True)
return 2
+53 -7
View File
@@ -104,7 +104,16 @@ def verify(root: str, candidate: list, family: str, model: str, max_steps: int,
client = _client()
before = float(client.cost_usd())
path, function, line = candidate
spot = Hotspot(path=path, function_name=function, score=1.0, band="candidate", signals={}, rationale="model-classified sink", test_hint=None, inventory_finding_ids=())
spot = Hotspot(
path=path,
function_name=function,
score=1.0,
band="candidate",
signals={},
rationale="model-classified sink",
test_hint=None,
inventory_finding_ids=(),
)
prompt = (
f"Operation to judge: in `{path}`, function `{function}`, around line {line}, code that {OPERATIONS[family]}. "
"The file is above. Trace where the operation's data comes from and whether it is guarded."
@@ -120,9 +129,19 @@ def verify(root: str, candidate: list, family: str, model: str, max_steps: int,
second = recording.usage()
usage = {k: usage.get(k, 0) + second.get(k, 0) for k in set(usage) | set(second)}
except Exception as error: # noqa: BLE001 -- recorded; an unverified candidate is not a clean one
return {"candidate": candidate, "unverified": f"{type(error).__name__}: {str(error)[:120]}", "usd": float(client.cost_usd()) - before}
return {
"candidate": candidate,
"unverified": f"{type(error).__name__}: {str(error)[:120]}",
"usd": float(client.cost_usd()) - before,
}
findings = [
{"site": f"{f.path}:{f.line or line}:{f.function_name or function}", "family": family, "title": f.title, "witness": f.rationale[:600], "candidate": f"{path}::{function}"}
{
"site": f"{f.path}:{f.line or line}:{f.function_name or function}",
"family": family,
"title": f.title,
"witness": f.rationale[:600],
"candidate": f"{path}::{function}",
}
for f in found
]
return {"candidate": candidate, "findings": findings, "usd": float(client.cost_usd()) - before, "escalated": escalated, "usage": usage}
@@ -134,7 +153,12 @@ def main() -> int:
parser.add_argument("--only", default="")
parser.add_argument("--model", default=DEFAULT_DETECTOR_MODEL)
parser.add_argument("--max-steps", type=int, default=6)
parser.add_argument("--escalate-steps", type=int, default=0, help="rerun a hunt that used every step without a verdict; 0 disables (measured: 88% escalate, +45% cost)")
parser.add_argument(
"--escalate-steps",
type=int,
default=0,
help="rerun a hunt that used every step without a verdict; 0 disables (measured: 88% escalate, +45% cost)",
)
parser.add_argument("--workers", type=int, default=4)
parser.add_argument("--budget-usd", type=float, default=40.0)
parser.add_argument("--fixed", action="store_true", help="recheck detected sites at the fixed pin")
@@ -185,7 +209,9 @@ def main() -> int:
with concurrent.futures.ProcessPoolExecutor(max_workers=args.workers) as pool, log.open("a") as sink:
futures = []
for candidate in pending:
futures.append(pool.submit(verify, str(checkout), list(candidate), case["family"], args.model, args.max_steps, args.escalate_steps))
futures.append(
pool.submit(verify, str(checkout), list(candidate), case["family"], args.model, args.max_steps, args.escalate_steps)
)
for future in concurrent.futures.as_completed(futures):
if future.cancelled(): # cancelled at the ceiling: never run, nothing to record
continue
@@ -219,9 +245,29 @@ def main() -> int:
} # fmt: skip
if complete:
out.write_text(json.dumps(result, indent=1) + "\n")
tokens = {k: sum(int(d.get("usage", {}).get(k, 0)) for d in done) for k in ("prompt_tokens", "prompt_cache_hit_tokens", "completion_tokens")}
tokens = {
k: sum(int(d.get("usage", {}).get(k, 0)) for d in done)
for k in ("prompt_tokens", "prompt_cache_hit_tokens", "completion_tokens")
}
result["usage"] = tokens | {"escalated": sum(1 for d in done if d.get("escalated"))}
print(json.dumps({"case": case["id"], "pin": label, "candidates": len(todo), "verified": len(verified), "findings": len(findings), "unverified": len(unverified), "escalated": result["usage"]["escalated"], "cache_share": round(tokens["prompt_cache_hit_tokens"] / max(tokens["prompt_tokens"], 1), 2), "usd": result["usd"], "total_usd": round(spent, 4), "complete": complete}), flush=True)
print(
json.dumps(
{
"case": case["id"],
"pin": label,
"candidates": len(todo),
"verified": len(verified),
"findings": len(findings),
"unverified": len(unverified),
"escalated": result["usage"]["escalated"],
"cache_share": round(tokens["prompt_cache_hit_tokens"] / max(tokens["prompt_tokens"], 1), 2),
"usd": result["usd"],
"total_usd": round(spent, 4),
"complete": complete,
}
),
flush=True,
)
if account_error:
print(json.dumps({"stopped": "account", "error": account_error, "spent_usd": round(spent, 4)}), flush=True)
return 2

Some files were not shown because too many files have changed in this diff Show More