marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-20 05:29:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

spec(search-with-proof): requirements amended after two Codex reviews; design v2

Browse Source 
False alarms count every visible finding; the verifier owns per-family oracles;
inconclusive on one-sided build failure; feasibility pilot separated from a
73-negative BLOCK qualification; coverage ceilings; spend reserved before calls;
key isolated from executed code; cluster model calls supersede the pool VM-only
rule for this arm only. Requirements approval reopened.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-05 13:35:18 +02:00
co-authored by Claude Opus 5.5
parent 8bbea86cf8
commit c3820e184c
3 changed files with 84 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/search-with-proof/requirements.md
+55 -19
View File
@@ -57,13 +57,21 @@ known instead of restarting.
#### Acceptance Criteria
1. A demonstration is an executable artefact (test, script or request sequence) that the system, not the agent,
re-runs in a fresh sandbox: it fails (shows the vulnerable behaviour) on the vulnerable revision and passes on the
fixed revision for a labelled pair; for a push it shows the behaviour on head (and not on base where the change
introduced it).
2. Only a re-verified demonstration counts as a demonstrated finding. The agent's own claim of success is never enough.
3. Outcomes are recorded as `demonstrated`, `plausible_unproven` (evidence but no passing demonstration),
`could_not_build` / `could_not_run` (environment failures), or `no_evidence`; none of the failure outcomes is ever
reported as "not vulnerable".
re-runs in fresh sandboxes. The **verifier observes the effect itself** through a per-family oracle it owns (for
example a request log, a planted resource it watches, a browser execution check for XSS); the artefact's own exit
code or output is never the oracle.
2. Each family's oracle states the attacker boundary (what the artefact may touch: only the documented external
inputs of the code under analysis, never the verifier's fixtures, the source tree, dependencies or the runtime).
Inputs are immutable during a run; untracked files, dependency replacement and runtime patching are refused.
3. Differential: for a labelled pair the effect is observed on the vulnerable revision and not on the fixed one; for a
push, on head and not on base. Both sides must build and pass a readiness check independently; a side that fails
to build or start makes the result **inconclusive**, never a pass. Each side is run at least three times from fresh
state; inconsistent runs are rejected.
4. Only a verified demonstration counts as `demonstrated`. Outcomes: `demonstrated`, `plausible_unproven`,
`could_not_build`, `could_not_run`, `inconclusive`, `no_oracle` (family without an owned oracle), `no_evidence`;
none of the failure outcomes is ever reported as "not vulnerable".
5. Families without an owned oracle (for now: deserialisation, access control, configuration and secrets) can reach
`plausible_unproven` at most until an oracle is specified and reviewed.
### Requirement 4: Triage and budget
@@ -74,7 +82,9 @@ known instead of restarting.
1. Candidates come from the existing layers (quick rules, engine, classifier scores); only the top K per push or per
pair side are searched; K and the per-search and per-push budgets (model spend, task count, wall time) are
configuration, enforced by the coordinator.
2. Every search records its model spend, task count and wall time; a budget is never exceeded.
2. Every search records its model spend, task count and wall time. Spend is **reserved before each model call** (the
call's maximum cost) and released after; wall time, memory, disk and retries are capped per task and in total; a
budget is never exceeded, including by calls in flight.
3. Paid runs need the maintainer's budget go-ahead with a ceiling, as all paid steps do.
### Requirement 5: Model access in the cluster
@@ -85,19 +95,45 @@ known instead of restarting.
1. The agents' model key is a scoped, budget-limited key provided by the operator inside the cluster (AX v0.3.1
injects only `gemini-api-secret` as `GEMINI_API_KEY`; the design names the exact mechanism agreed with the
operator). Store credentials stay out of the cluster (presigned links, as today).
2. The key never appears in a manifest, a log, a board entry or a record.
operator). Store credentials stay out of the cluster (presigned links, as today). For this arm, model calls inside
the cluster supersede `unseen-repo-evaluation` Req 4.4 (model calls on the VM); that boundary is unchanged for all
other arms. *(maintainer decision 2026-10-05: scoped key in the cluster)*
2. The key never appears in a manifest, a log, a board entry or a record, and is not readable by the repository code
the sandbox executes (the model client and the executed code are separated).
### Requirement 6: Pilot, then the unseen pool
### Requirement 6: Feasibility pilot, false-proof qualification, then the unseen pool
**User Story:** As the maintainer, I want to know quickly whether search with proof works before scaling it.
**User Story:** As the maintainer, I want to know quickly whether search with proof works, and to admit BLOCK only on
evidence that meets the 95% precision bar.
#### Acceptance Criteria
1. Pilot: about 20 known fix pairs across families and languages; for each, search runs on the vulnerable side and
on the fixed side under the same budget. Reported: demonstrated rate on vulnerable sides, any demonstration on a
fixed side (a false proof, which must be zero), environment-failure rate, spend and wall time per search.
2. Exit of the pilot: at least one family with demonstrations on vulnerable sides and zero false proofs; otherwise
the result is recorded and the arm is not scaled.
3. Then a registered G1 arm on the unseen-repo pool: catch rate at the pool's false-alarm budgets, with BLOCK only on
demonstrated findings, compared with the classifier arms.
1. **Feasibility pilot** (amended 2026-10-05 after review): about 20 known fix pairs from development corpora. It
measures feasibility only and is reported as such: demonstrated rate on vulnerable sides, false proofs on fixed
sides, `could_not_build` / `could_not_run` / `inconclusive` / `no_oracle` rates, spend, tasks and wall time. Its
advantages (known location, labelled family, runnable-project bias) are stated in the record. Fixed-side searches
never receive the vulnerable revision or its demo.
2. Before any paid search, an infrastructure probe without model calls proves each owned oracle works inside a gVisor
AX Task (database fixture for SQL, in-sandbox listener for SSRF, headless browser for XSS, file and process
watches), that the artefact cannot reach the observers directly, and records image size, RAM and the full
verification cost (up to 3 demos x 2 sides x 3 runs per search). Pilot exit: at least one family with at least 5 verified demonstrations on vulnerable sides and zero false proofs;
otherwise the result is recorded and the arm is not scaled.
3. **BLOCK admission per family** (deferred until the pilot passes; not part of the first tasks) requires a false-proof
qualification. A **negative** is a search on code that should be safe: a fixed revision searched on its own (the
search never sees the vulnerable revision), or an ordinary change. A **false proof** is the verifier observing the
effect on that safe code, confirmed by adjudication (an ordinary-change alarm may be a real new bug and is
adjudicated before it counts either way). Qualification needs zero false proofs over at least 73 negatives drawn
from at least 73 distinct repositories, so the Wilson 95% upper bound on the false-proof rate per negative is at
most 5%. This bounds the false-proof rate, not precision among flags; precision is reported separately from the
pool. Repeated runs and repeated searches of one repository are one negative.
4. **Coverage ceilings**: every run reports, separately, (a) the demonstrated ceiling: the share of vulnerable changes
with a true candidate selected, an owned oracle, both sides runnable and a genuine differential, build and run
failures in the denominator; and (b) the advisory ceiling: the share with a true candidate selected. Families
without an owned oracle (access control, deserialisation, configuration and secrets) have a demonstrated and BLOCK
ceiling of zero today; their advisory ceiling need not be zero.
5. **G1 arm on the unseen pool**: scored with the pool's rules on **every visible finding** (BLOCK and advisory), with
its family and location match; catch at the 1%, 5% and 10% false-alarm budgets; compared with the classifier arms;
adopted only under the pool's no-regression rule on a fresh slice. No evaluation label, fixing revision or board
from evaluation searches is ever an input to unseen search.
6. Delayed results (a search finishing after the push) reach the user through the existing background delivery and
are scored separately from results available at push time.