Open_Ultra_SAST/.kiro/specs/search-with-proof/requirements.md

# Requirements Document

## Introduction

Add a search-with-proof stage: for the top-ranked candidates of a push or a labelled pair, sandboxed tool-using agents
search for a demonstration of the vulnerability, coordinated through a fact/intent board, executing primarily in the
kube-ax cluster. BLOCK becomes "demonstrated", not "scored above a threshold". Context: `brief.md`, roadmap G1-G2,
`unseen-repo-evaluation` (the pool and metric this arm is judged by).

## Boundary Context

- In scope: the board (facts, intents, hints) in the memory store; reason and explore steps as AX Tasks; the
  demonstration contract and its verification; triage-to-search budgeting; a pilot on known fix pairs; a G1 arm on the
  unseen pool; model access through an operator-managed scoped key.
- Out of scope: copying Cairn code (AGPL-3.0); network actions against anything but the code under analysis;
  changing the existing quick rules, engine or classifier (they are inputs here).
- Adjacent expectations: `unseen-repo-evaluation` scores this arm; the AX batch dispatcher (`benchmarks/ax/batch.py`)
  and the kube-ax constraints (2 heavy lanes, `ousast-engine-` names, egress to public HTTPS) apply;
  `learned-decision-engine` Req 6.4 BLOCK rule is superseded for this arm by "demonstrated".

## Requirements

### Requirement 1: The board

**User Story:** As the maintainer, I want each search to keep its state, so exploration builds on what is already
known instead of restarting.

#### Acceptance Criteria

1. Per search (one candidate or one push), the board holds facts (confirmed, objective, with the evidence that
   confirmed them), intents (declared directions, open or concluded, with the facts they start from) and hints (human
   input, absorbed on the next read).
2. The board lives in the memory store, versioned, and every fact and intent carries the step and the task that wrote
   it.
3. A search ends when a fact meets the goal (Requirement 3), when its budget is spent, or when the reasoner proposes no
   new intent and none is open; the end reason is recorded.

### Requirement 2: Reason and explore steps

**User Story:** As the maintainer, I want the search driven by the board's state, not by fixed roles.

#### Acceptance Criteria

1. A reason step reads the board and returns: the goal is met (citing facts), or at most N new, independent,
   non-overlapping intents, or no new intent.
2. An explore step claims one intent, works it with tools inside the sandbox (read files, search the repository,
   follow callers, build, run tests and scripts against the code under analysis), and returns exactly one new fact or
   a recorded failure.
3. Both run as AX Tasks in the kube-ax cluster by default; the VM coordinates (board, dispatch, budgets) and is a
   fallback lane only.
4. Parallel explores of one search never race on the board (claims as in the batch dispatcher).

### Requirement 3: The demonstration contract

**User Story:** As the maintainer, I want BLOCK to mean "shown", so precision does not depend on calibration.

#### Acceptance Criteria

1. *(Amended 2026-10-05, maintainer decision after the kube-ax probe: AX tasks have no capabilities, so gVisor and one
   fresh task per side are the only isolation.)* A demonstration is **declarative data, never agent-written code**: a
   build recipe chosen from the verifier's fixed per-ecosystem recipes, a start command of the form runtime + path +
   arguments (no shell), and a request sequence (HTTP requests or CLI invocations, with values captured from earlier
   responses) that the verifier itself sends. The system, not the agent, runs it in fresh sandboxes. The **verifier observes the effect itself** through a per-family oracle it owns (for
   example a request log, a planted resource it watches, a browser execution check for XSS); the artefact's own exit
   code or output is never the oracle.
2. Each family's oracle states the attacker boundary (what the artefact may touch: only the documented external
   inputs of the code under analysis, never the verifier's fixtures, the source tree, dependencies or the runtime).
   Inputs are immutable during a run; untracked files, dependency replacement and runtime patching are refused.
3. Differential: for a labelled pair the effect is observed on the vulnerable revision and not on the fixed one; for a
   push, on head and not on base. Both sides must build and pass a readiness check independently; a side that fails
   to build or start makes the result **inconclusive**, never a pass. Each side is run at least three times from fresh
   state; inconsistent runs are rejected.
4. Only a verified demonstration counts as `demonstrated`. Outcomes: `demonstrated`, `plausible_unproven`,
   `could_not_build`, `could_not_run`, `inconclusive`, `no_oracle` (family without an owned oracle), `no_evidence`;
   none of the failure outcomes is ever reported as "not vulnerable".
5. Families without an owned oracle (for now: deserialisation, access control, configuration and secrets) can reach
   `plausible_unproven` at most until an oracle is specified and reviewed.

### Requirement 4: Triage and budget

**User Story:** As the maintainer, I want search spent where it matters, at a known cost.

#### Acceptance Criteria

1. Candidates come from the existing layers (quick rules, engine, classifier scores); only the top K per push or per
   pair side are searched; K and the per-search and per-push budgets (model spend, task count, wall time) are
   configuration, enforced by the coordinator.
2. Every search records its model spend, task count and wall time. Spend is **reserved before each model call** (the
   call's maximum cost) and released after; wall time, memory, disk and retries are capped per task and in total; a
   budget is never exceeded, including by calls in flight.
3. Paid runs need the maintainer's budget go-ahead with a ceiling, as all paid steps do.

### Requirement 5: Model access in the cluster

**User Story:** As the maintainer, I want agents in the cluster to reach a model without spreading keys.

#### Acceptance Criteria

1. The agents' model key is a scoped, budget-limited key provided by the operator inside the cluster (AX v0.3.1
   injects only `gemini-api-secret` as `GEMINI_API_KEY`; the design names the exact mechanism agreed with the
   operator). Store credentials stay out of the cluster (presigned links, as today). For this arm, model calls inside
   the cluster supersede `unseen-repo-evaluation` Req 4.4 (model calls on the VM); that boundary is unchanged for all
   other arms. *(maintainer decision 2026-10-05: scoped key in the cluster)*
2. The key never appears in a manifest, a log, a board entry or a record, and never shares a sandbox with repository
   code. *(Amended 2026-10-05, maintainer decision: trust domains are separated instead of isolated inside one task.)*
   The agent loop (the "brain", holding the key) runs on the VM or in its own AX task and executes no repository code;
   every tool call that touches the repository runs in a separate executor AX task without the key, exchanging
   commands and results through presigned store objects. A Firecracker executor runtime is considered only if the
   pilot's `could_not_build` rate shows gVisor cannot build the projects.

### Requirement 6: Feasibility pilot, false-proof qualification, then the unseen pool

**User Story:** As the maintainer, I want to know quickly whether search with proof works, and to admit BLOCK only on
evidence that meets the 95% precision bar.

#### Acceptance Criteria

1. **Feasibility pilot** (amended 2026-10-05 after review): about 20 known fix pairs from development corpora. It
   measures feasibility only and is reported as such: demonstrated rate on vulnerable sides, false proofs on fixed
   sides, `could_not_build` / `could_not_run` / `inconclusive` / `no_oracle` rates, spend, tasks and wall time. Its
   advantages (known location, labelled family, runnable-project bias) are stated in the record. Fixed-side searches
   never receive the vulnerable revision or its demo.
2. Before any paid search, an infrastructure probe without model calls proves each owned oracle works inside a gVisor
   AX Task (database fixture for SQL, in-sandbox listener for SSRF, headless browser for XSS, file and process
   watches), that the artefact cannot reach the observers directly, and records image size, RAM and the full
   verification cost (up to 3 demos x 2 sides x 3 runs per search). Pilot exit: at least one family with at least 5 verified demonstrations on vulnerable sides and zero false proofs;
   otherwise the result is recorded and the arm is not scaled.
3. **BLOCK admission per family** (deferred until the pilot passes; not part of the first tasks) requires a false-proof
   qualification. A **negative** is a search on code that should be safe: a fixed revision searched on its own (the
   search never sees the vulnerable revision), or an ordinary change. A **false proof** is the verifier observing the
   effect on that safe code, confirmed by adjudication (an ordinary-change alarm may be a real new bug and is
   adjudicated before it counts either way). Qualification needs zero false proofs over at least 73 negatives drawn
   from at least 73 distinct repositories, so the Wilson 95% upper bound on the false-proof rate per negative is at
   most 5%. This bounds the false-proof rate, not precision among flags; precision is reported separately from the
   pool. Repeated runs and repeated searches of one repository are one negative.
4. **Coverage ceilings**: every run reports, separately, (a) the demonstrated ceiling: the share of vulnerable changes
   with a true candidate selected, an owned oracle, both sides runnable and a genuine differential, build and run
   failures in the denominator; and (b) the advisory ceiling: the share with a true candidate selected. Families
   without an owned oracle (access control, deserialisation, configuration and secrets) have a demonstrated and BLOCK
   ceiling of zero today; their advisory ceiling need not be zero.
5. **G1 arm on the unseen pool**: scored with the pool's rules on **every visible finding** (BLOCK and advisory), with
   its family and location match; catch at the 1%, 5% and 10% false-alarm budgets; compared with the classifier arms;
   adopted only under the pool's no-regression rule on a fresh slice. No evaluation label, fixing revision or board
   from evaluation searches is ever an input to unseen search.
6. Delayed results (a search finishing after the push) reach the user through the existing background delivery and
   are scored separately from results available at push time.