docs: plain-English pass over the active specs

Ran the humanizer rules over 79 prose files in the active specs (brief,
design, requirements, research and narrative docs). Removed em dashes,
significance inflation, forced rule-of-three, bold-label list items and
filler. Skipped tasks.md checklists and the retired specs (harnessx-*,
openrouter-sast-harness, unseen-repo-evaluation).

Every number, path, command, hash and identifier was preserved: checked
each file against HEAD for drift in numbers, inline code and code fences,
zero drift across all 79 files.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-12 18:16:41 +02:00
co-authored by Claude Opus 4.8
parent b08f372022
commit 39a1873e5b
79 changed files with 1103 additions and 1103 deletions
+3 -3
View File
@@ -141,7 +141,7 @@ before `ax apply`, and a task without a Model gets none. The value is never rend
Every actor connection except DNS goes through Agent Substrate's egress gateway (`atenet-egress`), which denies
an actor without an `EgressPolicy`. Three pieces make the plane work under that rule:
- **Receiver by name.** `ops/ax/receiver-service.yaml.tmpl` (applied by `up.sh`) is a selector-less Service
- Receiver by name. `ops/ax/receiver-service.yaml.tmpl` (applied by `up.sh`) is a selector-less Service
`ousast-receiver` in `ax-system`, port 80, with an EndpointSlice on the kind gateway at `OUSAST_ARTIFACT_PORT`
(fixed, 18090), where the reconciler binds its receiver. Tasks get
`OUSAST_ARTIFACT_URL=http://ousast-receiver.ax-system.svc.cluster.local/` plus `OUSAST_ARTIFACT_DIAL=<the
@@ -150,12 +150,12 @@ an actor without an `EgressPolicy`. Three pieces make the plane work under that
`Host` but connects to the address the actor dialled (measured: dialling the kind gateway on port 80 was allowed
and then refused upstream), so the dial address is the ClusterIP, which kube-proxy maps to the host.
`OUSAST_ARTIFACT_HOST` set keeps the old direct URL (tests).
- **Per-task policy.** `policy_for` grants `http` on port 80 to the receiver name, and `tls_passthrough` on 443
- Per-task policy. `policy_for` grants `http` on port 80 to the receiver name, and `tls_passthrough` on 443
to the Git hosts of the bound Workspaces' `https` repos and to the bound Model's
`openultrasast.io/egress-hosts` (`api.deepseek.com` for `deepseek-flash`, validated as a plain DNS name).
Nothing else: no IP address or wildcard, and passthrough only -- an `https` rule intercepts TLS and the actor
would need the gateway CA, which ax templates do not mount.
- **Order.** `ax apply` creates the actor; the reconciler writes the policy with `kubectl ate create
- Order. `ax apply` creates the actor; the reconciler writes the policy with `kubectl ate create
egress-policy` (update when one exists), reads it back, waits `OUSAST_EGRESS_SETTLE` (11 s: the gateway caches a
missing policy as deny for 10 s), then resumes. kubectl-ate has no delete verb; Substrate's store deletes the
policy with its actor, so after `ax delete task` the reconciler confirms it is gone.
+4 -4
View File
@@ -14,19 +14,19 @@ a DeepSeek provider -- is carried by our thin `Run` layer and dissolves into ax
## Boundary Context
- **In scope**: the manifest kinds and a `Run` kind for dependencies, artifacts and budgets; bring-up of ax on
- In scope: the manifest kinds and a `Run` kind for dependencies, artifacts and budgets; bring-up of ax on
this host (kind + Agent Substrate + ax controller) with a doctor command; an ax-backed reconciler that
submits Tasks to ax (one engine-bearing task at a time) and is the only executor; the runner contract and a
runner image; the first two tasks, `repo-facts` and
`verify`, with the existing scripts as reference behaviour; measurement on the 46-candidate validation set.
- **Out of scope**: a remote or multi-node cluster; contributing providers or budgets to ax upstream (a
- Out of scope: a remote or multi-node cluster; contributing providers or budgets to ax upstream (a
follow-on once the increment is measured); changing what the model is asked (prompts, families, pass rules)
except as required to consume `repo-facts`; the judge stage, the config-family question and population v3
(they become tasks later, under their own specs); any change to `evaluate.py`'s protocol scoring.
- **Adjacent expectations**: the independent-population protocols and records stay as they are; `finding_dump`
- Adjacent expectations: the independent-population protocols and records stay as they are; `finding_dump`
and the engine path are untouched; `vulnerabilities-over-plumbing` applies -- this is plumbing that must pay
for itself in measured tokens and unchanged or better recall.
- **Target state (maintainer, 2026-09-29)**: this plane replaces HarnessX as the agentic plane; HarnessX is
- Target state (maintainer, 2026-09-29): this plane replaces HarnessX as the agentic plane; HarnessX is
removed by its own spec (`harnessx-removal`) once Requirement 6's measurement shows value; the `Run` layer
shrinks as ax gains dependencies, budgets and artifacts. No new HarnessX feature is added meanwhile.
@@ -8,9 +8,9 @@ Nothing in the pipeline can represent "this operation should have been guarded"
## Desired Outcome
Absence bugs become first-class, measured, and honest. An obligated operation (protected data access, privileged action, security configuration) reached from an entry point without a discharger (a dominating auth guard, an identity constraint bound from the authenticated context, a non-permissive value) is reported as an obligation violation, with the missing discharger kind, the evidence for why the obligation exists (sibling handlers that discharge it, or a declared policy clause), and the known fix shape the corpus taught. Leave-one-out on the absence pairs is the objective, per obligation kind and per provenance profile, so the number rises as the corpus grows. `agent-vfc` and `vibe-py` absence pairs move from 0 detected to a measured baseline that the engine specs can raise.
Absence bugs become first-class and are measured without overstating results. An obligated operation (protected data access, privileged action, security configuration) reached from an entry point without a discharger (a dominating auth guard, an identity constraint bound from the authenticated context, a non-permissive value) is reported as an obligation violation, with the missing discharger kind, the evidence for why the obligation exists (sibling handlers that discharge it, or a declared policy clause), and the known fix shape the corpus taught. Leave-one-out on the absence pairs is the objective, per obligation kind and per provenance profile, so the number rises as the corpus grows. `agent-vfc` and `vibe-py` absence pairs move from 0 detected to a measured baseline that the engine specs can raise.
Principle written into the roadmap: **an operation carries obligations, not only data; the application's own consistency and its declared policy say which, and the tool reports the missing discharger rather than a phantom flow.**
Principle written into the roadmap: **an operation carries data and obligations; the application's own consistency and its declared policy say which, and the tool reports the missing discharger rather than a phantom flow.**
## Approach
@@ -18,8 +18,8 @@ Obligated operations and discharger kinds are closed facts data, like sinks and
## Scope
- **In**: obligated-operation and discharger facts; obligation shapes derived from pairs; consistency detection across sibling handlers; optional declared policy file with a closed schema; path-aware violation evaluation with a function-local degraded mode; new evidence rungs and report wording for obligations; worth-fixing weighting by resource sensitivity; corpus recipe shape for absence pairs (handler plus registration context); leave-one-out per obligation kind; `obligations` improve lever; offline tests.
- **Out**: taint or facts changes for flow bugs; the call graph and path records themselves (reachability spec); guard-dominance IR (guard-dominance-regime); LLM-authored policy or shapes; any merge gate; vendoring unlicensed code; sandbox proof of absence bugs (they do not crash).
- In: obligated-operation and discharger facts; obligation shapes derived from pairs; consistency detection across sibling handlers; optional declared policy file with a closed schema; path-aware violation evaluation with a function-local degraded mode; new evidence rungs and report wording for obligations; worth-fixing weighting by resource sensitivity; corpus recipe shape for absence pairs (handler plus registration context); leave-one-out per obligation kind; `obligations` improve lever; offline tests.
- Out: taint or facts changes for flow bugs; the call graph and path records themselves (reachability spec); guard-dominance IR (guard-dominance-regime); LLM-authored policy or shapes; any merge gate; vendoring unlicensed code; sandbox proof of absence bugs (they do not crash).
## Boundary Candidates
@@ -40,13 +40,13 @@ Obligated operations and discharger kinds are closed facts data, like sinks and
## Upstream / Downstream
- **Upstream**: `EntryPointRecord` (route, access level, decorators, conditions); `FileIR`; path records and dominance from the two engine specs; `PairCase` with `review_tier`; `MechanismStore` and the lever machinery; facts loader.
- **Downstream**: reports and SARIF; worth-fixing ranking; the roadmap's leave-one-out table gains obligation kinds; `guard-dominance-regime` consumes obligation discharger kinds as guard candidates for web code.
- Upstream: `EntryPointRecord` (route, access level, decorators, conditions); `FileIR`; path records and dominance from the two engine specs; `PairCase` with `review_tier`; `MechanismStore` and the lever machinery; facts loader.
- Downstream: reports and SARIF; worth-fixing ranking; the roadmap's leave-one-out table gains obligation kinds; `guard-dominance-regime` consumes obligation discharger kinds as guard candidates for web code.
## Existing Spec Touchpoints
- **Extends**: `corpus-seeded-mechanisms` (second shape family, same store, same lever contract), `pair-corpus-honesty` (registration-context excerpt mode, obligation label), `three-stage-scan` (one more proposer in MAP, new rungs below proof).
- **Depends**: `reachability-flow-model` (entry point to operation paths); `guard-dominance-regime` (a guard dominates the path). Reachability is now on the critical path for two consumers.
- Extends: `corpus-seeded-mechanisms` (second shape family, same store, same lever contract), `pair-corpus-honesty` (registration-context excerpt mode, obligation label), `three-stage-scan` (one more proposer in MAP, new rungs below proof).
- Depends: `reachability-flow-model` (entry point to operation paths); `guard-dominance-regime` (a guard dominates the path). Reachability is now on the critical path for two consumers.
## Constraints
@@ -152,9 +152,9 @@ tests/
### Modified Files
- `src/openultrasast/cli.py` — `_check_obligations` after `_search_variants` in MAP (standard/deep); manifest `obligations` block; `mechanisms export` writes obligation rows too.
- `src/openultrasast/pairs.py` — `ExpectedFinding.obligation: str | None`; `_detections`-side additive rule: for a row with `obligation`, a finding with tag `obligation:<kind>` inside the labeled function counts.
- `src/openultrasast/semantic/mechanisms.py` — no schema change; `corpus_mechanisms()` unchanged; a helper `obligation_mechanisms(records)` selects `shape["family"] == "obligation"` (added in `obligations/shapes.py`, not in the store module).
- `src/openultrasast/cli.py`: `_check_obligations` after `_search_variants` in MAP (standard/deep); manifest `obligations` block; `mechanisms export` writes obligation rows too.
- `src/openultrasast/pairs.py`: `ExpectedFinding.obligation: str | None`; `_detections`-side additive rule: for a row with `obligation`, a finding with tag `obligation:<kind>` inside the labeled function counts.
- `src/openultrasast/semantic/mechanisms.py`: no schema change; `corpus_mechanisms()` unchanged; a helper `obligation_mechanisms(records)` selects `shape["family"] == "obligation"` (added in `obligations/shapes.py`, not in the store module).
- `src/openultrasast/reports.py`, `rank.py`, `config.py`, `improve/*`, `semantic/seed.py`, `semantic/loo.py`, `benchmarks/pairs/harvest.py`, `benchmarks/pairs/mechanisms.toml` as listed above.
## System Flows
@@ -177,7 +177,7 @@ flowchart LR
I --> J[StaticFinding suspicion; tags obligation:*, discharger:*, obligation_evidence:*]
```
Gating: the checker emits a finding only when (1) the operation is reached — from an entry point along a `PathRecord`, or, without paths, when the operation sits in a handler that is itself an entry point — and (2) no discharger of the required kind dominates, and (3) the obligation is justified by a sibling set (`consistency_violation`), a policy clause (`declared_policy_violation`) or, in degraded mode without either, by the operation fact alone (`function_local`). A declared public route suppresses path-guard obligations for that route only.
Gating: the checker emits a finding only when (1) the operation is reached (from an entry point along a `PathRecord`, or, without paths, when the operation sits in a handler that is itself an entry point) and (2) no discharger of the required kind dominates, and (3) the obligation is justified by a sibling set (`consistency_violation`), a policy clause (`declared_policy_violation`) or, in degraded mode without either, by the operation fact alone (`function_local`). A declared public route suppresses path-guard obligations for that route only.
### Teaching and measuring
@@ -8,9 +8,9 @@ AppSec engineers and maintainers of OpenUltraSAST scan web code, increasingly vi
## Boundary Context
- **In scope**: obligated operations and discharger kinds as closed facts data; obligation shapes learned from trusted pairs; consistency detection across sibling handlers; an optional declared policy file with a closed schema; path-aware violation evaluation with a function-local degraded mode; obligation evidence rungs, report wording and worth-fixing weighting; a corpus recipe shape that keeps the handler's registration context; leave-one-out per obligation kind; an `obligations` improve lever; offline tests.
- **Out of scope**: flow taint, walker or sink facts; the call graph, path records and dominance computation (owned by `reachability-flow-model` and `guard-dominance-regime`); model-authored policy or shapes; sandbox proof of absence bugs; any merge gate; vendoring unlicensed code.
- **Adjacent expectations**: the entry-point mapper keeps providing routes with an access level and their decorators; the reachability spec provides entry-point-to-operation paths and the guard-dominance spec provides "a guard dominates the path"; the mechanism store, its deterministic ids and the lever contract from `corpus-seeded-mechanisms` are reused unchanged; the pair scorer accepts one additive label field; `local` remains the only slice that can fail CI.
- In scope: obligated operations and discharger kinds as closed facts data; obligation shapes learned from trusted pairs; consistency detection across sibling handlers; an optional declared policy file with a closed schema; path-aware violation evaluation with a function-local degraded mode; obligation evidence rungs, report wording and worth-fixing weighting; a corpus recipe shape that keeps the handler's registration context; leave-one-out per obligation kind; an `obligations` improve lever; offline tests.
- Out of scope: flow taint, walker or sink facts; the call graph, path records and dominance computation (owned by `reachability-flow-model` and `guard-dominance-regime`); model-authored policy or shapes; sandbox proof of absence bugs; any merge gate; vendoring unlicensed code.
- Adjacent expectations: the entry-point mapper keeps providing routes with an access level and their decorators; the reachability spec provides entry-point-to-operation paths and the guard-dominance spec provides "a guard dominates the path"; the mechanism store, its deterministic ids and the lever contract from `corpus-seeded-mechanisms` are reused unchanged; the pair scorer accepts one additive label field; `local` remains the only slice that can fail CI.
## Requirements
@@ -2,9 +2,9 @@
## Summary
- **Feature**: `authorization-obligations`
- **Discovery Scope**: Extension of the semantic layer (facts, shapes, store, MAP proposers), the entry-point mapper, reports, the pair corpus, leave-one-out and the improve lever; complex integration because it consumes two unimplemented upstream specs.
- **Key Findings**:
- Feature: `authorization-obligations`
- Discovery Scope: Extension of the semantic layer (facts, shapes, store, MAP proposers), the entry-point mapper, reports, the pair corpus, leave-one-out and the improve lever; complex integration because it consumes two unimplemented upstream specs.
- Key Findings:
- The absence pairs are real and structured. `vampi-books-get-by-title` has identical code on both sides except a constraint `user=user` bound from `token_validator(request.headers.get('Authorization'))['sub']`; the vulnerable branch queries `Book` by title alone. The difference is the provenance of a constraint, not a flow. 15 of 29 `agent-vfc` rows and 6 of the 19 non-teaching `vibe-py` pairs carry absence labels (`missing_auth_guard`, `permissive_default`, `identity_from_request_body`, `secret_in_client`, `validation_strength`).
- The entry-point mapper already classifies every route as `public`, `authenticated` or `role-restricted` from its decorators (`mapping._python_route_access`), records the decorators as `access_evidence`, and lists `conditions` (`if` tests that look like checks). That is the sibling detector's upstream for path guards and needs no change.
- `FileIR` binds carry `value_text`, `names` and `call_name`; `variants.source_kind_of_name` already traces a name through binds to a fact source, a parameter or a container read. Provenance of a constraint's value (authenticated context versus request input versus constant) is the same trace with one more source class.
@@ -17,29 +17,29 @@
### What the absence pairs actually differ by
- **Context**: Decide whether absence bugs can be a shape family at all.
- **Sources Consulted**: `benchmarks/pairs/vibe-py` excerpts for `missing_auth_guard` and `permissive_default` rows; `benchmarks/pairs/agent-vfc/recipes.toml` mechanisms; `mechanisms.toml` definitions.
- **Findings**: three recurring differences: (a) a constraint added to a data access whose value comes from the authenticated context; (b) a guard call or decorator added before the operation (early return on a failed token check, `login_required`); (c) a literal changed from permissive to restrictive (`CORS(app)` → origins list, `DEBUG=True` → `False`, RLS enabled). A fourth, rarer: identity read from the body replaced by identity from the token (same as (a) seen from the other side).
- **Implications**: discharger kinds `identity_constraint`, `path_guard`, `ownership_check`, `non_permissive_value`, `validated_input`; the value provenance is the discriminating feature for (a) and (d).
- Context: Decide whether absence bugs can be a shape family at all.
- Sources Consulted: `benchmarks/pairs/vibe-py` excerpts for `missing_auth_guard` and `permissive_default` rows; `benchmarks/pairs/agent-vfc/recipes.toml` mechanisms; `mechanisms.toml` definitions.
- Findings: three recurring differences: (a) a constraint added to a data access whose value comes from the authenticated context; (b) a guard call or decorator added before the operation (early return on a failed token check, `login_required`); (c) a literal changed from permissive to restrictive (`CORS(app)` → origins list, `DEBUG=True` → `False`, RLS enabled). A fourth, rarer: identity read from the body replaced by identity from the token (same as (a) seen from the other side).
- Implications: discharger kinds `identity_constraint`, `path_guard`, `ownership_check`, `non_permissive_value`, `validated_input`; the value provenance is the discriminating feature for (a) and (d).
### Where the policy comes from
- **Context**: Absence is relative to intent; the tool must not invent intent.
- **Sources Consulted**: `mapping.analyze_entry_points` (routes, access level, evidence), Real-Vuln repositories (many handlers per resource), Clearwing's sibling-comparison practice.
- **Findings**: in every Real-Vuln web app the handlers on one resource share a router and a model; the vulnerable handler is the odd one out. Route access classification already exists. A declared policy is the only way to say "this route is meant to be public"; the hunter can only guess it.
- **Implications**: consistency first (no declaration needed, measurable now), declared policy second (stronger rung), hunter last (records intent, never raises).
- Context: Absence is relative to intent; the tool must not invent intent.
- Sources Consulted: `mapping.analyze_entry_points` (routes, access level, evidence), Real-Vuln repositories (many handlers per resource), Clearwing's sibling-comparison practice.
- Findings: in every Real-Vuln web app the handlers on one resource share a router and a model; the vulnerable handler is the odd one out. Route access classification already exists. A declared policy is the only way to say "this route is meant to be public"; the hunter can only guess it.
- Implications: consistency first (no declaration needed, measurable now), declared policy second (stronger rung), hunter last (records intent, never raises).
### Dominance before guard-dominance-regime exists
- **Context**: Req 5 needs "a discharger dominates the path"; the spec that owns dominance is not written.
- **Findings**: within one handler, the existing `FunctionIR` order of binds and calls gives statement order; a guard call followed by an early return before the operation line is dominance in the common case; decorators dominate the whole handler; middleware attached to the router dominates every route under it. Across calls, `PathRecord.hops` gives the sequence of nodes; a discharger on any hop before the operation node dominates the path when the hop's function returns early on failure.
- **Implications**: define `Dominance` as a narrow protocol (`dominates(witness, operation) -> bool`) with a built-in `OrderDominance` implementation (statement order in the handler, decorators, router middleware, hop order) that `guard-dominance-regime` may replace with nesting-aware dominance. The degraded mode (`function_local`) is `OrderDominance` without path records.
- Context: Req 5 needs "a discharger dominates the path"; the spec that owns dominance is not written.
- Findings: within one handler, the existing `FunctionIR` order of binds and calls gives statement order; a guard call followed by an early return before the operation line is dominance in the common case; decorators dominate the whole handler; middleware attached to the router dominates every route under it. Across calls, `PathRecord.hops` gives the sequence of nodes; a discharger on any hop before the operation node dominates the path when the hop's function returns early on failure.
- Implications: define `Dominance` as a narrow protocol (`dominates(witness, operation) -> bool`) with a built-in `OrderDominance` implementation (statement order in the handler, decorators, router middleware, hop order) that `guard-dominance-regime` may replace with nesting-aware dominance. The degraded mode (`function_local`) is `OrderDominance` without path records.
### Store and lever reuse
- **Context**: Avoid a second store and a second lever machinery.
- **Findings**: `Mechanism.shape` is a dict validated where used; `corpus_mechanism_id` hashes `shape.key()`; `validate_mechanism` checks guard and source kinds against `variants` constants. A second family needs its own closed kinds check and a `family` field in the shape dict; the tombstone, snapshot and profile clause need nothing.
- **Implications**: `ObligationShape.to_dict()` carries `family = "obligation"`; the lever validator dispatches on `family`; `search_tree` ignores obligation rows (they are not sink shapes) and the obligation checker ignores sink rows.
- Context: Avoid a second store and a second lever machinery.
- Findings: `Mechanism.shape` is a dict validated where used; `corpus_mechanism_id` hashes `shape.key()`; `validate_mechanism` checks guard and source kinds against `variants` constants. A second family needs its own closed kinds check and a `family` field in the shape dict; the tombstone, snapshot and profile clause need nothing.
- Implications: `ObligationShape.to_dict()` carries `family = "obligation"`; the lever validator dispatches on `family`; `search_tree` ignores obligation rows (they are not sink shapes) and the obligation checker ignores sink rows.
## Architecture Pattern Evaluation
@@ -54,23 +54,23 @@
### Decision: Obligation findings are `suspicion` with a data label, not new ladder rungs
- **Context**: Req 6.1 names three labels; the ladder is closed and owned elsewhere.
- **Selected Approach**: `StaticFinding.evidence_level = "suspicion"`, tags `obligation:<kind>`, `discharger:<kind>`, `obligation_evidence:<label>`; reports render the label. Dispositions and `ALLOWED_EVIDENCE` unchanged.
- **Trade-offs**: SARIF consumers see `suspicion` plus properties; honest and non-invasive.
- Context: Req 6.1 names three labels; the ladder is closed and owned elsewhere.
- Selected Approach: `StaticFinding.evidence_level = "suspicion"`, tags `obligation:<kind>`, `discharger:<kind>`, `obligation_evidence:<label>`; reports render the label. Dispositions and `ALLOWED_EVIDENCE` unchanged.
- Trade-offs: SARIF consumers see `suspicion` plus properties; honest and non-invasive.
### Decision: Dominance is a protocol with an order-based default
- **Context**: `guard-dominance-regime` is unwritten; this spec cannot wait for it to be useful.
- **Selected Approach**: `Dominance` protocol; `OrderDominance` default (decorator, router middleware, statement order before the operation, hop order along a path). The guard spec replaces it without touching callers.
- **Trade-offs**: order-based dominance misses guards inside conditionals; recorded as the known coarseness and measured on the fixed twins.
- Context: `guard-dominance-regime` is unwritten; this spec cannot wait for it to be useful.
- Selected Approach: `Dominance` protocol; `OrderDominance` default (decorator, router middleware, statement order before the operation, hop order along a path). The guard spec replaces it without touching callers.
- Trade-offs: order-based dominance misses guards inside conditionals; recorded as the known coarseness and measured on the fixed twins.
### Decision: Policy is never written by the tool
- **Selected Approach**: `declared policy` is a hand-maintained file with a closed schema; the lever may only *propose* a clause, journaled as pending until a human accepts; scans never read pending clauses.
- Selected Approach: `declared policy` is a hand-maintained file with a closed schema; the lever may only *propose* a clause, journaled as pending until a human accepts; scans never read pending clauses.
### Decision: Resource identity for sibling grouping
- **Selected Approach**: a handler's resource is the trailing segment of its route path with parameters stripped, joined with the operation's target token (model, table, collection name from the operation fact match); router or module is the second key. Coarse but deterministic and text-free.
- Selected Approach: a handler's resource is the trailing segment of its route path with parameters stripped, joined with the operation's target token (model, table, collection name from the operation fact match); router or module is the second key. Coarse but deterministic and text-free.
## Risks & Mitigations
+62 -62
View File
@@ -2,7 +2,7 @@
## Problem
`learning-harness` built the measurement layer honestly — a closed family taxonomy, a class-aware pair
`learning-harness` built an honest measurement layer with a closed family taxonomy, a class-aware pair
scorer, per-family detector configurations, a split enforced by construction, verifier-gated reporting,
and a journal with attribution. What it did **not** build is a search. `run_learning_round` is a
hand-rolled hill-climber: one proposal, one minibatch, one holdout check, a threshold rule, accept or
@@ -28,26 +28,26 @@ Round zero on vibe-py, `deepseek-v4-flash`, K = 5, $0.69, 23/30 pair-correct:
Four defects follow from the loop's shape, not from its bugs:
1. **The acceptance rule cannot discriminate.** `decide` compares raw deltas: any regression on train or
1. The acceptance rule cannot discriminate. `decide` compares raw deltas: any regression on train or
holdout rejects, a tie rejects. The measured floor says 11 of 20 injection pairs disagree with
*themselves* across five runs, so a delta over a minibatch of 8 is mostly noise. The spec built
`sign_test` and `reliable_change` for exactly this and the rule calls neither.
2. **The holdout is an input to the decision.** `decide` reads `holdout_delta`. Req 5.4 keeps the
2. The holdout is an input to the decision. `decide` reads `holdout_delta`. Req 5.4 keeps the
*proposer* blind to the holdout, and then the *selector* reads it every round. Repeated rounds
selecting on holdout performance is search-time overfitting of the test set — the same leak class
selecting on holdout performance is search-time overfitting of the test set, the same leak class
this project has already measured once, one level up.
3. **Noise is fought with repetition instead of diversity.** K = 5 spends five rollouts per pair per
3. Noise is fought with repetition instead of diversity. K = 5 spends five rollouts per pair per
side estimating variance. The published alternative spends that budget on more candidates and lets a
frontier keep whatever wins somewhere.
4. **The proposer is told a score, not what went wrong.** `FailureFacts` carries outcomes, run strings
4. The proposer is told a score, not what went wrong. `FailureFacts` carries outcomes, run strings
and counts. The single largest lever in the published work is textual feedback from the *evaluation
trace* — and our evaluator produces exactly the right material (which finding landed on which line,
trace*, and our evaluator produces exactly the right material (which finding landed on which line,
whether it fell inside the labeled span, whether the fixed side leaked, which family was claimed)
and then throws all of it away.
## What others built
**GEPA — Reflective Prompt Evolution (Agrawal et al., arXiv:2507.19457, ICLR 2026 oral).** Algorithm 1,
**GEPA, Reflective Prompt Evolution (Agrawal et al., arXiv:2507.19457, ICLR 2026 oral).** Algorithm 1,
read from the paper rather than from a summary:
- Split `D_train` into `D_feedback` and `D_pareto`. Initialise the candidate pool `P = [Φ]` and score
@@ -56,7 +56,7 @@ read from the paper rather than from a summary:
minibatch drawn from `D_feedback`; collect scores, traces and **textual feedback** via a feedback
function `μ_f`; have a reflection LM rewrite that module's prompt; re-run the mutated system on the
*same* minibatch.
- **Accept if and only if the minibatch average improved.** Only then is the candidate scored on all of
- Accept if and only if the minibatch average improved. Only then is the candidate scored on all of
`D_pareto` and added to the pool with its parent recorded.
- Return the candidate with the best average on `D_pareto`.
@@ -65,7 +65,7 @@ candidates that achieve it; drop strictly dominated ones; sample a survivor with
proportional to the number of instances it leads on.
Three things about that are directly relevant to us. The acceptance gate is **cheap and noisy on
purpose** — a single minibatch comparison — because the frontier, not the gate, is what protects
purpose** (a single minibatch comparison) because the frontier, not the gate, is what protects
against a bad accept: anything that wins on one instance is retained. Selection is **stochastic and
diversity-preserving**, which is what stops the search collapsing onto one lineage. And the expensive
full evaluation is spent only on candidates that already passed the cheap gate.
@@ -76,20 +76,20 @@ GRPO's 24,000. Sample efficiency is the claim that matters for us, because our r
**DSPy's optimizer interface.** `MIPROv2` proposes instruction and demonstration candidates and searches
them with Bayesian optimisation (Optuna TPE), scoring on minibatches and running a full validation pass
on the incumbent every `minibatch_full_eval_steps` — an explicit two-fidelity design for a noisy metric.
on the incumbent every `minibatch_full_eval_steps`, an explicit two-fidelity design for a noisy metric.
`dspy.GEPA` takes `metric`, `reflection_lm`, a budget (`auto` / `max_full_evals` / `max_metric_calls`),
`reflection_minibatch_size`, `candidate_selection_strategy` (`pareto` | `current_best`) and
`track_stats`, and its metric returns `Prediction(score, feedback)` rather than a float.
**SIMBA — Stochastic Introspective Mini-Batch Ascent (`dspy.SIMBA`).** Two of its ideas fit our corpus
**SIMBA, Stochastic Introspective Mini-Batch Ascent (`dspy.SIMBA`).** Two of its ideas fit our corpus
better than GEPA's do. It runs `num_candidates` (6 by default) program variants per example, buckets the
trajectories, and ranks examples by **max-to-min score gap** — deliberately spending the budget on the
trajectories, and ranks examples by **max-to-min score gap**, deliberately spending the budget on the
instances where variants disagree most. On our corpus that is not a nuisance, it is the training signal:
the 11 injection pairs that disagree with themselves are precisely the pairs where a rule change can flip
an outcome, and uniform minibatch sampling would spend most of its budget on the 9 that never move. It
also alternates two strategies rather than round-robining blindly — `append_a_demo` (a successful
also alternates two strategies rather than round-robining blindly, `append_a_demo` (a successful
trajectory becomes a demonstration) and `append_a_rule` (the model introspects on the bucket and writes a
rule) — which is exactly the split our levers already have between `counterexamples`/`hard_negatives` and
rule), which is exactly the split our levers already have between `counterexamples`/`hard_negatives` and
`checklist`/`prompt`. Candidates are sampled by softmax over historical scores at temperature 0.2, a
cheaper third option beside GEPA's `pareto` and `current_best`.
@@ -97,16 +97,16 @@ cheaper third option beside GEPA's `pareto` and `current_best`.
produce. It shows the reflector the current instruction, then "inputs, outputs and feedback" for the
minibatch, and asks it to infer the task, extract "all niche and domain specific factual information ...
as a lot of it may not be available to the assistant in the future", and name any generalizable strategy.
That is a specification for `μ_f`, not just a prompt: feedback has to carry the domain facts a detector
cannot re-derive — which sink, which span, which side leaked — rather than a verdict.
That is a prompt and a specification for `μ_f`: feedback has to carry the domain facts a detector
cannot re-derive (which sink, which span, which side leaked) rather than a verdict.
**GEPA's merge** is gated strictly: two candidates are crossed only when they share a common ancestor,
optimised *disjoint* sets of prompts, are both Pareto-optimal, and both beat the ancestor's aggregate.
Worth having once we have per-lever candidates, and worth not having before then.
**The standalone `gepa` library** (MIT) exposes `gepa.optimize(seed_candidate, trainset, valset,
task_lm, reflection_lm, max_metric_calls)` over a `GEPAAdapter` protocol of three methods —
`evaluate`, `make_reflective_dataset`, `propose_new_texts` — with a seed candidate that is just a dict
task_lm, reflection_lm, max_metric_calls)` over a `GEPAAdapter` protocol of three methods
(`evaluate`, `make_reflective_dataset`, `propose_new_texts`), with a seed candidate that is just a dict
of component name to text. That adapter shape is the interface we want whether or not we take the
dependency: it is the smallest surface that separates *what a detector is* from *how it is searched*.
@@ -114,26 +114,26 @@ dependency: it is the smallest surface that separates *what a detector is* from
Restate the detector as an optimisable program and put a published optimiser behind it.
- **A detector is a candidate**: a dict of named text components (`prompt`, `checklist`,
- A detector is a candidate: a dict of named text components (`prompt`, `checklist`,
`hard_negatives`, `counterexamples`) plus its frozen non-text settings (tools, budgets). That is
already what `FamilyConfig` holds; it becomes the seed candidate.
- **The metric returns a score and a reason.** `score_pair_family` gains a feedback string built from
- The metric returns a score and a reason. `score_pair_family` gains a feedback string built from
what the evaluation already knows. This is the highest-value change in the spec and it is independent
of which optimiser sits on top.
- **`D_pareto` is carved out of the train split.** The holdout leaves the acceptance rule entirely and
- `D_pareto` is carved out of the train split. The holdout leaves the acceptance rule entirely and
is scored once, for reporting, when the search is over. This is a requirements change to Req 9.5 and
the most important honesty consequence of the redesign.
- **An adapter protocol** (`evaluate`, `make_reflective_dataset`, `propose_new_texts`) with two backends:
- An adapter protocol (`evaluate`, `make_reflective_dataset`, `propose_new_texts`) with two backends:
our own implementation of Algorithms 1 and 2 by default, so the core install keeps
`dependencies = []`, and `gepa` behind an optional extra so its improvements arrive for free.
- **The Pareto pool replaces the accept/revert tree.** `Archive` already records per-pair winners; it
- The Pareto pool replaces the accept/revert tree. `Archive` already records per-pair winners; it
becomes the frontier, and the journal's parent records become the ancestry GEPA needs.
## Where the noise actually is, and why the architecture may be wrong
The maintainer's objection, and I think it is correct: the harness treats the detector as a black box
emitting a noisy scalar, and everything follows from that — K runs, majority vote, a measured floor, a
regression budget, a sign test. That is the stance you take when you cannot see inside the thing you are
emitting a noisy scalar, and everything follows from that (K runs, majority vote, a measured floor, a
regression budget, a sign test). That is the stance you take when you cannot see inside the thing you are
measuring. We can see inside it.
The variance is not in the judgement. `temperature: 0` is sent and thinking is disabled, and 11 of 20
@@ -147,8 +147,8 @@ than the symptom.
**Constraints as predicates, enforced at generation time** (`dspy.Assert`/`Suggest`, arXiv:2312.13382).
A constraint is a boolean function of the output; on failure the past output *and the error* are injected
into a retry. We already have the predicates — is the reported line inside a parsed function, does the
named sink occur at that line, is the claimed family in the taxonomy, did the run cite a tool result —
into a retry. We already have the predicates (is the reported line inside a parsed function, does the
named sink occur at that line, is the claimed family in the taxonomy, did the run cite a tool result),
and we apply every one of them **after the fact, at scoring time, to decide whether to count a finding**.
Applying them at generation time to decide whether to *accept* one is free, needs no dependency, and is
the single highest-value change available. The paper reports near-total intrinsic constraint satisfaction
@@ -156,10 +156,10 @@ and 5–15% downstream gains.
**Typed decomposition instead of one free-running loop.** Our detector is a single ReAct loop that decides
which files to read, what to grep, when to stop and what to report; every one of those is a variance
source. DSPy's thesis is that each step gets a signature — a typed input/output contract — and a little
source. DSPy's thesis is that each step gets a signature (a typed input/output contract), and a little
control flow composes them. Ours would be roughly: locate candidate sink sites; trace whether an untrusted
value reaches each; check for a guard between source and sink; judge. **Several of those steps need no
model at all** — the CST, the entry-point mapper, `flows` and `obligations` already answer them
model at all**. The CST, the entry-point mapper, `flows` and `obligations` already answer them
deterministically, and we are currently paying a model to re-derive facts we hold.
**Best-of-N against a checkable reward, instead of a majority vote** (`dspy.Refine`, `BestOfN`). Both
@@ -170,7 +170,7 @@ have checkable rewards for free. The same budget, used as evidence rather than a
**Why this matters more for our models, not less.** GEPA and SIMBA are demonstrated with a strong
reflection model over a weaker task model, and their gains come from evolving instructions. A cheap model
fails on an under-specified task and does well on a narrow typed one. So the cheaper the detector model,
the more of the return sits in decomposition and constraints and the less in prompt evolution — which is
the more of the return sits in decomposition and constraints and the less in prompt evolution, which is
the opposite of the order this spec originally proposed.
### The experiment, run (2026-09-06, `benchmarks/measurements/2026-09-06-trajectory-divergence.json`)
@@ -178,7 +178,7 @@ the opposite of the order this spec originally proposed.
Twelve injection pairs, both sides, K = 5, temperature 0, thinking disabled, recording the tool-call
sequence of every run. $0.23.
**The intended statistic could not be computed. Across 120 runs there was not one repeated trajectory** —
**The intended statistic could not be computed. Across 120 runs there was not one repeated trajectory**. There were
4.67 distinct trajectories per vulnerable side and 4.92 per fixed side, out of 5, and 78 distinct tool-*name*
sequences ignoring arguments entirely. The detector is not sampling one measurement five times. It is running
five different investigations and voting on the results.
@@ -193,21 +193,21 @@ Asked as a gradient instead, the relationship is monotone and clean:
Three more facts from the same run:
- **The variance is a search problem, and the search is `grep_repo`.** 555 grep calls against 181 `read_file`,
- The variance is a search problem, and the search is `grep_repo`. 555 grep calls against 181 `read_file`,
71 `entry_points`, 43 `find_refs`, 13 `read_definition`. The model spends its budget guessing patterns to
locate a sink — while the CST, the entry-point mapper, `flows` and `obligations` already know where the
locate a sink, while the CST, the entry-point mapper, `flows` and `obligations` already know where the
sinks are.
- **All of the instability is on the vulnerable side.** Every one of the twelve pairs produced exactly one
- All of the instability is on the vulnerable side. Every one of the twelve pairs produced exactly one
distinct finding set on the fixed side. The detector is reliably silent on the fix and unreliably vocal on
the bug, so the noise is entirely in detection and none of it in false positives.
- **Findings converge far more than trajectories do.** 78 distinct trajectories, but one to three distinct
- Findings converge far more than trajectories do. 78 distinct trajectories, but one to three distinct
finding sets per pair. Many different investigations reach the same answer; six of twelve pairs are stable
despite never once repeating a trajectory.
**What this does to the noise architecture.** The floor is not measuring the detector's uncertainty about a
question. It is measuring how often an unconstrained search happens to reach the sink. A majority vote over
five runs is therefore not variance reduction — it is an unweighted ensemble of five arbitrary search
procedures — and a regression budget derived from it is a budget for the search's luck. K runs, the floor,
five runs is therefore an unweighted ensemble of five arbitrary search
procedures rather than variance reduction, and a regression budget derived from it is a budget for the search's luck. K runs, the floor,
the budget and the sign test are all statistical treatment for a structural problem, and the 0% and 4.2%
rows of that table say what happens when the structure is fixed instead.
@@ -224,45 +224,45 @@ answer "is our own instability structural?" in twenty minutes is the thing that
**Was answering the wrong question.**
- **Req 3.5, K ≥ 3 runs per pair.** The premise is variance reduction over a repeated measurement. There is
- Req 3.5, K ≥ 3 runs per pair. The premise is variance reduction over a repeated measurement. There is
no repeated measurement: 120 runs, 120 trajectories. K runs is an *ensemble of distinct search
procedures*, and a majority vote over it is an unweighted ensemble, not an estimate. It should either be
renamed and published as an ensemble — which changes what every family number means — or dropped once the
renamed and published as an ensemble, which changes what every family number means, or dropped once the
search is constrained.
- **Req 8.2, the noise floor as the regression budget.** The floor measures how often an unconstrained
- Req 8.2, the noise floor as the regression budget. The floor measures how often an unconstrained
search happens to reach the sink. Deriving a regression budget from it means the worse the search wanders,
the more collateral damage a round is licensed to do. That is backwards, and it is why injection's budget
is 11 of 20.
- **Req 9.5, the acceptance rule.** Already flagged for reading the holdout every round. Now also: it
- Req 9.5, the acceptance rule. Already flagged for reading the holdout every round. Now also: it
compares deltas between two ensemble votes, and calls the difference an improvement.
- **The sign test and `reliable_change`.** Added this session in good faith; they test whether an ensemble's
- The sign test and `reliable_change`. Added this session in good faith; they test whether an ensemble's
vote moved, which is not the question either.
**What the data adds that neither spec anticipated.**
- **The instability is entirely one-sided.** All twelve pairs produced exactly one finding set on the fixed
- The instability is entirely one-sided. All twelve pairs produced exactly one finding set on the fixed
side. Specificity is solid and recall is the noisy half, so the leak ceiling and the false-positive
machinery are sound and it is the recall numbers that carry the caveat.
- **`grep_repo` is the variance engine** — 555 calls against 181 `read_file` and 71 `entry_points`. The
- `grep_repo` is the variance engine. It accounts for 555 calls against 181 `read_file` and 71 `entry_points`. The
single highest-leverage change in either spec is not a prompt, a lever or an optimiser: **stop asking the
model to locate the sink.** The CST, the entry-point mapper, `flows` and `obligations` already know. Give
the detector the candidate sites and ask it the question only a model can answer — does untrusted input
the detector the candidate sites and ask it the question only a model can answer. Does untrusted input
reach this one, and is there a guard.
- **The loop is partly self-correcting**, so this should be a large win rather than a marginal one: 78
- The loop is partly self-correcting, so this should be a large win rather than a marginal one: 78
distinct trajectories collapse to one to three finding sets per pair, and half the pairs are already
stable despite never repeating an investigation.
**Limits of the experiment, stated so the conclusion is not over-read.** Twelve pairs, one family, one
slice, one model. The high-similarity bucket holds only two run pairs, so its 0% is decoration; the
defensible number is the mid bucket's 4.2% over 24 run pairs against 30.9% over 94. And the direction of
causation is inferred, not established — similar trajectories may be a symptom of easy pairs rather than a
causation is inferred, not established. Similar trajectories may be a symptom of easy pairs rather than a
cause of agreement. The way to settle that is the constrained detector itself: fix the investigation, and
if the floor does not collapse, this reading was wrong.
## The constraint that outranks the algorithm
Both algorithms assume a dataset. GEPA splits `D_train` into `D_feedback` and `D_pareto`; SIMBA's default
batch size is 32. **Our injection family has 12 train teachers** — fewer than one SIMBA batch, and after a
batch size is 32. **Our injection family has 12 train teachers** (fewer than one SIMBA batch), and after a
GEPA split roughly six instances on each side. Every other family has one to three.
So the honest statement of what this spec can and cannot buy: GEPA and SIMBA are sample-efficient in
@@ -279,42 +279,42 @@ selection, the budget accounting, and the artifacts that make a run reproducible
Out, and deliberately: the taxonomy, the classifier, the pair scorer's *outcome* rules, the verifier
boundary, the write roots, redaction, the three gates. Those are the measurement, and a search whose
objective is not trustworthy is worse than no search. Nothing here relaxes them — in particular the
objective is not trustworthy is worse than no search. Nothing here relaxes them. In particular, the
proposer stays blind to the holdout, verifier code and gold labels stay outside every write root, and
no pattern or policy is written by a model.
## Decisions for the maintainer
0. **Is the optimiser even the next thing to build?** Proposed no, and this reverses the order above.
0. Is the optimiser even the next thing to build? Proposed no, and this reverses the order above.
Constraints at generation time, typed decomposition and best-of-N attack the variance at its source; an
optimiser only tunes text around a program whose freedom is the problem. Proposed sequence: measure where
the variance is, constrain generation, decompose the loop, then optimise — and at a trainset of twelve the
the variance is, constrain generation, decompose the loop, then optimise, and at a trainset of twelve the
optimiser that fits is `BootstrapFewShot` (collect traces that scored well, install them as demonstrations
in a lever we already have) rather than an evolutionary search with nothing to search over.
0b. **Adopt DSPy itself?** Proposed no. Our detector is not a DSPy program — it is an agentic tool loop over
0b. Adopt DSPy itself? Proposed no. Our detector is not a DSPy program. It is an agentic tool loop over
a repository with curated tools, redaction on every prompt, path clamping, a "no findings unless a tool
was called" rule and a final-answer turn, several of which are security properties rather than
conveniences. Handing that loop to `dspy.ReAct` and the calls to litellm would put them behind someone
else's abstraction, and DSPy could only ever be an optional extra under `dependencies = []`, which means
two code paths for the thing that *is* the product. What DSPy would genuinely buy — signatures, adapters,
multi-module composition — we barely need at one module per family. The optimisers are available without
two code paths for the thing that *is* the product. What DSPy would buy (signatures, adapters,
multi-module composition), we barely need at one module per family. The optimisers are available without
the framework: `gepa` ships standalone under MIT, and SIMBA's two ideas are a hundred lines of logic.
Proposed instead: take the interface shape and the algorithms, not the framework.
1. **Reimplement Algorithms 1 and 2, take `gepa` as an extra, or both?** Both is proposed: ours by
1. Reimplement Algorithms 1 and 2, take `gepa` as an extra, or both? Both is proposed: ours by
default (zero-dependency, and the security properties are ours to own), `gepa` behind an extra for
comparison. The cost is one adapter and one conformance test that both backends must pass.
2. **Drop the holdout from the acceptance rule?** Proposed yes, and it amends approved Req 9.5. The
2. Drop the holdout from the acceptance rule? Proposed yes, and it amends approved Req 9.5. The
holdout becomes what its name says.
3. **Sample instances by disagreement (SIMBA) or uniformly (GEPA)?** Proposed SIMBA's ranking. Our measured
3. Sample instances by disagreement (SIMBA) or uniformly (GEPA)? Proposed SIMBA's ranking. Our measured
floor makes it the better fit, and it turns the noise floor from an obstacle into an instance sampler.
4. **K = 1 inside the search, K ≥ 3 only for published numbers?** Proposed yes. That is roughly a 5×
4. K = 1 inside the search, K ≥ 3 only for published numbers? Proposed yes. That is roughly a 5×
budget reclaim spent on more candidates, which is where the sample efficiency comes from.
5. **Does the reflection model differ from the detector model?** The published work uses a stronger
5. Does the reflection model differ from the detector model? The published work uses a stronger
reflector than the task model. We have `deepseek-v4-pro` configured as the judge; proposed to reuse it.
6. **Is `access_control` still the family we care most about?** It has one train teacher. Until agent-vfc
6. Is `access_control` still the family we care most about? It has one train teacher. Until agent-vfc
rows are reviewed above tier `title`, the optimiser cannot work on it, whatever its algorithm.
## Open questions
@@ -322,7 +322,7 @@ no pattern or policy is written by a model.
- Does `gepa` run against an OpenAI-shaped base URL that is not OpenAI (our DeepSeek endpoint), and what
does it pull in? Both must be checked before the extra is offered, under the no-undeclared-dependency rule.
- Our "module" granularity: is a module a family, or a lever within a family? Round-robin over levers is
the closer analogue, but the levers are not independent — a checklist edit and a prompt edit interact.
the closer analogue, but the levers are not independent. A checklist edit and a prompt edit interact.
- The feedback function's failure mode is a reflection model that writes plausible prose about a trace it
misread. What check makes a feedback string answerable rather than persuasive?
- GEPA's acceptance is a single noisy comparison. Our floor says a single comparison is close to a coin
+16 -16
View File
@@ -29,10 +29,10 @@ already reaches, while the **semantic IR caps it at 96.6%** with a median of ten
### This Spec Owns
- `learning/candidates.py` — what a candidate is, how the IR enumerates one per family shape, and the ceiling report.
- `learning/judgments.py` — the typed answer, and the toolless call that produces it.
- `learning/predicates.py` — the checks that constrain a claim at generation time and score it afterwards, defined once.
- `learning/slices.py` — deciding versus development slices, the overfitting gap, and the cross-slice rejection rule.
- `learning/candidates.py`, what a candidate is, how the IR enumerates one per family shape, and the ceiling report.
- `learning/judgments.py`, the typed answer, and the toolless call that produces it.
- `learning/predicates.py`, the checks that constrain a claim at generation time and score it afterwards, defined once.
- `learning/slices.py`, deciding versus development slices, the overfitting gap, and the cross-slice rejection rule.
- `run_constrained_detector` in `learning/detectors.py`, beside the existing `run_family_detector`.
- `ousast learning candidates`, and a detector selector on `score` and `baseline`.
@@ -113,16 +113,16 @@ src/openultrasast/learning/
### Modified Files
- `src/openultrasast/learning/detectors.py` — `run_constrained_detector(root, region, config, *, client, model,
- `src/openultrasast/learning/detectors.py`, `run_constrained_detector(root, region, config, *, client, model,
taxonomy, ranges)`; `FamilyConfig` gains nothing, because a constrained detector is configured by the same
prompt/checklist/limits and reads no tools.
- `src/openultrasast/learning/scoring.py` — `_hits`, `_others` and `_fabricated` delegate to `predicates`; the
- `src/openultrasast/learning/scoring.py`, `_hits`, `_others` and `_fabricated` delegate to `predicates`; the
family metrics gain nothing. A test asserts the recorded baseline numbers are reproduced exactly.
- `src/openultrasast/learning/publish.py` — the overfitting gap and the per-slice table (Req 3.1, 3.4).
- `src/openultrasast/config.py` — `[learning] detector = "hunter" | "constrained"`, default `hunter` until the
- `src/openultrasast/learning/publish.py`, the overfitting gap and the per-slice table (Req 3.1, 3.4).
- `src/openultrasast/config.py`, `[learning] detector = "hunter" | "constrained"`, default `hunter` until the
comparison says otherwise.
- `src/openultrasast/cli.py` — `learning candidates`; `--detector` on `score` and `baseline`.
- `pyproject.toml` — unchanged.
- `src/openultrasast/cli.py`, `learning candidates`; `--detector` on `score` and `baseline`.
- `pyproject.toml`, unchanged.
## System Flows
@@ -282,13 +282,13 @@ def teachable(slice_name: str, cases) -> bool # False while every row of a
## Migration and Phases
1. **Candidates and the ceiling, offline.** `candidates.py`, `learning candidates`, the per-slice ceiling artifact
1. Candidates and the ceiling, offline. `candidates.py`, `learning candidates`, the per-slice ceiling artifact
including the real-world slices (Req 3.5). No model call, so the phase is free and its result can still cancel
the design — if the ceiling on a deciding slice is below that slice's current detection rate, stop here.
2. **Predicates extracted.** `predicates.py`, `scoring.py` delegating, the baseline-reproduction test.
3. **Judgment and the constrained detector.** `judgments.py`, `run_constrained_detector`, the determinism test.
4. **Slices and reporting.** `slices.py`, the overfitting gap, `publish` per slice.
5. **The comparison.** Constrained against unconstrained on every slice, the re-measured trajectory count and noise
the design. If the ceiling on a deciding slice is below that slice's current detection rate, stop here.
2. Predicates extracted. `predicates.py`, `scoring.py` delegating, the baseline-reproduction test.
3. Judgment and the constrained detector. `judgments.py`, `run_constrained_detector`, the determinism test.
4. Slices and reporting. `slices.py`, the overfitting gap, `publish` per slice.
5. The comparison. Constrained against unconstrained on every slice, the re-measured trajectory count and noise
floor, and the named `learning-harness` amendments if the floor collapses.
Phase 1 is deliberately able to kill the design before any money is spent on it.
@@ -5,7 +5,7 @@
## Introduction
The detector's instability is structural, not statistical. Measured on 2026-09-06 over twelve injection pairs at
temperature 0 with thinking disabled, **120 runs produced 120 distinct tool-call trajectories** — not one repeat —
temperature 0 with thinking disabled, **120 runs produced 120 distinct tool-call trajectories**, not one repeat,
and outcome disagreement tracks trajectory divergence (30.9% at low similarity against 4.2% at mid). The model
spends its budget searching: 555 `grep_repo` calls against 71 `entry_points`, guessing where sinks are while the
semantic IR already knows. So `learning-harness`'s K runs are an ensemble of distinct search procedures rather than
@@ -20,22 +20,22 @@ ruleset caps recall at 12.1%, below the 64.9% the unconstrained loop already rea
at 96.6% with a median of ten sites per labeled function.
Baseline to beat, `deepseek-v4-flash`, K = 5, 57 scorable web pairs: **detection 64.9%, leak 12.3%, pair-correct
52.6%** — and 83.3% detection on hand-written vibe-py against 44.4% on agent-written agent-vfc, which is the gap
52.6%**, and 83.3% detection on hand-written vibe-py against 44.4% on agent-written agent-vfc, which is the gap
this feature must close rather than widen. Context and measurements: `brief.md`,
`benchmarks/measurements/2026-09-06-trajectory-divergence.json`, `2026-09-06-candidate-ceiling.json`.
## Boundary Context
- **In scope**: candidate enumeration from the semantic IR, per family shape; the typed, toolless judgment step;
- In scope: candidate enumeration from the semantic IR, per family shape; the typed, toolless judgment step;
constraint predicates enforced at generation time with a bounded retry; the evidence that decides whether the
K-run and noise-floor machinery still earns its place; per-slice reporting that keeps real-world code in front.
- **Out of scope**: the family taxonomy, the classifier, the pair scorer's outcome rules, the split, the verifier
boundary and write roots, redaction, the three gates, and the publication machinery — all of which stand and are
- Out of scope: the family taxonomy, the classifier, the pair scorer's outcome rules, the split, the verifier
boundary and write roots, redaction, the three gates, and the publication machinery, all of which stand and are
what made this feature's measurements possible. Also out: prompt optimisation, reflective evolution and any
optimiser (deferred to a later spec, on this feature's evidence); finding bugs in JavaScript interpreters, WebGL
renderers or browser engines; memory-safety fuzzing for C and C++; fine-tuning model weights; adopting DSPy as a
dependency.
- **Adjacent expectations**: `learning-harness` keeps owning the taxonomy, the scorer, the split and publication;
- Adjacent expectations: `learning-harness` keeps owning the taxonomy, the scorer, the split and publication;
its Req 3.5 (K runs), 8.2 (noise floor) and 9.5 (acceptance) are amended by this feature only on evidence it
produces, never on argument. The pair corpus keeps owning labels, tiers and splits. The unconstrained tool hunter
stays as the generalist and as the baseline this feature is measured against.
@@ -49,8 +49,8 @@ model stops searching and the limit of what it can possibly find is a number rat
#### Acceptance Criteria
1. The Constrained Detector shall enumerate candidate sites for a region from the semantic IR — call sites and
bindings — and shall not use the pattern ruleset as its candidate generator.
1. The Constrained Detector shall enumerate candidate sites for a region from the semantic IR (call sites and
bindings), and shall not use the pattern ruleset as its candidate generator.
2. The Constrained Detector shall shape candidates by family: a call site for injection, path, deserialization,
untrusted destination and prototype; an operation on a protected resource with the guards in scope for access
control; a configuration assignment or literal for permissive configuration and secrets.
@@ -90,13 +90,13 @@ teaching examples, so that the tool is not tuned for cases nobody has.
1. The Constrained Detector shall report every number per slice and shall never publish an aggregate across slices
without the per-slice numbers beside it.
2. The Constrained Detector shall designate the slices harvested from real projects — agent-vfc, vfc-js, vfc and
github — as the deciding slices, and the hand-written vibe-py slice as a development slice that cannot on its
2. The Constrained Detector shall designate the slices harvested from real projects (agent-vfc, vfc-js, vfc and
github) as the deciding slices, and the hand-written vibe-py slice as a development slice that cannot on its
own justify accepting a change.
3. If a change improves the development slice and regresses any deciding slice, then the Constrained Detector shall
reject it, and the rejection shall name the slice that regressed.
4. The Constrained Detector shall publish the **overfitting gap** — development-slice detection minus deciding-slice
detection — beside every headline number, and shall state that the baseline gap is 83.3% against 44.4%.
4. The Constrained Detector shall publish the **overfitting gap** (development-slice detection minus deciding-slice
detection) beside every headline number, and shall state that the baseline gap is 83.3% against 44.4%.
5. The Constrained Detector shall measure and publish the candidate ceiling separately for the real-world slices,
because the functions there are larger than the teaching corpus's and the enumerator's behaviour on them is not
implied by its behaviour on vibe-py.
+9 -9
View File
@@ -1,4 +1,4 @@
# Brief: contributor-scan — make the model layer a tool someone runs
# Brief: contributor-scan, make the model layer a tool someone runs
**Status:** brief only, unapproved. Successor to `model-grounded-detection`, which is complete (16/16 tasks)
and is not reopened by this.
@@ -22,7 +22,7 @@ is wired. The spec is complete and the feature is unshipped, both at once.
> "a real world tool that helps developers when they want to contribute"
That reframes the work. A contributor does not have a labelled function and a known weakness class — they
That reframes the work. A contributor does not have a labelled function and a known weakness class; they
have a repository they are about to touch and a question about what is risky in it. Everything measured so
far is function-level pairs: a labelled function, a declared family, a known target. None of it exercises the
questions a whole repository asks.
@@ -33,11 +33,11 @@ questions a whole repository asks.
Without this nothing else matters.
2. **Whole-repo scanning.** Region selection from entry points instead of a labelled function; ranking across
thousands of candidates rather than one; a budget policy (the enumerator caps at 8 candidates per region,
which is meaningless at repo scale); and the performance envelope — a CPG build on a 100k-line tree, not a
which is meaningless at repo scale); and the performance envelope: a CPG build on a 100k-line tree, not a
40-line excerpt. **This has never been run.**
3. **A known-vulnerable repository as the target.** The 176 real-world CVE pairs already in the corpus are
single-function excerpts and do not exercise any of the above. Running against a real checkout — curl,
libpng, a WordPress plugin — is the only way to find what breaks at scale.
single-function excerpts and do not exercise any of the above. Running against a real checkout (curl,
libpng, a WordPress plugin) is the only way to find what breaks at scale.
4. **The memory decision.** 173 of those 176 pairs are `memory` family, routed to a deferred execution tier
that is a stub. Either adopt Clearwing for them or scope C memory-safety out explicitly. Right now it is
neither, and a sixth of the corpus sits idle.
@@ -49,12 +49,12 @@ questions a whole repository asks.
These were delivered during `model-grounded-detection` without being in any spec. They belong here so the
record is not scattered:
- **The pipeline itself** (`model/pipeline.py`) — Req 8 existed, task 3.2 built `judge()`, and nothing ever
- The pipeline itself (`model/pipeline.py`): Req 8 existed, task 3.2 built `judge()`, and nothing ever
said to connect an enumerator to it. Built, tested, measured, unspecified.
- **Ops** — a single image carrying tool and engine, a compose file with read-only target and tmpfs scratch,
- Ops: a single image carrying tool and engine, a compose file with read-only target and tmpfs scratch,
bash/zsh/PowerShell wrappers that make the container invisible, and a pyinfra deploy for host installs.
The Docker path is written and syntax-validated but **the image has never been built**.
- **Corpus** — the `owasp` slice (40 validated xss pairs; `output_encoding` went from 1 pair to 41) and
- Corpus: the `owasp` slice (40 validated xss pairs; `output_encoding` went from 1 pair to 41) and
authored sink/sanitizer facts for Java and JavaScript.
## Known limits to carry forward
@@ -63,7 +63,7 @@ record is not scattered:
`debug=True` is dangerous where `httponly=True` is safe and the model cannot tell them apart.
- Interprocedural resolution is the most repeated miss cause across every experiment: OWASP's helper classes,
`angular-http-server`'s `resolveUrl`. Joern supports it; our queries may not be configured for it.
- Five bugs in `model-grounded-detection` shared one shape — a taint-specific assumption applied to every
- Five bugs in `model-grounded-detection` shared one shape: a taint-specific assumption applied to every
family, each found by a measurement rather than a check. Any new family-specific path should be audited for
siblings before it is measured, not after.
@@ -1,4 +1,4 @@
# Task 2.19 — measured input retention and census completeness
# Task 2.19, measured input retention and census completeness
The maintainer accepted the proposed contributor-scan follow-up after the pre-push
NO-GO report. The Kiro root-cause-first workflow identified a concrete shared-boundary
+18 -18
View File
@@ -1,6 +1,6 @@
# Design Document: contributor-scan
## Integration amendment — 2026-09-12 (draft, unapproved)
## Integration amendment, 2026-09-12 (draft, unapproved)
The maintainer has clarified the product as an actionable pre-push safety net across supported
ecosystems. The proposed successor `../pre-push-safety-net/design.md` consumes this driver/backend
@@ -9,7 +9,7 @@ of both targets and graphs. Existing approvals below apply to the prior design,
Proposed shared seams needing requirements/design review before implementation:
- Accept an absolute deadline across graph build, overlays, retries and queries, not only region/call caps.
- Accept an absolute deadline across graph build, overlays, retries and queries as well as region/call caps.
- Accept an optional validated reusable graph lease while preserving ordinary fresh-build scans;
backend remains authoritative for artifact identity, census, compatibility and cleanup.
- Return exact selected/deferred question IDs and context boundaries from ranker decisions;
@@ -45,13 +45,13 @@ for someone who does not have Joern.
### Non-Goals
- New arbiters or families — the three forms exist and are measured.
- New arbiters or families; the three forms exist and are measured.
- Any optimiser, prompt search, or K-run averaging. The predecessor deleted that machinery; this does not
reintroduce it by another name.
- Building a sandbox. Req 10 of the predecessor stands: adopt, never build.
- Changing the gates, the taxonomy, or the fact-table schema.
## The fact tables are an ontology, and it is general-purpose — 2026-09-21
## The fact tables are an ontology, and it is general-purpose, 2026-09-21
The `[[source]]`, `[[sink]]`, `[[sanitizer]]`, `[[operation]]`, `[[discharger]]` and `[[layout]]` rows
are not a pattern list. They are an ontology of program elements and their security relations: what
@@ -87,18 +87,18 @@ repository is in front of the maintainer, which is what task 7.1 exists to make
### This Spec Owns
- `model/regions.py` — entry points and file fallbacks to regions, with the family routing for each.
- `model/scan.py` — the repository-level driver: rank, budget, dispatch to `model/pipeline`, collect.
- `model/regions.py`: entry points and file fallbacks to regions, with the family routing for each.
- `model/scan.py`: the repository-level driver: rank, budget, dispatch to `model/pipeline`, collect.
- The `Stage.MAP` wiring in `cli.py` and the scan's model-layer payload and report section.
- `benchmarks/repos/` — pinned known-vulnerable checkouts and their baselines.
- `benchmarks/repos/`: pinned known-vulnerable checkouts and their baselines.
- The decisions in Req 7 and their record.
### Out of Boundary
- `model/pipeline`, `taint`, `dominance`, `config_value`, `ladder`, `specs`, `candidates` — built, measured,
- `model/pipeline`, `taint`, `dominance`, `config_value`, `ladder`, `specs`, `candidates`: built, measured,
used unchanged. If this feature needs to change one, that is a signal the change belongs in its own commit
with its own measurement.
- `mapping.analyze_entry_points`, the overlay, the obligations checker, the gates — untouched.
- `mapping.analyze_entry_points`, the overlay, the obligations checker, the gates: untouched.
### Allowed Dependencies
@@ -117,8 +117,8 @@ repository is in front of the maintainer, which is what task 7.1 exists to make
### Existing Architecture Analysis
`ousast scan` runs `policy_load → ruleset_load → preprocess → entry_point_mapping → rank → calibrate →
STATIC → MAP → REGRESS → report`. Inside MAP it already builds a complexity map, runs the overlay, and — the
closest analogue to what we are adding — runs `_check_obligations(root, targets, entries, facts, settings,
STATIC → MAP → REGRESS → report`. Inside MAP it already builds a complexity map, runs the overlay, and (the
closest analogue to what we are adding) runs `_check_obligations(root, targets, entries, facts, settings,
runtime, hunter_model)`, which takes entry points, produces findings, and merges them by re-ranking.
The model layer slots in exactly there, with the same shape and the same failure discipline. `entry_points`
@@ -167,10 +167,10 @@ benchmarks/repos/
### Changed
- `cli.py` — a `model` stage inside MAP, mirroring `_check_obligations`; its payload into the manifest and a
- `cli.py`: a `model` stage inside MAP, mirroring `_check_obligations`; its payload into the manifest and a
report section.
- `model/report.py` — a per-rung summary block for a scan (it currently reports per-slice for corpus runs).
- `config.py` — a `ModelConfig`-style block: enabled, budget, max regions. Zero-dep, defaults conservative.
- `model/report.py`: a per-rung summary block for a scan (it currently reports per-slice for corpus runs).
- `config.py`: a `ModelConfig`-style block: enabled, budget, max regions. Zero-dep, defaults conservative.
### Unchanged and load-bearing
@@ -194,8 +194,8 @@ class ScanRegion:
def regions_for(entries, targets, *, taxonomy, language_of) -> tuple[ScanRegion, ...]
```
Ranking uses what `EntryPointRecord` already carries — an externally-reachable entry point at a trust
boundary outranks an internal one — rather than inventing a new signal.
Ranking uses what `EntryPointRecord` already carries (an externally-reachable entry point at a trust
boundary outranks an internal one) rather than inventing a new signal.
### model/scan.py
@@ -264,10 +264,10 @@ each; everything past exhaustion counted into `regions_unjudged`.
1. **Wire it (Req 1, 2).** Regions, the MAP stage, findings with rungs reaching a report. Gate-protected.
Nothing else matters until a user can see a rung.
2. **Scale it (Req 3, 4).** Budget and ranking, then the first run against a pinned known-vulnerable
checkout — the measurement that has never been taken and can reshape the rest.
checkout, the measurement that has never been taken and can reshape the rest.
3. **Ship it (Req 5, 6).** Contributor output; build and verify the image that is currently only defined.
4. **Close the deferred (Req 7).** The memory decision; retire `evidence_level`.
5. **Grow the corpus (Req 9).** PHP/WordPress first — the families the model arbitrates best. C CVEs only
5. **Grow the corpus (Req 9).** PHP/WordPress first: the families the model arbitrates best. C CVEs only
after 7.1.
6. **Regression baselines (Req 10).**
@@ -1,4 +1,4 @@
# Release remediation research — 2026-09-14
# Release remediation research, 2026-09-14
Status: investigation under the maintainer's instruction to continue towards release.
This record does not approve rollout, replace the existing design, or mark a production
+19 -19
View File
@@ -12,7 +12,7 @@ hunter's 76.7% / 12.3%, at a fifth of the model calls. All 16 of its tasks are c
from a measurement harness, not from the tool. This feature closes that gap and answers the question the
maintainer actually posed: *a real world tool that helps developers when they want to contribute.*
That framing changes the work. A contributor has no labelled function and no declared weakness class — they
That framing changes the work. A contributor has no labelled function and no declared weakness class; they
have a repository they are about to touch. Everything measured so far is function-level pairs, which
exercise none of the questions a whole repository asks: which regions to look at, how to rank thousands of
candidates, what to spend, and whether a CPG build on a 100k-line tree finishes at all.
@@ -22,13 +22,13 @@ Context: `brief.md`, `.kiro/specs/model-grounded-detection/`, and the committed
## Boundary Context
- **In scope**: wiring `model/pipeline` into `ousast scan`; region selection from entry points; ranking and
- In scope: wiring `model/pipeline` into `ousast scan`; region selection from entry points; ranking and
budget at repository scale; the performance envelope; running against known-vulnerable checkouts;
contributor-facing output; the packaging already built (image, compose, shell wrappers, pyinfra); the
deferred memory-family decision; retiring `evidence_level`.
- **Out of scope**: new arbiters or families (the three forms exist); prompt optimisation of any kind;
building a sandbox (Req 10 of the predecessor stands — adopt, never build); changing the gates.
- **Carried forward, not re-litigated**: the evidence ladder, the taxonomy, the fact tables, and the finding
- Out of scope: new arbiters or families (the three forms exist); prompt optimisation of any kind;
building a sandbox (Req 10 of the predecessor stands: adopt, never build); changing the gates.
- Carried forward, not re-litigated: the evidence ladder, the taxonomy, the fact tables, and the finding
that the model supplies precision while the LLM supplies recall.
## Requirements
@@ -45,7 +45,7 @@ built is something I can run rather than something a harness demonstrates.
2. The system shall report model findings alongside the existing pattern and overlay findings, without
removing either.
3. When no CPG engine is available, the scan shall behave exactly as it does today and record
`cpg_unavailable` — the model layer is additive, never a precondition.
`cpg_unavailable`; the model layer is additive, never a precondition.
4. The detection, map and local pair gates shall remain byte-identical to the committed baseline.
5. The system shall expose the model layer's per-rung counts in the scan's own output, so a user can see how
much of the result the model arbitrated rather than inferring it.
@@ -109,7 +109,7 @@ in the corpus exercises it.
#### Acceptance Criteria
1. Every finding shall carry its rung, its witness, and the file and line it concerns.
2. The system shall state plainly what a `suspicion` means — that the model could not confirm it — so a
2. The system shall state plainly what a `suspicion` means (that the model could not confirm it) so a
reader does not treat it as established.
3. The system shall write findings to the writable output location, never into the repository under analysis.
4. The system shall summarise a scan in a form readable without opening a JSON file.
@@ -121,7 +121,7 @@ in the corpus exercises it.
throughout, and no criterion here forbade it.)*
6. Where a family's abstraction cannot decide a weakness class, the system shall state that limit in the
report, and no scan shall present an absence of findings as evidence of absence. *(Added 2026-09-09.
`c/memory` scored zero true positives across 89k lines of C for a structural reason — it models string
`c/memory` scored zero true positives across 89k lines of C for a structural reason: it models string
functions and libpng overflows through arithmetic. A silent scan reading as a clean bill of health is
worse than noise because it is quiet, and it is the one failure mode the rung ladder does not guard.)*
@@ -148,7 +148,7 @@ silently caps what the corpus can tell us.
#### Acceptance Criteria
1. The system shall either reach the `memory` family through an adopted execution tier, or record explicitly
that C memory-safety is out of scope — 173 of the 176 real-world CVE pairs currently sit idle behind a
that C memory-safety is out of scope; 173 of the 176 real-world CVE pairs currently sit idle behind a
stub.
2. The system shall retire `evidence_level` in favour of `rung`, or record why both are kept.
3. Each decision shall be recorded with its reason where a later reader will find it.
@@ -167,7 +167,7 @@ this one, because each exists due to a specific past error.
4. The core install shall remain zero-dependency, with the CPG engine, the LLM endpoint and any execution
tier optional and degrading to a recorded reason.
5. Any new family-specific code path shall be checked for sibling paths that need the same change, before it
is measured — five bugs in the predecessor shared exactly that shape.
is measured; five bugs in the predecessor shared exactly that shape.
### Requirement 9: Grow the corpus where the engine can use it
@@ -177,14 +177,14 @@ what developers actually contribute to rather than against 30 pairs of one slice
#### Acceptance Criteria
1. The system shall add a **PHP/WordPress** slice covering plugin and core weaknesses. WordPress plugin CVEs
are overwhelmingly injection, XSS and authorization bypass — the families the model layer already
arbitrates well — and `php2cpg` works once a PHP interpreter is present, which the shipped image now
are overwhelmingly injection, XSS and authorization bypass (the families the model layer already
arbitrates well) and `php2cpg` works once a PHP interpreter is present, which the shipped image now
provides.
2. The system shall add **real-world C library CVEs** beyond the existing curl and OpenSSL rows — libpng and
2. The system shall add **real-world C library CVEs** beyond the existing curl and OpenSSL rows: libpng and
the image libraries, OpenSSH, libpam, and vulnerable kernel modules were named. These are gated on Req 7.1:
until the memory-family decision is closed they would join 173 pairs already sitting idle, so the decision
comes first and the harvest follows it.
3. The system shall add further **vibe-code** pairs — agent-written and human-written web application bugs —
3. The system shall add further **vibe-code** pairs (agent-written and human-written web application bugs)
because that slice is where the tool's stated audience actually works, and it is currently 35 pairs.
4. Before any slice is vendored at scale, the system shall verify on a sample that the model can **read** its
pairs, and shall vendor only the categories that a measurement shows it can arbitrate. The OWASP harvest
@@ -194,7 +194,7 @@ what developers actually contribute to rather than against 30 pairs of one slice
contiguous key-shaped literal.
6. Each new slice shall report as its own row and shall never be blended into a headline, and a synthetic
slice shall say in its catalog that it measures breadth rather than real-world performance.
7. The system shall record, per slice, how many pairs the model can arbitrate — so corpus growth is measured
7. The system shall record, per slice, how many pairs the model can arbitrate, so corpus growth is measured
by what it teaches rather than by how many rows it adds.
### Requirement 10: Regression tuning over a large codebase
@@ -206,7 +206,7 @@ tables and queries can be checked for regressions that pair scoring cannot see.
1. The system shall support a repeatable scan over a pinned checkout of a large repository, recorded as a
committed baseline.
2. When the fact tables or queries change, the system shall report the delta against that baseline —
2. When the fact tables or queries change, the system shall report the delta against that baseline:
findings gained, findings lost, and rung movements.
3. The system shall treat an unexplained loss of a previously-entailed finding as a regression to be
investigated, not a number to be accepted.
@@ -214,7 +214,7 @@ tables and queries can be checked for regressions that pair scoring cannot see.
---
## Amendment record — group 2 (approved 2026-09-09)
## Amendment record, group 2 (approved 2026-09-09)
Group 2 was specified as the phase that can reshape the rest, and it did. Three amendments were drafted from
what it measured and **approved by the maintainer on 2026-09-09**; they are folded into the numbered
@@ -227,7 +227,7 @@ requirements above and marked there with the evidence that produced them.
| C — a checkout that can demonstrate detection | Req 4.6 | libpng satisfies 4.1 and cannot demonstrate detection |
**A is only half implemented, deliberately.** Task 2.11 made pattern matches proportionate. The same argument
applies to the model layer's own suspicion band — 38 of vampi's 60 findings — which is an enumerated
applies to the model layer's own suspicion band (38 of vampi's 60 findings), which is an enumerated
candidate the graph could not decide and an LLM affirmed: a pattern match with an opinion attached, not a
reasoned claim. Task 2.17 carries that half.
@@ -238,5 +238,5 @@ reasoned claim. Task 2.17 carries that half.
(2.6), where the leak fell 42.9% → 0.0% with pair-correct unchanged.
* **Req 3.3's honesty counters were the load-bearing part.** `regions_unjudged` is what made a budget that
stopped free work visible at all.
* **Req 11.2's byte-identical gates held through every one of nine defect fixes** — which says the same thing
* **Req 11.2's byte-identical gates held through every one of nine defect fixes**, which says the same thing
from the other side: the corpus could not have caught any of them.
@@ -18,8 +18,8 @@ Derive a mechanism record from a pair by parsing the vulnerable excerpt with the
## Scope
- **In**: mechanism export from pairs; shape normalization; variant search as a proposer in MAP; leave-one-out measurement command and payload; `mechanisms` improve lever with validator rules; store hygiene (tier, provenance, dedupe, version); offline tests.
- **Out**: LLM-authored shapes; changing dispositions or evidence rungs; embedding retrieval changes (existing prove-order reuse stays); inter-file flow (reachability spec); any merge gate; vendoring more pairs (corpus spec).
- In: mechanism export from pairs; shape normalization; variant search as a proposer in MAP; leave-one-out measurement command and payload; `mechanisms` improve lever with validator rules; store hygiene (tier, provenance, dedupe, version); offline tests.
- Out: LLM-authored shapes; changing dispositions or evidence rungs; embedding retrieval changes (existing prove-order reuse stays); inter-file flow (reachability spec); any merge gate; vendoring more pairs (corpus spec).
## Boundary Candidates
@@ -35,13 +35,13 @@ Derive a mechanism record from a pair by parsing the vulnerable excerpt with the
## Upstream / Downstream
- **Upstream**: `PairCase`/`load_pair_catalog` with `review_tier`; `FileIR`; `MechanismStore`; `adjudicate`; `run_improvement` gate with per-profile holdout.
- **Downstream**: `reachability-flow-model` attaches paths to variant findings; `guard-dominance-regime` consumes the guard shapes as sanitizer candidates; reports show mechanism ids on findings.
- Upstream: `PairCase`/`load_pair_catalog` with `review_tier`; `FileIR`; `MechanismStore`; `adjudicate`; `run_improvement` gate with per-profile holdout.
- Downstream: `reachability-flow-model` attaches paths to variant findings; `guard-dominance-regime` consumes the guard shapes as sanitizer candidates; reports show mechanism ids on findings.
## Existing Spec Touchpoints
- **Extends**: `propose-adjudicate-prove` mechanism memory (additive fields, second write path), `three-stage-scan` MAP stage (one more proposer), `harnessx-self-improving-rulesets` improve loop (one more bounded lever).
- **Depends**: `pair-corpus-honesty` Req 9 (tiers) for which pairs may seed.
- Extends: `propose-adjudicate-prove` mechanism memory (additive fields, second write path), `three-stage-scan` MAP stage (one more proposer), `harnessx-self-improving-rulesets` improve loop (one more bounded lever).
- Depends: `pair-corpus-honesty` Req 9 (tiers) for which pairs may seed.
## Constraints
@@ -107,9 +107,9 @@ tests/
### Modified Files
- `semantic/overlay.py` — `OverlayRecord.mechanism_id: str | None = None` (additive).
- `regress/candidate.py` — exposes `hotspot_from_finding`; variant findings enter the ranked candidate slice as zero-score hotspots from `cli.py` (after every scored promotion, inside the cap, through the safety check).
- `config.py` — `[variants] enabled=true, max_mechanisms=500`.
- `semantic/overlay.py`: `OverlayRecord.mechanism_id: str | None = None` (additive).
- `regress/candidate.py`: exposes `hotspot_from_finding`; variant findings enter the ranked candidate slice as zero-score hotspots from `cli.py` (after every scored promotion, inside the cap, through the safety check).
- `config.py`: `[variants] enabled=true, max_mechanisms=500`.
## System Flows
@@ -8,15 +8,15 @@ OpenUltraSAST's labeled pair corpus measures the detector but does not improve i
## Boundary Context
- **In scope**: mechanism export from pairs; shape normalization per language; variant search proposer in MAP; leave-one-out evaluation; `mechanisms` improve lever; store schema additions; reports showing mechanism ids; offline tests.
- **Out of scope**: taint, facts, walker, dispositions, evidence rungs; embedding retrieval changes; inter-file flow; LLM-authored shapes; catalogs and scorer changes beyond reading `PairCase`; any merge gate.
- **Adjacent expectations**: `Mechanism` gains additive fields only; `order_promotions` keeps working on old records; the improve loop's existing clauses and the per-profile holdout clause apply unchanged; `local` remains the only slice that can fail CI.
- In scope: mechanism export from pairs; shape normalization per language; variant search proposer in MAP; leave-one-out evaluation; `mechanisms` improve lever; store schema additions; reports showing mechanism ids; offline tests.
- Out of scope: taint, facts, walker, dispositions, evidence rungs; embedding retrieval changes; inter-file flow; LLM-authored shapes; catalogs and scorer changes beyond reading `PairCase`; any merge gate.
- Adjacent expectations: `Mechanism` gains additive fields only; `order_promotions` keeps working on old records; the improve loop's existing clauses and the per-profile holdout clause apply unchanged; `local` remains the only slice that can fail CI.
## Requirements
### Requirement 1: Mechanisms seeded from trusted pairs
**Objective:** As a maintainer, I want each reviewed pair to become a mechanism record, so that the corpus is a library the detector can search, not only a ruler.
**Objective:** As a maintainer, I want each reviewed pair to become a mechanism record, so that the corpus is both a library the detector can search and a ruler.
#### Acceptance Criteria
@@ -64,7 +64,7 @@ OpenUltraSAST's labeled pair corpus measures the detector but does not improve i
### Requirement 5: A recall-raising lever in the improve loop
**Objective:** As a maintainer, I want the improve loop to be able to admit mechanisms, so that it can raise recall and not only shadow noisy rules.
**Objective:** As a maintainer, I want the improve loop to be able to admit mechanisms, so that it can raise recall as well as shadow noisy rules.
#### Acceptance Criteria
@@ -2,9 +2,9 @@
## Summary
- **Feature**: `corpus-seeded-mechanisms`
- **Discovery Scope**: Extension of mechanism memory, MAP stage, pair eval, and the improve loop
- **Key Findings**:
- Feature: `corpus-seeded-mechanisms`
- Discovery Scope: Extension of mechanism memory, MAP stage, pair eval, and the improve loop
- Key Findings:
- `semantic/mechanisms.py` already has `Mechanism`, `MechanismStore` (append-only JSONL), `append_mechanism`, embedding cache, hard filters by language/CWE/tags, and `order_promotions`. It is written only by `cli._append_proven_mechanisms` after a sandbox trigger, so it is empty on every real tree.
- Clearwing's recall lever is the variant loop: a proven mechanism becomes a structural search over the messy tree. OpenUltraSAST copied the store and the retrieval, not the loop.
- `FileIR` call sites already carry callee name, argument texts, argument identifier lists, and constant flags; binds carry source text and names. That is enough for a shape without new parsing.
@@ -16,30 +16,30 @@
### Mechanism store today
- **Context**: Can the existing store carry corpus-derived records?
- **Sources Consulted**: `semantic/mechanisms.py`, `cli.py` `_append_proven_mechanisms`, `propose-adjudicate-prove` design §Mechanism.
- **Findings**: `Mechanism` fields are summary, cwe, language, tags, keywords, what_made_it_exploitable, source_finding_id, source_repo. Identity is the pattern. JSONL is authoritative; vectors are a rebuildable cache keyed by id and model.
- **Implications**: add `origin` (`sandbox` | `corpus`), `review_tier`, `pairs`, `shape` (structured), `guard` as additive fields with defaults so old rows load. A second writer (`append_from_pair`) beside the sandbox writer.
- Context: Can the existing store carry corpus-derived records?
- Sources Consulted: `semantic/mechanisms.py`, `cli.py` `_append_proven_mechanisms`, `propose-adjudicate-prove` design §Mechanism.
- Findings: `Mechanism` fields are summary, cwe, language, tags, keywords, what_made_it_exploitable, source_finding_id, source_repo. Identity is the pattern. JSONL is authoritative; vectors are a rebuildable cache keyed by id and model.
- Implications: add `origin` (`sandbox` | `corpus`), `review_tier`, `pairs`, `shape` (structured), `guard` as additive fields with defaults so old rows load. A second writer (`append_from_pair`) beside the sandbox writer.
### Shape derivation from FileIR
- **Context**: Avoid regex and LLM authoring.
- **Sources Consulted**: `semantic/ir.py` (`CallSite.arg_names`, `arg_is_constant`, `Bind.names`, `FunctionIR.params`), `semantic/taint.py` (`_source_in`, parameter seeding).
- **Findings**: for the labeled function, the sink call site is the call whose trailing name equals the label's sink or whose line equals the label's line; source positions are argument indexes whose identifiers trace by binds to a fact source, a parameter, or a container read. Guard kind comes from statements on the fixed side absent on the vulnerable side: `if` with `== NULL`/`is None`/`!x` → `null_test`; comparison with `<`, `>`, `len(` → `bounds_test`; membership in a literal collection or `startswith`/regex → `allowlist_test`; identity read from an auth context (`request.state`, `req.user`, `session`) → `auth_check`; parameterized second argument → `parameterized_call`; changed declared type → `type_change`.
- **Implications**: `semantic/variants.py` implements `derive_shape(pair_ir_vuln, pair_ir_fixed, label)` and `match_shape(file_ir, shape)`. Guard classification is heuristic and closed; unknown → `none`.
- Context: Avoid regex and LLM authoring.
- Sources Consulted: `semantic/ir.py` (`CallSite.arg_names`, `arg_is_constant`, `Bind.names`, `FunctionIR.params`), `semantic/taint.py` (`_source_in`, parameter seeding).
- Findings: for the labeled function, the sink call site is the call whose trailing name equals the label's sink or whose line equals the label's line; source positions are argument indexes whose identifiers trace by binds to a fact source, a parameter, or a container read. Guard kind comes from statements on the fixed side absent on the vulnerable side: `if` with `== NULL`/`is None`/`!x` → `null_test`; comparison with `<`, `>`, `len(` → `bounds_test`; membership in a literal collection or `startswith`/regex → `allowlist_test`; identity read from an auth context (`request.state`, `req.user`, `session`) → `auth_check`; parameterized second argument → `parameterized_call`; changed declared type → `type_change`.
- Implications: `semantic/variants.py` implements `derive_shape(pair_ir_vuln, pair_ir_fixed, label)` and `match_shape(file_ir, shape)`. Guard classification is heuristic and closed; unknown → `none`.
### Where variant search sits
- **Context**: Propose, adjudicate, prove roles must stay separate.
- **Sources Consulted**: `three-stage-scan` design (MAP stage, hunter suspicion), `propose-adjudicate-prove` (overlay records, coverage), `regress/candidate.py`.
- **Findings**: MAP already runs overlay and the tool hunter; variant search is a third proposer that reads `FileIR` and the store. Findings at `suspicion`, origin `variant`; when the overlay has a flow at that call site, merge (attach mechanism id to the overlay record). Candidate eligibility follows the hunter-eligibility path being added in `reachability-flow-model` or the sev-5 forced-hotspot path today.
- **Implications**: no disposition change; one new finding origin; manifest counters.
- Context: Propose, adjudicate, prove roles must stay separate.
- Sources Consulted: `three-stage-scan` design (MAP stage, hunter suspicion), `propose-adjudicate-prove` (overlay records, coverage), `regress/candidate.py`.
- Findings: MAP already runs overlay and the tool hunter; variant search is a third proposer that reads `FileIR` and the store. Findings at `suspicion`, origin `variant`; when the overlay has a flow at that call site, merge (attach mechanism id to the overlay record). Candidate eligibility follows the hunter-eligibility path being added in `reachability-flow-model` or the sev-5 forced-hotspot path today.
- Implications: no disposition change; one new finding origin; manifest counters.
### Leave-one-out
- **Context**: Prove that corpus growth raises recall, honestly.
- **Findings**: pair eval materializes each side into a temp tree already; a leave-one-out runner seeds a temp store from the other trusted pairs and runs variant search on both sides, scoring with the pair scorer's detection and leak rules.
- **Implications**: `pairs.py` gains a read-only consumer path (`evaluate_loo`) or a sibling module `semantic/loo.py` to keep the scorer's boundary; report per slice, profile, mechanism; artifact only.
- Context: Prove that corpus growth raises recall, honestly.
- Findings: pair eval materializes each side into a temp tree already; a leave-one-out runner seeds a temp store from the other trusted pairs and runs variant search on both sides, scoring with the pair scorer's detection and leak rules.
- Implications: `pairs.py` gains a read-only consumer path (`evaluate_loo`) or a sibling module `semantic/loo.py` to keep the scorer's boundary; report per slice, profile, mechanism; artifact only.
## Architecture Pattern Evaluation
@@ -54,27 +54,27 @@
### Decision: Seed only from seeded and reviewed tiers
- **Context**: `title` pairs are unreviewed guesses; `advisory` pairs have correct fixes but unlabeled mechanisms.
- **Selected Approach**: export reads `review_tier`; `advisory`/`title` may be scored leave-one-out as held-out targets but never seed.
- **Trade-offs**: fewer seeds at first; honest about what the detector was taught.
- Context: `title` pairs are unreviewed guesses; `advisory` pairs have correct fixes but unlabeled mechanisms.
- Selected Approach: export reads `review_tier`; `advisory`/`title` may be scored leave-one-out as held-out targets but never seed.
- Trade-offs: fewer seeds at first; honest about what the detector was taught.
### Decision: Variant is a proposer at suspicion
- **Context**: Evidence honesty.
- **Selected Approach**: origin `variant`, `suspicion`; merge with overlay flow when present; sandbox may raise.
- **Trade-offs**: variant-only hits do not fail CI; correct.
- Context: Evidence honesty.
- Selected Approach: origin `variant`, `suspicion`; merge with overlay flow when present; sandbox may raise.
- Trade-offs: variant-only hits do not fail CI; correct.
### Decision: Lever admits records, not shapes
- **Context**: Keep the validator closed.
- **Selected Approach**: `mechanisms` lever edits reference record ids produced by the exporter; free-form shapes are rejected.
- Context: Keep the validator closed.
- Selected Approach: `mechanisms` lever edits reference record ids produced by the exporter; free-form shapes are rejected.
## Risks & Mitigations
- Coarse shapes over-match (`exec(x)` everywhere) — source-position requirement plus fact/parameter/container check; report leak rate on fixed sides leave-one-out; the per-profile gate rejects admissions that leak.
- Guard heuristics misclassify — closed set with `none`; reported, not used for demotion.
- Store growth — dedupe by shape; one record per shape with pair provenance list.
- Leave-one-out cost — one temp store per pair over tens of pairs; fine; cache parsed IR per excerpt.
- Coarse shapes over-match (`exec(x)` everywhere): source-position requirement plus fact/parameter/container check; report leak rate on fixed sides leave-one-out; the per-profile gate rejects admissions that leak.
- Guard heuristics misclassify: closed set with `none`; reported, not used for demotion.
- Store growth: dedupe by shape; one record per shape with pair provenance list.
- Leave-one-out cost: one temp store per pair over tens of pairs; fine; cache parsed IR per excerpt.
## References
@@ -9,13 +9,13 @@ numbers"), gated on the pool metric; it is not part of the G0 record-07 baseline
The engine produces **no finding** on real unseen vulnerability changes. Live 300 s long-deadline replays show
that the cause is usually not a detection-capability gap:
- **Clean Python injection change** (`aws/bedrock-agentcore-sdk-python`, `install_packages`, introducing, 3-file
- Clean Python injection change (`aws/bedrock-agentcore-sdk-python`, `install_packages`, introducing, 3-file
diff): `wall=303 s`, hit the deadline. The ranker found candidates (`static_rank=1.0`) but **`selected: []`** and
every candidate `deferred` with `reason: deadline_exhausted`, `analysis_not_started=deadline_exhausted`,
`change_context_incomplete`. No analyses ran. The ranked candidates were **repo-wide and unrelated to
the push** (`auth.py` access_control/config_secrets), not the changed `install_packages`. The engine is trying to
re-audit the whole repo and runs out of budget before analyzing the change.
- **Noisy last_touch config_secrets change** (`jenkinsci/metrics-plugin`, 14-file diff): finished in 206 s but
- Noisy last_touch config_secrets change (`jenkinsci/metrics-plugin`, 14-file diff): finished in 206 s but
`comparison_unknown` + `capability_unavailable` + `eligibility_provenance_mismatch` +
`contributor-scan ... projection_unavailable`. This has several causes. Correspondence was one (fixed separately,
commit c39fbb8); the capability registry is Task 8.3.
@@ -157,12 +157,12 @@ To be releasable, the net needs tier 2 and/or tier 3 to run and catch within a r
The target is **≤60–120 s** (developer patience), hard ceiling 300 s. The analysis records each tier's budget against
that window and identifies the measurements needed:
- **Engine barrier (tier 2).** Fixed floor `build(48) + evidence_fixed(~150) ≈ 198 s` precedes any dataflow answer.
- Engine barrier (tier 2). Fixed floor `build(48) + evidence_fixed(~150) ≈ 198 s` precedes any dataflow answer.
Measure (clean host): **E1** split the ~150 s into JVM-start / graph-load / census; **E2** does a warm session
with the graph **preloaded at build** eliminate `evidence_fixed` on the first query (the architectural
test)?; **E3** analysis (`reachableByFlows`) per-pair cost + warm reuse; **E4** build cost vs repo size.
Levers in priority order: warm session + graph preload → cached census → change-scope the per-pair tail.
- **LLM barrier (tier 3).** Unmeasured in the push path. Measure: **L1** model round-trip latency per call
- LLM barrier (tier 3). Unmeasured in the push path. Measure: **L1** model round-trip latency per call
(DeepSeek) and calls needed per push for the decision/proof stage; **L2** the recall lift of tier 3 over tiers
1+2 on the pool, does the LLM catch what regex and dataflow miss, and at what $ and latency per push?
- Produce a **barrier table** for budget ∈ {60, 120, 300, 600} s × each tier combination. Record whether it fits
@@ -258,7 +258,7 @@ Reprioritised engine levers (corrected, anchoring is not the lever):
([[search-with-proof]]) is that the LLM tier catches what regex+dataflow miss. The pilot supports
testing it: **L1** model latency + calls/push, **L2** recall lift on the same 5 (and beyond). These calls cost money, so stage
them (5–10 sizing → ~10–20 pilot → full only on lift).
- **#2 taint source/sink coverage + region ranking** so the dataflow recognises these vuln sources and promotes the
- #2 taint source/sink coverage + region ranking so the dataflow recognises these vuln sources and promotes the
vuln region (banks path: sink found, source unrecognised; graphiti injection: score 0).
- #3 change-anchoring robustness matters only after the static/LLM tiers produce vuln-family
findings to anchor. Deprioritised until (1)/(2) yield findings.
@@ -329,10 +329,10 @@ Two blockers to realising it in the live pipeline (both now concrete):
mislocation, not a detection failure.**
The budget is approved. Keep the revised next steps staged:
- **Validate the 5 pilot labels first** (cheap, LLM-assisted corpus audit): for each, is the real sink inside the
- Validate the 5 pilot labels first (cheap, LLM-assisted corpus audit): for each, is the real sink inside the
changed lines, and is it change-introduced vs pre-existing-in-base? Only sound, change-located vulns are fair
targets. This separates true detection gaps from SZZ artifacts before spending on detection runs.
- **Then run the agentic hunter** (search-with-proof) on the sound targets and size its per-vuln call count/cost;
- Then run the agentic hunter (search-with-proof) on the sound targets and size its per-vuln call count/cost;
measure catch + false-alarm (on ordinary changes) vs regex 1/5. The single-shot path is insufficient; the hunter
is the real "#1".
- Feed confirmed source shapes (e.g. library param → path join) back as guarded static facts (#2).
@@ -362,7 +362,7 @@ which becomes a fact after repeated confirmation (the #2 feedback loop). No new
user. This is now part of the "#2 guided by #1" workstream and must not be deferred. (DeepSeek
`deepseek-flash` is the working default; GLM 5.3 "cyber" model is future-spec.)
## POOL-P1 LABEL AUDIT (LLM, 2026-10-12) — 47% of labels are sound
## POOL-P1 LABEL AUDIT (LLM, 2026-10-12): 47% of labels are sound
The banks label-location check was extended to all **49 public** pool vulns via DeepSeek (advisory + head code +
pool label → sound/unsound). Full artifact: `benchmarks/measurements/2026-10-12-pool-p1-label-audit/`.
@@ -378,7 +378,7 @@ pool label → sound/unsound). Full artifact: `benchmarks/measurements/2026-10-1
- Next: run the agentic search-with-proof hunter on the sound targets (start bedrock + graphiti), size its
per-vuln call count/cost, measure catch + false-alarm. Then recompute record 07 on the sound subset.
## MODEL HUNTER on the 2 sound targets (2026-10-12) — first LLM catch the static tiers missed
## MODEL HUNTER on the 2 sound targets (2026-10-12): first LLM catch the static tiers missed
Ran `ousast scan --mode deep` (DeepSeek `deepseek-flash`, model layer + tool_hunter, 60-call/8-hotspot bound) on
bedrock + graphiti, the two sound pilot targets the static engine missed. Artifact:
@@ -417,7 +417,7 @@ Next: deep scan (regex/engine/hunter) on the 5 fair targets → per-tier catch r
stand up an app-shaped eval set from known-vulnerable apps (VAmPI/NodeGoat/DjanGoat/vibe-py) and recompute the
floor. Revisit the deferred bets (advisory-with-conditions, reachability prover) once the fair-target results are available.
## FAIR-TARGET RESULT: detection 3/5 on app-shaped+sound (vs ~0 library-heavy) — target-fix validated (2026-10-12)
## FAIR-TARGET RESULT: detection 3/5 on app-shaped+sound (vs ~0 library-heavy), target-fix validated (2026-10-12)
Deep scan on the 5 app-shaped+sound targets (`benchmarks/measurements/2026-10-12-hunter-fair-targets/`):
- 3/5 detected: graphiti (hunter, Cypher injection), mcp-remote (hunter, url-scheme injection), dcmcp (ENGINE
@@ -433,7 +433,7 @@ Detection appears on fair targets; release-grade BLOCK still needs (a) proof-arm
(b) family coverage (template XSS/SSTI) + the DSPy hint loop for misses. Next: recompute a floor on an app-shaped
sound subset / real app corpus (VAmPI, NodeGoat, DjanGoat, vibe-py), and measure the proof arm's suspicion→proven lift.
## PROOF-ARM MEASUREMENT on the 3 fair-target hits (2026-10-12) — push-time lift 0/3, structural
## PROOF-ARM MEASUREMENT on the 3 fair-target hits (2026-10-12): push-time lift 0/3, structural
Measured whether the proof arm can elevate the 3 fair-target suspicions (graphiti, mcp-remote, dcmcp) to
`exploit_demonstrated`.
@@ -448,7 +448,7 @@ Measured whether the proof arm can elevate the 3 fair-target suspicions (graphit
(`isolation_mode: userns`); the plane supplies a task-boundary dispatcher.
- The scan/push path reaches only `suspicion`/`static_corroboration` (verification.py levels). There is no code
path from a scan suspicion to `exploit_demonstrated`, and no caller wires the search Coordinator from the push
flow. So proof-arm lift at push time = 0/3: not because the vulns are unreal, but because proving is a
flow. The vulns are real, but proof-arm lift at push time = 0/3 because proving is a
runtime differential operation with an offline build for each target that cannot run inside a pre-push hook. (Prior
per-pair demo attempts were mostly inconclusive on this exact wiring, app startup, offline deps, oracle plumbing.)
@@ -462,7 +462,7 @@ suspicion:
proving, with authored (eventually DSPy-authored) demos. Run a bounded, separate test on one tractable target
(mcp-remote) to confirm that the arm demonstrates a real unseen vuln.
## BANKS FIXED-SEARCH MVP (2026-10-12) — "no wrong targets, only wrong searching" demonstrated
## BANKS FIXED-SEARCH MVP (2026-10-12): "no wrong targets, only wrong searching" demonstrated
Record: `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`. Target: banks (the "library" that gave
no engine path findings in the pilot). The target and CPG are unchanged; only the search changes.
@@ -12,7 +12,7 @@ Fixed via **option 1** (filter params to the declared signature) in `src/openult
- `JoernBackend.query()` now drops any param the script does not declare before building the `--param` list
(disposable route) and the in-session named-arg call (session route). This is faithful, not lossy: the
single-request branch of `taint.sc` (line 1542) calls `rowsFor(...)` with top-level params only and lets
`fixedOrigin`/`originAnchors`/`quotedSanitizers`/`guards`/`fieldParameterSources` default — it never could
`fixedOrigin`/`originAnchors`/`quotedSanitizers`/`guards`/`fieldParameterSources` default; it never could
consume them. The batch path keeps carrying them as JSON the script parses itself.
- No-op for `config`/`dominance`: their own `request_params` emit only params their `.sc` declares (verified),
so nothing is dropped. Only `taint` sheds the 5 extras.
@@ -21,12 +21,12 @@ Fixed via **option 1** (filter params to the declared signature) in `src/openult
queries now RUN (no `cpg query taint failed`); baseline = 0 findings, fixed search (annotation) = 2 scored
`path`/`model_entailed` findings (`from_prompt_path` → `prompt_file.write_text(prompt.raw)` L45; `_scan` →
`self._index_path.write_text(...)` L155). Regression test:
`tests/test_cpg_backend.py::test_a_single_query_passes_only_params_the_script_declares` (fake runner, no Joern
— asserts the 5 undeclared params never reach the command line and the query parses).
`tests/test_cpg_backend.py::test_a_single_query_passes_only_params_the_script_declares` (fake runner, no Joern;
asserts the 5 undeclared params never reach the command line and the query parses).
## Summary
Any call on the **single-query** taint path — `cpg.run("taint", request_params(...))` — fails on **both**
Any call on the **single-query** taint path (`cpg.run("taint", request_params(...))`) fails on **both**
of its sub-routes, returns `None`, and is read upstream as "no flows / clean". `request_params()` emits five
keys that `taint.sc`'s `@main def exec(...)` does not declare, and the single-query dispatch passes every
param straight to the script. The **batch** path (`run_batch` → `requestsFile` JSON) is unaffected, which is
@@ -53,10 +53,10 @@ contextFilter, contextPaths, requests, requestsFile, trace`.
`backend.query()` (`src/openultrasast/cpg/backend.py:1266`) dispatches a single query two ways, **both** of
which choke on the undeclared keys:
- **Disposable / CLI route** (lines 1276–1278): appends `--param {key}={value}` for every param. joern errors
- Disposable / CLI route (lines 1276–1278): appends `--param {key}={value}` for every param. joern errors
`Unknown arguments: --fixedOrigin --guards --quotedSanitizers --originAnchors --fieldParameterSources`
(and, as a knock-on, `Missing argument: --cpgFile`). Observed verbatim via `banks_debug.py` on 2026-10-12.
- **Session route** (lines 1321–1326): defines `def ousast_taint(...)` by renaming `@main def exec(` and then
- Session route (lines 1321–1326): defines `def ousast_taint(...)` by renaming `@main def exec(` and then
calls `ousast_taint(key = value, ...)` with named args. The same five keys are not parameters of that
method → the Scala named-arg call fails to compile. (By construction; the rename preserves the exact
`exec` parameter list.)
@@ -77,14 +77,14 @@ Unaffected:
- The push/scan **batch** path: `scan.py` uses `run_batch("taint", …)` (scan.py:1259, 1264, 1387). This is
why the engine-on pilot produced results.
- `config` and `dominance` single-queries (`config_value.py:46`, `dominance.py:54`) — their `.sc` signatures
- `config` and `dominance` single-queries (`config_value.py:46`, `dominance.py:54`): their `.sc` signatures
are presumed to match what their specs emit (confirm separately; not exercised here).
Open question — **CLOSED by call-graph trace (2026-10-12).** No shipped product path and no in-repo
Open question: **CLOSED by call-graph trace (2026-10-12).** No shipped product path and no in-repo
benchmark/test reached the real single-query `JoernBackend.query("taint")`; the bug was latent but unreached
by the product. Traced:
- **Shipped push/scan/replay** (`cli.py:1334` → `scan_repository` → `_scan_repository_impl`) is DOUBLE
- Shipped push/scan/replay (`cli.py:1334` → `scan_repository` → `_scan_repository_impl`) is DOUBLE
shielded: Phase 2 fetches taint via the **batch route** (`run_batch`, always wired by `build()`), and
Phase 3's `scan_region` (scan.py:559) runs against a **`_prefetched` CpgResult** whose `run` ignores the
query and returns the already-fetched batch rows (scan.py:1472). So `scan_region → taint_verdict →
@@ -105,7 +105,7 @@ The fix restores the single-query path for pair/direct use without altering push
## Evidence
- `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/` — the batch path works (11 flow rows,
- `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`: the batch path works (11 flow rows,
`write_text` sinks); the v1 single-query attempt returned 0/0 on `cpg query taint failed`.
- Raw joern error captured by `banks_debug.py` (scratchpad): `Missing argument: --cpgFile; Unknown
arguments: --fixedOrigin --guards --quotedSanitizers --originAnchors --fieldParameterSources`.
+8 -8
View File
@@ -1,4 +1,4 @@
# Brief: finding-feedback-loop — what happens after a finding is judged wrong
# Brief: finding-feedback-loop, what happens after a finding is judged wrong
**Status:** brief only, unapproved. Successor scope lifted out of `contributor-scan`, which is mid-implementation
and does not cover this.
@@ -7,13 +7,13 @@ and does not cover this.
Two task groups were drafted into `contributor-scan/tasks.md` on 2026-09-09 without passing through
requirements or design. That was a workflow error: they are not refinements of scan integration, they are a
different feature — a loop that learns from findings a human rejected — and writing them as tasks skipped the
different feature (a loop that learns from findings a human rejected) and writing them as tasks skipped the
two phases that decide whether the feature is shaped right. They are lifted here to be specified properly.
## The gap this exists to close
Nine engine defects were found in one session against two repositories. **Three of them were the same bug in
three different matchers**: a spec token matched as a substring rather than as a word — `resolve` inside
three different matchers**: a spec token matched as a substring rather than as a word: `resolve` inside
`resolveUrl` in the sanitizer matcher, `user` inside `users` in the discharger matcher, `gets` inside `fgets`
in the sink matcher. Each was found by a measurement, fixed locally, and left no trace. The third survived
until a 89k-line C library made it visible as 26 identical false positives.
@@ -60,17 +60,17 @@ sound. The same machinery writing sink tables is the thing that produced the lea
## What the shape looks like
* **Per-repository, declarative, committed by the developer.** A dismissal file in the *target* repository —
site, rung, reason, author — read by the tool and never written by it, the same rule the declared policy
* **Per-repository, declarative, committed by the developer.** A dismissal file in the *target* repository (
site, rung, reason, author), read by the tool and never written by it, the same rule the declared policy
file already follows. Reviewed like code, diffable, and a dismissal without a reason is how a suppression
list becomes a graveyard nobody can audit.
* **Dismissals are labels, not suppressions.** With the known vulnerabilities a `benchmarks/repos/` recipe
already declares, a dismissal file gives a per-repository evaluation set with *both* classes in it — which
already declares, a dismissal file gives a per-repository evaluation set with *both* classes in it, which
is what an optimiser needs and what a suppression list normally throws away.
* **A ledger for the maintainer.** Counter-examples grouped by what they share (same token, same matcher
clause, same witness shape), so a recurring cause is visible as a cause rather than as five separate bugs.
* **Split-respecting adoption.** Train split only, holdout never read, the three gates byte-identical, and
precision *and* recall reported on both sides — an adoption that quietly trades recall is worse than none.
precision *and* recall reported on both sides; an adoption that quietly trades recall is worse than none.
## The acceptance test for the model's role
@@ -80,7 +80,7 @@ recover a cause we already know should not be trusted to propose one we do not.
## Known limits to carry forward
* The pair corpus could not see any of the nine defects; whole-repository measurement is what found them, and
this loop inherits that. Its evaluation set has to include repositories, not only pairs.
this loop inherits that. Its evaluation set has to include repositories as well as pairs.
* Only two repositories exist today (one Python, one C), and the C one has never produced a true positive.
An optimiser tuned on that evidence base would be tuned on very little.
* `contributor-scan` Req 8 already warns about the +16% overfitting gap between the slice a change was
+10 -10
View File
@@ -10,7 +10,7 @@ anything.
The design turns on one asymmetry: **two of those three bugs needed no judgement at all.** Once the first was
understood, the other two were the same fix carried across. So the automation that pays is *propagation and
memory*, and the model is confined to the one step that genuinely needs inference — naming what a group has in
memory*, and the model is confined to the one step that genuinely needs inference: naming what a group has in
common.
### Goals
@@ -22,7 +22,7 @@ common.
### Non-Goals
- **Automated fact-table edits.** Not deferred — prohibited. See Boundary Commitments.
- **Automated fact-table edits.** Prohibited rather than deferred. See Boundary Commitments.
- Suppressing findings silently. A dismissal makes a finding *labelled*, never invisible.
- A general-purpose optimiser. The only surfaces evolved are two prompt constants.
- Replacing the pair corpus. It is the holdout that keeps per-repository tuning honest.
@@ -38,10 +38,10 @@ common.
### Out of Boundary
- `semantic/facts`, `semantic/obligations`, `ruleset/`, the declared policy file — **no automated writer, by
- `semantic/facts`, `semantic/obligations`, `ruleset/`, the declared policy file: **no automated writer, by
any path, at any time.** This is the boundary the whole design exists to respect.
- The three gates. They are the invariant an adoption is measured against, not a thing this spec changes.
- `model/taint.py`, `dominance.py`, `config_value.py` — arbiters are fixed here; a proposal that would change
- `model/taint.py`, `dominance.py`, `config_value.py`: arbiters are fixed here; a proposal that would change
one is a proposal for a human, not an edit.
### Allowed Dependencies
@@ -64,7 +64,7 @@ Three pieces already exist and are reused rather than rebuilt:
- **`benchmarks/repos/*.toml`** declares known vulnerabilities per checkout, with `in_scope` stating whether a
family can decide one at all. That is the positive half of an evaluation set, already committed.
- **`pairs.py`** already carries a train/holdout split and the machinery that respects it. The leak of
2026-09-05 was in the *lever*, not the splitter — `semantic/loo.evaluate_loo` was honest throughout.
2026-09-05 was in the *lever*, not the splitter; `semantic/loo.evaluate_loo` was honest throughout.
- **`redaction.py`** is applied to every prompt and persisted trajectory, and applies here unchanged.
### Boundary Map
@@ -87,21 +87,21 @@ benchmarks/repos/*.toml ── read ────▶ evaluation set
**A dismissal is a label, not a suppression.** A suppressed finding disappears and its reason rots; a
labelled one is reported as dismissed with its reason, and a dismissal that stops matching is reported too.
This is what turns a suppression list into the negative half of an evaluation set — the thing that makes the
This is what turns a suppression list into the negative half of an evaluation set, the thing that makes the
rest of the feature possible.
**The tool never writes the dismissal file.** Same rule as the declared policy file, for the same reason: a
file the tool edits is a file no reviewer trusts. It is committed by a developer and reviewed like code.
**Grouping is deterministic; only the hypothesis is not.** Entries are grouped by shared spec token, matcher
clause, family and witness shape — all mechanical. The model is asked one question about a group it did not
clause, family and witness shape; all mechanical. The model is asked one question about a group it did not
choose, and its answer goes to a queue. This keeps the inference confined to the step that needs it.
**Evolution targets the question, never the facts.** A prompt is how we ask; a fact table is what we claim.
`CANDIDATE_QUESTION` and `residual_question` are already the LLM-facing surface and are the only mutable
targets. Enforced by a test that fails if any writer touches the fact modules, not by convention.
**The holdout is never read.** Not "should not" — the evaluation-set API has no path to it during
**The holdout is never read.** The evaluation-set API has no path to it during
optimisation, and a test asserts the absence. The 2026-09-05 leak was possible because reading it was merely
discouraged.
@@ -123,9 +123,9 @@ benchmarks/ledger/
### Changed
- `cli.py` — a `feedback` subcommand: `ledger`, `groups`, `propose`, `evolve`. Maintainer surface; `scan`
- `cli.py`: a `feedback` subcommand: `ledger`, `groups`, `propose`, `evolve`. Maintainer surface; `scan`
gains only the dismissal read.
- `reports.py` — a dismissed finding is rendered as dismissed with its reason.
- `reports.py`: a dismissed finding is rendered as dismissed with its reason.
### Unchanged and load-bearing
@@ -5,22 +5,22 @@
## Introduction
Nine engine defects were found in one session against two repositories. **Three were the same bug in three
matchers** — a spec token matched as a substring rather than a word: `resolve` inside `resolveUrl` in the
matchers**: a spec token matched as a substring rather than a word: `resolve` inside `resolveUrl` in the
sanitizer matcher, `user` inside `users` in the discharger matcher, `gets` inside `fgets` in the sink matcher.
Each was found by a measurement, fixed locally, and left no trace; the third survived until an 89k-line C
library made it visible as 26 identical false positives.
Nothing in the system remembers that a finding was wrong. There is no artifact saying *this site, at this
rung, is not a defect, and here is why* — so a class recurs, and each recurrence costs another measurement.
rung, is not a defect, and here is why*, so a class recurs, and each recurrence costs another measurement.
Two audiences want that artifact for different reasons. A maintainer wants the *propagation* automated: two of
the three matcher bugs needed no judgement at all, only the fix carried across. A developer wants to dismiss a
finding in their own repository, have it stay dismissed, and have the reason reviewable beside the code — a
finding in their own repository, have it stay dismissed, and have the reason reviewable beside the code; a
tool that cannot be told it is wrong is one they stop running.
This feature is bounded by a rule the project has held from the start and a leak that shows why. An optimiser
may evolve **the question** — the judge and residual prompts, which are how we ask. It may never author **the
facts** — tables, rulesets, the declared policy file, which are what we claim. On 2026-09-05 the improve lever
may evolve **the question**: the judge and residual prompts, which are how we ask. It may never author **the
facts**: tables, rulesets, the declared policy file, which are what we claim. On 2026-09-05 the improve lever
admitted candidate shapes because they "recover a currently missed holdout pair"; five of eleven were taught by
the holdout pairs they then recovered, and a holdout Youden of **−0.059 was reported as +0.059**.
@@ -38,11 +38,11 @@ recorded and my colleagues can see why.
#### Acceptance Criteria
1. The system shall read dismissals from a declarative file in the **target** repository, and shall never
write to it — the same rule the declared policy file already follows.
write to it, the same rule the declared policy file already follows.
2. A dismissal shall carry the site it concerns, the rung it was dismissed at, a reason, and an author.
3. The system shall reject a dismissal that carries no reason, because a suppression list without reasons is
a graveyard nobody can audit.
4. The system shall report a dismissed finding as dismissed, with its reason, rather than omitting it — so a
4. The system shall report a dismissed finding as dismissed, with its reason, rather than omitting it, so a
stale dismissal is visible rather than silent.
5. When a dismissal no longer matches any finding, the system shall say so, because a dismissal that has
stopped applying is either a fixed bug or a moved target and both are worth knowing.
@@ -54,20 +54,20 @@ cause is visible as a cause rather than as five separate bugs.
#### Acceptance Criteria
1. When a finding is contradicted — by a dismissal, by the judge with coverage, or by a declared contract —
1. When a finding is contradicted (by a dismissal, by the judge with coverage, or by a declared contract)
the system shall record the site, the rung, the witness, the spec tokens that produced it, and the commit.
2. Each ledger entry shall be runnable as a regression case: this site, at this commit, shall not be reported
at that rung.
3. The ledger's regression cases shall run offline in CI, so that re-introducing a fixed cause fails a test
rather than a repository scan four weeks later.
4. The system shall group ledger entries by what they share — spec token, matcher clause, family, witness
shape — and report the groups by size.
4. The system shall group ledger entries by what they share (spec token, matcher clause, family, witness
shape) and report the groups by size.
5. The ledger shall be committed evidence, not a local artifact.
### Requirement 3: A model may propose a cause, and only propose it
**Objective:** As a maintainer, I want the model to do the part that needs judgement — naming what a group of
counter-examples has in common — without it deciding anything.
**Objective:** As a maintainer, I want the model to do the part that needs judgement (naming what a group of
counter-examples has in common) without it deciding anything.
#### Acceptance Criteria
@@ -132,7 +132,7 @@ quietly loses recall is worse than none.
1. The evaluation set shall include whole repositories, not pairs alone: the pair corpus could not see any of
the nine defects that motivate this feature, and the gates stayed byte-identical through every fix.
2. The system shall state how many repositories its evidence base contains, because two — one of which has
never produced a true positive — is very little to tune against.
2. The system shall state how many repositories its evidence base contains, because two (one of which has
never produced a true positive) is very little to tune against.
3. Where an adoption is measured on one repository only, the system shall label it as such rather than
reporting it as a general improvement.
+53 -53
View File
@@ -1,10 +1,10 @@
# Brief: flow-aware-ranking — spend the expensive query where the cheap evidence is strongest
# Brief: flow-aware-ranking, spend the expensive query where the cheap evidence is strongest
> **Status 2026-10-02:** dormant. Brief only and unapproved; its 2026-09-12 amendment placed ranking inside the pre-push safety net, where the ranker now lives.
**Status:** brief only, unapproved.
## Current amendment — 2026-09-12
## Current amendment, 2026-09-12
This section supersedes forward-looking interpretations below; the iteration history is retained.
The product goal is a scalable, actionable pre-push security safety net as AI accelerates code changes.
@@ -23,7 +23,7 @@ need explicit summaries or unresolved coverage, never automatic vendor reintrodu
- Latest recorded phase-2 budgets: PMPro **82**, WP Statistics **91**, MW WP Form **54**, VAmPI **16**.
The `<50` gate remains unmet. The evidence passes recorded no unanswered pairs. These are stored
ranking measurements, not fresh detection/precision results or a pre-push latency demonstration.
- **Equal-score ties are not the whole remaining problem.** At fixed scores, perfect tie-breaking
- Equal-score ties are not the whole remaining problem. At fixed scores, perfect tie-breaking
still needs at least **68** regions for PMPro and **86** for WP Statistics (67/85 strictly
higher-scored same-tier regions precede the targets). An LLM restricted to equal-score ties
cannot pass `<50`. Within-tier reordering across scores or a new feature is a distinct experiment.
@@ -95,7 +95,7 @@ Three regions from one plugin, with what the ranker sees today and what is actua
| `pmpro_rest_api_get_membership_level` | **0.80** | none — no SQL sink at all |
A region with no sink outranks two holding raw concatenated SQL. The discriminating signal is in the sink
call's own source text — interpolated versus bound — and nothing in the ranker reads it.
call's own source text (interpolated versus bound), and nothing in the ranker reads it.
One detail decides whether a keyword rule suffices, and it does not: `esc_sql` IS in scope for the CVE's
function, applied to a different statement. "A sanitizer is present, rank it lower" demotes precisely the
@@ -122,8 +122,8 @@ finding**, it decides what is looked at, and `regions_truncated` already reports
Each phase ships only if it beats the previous one **on a repository the change was not developed against**.
A phase that cannot show that is not an improvement, it is a story.
- **Phase 0 — the oracle and the baseline.** The evidence extractor (no model) and a scoring harness over the
pinned CVEs, reporting each one's rank position. Gate: reproduces today's numbers — CVE-2023-23488 at
- Phase 0, the oracle and the baseline. The evidence extractor (no model) and a scoring harness over the
pinned CVEs, reporting each one's rank position. Gate: reproduces today's numbers, CVE-2023-23488 at
position 390 of 4,463, 83.8% of regions in one tier. Without a baseline nothing later is measurable.
**Done 2026-09-10.** `benchmarks/ranking/position.py`, committed as
@@ -138,36 +138,36 @@ A phase that cannot show that is not an improvement, it is a story.
**And this corrects the cross-assessment above.** "Ranking's job is concentration, not inclusion" was
measured on `pmpro` alone, where the CVE is inside the budget. On the other two plugins the CVE sits at
512 and 503 — **outside a 500-region budget** — so the current ranker never examines it at all, and the
512 and 503, **outside a 500-region budget**, so the current ranker never examines it at all, and the
slices found those CVEs only because a slice has too few regions to exclude anything. Both jobs are
real: inclusion on two repositories, concentration on all three. The single metric `budget_at_recall`
covers both, which is why it is the one that matters.
The instrument is `evidenceOnly` mode on the taint query, recorded on every scan as `ModelScanResult.tiers`
and `tier_counts`, and acted on by nothing — so every later phase is measured against a record made by
and `tier_counts`, and acted on by nothing, so every later phase is measured against a record made by
scans that did not know they were being measured.
**Measured cost of the instrument**, 637-file graph, the scan's own 500-region request set:
2,500 pairs in 98.9 s including JVM load — about 40 ms per pair
2,500 pairs in 98.9 s including JVM load, about 40 ms per pair
tier 0 (no sink of that family in reach): 2,161 of 2,500 = 86%
Two attempts before that ran past an hour each without producing a payload, and the cause was mine both
times: repository-wide walks — the sink scan, the reachable-method BFS, `familyInRepo` — repeated per
times: repository-wide walks (the sink scan, the reachable-method BFS, `familyInRepo`) repeated per
request. Memoising them across the batch is what made "milliseconds per pair" true rather than claimed.
**And the same memoisation reaches the dataflow mode.** The identical 250-request set that cost 2,026.3 s
(8,105 ms per request) with both joins on now costs **287.0 s — 1,148 ms per request — for the same 5,779
(8,105 ms per request) with both joins on now costs **287.0 s (1,148 ms per request) for the same 5,779
rows.** Most of what contributor-scan 5.13 measured as "the base taint query" was the per-request scans,
not the dataflow. That correction is recorded there.
- **Phase 1 — tier 0 only, no model.** Exact pruning. Gate: **53% fewer taint requests issued, measured** —
- Phase 1, tier 0 only, no model. Exact pruning. Gate: **53% fewer taint requests issued, measured**,
and nothing more. 1,177 requests at 6.63 s is still 2.2 hours, so this phase does NOT make the scan finish;
it halves the problem. Claiming completion here would send someone looking for a bug where there is none.
**Done 2026-09-10, gate exceeded, scan still does not finish — and the reason corrects phase 0's cost
**Done 2026-09-10, gate exceeded, scan still does not finish, and the reason corrects phase 0's cost
figure.** On the 637-file plugin: `tier_counts = {0: 2161, 1: 60, 2: 20, 3: 211, 4: 48}`, **2,161 of 2,500
requests pruned (86%)**, evidence pass 94.5 s. The remaining **339 requests still hit the 2,400 s ceiling**
— more than 7 s each.
requests pruned (86%)**, evidence pass 94.5 s. The remaining **339 requests still hit the 2,400 s ceiling**,
more than 7 s each.
**The scan finishes, 2026-09-11.** Not because of anything in this spec: contributor-scan 5.13 found
that every query batch had been recomputing Joern's dataflow overlay at load since PHP builds went
@@ -188,12 +188,12 @@ A phase that cannot show that is not an improvement, it is a story.
which 88% were tier 0 and returned instantly; the ~30 real requests in it cost about 9.5 s each. Pruning
removes exactly the cheap questions and keeps the expensive ones, so it cannot make those faster. The
honest per-request cost of a taint question that *can* have an answer is **7–10 s at `callDepth = 3`**,
and that is the number phase 2 and 5.13 must work against — not 1.1 s, and not 6.6 s either.
- **Phase 2 — heuristic tiers, still no model.** The vector and tiers defined below, ordered within a tier
by a published weighted sum. Gate: `budget_at_recall` — the smallest K holding every pinned CVE — **below
and that is the number phase 2 and 5.13 must work against, not 1.1 s, and not 6.6 s either.
- Phase 2, heuristic tiers, still no model. The vector and tiers defined below, ordered within a tier
by a published weighted sum. Gate: `budget_at_recall` (the smallest K holding every pinned CVE) **below
50 on every pinned repository**, and `pmpro`'s taint query completing at that K. This is the phase that
makes a scan finish, because the job is concentration: the CVE is already inside 500, the question is
whether it is inside 50. **If this passes, the model is a refinement, not the plan** — and knowing that is
whether it is inside 50. **If this passes, the model is a refinement, not the plan**, and knowing that is
worth more than assuming it.
**Implemented 2026-09-11, measurement in flight.** `Evidence.score` is the published weighted sum, and
every weight is in one table in `model/evidence.py`: per open sink +1.0 (capped at 5), per cleansed or
@@ -304,17 +304,17 @@ A phase that cannot show that is not an improvement, it is a story.
facts and a weight table can do; the ties are phase 3's. After that, within-tier order is phase 3's -- a model reading the code the ties cannot --
or one more exact path fact (`distance`), and the gate stays unmet until one of them moves it.
- **Phase 3 — the LLM ranker.** Same evidence, ordered by a model. Gate: beats phase 2 leave-one-repository-out.
- Phase 3, the LLM ranker. Same evidence, ordered by a model. Gate: beats phase 2 leave-one-repository-out.
If it does not, phase 2 ships and phase 3 does not.
- **Phase 4 — break the closed loop.** An exploration slice (~10% of budget sampled outside the top-K) and
- Phase 4, break the closed loop. An exploration slice (~10% of budget sampled outside the top-K) and
full labelling on repositories small enough to arbitrate every region. Gate: a measured estimate of the
selection bias in phase 3's numbers.
- **Phase 5 — prompt optimisation (GEPA / SIMBA / DSPy).** Only after 3 and 4. An optimiser run against a
- Phase 5, prompt optimisation (GEPA / SIMBA / DSPy). Only after 3 and 4. An optimiser run against a
censored metric optimises the censorship.
## What the prompt has to carry
The question put to the model is **not** "is this vulnerable" — that is adjudication, it is the arbiter's job,
The question put to the model is **not** "is this vulnerable". That is adjudication, the arbiter's job,
and it is the one thing a ranker must never be asked. It is:
> Given a fixed budget of expensive dataflow queries, which of these sites are worth spending it on?
@@ -326,14 +326,14 @@ reachable from a request handler.
## Constraints, from measurement and from prior scars
- **Cost.** One model call per region is 4,463 calls a scan. Batch over a prefilter, or it does not ship.
- **The loop is closed.** A region the ranker excludes is never arbitrated, never labelled, so the ranker
- Cost. One model call per region is 4,463 calls a scan. Batch over a prefilter, or it does not ship.
- The loop is closed. A region the ranker excludes is never arbitrated, never labelled, so the ranker
never learns it was wrong. Every run confirms the ordering it already had.
- **N projects.** Four pinned CVEs is not a training set. This project already has the closed-loop
- N projects. Four pinned CVEs is not a training set. This project already has the closed-loop
train-on-test scar; the reported figure must come from a repository the optimiser never saw.
- **Determinism.** Two runs of one repository must agree or a baseline diff is meaningless. The ranking is
- Determinism. Two runs of one repository must agree or a baseline diff is meaningless. The ranking is
recorded in the manifest.
- **Degradation.** With no model configured, phases 0–2 still rank, and the report says which ranker ordered
- Degradation. With no model configured, phases 0–2 still rank, and the report says which ranker ordered
the scan.
## Cross-assessment against the measurements
@@ -342,7 +342,7 @@ Read against what was actually measured today, the plan above has two errors and
here, corrected, rather than silently rewritten, because the corrections are themselves results.
**Phase 1's gate is wrong.** It says pruning makes `pmpro`'s taint query complete. It does not. Pruning
removes 53% of requests, and 1,177 × 6.63 s is still **2.2 hours** — nowhere near the 300 s default or the
removes 53% of requests, and 1,177 × 6.63 s is still **2.2 hours**, nowhere near the 300 s default or the
2,400 s that was tried. Pruning halves the problem; it does not solve it. The honest gate is "53% fewer
requests issued, measured", and completion belongs to phase 2. Stating otherwise would have sent someone
looking for a bug in phase 1 when phase 1 had done exactly what it can.
@@ -356,8 +356,8 @@ the numbers it had; the numbers were the instrument's.
**Ranking's job is concentration, not inclusion.** The brief implies the CVE is missed because the ranker
excluded it. It did not: CVE-2023-23488's region sits at position 390 of 4,463, inside the 500-region budget,
and was examined. It goes unreported because the budget is spent on 500 regions when 50 would do, and 500
is unaffordable at 6.63 s per family per region. So the ranker's success metric is not "is the CVE in the
budget" — it already is — but **"how small can the budget be while the CVE stays in it"**. That is a
is unaffordable at 6.63 s per family per region. So the ranker's success metric is
**"how small can the budget be while the CVE stays in it"**; the CVE is already in the budget. That is a
different objective, and it is the one that makes a scan finish.
**The gap: nothing above says what the evidence IS.** "Sink shape" and "cleansing on the call" are named,
@@ -366,24 +366,24 @@ check for language leakage. The rest of this brief defines it.
Two things the measurements confirm and the plan should lean on harder:
- **The joins are cheap and must survive any ranker.** 22% of query cost, two of three CVEs. A ranker that
- The joins are cheap and must survive any ranker. 22% of query cost, two of three CVEs. A ranker that
demotes field- or hook-carried regions because they show no direct source is a ranker that loses those
CVEs. The evidence vector below therefore carries the join's own stage-one result as a field.
- **84% of regions in one tier means any separation is progress**, and the separation does not need to be
- 84% of regions in one tier means any separation is progress, and the separation does not need to be
clever to be large. A tier 0 that is merely exact already reorders the budget more than the current ranker
ever has.
## The evidence vector, defined
The unit is a **(region, family) pair**. For each, a fixed vector of fields is computed. Every field is
defined in terms of **fact-table categories** — source, sink, sanitizer, dispatch, access — never in terms of
defined in terms of **fact-table categories** (source, sink, sanitizer, dispatch, access), never in terms of
a language's syntax, which is what makes the definition language-agnostic: any language with a fact table
gets the same vector, and a heuristic that mentions `$wpdb` or `wp_ajax_` has by construction left the
vector and must be rejected in review.
**Where it is computed matters as much as what it is.** The graph is already built for the arbiter, at 76 s
for 637 files. Extracting these fields is a CPG query **without dataflow** — call nodes, their arguments,
their enclosing methods, the call graph one hop out — which is milliseconds per call rather than 6.63 s per
for 637 files. Extracting these fields is a CPG query **without dataflow** (call nodes, their arguments,
their enclosing methods, the call graph one hop out), which is milliseconds per call rather than 6.63 s per
request. The extractor is therefore a sibling of `taint.sc` that never calls `reachableByFlows`:
> **Tiering is the CPG query without dataflow. Arbitration is the CPG query with it. Same graph, same
@@ -393,7 +393,7 @@ That is the whole architecture in one line, and it is why the evidence is langua
merely language-neutral-in-intent: the CPG normalises syntax across frontends, so "the argument of this call
is a literal" means the same thing in PHP, Python and C.
### Sink evidence — from `[[sink]]` facts of family F
### Sink evidence, from `[[sink]]` facts of family F
| field | definition | exactness |
|---|---|---|
@@ -401,7 +401,7 @@ is a literal" means the same thing in PHP, Python and C.
| `shape` | per sink call, one of: **`literal`** (the vulnerable-position argument is a constant), **`bound`** (the call is a declared-safe form — a `parameterized` sanitizer fact wraps it, or its arity/shape matches the bound form the taint query already recognises), **`computed`** (the argument is an expression: concatenation, interpolation, a variable, a call result) | exact for `literal` and `bound`; `computed` is the residue |
| `cleansed_on_call` | per sink call, whether a sanitizer of F is an **ancestor of the vulnerable-position argument in the AST** — not merely present in the same function | exact, and this is the field a text scan gets wrong: `esc_sql` in scope on a different statement is `false` here |
### Source evidence — from `[[source]]` facts and the entry-point contract
### Source evidence, from `[[source]]` facts and the entry-point contract
| field | definition | exactness |
|---|---|---|
@@ -411,7 +411,7 @@ is a literal" means the same thing in PHP, Python and C.
| `access` | the declared access level, **with provenance**: `declared` (a registration or route contract says so) or `inferred` (nothing found) — because "public" meaning "no decorator found" is the false positive the project has already paid for | exact |
| `carried` | the region reads a **field** whose assignment is fed by a source anywhere in the file, or a **dispatch** value whose registered producer is — i.e. stage one of the two-stage join, which is already computed and memoised, reused here as evidence | exact where computed; absent (not false) where the join was not run |
### Path evidence — cheap structural proxies for what dataflow would establish
### Path evidence, cheap structural proxies for what dataflow would establish
| field | definition |
|---|---|
@@ -438,14 +438,14 @@ is a literal" means the same thing in PHP, Python and C.
Two properties are load-bearing and must hold in every implementation:
1. **Only tier 0 excludes; every other tier orders.** So the only place exactness is *required* is tier 0,
1. Only tier 0 excludes; every other tier orders. So the only place exactness is *required* is tier 0,
and tier 0's predicate has a proof: a family with no sink in reach cannot produce a flow, in any language,
under any dataflow model. Above tier 0, being wrong costs position, and `regions_truncated` reports what
position cost.
2. **Tier 0's scope must match the arbiter's.** At `callDepth = 0` the scope is the region; at depth k it is
2. Tier 0's scope must match the arbiter's. At `callDepth = 0` the scope is the region; at depth k it is
the region plus its reachable methods. A tier 0 computed on the file when the arbiter will look k hops
out is not exact and must not be called so. The repository-wide form (`family_in_repo == false`) is exact
at every depth and is the cheap first cut — it alone removes all 487 `deserialization` requests on `pmpro`.
at every depth and is the cheap first cut. It alone removes all 487 `deserialization` requests on `pmpro`.
**Within a tier**, order by a score. In phase 2 that score is a fixed weighted sum over the vector, published
in the manifest so two runs agree. In phase 3 it is the model's ordering **of the same vector**, which is
@@ -456,23 +456,23 @@ what makes the two comparable: same input, different ordering function, one held
The vector above is exact wherever exactness was cheap. What remains genuinely uncertain is narrower than
"ranking":
- **Ambiguous cleansing.** `cleansed_on_call` is exact when the sanitizer wraps the argument directly. It is
`false` — correctly, but unhelpfully — when cleansing happens two statements earlier into a variable that is
- Ambiguous cleansing. `cleansed_on_call` is exact when the sanitizer wraps the argument directly. It is
`false` (correctly, but unhelpfully) when cleansing happens two statements earlier into a variable that is
then interpolated. Deciding whether *that* cleansing reaches *this* call is reading the code, and it is the
single largest source of tier-2/tier-3 misplacement.
- **Unfamiliar shapes.** A sink call whose form the `bound` rule does not recognise, in a framework with no
- Unfamiliar shapes. A sink call whose form the `bound` rule does not recognise, in a framework with no
facts yet. The model can say "this looks bound" as an *ordering* opinion; it cannot add a `bound` fact,
because a fact is policy.
- **Within-tier priority** when the fixed score ties, which on a real repository it will, constantly.
- Within-tier priority when the fixed score ties, which on a real repository it will, constantly.
And the boundaries:
- **The model never computes tier 0.** Exclusion is exact or it is a silent miss, and a model's "no sink
- The model never computes tier 0. Exclusion is exact or it is a silent miss, and a model's "no sink
here" is not a proof.
- **The model never sees the question "is this vulnerable".** It sees "which of these is worth an expensive
- The model never sees the question "is this vulnerable". It sees "which of these is worth an expensive
query", with the vector and the sink call's source text. Adjudication is the arbiter's, and keeping it
there is what keeps the gates byte-identical.
- **The model's input is the vector plus text, never the arbiter's verdict** on the same run — that is the
- The model's input is the vector plus text, never the arbiter's verdict on the same run. That is the
closed loop feeding on itself.
## Metrics, so every phase can fail
@@ -498,8 +498,8 @@ answer is visible:
1. ~~Where does the evidence extractor live?~~ **It is a CPG query without dataflow, a sibling of `taint.sc`
in `cpg/queries/`.** Not the mapper: the mapper reads text and would have to re-derive per language what
the graph already normalises. The vector is language-agnostic *because* the graph is.
2. ~~Per file or per reachable set?~~ **Both, in that order.** `family_in_repo` first — exact at any depth,
free, a fifth of the budget on `pmpro` — then the reachable set at the arbiter's own `callDepth`. Never the
2. ~~Per file or per reachable set?~~ **Both, in that order.** `family_in_repo` first (exact at any depth,
free, a fifth of the budget on `pmpro`), then the reachable set at the arbiter's own `callDepth`. Never the
file alone above depth zero, because that is not exact and must not be called tier 0.
3. How many pinned repositories before leave-one-out means anything? Four is not enough. This is the
corpus ask, and it is the same ask contributor-scan 5.8 made: plugins chosen by installed base, CVEs read
@@ -507,7 +507,7 @@ answer is visible:
4. Does the arbiter's verdict feed back as a label? **Only through the exploration slice.** A verdict on a
region the ranker chose is a label the policy selected, and training on it closes the loop. A verdict on
an exploration region is the only unbiased label the scan produces, and it is the one worth keeping.
5. **New:** the weighted sum in phase 2 has weights. Where do they come from before any model exists?
5. New: the weighted sum in phase 2 has weights. Where do they come from before any model exists?
Proposed: set by hand from the tier semantics, published in the manifest, and treated as a baseline for
phase 3 to beat — not tuned, because tuning four weights on four CVEs is the overfit the plan exists to
phase 3 to beat, not tuned, because tuning four weights on four CVEs is the overfit the plan exists to
avoid.
+16 -16
View File
@@ -1,4 +1,4 @@
# Brief: ir-consolidation — stop maintaining an IR we did not need, and adopt one we do
# Brief: ir-consolidation, stop maintaining an IR we did not need, and adopt one we do
> **Status 2026-10-02:** dormant. Brief only and unapproved. The subtraction it asks for still needs a measured recall comparison before the flat IR goes.
@@ -12,16 +12,16 @@
Two different questions live inside that, and they have opposite answers. Keeping them apart is most of the
value of this brief.
## Question 1 — the source IR we already replaced and never retired
## Question 1, the source IR we already replaced and never retired
`semantic/` is **3,218 lines, 16% of the codebase**: a hand-written IR (`ir.py`, `cst.py`) and a flat taint
engine (`taint.py`), plus the overlay that runs them. `model-grounded-detection` measured that IR against
Joern's CPG and the result was decisive — **2.2% entailment against 41.3%** — which is why the CPG was
Joern's CPG and the result was decisive, **2.2% entailment against 41.3%**, which is why the CPG was
adopted for the model layer. The flat IR was never removed, and still runs as a proposer in every scan.
So the outgrowth is not something we are about to add. It is something already carried.
**Vine and P-Code do not help here.** Both are *binary* IRs — Vine lifts executables (BitBlaze, UC Berkeley,
**Vine and P-Code do not help here.** Both are *binary* IRs. Vine lifts executables (BitBlaze, UC Berkeley,
GPL-2.0, last significant activity 2009), and P-Code lifts machine code. Our source languages are Python,
PHP, JavaScript and Java, and the source IR we should adopt is the one we already have. The work is
subtraction, not substitution:
@@ -32,16 +32,16 @@ subtraction, not substitution:
A candidate ~1,000-line deletion with a number attached, which is the only kind worth doing.
## Question 2 — the abstraction we are missing, where an existing IR is exactly right
## Question 2, the abstraction we are missing, where an existing IR is exactly right
`c/memory` has produced **zero true positives** and three distinct classes of false one. The reason is
structural and now stated in the report (Req 5.6): it models unbounded string CALLS, while real C overflows
are *arithmetic* — an index, a pointer offset, a `memcpy` whose size is computed. Every one of libpng's is,
are *arithmetic* (an index, a pointer offset, a `memcpy` whose size is computed). Every one of libpng's is,
and its library code never calls a modelled string API at all.
"Can this expression exceed this allocation" is a value question, not a reachability one. **Rolf Rolles'
GhidraPAL carries a three-valued-logic abstract interpreter over Ghidra P-Code** — each bit definite 0,
definite 1, or unknown, with abstract transformers for `INT_ADD`, `INT_MULT`, `INT_AND`, the comparisons —
GhidraPAL carries a three-valued-logic abstract interpreter over Ghidra P-Code**, with each bit definite 0,
definite 1, or unknown, and with abstract transformers for `INT_ADD`, `INT_MULT`, `INT_AND`, the comparisons,
validated against concrete execution in its own tests. That is precisely the shape of analysis the C family
lacks, and it exists.
@@ -55,13 +55,13 @@ lacks, and it exists.
**angr is the strongest candidate and Vine is not one.** Three things decide it:
* **It is Python.** This codebase is Python with `dependencies = []` in the core, and every heavyweight
* It is Python. This codebase is Python with `dependencies = []` in the core, and every heavyweight
capability so far has been reached across a subprocess boundary rather than imported. angr fits that
pattern as an extra, the way `harnessx` and the tree-sitter grammars already do.
* **Value-set analysis is the abstraction we are missing**, by name. "Can this index exceed this allocation"
* Value-set analysis is the abstraction we are missing, by name. "Can this index exceed this allocation"
is a VSA question, and VSA is abstract interpretation rather than SMT -- which is the choice
`model-grounded-detection` already made and gave its reasons for.
* **BSD-2-Clause**, against Vine's GPL-2.0. This project has a standing rule about never vendoring code
* BSD-2-Clause, against Vine's GPL-2.0. This project has a standing rule about never vendoring code
without a stated licence, and a GPL dependency is a decision about the whole tool rather than about one
analysis.
@@ -71,19 +71,19 @@ thing to read before writing any abstract domain here, whether the platform unde
It is also the honest answer to a decision this project already took. `model-grounded-detection` chose
abstract interpretation over SMT, and chose to build the model from code because execution cannot be
assumed. A bit-level abstract interpreter over P-Code is *still abstract interpretation* and still static —
what changes is the artifact: it needs a **built** library rather than its source.
assumed. A bit-level abstract interpreter over P-Code is *still abstract interpretation* and still static.
What changes is the artifact: it needs a **built** library rather than its source.
## What this brief must decide
1. Whether the source-IR subtraction is safe, measured rather than assumed.
2. Whether the C arithmetic class is worth a second pipeline at all, or whether `c/memory` stays scoped to
string misuse and says so — which is where task 2.16 left it, deliberately.
string misuse and says so, which is where task 2.16 left it, deliberately.
3. If it is worth it: **angr** first, on language, licence and the fact that value-set analysis is the
named abstraction -- with GhidraPAL read as the worked example of a bit-precise domain rather than as a
platform. Vine is dormant since 2009 and is not a candidate.
4. What a second pipeline costs in the one currency this project has been strict about — **not lines, but
the number of things that can silently return nothing.** This session found three separate quiet failures
4. What a second pipeline costs in the one currency this project has been strict about,
**the number of things that can silently return nothing rather than lines.** This session found three separate quiet failures
in one PHP scan. A binary pipeline adds a build step, a lifter, and a domain, each of which can produce
an empty result that looks like a clean one.
+46 -46
View File
@@ -14,13 +14,13 @@ experiments.
Six facts from the repository and the recorded results shape the design:
- **The deployment target is a repository nobody trained on.** Maintainer: "the tool will never run against the
- The deployment target is a repository nobody trained on. Maintainer: "the tool will never run against the
test cases it's being trained with. it's purpose is a repo safety net." (Req 7). The primary unit is a candidate
in changed code (`ousast pre-push`, base..head), and whole-repository `scan` is the secondary mode. Every
reported number therefore comes from folds grouped by repository across all corpora. Features are
repository-agnostic by an allow-list. Fit to a user's own repository is an opt-in layer outside the reported
numbers.
- **Labels are scarce, and full plane signals exist on 43 rows.** The only recorded plane run with
- Labels are scarce, and full plane signals exist on 43 rows. The only recorded plane run with
verify/agree/tie-break is `validation-46`: 15 cases, 43 candidates after triage, 19 of them with
`site_match` true, all from population v2
(`~/ousast-results/plane/validation-46/*-final/agreed.json`). The pair corpus is larger (427 pairs and 452
@@ -30,19 +30,19 @@ Six facts from the repository and the recorded results shape the design:
classifier over verify signals cannot be evaluated today. The design starts from the signals every labelled row
can have, harvests the rest at a stated cost, and accepts that data will stay small: the engine learns by
memory and compilation, not by fitting parameters.
- **The core install is PyYAML only** (`pyproject.toml:11`). numpy, scipy and scikit-learn are not installed in
- The core install is PyYAML only (`pyproject.toml:11`). numpy, scipy and scikit-learn are not installed in
`.venv`. Retrieval (a Gower distance and a cosine over at most 40 vectors) and calibration (two-parameter Platt)
stay pure Python; the model calls go through the plane's metered client (section 4).
- **`pre-push` already has the decision seam.** `admit_candidates` (`push/policy.py:879`) applies one evidence
- `pre-push` already has the decision seam. `admit_candidates` (`push/policy.py:879`) applies one evidence
bar to `CandidateDelta`s with `novelty` (`push/policy.py:61-74`). Its capability registry is empty by design
(`push/policy.py:802-823`: "No shipped declaration is enabled"), so `pre-push` blocks nothing today. The
engine's operating point replaces the capability-eligibility reasons, and the structural reasons stay.
- **The memory store is the system of record and has the needed shape.** Rows are keyed by repository and pin,
- The memory store is the system of record and has the needed shape. Rows are keyed by repository and pin,
with a closed `KINDS` tuple (`plane/memory.py:52`). There are two backends (FileStore, S3Store) behind one
interface (`plane/memory.py:176-291`), content-addressed blobs (`put_facts`, `plane/memory.py:249-255`), and a
train-on-test `Guard` (`improve/memory.py:95-133`). Features, labels, decisions, model artifacts and experiment
outcomes become new row kinds and blob prefixes. No second store is added.
- **Hand-written framework knowledge is embedded in the detectors** (Req 8). Examples are WordPress `$wpdb`,
- Hand-written framework knowledge is embedded in the detectors (Req 8). Examples are WordPress `$wpdb`,
hooks and `maybe_unserialize` in `ruleset/semantic/php.toml` and `ruleset/php/rules.toml`, and Flask and
Django names in `ruleset/semantic/python.toml`. Section 2 tags each entry. Priors become optional features
that are off by default. Framework roles are inferred per repository instead.
@@ -548,7 +548,7 @@ If the pool cannot satisfy a minimum, take what exists and record `balance: "sho
**The evaluation boundary (Req 3.4, 7.1).** `learn/retrieve.py: eligible(example, target, fold)` is the only way
an example enters a prompt, for retrieved examples and for the compiled demonstrations alike:
- `example.group` is not in the fold's evaluation groups (all of them, not just the target's), and not the
- `example.group` is not in the fold's evaluation groups (every evaluation group, including the target's), and not the
target's group after the CVE/alias merge (`Groups`, `learn/labels.py:182`);
- leave-one-source-out: `example.source` is not the held-out source;
- leave-one-framework-out on F: `F not in example.frameworks`; a compiled demonstration tagged F is swapped for its
@@ -669,23 +669,23 @@ million. Token sizes are estimates to be replaced by the smoke run's measured `u
| candidate (80-line excerpt ~1,100, signals and roles ~300) | ~1,400 | no |
| output (JSON, rationale <= 3 sentences) | ~200 | -- |
- **Per candidate, `k = 1`**: 2,750 x 0.014 + 5,950 x 0.44 + 200 x 1.32 per million = **$0.0029**.
- **Each extra sample** (same prompt, fully cached): 8,700 x 0.014 + 200 x 1.32 per million = **$0.0004**. So
- Per candidate, `k = 1`: 2,750 x 0.014 + 5,950 x 0.44 + 200 x 1.32 per million = **$0.0029**.
- Each extra sample (same prompt, fully cached): 8,700 x 0.014 + 200 x 1.32 per million = **$0.0004**. So
`k = 5` is **$0.0045** with cache hits, and $0.0146 if the cache does not hit (the smoke run decides the default).
- **Embedding**: $0.000024 per candidate; the whole store about $0.10 once.
- **Per compile**: bootstrap 150 x $0.0029 = $0.44; proposals ~$0.01; scoring 600 x $0.0029 = $1.74; total
- Embedding: $0.000024 per candidate; the whole store about $0.10 once.
- Per compile: bootstrap 150 x $0.0029 = $0.44; proposals ~$0.01; scoring 600 x $0.0029 = $1.74; total
**~$2.2, ceiling $3.00**.
- **Per full evaluation** of one program, evaluated families only (475 verified rows outside memory,
- Per full evaluation of one program, evaluated families only (475 verified rows outside memory,
config_secrets and unknown) plus a 200-unit benign-delta sample for false BLOCKs:
- outer folds: 675 candidates, **$2.0 at k = 1, $3.0 at k = 5** (cached);
- leave-one-source-out: 475 more, **$1.4 / $2.1**;
- leave-one-framework-out: $0 today (no framework at the floor);
- total **~$3.4 (k = 1) to $5.2 (k = 5)**, ceiling $7.
- **Learning curves** (section 7): 6 memory-size points + 3 source-count points on a fixed 200-candidate subset at
- Learning curves (section 7): 6 memory-size points + 3 source-count points on a fixed 200-candidate subset at
`k = 1`: 9 x 200 x $0.0029 = **~$5.2**, ceiling $7.
- **A program-variant A/B** on 300 paired candidates: arm A replays cached responses from the evaluation ($0);
- A program-variant A/B on 300 paired candidates: arm A replays cached responses from the evaluation ($0);
arm B costs **$0.9 (k = 1) to $1.35 (k = 5)**.
- **Deployment**: `pre-push` with ~5 candidates is ~$0.02 per push at `k = 5`; a plane Run case with ~30 candidates
- Deployment: `pre-push` with ~5 candidates is ~$0.02 per push at `k = 5`; a plane Run case with ~30 candidates
~$0.14.
**Against the balance.** DeepSeek was ~$19.6 before the harvest; the harvest's verify passes are ~$6 expected and
@@ -807,10 +807,10 @@ flagged in any pass as 17 for the plane vs 19 for the reference, of 20. The plan
(`plane/tasks/verify.py:1-9, 164`, `CALLERS_CAP = 8` at :34), and the comparison is confounded (other changes,
stochastic passes, v2 is where it was seen).
- **Arms.** A is `verify` as today. B is `verify` with `OUSAST_VERIFY_CALLERS=off`, a new switch (about 3 lines)
- Arms. A is `verify` as today. B is `verify` with `OUSAST_VERIFY_CALLERS=off`, a new switch (about 3 lines)
that makes `callers_line` return empty. Same model (`deepseek-flash`), `PER_HUNT = 6`, `MAX_STEPS = 6`, same
candidates, same `facts.json`, same runner image digest, two passes (a, b) per arm.
- **Units: the development corpus, not v2** (v2 generated the hypothesis):
- Units: the development corpus, not v2 (v2 generated the hypothesis):
- population v1, 11 cases, whose engine scans are recorded under `~/ousast-results/independent-v1/scans/` (44
scans);
- pmpro, MediaWiki, WP Statistics (dev-php);
@@ -821,15 +821,15 @@ stochastic passes, v2 is where it was seen).
model-free and frozen in the units file: per pin, the fix-changed functions, plus the top engine- and
quick-alert functions, up to 8 candidates per pin, on the vulnerable and the fixed pin. Only units where arm A's
callers line is non-empty are *treated*. The primary analysis is restricted to them, and the rest are reported.
- **Metric.** Primary: the per-positive-unit rate of "flagged in any pass" (the 17-vs-19 metric), with positives
- Metric. Primary: the per-positive-unit rate of "flagged in any pass" (the 17-vs-19 metric), with positives
from the label builder (fix-range functions on the vulnerable pin). Secondary: the flag rate on fixed-pin
negatives (false flags), and usd per candidate.
- **Sample size and MDE.** About 55 positive units across 41 repositories (about 28 dev + 27 pointer pairs). With
- Sample size and MDE. About 55 positive units across 41 repositories (about 28 dev + 27 pointer pairs). With
a discordance rate of about 0.15, alpha 0.05 and power 0.8, the paired MDE is `(1.96 + 0.84) * sqrt(0.15) /
sqrt(55)`, about **0.15**. The observed gap of 0.10 needs about 115 positive units, so an inconclusive result is
the likely and honest outcome. It is registered as such in advance, and a follow-up needs new units, not a
continuation.
- **Budget.** Hunts per pass per arm: 14 dev repositories x 2 pins x up to 3 files (84) + 27 pairs x 2 sides (54)
- Budget. Hunts per pass per arm: 14 dev repositories x 2 pins x up to 3 files (84) + 27 pairs x 2 sides (54)
= 138. Four arm-passes give 552 hunts x $0.014, **about $7.7 expected**. The ceiling is **$15 total, $7.5 per
arm**, enforced by the task budgets (`plane/budget.py`). Candidate generation reuses the recorded v1 engine
scans. For dev-php it runs the engine once per pin on the host, one container at a time, from a frozen source
@@ -1111,30 +1111,30 @@ Each commit is gated on the full suite's own exit code, `ruff` and `mypy`. None
commit 9 adopts a program, and without an adopted program every path is today's. Steps that spend model money
start only after the maintainer confirms their budget against the balance read just before.
1. **Schema and features** (1.1-1.3, 7.2) -- done.
2. **Framework tags** (8.2, 8.4) -- done.
3. **Roles** (8.1) -- done.
4. **Labels** (2.1-2.3, 7.1) -- done.
5. **Harvest** (1.1, data) -- in progress: verify a/b on the verify-family pairs and the development cases (about
1. Schema and features (1.1-1.3, 7.2) -- done.
2. Framework tags (8.2, 8.4) -- done.
3. Roles (8.1) -- done.
4. Labels (2.1-2.3, 7.1) -- done.
5. Harvest (1.1, data) -- in progress: verify a/b on the verify-family pairs and the development cases (about
$6, ceiling $10), model roles on the candidate files (ceiling $10), engine features on the host.
6. **Program, memory, compile, first evaluation** (3.1-3.6, 5.1-5.2, 7.3-7.4, 8.3): excerpts and the example
6. Program, memory, compile, first evaluation (3.1-3.6, 5.1-5.2, 7.3-7.4, 8.3): excerpts and the example
store, metered embeddings and the cache, retrieval with the boundary test, the program and response cache, folds,
compile, cross-fitted calibration; then the paid slice (smoke, embed, compile, outer evaluation, ~$5, ceiling $7)
and, after a second go-ahead, leave-one-source-out and the memory-size curves (~$7, ceiling $10), committed under
`benchmarks/measurements/<date>-decision-engine-v1/`.
7. **Experiments** (4.1-4.4): `learn/experiments.py`, `register`, `experiment-outcome`, the Run annotation checks,
7. Experiments (4.1-4.4): `learn/experiments.py`, `register`, `experiment-outcome`, the Run annotation checks,
`OUSAST_VERIFY_CALLERS`, exp-001 registered, run and recorded, and the program-variant experiment machinery.
8. **Integration** (6.1-6.3, 7.3, 7.5): the `embed` and `decide` tasks with the seeded memory, the `scan` stage,
8. Integration (6.1-6.3, 7.3, 7.5): the `embed` and `decide` tasks with the seeded memory, the `scan` stage,
`pre-push` admission with the engine, report fields, the fallback line, and the opt-in adaptation layer.
9. **Adopt** (3.3, 8.2): `ousast learn adopt`, the packaged program with the licence filter, the priors-on/off and
9. Adopt (3.3, 8.2): `ousast learn adopt`, the packaged program with the licence filter, the priors-on/off and
`k` experiments with recorded decisions.
10. **v3 check** (5.3): `prediction-v3.json` is committed first, then v3 is run by its owner under protocol v3, and
10. v3 check (5.3): `prediction-v3.json` is committed first, then v3 is run by its owner under protocol v3, and
`result-v3-engine.json` is committed. Nothing between the two commits may touch the program, memory snapshot,
features, priors or thresholds.
## Risks
- **Label scarcity decides what can be evaluated.** The status of each family today:
- Label scarcity decides what can be evaluated. The status of each family today:
- memory (5 repository groups, 173/173 rows, almost all openssl and curl), config_secrets (5) and prototype are
**insufficient data** (`labels.json` `verified_pin_labels`). Users get today's behaviour there, labelled as
such, and no money is spent evaluating them.
@@ -1143,16 +1143,16 @@ start only after the maintainer confirms their budget against the balance read j
- The plane profile has 43 rows from one population until the harvest Run.
- BLOCK at 95% precision is likely **not reachable** for most families on this data. The design reports "BLOCK
not offered" instead of lowering the target.
- **Weak signals, strong code.** The pair baseline has quick-rule labelled recall 0.24 and Youden -0.013 overall
- Weak signals, strong code. The pair baseline has quick-rule labelled recall 0.24 and Youden -0.013 overall
(`pairs.json` `overall`); the v2 engine recall is 0/17 (`results-v2.json`). The program reads the code, so it
can beat the signals, but the verify history says the model over-flags (v1 1/12, v2 0/17 adjudicated true). A
calibrated classifier that mostly says "advisory" is honest; recall at ADVISORY 0.90 may need a threshold with
poor precision. That is reported, not tuned away on outer folds.
- **LLM nondeterminism.** The same prompt can answer differently, and temperature 0 is not a provider guarantee.
- LLM nondeterminism. The same prompt can answer differently, and temperature 0 is not a provider guarantee.
Guards: every response cached and replayable (numbers reproduce at $0); sample 1 at temperature 0; the vote mean
over `k` samples as the score; the A/B unit is paired and both arms' caches are recorded; a re-ask agreement rate
on a fixed 30-candidate canary set is reported with each evaluation.
- **Label leakage into prompts.**
- Label leakage into prompts.
- Through rationales: memory examples carry no rationale; demonstration rationales are produced without the label
and scanned for identities and advisory ids; the candidate's prompt never holds `EXCLUDED_FIELDS` (test).
- Through code: fixed-side excerpts can carry fix comments ("prevent XSS", a CVE id). Advisory ids and security
@@ -1160,11 +1160,11 @@ start only after the maintainer confirms their budget against the balance read j
the change under test, never the later fix.
- Through memorisation: the chat model may have seen public CVE fixes in pre-training. Not removable; the v3
check (unpublished fixes are rarer there) and per-source numbers expose it, and the report names it.
- **Retrieval leakage.** Same repository under another name or corpus: group normalisation, CVE merge, the
- Retrieval leakage. Same repository under another name or corpus: group normalisation, CVE merge, the
near-duplicate drop at cosine >= 0.98 in evaluation, and `test_boundary`. Demonstrations from the compile split
are, by construction, from repositories never evaluated. The seed for plane Runs filters on the host, where the
labels are; the sandbox never receives an example of its own case's group.
- **Label-tracking signals (maintainer decision 2026-10-01: "Neutralise both for v1").** The harvest found two
- Label-tracking signals (maintainer decision 2026-10-01: "Neutralise both for v1"). The harvest found two
signals that follow the pair label instead of the code: verify candidates on fixed sides named functions the side
no longer declares (now never produced: a side's candidates are only the functions it declares, the rest counted
`absent_on_side`), and the engine's "read the file, asked no question" state, which a fixed side reaches by losing
@@ -1173,40 +1173,40 @@ start only after the maintainer confirms their budget against the balance read j
the stored record keeps it. The engine returns only through a pre-registered A/B evaluated on assumed-benign and
benign-control negatives, where it cannot track a fix. `ousast learn audit-leaks` reports how well every signal
alone separates the sides of a pair and flags any above 0.20 (counts under `benchmarks/measurements/`).
- **Earlier leakage paths still apply.** Hand rules and framework priors written after seeing a training repository
- Earlier leakage paths still apply. Hand rules and framework priors written after seeing a training repository
(the `$wpdb` rule from PMPro, the hook source from WP Statistics, `taint.sc:911-949`): priors off by default,
leave-one-framework-out, best-effort `taught_by = [repo]`. The mechanism exporter's train-on-test leak
(`closed-loop-train-on-test-leak.md`): no exporter output is a feature. Verify and tie-break fitted on v2: the v2
fold is design-informed. Thresholds and instructions chosen on reported folds: cross-fitting and the compile
split. v3: fail-closed sources, never opened. User dismissals: refused by both builders.
- **Cost.** Model spend is now per candidate and per evaluation, not one-off. Measured (section 4.6) the first
- Cost. Model spend is now per candidate and per evaluation, not one-off. Measured (section 4.6) the first
slice fits today's balance; everything after it needs a top-up. Guards: `MeteredClient` ceilings on every task
and command, a smoke check that measures real tokens and cache hits before any paid run, cached replay for
re-analysis, unevaluable families skipped, `k = 1` if the prefix cache does not hit. Deployment cost (~$0.02 per
push) is a user-facing change: `pre-push` without a key falls back, and the report states the spend.
- **Model and version drift.** `deepseek-flash` is an alias whose weights can change, and so can OpenRouter's
- Model and version drift. `deepseek-flash` is an alias whose weights can change, and so can OpenRouter's
embedding route. The artifact records model ids, the Model parameters digest and the response `model` field;
before adoption and before the v3 prediction the canary set is re-asked, and agreement below 0.9 with the
cached answers, or an embedding dimension change, invalidates the evaluation (re-embed, re-evaluate) rather than
being averaged in. Prices are data in the Model manifest; a price change changes only cost reports.
- **Latency.** A model call per candidate is slower than a dot product. The 30 s `pre-push` deadline is met by
- Latency. A model call per candidate is slower than a dot product. The 30 s `pre-push` deadline is met by
concurrency and the `undecided: deadline` state; a push with many candidates gets partial decisions, stated.
- **Packaging third-party code.** Demonstrations and memory hold excerpts of corpus repositories. Only
- Packaging third-party code. Demonstrations and memory hold excerpts of corpus repositories. Only
permissively licensed excerpts may ship in the package; the store is never shipped. A user without a store gets
the demonstrations-only program, whose own curve point (memory 0%) and calibration are what they are told.
- **Framework inference is itself a model.** Model roles cost money and can be wrong. `scan` without `[models]`
- Framework inference is itself a model. Model roles cost money and can be wrong. `scan` without `[models]`
gets only wrapper inference, whose reach on framework-heavy code without dependency source is poor. The report
says which role source ran. Both are measured by the roles-on/off experiment. Nothing assumes they help.
- **Host time on the 7 GB host.** Engine features remain the time cost (7-14 hours serial for the pair corpus,
- Host time on the 7 GB host. Engine features remain the time cost (7-14 hours serial for the pair corpus,
dev-php scans of 200-2,000 s each), run once per engine image from a frozen source export, one container at
`--memory 3g` at a time, never beside the kind cluster's verify pods, cached by (content sha, image id). Model
calls add wall time but no memory pressure; excerpts and embeddings are small (~8 KB per example with a 1,536-float
vector, ~45 MB for 5,600 examples).
- **Experiments will often be inconclusive.** With 41 development repositories the MDE is about 15 points.
- Experiments will often be inconclusive. With 41 development repositories the MDE is about 15 points.
Pre-registering that and recording "inconclusive" is the intended behaviour. The failure mode to avoid is
extending an experiment after looking.
- **First paid slice, 2026-10-01 (tasks 6.2, 6.6; injection only).** Four instrument defects surfaced before any
- First paid slice, 2026-10-01 (tasks 6.2, 6.6; injection only). Four instrument defects surfaced before any
number was believed, each fixed with a test:
- *The memory build joined no pair.* The harvest stores a pair side's record under its unit pin (the blob sha1 of
the side's excerpt), not under `label_pin`; `ousast learn memory build --units <harvest units.json>` joins them.
@@ -1227,7 +1227,7 @@ start only after the maintainer confirms their budget against the balance read j
0.90 has precision 0.54 (prevalence 0.53); the majority `vulnerable` verdict alone has precision 0.86, recall 0.43.
Canary re-ask agreement at temperature 0 was 0.83, below the 0.9 the adoption check requires.
- **Slice 2, 2026-10-02 (six families, after the advisory-fix harvest).** Two instrument defects stopped runs before any
- Slice 2, 2026-10-02 (six families, after the advisory-fix harvest). Two instrument defects stopped runs before any
number: the memory build's side reader knew only the first pair-kind source (all 194 advisory sides unread), and
compiled demonstrations skipped the near-duplicate rule that retrieval and the pre-call assertion apply (the
assertion stopped the untrusted_destination evaluation). Both fixed with tests. A confound to read with the new
@@ -25,29 +25,29 @@ money.
## Requirements (proposed)
1. **Mining.** From each fix pair, extract what the fixed function adds on the vulnerable flow: new calls,
1. Mining. From each fix pair, extract what the fixed function adds on the vulnerable flow: new calls,
new conditions guarding the sink, new or changed sink arguments. Record each as a candidate fact with its
language, family, form (sanitizer call, guard predicate, safe argument, sink restriction) and the pairs
that propose it. Mining is deterministic (tree-sitter diff of the two function bodies). No model calls.
2. **Generalisation is explicit.** A candidate is a pattern, not a repository name: a callee name, an
2. Generalisation is explicit. A candidate is a pattern, not a repository name: a callee name, an
argument key, a predicate shape. Candidates naming project-specific identifiers are rejected (Req 8: no
application-specific trees).
3. **Admission by held-out separation.** Folds are by repository, as in the decision engine. A candidate is
3. Admission by held-out separation. Folds are by repository, as in the decision engine. A candidate is
admitted for a fold only if, on that fold's held-out pairs, adding it to the fact tables:
- increases pairs where the vulnerable side keeps a path and the fixed side loses it or becomes
sanitized;
- and does not remove any held-out vulnerable-side path (no recall loss).
A candidate proposed by fewer than 2 distinct repositories in the training folds is not tried.
4. **No train-on-test.** Mining and admission never read held-out pairs; the closed-loop leak recorded in
4. No train-on-test. Mining and admission never read held-out pairs; the closed-loop leak recorded in
memory (`closed-loop-train-on-test-leak`) is the failure this rule exists to prevent. Population v3 is
never read.
5. **Measurement by the trace runner.** Each admission round re-runs the engine on the affected pins only
5. Measurement by the trace runner. Each admission round re-runs the engine on the affected pins only
(the runner's resume and selection already support this; on k3s the worker Deployment runs them in
parallel). The outcome is the pair-asymmetry table per family, before and after, on held-out folds.
6. **Gate before production.** An admitted fact enters `src/openultrasast/ruleset/semantic/*.toml` only after
6. Gate before production. An admitted fact enters `src/openultrasast/ruleset/semantic/*.toml` only after
the detection gates (`gate`, `map_gate`, `pair_gate`) show no regression, and only as a registered
experiment whose decision rule was fixed before the run.
7. **Coverage second.** After discharge facts, the same loop proposes source and sink additions from pairs
7. Coverage second. After discharge facts, the same loop proposes source and sink additions from pairs
where neither side has a path (91 of 105 today), admitted under the same held-out rule.
## How it relates to what exists
@@ -9,13 +9,13 @@ Maintainer direction and context: `brief.md`.
## Boundary Context
- **In scope**: a feature record per candidate built from every instrument and the memory store; a labelled
- In scope: a feature record per candidate built from every instrument and the memory store; a labelled
dataset with provenance; training, calibration and versioning of the decision engine; A/B experiments as plane
Runs; learning curves and leave-one-population-out estimates; the engine's use in `ousast scan`, `ousast
pre-push` and plane Runs; a final one-time check on frozen population v3.
- **Out of scope**: new instruments (they plug in as signal producers later); deep or large models; any change to
- Out of scope: new instruments (they plug in as signal producers later); deep or large models; any change to
population v3 or its protocol; publishing results (licences still gate that).
- **Adjacent expectations**: instruments keep working standalone; the memory store (harnessx-removal Req 6) is
- Adjacent expectations: instruments keep working standalone; the memory store (harnessx-removal Req 6) is
the system of record; the plane attributes every experiment's token spend per task.
## Requirements
@@ -121,7 +121,7 @@ rule hits.
reported unless the engine's decision changes.
4. *(Amended 2026-10-03, maintainer decision; refines the BLOCK rule of 7.3 and 7.4. Tested by exp-004.)* BLOCK
precision is demonstrated pooled across families, not per family:
- **Normalisation**, declared before any run: each family's raw score is put on one scale by a per-family Platt
- Normalisation, declared before any run: each family's raw score is put on one scale by a per-family Platt
map (`learn.calibrate.fit_map` with its floors) fitted on training folds only. A family the training folds
cannot map does not block on that fold.
- **t_block**: one threshold on the normalised score for all families, chosen on training folds as the lowest
+7 -7
View File
@@ -62,8 +62,8 @@ Principle for the roadmap: **classify first, verify before reporting, improve on
## Scope
- **In**: items 1–8; roadmap rewrite from one command; a closure decision on `authorization-obligations` 4.3, 5 and 6.
- **Out**: new static shape families; hand-written facts beyond what a round proposes and validates; changing the evidence ladder or dispositions; any merge gate on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights.
- In: items 1–8; roadmap rewrite from one command; a closure decision on `authorization-obligations` 4.3, 5 and 6.
- Out: new static shape families; hand-written facts beyond what a round proposes and validates; changing the evidence ladder or dispositions; any merge gate on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights.
## Boundary Candidates
@@ -80,14 +80,14 @@ Principle for the roadmap: **classify first, verify before reporting, improve on
## Upstream / Downstream
- **Upstream**: `pair-corpus-honesty` (labels, tiers, splits, `known_limit`), `corpus-seeded-mechanisms` (exporter, LOO, lever), `tool_hunter` and `provider/openrouter`, `harness_ext`, HarnessX 0.1.0 (`meta_harness`, `rl`, `tracing.journal`), `tree-sitter-overlay-extra`, the sandbox and regress recipes.
- **Downstream**: every roadmap number; `harnessx-self-improving-rulesets` (its evolve step is this spec); `authorization-obligations` 4.3/5/6; scan reports (findings carry class, detector and verifier).
- Upstream: `pair-corpus-honesty` (labels, tiers, splits, `known_limit`), `corpus-seeded-mechanisms` (exporter, LOO, lever), `tool_hunter` and `provider/openrouter`, `harness_ext`, HarnessX 0.1.0 (`meta_harness`, `rl`, `tracing.journal`), `tree-sitter-overlay-extra`, the sandbox and regress recipes.
- Downstream: every roadmap number; `harnessx-self-improving-rulesets` (its evolve step is this spec); `authorization-obligations` 4.3/5/6; scan reports (findings carry class, detector and verifier).
## Existing Spec Touchpoints
- **Extends**: `pair-corpus-honesty` Req 1 (scorer) and Req 9/10 (row classification); `corpus-seeded-mechanisms` Req 4/5 (split by construction); `three-stage-scan` MAP (per-class hunters as proposers); `propose-adjudicate-prove` (verifiers keyed by class); `harnessx-self-improving-rulesets` (the evolve loop lands here).
- **Folds in**: the retired `closed-loop-integrity` brief.
- **Re-homes**: `authorization-obligations` 4.3 and the corpus half of 6.1; holds 5.
- Extends: `pair-corpus-honesty` Req 1 (scorer) and Req 9/10 (row classification); `corpus-seeded-mechanisms` Req 4/5 (split by construction); `three-stage-scan` MAP (per-class hunters as proposers); `propose-adjudicate-prove` (verifiers keyed by class); `harnessx-self-improving-rulesets` (the evolve loop lands here).
- Folds in: the retired `closed-loop-integrity` brief.
- Re-homes: `authorization-obligations` 4.3 and the corpus half of 6.1; holds 5.
## Constraints
+13 -13
View File
@@ -150,20 +150,20 @@ tests/test_learning_families.py, test_learning_classify.py, test_learning_scorin
```
### Modified Files
- `src/openultrasast/pairs.py` — `PairCase.unscorable: str | None` computed at load (identical twin via normalized body hash, `known_limit` passthrough); duplicate report across pairs before split; `split_by_repository` helper honoring existing `split` fields and reporting straddles; `_evaluate_hunter_pair` delegates matching and leaks to `learning.scoring` through an injected callable so `pairs` does not import `learning`; `build_pair_signals` gains a `split` filter.
- `src/openultrasast/benchmark.py` — `ExpectedFinding.family: str | None = None` (additive, stored as an opaque string; the Classifier validates loaded labels against the taxonomy so `pairs` and `benchmark` never import `learning`).
- `src/openultrasast/tool_hunter.py` — `run_tool_hunter` gains `system_prompt`, `context_files`, `tools`, `max_chars` and `tags` parameters with today's values as defaults; finding parsing accepts an optional `family` field; the reply loop replays `reasoning_content`.
- `src/openultrasast/hunter_tools.py` — four new tools beside `read_file`, `grep_repo`, `find_refs`: `read_definition(symbol)`, `entry_points(path)`, `flows(path, function)`, `obligations(path, function)`, each clamped to the repository and returning identifiers and line spans only; `HUNTER_TOOLS` schemas extended.
- `src/openultrasast/provider/openrouter.py` — `OpenRouterChatClient` gains `extra_body` (thinking, response_format, logprobs) and retains the `reasoning_content` field on assistant messages; replaying it across tool turns is the hunter loop's job (`tool_hunter`), not the client's. No behavior change for existing callers.
- `src/openultrasast/config.py` — `[models]` gains `chat_base_url`, `chat_api_key_env`, `judge`; `[learning]` section (`families_path`, `configs_dir`, `k_runs = 5`, `round_cost_cap_usd`, `minibatch`, `cutoff_date`); `load_dotenv` unchanged, tests inject endpoints explicitly.
- `src/openultrasast/semantic/seed.py`, `semantic/loo.py`, `improve/evolve.py` — teacher selection through `learning.split.teachers`; refusals recorded; `evaluate_loo` teaches from train only.
- `src/openultrasast/regress/candidate.py`, `regress/verdict.py` — `run_regression` accepts an optional `oracle: Callable[[SandboxResult], bool]`; `verdict_from_result` uses the oracle when given, exit code otherwise.
- `src/openultrasast/cli.py` — `ousast learning classify|score|baseline|round|publish`; `pairs --hunter` uses the family scorer and `--k-runs`; `[models].hunter` resolution through `learning.endpoint`.
- `pyproject.toml` — `tree-sitter-typescript>=0.23` in the semantic extra; `src/openultrasast/semantic/extra.py` maps `typescript`/`tsx`.
- `benchmarks/pairs/vibe-py/catalog.toml` and `recipes.toml` — the five identical twins gain `known_limit = "identical_twin"`; absence rows gain `family` and `mode = "handler_context"` (re-harvest is a maintainer step).
- `src/openultrasast/pairs.py`: `PairCase.unscorable: str | None` computed at load (identical twin via normalized body hash, `known_limit` passthrough); duplicate report across pairs before split; `split_by_repository` helper honoring existing `split` fields and reporting straddles; `_evaluate_hunter_pair` delegates matching and leaks to `learning.scoring` through an injected callable so `pairs` does not import `learning`; `build_pair_signals` gains a `split` filter.
- `src/openultrasast/benchmark.py`: `ExpectedFinding.family: str | None = None` (additive, stored as an opaque string; the Classifier validates loaded labels against the taxonomy so `pairs` and `benchmark` never import `learning`).
- `src/openultrasast/tool_hunter.py`: `run_tool_hunter` gains `system_prompt`, `context_files`, `tools`, `max_chars` and `tags` parameters with today's values as defaults; finding parsing accepts an optional `family` field; the reply loop replays `reasoning_content`.
- `src/openultrasast/hunter_tools.py`: four new tools beside `read_file`, `grep_repo`, `find_refs`: `read_definition(symbol)`, `entry_points(path)`, `flows(path, function)`, `obligations(path, function)`, each clamped to the repository and returning identifiers and line spans only; `HUNTER_TOOLS` schemas extended.
- `src/openultrasast/provider/openrouter.py`: `OpenRouterChatClient` gains `extra_body` (thinking, response_format, logprobs) and retains the `reasoning_content` field on assistant messages; replaying it across tool turns is the hunter loop's job (`tool_hunter`), not the client's. No behavior change for existing callers.
- `src/openultrasast/config.py`: `[models]` gains `chat_base_url`, `chat_api_key_env`, `judge`; `[learning]` section (`families_path`, `configs_dir`, `k_runs = 5`, `round_cost_cap_usd`, `minibatch`, `cutoff_date`); `load_dotenv` unchanged, tests inject endpoints explicitly.
- `src/openultrasast/semantic/seed.py`, `semantic/loo.py`, `improve/evolve.py`: teacher selection through `learning.split.teachers`; refusals recorded; `evaluate_loo` teaches from train only.
- `src/openultrasast/regress/candidate.py`, `regress/verdict.py`: `run_regression` accepts an optional `oracle: Callable[[SandboxResult], bool]`; `verdict_from_result` uses the oracle when given, exit code otherwise.
- `src/openultrasast/cli.py`: `ousast learning classify|score|baseline|round|publish`; `pairs --hunter` uses the family scorer and `--k-runs`; `[models].hunter` resolution through `learning.endpoint`.
- `pyproject.toml`: `tree-sitter-typescript>=0.23` in the semantic extra; `src/openultrasast/semantic/extra.py` maps `typescript`/`tsx`.
- `benchmarks/pairs/vibe-py/catalog.toml` and `recipes.toml`: the five identical twins gain `known_limit = "identical_twin"`; absence rows gain `family` and `mode = "handler_context"` (re-harvest is a maintainer step).
- **Cross-file registration context (amended during task 4.1).** Some frameworks keep the route table out of the handler module: connexion registers by `operationId` in an OpenAPI document, an Express application registers in a router file. A single-file excerpt of such a handler can never contain its registration, so Req 10.2 is unreachable for those rows without a second file. A recipe may therefore name `context = ["<repo-relative path>"]`; `benchmarks/pairs/harvest.py` fetches each document at the same two commits, extracts only the stanza that names the handler (`extract_registration_context`), and writes it under `<name>/context/<side>/<relpath>` with its own provenance header and the same redaction. The catalog carries `[[pair.context]]` rows, `PairCase.context_files` holds `(relpath, vuln, fixed)`, and `_materialize_side` copies them beside the excerpt at their repository-relative paths so the entry-point mapper sees the same layout the project has. `src/openultrasast/mapping.py` gains `_route_manifest_entry_points`, which reads `operationId` and the operation's `security` block out of a YAML or JSON document and attributes the record to the module the operation id points at. Trap pairs whose fixed side is a different, correctly guarded handler carry `fix_function` so the fixed side gets *its* registration rather than the vulnerable side's.
- `benchmarks/pairs/mechanisms.toml` — unchanged; families reference mechanism ids.
- `.kiro/steering/roadmap.md` — rewritten by `publish` from artifacts.
- `benchmarks/pairs/mechanisms.toml`: unchanged; families reference mechanism ids.
- `.kiro/steering/roadmap.md`: rewritten by `publish` from artifacts.
## System Flows
+4 -4
View File
@@ -11,9 +11,9 @@ OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The
## Boundary Context
- **In scope**: the class taxonomy and its classifier; the class-aware pair scorer and unscorable-row accounting; split enforcement for every learning path; per-family detector configurations and their routing; class-keyed verifiers and reporting rungs; the round-based evolve loop with attribution; the corpus repairs the loop needs; publishing every number with its denominator and noise floor.
- **Out of scope**: finding bugs in JavaScript interpreters, WebGL renderers or browser engines; memory-safety fuzzing or crash oracles for C and C++ (the vfc slice stays a measured, non-gating slice under one coarse family); new hand-written static shape families; changes to the evidence ladder or dispositions; merge gates on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights.
- **Adjacent expectations**: the pair corpus (`pair-corpus-honesty`) keeps owning labels, tiers, splits and `known_limit`; `corpus-seeded-mechanisms` keeps its exporter, leave-one-out and lever, which this feature subordinates to the split and the class attribution; `authorization-obligations` keeps its checker, whose corpus tasks 4.3 and 6.1 move here and whose lever task 5 becomes the access-control family's first round; the local detection, map and pair gates stay byte-identical; HarnessX and tree-sitter grammars remain optional extras and every step that needs them degrades to a recorded reason without them.
- In scope: the class taxonomy and its classifier; the class-aware pair scorer and unscorable-row accounting; split enforcement for every learning path; per-family detector configurations and their routing; class-keyed verifiers and reporting rungs; the round-based evolve loop with attribution; the corpus repairs the loop needs; publishing every number with its denominator and noise floor.
- Out of scope: finding bugs in JavaScript interpreters, WebGL renderers or browser engines; memory-safety fuzzing or crash oracles for C and C++ (the vfc slice stays a measured, non-gating slice under one coarse family); new hand-written static shape families; changes to the evidence ladder or dispositions; merge gates on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights.
- Adjacent expectations: the pair corpus (`pair-corpus-honesty`) keeps owning labels, tiers, splits and `known_limit`; `corpus-seeded-mechanisms` keeps its exporter, leave-one-out and lever, which this feature subordinates to the split and the class attribution; `authorization-obligations` keeps its checker, whose corpus tasks 4.3 and 6.1 move here and whose lever task 5 becomes the access-control family's first round; the local detection, map and pair gates stay byte-identical; HarnessX and tree-sitter grammars remain optional extras and every step that needs them degrades to a recorded reason without them.
## Requirements
@@ -87,7 +87,7 @@ OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The
1. The Learning Harness shall hold one detector configuration per family consisting of a prompt under a stated length cap, a curated tool set, a step budget, a cost budget, an evidence checklist and a set of hard negatives and counterexamples.
2. While a round evolves one family, the Learning Harness shall keep every other family's configuration unchanged.
3. When a code region is classified, the Learning Harness shall run the detector of every admitted family plus a generalist detector, and shall record on each finding the family, the detector configuration version and the verifier outcome.
4. The Learning Harness shall give a detector the whole file of a region and the files its curated tools resolve, not only an excerpt around a hotspot.
4. The Learning Harness shall give a detector the whole file of a region and the files its curated tools resolve instead of limiting input to an excerpt around a hotspot.
5. The Learning Harness shall expose to a detector only curated tools that read, search and resolve symbols within the scanned repository, and no general shell.
6. The Learning Harness shall run detectors, the classifier's model tier and the evolve proposer on a configurable chat endpoint that is independent of the embedding endpoint, with a cheap model as the default and a stronger model reserved for the second judge and for review-queue disagreements.
+29 -29
View File
@@ -2,9 +2,9 @@
## Summary
- **Feature**: `learning-harness`
- **Discovery Scope**: Complex Integration (extension of pairs, tool hunter, provider, regress, improve; adoption of HarnessX meta-harness; corpus repairs)
- **Key Findings**:
- Feature: `learning-harness`
- Discovery Scope: Complex Integration (extension of pairs, tool hunter, provider, regress, improve; adoption of HarnessX meta-harness; corpus repairs)
- Key Findings:
- The tool hunter already beats the overlay 12/14 vs 5/14 on the class-aware, scorable vibe-py holdout; the published 0/17 was the text-token scorer.
- Every learning path ignores the split (`evaluate_profiles`, `evaluate_mechanism_profiles`, `propose_mechanism_edits`, `evaluate_loo`, `build_pair_signals`, `export_mechanisms`); one teacher rule fixes all of them.
- The sandbox verdict is exit-code only and snippets are compile-only templates; a canary oracle is an additive callback that respects the existing safety bans.
@@ -15,13 +15,13 @@
## Research Log
### Codebase seams (2026-09-06, Explore subagent over `src/openultrasast`)
- **Context**: design an extension without re-reading every module.
- **Findings**: `run_tool_hunter(root, hotspots, *, client, model, max_steps)` with three curated tools and `suspicion` findings tagged `tool-hunter`; `ChatClient.complete(*, model, messages, tools, timeout_seconds)`; `OpenRouterChatClient.complete_chat` always sends `temperature: 0` and drops `reasoning_content`; `_hunter_matches_expected` is text-token based and every fixed-side finding leaks under `fix_policy = silent`; `run_regression(..., sandbox, sandbox_limits, images)` with `verdict_from_result` on exit code; `check_snippet_safety` bans sockets, curl and workspace writes; `MechanismStore` append-only JSONL with tombstones; `improve.run_round` journal has no hypothesis, attribution or cost fields and its regression check is per provenance profile; `load_dotenv` no-ops under pytest; `pyproject` semantic extra lacks TypeScript; `pair_gate` scores only vendored `local` pairs.
- **Implications**: the design adds parameters and additive fields instead of replacing modules; the family scorer is injected into the hunter path so `pairs` never imports `learning`; the oracle is an optional callback on `run_regression`; endpoint resolution takes an explicit override for tests.
- Context: design an extension without re-reading every module.
- Findings: `run_tool_hunter(root, hotspots, *, client, model, max_steps)` with three curated tools and `suspicion` findings tagged `tool-hunter`; `ChatClient.complete(*, model, messages, tools, timeout_seconds)`; `OpenRouterChatClient.complete_chat` always sends `temperature: 0` and drops `reasoning_content`; `_hunter_matches_expected` is text-token based and every fixed-side finding leaks under `fix_policy = silent`; `run_regression(..., sandbox, sandbox_limits, images)` with `verdict_from_result` on exit code; `check_snippet_safety` bans sockets, curl and workspace writes; `MechanismStore` append-only JSONL with tombstones; `improve.run_round` journal has no hypothesis, attribution or cost fields and its regression check is per provenance profile; `load_dotenv` no-ops under pytest; `pyproject` semantic extra lacks TypeScript; `pair_gate` scores only vendored `local` pairs.
- Implications: the design adds parameters and additive fields instead of replacing modules; the family scorer is injected into the hunter path so `pairs` never imports `learning`; the oracle is an optional callback on `run_regression`; endpoint resolution takes an explicit override for tests.
### DeepSeek platform API (2026-09-06, api-docs.deepseek.com)
- **Findings**: base `https://api.deepseek.com` (no `/v1` in current docs), Anthropic-shape endpoint at `/anthropic`; models `deepseek-v4-flash`, `deepseek-v4-pro`, `deepseek-v4-flash-vision-exp`; 1M context, 384K output; thinking on by default (`thinking: {"type": "disabled"}` to turn off; `temperature` ignored while thinking); tools with `tool_choice`, `reasoning_content` must be replayed on tool turns; `response_format json_object` only, occasional empty content; `logprobs`/`top_logprobs` documented, unverified in thinking mode; no `seed`, `n`, `logit_bias`; flash 0.44/1.32 USD per M in/out peak, cache hit 0.014, pro 1.32/3.96, off-peak half; concurrency 2500 flash / 500 pro; no embeddings endpoint.
- **Implications**: detectors and classifier run with thinking disabled and temperature 0; determinism is best-effort and the K-run noise floor absorbs it; the classifier's model tier uses `logprobs` averaged and falls to `unknown`; embeddings stay on OpenRouter; cost accounting reads DeepSeek's cache-hit usage fields.
- Findings: base `https://api.deepseek.com` (no `/v1` in current docs), Anthropic-shape endpoint at `/anthropic`; models `deepseek-v4-flash`, `deepseek-v4-pro`, `deepseek-v4-flash-vision-exp`; 1M context, 384K output; thinking on by default (`thinking: {"type": "disabled"}` to turn off; `temperature` ignored while thinking); tools with `tool_choice`, `reasoning_content` must be replayed on tool turns; `response_format json_object` only, occasional empty content; `logprobs`/`top_logprobs` documented, unverified in thinking mode; no `seed`, `n`, `logit_bias`; flash 0.44/1.32 USD per M in/out peak, cache hit 0.014, pro 1.32/3.96, off-peak half; concurrency 2500 flash / 500 pro; no embeddings endpoint.
- Implications: detectors and classifier run with thinking disabled and temperature 0; determinism is best-effort and the K-run noise floor absorbs it; the classifier's model tier uses `logprobs` averaged and falls to `unknown`; embeddings stay on OpenRouter; cost accounting reads DeepSeek's cache-hit usage fields.
### Measurement that triggered the spec (2026-09-06)
- See `brief.md` Problem section and `benchmarks/measurements/2026-09-06-*`.
@@ -38,37 +38,37 @@
## Design Decisions
### Decision: families are the only routing and scoring key
- **Context**: 1.4, 3.6; text tokens and CWE ids were the proxy.
- **Alternatives**: fine CWE routing; mechanism ids as key.
- **Selected**: ten coarse families with CWE and mechanism as attributes; hierarchical relation for partial credit.
- **Rationale**: fine CWE from code is unsolved (≤14.7% top-1) and coarse families are what worked in AIxCC and MoEVD.
- **Trade-offs**: less granular reporting; compensated by attributes on findings.
- Context: 1.4, 3.6; text tokens and CWE ids were the proxy.
- Alternatives: fine CWE routing; mechanism ids as key.
- Selected: ten coarse families with CWE and mechanism as attributes; hierarchical relation for partial credit.
- Rationale: fine CWE from code is unsolved (≤14.7% top-1) and coarse families are what worked in AIxCC and MoEVD.
- Trade-offs: less granular reporting; compensated by attributes on findings.
### Decision: the scorer is injected into the hunter path, not imported by `pairs`
- **Context**: dependency direction; `pairs` is owned by `pair-corpus-honesty`.
- **Selected**: `_evaluate_hunter_pair` accepts a matcher/leak callable; the CLI wires `learning.scoring`.
- **Trade-offs**: one more parameter; keeps `pairs` free of `learning` and the overlay path unchanged.
- Context: dependency direction; `pairs` is owned by `pair-corpus-honesty`.
- Selected: `_evaluate_hunter_pair` accepts a matcher/leak callable; the CLI wires `learning.scoring`.
- Trade-offs: one more parameter; keeps `pairs` free of `learning` and the overlay path unchanged.
### Decision: canary oracle as an optional callback on `run_regression`
- **Context**: 7.2; exit code cannot express "the vulnerability fired".
- **Alternatives**: a parallel verifier runner; parsing stderr.
- **Selected**: `oracle: Callable[[SandboxResult], bool] | None`; token in stdout.
- **Trade-offs**: minimal change to regress; templates per family live in `learning/verifiers/`.
- Context: 7.2; exit code cannot express "the vulnerability fired".
- Alternatives: a parallel verifier runner; parsing stderr.
- Selected: `oracle: Callable[[SandboxResult], bool] | None`; token in stdout.
- Trade-offs: minimal change to regress; templates per family live in `learning/verifiers/`.
### Decision: HarnessX proposer behind the seam, scripted proposer first
- **Context**: 9.x with core `dependencies = []`.
- **Selected**: `Proposer` protocol; `HarnessXProposer` composes `MetaAgent.evolve` with `allowed_write_roots` = the family directory; `ScriptedProposer` for tests and phase 4 start.
- **Trade-offs**: two proposers; the loop is testable offline and the extra stays optional.
- Context: 9.x with core `dependencies = []`.
- Selected: `Proposer` protocol; `HarnessXProposer` composes `MetaAgent.evolve` with `allowed_write_roots` = the family directory; `ScriptedProposer` for tests and phase 4 start.
- Trade-offs: two proposers; the loop is testable offline and the extra stays optional.
### Decision: noise floor from K = 5 baseline runs is the regression budget
- **Context**: 8.2, 9.5; no public numeric negative-flip budget exists.
- **Selected**: per-family negative-flip rate between seed runs; `budget_flips = ceil(rate × scorable)`.
- **Trade-offs**: a noisy family gets a loose budget; reported beside the number so it is visible.
- Context: 8.2, 9.5; no public numeric negative-flip budget exists.
- Selected: per-family negative-flip rate between seed runs; `budget_flips = ceil(rate × scorable)`.
- Trade-offs: a noisy family gets a loose budget; reported beside the number so it is visible.
### Decision: DeepSeek adapter wraps the existing OpenRouter client
- **Context**: 6.6; both clients share one base URL today.
- **Selected**: `ChatEndpoint` resolution order DeepSeek → OpenRouter → scripted; adapter adds `extra_body` and `reasoning_content` replay; embeddings untouched.
- **Trade-offs**: one adapter class; no new dependency.
- Context: 6.6; both clients share one base URL today.
- Selected: `ChatEndpoint` resolution order DeepSeek → OpenRouter → scripted; adapter adds `extra_body` and `reasoning_content` replay; embeddings untouched.
- Trade-offs: one adapter class; no new dependency.
## Risks & Mitigations
- Router error routes a region to the wrong family → multi-label routing plus the generalist; router measured separately against random and oracle.
+6 -6
View File
@@ -8,8 +8,8 @@ skills. Related: `proof-arm-opencode` (the proof model), `pre-push-safety-net`
## Motivation
The pre-push hook is the gate. Developers and their coding agents want the same
security feedback **earlier**, while the code is still being written, not only at
push. And in an agent-driven workflow, a mandatory security check is how a human
security feedback **earlier**, while the code is still being written, before
push. In an agent-driven workflow, a mandatory security check is how a human
oversees agents: the agent's output has to clear it before it ships.
Proposal: expose the safety net through an MCP agent and a small set of skills so
@@ -19,20 +19,20 @@ proposes a push.
## Shape
- **Transport:** extend the existing narrow MCP server (`ousast mcp`, stdio) that
- Transport: extend the existing narrow MCP server (`ousast mcp`, stdio) that
OpenCode and other agents already speak. No new network surface; same
no-secret, bounded contract.
- **What the agent offers:**
- What the agent offers:
1. Scan the current diff with the static arm (fast, no model, no cluster) and
return findings scoped to the change, with location and a concrete repair
direction, not a confidence label.
2. Triage a finding: explain it in plain terms and whether it is reachable.
3. Prove on demand: hand the top candidate to the proof arm when a runnable
harness exists, and return the demonstrated proof-of-concept.
- **Skills:** the existing `openultrasast-scan` and `openultrasast-triage`
- Skills: the existing `openultrasast-scan` and `openultrasast-triage`
product skills, plus shift-left additions (scan-the-diff, explain-a-finding,
suggest-a-fix-direction) that a coding agent invokes mid-task.
- **Oversight mode:** a coding agent can be configured to call the scan as a
- Oversight mode: a coding agent can be configured to call the scan as a
required step before it finalizes a change, so security review is automatic
rather than remembered.
+61 -61
View File
@@ -7,7 +7,7 @@
> deterministically from code**; execution is a later, conditional tier. The correction is integrated below.
**Status:** course-setting. Supersedes `learning-harness` and `constrained-detector`. Not a requirements
document — a decision about what to build, what to delete, and what to adopt, written after the
document, a decision about what to build, what to delete, and what to adopt, written after the
learning-harness line produced numbers its own measurements say to discard.
## 1. The premise we got wrong
@@ -17,19 +17,19 @@ averaged (K runs), floored (per-family noise budget), gated (sign test), and slo
Every mechanism in `learning-harness` follows from that assumption. This session measured all of it and it
does not hold:
- Five evolve rounds, $2.17, **zero decisions distinguishable from chance** — because the acceptance rule
- Five evolve rounds, $2.17, **zero decisions distinguishable from chance**, because the acceptance rule
compares two ensemble votes over a detector that never runs the same investigation twice.
- 120 runs of one detector produced **120 distinct trajectories**. The variance is the loop's freedom to
search, not the model's uncertainty about an answer.
- The noise floor a regression budget is derived from measures *how often an unconstrained search happens
to reach the sink* — so the worse the search wanders, the more collateral damage a round is licensed to do.
to reach the sink*, so the worse the search wanders, the more collateral damage a round is licensed to do.
The assumption is also empirically false as a claim about LLMs, and the evidence is now overwhelming:
- **Google Big Sleep** (Nov 2024): the first LLM-discovered zero-day in widely-used real-world software — a
stack buffer underflow in SQLite — found by *variant analysis*, a sandboxed test case, and an
- Google Big Sleep (Nov 2024): the first LLM-discovered zero-day in widely-used real-world software, a
stack buffer underflow in SQLite, found by *variant analysis*, a sandboxed test case, and an
auto-generated root-cause report. Not pattern-matching; reproduction.
- **DARPA AIxCC finals** (DEF CON 33, Aug 2025): seven autonomous systems processed 54M lines of code,
- DARPA AIxCC finals (DEF CON 33, Aug 2025): seven autonomous systems processed 54M lines of code,
patched 43 of 54 synthetic vulnerabilities and **uncovered 18 previously-unknown real 0-days**. Buttercup
(Trail of Bits, 2nd place) found **28 vulnerabilities across 20 CWEs at 90% accuracy, $181 per point,
using non-reasoning LLMs only**, ~$152 per task, 45 minutes each.
@@ -47,11 +47,11 @@ Lazarus-AI, MIT, an open reproduction of Anthropic's Glasswing) share one shape:
> **The LLM is a judgement layer on top of a deterministic substrate that arbitrates its claims. Truth comes
> from that substrate, not from a score against a labeled corpus.**
In the memory-safety systems above the substrate is *execution* — a crash is the arbiter. But execution is
In the memory-safety systems above the substrate is *execution*. A crash is the arbiter. But execution is
only the arbiter you reach for when the target builds in isolation and the bug crashes, which is the C world,
not ours. The general form of the substrate is a **program model built from the code**: a deterministic
abstraction rich enough to *check* a claim — does the source actually reach the sink through resolved
dataflow; is there a guard on the path or isn't there; is the entry point attacker-reachable. Execution is one
abstraction rich enough to *check* a claim (does the source actually reach the sink through resolved
dataflow; is there a guard on the path or isn't there; is the entry point attacker-reachable). Execution is one
way to arbitrate; a model is the way that works for a SAST project and for bugs that never crash.
Clearwing's evidence ladder is the whole thesis in one type:
@@ -81,24 +81,24 @@ real. **We built K-runs and noise floors because we had no ground truth. They ge
This is not a total loss, and the course is not "throw everything away." Two results here are real,
measured, and *absent from Clearwing*:
1. **IR-driven candidate enumeration with a published ceiling.** We measured that the pattern ruleset caps
1. IR-driven candidate enumeration with a published ceiling. We measured that the pattern ruleset caps
recall at 12.1% while call-sites-and-bindings from the semantic IR reach **96.6%** across the web slices,
a median of ten sites per function. Clearwing's candidate feeder is C-first (taint tables for
`memcpy`-shaped bugs); its `web_framework` specialist is a *single unconstrained prompt* with no
equivalent candidate enumerator for logic/web classes.
2. **Absence-bug detection.** Our `obligations` work — an access-control bug is the *absence* of a guard
around an operation its siblings protect — has no analogue in Clearwing, whose ladder is built around a
2. Absence-bug detection. Our `obligations` work (an access-control bug is the *absence* of a guard
around an operation its siblings protect) has no analogue in Clearwing, whose ladder is built around a
*crash*. An IDOR or a missing authorization check never crashes a sandbox. Clearwing's execution
ground-truth model does not reach these classes at all, and they are exactly the "dangerous but
low-hanging" web bugs this project was chartered to catch.
So the gap Clearwing leaves open is **the web/logic taxonomy without a crash oracle** — which is where our
So the gap Clearwing leaves open is **the web/logic taxonomy without a crash oracle**, which is where our
IR candidate enumerator, our family taxonomy, and our obligation checker are genuinely ahead.
## 4b. The model layer, and the evidence ladder it produces
The arbiter is a program model constructed deterministically from the code — the thing an LLM claim is
The arbiter is a program model constructed deterministically from the code, the thing an LLM claim is
*checked against*, so that a finding's truth does not rest on the LLM that proposed it. We already hold the
first rungs of it; they were buried under a noise layer instead of being made the arbiter.
@@ -113,16 +113,16 @@ The evidence ladder is the model layer made legible. Each rung is a stronger, st
Two properties are the whole point:
- **`model_corroborated` and above are deterministic.** Run the checker twice over the same model and it
returns the same verdict. That removes the run-to-run noise the same way a crash would — no K-runs, no
floor, no sign test — *without needing a runnable target*. The noise this session measured was the LLM
- `model_corroborated` and above are deterministic. Run the checker twice over the same model and it
returns the same verdict. That removes the run-to-run noise the same way a crash would (no K-runs, no
floor, no sign test), *without needing a runnable target*. The noise this session measured was the LLM
free-searching; a model check does not search.
- **The LLM's job shrinks to the residual the model cannot decide** — is this string genuinely
attacker-controlled, is this guard semantically sufficient — and even that answer is checked against the
- The LLM's job shrinks to the residual the model cannot decide (is this string genuinely
attacker-controlled, is this guard semantically sufficient), and even that answer is checked against the
model rather than averaged over runs. The LLM stops being the oracle and becomes a proposer the model
audits.
**Measured, 2026-09-08 — the entailment ceiling** (`benchmarks/measurements/2026-09-08-model-entailment-ceiling.json`,
**Measured, 2026-09-08, the entailment ceiling** (`benchmarks/measurements/2026-09-08-model-entailment-ceiling.json`,
offline, no LLM):
| rung | web/logic taxonomy | for comparison |
@@ -134,17 +134,17 @@ offline, no LLM):
The gap between 96.6% (a site exists) and 2.2% (the model *decides* the vulnerability) is the gap between
enumerating and arbitrating, and it is the whole finding. The model cannot be a *pure* arbiter at v1. But it
is not a wall — the measurement diagnoses depth, not impossibility: one-hop source resolution (real taint sits
is not a wall. The measurement diagnoses depth, not impossibility: one-hop source resolution (real taint sits
1–3 binds upstream: `q = "…" + id; execute(q)`), a closed literal sink table (misses project wrappers and the
path/xss/prototype sinks), and the function-naming gap that eats 20/50 injection pairs as `no_function`.
Deepening the taint to reuse the candidate enumerator's depth-4 chain already moved injection 24% → 48%.
So the architecture the numbers support is a **layered arbiter**, not a pure one:
- where the model **entails** — report `model_entailed`, deterministic, the LLM not needed;
- where the model **corroborates a taint path** (~35% and rising) — the LLM judges, but its claim is *checked*
- where the model **entails**, report `model_entailed`, deterministic, the LLM not needed;
- where the model **corroborates a taint path** (~35% and rising), the LLM judges, but its claim is *checked*
against the resolved path, so run-to-run noise is removed on this band;
- where the model can only **enumerate** (the rest, up to 96.6%) — honest `suspicion`, the LLM proposes with
- where the model can only **enumerate** (the rest, up to 96.6%), honest `suspicion`, the LLM proposes with
no deterministic backstop, and this is exactly the band where the deferred execution / differential tier
earns its place.
@@ -152,32 +152,32 @@ Pure `model_entailed`-as-arbiter is the north star. The v1 is the layered arbite
named by the measurement: fix function naming, build per-family sink/sanitizer tables, and lift the flow
model onto the multi-hop binding chain we already have.
This is the classical SAST arbiter — dataflow / abstract interpretation, the CodeQL and Infer lineage —
This is the classical SAST arbiter (dataflow / abstract interpretation, the CodeQL and Infer lineage),
with the LLM supplying the semantic judgement the model is weakest at (aliasing intent, whether a value is
really untrusted, whether a guard really covers the case). It does not need soundness. A partial model that
can confirm *some* claims and honestly returns "could not confirm — stays suspicion" for the rest is already
can confirm *some* claims and honestly returns "could not confirm, stays suspicion" for the rest is already
the independent arbiter we never had.
## 4c. IL or AST, build or adopt — the substrate decision
## 4c. IL or AST, build or adopt, the substrate decision
**IL, and adopt one.** Both halves are forced by the entailment measurement.
*Why an IL, not the AST we have.* Our `semantic/ir.py` is not an IL — it is a lossy AST projection: per-function
*Why an IL, not the AST we have.* Our `semantic/ir.py` is a lossy AST projection rather than an IL: per-function
flat lists of calls and binds, no control-flow graph, no basic blocks, no data-dependence edges. That is
*why* entailment sits at 2.2%. Real abstract interpretation is a fixpoint over a **CFG**: guard *dominance*
(is the check on every path to the operation) is a dominator-tree question, and path-sensitive taint (a
sanitizer on the taint path specifically) needs the branch structure. A flat bind list cannot express either;
it can only over-approximate taint flow-insensitively by name, which is the one-hop ceiling we measured. The
model layer wants the classical IL that merges AST + CFG + PDG + call graph into one graph — a **Code Property
Graph (CPG)** — over which taint is a reachability query (source → sink minus sanitizer) and absence is a
model layer wants the classical IL that merges AST + CFG + PDG + call graph into one graph (a **Code Property
Graph (CPG)**), over which taint is a reachability query (source → sink minus sanitizer) and absence is a
dominance query.
*Why adopt, not build.* Building a multi-language CFG/PDG/taint engine on top of tree-sitter is precisely
reinventing CodeQL, Semgrep and Joern — a multi-year effort, and the maximal case of the failure mode this
reinventing CodeQL, Semgrep and Joern, a multi-year effort, and the maximal case of the failure mode this
session logged eighteen times. **Joern** is the adoptable answer: Apache-2.0, a CPG engine for exactly our
corpus (C/C++/Java/JavaScript/Python/Kotlin/binary), a built-in taint-propagation engine, embeddable, and
there is already a published CPG-plus-LLM bridge (Codebadger, an MCP server over Joern) and an arXiv line on
CPGs as the substrate for LLM program analysis. Joern *is* "a model built up from code" as a graph — the exact
CPGs as the substrate for LLM program analysis. Joern *is* "a model built up from code" as a graph, the exact
thing the maintainer described. The alternatives: CodeQL is more powerful but least adoptable (restrictive
license, query-only, GitHub-owned); Semgrep's taint engine is lighter and we already have integration
precedent through Clearwing's sidecar, but interprocedural taint is behind the paid Pro engine.
@@ -192,14 +192,14 @@ source ──▶ Joern: compile to CPG (one IL: AST + CFG + PDG + call graph, la
```
The CPG is language-agnostic once its frontend exists, so the taint and dominance logic is written *once*, not
per language — which is the whole reason our per-language tree-sitter approach kept hitting naming and grammar
per language, which is the whole reason our per-language tree-sitter approach kept hitting naming and grammar
gaps.
*What stays ours, and is the contribution.* Joern is a workbench, not an opinionated web/logic detector with
an LLM judge. What we own and port onto it: the closed **family taxonomy**; the **source / sink / sanitizer /
guard models** for the web/logic CWEs — our obligation facts and closed guard vocabulary become CPG queries
and taint specifications; and the **LLM-adjudication-checked-against-the-CPG** orchestration. That security
modelling — expressed as queries over an adopted CPG — is where our work is genuinely ahead of both Joern
guard models** for the web/logic CWEs (our obligation facts and closed guard vocabulary become CPG queries
and taint specifications); and the **LLM-adjudication-checked-against-the-CPG** orchestration. That security
modelling, expressed as queries over an adopted CPG, is where our work is ahead of both Joern
(no opinion on these classes) and Clearwing (crash-oracle, no arbiter for bugs that do not crash).
*The one real cost.* Joern is JVM/Scala; embedding it in a Python tool means its server mode or the MCP bridge,
@@ -212,38 +212,38 @@ noise architecture; keep execution as a later, conditional tier.**
Concretely, superseding both prior specs:
1. **Delete, do not fix.** The K-run machinery, the per-family noise floor, the regression budget, the sign
1. Delete, do not fix. The K-run machinery, the per-family noise floor, the regression budget, the sign
test, the evolve loop, the acceptance rule, the mechanism-lever optimizer. All of it existed to average an
LLM oracle we could not otherwise trust. A deterministic model arbiter removes the need for every piece.
`learning-harness` Req 3.5, 8.2, 9.5 and the evolve tasks are retired, not amended.
2. **Adopt a CPG engine (Joern) as the arbiter substrate; port our security models onto it.** We do not lift
2. Adopt a CPG engine (Joern) as the arbiter substrate; port our security models onto it. We do not lift
our flat IR to a CFG ourselves (§4c). The taxonomy, the source/sink/sanitizer/guard models, and the
obligation/absence logic move from prompt context to **CPG taint specifications and dominance queries** —
obligation/absence logic move from prompt context to **CPG taint specifications and dominance queries**,
deterministic `model_corroborated` and `model_entailed` verdicts over an engine that already has the CFG
and PDG our IR lacks. Our tree-sitter IR retains one job: cheap candidate enumeration for the `suspicion`
band where the CPG has no verdict.
3. **The LLM proposes; the model disposes.** One bounded, typed question per candidate — the constrained
detector we already started — but the answer is not scored against a corpus, it is *checked against the
model*. A claim the model corroborates advances; a claim it contradicts is dropped with the contradiction
3. The LLM proposes; the model disposes. One bounded, typed question per candidate (the constrained
detector we already started), with the answer *checked against the
model* rather than scored against a corpus. A claim the model corroborates advances; a claim it contradicts is dropped with the contradiction
recorded; a claim it can neither confirm nor deny stays `suspicion` and is reported as such. No K-runs, no
evolve loop.
4. **Corpus is the model's calibration set, not the LLM's training signal.** Keep the vulnerable-parent /
fix-commit pairs — but their role changes. The pair is ground truth for *the model's precision*: the fix
4. Corpus is the model's calibration set, not the LLM's training signal. Keep the vulnerable-parent /
fix-commit pairs, but their role changes. The pair is ground truth for *the model's precision*: the fix
is what the model should be able to see (a guard added, a sanitizer inserted, a bound checked), and a pair
the model cannot distinguish is a gap in the model, named and measured, not a detector miss to average.
Harvest more pairs at repository scale from CVE history, as Clearwing does; truth is the fix diff read by
the model, not a label we hand-assign.
5. **Execution is a deferred, conditional tier.** For the classes a static model genuinely cannot arbitrate —
memory-safety-in-C, where reachability and exploitability turn on runtime layout — a sandbox crash oracle
5. Execution is a deferred, conditional tier. For the classes a static model genuinely cannot arbitrate
(memory-safety-in-C, where reachability and exploitability turn on runtime layout), a sandbox crash oracle
is the right arbiter, and Clearwing already is that system. Adopt it *there*, later, only when a class
needs it. It is the `execution_confirmed` rung, not the foundation.
6. **Reporting keeps the honesty disciplines that stand:** per-slice numbers with the real-world slice
deciding, the overfitting gap published, and the evidence rung on every finding — now meaning a model
6. Reporting keeps the honesty disciplines that stand: per-slice numbers with the real-world slice
deciding, the overfitting gap published, and the evidence rung on every finding, now meaning a model
verdict, not a sandbox result.
## 6. Build vs. adopt, revised
@@ -252,14 +252,14 @@ The earlier draft of this brief over-rotated to "adopt Clearwing wholesale," bec
foregrounds execution. The maintainer's correction re-centres it, and the honest split is cleaner than either
extreme:
- **The model layer is adopt-plus-contribute, not build.** The arbiter *engine* is Joern's CPG (Apache-2.0,
our languages, taint + dominance built in); the *contribution* is our security modelling on top of it — the
- The model layer is adopt-plus-contribute, not build. The arbiter *engine* is Joern's CPG (Apache-2.0,
our languages, taint + dominance built in); the *contribution* is our security modelling on top of it, the
taxonomy and the source/sink/sanitizer/guard specs for the web/logic classes, plus the LLM adjudication
checked against the CPG. The entailment measurement (2.2% on our flat IR) is what rules out building the
engine ourselves: that number is the distance between our AST projection and a real CPG, and closing it by
hand is reimplementing Joern.
- **The execution tier is Clearwing's to adopt**, later, for the memory-safety classes where it is strong and
we are weak — via the fork the maintainer already maintains. We do not reimplement its sandbox lifecycle,
- The execution tier is Clearwing's to adopt, later, for the memory-safety classes where it is strong and
we are weak, via the fork the maintainer already maintains. We do not reimplement its sandbox lifecycle,
container pooling, sanitizer images or PoC stability; those are exactly the parts this session proved we get
wrong.
@@ -268,24 +268,24 @@ arbiter we cannot, when a class demands it.
## Open questions for the requirements phase
- **How far up the ladder can a partial model reach without soundness?** `model_corroborated` needs only a
- How far up the ladder can a partial model reach without soundness? `model_corroborated` needs only a
consistency check; `model_entailed` needs a real interprocedural dataflow/guard-dominance result. What
fraction of our corpus can the model *entail* today, and where does it honestly fall back to `suspicion`?
This is the first measurement to run — the model analogue of the 96.6% candidate ceiling — and it can end
This is the first measurement to run (the model analogue of the 96.6% candidate ceiling), and it can end
or reshape the design before any LLM cost, exactly as the ceiling did.
- **What is the model's arbiter for an absence bug?** A missing guard has no dataflow path to confirm. The
- What is the model's arbiter for an absence bug? A missing guard has no dataflow path to confirm. The
obligation/sibling-dominance model is the candidate arbiter; does it distinguish the vulnerable-parent from
the fixed twin across the corpus on its own, deterministically?
- **Where does the LLM's residual judgement resist checking?** Some claims — "this value is attacker-
controlled by convention", "this guard is semantically insufficient" — the model cannot audit. How large is
- Where does the LLM's residual judgement resist checking? Some claims ("this value is attacker-
controlled by convention", "this guard is semantically insufficient") the model cannot audit. How large is
that residual, and is `suspicion` an acceptable home for it or does it need the execution tier sooner?
- **Which classes cross into execution, and when.** Memory-safety-in-C is the clear case for the deferred
- Which classes cross into execution, and when. Memory-safety-in-C is the clear case for the deferred
tier. Is anything in the web/logic taxonomy genuinely un-arbitrable by a model, forcing the differential
request oracle earlier than "later, if needed"?
- **What of the existing corpus survives** as the model's calibration set, given the twin/straddle/tier
- What of the existing corpus survives as the model's calibration set, given the twin/straddle/tier
defects the learning-harness reviews surfaced?
Sources: [Big Sleep / SQLite zero-day](https://thehackernews.com/2024/11/googles-ai-tool-big-sleep-finds-zero.html) ·
[AIxCC finals — Buttercup, 28 vulns / 20 CWEs / non-reasoning LLMs](https://blog.trailofbits.com/2025/08/09/trail-of-bits-buttercup-wins-2nd-place-in-aixcc-challenge/) ·
[AIxCC results — 18 real 0-days, 43/54 patched](https://semiengineering.com/aixcc-2025-what-it-means-for-device-security/) ·
[AIxCC finals, Buttercup, 28 vulns / 20 CWEs / non-reasoning LLMs](https://blog.trailofbits.com/2025/08/09/trail-of-bits-buttercup-wins-2nd-place-in-aixcc-challenge/) ·
[AIxCC results, 18 real 0-days, 43/54 patched](https://semiengineering.com/aixcc-2025-what-it-means-for-device-security/) ·
[Clearwing](https://github.com/Lazarus-AI/clearwing) · local: `/home/mc/Source/clearwing`, fork `norandom/deepwing`.
+20 -20
View File
@@ -4,14 +4,14 @@
The detector stops being a noisy oracle and becomes a proposer that a deterministic program model audits. The
model is a **Code Property Graph** (Joern, Apache-2.0): one graph merging AST, CFG, PDG and call graph, over
which taint reachability and guard dominance produce verdicts on the evidence ladder — `suspicion`,
which taint reachability and guard dominance produce verdicts on the evidence ladder, `suspicion`,
`model_corroborated`, `model_entailed`, `execution_confirmed`. The LLM is asked one bounded question per
candidate; its answer is checked against the CPG, not averaged over runs.
Two disciplines shape every decision below. **Subtraction first:** the noise architecture and the outgrown
test surface are removed and audited before the CPG arbiter is adopted, because the session that produced this
spec logged eighteen defects building on an overgrown surface. **Adopt, don't build:** the CFG/PDG our flat IR
lacks comes from Joern, not from a hand-built engine — the 2.2% entailment ceiling is the measured distance
lacks comes from Joern, not from a hand-built engine. The 2.2% entailment ceiling is the measured distance
between our AST projection and a real CPG, and closing it by hand is reimplementing Joern.
### Goals
@@ -19,12 +19,12 @@ between our AST projection and a real CPG, and closing it by hand is reimplement
- Remove the noise architecture and reduce the test suite with it, gate-protected and reversible.
- A deterministic model verdict on the band the model covers; honest `suspicion` on the rest.
- The CPG engine, the LLM endpoint and any execution tier are optional; the core stays zero-dependency.
- The security modelling — taxonomy, source/sink/sanitizer/guard specs — is ours, expressed as CPG queries.
- The security modelling (taxonomy, source/sink/sanitizer/guard specs) is ours, expressed as CPG queries.
### Non-Goals
- Building a control-flow or dataflow engine (adopted, Req 4). Prompt optimisation, evolve loops, acceptance
rules (deleted, Req 8.5). The execution tier itself (deferred to Clearwing, Req 10). Soundness — the
rules (deleted, Req 8.5). The execution tier itself (deferred to Clearwing, Req 10). Soundness, the
north-star `model_entailed`-as-pure-arbiter is not v1.
## Boundary Commitments
@@ -32,18 +32,18 @@ between our AST projection and a real CPG, and closing it by hand is reimplement
### This Spec Owns
- The subtraction: which modules are removed, the dangling-import check, the module-audit manifest.
- `cpg/` — the Joern seam (`CpgBackend`, the subprocess/query implementation, the null backend).
- `model/` — `Verdict`/`Rung`, taint reachability, guard dominance, constant abstraction; the security specs
- `cpg/`, the Joern seam (`CpgBackend`, the subprocess/query implementation, the null backend).
- `model/`, `Verdict`/`Rung`, taint reachability, guard dominance, constant abstraction; the security specs
ported from the deleted modules' vocabularies.
- The proposer/judge orchestration where the LLM's claim is checked against the model.
- The corpus-as-calibration harness and its model-gap report.
### Out of Boundary
- The family **taxonomy** (`learning/families.py`) — retained unchanged, re-homed under `model/`.
- The corpus loader, per-slice split and gates — retained; the gates stay byte-identical.
- Redaction, the LLM endpoint client (`learning/endpoint.py`) — retained.
- Clearwing's sandbox lifecycle — never reimplemented; the execution tier adopts it.
- The family **taxonomy** (`learning/families.py`), retained unchanged, re-homed under `model/`.
- The corpus loader, per-slice split and gates, retained; the gates stay byte-identical.
- Redaction, the LLM endpoint client (`learning/endpoint.py`), retained.
- Clearwing's sandbox lifecycle, never reimplemented; the execution tier adopts it.
### Allowed Dependencies
@@ -63,7 +63,7 @@ between our AST projection and a real CPG, and closing it by hand is reimplement
### Existing Architecture Analysis
The `learning/` package (4,091 lines, 16 modules) implements the noise architecture and is **absent from the
gate path** — the gates import only from `pairs`, `benchmark`, `findings`, `mapping`, `preprocess`, `rank`,
gate path**. The gates import only from `pairs`, `benchmark`, `findings`, `mapping`, `preprocess`, `rank`,
`complexity.map`. So the subtraction is clean: removing `rounds`, `acceptance`, `proposer`, `journal`, the
noise machinery in `scoring`, and the mechanism loop cannot move a gate. What survives and re-homes: the
taxonomy (`families.py`), the candidate enumerator (`candidates.py`, the `suspicion`-band feeder), the LLM
@@ -140,7 +140,7 @@ src/openultrasast/model/
- `learning/candidates.py` → `model/candidates.py` (the suspicion-band enumerator, Req 8.4).
- `learning/endpoint.py` → `model/endpoint.py` (the judge client).
- `semantic/obligations/` retained; its operation/discharge facts feed `model/dominance.py`.
- `pairs.py`, the gates, redaction — unchanged.
- `pairs.py`, the gates, redaction, unchanged.
### The audit manifest (Req 3)
@@ -288,21 +288,21 @@ is dominated by no discharger and its siblings are, else `CORROBORATED`/`None`.
## Migration and Phases
1. **Subtract and audit (Req 1–3).** Remove the noise architecture; port the guard/sink vocabularies into
1. Subtract and audit (Req 1–3). Remove the noise architecture; port the guard/sink vocabularies into
`model/specs.py` before deleting their modules; run the import-graph check and the gate-baseline; reduce
the tests in the same change; commit the audit manifest. Gate-protected and reversible; no CPG yet.
2. **Joern spike on one family (Req 4, 6.1).** The `CpgBackend` seam and the injection `TaintSpec`, proven
2. Joern spike on one family (Req 4, 6.1). The `CpgBackend` seam and the injection `TaintSpec`, proven
end to end on the injection slice: build a CPG, run the taint query, produce entailed/corroborated
verdicts, assert determinism. This phase can stop the design if Joern cannot deliver a deterministic
verdict at acceptable cost — the analogue of the candidate-ceiling gate.
3. **The ladder and the judge (Req 5, 8).** `Verdict`/`Rung`, and `judge.py` checking the LLM's residual
verdict at acceptable cost, the analogue of the candidate-ceiling gate.
3. The ladder and the judge (Req 5, 8). `Verdict`/`Rung`, and `judge.py` checking the LLM's residual
answer against the verdict. No K-runs, nothing to average.
4. **Absence and the rest of the taxonomy (Req 6.2, 7).** Dominance queries from the obligation facts; the
4. Absence and the rest of the taxonomy (Req 6.2, 7). Dominance queries from the obligation facts; the
remaining `TaintSpec`s. Regenerate the entailment ceiling (Req 6.5) and watch it rise as depth improves.
5. **Corpus calibration and reporting (Req 9, 11).** The vulnerable-parent/fix differential, the model-gap
5. Corpus calibration and reporting (Req 9, 11). The vulnerable-parent/fix differential, the model-gap
report, per-slice + overfitting gap.
6. **Execution tier stub (Req 10).** The `execution_confirmed` rung and the Clearwing adoption seam, built
only when a class needs it — memory/C first, and not for web/logic v1.
6. Execution tier stub (Req 10). The `execution_confirmed` rung and the Clearwing adoption seam, built
only when a class needs it, memory/C first, and not for web/logic v1.
Phase 1 removes ~5,000 lines before a line of Joern is written. Phase 2 can end the design before the
integration cost is paid, exactly as the candidate ceiling could have in the prior spec.
@@ -6,12 +6,12 @@
This feature supersedes `learning-harness` and `constrained-detector`. It replaces the assumption that the LLM
is a noisy oracle to be averaged, floored and gated with an architecture in which a **deterministic program
model arbitrates the LLM's claims**. The arbiter is a Code Property Graph (CPG) — an intermediate language
merging AST, control-flow graph, program-dependence graph and call graph — over which taint reachability and
model arbitrates the LLM's claims**. The arbiter is a Code Property Graph (CPG), an intermediate language
merging AST, control-flow graph, program-dependence graph and call graph, over which taint reachability and
guard dominance produce verdicts the LLM's assertion is no longer needed for. The measurements that force this
design are committed: the candidate ceiling (96.6% of labeled functions have an enumerable site), and the
entailment ceiling (a deterministic model *decides* only 2.2% today on our flat AST-projection IR, ~24-35%
with deeper taint — the distance to a real CPG). Context: `brief.md`,
with deeper taint, the distance to a real CPG). Context: `brief.md`,
`benchmarks/measurements/2026-09-06-trajectory-divergence.json`, `2026-09-06-candidate-ceiling-all-slices.json`,
`2026-09-08-model-entailment-ceiling.json`.
@@ -22,15 +22,15 @@ overgrown, superseded surface is how this session logged eighteen defects.
## Boundary Context
- **In scope**: removing the noise architecture and the superseded arbiter; reducing the test suite to
- In scope: removing the noise architecture and the superseded arbiter; reducing the test suite to
surviving behavior; auditing the whole tree for orphaned modules; adopting a CPG engine (Joern) as the
model-layer substrate behind an optional extra; the evidence ladder as the verdict type; taint + dominance
verdicts; porting our taxonomy and source/sink/sanitizer/guard models to the CPG; the LLM as a proposer
checked against the model; the corpus as the model's calibration set; per-slice reporting.
- **Out of scope**: building a dataflow/CFG engine by hand (the CPG is adopted, §Req 4); the execution tier
- Out of scope: building a dataflow/CFG engine by hand (the CPG is adopted, §Req 4); the execution tier
itself (deferred to Clearwing, `execution_confirmed` rung only, Req 10); prompt optimisation, reflective
evolution, any optimiser (deleted, not deferred); fine-tuning weights.
- **North star, not v1**: pure `model_entailed`-as-arbiter. The entailment ceiling says the model decides a
- North star, not v1: pure `model_entailed`-as-arbiter. The entailment ceiling says the model decides a
minority today; v1 is the *layered* arbiter (entail where it can, corroborate the taint band, honest
suspicion for the rest). This is stated so the requirements are not read as promising a soundness the
measurement denies.
@@ -45,12 +45,12 @@ so that nothing is maintained, tested, or built upon that this feature's measure
#### Acceptance Criteria
1. The system shall remove the round loop, the per-family noise floor, the regression budget, the acceptance
rule, the sign test, the evolve journal and the meta-agent proposer — concretely `learning/rounds.py`,
rule, the sign test, the evolve journal and the meta-agent proposer, concretely `learning/rounds.py`,
`learning/acceptance.py`, `learning/proposer.py`, `learning/journal.py`, and the K-run / floor / sign-test
machinery in `learning/scoring.py`.
2. The system shall remove the mechanism-optimisation loop — the mechanism store and lever in
2. The system shall remove the mechanism-optimisation loop (the mechanism store and lever in
`improve/evolve.py`, and the shape-search machinery in `semantic/mechanisms.py` and the optimisation
surface of `semantic/variants.py` — while preserving the *security models* those modules carried (the
surface of `semantic/variants.py`), while preserving the *security models* those modules carried (the
guard vocabulary and source/sink classification) for porting under Req 7.
2b. The system shall remove the per-family evolvable detector configuration and its canary/verifier surface
(`learning/detectors.py` config-evolution parts, `learning/canaries.py`, `learning/verifiers.py`,
@@ -70,7 +70,7 @@ ratchet that pins deleted mechanisms and the suite is once again a description o
1. When a mechanism is removed under Req 1, the tests that existed only to pin its behavior shall be removed
with it, in the same change, not left skipped or xfailed.
2. Every surviving test shall map to a surviving requirement — of this spec or of a still-shipping feature —
2. Every surviving test shall map to a surviving requirement (of this spec or of a still-shipping feature)
and a test that maps to nothing shall be removed.
3. The reduced suite shall be smaller than the current 832 tests, and the change shall report the count before
and after with the requirement each removed test cluster served.
@@ -78,7 +78,7 @@ ratchet that pins deleted mechanisms and the suite is once again a description o
determinism test (Req 6.4), the per-slice honesty tests, and the two ceiling measurements' regeneration
tests.
5. A test that asserts a shape "does nothing" or greps `--help`, or that passes with its capability absent,
shall be removed rather than kept — the session logged several; none survive.
shall be removed rather than kept. The session logged several; none survive.
### Requirement 3: Audit the whole tree for outgrowth
@@ -89,7 +89,7 @@ found and removed rather than carried indefinitely.
1. The system shall classify every source module as one of: load-bearing for the model-grounded architecture;
a retained standalone capability with its own users; or orphaned.
2. If a module is orphaned — no importer, no CLI entry, no gate dependency, no surviving requirement — then the
2. If a module is orphaned (no importer, no CLI entry, no gate dependency, no surviving requirement), then the
system shall remove it and record the classification.
3. The system shall name, for the audit, the older-phase subsystems whose status is not obvious
(`fusion.py`, `mcp.py`, `skills.py`, `harness_ext.py`, `hunter_harness.py`) and give each an explicit
@@ -122,7 +122,7 @@ model result is never confused with an LLM suspicion.
#### Acceptance Criteria
1. The system shall assign every finding one rung: `suspicion` (a site was enumerated / the LLM flagged it),
`model_corroborated` (the LLM's claim is consistent with the CPG — the source it names reaches the sink
`model_corroborated` (the LLM's claim is consistent with the CPG, the source it names reaches the sink
through resolved dataflow, or a guard its siblings carry is absent), `model_entailed` (the CPG alone
establishes it, independent of trusting the LLM), or `execution_confirmed` (a sandbox reproduced it).
2. The system shall establish `model_corroborated` and `model_entailed` from the CPG deterministically, with
@@ -145,21 +145,21 @@ project measured is removed on the band the model covers.
4. When the system runs the model layer twice over the same CPG, the sequence of verdicts shall be identical,
and a test shall assert it.
5. The system shall report, per family and per slice, the fraction of the corpus it can `model_entail` and
`model_corroborate` — the regeneration of the committed entailment ceiling — and shall name where it falls
`model_corroborate` (the regeneration of the committed entailment ceiling), and shall name where it falls
back to `suspicion`.
### Requirement 7: Port the taxonomy and security models onto the CPG
**Objective:** As a maintainer, I want our security knowledge — the classes we target and how their bugs look
— expressed as queries over the adopted CPG, so that the contribution is the modelling, not the engine.
**Objective:** As a maintainer, I want our security knowledge (the classes we target and how their bugs look)
expressed as queries over the adopted CPG, so that the contribution is the modelling, not the engine.
#### Acceptance Criteria
1. The system shall retain the closed family taxonomy as the routing key, unchanged in meaning.
2. The system shall express the source, sink and sanitizer models for the web/logic CWEs as CPG taint
specifications, seeded from the models the deleted modules carried.
3. The system shall express the obligation / absence model — an operation on a protected resource, and the
guards that discharge it — as CPG dominance queries, as the arbiter for the classes that never crash.
3. The system shall express the obligation / absence model (an operation on a protected resource, and the
guards that discharge it) as CPG dominance queries, as the arbiter for the classes that never crash.
4. The system shall keep the verifier boundary: the models and gold labels stay outside anything an LLM or a
future optimiser may write.
@@ -177,7 +177,7 @@ with its claim checked against the model, so that there is no oracle to average.
`model_corroborated`.
4. When the model can neither confirm nor contradict, the system shall report the claim at `suspicion`, and
this band is where a future execution tier earns its place.
5. The system shall contain no K-run averaging, no noise floor, no acceptance rule, no evolve loop — the model
5. The system shall contain no K-run averaging, no noise floor, no acceptance rule, no evolve loop. The model
check replaces all of them.
### Requirement 9: The corpus is the model's calibration set
@@ -188,7 +188,7 @@ vote, so that a gap is a gap in the model, named and fixable.
#### Acceptance Criteria
1. The system shall use the vulnerable-parent / fix-commit pair as ground truth for whether the model can
distinguish the two — the fix is what the model should be able to see.
distinguish the two. The fix is what the model should be able to see.
2. If the model cannot distinguish a pair's two sides, then the system shall record it as a named model gap
with the reason, not as a detector miss to be averaged away.
3. The system shall keep the per-slice split with the real-world slices deciding, and shall not present the
@@ -204,7 +204,7 @@ arbitrate, without building or requiring it at v1.
#### Acceptance Criteria
1. The system shall treat `execution_confirmed` as an optional rung reached only for classes where the model
layer cannot decide — memory-safety in C first.
layer cannot decide, memory-safety in C first.
2. When execution confirmation is used, the system shall adopt it from Clearwing (the existing fork), not
reimplement the sandbox lifecycle, container pooling, sanitizer images or PoC stability.
3. The system shall not require the execution tier for any web/logic class at v1.
+7 -7
View File
@@ -25,8 +25,8 @@ Fix the scorer and labels first (no engine change). Anchor harvest at the declar
## Scope
- **In**: scorer fidelity; label schema; harvest anchor and modes; corpus hygiene for sast; three new slices; dataset pointers; provenance fingerprint; per-profile and per-mechanism dashboard; per-profile improve-loop gate; offline tests.
- **Out**: any change to taint, facts, CST walker, or dispositions (engine specs); tolerating tree-sitter ERROR nodes (engine); BaxBench generation and exploit-labelled slices (needs deep mode; later); profile-specific fact tables; making any dashboard a merge gate; vendoring SecBench.js test files (no license).
- In: scorer fidelity; label schema; harvest anchor and modes; corpus hygiene for sast; three new slices; dataset pointers; provenance fingerprint; per-profile and per-mechanism dashboard; per-profile improve-loop gate; offline tests.
- Out: any change to taint, facts, CST walker, or dispositions (engine specs); tolerating tree-sitter ERROR nodes (engine); BaxBench generation and exploit-labelled slices (needs deep mode; later); profile-specific fact tables; making any dashboard a merge gate; vendoring SecBench.js test files (no license).
## Boundary Candidates
@@ -42,14 +42,14 @@ Fix the scorer and labels first (no engine change). Anchor harvest at the declar
## Upstream / Downstream
- **Upstream**: pair catalog schema; overlay eval path; `build_pair_signals`; `run_improvement` accept gate; preprocess tags.
- **Downstream**: `overlay-ir-completeness` and `guard-dominance-regime` measure against these slices and vocabularies; later `baxgen` slice reuses provenance and mechanism fields.
- Upstream: pair catalog schema; overlay eval path; `build_pair_signals`; `run_improvement` accept gate; preprocess tags.
- Downstream: `overlay-ir-completeness` and `guard-dominance-regime` measure against these slices and vocabularies; later `baxgen` slice reuses provenance and mechanism fields.
## Existing Spec Touchpoints
- **Supersedes**: `real-world-vfc-slice` (corpus/scorer plane). Its seed pairs, harvest script, and dual scoreboard are kept and extended.
- **Extends**: roadmap direct candidate "record inventory vs overlay Youden side-by-side in sast JSON".
- **Adjacent**: `propose-adjudicate-prove` (one additive field on `OverlayRecord`), `harnessx-self-improving-rulesets` (accept gate grows a per-profile clause, ledger format unchanged).
- Supersedes: `real-world-vfc-slice` (corpus/scorer plane). Its seed pairs, harvest script, and dual scoreboard are kept and extended.
- Extends: roadmap direct candidate "record inventory vs overlay Youden side-by-side in sast JSON".
- Adjacent: `propose-adjudicate-prove` (one additive field on `OverlayRecord`), `harnessx-self-improving-rulesets` (accept gate grows a per-profile clause, ledger format unchanged).
## Constraints
+5 -5
View File
@@ -172,11 +172,11 @@ tests/
### Modified Files
- `src/openultrasast/pairs.py` — see above; `OVERLAY_SLICES` grows; `PairCase` gains `provenance`, `mechanism` (on expected), `split`, `known_limit`.
- `src/openultrasast/semantic/overlay.py` — fill `function` from the `FunctionIR` whose range contains the record line; default `None`.
- `src/openultrasast/improve/evolve.py` — `_Eval` computes `per_profile` on holdout; new reject reason `profile_regression:<profile>`.
- `src/openultrasast/preprocess.py`, `reports.py`, `cli.py` — wiring only.
- `benchmarks/pairs/README.md`, `.kiro/steering/roadmap.md` — slice list and spec order.
- `src/openultrasast/pairs.py`: see above; `OVERLAY_SLICES` grows; `PairCase` gains `provenance`, `mechanism` (on expected), `split`, `known_limit`.
- `src/openultrasast/semantic/overlay.py`: fill `function` from the `FunctionIR` whose range contains the record line; default `None`.
- `src/openultrasast/improve/evolve.py`: `_Eval` computes `per_profile` on holdout; new reject reason `profile_regression:<profile>`.
- `src/openultrasast/preprocess.py`, `reports.py`, `cli.py`: wiring only.
- `benchmarks/pairs/README.md`, `.kiro/steering/roadmap.md`: slice list and spec order.
## System Flows
@@ -8,9 +8,9 @@ Engineers evolving OpenUltraSAST's overlay cannot trust the pair scoreboard: cov
## Boundary Context
- **In scope**: pair scorer fidelity; catalog label schema (function, mechanism, provenance, known limits, held-out split); harvest anchoring and extraction modes; sast corpus hygiene; `vibe-py`, `vfc-js`, `agent-vfc` slices; dataset pointers; deterministic provenance fingerprint at scan time; per-profile and per-mechanism dashboard; per-profile acceptance in the improve loop; offline tests.
- **Out of scope**: taint, facts, CST walker, dispositions, or tolerance of parser ERROR nodes (engine specs); BaxBench generation slices; profile-specific fact tables or thresholds; any new merge gate; vendoring files without a stated license; LLM-based classification.
- **Adjacent expectations**: `local` stays the 100% pair gate; the stage-1 smoke gate stays inventory-only; `OverlayRecord` may gain one optional additive field; the improve loop's existing accept gate (recall floor, FP ceiling, score, no matched regression) stays and grows one clause; ledger file formats do not change.
- In scope: pair scorer fidelity; catalog label schema (function, mechanism, provenance, known limits, held-out split); harvest anchoring and extraction modes; sast corpus hygiene; `vibe-py`, `vfc-js`, `agent-vfc` slices; dataset pointers; deterministic provenance fingerprint at scan time; per-profile and per-mechanism dashboard; per-profile acceptance in the improve loop; offline tests.
- Out of scope: taint, facts, CST walker, dispositions, or tolerance of parser ERROR nodes (engine specs); BaxBench generation slices; profile-specific fact tables or thresholds; any new merge gate; vendoring files without a stated license; LLM-based classification.
- Adjacent expectations: `local` stays the 100% pair gate; the stage-1 smoke gate stays inventory-only; `OverlayRecord` may gain one optional additive field; the improve loop's existing accept gate (recall floor, FP ceiling, score, no matched regression) stays and grows one clause; ledger file formats do not change.
## Requirements
@@ -23,7 +23,7 @@ Engineers evolving OpenUltraSAST's overlay cannot trust the pair scoreboard: cov
1. When an overlay slice pair is scored, the system shall count a `coverage` record that carries a source-to-sink flow as a detection on the vulnerable side and as a leak on the fixed side, symmetrically with `promote`.
2. When an expected row names a function, the system shall match a promotion or coverage record only if the record lies inside that function's line range.
3. When an expected row names neither a rule id nor a sink nor a function, the system shall not count a CWE-only match as a detection and shall flag the row as `weak_label` in the payload.
4. When any overlay slice is evaluated, the system shall report inventory and overlay metrics side by side for that slice, not only for `vfc`.
4. When any overlay slice is evaluated, the system shall report inventory and overlay metrics side by side for that slice as well as for `vfc`.
5. When a pair is evaluated, the system shall report how many files on each side were `parse_failed` and how many proposals were `unadjudicated`, so parser loss is visible on the dashboard.
6. The system shall keep `local` as the only slice whose result can fail CI.
7. Where a hunter model is configured, the system shall report a third scorer, `hunter`, that runs the tool hunter on each isolated pair function and scores it with the same pair rules, so the LLM plane is measured on the same corpus; absence of a model shall skip the scorer with a recorded degradation.
+47 -47
View File
@@ -2,9 +2,9 @@
## Summary
- **Feature**: `pair-corpus-honesty`
- **Discovery Scope**: Extension of the pair catalog, scorer, and harvest; supersedes `real-world-vfc-slice`
- **Key Findings**:
- Feature: `pair-corpus-honesty`
- Discovery Scope: Extension of the pair catalog, scorer, and harvest; supersedes `real-world-vfc-slice`
- Key Findings:
- The overlay scorer ignores `coverage`. The Java sqli vuln file already produces a coverage record on `prepareCall` from a servlet source; the scorer reports MISS.
- The overlay column equals the inventory column on all 176 vfc pairs because C facts have no memory sink, so every proposal is `flow_incomplete` and the scorer falls back to inventory. The dashboard cannot show engine progress until the scorer reports parser and adjudication loss explicitly.
- Harvest anchors on the first textual occurrence of the function name. For curl that is the doc comment mention (`Curl_follow()`), so 25 pairs start mid-comment and fail to parse.
@@ -18,23 +18,23 @@
### Scorer blind spots
- **Context**: Why the sast overlay number did not move after the tree-sitter extra.
- **Sources Consulted**: `src/openultrasast/pairs.py` (`_evaluate_overlay_pair`, `_overlay_matches_expected`, `_overlay_leaks`), `semantic/overlay.py`, direct runs of `adjudicate` on the sast files.
- **Findings**: only `promote` is counted; `coverage` is excluded on both sides. `_overlay_matches_expected` accepts a CWE-only match. `ExpectedFinding` already has `function` and `line`, and `benchmark._matches_expected` honors them, but the overlay path does not. `OverlayRecord` has no function field; `FunctionIR` has line ranges.
- **Implications**: count coverage symmetrically; add optional `function` to `OverlayRecord`; require function or sink or rule id for a match when present; flag CWE-only rows.
- Context: Why the sast overlay number did not move after the tree-sitter extra.
- Sources Consulted: `src/openultrasast/pairs.py` (`_evaluate_overlay_pair`, `_overlay_matches_expected`, `_overlay_leaks`), `semantic/overlay.py`, direct runs of `adjudicate` on the sast files.
- Findings: only `promote` is counted; `coverage` is excluded on both sides. `_overlay_matches_expected` accepts a CWE-only match. `ExpectedFinding` already has `function` and `line`, and `benchmark._matches_expected` honors them, but the overlay path does not. `OverlayRecord` has no function field; `FunctionIR` has line ranges.
- Implications: count coverage symmetrically; add optional `function` to `OverlayRecord`; require function or sink or rule id for a match when present; flag CWE-only rows.
### Harvest anchor defect
- **Context**: 50 vfc files `parse_failed`.
- **Sources Consulted**: `benchmarks/pairs/vfc/harvest.py` `extract_function`; tree-sitter ERROR nodes on `curl-cve-2022-27774/vuln.c` and `curl-cve-2023-38545-do_socks5/vuln.c`.
- **Findings**: the name search accepts a match followed by `(` even inside a comment, and the brace search then finds the real body. A second, smaller class is `#if`-split `else if` chains, which tree-sitter cannot attach; that is an engine tolerance question, not harvest.
- **Implications**: skip comment and string regions before searching; anchor on a line that starts a declarator; add line-range and hunk modes so Python and JS/TS are covered; report parse loss on the dashboard so the engine spec can measure its tolerance work.
- Context: 50 vfc files `parse_failed`.
- Sources Consulted: `benchmarks/pairs/vfc/harvest.py` `extract_function`; tree-sitter ERROR nodes on `curl-cve-2022-27774/vuln.c` and `curl-cve-2023-38545-do_socks5/vuln.c`.
- Findings: the name search accepts a match followed by `(` even inside a comment, and the brace search then finds the real body. A second, smaller class is `#if`-split `else if` chains, which tree-sitter cannot attach; that is an engine tolerance question, not harvest.
- Implications: skip comment and string regions before searching; anchor on a line that starts a declarator; add line-range and hunk modes so Python and JS/TS are covered; report parse loss on the dashboard so the engine spec can measure its tolerance work.
### Candidate corpora
- **Context**: User asked for vibe-coded and Node/TypeScript/Python web corpora.
- **Sources Consulted**: Real-Vuln-Benchmark repo and paper; SecBench.js repo and ICSE 2023 paper; AIDev dataset card and paper; BaxBench site, repo, and paper; SecurityEval, CodeSecEval, CWEval; CyberSecEval benchmarks README; VibeApps paper; TS-VulBench page; SecureVibeBench repo; GitHub commit search via `gh api search/commits`.
- **Findings**:
- Context: User asked for vibe-coded and Node/TypeScript/Python web corpora.
- Sources Consulted: Real-Vuln-Benchmark repo and paper; SecBench.js repo and ICSE 2023 paper; AIDev dataset card and paper; BaxBench site, repo, and paper; SecurityEval, CodeSecEval, CWEval; CyberSecEval benchmarks README; VibeApps paper; TS-VulBench page; SecureVibeBench repo; GitHub commit search via `gh api search/commits`.
- Findings:
| Corpus | Languages | Labels | Fixed twin | License | Verdict |
|---|---|---|---|---|---|
| Real-Vuln-Benchmark | Python web | file, line range, function, CWEs, authorship, FP traps | traps | Apache-2.0 | `vibe-py` |
@@ -46,14 +46,14 @@
| VibeApps/VibeVulns | TS, JS, React | not released | no | n/a | method only |
| TS-VulBench | TS synthetic | paywalled | no | unknown | skip |
| SecureVibeBench | C/C++ | agent tasks | n/a | MIT | skip for web |
- **Implications**: three slices now; BaxBench after deep mode is stable; CyberSecEval never as recall ground truth.
- Implications: three slices now; BaxBench after deep mode is stable; CyberSecEval never as recall ground truth.
### Classification and branching
- **Context**: User expects tuning to split into branches per corpus class.
- **Sources Consulted**: `improve/evolve.py` accept gate; `ruleset/store.py` ledger; `calibration.py` scope; `preprocess.detect_tags`; `build_pair_signals`.
- **Findings**: the accept gate already ANDs four clauses and reverts byte-for-byte. Pair signals carry pair name but no profile. Ledger is one file per target. Preprocess tags are deterministic and per file.
- **Implications**: classify on two axes (provenance for gating, mechanism for reporting). Branch the acceptance decision, not the ledger format: reject a change if any profile's holdout regresses. Profile-specific facts and thresholds are deferred to the engine specs where those data live.
- Context: User expects tuning to split into branches per corpus class.
- Sources Consulted: `improve/evolve.py` accept gate; `ruleset/store.py` ledger; `calibration.py` scope; `preprocess.detect_tags`; `build_pair_signals`.
- Findings: the accept gate already ANDs four clauses and reverts byte-for-byte. Pair signals carry pair name but no profile. Ledger is one file per target. Preprocess tags are deterministic and per file.
- Implications: classify on two axes (provenance for gating, mechanism for reporting). Branch the acceptance decision, not the ledger format: reject a change if any profile's holdout regresses. Profile-specific facts and thresholds are deferred to the engine specs where those data live.
## Architecture Pattern Evaluation
@@ -68,48 +68,48 @@
### Decision: Count coverage-with-flow as detection and as leak
- **Context**: Sink aliases will always land as coverage.
- **Alternatives Considered**: promote coverage to `promote` in the engine; count coverage only on the vulnerable side.
- **Selected Approach**: scorer treats `coverage` with a non-empty source the same as `promote` on both sides.
- **Rationale**: `propose-adjudicate-prove` already states coverage enters the same state machine. Asymmetry would inflate recall.
- **Trade-offs**: some pairs move from MISS to LEAK; that is the honest reading.
- Context: Sink aliases will always land as coverage.
- Alternatives Considered: promote coverage to `promote` in the engine; count coverage only on the vulnerable side.
- Selected Approach: scorer treats `coverage` with a non-empty source the same as `promote` on both sides.
- Rationale: `propose-adjudicate-prove` already states coverage enters the same state machine. Asymmetry would inflate recall.
- Trade-offs: some pairs move from MISS to LEAK; that is the honest reading.
### Decision: Function-scoped matching with weak-label flagging
- **Context**: CWE-only matching over-counts.
- **Selected Approach**: expected rows carry `function`; match requires the record inside that function's lines; rows with no function, sink, or rule id are `weak_label` and never match by CWE alone.
- **Trade-offs**: vfc recall may drop on first run; the drop is real.
- Context: CWE-only matching over-counts.
- Selected Approach: expected rows carry `function`; match requires the record inside that function's lines; rows with no function, sink, or rule id are `weak_label` and never match by CWE alone.
- Trade-offs: vfc recall may drop on first run; the drop is real.
### Decision: Provenance is deterministic and closed
- **Context**: No LLM in classification.
- **Selected Approach**: signals from `.claude/`, `CLAUDE.md`, `AGENTS.md`, `.cursor/`, `.github/copilot-instructions.md`, Lovable meta tag, "Generated with Claude Code", and co-author trailers for Claude, Codex, Copilot, Cursor, Devin, Jules when a git log is readable. Threshold for `agent` on a git tree: at least 85% of commits or lines in the last N commits carry a signal, following the VibeApps method; otherwise `mixed` when any signal exists; `human` when none; `synthetic` only from catalog declaration.
- **Trade-offs**: repos that scrub trailers read as human. Acceptable; the label is a profile, not a verdict.
- Context: No LLM in classification.
- Selected Approach: signals from `.claude/`, `CLAUDE.md`, `AGENTS.md`, `.cursor/`, `.github/copilot-instructions.md`, Lovable meta tag, "Generated with Claude Code", and co-author trailers for Claude, Codex, Copilot, Cursor, Devin, Jules when a git log is readable. Threshold for `agent` on a git tree: at least 85% of commits or lines in the last N commits carry a signal, following the VibeApps method; otherwise `mixed` when any signal exists; `human` when none; `synthetic` only from catalog declaration.
- Trade-offs: repos that scrub trailers read as human. Acceptable; the label is a profile, not a verdict.
### Decision: Branch the accept decision, not the data
- **Context**: Overfitting risk with per-slice tuning.
- **Selected Approach**: `run_improvement` evaluates holdout pairs per profile; any profile regression beyond tolerance rejects. Profiles capped at four. Minimum holdout size reported.
- **Follow-up**: profile-keyed facts land with the engine specs, gated by this clause.
- Context: Overfitting risk with per-slice tuning.
- Selected Approach: `run_improvement` evaluates holdout pairs per profile; any profile regression beyond tolerance rejects. Profiles capped at four. Minimum holdout size reported.
- Follow-up: profile-keyed facts land with the engine specs, gated by this clause.
## Risks & Mitigations
- Recall numbers drop after function-scoped matching and weak-label flags — expected; record before and after in the roadmap.
- SecBench.js has no license — never vendor its files; harvest from upstream package repos and record each package license.
- Agent-authored commits may be false labels (a "fix" that was not a vuln) — human review required; keep `agent-vfc` small.
- Real-Vuln-Benchmark apps are educational and CTF style — say so in the slice README; do not treat `vibe-py` as production.
- Held-out halves are small on day one — report minimum-size warnings; do not gate on a profile under the minimum.
- Recall numbers drop after function-scoped matching and weak-label flags; expected; record before and after in the roadmap.
- SecBench.js has no license; never vendor its files; harvest from upstream package repos and record each package license.
- Agent-authored commits may be false labels (a "fix" that was not a vuln); human review required; keep `agent-vfc` small.
- Real-Vuln-Benchmark apps are educational and CTF style; say so in the slice README; do not treat `vibe-py` as production.
- Held-out halves are small on day one; report minimum-size warnings; do not gate on a profile under the minimum.
## References
- [Real-Vuln-Benchmark](https://github.com/kolega-ai/Real-Vuln-Benchmark) — Apache-2.0, manifest with pinned SHAs, ground-truth schema
- [RealVuln paper](https://arxiv.org/abs/2604.13764) — 26 human plus 40 LLM-generated Python repos, FP traps
- [SecBench.js](https://github.com/cristianstaicu/SecBench.js) — 600 npm vulns, `fixCommit`, `sinkLocation`
- [Real-Vuln-Benchmark](https://github.com/kolega-ai/Real-Vuln-Benchmark): Apache-2.0, manifest with pinned SHAs, ground-truth schema
- [RealVuln paper](https://arxiv.org/abs/2604.13764): 26 human plus 40 LLM-generated Python repos, FP traps
- [SecBench.js](https://github.com/cristianstaicu/SecBench.js): 600 npm vulns, `fixCommit`, `sinkLocation`
- [SecBench.js paper](https://software-lab.org/publications/icse2023_SecBenchJS.pdf)
- [AIDev dataset](https://huggingface.co/datasets/hao-li/AIDev) — agent-authored PRs, CC-BY-4.0 index
- [BaxBench](https://github.com/logic-star-ai/baxbench) — MIT, scenarios and exploits, generation by CLI
- [AIDev dataset](https://huggingface.co/datasets/hao-li/AIDev): agent-authored PRs, CC-BY-4.0 index
- [BaxBench](https://github.com/logic-star-ai/baxbench): MIT, scenarios and exploits, generation by CLI
- [SecurityEval](https://github.com/s2e-lab/SecurityEval), [CodeSecEval](https://arxiv.org/pdf/2407.02395), [CWEval](https://github.com/Co1lin/CWEval)
- [CyberSecEval](https://github.com/meta-llama/PurpleLlama/tree/main/CybersecurityBenchmarks) — labels defined by ICD rules
- [VibeApps paper](https://arxiv.org/html/2606.23130v1) — identification method
- `.kiro/specs/real-world-vfc-slice/` — superseded corpus spec
- Memory note `overlay-gaps-diagnosis-2026-09` — measured state and engine gaps
- [CyberSecEval](https://github.com/meta-llama/PurpleLlama/tree/main/CybersecurityBenchmarks): labels defined by ICD rules
- [VibeApps paper](https://arxiv.org/html/2606.23130v1): identification method
- `.kiro/specs/real-world-vfc-slice/`: superseded corpus spec
- Memory note `overlay-gaps-diagnosis-2026-09`: measured state and engine gaps
+21 -21
View File
@@ -206,25 +206,25 @@ URL; the CLI mints them and polls the store. The URLs travel in the start body,
"inputs": {"CANDIDATES": "<presigned GET runs/<run>/<producer>/<artifact>>"}}}
```
- **Why the start body.** A presigned URL is a bearer capability, so the credentials-only-in-the-start-request
- Why the start body. A presigned URL is a bearer capability, so the credentials-only-in-the-start-request
rule applies; env would bake it into the ActorTemplate and hand it to the golden boot; and the body has no
32 KB ceiling (Substrate rejects env values over 32768 characters, `reconciler.py:141`). `StartGate.offer`
(`runner.py:250-275`) validates `delivery` the way it validates `credentials`: `https` URLs only, hostnames,
no echo. Expiry = `OUSAST_TASK_TIMEOUT` (7200 s) + 600 s; a re-start after a re-resume re-mints.
- **Why presigned rather than a scoped delivery credential.** A per-task S3 key needs a per-task policy on RustFS,
- Why presigned rather than a scoped delivery credential. A per-task S3 key needs a per-task policy on RustFS,
gives the sandbox `ListBucket` and reads beyond its inputs, and would travel in the same body. Presigned URLs
name exactly the objects a task may write and read, which is what the receiver enforces today
(`Receiver.input_file`, `egress.py:230-247`).
- **Why not an in-cluster receiver Service.** It is a server to run, holds state in a process, and is reachable
- Why not an in-cluster receiver Service. It is a server to run, holds state in a process, and is reachable
only from a pod; that is the gap remote mode has. **Rejected.**
- **Objects.** `runs/<run>/<task>/output.tar` (the runner's PUT); after the CLI extracts it, the declared
- Objects. `runs/<run>/<task>/output.tar` (the runner's PUT); after the CLI extracts it, the declared
`outputs[]` are re-put as `runs/<run>/<task>/<artifact>` so a consumer's presigned GET names one artifact, and
`runs/<run>/<task>/done.json` carries the summary's status. The 30-day `runs/` expiry already applies; memory
rows are ingested from the local extraction by `remember` as today (f: annotations and `memory_key` seeding
unchanged, `generate.py:_chain_inputs`, `memory.seed`).
- **Completion.** `_await`'s `delivered()` becomes `Delivery.delivered(task)`: a HEAD on `output.tar`, polled at
- Completion. `_await`'s `delivered()` becomes `Delivery.delivered(task)`: a HEAD on `output.tar`, polled at
`OUSAST_POLL_SECONDS` (3 s). The runner's completion marker logic (`runner.py`, `_run_once`) is unchanged.
- **Inputs no task produces** (`candidates.json`, `functions.json`, `case.json`) may also be store objects
- Inputs no task produces (`candidates.json`, `functions.json`, `case.json`) may also be store objects
declared as inputs, which lifts the ~20 KB inline-files limit (`ops/ax/README.md:117`); Workspace `files`
stay for the smoke Task and small cases.
@@ -251,7 +251,7 @@ shim and `/workspace`), so one *contract* in two images: `ousast-runner` (slim,
2.8 GB and every golden snapshot carry it; the engine runs for a minority of cases); Docker-in-pod (no socket on
containerd; `Dockerfile:1-7` already argues against it).
- **Task.** Template `plane/tasks/engine.yaml`: `command: ["engine"]`, env `OUSAST_WORKSPACE_DIR`,
- Task. Template `plane/tasks/engine.yaml`: `command: ["engine"]`, env `OUSAST_WORKSPACE_DIR`,
`OUSAST_FIXED_DIR`, `OUSAST_INPUT_CASE`, pins and `OUSAST_FAMILIES`, bound to the same two Workspaces the
`alerts` task binds (`generate.py:342-353`). The module lifts `benchmarks/push/finding_dump.py`'s scan into
`openultrasast.cpg.dump` (the image ships `src/`, not `benchmarks/`) and reuses `engine_alerts.engine_rows`,
@@ -259,7 +259,7 @@ containerd; `Dockerfile:1-7` already argues against it).
`alerts.jsonl` rows (`rule_id: engine:<family>`, `source: "engine"`) and `summary.json` (coverage with the
engine's languages, files and bytes read, questions, seconds, degradations per pin) as `alerts-engine`, with
`host: false`. The instrument rule stays: no `read N files` line or `questions == 0` fails the task loudly.
- **Sizing from the record** (`benchmarks/measurements/2026-09-30-php-engine-alerts/record.json`): container
- Sizing from the record (`benchmarks/measurements/2026-09-30-php-engine-alerts/record.json`): container
peak 2.129 GiB under a 3 GiB limit, 789 s for both pins of froxlor, deadline 1800 s per pin. Task resources:
requests 3 Gi / 1 CPU, limits 4 Gi / 2 CPU; `engine-pool` workers 4 Gi / 2 CPU, replicas 1 on kind (the 7 GB
host holds it only with `ousast-pool` at one worker), production replicas in `k3s.toml`. Task timeout: two pins
@@ -314,23 +314,23 @@ runbook: a Run with `--workers N` beyond the pool's replicas queues in ax as Sus
`plane/scan.py`, reusing `generate.py`'s render path with a new case source:
1. **Workspace generation** (`generate.scan_cases(url, base, head, cache)`): a shallow clone into the case cache
1. Workspace generation (`generate.scan_cases(url, base, head, cache)`): a shallow clone into the case cache
(`runner.find_cached_checkout` layout), two pinned Workspaces `<repo_dir>-<head7>` and `<repo_dir>-<base7>`
(`workspaces.py` shape, pins in `openultrasast.io/git-commits`), repository key by `memory.repo_key`.
2. **Candidates from changed functions**: `generate.fix_ranges(repo, base, head, side="new")` gives the head-side
2. Candidates from changed functions: `generate.fix_ranges(repo, base, head, side="new")` gives the head-side
hunks (the `push/` delta primitive `_diff_records` is the same `git diff -U0`), `repo_facts.enclosing` names
the function per hunk line, filtered to `repo_facts.product_files`. Quick rules on the changed files
(`tasks.alerts.scan`) add alerted functions; `roles` runs once per repository when memory has no `roles` rows
for it; `repo-facts` supplies callers. `candidates.json` and `functions.json` are store inputs (section 3).
3. **Run composition**: per changed-function set `facts -> va, vb -> agree -> vc -> final -> features ->
3. Run composition: per changed-function set `facts -> va, vb -> agree -> vc -> final -> features ->
remember`, plus `engine` when a changed file's language is engine-covered and not quick-covered, or PHP
(`ENGINE_ALONGSIDE_QUICK`). Budgets from `generate.verify_budget` (0.022 USD per candidate, headroom 1.5);
`--ceiling` caps the sum. `--all` (whole repository, secondary) takes every product function.
4. **Results** land in memory under `repos/<host>__<owner>__<name>/<head>.jsonl` through `remember` and
4. Results land in memory under `repos/<host>__<owner>__<name>/<head>.jsonl` through `remember` and
`ousast plane remember` (rows already carry `repo` and `pin`, `memory.py:77`); `runs/<run>/` holds the raw
artifacts for 30 days. The report prints the attribution table and the plane's agreement (`final`); the
decision engine's verdict only when a model is adopted (learned-decision-engine Req 6 unchanged).
5. **Latency split (e).** `ousast pre-push` stays the blocking hook: quick rules and the delta engine inside
5. Latency split (e). `ousast pre-push` stays the blocking hook: quick rules and the delta engine inside
`OUSAST_PUSH_DEADLINE` (30 s, `ops/README.md:322`). `ousast plane scan --exec remote --no-wait` submits and
prints the Run name in the hook's report; `--wait SECONDS` blocks for CI. `ousast plane status <run>` reports
later. The wrapper `ops/pre-push` gains an opt-in `OUSAST_PLANE_SCAN=1`; it never blocks on the plane.
@@ -378,7 +378,7 @@ template names `imagePullSecret` from the profile.
- `tests/test_plane_engine_task.py`: fixture record -> rows and summary equal `engine_alerts.produce`'s shape;
the instrument failures.
- `tests/test_plane_scan.py`: candidates from a two-commit fixture repository; Run composition; budgets.
- **Only live on kind:** placement on `engine-pool`; Substrate accepting the SA token for `egress-policy`;
- Only live on kind: placement on `engine-pool`; Substrate accepting the SA token for `egress-policy`;
the gateway passing TLS to the store host; GHCR pulls; the HPA (k3s only). These are the kind proofs above.
## Data Models
@@ -411,16 +411,16 @@ from `StartGate.offer`. The engine task fails loudly on an unread input, as `rea
## Open Questions
1. **Substrate authorization for the CI token.** `kubectl-ate` sends a ServiceAccount token; which audience and
1. Substrate authorization for the CI token. `kubectl-ate` sends a ServiceAccount token; which audience and
which Substrate-side grant let `ousast-ci` create `egress-policy` in `default` is not in the checkouts read
here. Proven or refuted in increment 3.
2. **Engine placement.** Whether Substrate places a 3 Gi actor only on workers that fit it (section 4). The
2. Engine placement. Whether Substrate places a 3 Gi actor only on workers that fit it (section 4). The
fallback is named.
3. **gVisor tarball reachability on k3s** (`gs://gvisor/...` in the SandboxConfig): mirror or egress needed.
4. **Store endpoint hostname** for any future in-cluster RustFS; today's external endpoint is fine.
5. **Two images under one contract** versus the maintainer's "one runner image for all tasks": the design
3. gVisor tarball reachability on k3s (`gs://gvisor/...` in the SandboxConfig): mirror or egress needed.
4. Store endpoint hostname for any future in-cluster RustFS; today's external endpoint is fine.
5. Two images under one contract versus the maintainer's "one runner image for all tasks": the design
reads it as one contract; confirm.
6. **GHCR visibility** (public packages need no pull secret) and whether Substrate's and ax's `ko` images go to
6. GHCR visibility (public packages need no pull secret) and whether Substrate's and ax's `ko` images go to
the same registry namespace.
7. **Reconciler size**: remote mode and `Delivery` must stay under the 500-line test; if they do not, the test's
7. Reconciler size: remote mode and `Delivery` must stay under the 500-line test; if they do not, the test's
bound moves with a reason, not silently.
@@ -9,24 +9,24 @@ production entry point (a delta Run for a push) is added. Context: `brief.md`, `
## Boundary Context
- **In scope**: two profiles of one code base, `kind` (local, kept) and `k3s` (production); a remote execution mode of the CLI; artifacts delivered to the store; configurable
- In scope: two profiles of one code base, `kind` (local, kept) and `k3s` (production); a remote execution mode of the CLI; artifacts delivered to the store; configurable
cluster context, registry and image pins; our own snapshot and memory buckets; the Joern engine as a plane task
on a larger worker pool; worker pool sizing and autoscaling; a delta Run command and Workspaces from URL +
commit; a deployment runbook with a proof on kind and a checklist for the target cluster.
- **Out of scope**: provisioning or operating the target cluster; modifying Agent Substrate or ax; any change to
- Out of scope: provisioning or operating the target cluster; modifying Agent Substrate or ax; any change to
detection, scoring, populations or the decision engine; replacing the `ax` CLI with a bespoke client beyond what
remote submission needs.
- **Adjacent expectations**: `ai-service-plane` requirements (one Model + budget per task, attribution,
- Adjacent expectations: `ai-service-plane` requirements (one Model + budget per task, attribution,
credentials only in the start request, egress policies, verify-at-startup memory store) keep holding;
`harnessx-removal` Req 6 (memory on the S3 store) keeps holding; nothing is deleted before its replacement
passes its test (the `harnessx-removal` sequencing rule).
## Target Environment (maintainer, 2026-10-02)
- **Cluster: k3s.** Consequences: containerd (no Docker socket; the engine's host-Docker path does not exist there),
- Cluster: k3s. Consequences: containerd (no Docker socket; the engine's host-Docker path does not exist there),
gVisor needs `runsc` and a `RuntimeClass` installed on the nodes Substrate's workers run on, k3s ships Traefik
and local-path storage, and single-node k3s is close enough to kind that the kind proof transfers.
- **Images and source: GitHub.** Images are published to GitHub Container Registry (`ghcr.io/norandom/...`),
- Images and source: GitHub. Images are published to GitHub Container Registry (`ghcr.io/norandom/...`),
built by GitHub Actions from tagged commits, digest-pinned in the manifests; private packages need an
`imagePullSecret` on the worker ServiceAccounts and the CI principal. Substrate's and ax's own images are built with
`ko` from their checkouts and pushed to the same registry.
+1 -1
View File
@@ -69,7 +69,7 @@ family semantics remain with their existing specs rather than being reimplemente
- No promise of all-language coverage; capabilities are admitted per language/framework/family.
- Initial work must establish feasibility and noise before investing in an LLM ranker.
## Cross-language amendment — 2026-09-12
## Cross-language amendment, 2026-09-12
Keep PMPro as the primary worked optimization example, and require a JavaScript/Express/Node API
transfer experiment before claiming the core improvement generalizes. Abstract evidence, ranking,
+4 -4
View File
@@ -24,7 +24,7 @@ an irrelevant old defect, or an unactionable witness can still produce noise.
- Do not build a second ranker/scope heuristic, an incremental CPG engine, a daemon, an automatic fixer,
or a mandatory model service. Do not scan intermediate pushed history in this feature.
## What this hook is, and is not — 2026-09-21
## What this hook is, and is not, 2026-09-21
A general-purpose pre-commit hook, not an application-specific one. It may be measured on WordPress
plugins and Node applications, and it may not become a tool that only works on them. The practical
@@ -63,8 +63,8 @@ The fourth and fifth cost the declared v0.1 target its entire admissibility: a W
PHP plus admin JavaScript, so the product could detect defects on its own target and never admit
one. Neither passing subject could reveal it, because both hold one language.
**In review:** every new degradation reason must state the scope it reaches — one question, one file,
one partition, or the whole read — and a reason that demotes an outcome must justify reaching that
**In review:** every new degradation reason must state the scope it reaches (one question, one file,
one partition, or the whole read), and a reason that demotes an outcome must justify reaching that
far. A gap that costs coverage without threatening correctness is a declared exclusion: it stays
published and it does not demote.
@@ -462,7 +462,7 @@ decision in the terminal; do not fabricate a successful cache entry or hang whil
separate from a future C/C++ positive control. See `evaluation.md` for selection and labeling gates.
Measure the accepted experimental acceptance profile in requirements on declared hardware and fixed workloads.
Warm means prior reusable work from the preceding state, not only repeating the identical tip.
Warm means prior reusable work from the preceding state. Repeating the identical tip alone is insufficient.
Report identical-tip hits separately from a one-function change and a dependency/configuration
change. Include first installs, cold graphs, growing repository size, multi-ref pushes and incomplete
outcomes. Freeze the applicable population before ranker selection; deferred questions count against
@@ -68,7 +68,7 @@ real capability; task 8.3 still owns actual eligibility. Task 4.2 brings progres
The agent service rejected the task 4.3 dispatch with `agent thread limit reached`.
The [kiro-impl skill](../../../.agents/skills/kiro-impl/SKILL.md) explicitly provides:
“If multi-agent is not available, fall back to manual mode execution for all tasks.”
"If multi-agent is not available, fall back to manual mode execution for all tasks."
Implementation, review and completion checks therefore ran in the main context.
This is a manual review record, not an independent-review claim.
@@ -1,6 +1,6 @@
# Implementation verification: group 5
Group 5 was authorized by the maintainer's “approved” after group 4 completion.
Group 5 was authorized by the maintainer's "approved" after group 4 completion.
Tasks 5.1–5.2 are implemented and verified (16/27 total); the measured feasibility decision is NO-GO.
Real capability eligibility and rollout readiness remain unproven.
@@ -8,7 +8,7 @@ Real capability eligibility and rollout readiness remain unproven.
Fresh implementer dispatch for 5.1 returned `agent thread limit reached`.
The [Kiro implementation skill](../../../.agents/skills/kiro-impl/SKILL.md) explicitly
states: “If multi-agent is not available, fall back to manual mode execution for all tasks.”
states: "If multi-agent is not available, fall back to manual mode execution for all tasks."
Its manual review rule permits review in the main context when a fresh reviewer is unavailable.
This run therefore uses manual implementation and adversarial review, not independent review.
@@ -300,15 +300,15 @@ and its [bundle](../../../benchmarks/measurements/2026-09-16-nodegoat-m1b-explai
Three evaluation findings were rendered on the vulnerable revision, one per eval site. For
line 32:
- **Location** app/routes/contributions.js:32, family injection, novelty new with
- Location: app/routes/contributions.js:32, family injection, novelty new with
`operation_absent_from_comparable_base`.
- **Witness** `eval(req.body.preTax) -> const preTax = eval(req.body.preTax)`.
- **Change attribution** `base:app/routes/contributions.js:32-34`.
- **Provenance** `file_scope`: the question named no function, the engine reported `<lambda>2`.
- **Consequence** "A request field reaches const preTax = eval(req.body.preTax), where it is
- Witness: `eval(req.body.preTax) -> const preTax = eval(req.body.preTax)`.
- Change attribution: `base:app/routes/contributions.js:32-34`.
- Provenance: `file_scope`: the question named no function, the engine reported `<lambda>2`.
- Consequence: "A request field reaches const preTax = eval(req.body.preTax), where it is
evaluated as JavaScript, so an attacker controlling that field can execute arbitrary code in
the server process."
- **Repair** "Replace the evaluation at const preTax = eval(req.body.preTax) with a numeric
- Repair: "Replace the evaluation at const preTax = eval(req.body.preTax) with a numeric
conversion such as Number(...) or parseInt(...), then validate the result before use."
Both sentences come from the declaration's templates through the grounded-template path, not from
@@ -64,9 +64,9 @@ code the precision estimate will move, against a bar of 95% with zero false aler
benign sides. Expect the first M4 evaluation to fail and budget for a second. That is the strongest
argument for decision 2.
## M1 — One real security change works end to end
## M1: One real security change works end to end
Status: **NEXT — not verified**. Dependencies: existing immutable replay and scan machinery.
Status: **NEXT, not verified**. Dependencies: existing immutable replay and scan machinery.
M1 is split into M1a (diagnose with vetoes recorded, not applied) and M1b (repair only
the vetoes M1a showed to matter). M1 is verified only when M1b passes.
@@ -80,7 +80,7 @@ a global completion veto, line ambiguity inherited by every question, and query
differing from witness identity. Requiring all four repaired before the first visible
result is a full production pass, not one detection.
### M1a — Show the detection with every veto recorded
### M1a: Show the detection with every veto recorded
Status: **MET 2026-09-16**, inside its one-week box. Evidence:
[M1a record](m1a-recorded-vetoes.md) and
@@ -112,9 +112,9 @@ source and census receipts. **Kill criterion:** if the fixed twin is not quiet u
recorded-veto run, M1 has a detector problem, and M1b does not start until that is
diagnosed and recorded.
### M1b — Repair only the vetoes M1a showed to matter
### M1b: Repair only the vetoes M1a showed to matter
Status: **NEXT — not verified**. M1a supplied the veto list: per-family change context,
Status: **NEXT, not verified**. M1a supplied the veto list: per-family change context,
question-owned vendor relevance, operation correspondence for an operation on a changed line,
file-wide correspondence ambiguity scope, and query-to-operation provenance. The owner
diagnosis expected the first and the last two; operation correspondence is the addition M1a
@@ -144,7 +144,7 @@ Owners: contributor-scan for query/operation/context evidence; pre-push-safety-n
comparison, admission, the recorded-veto flag and reporting. Requirements: 1.1–1.4,
2.1–2.4, 3.1–3.5, 6.3, 7.1–7.3.
## M2 — The same useful analysis fits the push budget
## M2: The same useful analysis fits the push budget
Status: **MEASURED AND UNMET, 2026-09-17.** Two attacks on engine startup were measured and
neither reaches the gate. A transaction-owned session is implemented and proven equivalent for
@@ -179,7 +179,7 @@ completion at least 95%, under the predeclared profile. A fast census alone does
Owner: contributor-scan runtime, with pre-push-safety-net deadline/cache integration.
Requirements: 4.1–4.4, 6.2–6.5, 7.3.
## M3 — The shared design transfers across languages
## M3: The shared design transfers across languages
Status: **PARTIAL, 2026-09-17. Not met.** Python is established and PHP is blocked by runtime.
@@ -218,7 +218,7 @@ remain a separate unsupported abstraction; adding a new engine is not a prerequi
Owners: contributor-scan shared evidence and framework adapters; pre-push-safety-net evaluation.
Requirements: 6.2–6.4, 8.1–8.4.
## M4 — Independent examples establish useful detection and low noise
## M4: Independent examples establish useful detection and low noise
Status: **PENDING M3**. Population preparation may proceed earlier, but evaluation follows
the frozen candidate. Reserve untouched cases before examining their scan results.
@@ -241,7 +241,7 @@ do not require every possible language or family to become eligible for a narrow
Owner: pre-push-safety-net task 8.3, consuming verified contributor-scan semantics.
Requirements: 6.1–6.5, 8.1–8.4 and the accepted experimental acceptance profile.
## M5 — Enable a narrow advisory rollout
## M5: Enable a narrow advisory rollout
Status: **PENDING M4**.
@@ -196,6 +196,6 @@ failures without lowering the bar after inspecting holdout results.
- >= 95% observed actionable-alert precision with counts and confidence interval; zero known false
alerts on the fixed/benign regression suite. Small samples do not establish broad precision.
- >= 90% recall on the predeclared supported security-change suite; unsupported classes remain in
coverage reporting, not reclassified after a miss. Publish per-capability results, not only an aggregate.
coverage reporting, not reclassified after a miss. Publish per-capability results; an aggregate alone is insufficient.
- No capability is admitted with zero positive controls, zero reviewed emitted alerts, or no untouched
evaluation repository. Failed profiles remain experimental and silent in the normal hook.
+1 -1
View File
@@ -89,7 +89,7 @@ feasibility gate. [Official CPG slicing documentation](https://docs.joern.io/cpg
2. **Separate claim from interruption.** Existing rungs describe analysis evidence. Hook admission
additionally requires evaluated capability, change attribution and actionability. This meets the
user's low-noise requirement without pretending an LLM confidence score proves a bug.
3. **Compare evidence, not just finding lists.** A missing base finding can mean truncated analysis.
3. **Compare evidence as well as finding lists.** A missing base finding can mean truncated analysis.
Targeted comparable base checks are required for novelty, including removed-guard cases.
4. **Adopt Git and the existing engine.** Build only the push transaction, cache and admission adapters.
Do not add an IR, database or service merely to describe future scale.
+2 -2
View File
@@ -1,4 +1,4 @@
# Pre-push safety net status — v1.2.0-alpha.1
# Pre-push safety net status, v1.2.0-alpha.1
Updated 2026-09-16. Phase: **implementation**. Requirements, design and tasks are
approved. **28/29 executable tasks complete; task 8.3 remains blocked.**
@@ -141,7 +141,7 @@ Four of seven declared families establish findings on the two development subjec
injection, access_control, config_secrets and untrusted_destination. Output encoding, path,
prototype and deserialization have established nothing anywhere. Every question completed on
both subjects, 299 of 299 on NodeGoat and 35 of 35 on VAmPI, so the silences are answers
rather than failures — but three of them are diagnosed gaps rather than clean code:
rather than failures, but three of them are diagnosed gaps rather than clean code:
NodeGoat's documented SSRF is invisible because `needle.get` is not in the sink vocabulary,
its XSS is invisible because output encoding declares only DOM sinks, and its two
access-control defects raise no obligation because the modelled data operations live in DAO
@@ -1,9 +1,9 @@
# Packaged acceptance validation — group 8
# Packaged acceptance validation, group 8
## Validation Report
- DECISION: NO-GO
- FEATURE_COMPLETE: false
- BLOCKED_TASKS: 8.3 — independent reviewed populations and verified VAmPI transitive evidence are missing; useful completion and runtime gates fail.
- BLOCKED_TASKS: 8.3, independent reviewed populations and verified VAmPI transitive evidence are missing; useful completion and runtime gates fail.
- OWNERSHIP: UPSTREAM for engine/census/context prerequisites; LOCAL for evaluation population and capability qualification.
- UPSTREAM_SPEC: contributor-scan; see `pre-push-runtime-followup.md` there.
+11 -11
View File
@@ -7,11 +7,11 @@ Status: brief / not yet specced. Proposed 2026-10-07 by the maintainer. Builds o
The search arm (directed, Cairn-inspired) works. The **proof-authoring** last mile is the drag:
1. **Safety-classifier refusals.** Authoring exploit demos repeatedly tripped a safety classifier
1. Safety-classifier refusals. Authoring exploit demos repeatedly tripped a safety classifier
this session when exploit payloads or step-by-step attack prose passed through the model or a
tool prompt. The mitigation so far (keep payloads on disk, treat demos as opaque) is a workaround,
not a fix, and it limits how directly the proof model can reason about the exploit.
2. **Nondeterministic wiring.** The agent authors a demo but intermittently mis-wires the app to
2. Nondeterministic wiring. The agent authors a demo but intermittently mis-wires the app to
the oracle fixture (served directory for path, database URL for SQL), so a demonstrable pair
comes back inconclusive until a retry. Each miss has cost a paid run.
@@ -21,20 +21,20 @@ proof-authoring policy with DSPy from **traces logged in OpenObserve**.
## Shape
- **Search model:** DeepSeek (current). Good at the directed fact/intent search.
- **Proof model:** GLM 5.3 (Anthropic-recommended) and/or an abliterated model via
- Search model: DeepSeek (current). Good at the directed fact/intent search.
- Proof model: GLM 5.3 (Anthropic-recommended) and/or an abliterated model via
`ABLITERATION_BASE_URL` (`abliterated large v2`). The abliterated model authors exploit **demos**
without the refusals that block a safety-trained model on this defensive task.
- **Proof driver:** OpenCode MCP (github.com/AlaeddineMessadi/opencode-mcp) as the agentic
- Proof driver: OpenCode MCP (github.com/AlaeddineMessadi/opencode-mcp) as the agentic
demo-author, running inside the gVisor verification task so the exploit-authoring is sandboxed.
- **Learning:** DSPy compiles/optimizes the proof-authoring program (demo schema, wiring, steps)
- Learning: DSPy compiles/optimizes the proof-authoring program (demo schema, wiring, steps)
from past traces, instead of the per-family hand-guidance we keep adding (which is now hitting
the prompt character budget — a signal the hand-rule approach is near its limit). Matches the
the prompt character budget, a signal the hand-rule approach is near its limit). Matches the
maintainer's "learned decisions, not hand-tuning" direction.
- **Tracing:** proof runs (reason/explore/author/verify, timings, outcomes, error classes) log to
- Tracing: proof runs (reason/explore/author/verify, timings, outcomes, error classes) log to
**OpenObserve AI** at `OPENOBSERVE_URL` (org `OPENOBSERVE_ORG`, stream `OPENOBSERVE_TRACES_STREAM`).
DSPy consumes these traces as its training signal.
- **Credentials:** stored in `.env` like the existing keys, referenced by name only:
- Credentials: stored in `.env` like the existing keys, referenced by name only:
`ABLITERATION_API_KEY`, `ABLITERATION_BASE_URL`, `ABLITERATION_MODEL`, `OPENOBSERVE_URL`,
`OPENOBSERVE_ORG`, `OPENOBSERVE_TRACES_STREAM`, `OPENOBSERVE_TOKEN`. No secret value ever enters a
tracked file, a prompt, a log stream, or a commit.
@@ -83,8 +83,8 @@ in a **gVisor-isolated** corpus, to demonstrate a regression before push. Guardr
## OpenCode MCP as the endpoint for both PoC and fix
Maintainer note (2026-10-07): OpenCode MCP is a good server endpoint for the PoC **and** the fix,
not only proof authoring. The same agentic endpoint can:
Maintainer note (2026-10-07): OpenCode MCP is a good server endpoint for both the PoC **and** the fix.
The same agentic endpoint can:
1. Author the PoC (the demonstrated proof) for a caught finding.
2. Write a candidate fix against the finding's repair direction and the failing PoC.
@@ -2,7 +2,7 @@
## Problem
OpenUltraSAST already has three scan stages (`quick` inventory, `standard` map, `deep` regression), but **only the first stage actually decides**. Pattern rules propose sinks; the map ranks files; the sandbox can run recipes. Nothing in the middle answers “does this sink receive attacker-controlled data, or is it a Juliet-style good twin / dead assignment / hardcoded command?”
OpenUltraSAST already has three scan stages (`quick` inventory, `standard` map, `deep` regression), but **only the first stage actually decides**. Pattern rules propose sinks; the map ranks files; the sandbox can run recipes. Nothing in the middle answers "does this sink receive attacker-controlled data, or is it a Juliet-style good twin / dead assignment / hardcoded command?"
Calibration corpora (cheat-sheet fixtures, OWASP Benchmark true/false files, Juliet `bad`/`good`, CVE pairs) are **nice**: one CWE per file, named sinks, labeled counterparts. Real working trees are not: mixed languages, generated code, unparseable files, framework wrappers, comments, one-hop assignments (`sql = f"…{x}"; cur.execute(sql)`), and no ground-truth labels.
@@ -12,17 +12,17 @@ Measured honesty on the labeled SAST slice is about **+9% Youden**. Widening reg
Keep three roles, never collapse them:
1. **Propose** — pattern rules + optional SARIF. Stage 1. Does not decide worth-fixing.
2. **Adjudicate** — tree-sitter IR (multi-language) + taint facts; optional Joern CPG sidecar; Python stdlib parse only as last-resort fallback. Promote, demote, or **unadjudicated**.
3. **Budget** — ranker + **mechanism memory** (JSONL + OpenRouter embeddings, hard CWE/language/tag filters) **before** prove. Not a verdict. Not TF-IDF. Not raw file-slice embeddings.
4. **Prove** — sandbox only on promoted, reachable, budgeted candidates.
1. **Propose**, pattern rules + optional SARIF. Stage 1. Does not decide worth-fixing.
2. **Adjudicate**, tree-sitter IR (multi-language) + taint facts; optional Joern CPG sidecar; Python stdlib parse only as last-resort fallback. Promote, demote, or **unadjudicated**.
3. **Budget**, ranker + **mechanism memory** (JSONL + OpenRouter embeddings, hard CWE/language/tag filters) **before** prove. It is a budget rather than a verdict; it uses neither TF-IDF nor raw file-slice embeddings.
4. **Prove**, sandbox only on promoted, reachable, budgeted candidates.
The primary in-scope target is a **messy working tree the operator is writing**, not a labeled benchmark. Known-vuln corpora remain calibration scoreboards (smoke 90/10, pair Youden, GitHub silent-on-fix). They do not define success on unlabeled code.
## Scope
- Semantic overlay as MAP-stage adjudication; **tree-sitter is the prio engine** (CLI/API, fail-closed). Joern optional. Embeddings tune prove order.
- Explicit `unadjudicated` when parse or flow fails — do not treat silence as safe.
- Explicit `unadjudicated` when parse or flow fails. Do not treat silence as safe.
- Sandbox prove only after promote.
- Pair eval scores the overlay on the sast slice; smoke gate stays inventory-only.
+18 -18
View File
@@ -10,10 +10,10 @@
### Goals
- Map every current scan module onto propose, adjudicate, budget, or prove — rank/embed is not a verdict.
- Map every current scan module onto propose, adjudicate, budget, or prove (rank/embed is not a verdict).
- Treat messy unlabeled trees as the default; labeled corpora as calibration.
- Fail closed: cannot parse or cannot follow flow → `unadjudicated`, not demote.
- **Prioritize the semantic engine now**: tree-sitter IR for in-scope languages; Joern optional CPG sidecar; stdlib Python parse only as last resort.
- Prioritize the semantic engine now: tree-sitter IR for in-scope languages; Joern optional CPG sidecar; stdlib Python parse only as last resort.
- Tunable embeddings + existing ranker order **prove** candidates.
- SAST pair slice scores the overlay; smoke gate stays inventory-only.
@@ -51,11 +51,11 @@
- Preprocess, mapping, rank (`heuristic_rank` 0.5/0.2/0.3), `quick_scan_findings`, complexity map, `ousast index` / json-local vectorstore, tool hunter, sandbox runner, pair eval, reports.
- Host binaries when present: `tree-sitter`, `joern` / `joern-parse` (probed, never required for inventory).
- Optional embedding model from existing `[embeddings]` config.
- Clearwing as an **oracle** for tree-sitter callgraph + taint-pattern tables + mechanism retrieval — not a dependency.
- Clearwing as an **oracle** for tree-sitter callgraph + taint-pattern tables + mechanism retrieval, not a dependency.
### Revalidation Triggers
- Adding a fourth disposition that implies “safe.”
- Adding a fourth disposition that implies "safe."
- Demoting when flow is incomplete.
- Stamping overlay above `static_corroboration`.
- Selecting sandbox candidates from demoted or unadjudicated sets.
@@ -84,7 +84,7 @@ Concrete mapping of **what exists today** onto the three roles:
| Prove | REGRESS / `deep` | `regress` + sandbox | Recipes on ranked findings | Recipes on **promoted then budgeted** findings |
| Calibrate | CI | `gate.py` / `pairs` | Cheat-sheet vs Youden | Inventory vs overlay, unchanged split |
**Selected pattern.** Propose → adjudicate → **budget** → prove. Clearwing’s tree-sitter callgraph + taint tables + mechanism memory are the oracle, not a library import. Embeddings never demote.
**Selected pattern.** Propose → adjudicate → **budget** → prove. Clearwing's tree-sitter callgraph + taint tables + mechanism memory are the oracle, not a library import. Embeddings never demote.
```mermaid
flowchart TB
@@ -161,13 +161,13 @@ src/openultrasast/ruleset/semantic/
### Modified Files
- `stages.py` / `cli.py` — overlay in MAP; prove-budget before regress.
- `findings.py` — inventory unchanged.
- `regress/candidate.py` — `promote` only, then budget order.
- `rank.py` / `index.py` / `vectorstore.py` — reuse; do not invent a second store.
- `reports.py` — disposition columns.
- `pairs.py` — sast slice uses overlay.
- `mcp.py` — still no `deep`.
- `stages.py` / `cli.py`, overlay in MAP; prove-budget before regress.
- `findings.py`, inventory unchanged.
- `regress/candidate.py`, `promote` only, then budget order.
- `rank.py` / `index.py` / `vectorstore.py`, reuse; do not invent a second store.
- `reports.py`, disposition columns.
- `pairs.py`, sast slice uses overlay.
- `mcp.py`, still no `deep`.
## System Flows
@@ -301,11 +301,11 @@ Identity is the *pattern*, not a file slice:
| what_made_it_exploitable | why a near-miss became a bug |
| source_finding_id, source_repo | provenance only |
Persistence: append-only JSONL (portable). Recall is **not** Clearwing’s TF-IDF ladder.
Persistence: append-only JSONL (portable). Recall is **not** Clearwing's TF-IDF ladder.
1. **Filter** — language, CWE, tags (exact overlap). This is a hard prefilter, not a ranker.
2. **Rank** — OpenRouter `/embeddings` (`OpenRouterEmbeddingClient`, existing `[embeddings].model`) over `summary + what_made_it_exploitable + tags`. Cosine against the promotion’s overlay text. Cache vectors in json-local `mechanisms` namespace, keyed by mechanism id + model id.
3. **Degrade** — no key / no model / embed failure → heuristic ranker only, degradation recorded. Never bag-of-words cosine.
1. **Filter**, language, CWE, tags (exact overlap). This is a hard prefilter, not a ranker.
2. **Rank**, OpenRouter `/embeddings` (`OpenRouterEmbeddingClient`, existing `[embeddings].model`) over `summary + what_made_it_exploitable + tags`. Cosine against the promotion's overlay text. Cache vectors in json-local `mechanisms` namespace, keyed by mechanism id + model id.
3. **Degrade**, no key / no model / embed failure → heuristic ranker only, degradation recorded. Never bag-of-words cosine.
Query for a promotion: embed the overlay rationale (sources, sink, CWE). Result is a **boost to prove order**, never a demote.
@@ -327,7 +327,7 @@ Repo-code chunks (`ousast index` 80-line slices) stay for hunter/verifier retrie
- Unit: parse fail → unadjudicated; constant `exec("ls")` → demote; `eval(request.args)` → promote; `sql = f"…{x}"; execute(sql)` → coverage or promote.
- Pair: `--slice sast` uses overlay; `juliet-c-cwe134-printf` stays PASS; `owasp-python-deser` stays PASS; Juliet `goodG2B` exec/strcpy must **demote or unadjudicated**, not inventory-leak counted as pair fail for overlay.
- Unlabeled: tmp tree with a syntax-error file next to a clean `eval(request.args)` — error file unadjudicated, clean file promoted; no expected.toml required.
- Unlabeled: tmp tree with a syntax-error file next to a clean `eval(request.args)`. Error file unadjudicated, clean file promoted; no expected.toml required.
- Gate: `python -m openultrasast.gate` still inventory-only.
- Deep: candidate list excludes demoted.
@@ -346,4 +346,4 @@ Repo-code chunks (`ousast index` 80-line slices) stay for hunter/verifier retrie
- Default `standard` runs overlay when facts load; if facts missing, degrade as Requirement 7.5.
- Existing `quick` CI unchanged.
- Pair gate local 100% remains inventory-or-overlay-equivalent on fixture counterparts (those files are still “nice”; unlabeled tests are additional).
- Pair gate local 100% remains inventory-or-overlay-equivalent on fixture counterparts (those files are still "nice"; unlabeled tests are additional).
@@ -4,15 +4,15 @@
## Introduction
OpenUltraSAST users scan **working trees they are writing**: mixed files, incomplete edits, generated code, framework wrappers, and no labels. Today the harness still *decides* with pattern inventory. The complexity map ranks; the sandbox can run; neither adjudicates “this sink is fed by a source” versus “this sink is hardcoded, sanitized, or unreadable.”
OpenUltraSAST users scan **working trees they are writing**: mixed files, incomplete edits, generated code, framework wrappers, and no labels. Today the harness still *decides* with pattern inventory. The complexity map ranks; the sandbox can run; neither adjudicates "this sink is fed by a source" versus "this sink is hardcoded, sanitized, or unreadable."
This feature splits the scan into three operator-visible roles that must not collapse: **regex proposes, semantics adjudicates, sandbox proves.** Labeled suites (cheat-sheet fixtures, OWASP Benchmark, Juliet, CVE pairs) remain **calibration**. They are not the definition of in-scope code.
## Boundary Context
- **In scope**: Role split on `quick` / `standard` / `deep`; overlay dispositions; tree-sitter (and optional Joern) as the semantic engine; tunable retrieval/rank **before prove**; `unadjudicated` on messy files; sandbox only after promote; calibration vs unlabeled working trees.
- **Out of scope**: Deleting pattern rules; making an LLM the adjudicator; rewriting CWE policy or project-score formula; requiring Joern or an embedding model for `quick`; auto-generated flow facts from the improve loop; treating benchmark Youden as a merge gate that can unblock a smoke-gate breach. Embedding rank is a **budget**, not a fourth verdict.
- **Adjacent expectations**: Stage plan, complexity map, Docker sandbox, pair catalog, and 90/10 detection gate from `three-stage-scan` keep their contracts. This spec overlays adjudication on that pipeline; it does not replace stage accounting, sandbox flags, or `LANGUAGE_MANIFESTS`.
- In scope: Role split on `quick` / `standard` / `deep`; overlay dispositions; tree-sitter (and optional Joern) as the semantic engine; tunable retrieval/rank **before prove**; `unadjudicated` on messy files; sandbox only after promote; calibration vs unlabeled working trees.
- Out of scope: Deleting pattern rules; making an LLM the adjudicator; rewriting CWE policy or project-score formula; requiring Joern or an embedding model for `quick`; auto-generated flow facts from the improve loop; treating benchmark Youden as a merge gate that can unblock a smoke-gate breach. Embedding rank is a **budget**, not a fourth verdict.
- Adjacent expectations: Stage plan, complexity map, Docker sandbox, pair catalog, and 90/10 detection gate from `three-stage-scan` keep their contracts. This spec overlays adjudication on that pipeline; it does not replace stage accounting, sandbox flags, or `LANGUAGE_MANIFESTS`.
## Requirements
@@ -36,9 +36,9 @@ This feature splits the scan into three operator-visible roles that must not col
1. When a scanned file is incomplete, generated, minified, or otherwise not interpretable as structured source, the system shall leave matching inventory proposals **unadjudicated** and shall record that reason on the finding.
2. When no available semantic engine can follow data from a source to a sink for that file (parse failure, missing grammar, cross-file call, dynamic dispatch, or missing callee), the system shall leave the proposal unadjudicated rather than demote it.
3. When OpenUltraSAST scans a working tree with no benchmark manifest and no expected-finding labels, the system shall still emit proposals, dispositions, and (in `deep`) proofs without requiring a counterpart “fixed” tree.
3. When OpenUltraSAST scans a working tree with no benchmark manifest and no expected-finding labels, the system shall still emit proposals, dispositions, and (in `deep`) proofs without requiring a counterpart "fixed" tree.
4. If the operator reads a report for an unadjudicated proposal, then the system shall present it as unresolved inventory, not as a verified vulnerability and not as a cleared finding.
5. The system shall not require a file to resemble an OWASP Benchmark or Juliet test case in order to produce a proposal.
5. The system shall not require a file to resemble an OWASP Benchmark or Juliet test case to produce a proposal.
### Requirement 3: Adjudication promote, demote, coverage
@@ -46,8 +46,8 @@ This feature splits the scan into three operator-visible roles that must not col
#### Acceptance Criteria
1. When a proposal’s sink is reachable from an identified untrusted source without an identified sanitizer in the adjudicated region, the system shall **promote** the proposal.
2. When a proposal’s sink is fed only by a constant, a sanitizer, or an identified non-attacker source in the adjudicated region, the system shall **demote** the proposal and shall not list it as a promoted finding.
1. When a proposal's sink is reachable from an identified untrusted source without an identified sanitizer in the adjudicated region, the system shall **promote** the proposal.
2. When a proposal's sink is fed only by a constant, a sanitizer, or an identified non-attacker source in the adjudicated region, the system shall **demote** the proposal and shall not list it as a promoted finding.
3. When adjudication finds an untrusted source flowing to a dangerous sink that inventory did not propose, the system shall emit a **coverage** finding at corroboration-or-lower evidence, tagged as overlay-originated.
4. When both a source-to-sink path and a sanitizer appear to apply, the system shall not promote unless the sanitizer does not dominate that path.
5. The system shall persist promote, demote, unadjudicated, and coverage outcomes in the scan artifacts with the proposal identifier they refer to.
@@ -62,7 +62,7 @@ This feature splits the scan into three operator-visible roles that must not col
2. When the pair-efficiency evaluation runs on the SAST slice, the system shall score **adjudicated** outcomes (promote on the vulnerable side, demote or silent on the false or good side), not raw inventory hits.
3. When overlay changes raise SAST Youden and also cause the stage-1 detection gate to fail, the system shall treat the detection-gate failure as blocking.
4. The system shall keep labeled pair fixtures and SAST-slice cases out of the stage-1 cheat-sheet corpus map.
5. The system shall not require a labeled counterpart tree in order to scan an operator working tree.
5. The system shall not require a labeled counterpart tree to scan an operator working tree.
### Requirement 5: Evidence and models stay honest
@@ -74,7 +74,7 @@ This feature splits the scan into three operator-visible roles that must not col
2. When the tool hunter emits a finding, the system shall keep that finding at `suspicion` unless an independent corroborating stage has already raised it.
3. If an overlay or hunter result claims an evidence level above corroboration without a sandbox or verifier step that earned it, then the system shall reject that claim.
4. The system shall not use a language model as the adjudication engine that decides promote or demote.
5. Where CWE policy applies, the system shall resolve severity from the central CWE policy, not from the overlay or the proposing rule’s private severity string.
5. Where CWE policy applies, the system shall resolve severity from the central CWE policy, not from the overlay or the proposing rule's private severity string.
### Requirement 6: Sandbox proves only promoted work
@@ -106,7 +106,7 @@ This feature splits the scan into three operator-visible roles that must not col
#### Acceptance Criteria
1. When a tree-sitter grammar for the file’s language is available, the system shall use that parse as the primary intra-file IR for adjudication.
1. When a tree-sitter grammar for the file's language is available, the system shall use that parse as the primary intra-file IR for adjudication.
2. When tree-sitter is unavailable for that language, the system shall try a stdlib parse only for Python and shall otherwise leave the proposal unadjudicated with language_unsupported or parse_failed.
3. Where Joern is installed and the language is one Joern can export, the system shall be allowed to attach Joern flow facts as **additional** corroboration; Joern absence shall not fail `quick` or `standard`.
4. If Joern and tree-sitter disagree or Joern returns incomplete paths, then the system shall not demote on that disagreement; it shall promote only on a complete dominating path or leave the item unadjudicated.
@@ -124,6 +124,6 @@ This feature splits the scan into three operator-visible roles that must not col
3. The system shall not demote or drop a promotion solely because mechanism similarity is low.
4. When the embedding index is missing or stale, the system shall rebuild it from the mechanism log via OpenRouter embeddings; if that rebuild fails, the system shall record the degradation and continue on heuristic order.
5. The system shall keep hard filters operator-tunable (language, CWE, tags) without changing CWE severity policy.
6. When a finding is sandbox-proven triggerable (or otherwise verified at corroboration-or-higher with a prove step), the system shall append an abstract mechanism record to the mechanism log: summary, CWE, language, tags, what made it exploitable — without file:line as the identity.
6. When a finding is sandbox-proven triggerable (or otherwise verified at corroboration-or-higher with a prove step), the system shall append an abstract mechanism record to the mechanism log: summary, CWE, language, tags, what made it exploitable, without file:line as the identity.
7. The system shall treat the append-only mechanism log as the source of truth; any vector index shall be a rebuildable cache of that log.
8. The system shall not persist mechanisms from inventory-only, demoted, or unadjudicated proposals.
@@ -2,9 +2,9 @@
## Summary
- **Feature**: `propose-adjudicate-prove`
- **Discovery Scope**: Extension of `three-stage-scan` (MAP grows an overlay; STATIC and REGRESS stay)
- **Key Findings**:
- Feature: `propose-adjudicate-prove`
- Discovery Scope: Extension of `three-stage-scan` (MAP grows an overlay; STATIC and REGRESS stay)
- Key Findings:
- Inventory already decides; MAP ranks; REGRESS can run. Nothing adjudicates flow.
- Labeled SAST slice Youden is about +9%. Leaks are OWASP-false / Juliet `goodG2B` that still contain the sink. Misses are one-hop assignments and sink aliases.
- Clearwing already does the split we want: tree-sitter callgraph + pattern taint tables, then ranker (surface/influence/reachability 0.5/0.2/0.3), then mechanism-memory embeddings. OpenUltraSAST already copied the rank formula and has `json-local` vectors; they are not wired as prove-budget.
@@ -15,55 +15,55 @@
### Labeled vs unlabeled code
- **Context**: Operator: known-vuln corpora are “nice”; written code will not be.
- **Sources Consulted**: In-tree `LANGUAGE_MANIFESTS`, `benchmarks/pairs/`, OWASP Benchmark true/false files, Juliet `bad`/`goodG2B`/`goodB2G`, CWE-Bench-Java catalog.
- **Findings**:
- Context: Operator: known-vuln corpora are "nice"; written code will not be.
- Sources Consulted: In-tree `LANGUAGE_MANIFESTS`, `benchmarks/pairs/`, OWASP Benchmark true/false files, Juliet `bad`/`goodG2B`/`goodB2G`, CWE-Bench-Java catalog.
- Findings:
- Cheat-sheets: one CWE, named sink, often same-line source.
- OWASP: true and false files both often contain `exec` / `eval`; the difference is source constancy or infeasible branches.
- Juliet `goodG2B`: hardcoded source, **same sink**.
- Juliet `goodB2G`: same source, **sanitized sink** (`printf("%s")`).
- Working trees: syntax errors, generated bundles, wrappers (`helpers.db_sqlite.execute`), no labels.
- **Implications**: Overlay scored on sast slice; unlabeled scans must not pretend demote. Unadjudicated is a first-class disposition.
- Implications: Overlay scored on sast slice; unlabeled scans must not pretend demote. Unadjudicated is a first-class disposition.
### Engine choice
- **Context**: Operator prefers tree-sitter + Joern look, embeddings before prove, still v1-focused.
- **Sources Consulted**: `clearwing/sourcehunt/callgraph.py`, `taint.py`, `ranker.py`, `mechanism_memory.py`; `openultrasast/rank.py`, `vectorstore.py`, `index.py`.
- **Findings**:
- Context: Operator prefers tree-sitter + Joern look, embeddings before prove, still v1-focused.
- Sources Consulted: `clearwing/sourcehunt/callgraph.py`, `taint.py`, `ranker.py`, `mechanism_memory.py`; `openultrasast/rank.py`, `vectorstore.py`, `index.py`.
- Findings:
- Clearwing tree-sitter is optional: missing grammar disables callgraph, does not crash.
- Clearwing taint is intra-procedural, pattern tables for sources/sinks — same shape as our fact TOML.
- Clearwing taint is intra-procedural, pattern tables for sources/sinks, same shape as our fact TOML.
- Clearwing ranker fills reachability from the callgraph; taint_paths boost surface. Embeddings retrieve *mechanisms*, they do not decide taint.
- Our `heuristic_rank` already uses surface·0.5 + influence·0.2 + reachability·0.3. Vector store bakeoff selected `json-local`. `VALID_NAMESPACES` already includes `mechanisms` but nothing writes Clearwing-style records into it.
### Clearwing vector-store reference (real-world bugs)
- **Context**: Operator: Clearwing’s store is effective on real bugs, not textbook files.
- **Sources Consulted**: `clearwing/sourcehunt/mechanism_memory.py`, `data/memory/semantic_memory.py`, `docs/architecture.md` steps 7–8; `openultrasast/index.py` namespaces.
- **Findings**:
- Store **mechanisms**, not findings and not 80-line file windows. Example: “length field trusted before alloc; 16-bit value widened to size_t.”
- Context: Operator: Clearwing's store is effective on real bugs, not textbook files.
- Sources Consulted: `clearwing/sourcehunt/mechanism_memory.py`, `data/memory/semantic_memory.py`, `docs/architecture.md` steps 7–8; `openultrasast/index.py` namespaces.
- Findings:
- Store **mechanisms**, not findings and not 80-line file windows. Example: "length field trusted before alloc; 16-bit value widened to size_t."
- JSONL is the source of truth (`~/.clearwing/sourcehunt/mechanisms.jsonl`). Chroma/sqlite embeddings are ephemeral caches rebuilt from JSONL.
- Write path is **verified findings only** (after adversarial verify / crash). Suspicion never seeds the store.
- Clearwing’s JSONL + verified-write + metadata filters are the reference. Its TF-IDF default is **not**: bag-of-words will not match paraphrased real bugs (`length field trusted before alloc` vs `user-controlled size_t in malloc`).
- Clearwing's JSONL + verified-write + metadata filters are the reference. Its TF-IDF default is **not**: bag-of-words will not match paraphrased real bugs (`length field trusted before alloc` vs `user-controlled size_t in malloc`).
- OpenUltraSAST already has `OpenRouterEmbeddingClient` and json-local cosine. Use that as the only semantic ranker; CWE/language/tags are hard filters. No TF-IDF, no Chroma, no local MiniLM.
- Query is language + file tags + optional free text; same-language boost +0.5. Top-N injects hunter context and variant-grep seeds — it does not decide taint.
- Query is language + file tags + optional free text; same-language boost +0.5. Top-N injects hunter context and variant-grep seeds. It does not decide taint.
- Variant loop then greps the *messy* tree for structural matches of that mechanism. That is how it leaves Juliet behind.
- **Implications**: Copy JSONL + verified-write + metadata filters. Rank with OpenRouter embeddings only. Do not copy TF-IDF. Do not embed 80-line file windows as the prove store. Seed from sandbox-proven mechanisms, not OWASP `BenchmarkTest00007`.
- Implications: Copy JSONL + verified-write + metadata filters. Rank with OpenRouter embeddings only. Do not copy TF-IDF. Do not embed 80-line file windows as the prove store. Seed from sandbox-proven mechanisms, not OWASP `BenchmarkTest00007`.
- Joern CLI (`joern-parse` / scripts) is the only practical CPG without a Scala runtime in-process.
- **Implications**: Probe tree-sitter like Docker. Joern sidecar cannot demote. Embeddings order prove only.
- Implications: Probe tree-sitter like Docker. Joern sidecar cannot demote. Embeddings order prove only.
### Decision: Python intra-file first
- **Context**: Zero-dep core vs multi-language trees.
- **Selected**: **Superseded.** Tree-sitter is the prio engine; stdlib `ast` is Python last resort; `quick` stays zero-dep.
- **Rationale**: Operator working trees are mixed-language; Python-only adjudication would leave JS/C/Java as perpetual unadjudicated.
- **Trade-offs**: `standard` needs a grammar or CLI on PATH; missing tools → unadjudicated, not skip the scan.
- Context: Zero-dep core vs multi-language trees.
- Selected: **Superseded.** Tree-sitter is the prio engine; stdlib `ast` is Python last resort; `quick` stays zero-dep.
- Rationale: Operator working trees are mixed-language; Python-only adjudication would leave JS/C/Java as perpetual unadjudicated.
- Trade-offs: `standard` needs a grammar or CLI on PATH; missing tools → unadjudicated, not skip the scan.
### Evidence ladder
- **Context**: Three-stage-scan forbids hunter JSON as corroboration.
- **Sources Consulted**: `verification.EvidenceLevel`, `tool_hunter` suspicion rule, `worth_fixing` gate.
- **Findings**: Overlay can justify promote at `static_corroboration`. It cannot skip to crash or exploit. Worth-fixing stays sandbox + reachability.
- **Implications**: Disposition is not an evidence rung. Do not add a seventh ladder value.
- Context: Three-stage-scan forbids hunter JSON as corroboration.
- Sources Consulted: `verification.EvidenceLevel`, `tool_hunter` suspicion rule, `worth_fixing` gate.
- Findings: Overlay can justify promote at `static_corroboration`. It cannot skip to crash or exploit. Worth-fixing stays sandbox + reachability.
- Implications: Disposition is not an evidence rung. Do not add a seventh ladder value.
## Architecture Pattern Evaluation
@@ -78,28 +78,28 @@
### Decision: Unadjudicated is not demote
- **Context**: Messy files and cross-file flow.
- **Alternatives**: Silent drop; treat as safe; force demote.
- **Selected**: Explicit `unadjudicated` with reason.
- **Rationale**: Operator writing code must not read silence as “clean.”
- **Trade-offs**: Reports stay noisy until languages and inter-file land.
- **Follow-up**: Count unadjudicated rate on unlabeled trees as an honesty metric.
- Context: Messy files and cross-file flow.
- Alternatives: Silent drop; treat as safe; force demote.
- Selected: Explicit `unadjudicated` with reason.
- Rationale: Operator writing code must not read silence as "clean."
- Trade-offs: Reports stay noisy until languages and inter-file land.
- Follow-up: Count unadjudicated rate on unlabeled trees as an honesty metric.
### Decision: Score overlay on sast slice, inventory on smoke gate
- **Context**: +9% Youden vs 93% cheat-sheet recall.
- **Selected**: Two gates stay separate; overlay Youden cannot unblock smoke failure.
- **Rationale**: Same as detection-gate vs score in the improve loop.
- Context: +9% Youden vs 93% cheat-sheet recall.
- Selected: Two gates stay separate; overlay Youden cannot unblock smoke failure.
- Rationale: Same as detection-gate vs score in the improve loop.
## Risks & Mitigations
- False demote on real bugs (infeasible-branch OWASP tricks, wrappers) — never demote when flow is incomplete; only demote when the engine *has* a dominating constant or sanitizer.
- Youden chase via extra sinks — each coverage rule needs a false/good counterpart that must stay demoted or silent.
- Parse failures on operator WIP — unadjudicated, `quick` still works.
- False demote on real bugs (infeasible-branch OWASP tricks, wrappers). Never demote when flow is incomplete; only demote when the engine *has* a dominating constant or sanitizer.
- Youden chase via extra sinks. Each coverage rule needs a false/good counterpart that must stay demoted or silent.
- Parse failures on operator WIP, unadjudicated, `quick` still works.
## References
- `.kiro/specs/three-stage-scan/design.md` — stage plan, sandbox, hunter suspicion
- `benchmarks/pairs/sast/README.md` — OWASP/Juliet slice
- `.kiro/specs/three-stage-scan/design.md`, stage plan, sandbox, hunter suspicion
- `benchmarks/pairs/sast/README.md`, OWASP/Juliet slice
- NIST Juliet `goodG2B` vs `goodB2G` templates
- OWASP Benchmark expectedresults true/false protocol
+7 -7
View File
@@ -24,8 +24,8 @@ Build a name-resolved call graph from the `FunctionIR` records tree-sitter alrea
## Scope
- **In**: call graph, summaries, path records, reachability attachment, rank and map integration, hunter on paths, hunter-to-candidate path, fact oracle with cache, `facts` lever with validator, fix-point ranking and report section, measurement on split-sink, vibe-py, vfc slices, offline tests.
- **Out**: guard dominance semantics (`guard-dominance-regime` consumes path records), CST walker fixes (`overlay-ir-completeness`), a whole-program points-to or symbolic engine, concolic execution, executing target code outside the sandbox, new dispositions, LLM as parser, any merge gate change.
- In: call graph, summaries, path records, reachability attachment, rank and map integration, hunter on paths, hunter-to-candidate path, fact oracle with cache, `facts` lever with validator, fix-point ranking and report section, measurement on split-sink, vibe-py, vfc slices, offline tests.
- Out: guard dominance semantics (`guard-dominance-regime` consumes path records), CST walker fixes (`overlay-ir-completeness`), a whole-program points-to or symbolic engine, concolic execution, executing target code outside the sandbox, new dispositions, LLM as parser, any merge gate change.
## Boundary Candidates
@@ -42,14 +42,14 @@ Build a name-resolved call graph from the `FunctionIR` records tree-sitter alrea
## Upstream / Downstream
- **Upstream**: `FileIR`/`FunctionIR`, `analyze_entry_points`, `analyze_ir`, `load_facts`, `Hotspot`, `run_tool_hunter`, `select_candidates`, `run_improvement`, pair holdout metrics per profile.
- **Downstream**: `guard-dominance-regime` attaches guards to path records; later web-authorization detection is a path whose hops include no auth guard; reports and MCP read fix points.
- Upstream: `FileIR`/`FunctionIR`, `analyze_entry_points`, `analyze_ir`, `load_facts`, `Hotspot`, `run_tool_hunter`, `select_candidates`, `run_improvement`, pair holdout metrics per profile.
- Downstream: `guard-dominance-regime` attaches guards to path records; later web-authorization detection is a path whose hops include no auth guard; reports and MCP read fix points.
## Existing Spec Touchpoints
- **Extends**: `three-stage-scan` (hunter tools, candidate selection, worth-fixing input), `propose-adjudicate-prove` (facts consumed unchanged; one new fact source), `harnessx-self-improving-rulesets` (one new lever under the same gate).
- **Depends**: `pair-corpus-honesty` for per-profile holdout metrics and the hunter scorer.
- **Adjacent**: `overlay-ir-completeness` improves the records this spec walks; neither blocks the other.
- Extends: `three-stage-scan` (hunter tools, candidate selection, worth-fixing input), `propose-adjudicate-prove` (facts consumed unchanged; one new fact source), `harnessx-self-improving-rulesets` (one new lever under the same gate).
- Depends: `pair-corpus-honesty` for per-profile holdout metrics and the hunter scorer.
- Adjacent: `overlay-ir-completeness` improves the records this spec walks; neither blocks the other.
## Constraints
@@ -146,10 +146,10 @@ tests/
### Modified Files
- `semantic/cst.py`, `semantic/ir.py` — collect import statements into `FileIR.imports`; no other IR change.
- `mapping.py` — new function; `EntryPointRecord` unchanged.
- `stage_processors.py` — MAP stage runs graph, summaries, paths before overlay; overlay unchanged.
- `map_gate.py` — path assertion per language family.
- `semantic/cst.py`, `semantic/ir.py`, collect import statements into `FileIR.imports`; no other IR change.
- `mapping.py`, new function; `EntryPointRecord` unchanged.
- `stage_processors.py`, MAP stage runs graph, summaries, paths before overlay; overlay unchanged.
- `map_gate.py`, path assertion per language family.
## System Flows
@@ -8,9 +8,9 @@ OpenUltraSAST cannot yet steer effort where risk is reduced because it has no mo
## Boundary Context
- **In scope**: call graph, taint summaries, path records, reachability attachment to findings, rank, complexity map and worth-fixing; hunter path context; hunter findings as sandbox candidates; fact oracle and candidate-fact cache; `facts` improve lever; fix-point ranking and report section; offline tests and measurement on existing slices.
- **Out of scope**: guard semantics; CST walker fixes; whole-program alias analysis; symbolic or concolic execution; new dispositions or evidence rungs; LLM as parser; any merge gate change; profile-specific fact tables beyond the gate defined by `pair-corpus-honesty`.
- **Adjacent expectations**: `FileIR` shape is frozen except additive fields; `load_facts` merges candidate facts below hand-written facts; the improve loop's existing accept gate and per-profile holdout clause apply unchanged to the new lever; sandbox safety checks apply to hunter snippets as today.
- In scope: call graph, taint summaries, path records, reachability attachment to findings, rank, complexity map and worth-fixing; hunter path context; hunter findings as sandbox candidates; fact oracle and candidate-fact cache; `facts` improve lever; fix-point ranking and report section; offline tests and measurement on existing slices.
- Out of scope: guard semantics; CST walker fixes; whole-program alias analysis; symbolic or concolic execution; new dispositions or evidence rungs; LLM as parser; any merge gate change; profile-specific fact tables beyond the gate defined by `pair-corpus-honesty`.
- Adjacent expectations: `FileIR` shape is frozen except additive fields; `load_facts` merges candidate facts below hand-written facts; the improve loop's existing accept gate and per-profile holdout clause apply unchanged to the new lever; sandbox safety checks apply to hunter snippets as today.
## Requirements
@@ -79,7 +79,7 @@ OpenUltraSAST cannot yet steer effort where risk is reduced because it has no mo
### Requirement 6: Facts as an improve-loop lever
**Objective:** As a maintainer, I want the improve loop to be able to accept or reject facts, so that the evolutionary machinery can move detection and not only rule status.
**Objective:** As a maintainer, I want the improve loop to be able to accept or reject facts, so that the evolutionary machinery can move both detection and rule status.
#### Acceptance Criteria
+49 -49
View File
@@ -2,9 +2,9 @@
## Summary
- **Feature**: `reachability-flow-model`
- **Discovery Scope**: Complex Integration across semantic, mapping, rank, complexity, hunter, regress, improve
- **Key Findings**:
- Feature: `reachability-flow-model`
- Discovery Scope: Complex Integration across semantic, mapping, rank, complexity, hunter, regress, improve
- Key Findings:
- `FunctionIR` already carries every call site per function for Python, JS, TS, C, and Java when the extra is present. A name-resolved call graph is a few hundred lines over data that exists.
- Reachability today is "line inside an entry-point handler" (`mapping.attach_reachability_hints`); `rank` bumps an integer from tags and hints; `worth_fixing` needs `reachable` or `inferred-file-surface`. No call edge exists anywhere.
- Sandbox candidates come only from hotspots filtered to overlay-promoted inventory ids (`cli.py:404`, `prove_filter.filter_promoted_hotspots`); a hunter snippet is used only for an already promoted hotspot (`candidate.py:199`). The LLM plane cannot create a candidate.
@@ -18,38 +18,38 @@
### Where a call graph can come from
- **Context**: Avoid a new parser or a whole-program engine.
- **Sources Consulted**: `semantic/ir.py`, `semantic/cst.py`, `mapping.py`, `preprocess.py`, `hunter_tools.find_refs`.
- **Findings**: `FileIR.functions[*].calls[*].name` is the callee text. Python stdlib parse yields the same records. Imports are not in the IR; the walker can add an `imports` tuple per file cheaply. `find_refs` is a word-boundary regex over files and is the only cross-file lookup today.
- **Implications**: `semantic/callgraph.py` consumes `FileIR` plus an additive `imports` field. Resolution order: same-file definition, imported module definition, any definition with that name (ambiguous edges). Method calls resolve by trailing name. C resolves by bare name across the tree.
- Context: Avoid a new parser or a whole-program engine.
- Sources Consulted: `semantic/ir.py`, `semantic/cst.py`, `mapping.py`, `preprocess.py`, `hunter_tools.find_refs`.
- Findings: `FileIR.functions[*].calls[*].name` is the callee text. Python stdlib parse yields the same records. Imports are not in the IR; the walker can add an `imports` tuple per file cheaply. `find_refs` is a word-boundary regex over files and is the only cross-file lookup today.
- Implications: `semantic/callgraph.py` consumes `FileIR` plus an additive `imports` field. Resolution order: same-file definition, imported module definition, any definition with that name (ambiguous edges). Method calls resolve by trailing name. C resolves by bare name across the tree.
### Summaries versus whole-program taint
- **Context**: `propose-adjudicate-prove` deferred inter-file taint as too large and risky for false demotes.
- **Sources Consulted**: `semantic/taint.py` (`_analyze_function` seeds parameters as tainted, tracks names, emits `TaintPath` and `Demotion`), overlay invariants.
- **Findings**: Because parameters are already seeded tainted, a per-function run already answers "which parameter reaches which sink." Adding "which parameter reaches the return" is a small extension. Composition is then a graph walk with a depth bound. Nothing needs alias analysis to produce a useful path; anything unresolved stays `flow_incomplete`.
- **Implications**: Summaries are a pure function of `FunctionIR` plus facts, cacheable by file hash. Composition never demotes; it only adds paths or marks incompleteness.
- Context: `propose-adjudicate-prove` deferred inter-file taint as too large and risky for false demotes.
- Sources Consulted: `semantic/taint.py` (`_analyze_function` seeds parameters as tainted, tracks names, emits `TaintPath` and `Demotion`), overlay invariants.
- Findings: Because parameters are already seeded tainted, a per-function run already answers "which parameter reaches which sink." Adding "which parameter reaches the return" is a small extension. Composition is then a graph walk with a depth bound. Nothing needs alias analysis to produce a useful path; anything unresolved stays `flow_incomplete`.
- Implications: Summaries are a pure function of `FunctionIR` plus facts, cacheable by file hash. Composition never demotes; it only adds paths or marks incompleteness.
### Hunter to candidate
- **Context**: The LLM plane is the recall lever the field data supports, but it is walled off from prove.
- **Sources Consulted**: `tool_hunter.py`, `regress/candidate.py`, `three-stage-scan` design (hunter snippets run only in the sandbox after safety check), `verification.py`.
- **Findings**: The safety check and sandbox are origin-agnostic. The wall is `filter_promoted_hotspots` and the candidate selector's reliance on inventory ids. A hunter finding with a locatable sink line can be given a synthetic hotspot the same way `_hotspot_from_forced_finding` already does for severity-5 inventory.
- **Implications**: Add eligibility for hunter findings that name a sink line resolvable to a path record or an overlay sink. Evidence stays suspicion until sandbox trigger or path agreement.
- Context: The LLM plane is the recall lever the field data supports, but it is walled off from prove.
- Sources Consulted: `tool_hunter.py`, `regress/candidate.py`, `three-stage-scan` design (hunter snippets run only in the sandbox after safety check), `verification.py`.
- Findings: The safety check and sandbox are origin-agnostic. The wall is `filter_promoted_hotspots` and the candidate selector's reliance on inventory ids. A hunter finding with a locatable sink line can be given a synthetic hotspot the same way `_hotspot_from_forced_finding` already does for severity-5 inventory.
- Implications: Add eligibility for hunter findings that name a sink line resolvable to a path record or an overlay sink. Evidence stays suspicion until sandbox trigger or path agreement.
### Facts from a model, safely
- **Context**: The engine lacks callee knowledge: nullable returns (`tls1_lookup_sigalg`), bounded arguments, transformer set/get (`ConfigParser`), macro out-params (`n2s`).
- **Sources Consulted**: `semantic/facts.py` loader, `improve/validator.py`, `improve/journal.py`, `calibration.py` ledger patterns, `redaction.py`.
- **Findings**: Facts are versioned TOML loaded by `load_facts`. The loader can merge a second directory at lower precedence. The improve loop has a strict validator and byte-for-byte revert. Redaction exists for prompts.
- **Implications**: Model answers are candidate facts in `.openultrasast/calibration/candidate_facts.toml`; accepted facts in `accepted_facts.toml`; loader precedence hand-written > accepted > none. The `facts` lever moves rows between candidate and accepted under the same gate. No pattern text is ever model-authored; question kinds are closed.
- Context: The engine lacks callee knowledge: nullable returns (`tls1_lookup_sigalg`), bounded arguments, transformer set/get (`ConfigParser`), macro out-params (`n2s`).
- Sources Consulted: `semantic/facts.py` loader, `improve/validator.py`, `improve/journal.py`, `calibration.py` ledger patterns, `redaction.py`.
- Findings: Facts are versioned TOML loaded by `load_facts`. The loader can merge a second directory at lower precedence. The improve loop has a strict validator and byte-for-byte revert. Redaction exists for prompts.
- Implications: Model answers are candidate facts in `.openultrasast/calibration/candidate_facts.toml`; accepted facts in `accepted_facts.toml`; loader precedence hand-written > accepted > none. The `facts` lever moves rows between candidate and accepted under the same gate. No pattern text is ever model-authored; question kinds are closed.
### Fix points
- **Context**: The principle is risk reduced per fix, not findings per file.
- **Sources Consulted**: `complexity/map.py`, `complexity/ledger.py`, `regress/verdict.py`, `scoring/project_score.py`.
- **Findings**: The path graph is small (functions, not statements). Dominators over a DAG of entry-to-sink paths are cheap. Severity is available per CWE from policy.
- **Implications**: `risk/fixpoints.py` computes dominator sets per sink path and aggregates risk removed. It is a report and artifact, not a gate; it cannot hide severity-5 reachable findings.
- Context: The principle is risk reduced per fix, not findings per file.
- Sources Consulted: `complexity/map.py`, `complexity/ledger.py`, `regress/verdict.py`, `scoring/project_score.py`.
- Findings: The path graph is small (functions, not statements). Dominators over a DAG of entry-to-sink paths are cheap. Severity is available per CWE from policy.
- Implications: `risk/fixpoints.py` computes dominator sets per sink path and aggregates risk removed. It is a report and artifact, not a gate; it cannot hide severity-5 reachable findings.
## Architecture Pattern Evaluation
@@ -65,49 +65,49 @@
### Decision: Over-approximate edges, never infer safety
- **Context**: Missing edges silently hide risk; false edges cost hunter budget.
- **Selected Approach**: Ambiguous names produce edges to every definition, flagged. Unresolved calls produce an unresolved edge. Distance is infinite only within the depth bound and is reported as `unknown`.
- **Rationale**: Consistent with the fail-closed rule in the overlay.
- **Trade-offs**: More hunter work on large trees; bounded by existing hotspot caps.
- Context: Missing edges silently hide risk; false edges cost hunter budget.
- Selected Approach: Ambiguous names produce edges to every definition, flagged. Unresolved calls produce an unresolved edge. Distance is infinite only within the depth bound and is reported as `unknown`.
- Rationale: Consistent with the fail-closed rule in the overlay.
- Trade-offs: More hunter work on large trees; bounded by existing hotspot caps.
### Decision: Summaries are pure and cached
- **Context**: Cost on large repos.
- **Selected Approach**: Summary keyed by file hash plus facts version; recomputed only when either changes.
- **Trade-offs**: Cache invalidation on facts version bump; acceptable.
- Context: Cost on large repos.
- Selected Approach: Summary keyed by file hash plus facts version; recomputed only when either changes.
- Trade-offs: Cache invalidation on facts version bump; acceptable.
### Decision: Evidence rises on checks, not on origin
- **Context**: Regex stamped corroboration, hunter stamped suspicion.
- **Selected Approach**: This spec does not change existing stamps. It adds the checks that raise hunter evidence: composed path agreement or sandbox trigger. A follow-up may lower regex-only hits to inventory; recorded as a revalidation trigger for `propose-adjudicate-prove`.
- **Rationale**: Keep this spec additive; changing existing stamps changes gate baselines.
- Context: Regex stamped corroboration, hunter stamped suspicion.
- Selected Approach: This spec does not change existing stamps. It adds the checks that raise hunter evidence: composed path agreement or sandbox trigger. A follow-up may lower regex-only hits to inventory; recorded as a revalidation trigger for `propose-adjudicate-prove`.
- Rationale: Keep this spec additive; changing existing stamps changes gate baselines.
### Decision: Closed question kinds, gated acceptance
- **Context**: Auto-facts were excluded for good reason: unbounded model-authored patterns.
- **Selected Approach**: Five question kinds, structured answers, candidate cache with provenance, acceptance only via the `facts` lever under the per-profile holdout gate.
- **Trade-offs**: Slower than trusting answers directly; the gate is the point.
- Context: Auto-facts were excluded for good reason: unbounded model-authored patterns.
- Selected Approach: Five question kinds, structured answers, candidate cache with provenance, acceptance only via the `facts` lever under the per-profile holdout gate.
- Trade-offs: Slower than trusting answers directly; the gate is the point.
### Decision: Fix points are dominators, reported not gated
- **Context**: Do not add a seventh evidence rung or a new CI condition.
- **Selected Approach**: Compute and report; the severity-5 reachable invariant from `three-stage-scan` stays.
- Context: Do not add a seventh evidence rung or a new CI condition.
- Selected Approach: Compute and report; the severity-5 reachable invariant from `three-stage-scan` stays.
## Risks & Mitigations
- Dynamic dispatch and callbacks in JS and Python produce unresolved edges — report unresolved counts; hunter `find_refs` remains available as a fallback.
- Graph blow-up on large trees — depth and fan-out bounds in config; hotspot caps unchanged.
- Model answers wrong — never used unless accepted by the gate; provenance recorded; hand facts win.
- Hunter candidates increase sandbox load — existing `max_candidates` cap; hunter candidates ranked after overlay promotions unless a path agrees.
- Fix-point estimates misread as guarantees — label as estimates; show witness paths.
- Dynamic dispatch and callbacks in JS and Python produce unresolved edges. Report unresolved counts; hunter `find_refs` remains available as a fallback.
- Graph blow-up on large trees, depth and fan-out bounds in config; hotspot caps unchanged.
- Model answers can be wrong. They are never used unless accepted by the gate; provenance recorded; hand facts win.
- Hunter candidates increase sandbox load. The existing `max_candidates` cap applies; hunter candidates ranked after overlay promotions unless a path agrees.
- Fix-point estimates misread as guarantees. Label as estimates; show witness paths.
## References
- Clearwing sourcehunt callgraph, taint, ranker (reference architecture; not a dependency)
- [RealVuln](https://arxiv.org/abs/2604.13764) — rule-based versus LLM scanners on real Python web code
- [CVE-Bench](https://arxiv.org/abs/2503.17332) — agent exploitation ceiling on real web CVEs
- `.kiro/specs/three-stage-scan/design.md` — hunter tools, candidate selection, worth-fixing
- `.kiro/specs/propose-adjudicate-prove/design.md` — facts as data, dispositions, evidence honesty
- `.kiro/specs/harnessx-self-improving-rulesets/design.md` — bounded levers, accept gate, revert
- `.kiro/specs/pair-corpus-honesty/` — per-profile holdout gate, hunter scorer
- [RealVuln](https://arxiv.org/abs/2604.13764), rule-based versus LLM scanners on real Python web code
- [CVE-Bench](https://arxiv.org/abs/2503.17332), agent exploitation ceiling on real web CVEs
- `.kiro/specs/three-stage-scan/design.md`, hunter tools, candidate selection, worth-fixing
- `.kiro/specs/propose-adjudicate-prove/design.md`, facts as data, dispositions, evidence honesty
- `.kiro/specs/harnessx-self-improving-rulesets/design.md`, bounded levers, accept gate, revert
- `.kiro/specs/pair-corpus-honesty/`, per-profile holdout gate, hunter scorer
- Memory note `overlay-gaps-diagnosis-2026-09`
+9 -9
View File
@@ -2,11 +2,11 @@
## Problem
Operators care whether the harness fires on a real buggy snapshot and stays quiet after the patch — Firefox, Chromium, OpenSSL — not whether OWASP BenchmarkTest00074 is a TP. The labeled SAST slice is 11 textbook files. Overlay Youden there is **worse** than inventory (+9.09% → 0%). We cannot tell if the tool got better on messy product code.
Operators care whether the harness fires on a real buggy snapshot and stays quiet after the patch (Firefox, Chromium, OpenSSL), not whether OWASP BenchmarkTest00074 is a TP. The labeled SAST slice is 11 textbook files. Overlay Youden there is **worse** than inventory (+9.09% → 0%). We cannot tell if the tool got better on messy product code.
## Current State
Slices today: `local` (CI 100%), `github` (6 vendored functions: SVEN, hutool, node-serialize), `sast` (OWASP/Juliet). `datasets.toml` already points at CVEfixes, MoreFixes, CWE-Bench-Java, Vul4J. Nothing is harvested from gecko-dev / Firefox. Whole mozilla-central is not in-tree (and should not be). FixFox (Zenodo) is embargoed until 2030. Firefox security commit messages are often scrubbed of “security” words, so grep-the-log is a bad miner.
Slices today: `local` (CI 100%), `github` (6 vendored functions: SVEN, hutool, node-serialize), `sast` (OWASP/Juliet). `datasets.toml` already points at CVEfixes, MoreFixes, CWE-Bench-Java, Vul4J. Nothing is harvested from gecko-dev / Firefox. Whole mozilla-central is not in-tree (and should not be). FixFox (Zenodo) is embargoed until 2030. Firefox security commit messages are often scrubbed of "security" words, so grep-the-log is a bad miner.
Measured on the current 11-pair **sast** slice (2026-09-04):
@@ -27,8 +27,8 @@ Do not clone Firefox. Filter public VFC databases for `gecko-dev` / `mozilla-fir
## Scope
- **In**: harvest script or documented procedure; catalog slice; provenance headers; overlay-vs-inventory scoreboard on that slice; dataset.toml pointers for Firefox/Chromium.
- **Out**: Vendoring mozilla-central; live Bugzilla scraping; making Youden a merge gate; buffer-size semantics; tree-sitter packaging (owned by `tree-sitter-overlay-extra`).
- In: harvest script or documented procedure; catalog slice; provenance headers; overlay-vs-inventory scoreboard on that slice; dataset.toml pointers for Firefox/Chromium.
- Out: Vendoring mozilla-central; live Bugzilla scraping; making Youden a merge gate; buffer-size semantics; tree-sitter packaging (owned by `tree-sitter-overlay-extra`).
## Boundary Candidates
@@ -41,17 +41,17 @@ Do not clone Firefox. Filter public VFC databases for `gecko-dev` / `mozilla-fir
## Upstream / Downstream
- **Upstream**: `pairs.py` catalog format; `propose-adjudicate-prove` overlay scoring
- **Downstream**: improve-loop signals from real misses; later memory-safety spec can reuse the same slice
- Upstream: `pairs.py` catalog format; `propose-adjudicate-prove` overlay scoring
- Downstream: improve-loop signals from real misses; later memory-safety spec can reuse the same slice
## Existing Spec Touchpoints
- **Extends**: pair catalog only (new slice). Does not change smoke gate.
- **Adjacent**: `tree-sitter-overlay-extra` (C/Java overlay on this slice once the extra exists)
- Extends: pair catalog only (new slice). Does not change smoke gate.
- Adjacent: `tree-sitter-overlay-extra` (C/Java overlay on this slice once the extra exists)
## Constraints
Isolated functions, same `relpath` vuln/fixed. License in the header. No 16 GB dumps in git. Firefox pairs only when a public git commit pair exists — MFSA text without a commit is not a pair.
Isolated functions, same `relpath` vuln/fixed. License in the header. No 16 GB dumps in git. Firefox pairs only when a public git commit pair exists; MFSA text without a commit is not a pair.
## Closure (2026-09-04)
+8 -8
View File
@@ -132,12 +132,12 @@ benchmarks/pairs/
### Modified Files
- `src/openultrasast/pairs.py` — concatenate vfc catalog; overlay path for `vfc`; `scorers` on the eval result; `DEFAULT_VFC_CATALOG`.
- `src/openultrasast/cli.py` — `--slice` choice `vfc`.
- `src/openultrasast/pair_gate.py` — comment that vfc is honesty-only; fail condition stays local.
- `tests/test_pair_corpus.py` — vfc load, provenance, CLI, not in `LANGUAGE_MANIFESTS`, pair_gate still passes.
- `benchmarks/pairs/datasets.toml` — project pointers.
- `benchmarks/pairs/README.md` — slice list and harvest pointer.
- `src/openultrasast/pairs.py`: concatenate vfc catalog; overlay path for `vfc`; `scorers` on the eval result; `DEFAULT_VFC_CATALOG`.
- `src/openultrasast/cli.py`: `--slice` choice `vfc`.
- `src/openultrasast/pair_gate.py`: comment that vfc is honesty-only; fail condition stays local.
- `tests/test_pair_corpus.py`: vfc load, provenance, CLI, not in `LANGUAGE_MANIFESTS`, pair_gate still passes.
- `benchmarks/pairs/datasets.toml`: project pointers.
- `benchmarks/pairs/README.md`: slice list and harvest pointer.
## System Flows
@@ -203,8 +203,8 @@ Harvest (maintainer, not CI): recipe → fetch parent blob and fix blob → extr
- Runnable set stays small; recipes may list more than CI runs.
**Dependencies**
- Inbound: pair loader — concatenate file (P0)
- Outbound: evaluate_pair — cases (P0)
- Inbound: pair loader: concatenate file (P0)
- Outbound: evaluate_pair: cases (P0)
**Contracts**: State [x]
@@ -8,9 +8,9 @@ AppSec engineers cannot tell whether OpenUltraSAST improved on real product code
## Boundary Context
- **In scope**: A new pair slice of reviewed isolated function pairs from OpenSSL, Firefox (gecko-dev / mozilla-central / mozilla-firefox/firefox), and Chromium; provenance on every vendored file; overlay-versus-inventory scoreboard on that slice; a harvest procedure that does not clone megarepos; offline runnable catalog; testability of catalog load and slice evaluation.
- **Out of scope**: Cloning mozilla-central or Chromium into the repository or CI; FixFox (embargoed until 2030); live Bugzilla scraping; making overlay or vfc Youden a merge or smoke gate; buffer-size / CWE-121 semantics; inter-file taint; parser packaging; changing pattern inventory rules so the seed pairs pass.
- **Adjacent expectations**: Existing pair evaluation still scores `local` as the 100% CI gate and `github` / `sast` as honesty dashboards. Overlay dispositions, taint, and the stage-1 detection gate stay owned by their current specs. Dataset pointers may list Firefox/Chromium/OpenSSL sources; the dumps themselves are not vendored.
- In scope: A new pair slice of reviewed isolated function pairs from OpenSSL, Firefox (gecko-dev / mozilla-central / mozilla-firefox/firefox), and Chromium; provenance on every vendored file; overlay-versus-inventory scoreboard on that slice; a harvest procedure that does not clone megarepos; offline runnable catalog; testability of catalog load and slice evaluation.
- Out of scope: Cloning mozilla-central or Chromium into the repository or CI; FixFox (embargoed until 2030); live Bugzilla scraping; making overlay or vfc Youden a merge or smoke gate; buffer-size / CWE-121 semantics; inter-file taint; parser packaging; changing pattern inventory rules so the seed pairs pass.
- Adjacent expectations: Existing pair evaluation still scores `local` as the 100% CI gate and `github` / `sast` as honesty dashboards. Overlay dispositions, taint, and the stage-1 detection gate stay owned by their current specs. Dataset pointers may list Firefox/Chromium/OpenSSL sources; the dumps themselves are not vendored.
## Requirements
+41 -41
View File
@@ -2,49 +2,49 @@
## Summary
- **Feature**: `real-world-vfc-slice`
- **Discovery Scope**: Extension of the existing pair catalog and pair eval CLI
- **Key Findings**:
- Feature: `real-world-vfc-slice`
- Discovery Scope: Extension of the existing pair catalog and pair eval CLI
- Key Findings:
- Pair eval already concatenates `benchmarks/pairs/catalog.toml` with `sast/catalog.toml`. A third file `vfc/catalog.toml` is the smallest seam.
- CLI `--slice` choices are a closed set (`all`, `local`, `github`, `sast`). `vfc` must be added there and in tests that assert slice names.
- `sast` already scores overlay with inventory fallback when nothing is promoted or demoted. `vfc` should use that path as the primary scorer and add a second inventory-only figure for the honesty dashboard.
- `pair_gate` evaluates the full catalog but fails CI only on `local` 100%. Adding vfc pairs does not change the fail condition if names stay out of `LANGUAGE_MANIFESTS`.
- Seed commits with public VCS snapshots exist: OpenSSL Heartbleed (`tls1_process_heartbeat`), Firefox libmar (`mar_insert_item`, CVE-2020-15667), Chromium FileReader (`ArrayBufferResult`, CVE-2019-5786). All three keep a sink-shaped call after the fix, so overlay/inventory may LEAK — that is the dashboard, not a 100% gate.
- Seed commits with public VCS snapshots exist: OpenSSL Heartbleed (`tls1_process_heartbeat`), Firefox libmar (`mar_insert_item`, CVE-2020-15667), Chromium FileReader (`ArrayBufferResult`, CVE-2019-5786). All three keep a sink-shaped call after the fix, so overlay/inventory may LEAK; that is the dashboard, not a 100% gate.
- mozilla-central / gecko-dev must not be cloned. Phabricator and GitHub raw blobs are enough to extract one function. FixFox stays embargoed.
## Research Log
### Existing pair catalog and scoring
- **Context**: Brownfield extension; must not invent a second eval engine.
- **Sources Consulted**: `src/openultrasast/pairs.py`, `cli.py`, `pair_gate.py`, `tests/test_pair_corpus.py`, `benchmarks/pairs/catalog.toml`, `benchmarks/pairs/sast/catalog.toml`, `benchmarks/pairs/README.md`
- **Findings**:
- Context: Brownfield extension; must not invent a second eval engine.
- Sources Consulted: `src/openultrasast/pairs.py`, `cli.py`, `pair_gate.py`, `tests/test_pair_corpus.py`, `benchmarks/pairs/catalog.toml`, `benchmarks/pairs/sast/catalog.toml`, `benchmarks/pairs/README.md`
- Findings:
- `PairCase.slice` is a free string in the catalog loader; the closed set lives only in argparse and tests.
- Overlay pair eval is gated on `case.slice == "sast"`.
- GitHub pairs already use provenance comments plus catalog fields (`repo`, `commit`, `commit_url`, `cve`, `license`).
- Pair names must not appear in `LANGUAGE_MANIFESTS`.
- **Implications**: Add `vfc` as a concatenated catalog; extend the overlay branch to `sast` **or** `vfc`; keep the gate local-only.
- Implications: Add `vfc` as a concatenated catalog; extend the overlay branch to `sast` **or** `vfc`; keep the gate local-only.
### Harvestable public commits (OpenSSL, Firefox, Chromium)
- **Context**: User asked for those three projects and a labeled corpus that can actually be tested.
- **Sources Consulted**:
- Context: User asked for those three projects and a labeled corpus that can actually be tested.
- Sources Consulted:
- OpenSSL `96db9023b881d7cd9f379b0c154650d6c108e9a3` (parent `0d7717fc9c83dafab8153cbd5e2180e6e04cc802`), `ssl/t1_lib.c` `tls1_process_heartbeat`, CVE-2014-0160
- Firefox hg `b79b6cc78248eea7fda10bfb76aa273c19c9fa65` (parent `0c0f777161a9499dd149853ff62d356f75d16c2a`), `modules/libmar/src/mar_read.c` `mar_insert_item`, CVE-2020-15667 / Bug 1653371; Phabricator raw diff
- Chromium `ba9748e78ec7e9c0d594e7edf7b2c07ea2a90449` (parent `9f3fccdba567f30ea39b08f57d8d5b49cd832b4a`), `FileReaderLoader::ArrayBufferResult`, CVE-2019-5786
- Rejected: CVE-2019-11730 (SOP/file URI policy, not a sink pair); FixFox (embargoed to 2030); MFSA text without a commit
- **Findings**:
- Findings:
- Heartbleed fix adds a record-length check; `memcpy(bp, pl, payload)` remains. Inventory `c-unsafe-memory-copy` will fire on both sides (LEAK) until CWE-121 semantics exist.
- Firefox fix is `int namelen` → `uint32_t namelen`; `memcpy(item->name, name, namelen + 1)` remains. Same LEAK class.
- Chromium fix copies the ArrayBuffer on partial results (UAF). No memcpy/eval sink; expected MISS on inventory and overlay. Still a labeled pair.
- GitHub raw and Phabricator `?diff=1` return the function text without cloning.
- **Implications**: Seed three reviewed pairs. Tests assert presence and that the slice **runs**, not that pair_correct is 100%. Memory-safety pairs are allowed as known leaks (Req 6.3).
- Implications: Seed three reviewed pairs. Tests assert presence and that the slice **runs**, not that pair_correct is 100%. Memory-safety pairs are allowed as known leaks (Req 6.3).
### Dataset pointers versus vendored dumps
- **Context**: `datasets.toml` already lists CVEfixes, MoreFixes, SVEN, CWE-Bench-Java. Roadmap asked for Firefox/Chromium/OpenSSL pointers.
- **Findings**: Pointers are metadata only. CI must stay offline on vendored excerpts.
- **Implications**: Add dataset rows for OpenSSL, gecko-dev / mozilla-firefox/firefox, and Chromium. Do not fetch dumps in tests.
- Context: `datasets.toml` already lists CVEfixes, MoreFixes, SVEN, CWE-Bench-Java. Roadmap asked for Firefox/Chromium/OpenSSL pointers.
- Findings: Pointers are metadata only. CI must stay offline on vendored excerpts.
- Implications: Add dataset rows for OpenSSL, gecko-dev / mozilla-firefox/firefox, and Chromium. Do not fetch dumps in tests.
## Architecture Pattern Evaluation
@@ -59,48 +59,48 @@
### Decision: Concatenated `vfc` catalog, overlay primary, inventory sidecar
- **Context**: Need a labeled corpus that is offline, isolated, and honest about overlay versus regex.
- **Alternatives Considered**:
1. Inventory-only like `github` — misses the overlay question the user is evolving.
2. Overlay-only like `sast` without a sidecar — cannot see whether overlay helped or hurt.
3. Dual scoreboard with overlay primary and inventory sidecar — **selected**.
- **Selected Approach**: `slice = "vfc"` uses the overlay pair path (inventory fallback when unadjudicated). `PairEvalResult.scorers["vfc"]` holds both `overlay` and `inventory` metrics. Pair gate still fails only on `local`.
- **Rationale**: Reuses `sast` overlay machinery; satisfies Req 4 without making Youden a merge gate.
- **Trade-offs**: Pair gate runtime grows by three small overlay scans. Acceptable.
- **Follow-up**: Do not retune memcpy rules to force these pairs to PASS.
- Context: Need a labeled corpus that is offline, isolated, and honest about overlay versus regex.
- Alternatives Considered:
1. Inventory-only like `github`: misses the overlay question the user is evolving.
2. Overlay-only like `sast` without a sidecar: cannot see whether overlay helped or hurt.
3. Dual scoreboard with overlay primary and inventory sidecar: **selected**.
- Selected Approach: `slice = "vfc"` uses the overlay pair path (inventory fallback when unadjudicated). `PairEvalResult.scorers["vfc"]` holds both `overlay` and `inventory` metrics. Pair gate still fails only on `local`.
- Rationale: Reuses `sast` overlay machinery; satisfies Req 4 without making Youden a merge gate.
- Trade-offs: Pair gate runtime grows by three small overlay scans. Acceptable.
- Follow-up: Do not retune memcpy rules to force these pairs to PASS.
### Decision: Maintainer harvest script, vendored seed, no CI fetch
- **Context**: Grow the catalog later without cloning mozilla-central.
- **Selected Approach**: `benchmarks/pairs/vfc/harvest.py` plus `recipes.toml`. CI never runs harvest. Seed files are committed excerpts with provenance headers.
- **Rationale**: Req 5 and Req 7 (offline testability).
- **Trade-offs**: Harvest needs network when a maintainer grows the set; tests mock extraction locally.
- Context: Grow the catalog later without cloning mozilla-central.
- Selected Approach: `benchmarks/pairs/vfc/harvest.py` plus `recipes.toml`. CI never runs harvest. Seed files are committed excerpts with provenance headers.
- Rationale: Req 5 and Req 7 (offline testability).
- Trade-offs: Harvest needs network when a maintainer grows the set; tests mock extraction locally.
### Decision: Public hg counts as a version-control commit
- **Context**: Firefox canonical history is hg.mozilla.org; gecko-dev git SHAs are not the hg SHAs; GitHub 422s on hg identifiers.
- **Selected Approach**: Record the hg revision and Phabricator/hg URL. Dataset pointers still list gecko-dev and mozilla-firefox/firefox.
- **Rationale**: Brief forbids MFSA-without-commit, not hg. The MAR fix is a public VCS snapshot.
- **Follow-up**: If a git SHA is later mapped, add it as an extra catalog field without changing the excerpt.
- Context: Firefox canonical history is hg.mozilla.org; gecko-dev git SHAs are not the hg SHAs; GitHub 422s on hg identifiers.
- Selected Approach: Record the hg revision and Phabricator/hg URL. Dataset pointers still list gecko-dev and mozilla-firefox/firefox.
- Rationale: Brief forbids MFSA-without-commit, not hg. The MAR fix is a public VCS snapshot.
- Follow-up: If a git SHA is later mapped, add it as an extra catalog field without changing the excerpt.
## Synthesis
- **Generalization**: `vfc` is another honesty slice, not a new product. Overlay-versus-inventory is a report field, not a new engine.
- **Build vs adopt**: Adopt pair catalog schema, overlay eval, provenance comment style. Build only harvest extract + dual scoreboard payload.
- **Simplification**: No Joern, no megarepo, no 90% gate, no rule changes to chase Youden on three memory/UAF pairs.
- Generalization: `vfc` is another honesty slice, not a new product. Overlay-versus-inventory is a report field, not a new engine.
- Build vs adopt: Adopt pair catalog schema, overlay eval, provenance comment style. Build only harvest extract + dual scoreboard payload.
- Simplification: No Joern, no megarepo, no 90% gate, no rule changes to chase Youden on three memory/UAF pairs.
## Risks & Mitigations
- Overlay/inventory LEAK on Heartbleed and MAR memcpy-after-fix — document as known; tests must not assert PASS.
- Chromium UAF is a MISS — still required so the corpus is not only memcpy.
- Overlay/inventory LEAK on Heartbleed and MAR memcpy-after-fix; document as known; tests must not assert PASS.
- Chromium UAF is a MISS; still required so the corpus includes more than memcpy.
- `pair_gate` currently scans the full catalog; three extra pairs are small. If the catalog grows, gate should keep evaluating local-only for the fail condition (already true) and may later skip overlay slices for speed.
- License: OpenSSL (OpenSSL License / SSLeay), Firefox MPL-2.0, Chromium BSD-3-Clause — headers must state them.
- License: OpenSSL (OpenSSL License / SSLeay), Firefox MPL-2.0, Chromium BSD-3-Clause; headers must state them.
- gecko-dev is archived; harvest should accept hg.mozilla.org and github.com/chromium/chromium / openssl/openssl as hosts.
## References
- [OpenSSL Heartbleed fix](https://github.com/openssl/openssl/commit/96db9023b881d7cd9f379b0c154650d6c108e9a3) — CVE-2014-0160
- [Firefox MAR heap overflow](https://bugzilla.mozilla.org/show_bug.cgi?id=1653371) — CVE-2020-15667, hg `b79b6cc78248`
- [Chromium FileReader UAF](https://github.com/chromium/chromium/commit/ba9748e78ec7e9c0d594e7edf7b2c07ea2a90449) — CVE-2019-5786
- [OpenSSL Heartbleed fix](https://github.com/openssl/openssl/commit/96db9023b881d7cd9f379b0c154650d6c108e9a3): CVE-2014-0160
- [Firefox MAR heap overflow](https://bugzilla.mozilla.org/show_bug.cgi?id=1653371): CVE-2020-15667, hg `b79b6cc78248`
- [Chromium FileReader UAF](https://github.com/chromium/chromium/commit/ba9748e78ec7e9c0d594e7edf7b2c07ea2a90449): CVE-2019-5786
- `.kiro/specs/real-world-vfc-slice/brief.md`
- `.kiro/steering/roadmap.md`
+17 -17
View File
@@ -1,32 +1,32 @@
# Search-with-proof — handoff (2026-10-06)
# Search-with-proof, handoff (2026-10-06)
State at handoff, for resuming on the search problem and abstract interpretation.
## RESOLVED 2026-10-07 — the one open question is CLOSED: first cluster demonstration achieved
## RESOLVED 2026-10-07, the one open question is CLOSED: first cluster demonstration achieved
The "first `demonstrated`" question below is answered. search-with-proof produced its first autonomous
`demonstrated` finding end to end on kube-ax, and a blind negative confirmed no false proof.
- **Pair:** hanxi/xiaomusic (pilot pair 13), family `path`, vulnerability class **prefix-check bypass**
- Pair: hanxi/xiaomusic (pilot pair 13), family `path`, vulnerability class **prefix-check bypass**
(`startswith(root)` without a trailing separator; the fix adds `os.sep`). Image `e2c8b460`.
- **Vulnerable side:** `demonstrated` — owned effect in 3/3 affected runs, 0/3 safe runs. $0.37, 55 calls.
- **Fixed side (blind negative, Req 6.3):** `inconclusive`, **0 false proofs**. $0.56, 85 calls.
- **Pair total:** $1.22. Record: `benchmarks/measurements/2026-10-07-search-first-cluster-demonstration/`.
- Vulnerable side: `demonstrated`, owned effect in 3/3 affected runs, 0/3 safe runs. $0.37, 55 calls.
- Fixed side (blind negative, Req 6.3): `inconclusive`, **0 false proofs**. $0.56, 85 calls.
- Pair total: $1.22. Record: `benchmarks/measurements/2026-10-07-search-first-cluster-demonstration/`.
**The original pair-6 both-sides-false was a TWO-part diagnosis, both now fixed:** the demo did not reach the
observable (demo/agent), and — the decisive one — the path oracle only modelled **classic parent traversal** while
observable (demo/agent), and (the decisive one) the path oracle only modelled **classic parent traversal** while
many real path CVEs (xiaomusic included) are **prefix-bypass**, which reaches a sibling that string-extends the served
dir, not the parent. The oracle now also plants a prefix-sibling canary.
**Fix chain that enabled the demonstration (each merged to main and validated by a paid run):**
1. Parity — agent is told the observable's location (path oracle reached parity with command/ssrf).
2. Verify-time-only — the observable is injected at verify time and absent during exploration; stop the explore-time
1. Parity. The agent is told the observable's location (path oracle reached parity with command/ssrf).
2. Verify-time-only. The observable is injected at verify time and absent during exploration; stop the explore-time
reachability trap that burned budget without authoring a demo.
3. Fair-share pilot budget — each search gets a fair share of `--ceiling-usd` with rollover, so one expensive search
3. Fair-share pilot budget. Each search gets a fair share of `--ceiling-usd` with rollover, so one expensive search
no longer starves the rest.
4. Served-root guidance — the demo MUST set the app's served directory to the `${served_root}` placeholder.
5. Oracle prefix-bypass coverage — plant a sibling canary for the `startswith`-without-separator class.
6. Control-signal feedback — the keystone control (does the app serve `${served_root}`?) is fed back to the agent as a
4. Served-root guidance. The demo MUST set the app's served directory to the `${served_root}` placeholder.
5. Oracle prefix-bypass coverage. Plant a sibling canary for the `startswith`-without-separator class.
6. Control-signal feedback. The keystone control (does the app serve `${served_root}`?) is fed back to the agent as a
`served-root` hint so it self-corrects its wiring. This was the decisive last step.
**Keystone (honest measurement), merged earlier:** the verifier runs a benign positive control; a broken or mis-wired
@@ -38,7 +38,7 @@ dependency build-break (`filed`→`mime@4`) is pinned.
---
## The one open question (HISTORICAL — resolved above)
## The one open question (HISTORICAL, resolved above)
The full chain runs end to end on kube-ax. Every execution layer is cleared. The **first `demonstrated` has not
happened yet**, and the remaining gap is narrow and reproducible at **$0 model cost** via `demo_replay`.
@@ -59,7 +59,7 @@ Both-sides-false means it is **not the harness** any more. It is one of:
capture in `oracles.py` `PathOracle.observe`).
- Re-run with `python -m benchmarks.search.demo_replay --both --demo <stored demo.json> --family path --lane ax
--image <digest> ...`. Stored demos live under `benchmarks/search/private/runs/*/<pair>/demo/demo.json` (gitignored).
- **Classifier note:** keep exploit payloads on disk. Hand Codex file-path tasks that treat demo contents as opaque;
- Classifier note: keep exploit payloads on disk. Hand Codex file-path tasks that treat demo contents as opaque;
print outcomes / phase timings / error *classes*, never request strings, env values, or step bodies. See memory
`search-with-proof-direction` and the classifier false-positive note. Code changes never tripped it; echoing
payloads or step-by-step attack prose did.
@@ -95,9 +95,9 @@ image (pip 3.13 vs venv 3.12 → use `sys.executable`) → checkout root on sys.
- `/workspace` ~3 GiB free per node until the volume move; scratch guard 2 GiB/task.
- Pilot spend to date ~$11 across 9 paid runs; the decisive replay was $0.
## Decision model vs search — current read
## Decision model vs search (current read)
- Decision/classification: measured, **promising as triage** (within-pair AUC 0.762; trained logistic ties at 0.721),
**capped as a blocker** (cannot prove 95% precision). Reframed as triage (`learned-decision-engine` Req 6.5).
- Search-with-proof: **not yet demonstrated**, but no observation so far argues against the idea — every failure was
- Search-with-proof: **not yet demonstrated**, but no observation so far argues against the idea. Every failure was
our execution, now cleared. The pair-6 demo/oracle question is the first verdict on the method itself.
+3 -3
View File
@@ -16,13 +16,13 @@ each confirm one fact; the loop ends when the goal is demonstrably met.
Three things follow for OpenUltraSAST:
1. **An objective success condition.** A finding is demonstrated when a test or proof of concept fails on the
1. An objective success condition. A finding is demonstrated when a test or proof of concept fails on the
vulnerable code and passes on the fixed code (or, for a push, fails on head and passes on base where applicable).
A demonstrated finding is right by construction; it needs no calibration and no training data, so it generalises
to unseen repositories by construction.
2. **Exploration with tools.** Agents that read other files, follow callers, build and run the code, instead of a
2. Exploration with tools. Agents that read other files, follow callers, build and run the code, instead of a
single-shot judgement of one excerpt.
3. **Search state per repository and push.** Facts and intents accumulate on a board instead of every question
3. Search state per repository and push. Facts and intents accumulate on a board instead of every question
starting from zero.
## Shape
@@ -73,7 +73,7 @@ Other files already modified/untracked at session start remain uncommitted.
- Requested suite, with `-rs` for skip reasons:
`/home/mc/Source/OpenUltraSAST/.venv/bin/pytest -q tests/test_search_*.py tests/test_ax_batch.py tests/test_plane_literals.py -rs`
— **181 passed, 3 skipped, exit 0**, 24.94 seconds.
Result: **181 passed, 3 skipped, exit 0**, 24.94 seconds.
- Final executor/storage regression and module audit tests after tightening stderr
truncation: **34 passed, exit 0**, including all **6 module audit tests**.
- Shared dispatcher consumer `tests/test_unseen_calibrate.py`: **10 passed, exit 0**.
@@ -9,12 +9,12 @@ kube-ax cluster. BLOCK becomes "demonstrated", not "scored above a threshold". C
## Boundary Context
- **In scope**: the board (facts, intents, hints) in the memory store; reason and explore steps as AX Tasks; the
- In scope: the board (facts, intents, hints) in the memory store; reason and explore steps as AX Tasks; the
demonstration contract and its verification; triage-to-search budgeting; a pilot on known fix pairs; a G1 arm on the
unseen pool; model access through an operator-managed scoped key.
- **Out of scope**: copying Cairn code (AGPL-3.0); network actions against anything but the code under analysis;
- Out of scope: copying Cairn code (AGPL-3.0); network actions against anything but the code under analysis;
changing the existing quick rules, engine or classifier (they are inputs here).
- **Adjacent expectations**: `unseen-repo-evaluation` scores this arm; the AX batch dispatcher (`benchmarks/ax/batch.py`)
- Adjacent expectations: `unseen-repo-evaluation` scores this arm; the AX batch dispatcher (`benchmarks/ax/batch.py`)
and the kube-ax constraints (2 heavy lanes, `ousast-engine-` names, egress to public HTTPS) apply;
`learned-decision-engine` Req 6.4 BLOCK rule is superseded for this arm by "demonstrated".
@@ -13,7 +13,7 @@ directed state-space **search**, then a DAST-style **proof (verification)**.
between the vulnerable and fixed revisions: the owned effect appears on the vulnerable side and
not on the fixed side (3/3 vs 0/3). Only this differential earns `demonstrated`.
## Trust boundary — what runs where
## Trust boundary (what runs where)
The brain runs on the VM; the agent's tool executions and the DAST proof run as isolated tasks in
the kube-ax cluster. Secrets never enter the cluster.
+10 -10
View File
@@ -19,9 +19,9 @@ AppSec engineers and DevOps teams running OpenUltraSAST in CI get a regex static
A scan answers three operator questions, in order:
1. **What exists?** A static inventory of files, surfaces, and pattern/SARIF hits — honest about being inventory, not verified bugs.
2. **Where are problems most likely, and how do I tune tests?** A complexity map of hotspots plus test-gap hints a DevOps team can act on.
3. **Is this worth fixing?** Isolated regression against the top hotspots. Triggerable and reachable → worth a CI fail. Not triggerable → do not fail the build.
1. What exists? A static inventory of files, surfaces, and pattern/SARIF hits, honest about being inventory, not verified bugs.
2. Where are problems most likely, and how do I tune tests? A complexity map of hotspots plus test-gap hints a DevOps team can act on.
3. Is this worth fixing? Isolated regression against the top hotspots. Triggerable and reachable → worth a CI fail. Not triggerable → do not fail the build.
Stages 2–3 use a tool-calling hunter (read / grep / callers) and a Docker-isolated runner. Evaluation uses split-sink and mutation corpora, not cheat sheets. Missing Docker or a model degrades visibly; stage 1 still runs.
@@ -37,12 +37,12 @@ Do not require the HarnessX extra for the LLM path. Use the existing stdlib Open
Do not add a Python Docker SDK. Drive the Docker CLI via subprocess with the threat-model flags (no network, read-only source, tmpfs scratch, memory/pids/timeout, cap-drop). Core `dependencies = []` stays.
Complexity map is a new artifact built from existing ranking, entry points, finding density, nesting/size, and test-file presence — not a rewrite of `RankingScore`. Stage 3 writes a worth-fixing verdict that feeds back into a complexity ledger (evolutionary), without editing detection pattern text.
Complexity map is a new artifact built from existing ranking, entry points, finding density, nesting/size, and test-file presence, not a rewrite of `RankingScore`. Stage 3 writes a worth-fixing verdict that feeds back into a complexity ledger (evolutionary), without editing detection pattern text.
## Scope
- **In**: three-stage scan contract; complexity map + test-tuning hints; Docker-isolated regression and worth-fixing verdicts; tool-calling hunter on hotspots; split-sink/mutation evaluation; CI mapping (PR = stage 1, nightly = stage 2/3); visible degradation; evolutionary feedback from regress → map.
- **Out**: rewriting CWE policy or project-score formula; auto-authoring regex patterns; full fuzzing campaigns; patch generation/application; network pentest; importing Clearwing.
- In: three-stage scan contract; complexity map + test-tuning hints; Docker-isolated regression and worth-fixing verdicts; tool-calling hunter on hotspots; split-sink/mutation evaluation; CI mapping (PR = stage 1, nightly = stage 2/3); visible degradation; evolutionary feedback from regress → map.
- Out: rewriting CWE policy or project-score formula; auto-authoring regex patterns; full fuzzing campaigns; patch generation/application; network pentest; importing Clearwing.
## Boundary Candidates
@@ -61,13 +61,13 @@ Complexity map is a new artifact built from existing ranking, entry points, find
## Upstream / Downstream
- **Upstream**: existing preprocess, rank, mapping, findings, verification, config, CLI modes, OpenRouter client, threat-model sandbox limits.
- **Downstream**: CI fail-on `worth-fixing`; later fuzz/patch specs; MCP scan status.
- Upstream: existing preprocess, rank, mapping, findings, verification, config, CLI modes, OpenRouter client, threat-model sandbox limits.
- Downstream: CI fail-on `worth-fixing`; later fuzz/patch specs; MCP scan status.
## Existing Spec Touchpoints
- **Extends**: `openrouter-sast-harness` gates `standard_security_harness` and `sandboxed_dynamic_harness` (still unsatisfied).
- **Adjacent**: `harnessx-self-improving-rulesets` owns CWE policy, score, shadow, optional HarnessX extra. This spec must not reopen those.
- Extends: `openrouter-sast-harness` gates `standard_security_harness` and `sandboxed_dynamic_harness` (still unsatisfied).
- Adjacent: `harnessx-self-improving-rulesets` owns CWE policy, score, shadow, optional HarnessX extra. This spec must not reopen those.
## Constraints
+16 -16
View File
@@ -32,13 +32,13 @@
### This Spec Owns
- **Stage plan**: which stages a mode runs, skip/degrade accounting, `stages_requested` / `stages_completed` in the manifest.
- **Complexity map**: `Hotspot` records, signals, test-gap hints, `complexity_map.json`.
- **Complexity ledger**: overlay of hotspot scores from regression verdicts.
- **Sandbox**: capability probe and Docker CLI runner that never mounts the host Docker socket.
- **Regression / worth-fixing**: candidate selection, language recipes, verdicts, `--fail-on worth-fixing`.
- **Tool hunter**: repo-bound read / grep / callers loop for stages 2–3, evidence honesty.
- **Split-sink evaluation**: corpora and the "beats pattern-hit ranking" gate.
- Stage plan: which stages a mode runs, skip/degrade accounting, `stages_requested` / `stages_completed` in the manifest.
- Complexity map: `Hotspot` records, signals, test-gap hints, `complexity_map.json`.
- Complexity ledger: overlay of hotspot scores from regression verdicts.
- Sandbox: capability probe and Docker CLI runner that never mounts the host Docker socket.
- Regression / worth-fixing: candidate selection, language recipes, verdicts, `--fail-on worth-fixing`.
- Tool hunter: repo-bound read / grep / callers loop for stages 2–3, evidence honesty.
- Split-sink evaluation: corpora and the "beats pattern-hit ranking" gate.
### Out of Boundary
@@ -167,14 +167,14 @@ benchmarks/manifests/split-sink-*.toml
### Modified Files
- `cli.py` — run stage 2/3; stop exiting on `deep`; `--fail-on worth-fixing`; write new artifacts.
- `reports.py` — `scan_exit_code` grows `worth-fixing`; report sections for inventory, map, and worth-fixing.
- `config.py` — `ComplexityConfig` (top_k, max_hunter_hotspots) and `RegressConfig` (max_candidates, image pins); **consume** existing `SandboxConfig` (do not add a second sandbox TOML).
- `hunter.py` — schedule remains; execution delegates to `tool_hunter` when a model is set, else inventory-only.
- `verification.py` / reports — inventory vs worth-fixing labeling; hunter findings stay `suspicion`.
- `gate.py` — stage-1 smoke gate unchanged; add split-sink map gate.
- `reports.py` — map and verdict sections in markdown/manifest.
- `.github/workflows/ci.yml` — PR stays quick; optional nightly deep when Docker is available.
- `cli.py`, run stage 2/3; stop exiting on `deep`; `--fail-on worth-fixing`; write new artifacts.
- `reports.py`, `scan_exit_code` grows `worth-fixing`; report sections for inventory, map, and worth-fixing.
- `config.py`, `ComplexityConfig` (top_k, max_hunter_hotspots) and `RegressConfig` (max_candidates, image pins); **consume** existing `SandboxConfig` (do not add a second sandbox TOML).
- `hunter.py`, schedule remains; execution delegates to `tool_hunter` when a model is set, else inventory-only.
- `verification.py` / reports, inventory vs worth-fixing labeling; hunter findings stay `suspicion`.
- `gate.py`, stage-1 smoke gate unchanged; add split-sink map gate.
- `reports.py`, map and verdict sections in markdown/manifest.
- `.github/workflows/ci.yml`, PR stays quick; optional nightly deep when Docker is available.
## System Flows
@@ -468,7 +468,7 @@ Degradations reuse the existing list: `{ "stage": "regress", "reason": "sandbox_
- Mode→stage mapping and skip accounting.
- Path clamp rejects `../`.
- Map order on a fixture where regex hits a trivial helper and the real bug is a split-sink parser — parser ranks higher.
- Map order on a fixture where regex hits a trivial helper and the real bug is a split-sink parser, parser ranks higher.
- Covered hotspot yields `gap=covered` and no test kind.
- Safety check rejects docker.sock and network snippets.
- Ledger boost/demote; sev-5 reachable remains a candidate.
+3 -3
View File
@@ -13,9 +13,9 @@ The work is a follow-up to the HarnessX ruleset-governance spec, not a replaceme
## Boundary Context
- **In scope**: a three-stage scan contract mapped onto existing scan depths; a static inventory that reports what exists without claiming verification; a complexity map of hotspots with test-tuning hints; isolated regression that produces a worth-fixing verdict; a tool-using hunter confined to stages 2–3; evaluation against split-sink and mutation corpora; visible degradation when the sandbox or a model is unavailable; evolutionary feedback from regression verdicts back into the complexity map.
- **Out of scope**: rewriting CWE-to-severity policy or the project-score formula; authoring or rewriting detection pattern text; full fuzzing campaigns; generating or applying patches; network pentesting; importing Clearwing.
- **Adjacent expectations**: stage 1 continues to run in the zero-dependency default install with no model and no sandbox. CWE policy, scoring, traces, and SARIF remain owned by the existing governance spec. The operator supplies an isolated-sandbox runtime (Docker CLI) for stage 3 and an LLM credential for the tool-using hunter; neither is a packaged library dependency of the core install.
- In scope: a three-stage scan contract mapped onto existing scan depths; a static inventory that reports what exists without claiming verification; a complexity map of hotspots with test-tuning hints; isolated regression that produces a worth-fixing verdict; a tool-using hunter confined to stages 2–3; evaluation against split-sink and mutation corpora; visible degradation when the sandbox or a model is unavailable; evolutionary feedback from regression verdicts back into the complexity map.
- Out of scope: rewriting CWE-to-severity policy or the project-score formula; authoring or rewriting detection pattern text; full fuzzing campaigns; generating or applying patches; network pentesting; importing Clearwing.
- Adjacent expectations: stage 1 continues to run in the zero-dependency default install with no model and no sandbox. CWE policy, scoring, traces, and SARIF remain owned by the existing governance spec. The operator supplies an isolated-sandbox runtime (Docker CLI) for stage 3 and an LLM credential for the tool-using hunter; neither is a packaged library dependency of the core install.
## Requirements
+72 -72
View File
@@ -5,9 +5,9 @@
---
## Summary
- **Feature**: `three-stage-scan`
- **Discovery Scope**: Extension of the existing scan harness (brownfield), with one new runtime prerequisite (Docker CLI for stage 3) and one new evaluation contract (split-sink / mutation corpora).
- **Key Findings**:
- Feature: `three-stage-scan`
- Discovery Scope: Extension of the existing scan harness (brownfield), with one new runtime prerequisite (Docker CLI for stage 3) and one new evaluation contract (split-sink / mutation corpora).
- Key Findings:
- Stage 1 already exists as `quick` mode: preprocess, mapping, rank, `quick_scan_findings`, structural verify, score, SARIF. Do not rebuild it.
- Stage 2 is not a hunter dump and is not `RankingScore`. Ranking is per-file attention. A complexity map is per-file-and-function hotspot attention plus test-gap hints. Existing signals (tags, loc, reachability hints, finding density, `has_fuzz_entry_point`) are enough for a heuristic map; the tool-using hunter refines it.
- Stage 3 has config and a threat-model table, but `cli.py` exits on `--mode deep`. No container is ever started. The first execution of untrusted code must be Docker CLI with the documented isolation flags, not a Python SDK and not host `subprocess` of target code.
@@ -17,69 +17,69 @@
## Research Log
### Existing scan pipeline and mode mapping
- **Context**: Whether to add `--stage` or reuse `--mode quick|standard|deep`.
- **Sources Consulted**: `src/openultrasast/cli.py` (`_run_scan`, mode choices), `README.md` maturity legend, `.kiro/specs/openrouter-sast-harness/requirements.md` (hunter pool, docker sandbox, dynamic analysis), `.kiro/specs/harnessx-self-improving-rulesets/design.md`.
- **Findings**:
- Context: Whether to add `--stage` or reuse `--mode quick|standard|deep`.
- Sources Consulted: `src/openultrasast/cli.py` (`_run_scan`, mode choices), `README.md` maturity legend, `.kiro/specs/openrouter-sast-harness/requirements.md` (hunter pool, docker sandbox, dynamic analysis), `.kiro/specs/harnessx-self-improving-rulesets/design.md`.
- Findings:
- `quick` is implemented and deterministic. `standard` schedules a hunter pool that calls `quick_scan_findings` unless HarnessX extra + hunter model are set. `deep` raises `SystemExit`.
- OpenRouter spec already defined: quick = static + bounded LLM review; standard = sandboxed tool-calling hunters; deep = harness/fuzz/dynamic. Implementation never reached that.
- Existing CI and README teach `--mode`. A parallel `--stage` flag would fork the contract.
- **Implications**: Map modes onto stages 1 / 1+2 / 1+2+3. Record `stages_requested` and `stages_completed` in the manifest so the three questions are explicit even if the flag names stay.
- Implications: Map modes onto stages 1 / 1+2 / 1+2+3. Record `stages_requested` and `stages_completed` in the manifest so the three questions are explicit even if the flag names stay.
### Ranking vs complexity map
- **Context**: `rank.py` already scores files. Is stage 2 just ranking?
- **Sources Consulted**: `src/openultrasast/rank.py` (`heuristic_rank`, `composite_priority` = surface 0.5 + influence 0.2 + reachability 0.3), `preprocess.py` (`FileTarget` tags, loc, `has_fuzz_entry_point`), `mapping.py` (`analyze_entry_points`).
- **Findings**:
- Context: `rank.py` already scores files. Is stage 2 just ranking?
- Sources Consulted: `src/openultrasast/rank.py` (`heuristic_rank`, `composite_priority` = surface 0.5 + influence 0.2 + reachability 0.3), `preprocess.py` (`FileTarget` tags, loc, `has_fuzz_entry_point`), `mapping.py` (`analyze_entry_points`).
- Findings:
- Ranking is file-level and exists to budget the hunter. It does not name functions, test gaps, or "tune your tests."
- Tags already encode likely problem classes (`memory_unsafe`, `parser`, `network_entry`, `syscall_entry`). Loc ≥ 500 boosts influence. No nesting/branch signal. No test-file presence.
- A hotspot that stage 1 missed (split-sink) can still rank high if the file is a reachable parser. That is the point of stage 2.
- **Implications**: Keep `RankingScore`. Add `ComplexityMap` / `Hotspot` with function identity when recoverable, test-gap hints, and a ledger overlay. Ranking still feeds hunter budgets; the map feeds humans and stage 3.
- Implications: Keep `RankingScore`. Add `ComplexityMap` / `Hotspot` with function identity when recoverable, test-gap hints, and a ledger overlay. Ranking still feeds hunter budgets; the map feeds humans and stage 3.
### Hunter: regex, HarnessX dump, or tools
- **Context**: Previous spec adopted HarnessX "at the hunter loop" but the default hunter is regex and the HarnessX path dumps source into a prompt.
- **Sources Consulted**: `hunter.py` (`_run_hunter_task` → `quick_scan_findings`), `hunter_harness.py` (`_describe`, `_extract_findings`), `provider/openrouter.py` (`OpenRouterChatClient.complete_json`, stdlib `urllib`), `harness_ext.py` (optional extra).
- **Findings**:
- Context: Previous spec adopted HarnessX "at the hunter loop" but the default hunter is regex and the HarnessX path dumps source into a prompt.
- Sources Consulted: `hunter.py` (`_run_hunter_task` → `quick_scan_findings`), `hunter_harness.py` (`_describe`, `_extract_findings`), `provider/openrouter.py` (`OpenRouterChatClient.complete_json`, stdlib `urllib`), `harness_ext.py` (optional extra).
- Findings:
- OpenRouter client is already zero-extra (stdlib). It is used for optional ranking, not for hunting.
- HarnessX extra is SHA-pinned, heavy, absent in CI. Requiring it repeats the last spec's failure mode.
- Clearwing hunters use `read_source_file`, `grep_source`, `find_callers` with path clamping. That is the evolutionary piece, not MetaAgent regex-status flips.
- **Implications**: Stage 2/3 hunter is a bounded tool loop over the OpenRouter client (or HarnessX if already wired). Tools are stdlib (read, Python grep, caller search from mapping + text). Path clamp at repo root. Hunter output evidence = `suspicion`.
- Implications: Stage 2/3 hunter is a bounded tool loop over the OpenRouter client (or HarnessX if already wired). Tools are stdlib (read, Python grep, caller search from mapping + text). Path clamp at repo root. Hunter output evidence = `suspicion`.
### Docker sandbox: CLI vs SDK vs skip
- **Context**: Threat model already lists sandbox limits. Need a runner that does not invert zero-dep posture.
- **Sources Consulted**: `docs/threat-model.md` (sandbox table), `config.py` `SandboxConfig`, Clearwing `hunter_sandbox.py` (image build + spawn, Docker SDK), Docker isolation guidance (`docker run --network none --read-only --tmpfs --memory --pids-limit --cap-drop ALL --security-opt no-new-privileges`), initrunner docker-sandbox docs (CLI `docker run --rm --init`, optional `runsc`).
- **Findings**:
- Context: Threat model already lists sandbox limits. Need a runner that does not invert zero-dep posture.
- Sources Consulted: `docs/threat-model.md` (sandbox table), `config.py` `SandboxConfig`, Clearwing `hunter_sandbox.py` (image build + spawn, Docker SDK), Docker isolation guidance (`docker run --network none --read-only --tmpfs --memory --pids-limit --cap-drop ALL --security-opt no-new-privileges`), initrunner docker-sandbox docs (CLI `docker run --rm --init`, optional `runsc`).
- Findings:
- Python `docker` SDK would be a new core or extra dependency. The operator already has the Docker CLI if they can run containers.
- Subprocess `docker run --rm --init` with bind-mount of the repo read-only and a tmpfs/scratch write area matches the threat model and keeps `dependencies = []`.
- Unit tests must not require a daemon: inject a `SandboxRunner` protocol, ship a fake. Integration tests skip when `docker info` fails.
- Full sanitizer-image builds (Clearwing) are out of scope for this spec. Stage 3 runs language-specific commands on a small pinned image per language family, or `inconclusive` if the recipe is missing.
- **Implications**: `SandboxProbe` + `DockerCliRunner`. No SDK. gVisor/`runsc` is optional `security-opt` when `docker info` reports it; not required.
- Implications: `SandboxProbe` + `DockerCliRunner`. No SDK. gVisor/`runsc` is optional `security-opt` when `docker info` reports it; not required.
### Evaluation corpora
- **Context**: User rejected cheat-sheet fixtures. Current gate is ≥90% recall on those fixtures.
- **Sources Consulted**: `benchmarks/fixtures/python-vulnerable/app.py` (line 77 split-sink comment), `javascript-vulnerable/src/app.js` (prototype pollution comment), `tests/test_detection_benchmarks.py`, `gate.py`.
- **Findings**:
- Context: User rejected cheat-sheet fixtures. Current gate is ≥90% recall on those fixtures.
- Sources Consulted: `benchmarks/fixtures/python-vulnerable/app.py` (line 77 split-sink comment), `javascript-vulnerable/src/app.js` (prototype pollution comment), `tests/test_detection_benchmarks.py`, `gate.py`.
- Findings:
- Stage 1 smoke fixtures are still useful as unit tests. They must not be the stage 2/3 merge gate.
- A split-sink corpus is small and owned: source and sink on different lines; mutations through wrappers/variables. Success = map ranks the true function in the top band; stage 3 triggers at least one stage-1 miss when Docker is present.
- Comparing map ranking to "sort by pattern-hit count" is an anti-cheat-sheet test: if regex density is the map, the gate fails.
- **Implications**: New `benchmarks/fixtures/split-sink-*` and a stage-2 ranking gate. Keep existing `openultrasast.gate` for stage 1 only.
- Implications: New `benchmarks/fixtures/split-sink-*` and a stage-2 ranking gate. Keep existing `openultrasast.gate` for stage 1 only.
### Seams that already exist and must not be confused
- **Context**: Brownfield traps for a spec named "map" that also adds Docker.
- **Sources Consulted**: `skills.py` (`STAGES = ("map", "hunt", "verify", "fix")`), `mcp.py:68` (rejects `deep`), `reports.py:115` (`scan_exit_code`), `mapping.py` (no Java/Groovy AST), `config.py` (`SandboxConfig` parsed, never consumed by `_run_scan`), `rank.py` (`rank_targets_with_model` unused by CLI).
- **Findings**:
- Context: Brownfield traps for a spec named "map" that also adds Docker.
- Sources Consulted: `skills.py` (`STAGES = ("map", "hunt", "verify", "fix")`), `mcp.py:68` (rejects `deep`), `reports.py:115` (`scan_exit_code`), `mapping.py` (no Java/Groovy AST), `config.py` (`SandboxConfig` parsed, never consumed by `_run_scan`), `rank.py` (`rank_targets_with_model` unused by CLI).
- Findings:
- Trail of Bits skill routing already uses a stage named `map` (mapping-discipline skills: Semgrep/CodeQL/sharp-edge). That is not the complexity map. `stages.py` must not reuse `skills.STAGES`.
- MCP scan tools reject `deep` and forbid docker/shell. This spec does not widen the MCP surface; deep is CLI/CI only.
- `--fail-on` is implemented in `scan_exit_code`, not only argparse. `worth-fixing` must be added there.
- `--fail-on` is implemented in `scan_exit_code` as well as argparse. `worth-fixing` must be added there.
- Java/Groovy entry points are tag-only (`line=None` → `inferred-file-surface`). Hotspot `function_name` will often be empty on Java; the map must still rank the file.
- `[sandbox]`, `[dynamic]`, `[evidence]`, and `[models].ranker` are loaded into resolved config and then ignored. Stage 3 consumes `SandboxConfig`; it does not invent a parallel sandbox TOML.
- **Implications**: Naming, MCP, fail-on, and Java identity are integration constraints, not new products.
- Implications: Naming, MCP, fail-on, and Java identity are integration constraints, not new products.
### Evidence ladder honesty
- **Context**: HarnessX hunter stamped LLM JSON as `static_corroboration`. Original spec forbids reporting LLM-only claims as verified.
- **Sources Consulted**: `verification.py` (`verify_finding` accepts when reachability is not `unknown`), `hunter_harness.py` evidence_level assignment, openrouter requirements Evidence Ladder / Verification.
- **Findings**:
- Context: HarnessX hunter stamped LLM JSON as `static_corroboration`. Original spec forbids reporting LLM-only claims as verified.
- Sources Consulted: `verification.py` (`verify_finding` accepts when reachability is not `unknown`), `hunter_harness.py` evidence_level assignment, openrouter requirements Evidence Ladder / Verification.
- Findings:
- Pattern hits as `static_corroboration` is consistent with the ladder. LLM hits as `suspicion` is the missing rule.
- `--fail-on verified` today means reachable + static corroboration. That remains the cheap PR gate. `--fail-on worth-fixing` is the rugged gate.
- **Implications**: Do not change CWE policy. Change hunter evidence. Add fail-on value. Report sections: Inventory, Complexity map, Worth fixing.
- Implications: Do not change CWE policy. Change hunter evidence. Add fail-on value. Report sections: Inventory, Complexity map, Worth fixing.
## Architecture Pattern Evaluation
@@ -93,64 +93,64 @@
## Design Decisions
### Decision D1: Modes map to stages
- **Context**: Need a three-question scan without breaking CI that already passes `--mode`.
- **Selected Approach**: `quick`→stage 1, `standard`→1+2, `deep`→1+2+3. Manifest records both.
- **Rationale**: Evolutionary adoption of the CLI the project already shipped.
- **Trade-offs**: Operators must learn that `standard` now means "map" not "regex hunter pool." README must say so.
- Context: Need a three-question scan without breaking CI that already passes `--mode`.
- Selected Approach: `quick`→stage 1, `standard`→1+2, `deep`→1+2+3. Manifest records both.
- Rationale: Evolutionary adoption of the CLI the project already shipped.
- Trade-offs: Operators must learn that `standard` now means "map" not "regex hunter pool." README must say so.
### Decision D2: Docker CLI subprocess, no SDK
- **Context**: Zero-dep core vs isolated execution.
- **Selected Approach**: `docker` binary via `subprocess`; probe with `docker info`; fake runner in unit tests.
- **Rationale**: Operator runtime, not a Python dependency. Matches threat-model flags.
- **Trade-offs**: Windows/macOS Docker Desktop variance; integration tests skip without a daemon.
- Context: Zero-dep core vs isolated execution.
- Selected Approach: `docker` binary via `subprocess`; probe with `docker info`; fake runner in unit tests.
- Rationale: Operator runtime, not a Python dependency. Matches threat-model flags.
- Trade-offs: Windows/macOS Docker Desktop variance; integration tests skip without a daemon.
### Decision D3: OpenRouter client for the tool hunter; HarnessX optional
- **Context**: Last spec hid the LLM behind an extra CI does not install.
- **Selected Approach**: Tool loop calls `OpenRouterChatClient` (or any `RankerClient`-shaped chat client) when `[models].hunter` is set. If HarnessX extra is present, `HxScanOrchestrator` may host the same tools; it is not required.
- **Rationale**: Evolutionary use of code already in tree. Stdlib HTTP.
- **Trade-offs**: Two possible runtimes until HarnessX is either used or ignored; tests mock the client.
- Context: Last spec hid the LLM behind an extra CI does not install.
- Selected Approach: Tool loop calls `OpenRouterChatClient` (or any `RankerClient`-shaped chat client) when `[models].hunter` is set. If HarnessX extra is present, `HxScanOrchestrator` may host the same tools; it is not required.
- Rationale: Evolutionary use of code already in tree. Stdlib HTTP.
- Trade-offs: Two possible runtimes until HarnessX is either used or ignored; tests mock the client.
### Decision D4: Complexity map is a new artifact
- **Context**: Ranking already exists.
- **Selected Approach**: `ComplexityMap` with `Hotspot` records. Ranking remains the hunter budget. Map is what humans and stage 3 consume.
- **Rationale**: Different question (where to look / how to test vs which file to hunt first).
- Context: Ranking already exists.
- Selected Approach: `ComplexityMap` with `Hotspot` records. Ranking remains the hunter budget. Map is what humans and stage 3 consume.
- Rationale: Different question (where to look / how to test vs which file to hunt first).
### Decision D5: Worth-fixing is a gate, not an evidence level
- **Context**: Evidence ladder is already six rungs. Do not invent a seventh for CI.
- **Selected Approach**: Verdicts `triggerable|not_triggerable|already_covered|inconclusive` → `worth_fixing|not_worth_fixing`. Evidence levels unchanged. Hunter stays at `suspicion`.
- **Rationale**: CI owners need a fail-on that means "we proved it," without collapsing the ladder.
- Context: Evidence ladder is already six rungs. Do not invent a seventh for CI.
- Selected Approach: Verdicts `triggerable|not_triggerable|already_covered|inconclusive` → `worth_fixing|not_worth_fixing`. Evidence levels unchanged. Hunter stays at `suspicion`.
- Rationale: CI owners need a fail-on that means "we proved it," without collapsing the ladder.
### Decision D6: Split-sink / mutation gate for stages 2–3
- **Context**: 90% recall on cheat sheets is circular.
- **Selected Approach**: New corpus + "beats pattern-hit ranking" test. Stage 1 smoke gate stays.
- **Rationale**: Directly answers the user's evaluation complaint.
- Context: 90% recall on cheat sheets is circular.
- Selected Approach: New corpus + "beats pattern-hit ranking" test. Stage 1 smoke gate stays.
- Rationale: Directly answers the user's evaluation complaint.
### Decision D7: Ledger overlays hotspot scores, not patterns
- **Context**: Evolutionary feedback without repeating "loop must not edit patterns."
- **Selected Approach**: `.openultrasast/calibration/complexity_ledger.json` keyed by `repo_fingerprint + path + function`. Triggerable boosts, not_triggerable demotes. Cannot hide reachable severity-5 inventory from worth-fixing consideration.
- **Rationale**: Same overlay pattern as `rule_policy.json`, different object.
- Context: Evolutionary feedback without repeating "loop must not edit patterns."
- Selected Approach: `.openultrasast/calibration/complexity_ledger.json` keyed by `repo_fingerprint + path + function`. Triggerable boosts, not_triggerable demotes. Cannot hide reachable severity-5 inventory from worth-fixing consideration.
- Rationale: Same overlay pattern as `rule_policy.json`, different object.
### Decision D8: Language recipes, not a universal executor
- **Context**: Running untrusted Python vs C vs Java is different.
- **Selected Approach**: Small recipe registry (Python and JavaScript required for MVP; C compile+run when a compiler exists in the image; Java `inconclusive` until a recipe ships). Missing recipe → `inconclusive`, not a crash.
- **Rationale**: Simplification. Do not build Clearwing's sanitizer image matrix in this spec.
- Context: Running untrusted Python vs C vs Java is different.
- Selected Approach: Small recipe registry (Python and JavaScript required for MVP; C compile+run when a compiler exists in the image; Java `inconclusive` until a recipe ships). Missing recipe → `inconclusive`, not a crash.
- Rationale: Simplification. Do not build Clearwing's sanitizer image matrix in this spec.
## Synthesis
- **Generalization**: Inventory → attention → proof is one pipeline with a `StagePlan`. Map, hints, and regress are processors on that plan, not three products.
- **Build vs adopt**: Adopt Docker CLI and the in-tree OpenRouter client. Build the map, ledger, recipes, and split-sink corpus. Do not adopt Clearwing, docker SDK, or a cyclomatic-complexity library (nesting depth from indentation/braces is enough).
- **Simplification**: No `--stage` flag, no HarnessX requirement, no fuzz campaign, no patch oracle, no Python Docker extra.
- Generalization: Inventory → attention → proof is one pipeline with a `StagePlan`. Map, hints, and regress are processors on that plan, not three products.
- Build vs adopt: Adopt Docker CLI and the in-tree OpenRouter client. Build the map, ledger, recipes, and split-sink corpus. Do not adopt Clearwing, docker SDK, or a cyclomatic-complexity library (nesting depth from indentation/braces is enough).
- Simplification: No `--stage` flag, no HarnessX requirement, no fuzz campaign, no patch oracle, no Python Docker extra.
## Risks & Mitigations
- Docker-in-Docker / rootless CI — probe, skip stage 3, degrade; do not fail stage 1.
- Hunter cost on large repos — hotspot cap + existing `[harnessx] max_cost_usd` / hardening `max_findings` / new `max_regress_candidates`.
- Untrusted snippet execution — structural deny (network, writes outside scratch, curl, docker socket mount). Never mount `/var/run/docker.sock`.
- Stage 2 still heuristic without a model — allowed; gate is ranking quality on split-sink, which heuristics can pass if tags+reachability work.
- Breaking `deep` from "exits" to "runs" — intended; document in README.
- Docker-in-Docker / rootless CI, probe, skip stage 3, degrade; do not fail stage 1.
- Hunter cost on large repos, hotspot cap + existing `[harnessx] max_cost_usd` / hardening `max_findings` / new `max_regress_candidates`.
- Untrusted snippet execution, structural deny (network, writes outside scratch, curl, docker socket mount). Never mount `/var/run/docker.sock`.
- Stage 2 still heuristic without a model, allowed; gate is ranking quality on split-sink, which heuristics can pass if tags+reachability work.
- Breaking `deep` from "exits" to "runs", intended; document in README.
## References
- OpenUltraSAST threat model — `docs/threat-model.md`
- OpenRouter SAST harness requirements — Docker sandbox, hunter pool, evidence ladder
- HarnessX self-improving rulesets — adjacent governance; do not reopen
- Docker run isolation flags — `--network none`, `--read-only`, `--cap-drop ALL`, `--security-opt no-new-privileges`
- Clearwing sourcehunt hunter tools and sandbox — implementation oracle, not a dependency
- OpenUltraSAST threat model, `docs/threat-model.md`
- OpenRouter SAST harness requirements, Docker sandbox, hunter pool, evidence ladder
- HarnessX self-improving rulesets, adjacent governance; do not reopen
- Docker run isolation flags, `--network none`, `--read-only`, `--cap-drop ALL`, `--security-opt no-new-privileges`
- Clearwing sourcehunt hunter tools and sandbox, implementation oracle, not a dependency
@@ -6,7 +6,7 @@ AppSec scanning mixed working trees gets `language_unsupported` on JS/C/Java, an
## Current State
`parse_tree_sitter_cli` returns `None` even on a successful parse. Python falls back to stdlib `ast`. JS/C/Java stay unadjudicated. `quick` is correctly zero-dep. SAST overlay Youden on the 11-pair textbook slice is 0% vs +9% inventory — not because regex got worse, but because Python overlay fail-closes on ConfigParser-style heap and C/Java never parse.
`parse_tree_sitter_cli` returns `None` even on a successful parse. Python falls back to stdlib `ast`. JS/C/Java stay unadjudicated. `quick` is correctly zero-dep. SAST overlay Youden on the 11-pair textbook slice is 0% vs +9% inventory, not because regex got worse, but because Python overlay fail-closes on ConfigParser-style heap and C/Java never parse.
## Desired Outcome
@@ -18,8 +18,8 @@ Add optional extra `openultrasast[semantic]` (`tree-sitter` + official language
## Scope
- **In**: extra, probe that prefers the Python API when importable, CST → FileIR, tests with and without the extra.
- **Out**: Joern queries, inter-file taint, CWE-121 buffer semantics, new pair corpora (owned by `real-world-vfc-slice`).
- In: extra, probe that prefers the Python API when importable, CST → FileIR, tests with and without the extra.
- Out: Joern queries, inter-file taint, CWE-121 buffer semantics, new pair corpora (owned by `real-world-vfc-slice`).
## Boundary Candidates
@@ -32,13 +32,13 @@ Add optional extra `openultrasast[semantic]` (`tree-sitter` + official language
## Upstream / Downstream
- **Upstream**: `propose-adjudicate-prove` overlay records and taint
- **Downstream**: `real-world-vfc-slice` overlay scoring on C/JS/Java pairs
- Upstream: `propose-adjudicate-prove` overlay records and taint
- Downstream: `real-world-vfc-slice` overlay scoring on C/JS/Java pairs
## Existing Spec Touchpoints
- **Extends**: none (new spec). Does not reopen `propose-adjudicate-prove` tasks.
- **Adjacent**: `three-stage-scan` modes; `quick` must remain zero-dep.
- Extends: none (new spec). Does not reopen `propose-adjudicate-prove` tasks.
- Adjacent: `three-stage-scan` modes; `quick` must remain zero-dep.
## Constraints
@@ -116,13 +116,13 @@ tests/test_semantic_cst.py
### Modified Files
- `pyproject.toml` — extra `semantic`; pytest marker; mypy overrides.
- `src/openultrasast/semantic/ir.py` — dispatch; delete `parse_tree_sitter_cli` as success path.
- `src/openultrasast/semantic/engines.py` — `tree_sitter_available` must not mean CLI-only; extra/grammar helpers live in `extra.py`.
- `src/openultrasast/semantic/__init__.py` — export `has_semantic_extra` only if needed; do not import cst at package load.
- `tests/test_semantic_overlay.py` — extra-free JS stays `language_unsupported` without monkeypatching a fake IR as “tree-sitter success”.
- `.github/workflows/ci.yml` — second job with `--extra semantic` running `@pytest.mark.semantic`.
- `dagger/ci.py` — keep extra-free unless the same extra-on command is cheap to add; default is GHA second job only.
- `pyproject.toml`: extra `semantic`; pytest marker; mypy overrides.
- `src/openultrasast/semantic/ir.py`: dispatch; delete `parse_tree_sitter_cli` as success path.
- `src/openultrasast/semantic/engines.py`: `tree_sitter_available` must not mean CLI-only; extra/grammar helpers live in `extra.py`.
- `src/openultrasast/semantic/__init__.py`: export `has_semantic_extra` only if needed; do not import cst at package load.
- `tests/test_semantic_overlay.py`: extra-free JS stays `language_unsupported` without monkeypatching a fake IR as "tree-sitter success".
- `.github/workflows/ci.yml`: second job with `--extra semantic` running `@pytest.mark.semantic`.
- `dagger/ci.py`: keep extra-free unless the same extra-on command is cheap to add; default is GHA second job only.
Do not modify `taint.py`, `overlay.py`, `facts.py`, `gate.py`, `pairs.py`.
@@ -8,9 +8,9 @@ AppSec engineers scanning mixed working trees need overlay adjudication on Pytho
## Boundary Context
- **In scope**: Optional extra install; overlay using a structured parse for Python, JavaScript, C/C++, and Java when that extra provides a grammar; fail-closed `unadjudicated` when the extra or grammar is missing; `quick` unchanged and extra-free; existing overlay dispositions unchanged.
- **Out of scope**: New pair corpora or Firefox/VFC harvest (`real-world-vfc-slice`); Joern CPG queries; inter-file or whole-program taint; buffer-size / CWE-121 semantics; language-model parse or taint; replacing pattern inventory.
- **Adjacent expectations**: `propose-adjudicate-prove` still owns dispositions, facts, taint rules, and prove-budget. `three-stage-scan` still owns `quick` / `standard` / `deep` and the sandbox. This spec only supplies the structured parse those stages already asked for.
- In scope: Optional extra install; overlay using a structured parse for Python, JavaScript, C/C++, and Java when that extra provides a grammar; fail-closed `unadjudicated` when the extra or grammar is missing; `quick` unchanged and extra-free; existing overlay dispositions unchanged.
- Out of scope: New pair corpora or Firefox/VFC harvest (`real-world-vfc-slice`); Joern CPG queries; inter-file or whole-program taint; buffer-size / CWE-121 semantics; language-model parse or taint; replacing pattern inventory.
- Adjacent expectations: `propose-adjudicate-prove` still owns dispositions, facts, taint rules, and prove-budget. `three-stage-scan` still owns `quick` / `standard` / `deep` and the sandbox. This spec only supplies the structured parse those stages already asked for.
## Requirements
@@ -21,7 +21,7 @@ AppSec engineers scanning mixed working trees need overlay adjudication on Pytho
#### Acceptance Criteria
1. When OpenUltraSAST runs a `quick` scan, the system shall emit inventory proposals without requiring the semantic extra.
2. The system shall not require the semantic extra to be installed in order to import or run the default OpenUltraSAST command.
2. The system shall not require the semantic extra to be installed to import or run the default OpenUltraSAST command.
3. If the semantic extra is absent, then a `quick` scan shall still complete and shall not abort because a parser package is missing.
### Requirement 2: Extra enables structured parse on in-scope languages
@@ -30,7 +30,7 @@ AppSec engineers scanning mixed working trees need overlay adjudication on Pytho
#### Acceptance Criteria
1. Where the semantic extra is installed and a grammar for the file’s language is available, when OpenUltraSAST runs a `standard` or `deep` scan, the system shall use that structured parse as the primary intra-file input to overlay adjudication for Python, JavaScript, C, C++, and Java.
1. Where the semantic extra is installed and a grammar for the file's language is available, when OpenUltraSAST runs a `standard` or `deep` scan, the system shall use that structured parse as the primary intra-file input to overlay adjudication for Python, JavaScript, C, C++, and Java.
2. When a JavaScript file contains `eval` of a request query (or the language-equivalent untrusted source to a known sink) and the extra provides a JavaScript grammar, the system shall be able to emit a `promote` overlay disposition for that proposal.
3. When a C or Java file contains a sink fed only by a constant or a dominating sanitizer (for example a literal format string to `printf`) and the extra provides a grammar for that language, the system shall `demote` that proposal rather than leave it `language_unsupported`.
4. The system shall feed the structured parse into the existing overlay dispositions (`promote`, `demote`, `unadjudicated`, `coverage`) and shall not introduce a new disposition.
@@ -42,7 +42,7 @@ AppSec engineers scanning mixed working trees need overlay adjudication on Pytho
#### Acceptance Criteria
1. If the semantic extra is not installed, then when OpenUltraSAST adjudicates a JavaScript, C, C++, or Java file, the system shall leave matching inventory proposals `unadjudicated` with `language_unsupported` or `parse_failed`.
2. If the extra is installed but no grammar is available for that file’s language, then the system shall leave those proposals `unadjudicated` with `language_unsupported` or `parse_failed` and shall not crash the scan.
2. If the extra is installed but no grammar is available for that file's language, then the system shall leave those proposals `unadjudicated` with `language_unsupported` or `parse_failed` and shall not crash the scan.
3. When the extra is absent, the system shall still try a stdlib parse only for Python and shall otherwise follow the fail-closed rule in criterion 1.
4. If a file cannot be parsed even with a grammar present (syntax error or incomplete source), then the system shall leave matching proposals `unadjudicated` with `parse_failed`.
5. The system shall not treat a missing extra as a failed `standard` scan by itself.
@@ -55,7 +55,7 @@ AppSec engineers scanning mixed working trees need overlay adjudication on Pytho
1. When the extra supplies grammars for more than one in-scope language, the system shall extract overlay parse facts (assignments, calls, constants) through one shared structured-parse path rather than a separate visitor per language.
2. The system shall not add a new language-specific stdlib or regex AST as the product parse engine for JavaScript, C, C++, or Java.
3. Where a host parser CLI is present but the semantic extra is not, the system shall not claim a successful structured parse for overlay unless the extra’s grammar actually produced one.
3. Where a host parser CLI is present but the semantic extra is not, the system shall not claim a successful structured parse for overlay unless the extra's grammar actually produced one.
### Requirement 5: Overlay and stages stay in their existing contracts
@@ -26,12 +26,12 @@ Steering `product.md` / `tech.md` / `structure.md` are absent. Context is the ov
- Optional native/heavy deps stay off the default install; capability probe must not import at module load (`harness_ext`, Docker sandbox probe).
- mypy `ignore_missing_imports` for extras (`harnessx.*`).
- pytest marker already exists for `docker`; extra-enabled tests would follow that skip pattern.
- CI `checks` job: `uv sync --group dev` then pytest — **no optional extras**.
- CI `checks` job: `uv sync --group dev` then pytest, **no optional extras**.
### Integration surfaces
- `parse_file(path, text, language) -> FileIR` is the only seam taint/overlay need.
- `tree_sitter_available()` is a **presence** probe, not “grammar produced IR”. Host CLI on PATH makes it True even with no PyPI wheels (this environment has `/home/linuxbrew/.../tree-sitter` and **no languages**).
- `tree_sitter_available()` is a **presence** probe, not "grammar produced IR". Host CLI on PATH makes it True even with no PyPI wheels (this environment has `/home/linuxbrew/.../tree-sitter` and **no languages**).
- Pair eval already falls back to inventory when a language never promotes/demotes.
## 2. Requirements Feasibility
@@ -60,23 +60,23 @@ Steering `product.md` / `tech.md` / `structure.md` are absent. Context is the ov
Put extra pins in `pyproject.toml`, implement API parse + CST walk inside `ir.py`, tighten `tree_sitter_available()`.
- **Pros:** Few files; `parse_file` stays the seam.
- **Cons:** `ir.py` is already ~277 lines (Python ast + dead CLI). A walker + language table will bloat it and mix packaging probe with extract.
- Pros: Few files; `parse_file` stays the seam.
- Cons: `ir.py` is already ~277 lines (Python ast + dead CLI). A walker + language table will bloat it and mix packaging probe with extract.
### Option B: New components only
New `semantic/cst.py` + `semantic/extra.py`; leave `ir.py` as Python ast; overlay calls a new entrypoint.
- **Pros:** Clean extra guard.
- **Cons:** Two parse entrypoints; easy to fork `parse_file` consumers.
- Pros: Clean extra guard.
- Cons: Two parse entrypoints; easy to fork `parse_file` consumers.
### Option C: Hybrid (recommended for design)
- **New:** `semantic/cst.py` — one walker: tree-sitter `Node` → `Bind` / `CallSite` / `FunctionIR` using a node-kind map.
- **New or thin:** extra guard modeled on `harness_ext` (`has_semantic_extra`, lazy grammar load, never import wheels at module load).
- **Extend:** `pyproject.toml` `[project.optional-dependencies] semantic = [...]`; mypy override `tree_sitter*`; pytest marker `semantic`; CI extra-free job unchanged; optional second job or skip.
- **Extend:** `parse_file` prefers extra CST when a grammar loads; else Python ast; else `language_unsupported`. **Delete or stop calling** `parse_tree_sitter_cli` as a success path (Req 4.3).
- **Do not extend:** `taint.py`, `overlay.py`, facts, gate, pair catalog.
- New: `semantic/cst.py`: one walker: tree-sitter `Node` → `Bind` / `CallSite` / `FunctionIR` using a node-kind map.
- New or thin: extra guard modeled on `harness_ext` (`has_semantic_extra`, lazy grammar load, never import wheels at module load).
- Extend: `pyproject.toml` `[project.optional-dependencies] semantic = [...]`; mypy override `tree_sitter*`; pytest marker `semantic`; CI extra-free job unchanged; optional second job or skip.
- Extend: `parse_file` prefers extra CST when a grammar loads; else Python ast; else `language_unsupported`. **Delete or stop calling** `parse_tree_sitter_cli` as a success path (Req 4.3).
- Do not extend: `taint.py`, `overlay.py`, facts, gate, pair catalog.
**Trade-offs:** Extra files, but `ir.py` shrinks (drop dead CLI), taint stays language-agnostic, extra follows an existing packaging seam.
@@ -89,7 +89,7 @@ New `semantic/cst.py` + `semantic/extra.py`; leave `ir.py` as Python ast; overla
- Prefer **Option C**.
- Treat `FileIR` as frozen contract; do not add dispositions.
- Decide probe semantics: `tree_sitter_available` should mean “can produce FileIR for this language”, not “CLI binary exists”.
- Decide probe semantics: `tree_sitter_available` should mean "can produce FileIR for this language", not "CLI binary exists".
- Carry research items 1–4 into design (ABI pins, node-kind map, cpp package, CI job vs skip).
- Do not pull Joern, pair harvest, or CWE-121 into this spec.
@@ -97,9 +97,9 @@ New `semantic/cst.py` + `semantic/extra.py`; leave `ir.py` as Python ast; overla
## Summary (design discovery)
- **Feature**: tree-sitter-overlay-extra
- **Discovery Scope**: Extension of `parse_file` / overlay IR; optional extra packaging
- **Key Findings**:
- Feature: tree-sitter-overlay-extra
- Discovery Scope: Extension of `parse_file` / overlay IR; optional extra packaging
- Key Findings:
- Official PyPI wheels: `tree-sitter` (0.26), `tree-sitter-python`, `tree-sitter-javascript` (0.25), `tree-sitter-c` (0.24.x), `tree-sitter-cpp` (0.23.4), `tree-sitter-java`. API is `Parser(Language(mod.language()))`; `build_library` is gone.
- `FileIR` is the only overlay seam; taint does not need language visitors.
- Host CLI presence must not count as a successful overlay parse (Req 4.3).
@@ -108,16 +108,16 @@ New `semantic/cst.py` + `semantic/extra.py`; leave `ir.py` as Python ast; overla
### PyPI ABI and load API
- **Context**: Pin wheels that work on Python 3.11+ without compiling grammars in CI.
- **Sources Consulted**: https://pypi.org/project/tree-sitter/ ; https://pypi.org/project/tree-sitter-javascript/ ; https://pypi.org/project/tree-sitter-cpp/ ; py-tree-sitter 0.23 release notes (Language capsule, Parser constructor).
- **Findings**: Grammars ship abi3 wheels. Load with `Language(tree_sitter_<lang>.language())` then `Parser(language)`. Do not use `Language.build_library`.
- **Implications**: Extra lists core + five grammar packages. mypy ignores `tree_sitter*`.
- Context: Pin wheels that work on Python 3.11+ without compiling grammars in CI.
- Sources Consulted: https://pypi.org/project/tree-sitter/ ; https://pypi.org/project/tree-sitter-javascript/ ; https://pypi.org/project/tree-sitter-cpp/ ; py-tree-sitter 0.23 release notes (Language capsule, Parser constructor).
- Findings: Grammars ship abi3 wheels. Load with `Language(tree_sitter_<lang>.language())` then `Parser(language)`. Do not use `Language.build_library`.
- Implications: Extra lists core + five grammar packages. mypy ignores `tree_sitter*`.
### Probe vs grammar
- **Context**: `tree_sitter_available()` is true when CLI is on PATH with zero grammars.
- **Findings**: CLI parse cannot populate `FileIR` without a walker. Extra Python API is the product parse.
- **Implications**: Split `has_semantic_extra()` from per-language `grammar_for`. Remove CLI as a success path.
- Context: `tree_sitter_available()` is true when CLI is on PATH with zero grammars.
- Findings: CLI parse cannot populate `FileIR` without a walker. Extra Python API is the product parse.
- Implications: Split `has_semantic_extra()` from per-language `grammar_for`. Remove CLI as a success path.
## Architecture Pattern Evaluation
@@ -131,37 +131,37 @@ New `semantic/cst.py` + `semantic/extra.py`; leave `ir.py` as Python ast; overla
### Decision: Adopt official grammar wheels, build one walker
- **Context**: Avoid per-language AST visitors (Req 4).
- **Alternatives**: tree-sitter CLI sexp; py-tree-sitter-languages (unmaintained); hand-rolled JS/C parsers.
- **Selected**: `openultrasast[semantic]` = `tree-sitter` plus python/javascript/c/cpp/java grammar packages. One CST walk keyed by a node-kind map.
- **Rationale**: Wheels are MIT, maintained, match in-scope languages. Taint stays language-agnostic.
- **Trade-offs**: Native wheels in the extra; ABI pins. Core install stays empty.
- **Follow-up**: Verify Parser constructor against the pinned `tree-sitter` minor at implement time.
- Context: Avoid per-language AST visitors (Req 4).
- Alternatives: tree-sitter CLI sexp; py-tree-sitter-languages (unmaintained); hand-rolled JS/C parsers.
- Selected: `openultrasast[semantic]` = `tree-sitter` plus python/javascript/c/cpp/java grammar packages. One CST walk keyed by a node-kind map.
- Rationale: Wheels are MIT, maintained, match in-scope languages. Taint stays language-agnostic.
- Trade-offs: Native wheels in the extra; ABI pins. Core install stays empty.
- Follow-up: Verify Parser constructor against the pinned `tree-sitter` minor at implement time.
### Decision: Extra-free CI stays default; extra-on is a second job plus skip marker
- **Context**: Req 6 dual matrix.
- **Selected**: pytest `semantic` marker skips when extra absent. GitHub Actions adds a job `uv sync --extra semantic --group dev` that runs marked tests. Default `checks` job unchanged.
- **Rationale**: Default CI must not fail without wheels. Extra-on tests must actually run somewhere.
- **Trade-offs**: Two pytest invocations in CI.
- Context: Req 6 dual matrix.
- Selected: pytest `semantic` marker skips when extra absent. GitHub Actions adds a job `uv sync --extra semantic --group dev` that runs marked tests. Default `checks` job unchanged.
- Rationale: Default CI must not fail without wheels. Extra-on tests must actually run somewhere.
- Trade-offs: Two pytest invocations in CI.
### Decision: has_error implies parse_failed
- **Context**: tree-sitter is error-tolerant; Req 3.4 wants fail-closed on syntax errors.
- **Selected**: If `tree.root_node.has_error` (or ERROR children), return `FileIR(parse_ok=False, reason=parse_failed)`.
- **Rationale**: Partial IR would invite false demote.
- Context: tree-sitter is error-tolerant; Req 3.4 wants fail-closed on syntax errors.
- Selected: If `tree.root_node.has_error` (or ERROR children), return `FileIR(parse_ok=False, reason=parse_failed)`.
- Rationale: Partial IR would invite false demote.
## Synthesis
- **Generalization**: One `parse_file` dispatch; extra is a grammar source, not a second overlay.
- **Build vs adopt**: Adopt tree-sitter wheels; build only the FileIR projection (binds/calls).
- **Simplification**: No CLI sexp walker; no second parse API; no taint/overlay changes.
- Generalization: One `parse_file` dispatch; extra is a grammar source, not a second overlay.
- Build vs adopt: Adopt tree-sitter wheels; build only the FileIR projection (binds/calls).
- Simplification: No CLI sexp walker; no second parse API; no taint/overlay changes.
## Risks & Mitigations
- Grammar ABI mismatch — pin extras together; extra-on CI job catches import/parse failure.
- Node-type drift — table in cst.py; tests assert JS eval and C printf on real trees.
- CLI still on PATH — must not set parse_ok.
- Grammar ABI mismatch: pin extras together; extra-on CI job catches import/parse failure.
- Node-type drift: table in cst.py; tests assert JS eval and C printf on real trees.
- CLI still on PATH: must not set parse_ok.
## References
+13 -13
View File
@@ -1,4 +1,4 @@
# Brief: zero-with-a-reason — a finding count is meaningless without a denominator
# Brief: zero-with-a-reason, a finding count is meaningless without a denominator
> **Status 2026-10-02:** dormant. Brief only and unapproved; not touched since 2026-09-10.
@@ -33,7 +33,7 @@ produced. There is nothing to put on a ladder.
**Every zero must be decomposable, and every scan must be able to prove it did work.** Three mechanisms,
independent, in order of value per line of code.
### 1. The funnel — make the denominator first-class
### 1. The funnel, make the denominator first-class
The scan is already a chain of narrowing stages, each of which computes a count and throws it away:
@@ -57,10 +57,10 @@ Cheap, because every number already exists at the moment it is computed. `ModelS
the coverage disclosure (Req 5.6, already written) is where it prints. The hard rule that gives it teeth: a
report may not describe a repository as clean without the funnel beside it.
### 2. The dumb oracle — an independent lower bound on the graph
### 2. The dumb oracle, an independent lower bound on the graph
The CPG is an elaborate instrument. A regex over the source text is a stupid one, and stupid instruments
fail differently — which is the entire reason to have two.
fail differently, which is the entire reason to have two.
Before querying, count the fact tables' sink names in the raw source text. That is a floor on what the graph
must contain:
@@ -72,13 +72,13 @@ the CPG's sink match returns 0
```
This catches an empty graph, the wrong frontend, dropped files, a broken name matcher, and a missing
overlay — bugs 2, 3, 5, 6 and 7 above, none of which it needs to know about in advance. It is the same trick
overlay (bugs 2, 3, 5, 6 and 7 above), none of which it needs to know about in advance. It is the same trick
the pair corpus already uses: two independent measurements of one thing must agree.
What it cannot do: say anything about over-reporting, or about flows. It is purely "does the graph contain
the syntax the text contains". That narrowness is why it is trustworthy.
### 3. The canary — an end-to-end positive control
### 3. The canary, an end-to-end positive control
One synthetic file with an unmistakable flow, one extra request in the batch:
@@ -91,7 +91,7 @@ ran, the file reached the graph, overlays applied, the fact tables loaded, the q
round-tripped, the payload parsed, the arbiter ran and the ladder assigned a rung.
With one honest caveat that must be designed around: **adding a file to a PHP build can itself trigger the
php2cpg race** — a two-line file added to `class.memberorder.php` is exactly what emptied that graph. So on
php2cpg race**. A two-line file added to `class.memberorder.php` is exactly what emptied that graph. So on
PHP the canary belongs in its own build (`ousast doctor`), where it proves the toolchain works, and the
funnel and the oracle prove that *this* build worked. They are complementary, not redundant.
@@ -101,7 +101,7 @@ Every instrument failure in this project has the same shape, and it is not a coi
unreadable and the tool reported a plausible zero.** `file_exists()` returns false, the parser writes nothing,
the frontend exits 0, and "0 files dropped, 5,703-byte graph" reads exactly like success.
It has now happened four times in one day — a docker shim that did not mount `~/.cache` (twice, hours apart,
It has now happened four times in one day. The failures were a docker shim that did not mount `~/.cache` (twice, hours apart,
the second time after I had diagnosed the first), a measurement harness whose script path did not resolve
from its working directory, and a container that could not follow the symlinks in the tree it was handed.
@@ -132,15 +132,15 @@ prevents recurrence rather than detecting it.
## What this is not, and must not become
- **Not a confidence score.** The rung ladder exists precisely so uncertainty is carried as a claim about
- Not a confidence score. The rung ladder exists precisely so uncertainty is carried as a claim about
evidence rather than a number nobody can act on. A "coverage percentage" that averages the funnel away
would undo it.
- **Not a retry loop.** Retrying a failed build until it succeeds hides bug 6's race instead of reporting it.
- Not a retry loop. Retrying a failed build until it succeeds hides bug 6's race instead of reporting it.
Three of five builds succeeding is a fact about the toolchain and belongs in the output.
- **Not anomaly detection over historical finding counts.** 6.1's baselines already give the diff, and
"findings dropped 90%" is a lagging indicator that needs a known-good prior run — which is exactly what a
- Not anomaly detection over historical finding counts. 6.1's baselines already give the diff, and
"findings dropped 90%" is a lagging indicator that needs a known-good prior run, which is exactly what a
first scan of a new repository does not have.
- **Not more code than it saves.** The funnel is plumbing that already exists, the oracle is one pass over
- Not more code than it saves. The funnel is plumbing that already exists, the oracle is one pass over
files the preprocessor already reads, the canary is one file and one request.
## Open questions for the design phase