mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
docs: plain-English pass over the active specs
Ran the humanizer rules over 79 prose files in the active specs (brief, design, requirements, research and narrative docs). Removed em dashes, significance inflation, forced rule-of-three, bold-label list items and filler. Skipped tasks.md checklists and the retired specs (harnessx-*, openrouter-sast-harness, unseen-repo-evaluation). Every number, path, command, hash and identifier was preserved: checked each file against HEAD for drift in numbers, inline code and code fences, zero drift across all 79 files. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
b08f372022
commit
39a1873e5b
@@ -141,7 +141,7 @@ before `ax apply`, and a task without a Model gets none. The value is never rend
|
||||
Every actor connection except DNS goes through Agent Substrate's egress gateway (`atenet-egress`), which denies
|
||||
an actor without an `EgressPolicy`. Three pieces make the plane work under that rule:
|
||||
|
||||
- **Receiver by name.** `ops/ax/receiver-service.yaml.tmpl` (applied by `up.sh`) is a selector-less Service
|
||||
- Receiver by name. `ops/ax/receiver-service.yaml.tmpl` (applied by `up.sh`) is a selector-less Service
|
||||
`ousast-receiver` in `ax-system`, port 80, with an EndpointSlice on the kind gateway at `OUSAST_ARTIFACT_PORT`
|
||||
(fixed, 18090), where the reconciler binds its receiver. Tasks get
|
||||
`OUSAST_ARTIFACT_URL=http://ousast-receiver.ax-system.svc.cluster.local/` plus `OUSAST_ARTIFACT_DIAL=<the
|
||||
@@ -150,12 +150,12 @@ an actor without an `EgressPolicy`. Three pieces make the plane work under that
|
||||
`Host` but connects to the address the actor dialled (measured: dialling the kind gateway on port 80 was allowed
|
||||
and then refused upstream), so the dial address is the ClusterIP, which kube-proxy maps to the host.
|
||||
`OUSAST_ARTIFACT_HOST` set keeps the old direct URL (tests).
|
||||
- **Per-task policy.** `policy_for` grants `http` on port 80 to the receiver name, and `tls_passthrough` on 443
|
||||
- Per-task policy. `policy_for` grants `http` on port 80 to the receiver name, and `tls_passthrough` on 443
|
||||
to the Git hosts of the bound Workspaces' `https` repos and to the bound Model's
|
||||
`openultrasast.io/egress-hosts` (`api.deepseek.com` for `deepseek-flash`, validated as a plain DNS name).
|
||||
Nothing else: no IP address or wildcard, and passthrough only -- an `https` rule intercepts TLS and the actor
|
||||
would need the gateway CA, which ax templates do not mount.
|
||||
- **Order.** `ax apply` creates the actor; the reconciler writes the policy with `kubectl ate create
|
||||
- Order. `ax apply` creates the actor; the reconciler writes the policy with `kubectl ate create
|
||||
egress-policy` (update when one exists), reads it back, waits `OUSAST_EGRESS_SETTLE` (11 s: the gateway caches a
|
||||
missing policy as deny for 10 s), then resumes. kubectl-ate has no delete verb; Substrate's store deletes the
|
||||
policy with its actor, so after `ax delete task` the reconciler confirms it is gone.
|
||||
|
||||
@@ -14,19 +14,19 @@ a DeepSeek provider -- is carried by our thin `Run` layer and dissolves into ax
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: the manifest kinds and a `Run` kind for dependencies, artifacts and budgets; bring-up of ax on
|
||||
- In scope: the manifest kinds and a `Run` kind for dependencies, artifacts and budgets; bring-up of ax on
|
||||
this host (kind + Agent Substrate + ax controller) with a doctor command; an ax-backed reconciler that
|
||||
submits Tasks to ax (one engine-bearing task at a time) and is the only executor; the runner contract and a
|
||||
runner image; the first two tasks, `repo-facts` and
|
||||
`verify`, with the existing scripts as reference behaviour; measurement on the 46-candidate validation set.
|
||||
- **Out of scope**: a remote or multi-node cluster; contributing providers or budgets to ax upstream (a
|
||||
- Out of scope: a remote or multi-node cluster; contributing providers or budgets to ax upstream (a
|
||||
follow-on once the increment is measured); changing what the model is asked (prompts, families, pass rules)
|
||||
except as required to consume `repo-facts`; the judge stage, the config-family question and population v3
|
||||
(they become tasks later, under their own specs); any change to `evaluate.py`'s protocol scoring.
|
||||
- **Adjacent expectations**: the independent-population protocols and records stay as they are; `finding_dump`
|
||||
- Adjacent expectations: the independent-population protocols and records stay as they are; `finding_dump`
|
||||
and the engine path are untouched; `vulnerabilities-over-plumbing` applies -- this is plumbing that must pay
|
||||
for itself in measured tokens and unchanged or better recall.
|
||||
- **Target state (maintainer, 2026-09-29)**: this plane replaces HarnessX as the agentic plane; HarnessX is
|
||||
- Target state (maintainer, 2026-09-29): this plane replaces HarnessX as the agentic plane; HarnessX is
|
||||
removed by its own spec (`harnessx-removal`) once Requirement 6's measurement shows value; the `Run` layer
|
||||
shrinks as ax gains dependencies, budgets and artifacts. No new HarnessX feature is added meanwhile.
|
||||
|
||||
|
||||
@@ -8,9 +8,9 @@ Nothing in the pipeline can represent "this operation should have been guarded"
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
Absence bugs become first-class, measured, and honest. An obligated operation (protected data access, privileged action, security configuration) reached from an entry point without a discharger (a dominating auth guard, an identity constraint bound from the authenticated context, a non-permissive value) is reported as an obligation violation, with the missing discharger kind, the evidence for why the obligation exists (sibling handlers that discharge it, or a declared policy clause), and the known fix shape the corpus taught. Leave-one-out on the absence pairs is the objective, per obligation kind and per provenance profile, so the number rises as the corpus grows. `agent-vfc` and `vibe-py` absence pairs move from 0 detected to a measured baseline that the engine specs can raise.
|
||||
Absence bugs become first-class and are measured without overstating results. An obligated operation (protected data access, privileged action, security configuration) reached from an entry point without a discharger (a dominating auth guard, an identity constraint bound from the authenticated context, a non-permissive value) is reported as an obligation violation, with the missing discharger kind, the evidence for why the obligation exists (sibling handlers that discharge it, or a declared policy clause), and the known fix shape the corpus taught. Leave-one-out on the absence pairs is the objective, per obligation kind and per provenance profile, so the number rises as the corpus grows. `agent-vfc` and `vibe-py` absence pairs move from 0 detected to a measured baseline that the engine specs can raise.
|
||||
|
||||
Principle written into the roadmap: **an operation carries obligations, not only data; the application's own consistency and its declared policy say which, and the tool reports the missing discharger rather than a phantom flow.**
|
||||
Principle written into the roadmap: **an operation carries data and obligations; the application's own consistency and its declared policy say which, and the tool reports the missing discharger rather than a phantom flow.**
|
||||
|
||||
## Approach
|
||||
|
||||
@@ -18,8 +18,8 @@ Obligated operations and discharger kinds are closed facts data, like sinks and
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: obligated-operation and discharger facts; obligation shapes derived from pairs; consistency detection across sibling handlers; optional declared policy file with a closed schema; path-aware violation evaluation with a function-local degraded mode; new evidence rungs and report wording for obligations; worth-fixing weighting by resource sensitivity; corpus recipe shape for absence pairs (handler plus registration context); leave-one-out per obligation kind; `obligations` improve lever; offline tests.
|
||||
- **Out**: taint or facts changes for flow bugs; the call graph and path records themselves (reachability spec); guard-dominance IR (guard-dominance-regime); LLM-authored policy or shapes; any merge gate; vendoring unlicensed code; sandbox proof of absence bugs (they do not crash).
|
||||
- In: obligated-operation and discharger facts; obligation shapes derived from pairs; consistency detection across sibling handlers; optional declared policy file with a closed schema; path-aware violation evaluation with a function-local degraded mode; new evidence rungs and report wording for obligations; worth-fixing weighting by resource sensitivity; corpus recipe shape for absence pairs (handler plus registration context); leave-one-out per obligation kind; `obligations` improve lever; offline tests.
|
||||
- Out: taint or facts changes for flow bugs; the call graph and path records themselves (reachability spec); guard-dominance IR (guard-dominance-regime); LLM-authored policy or shapes; any merge gate; vendoring unlicensed code; sandbox proof of absence bugs (they do not crash).
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
@@ -40,13 +40,13 @@ Obligated operations and discharger kinds are closed facts data, like sinks and
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `EntryPointRecord` (route, access level, decorators, conditions); `FileIR`; path records and dominance from the two engine specs; `PairCase` with `review_tier`; `MechanismStore` and the lever machinery; facts loader.
|
||||
- **Downstream**: reports and SARIF; worth-fixing ranking; the roadmap's leave-one-out table gains obligation kinds; `guard-dominance-regime` consumes obligation discharger kinds as guard candidates for web code.
|
||||
- Upstream: `EntryPointRecord` (route, access level, decorators, conditions); `FileIR`; path records and dominance from the two engine specs; `PairCase` with `review_tier`; `MechanismStore` and the lever machinery; facts loader.
|
||||
- Downstream: reports and SARIF; worth-fixing ranking; the roadmap's leave-one-out table gains obligation kinds; `guard-dominance-regime` consumes obligation discharger kinds as guard candidates for web code.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: `corpus-seeded-mechanisms` (second shape family, same store, same lever contract), `pair-corpus-honesty` (registration-context excerpt mode, obligation label), `three-stage-scan` (one more proposer in MAP, new rungs below proof).
|
||||
- **Depends**: `reachability-flow-model` (entry point to operation paths); `guard-dominance-regime` (a guard dominates the path). Reachability is now on the critical path for two consumers.
|
||||
- Extends: `corpus-seeded-mechanisms` (second shape family, same store, same lever contract), `pair-corpus-honesty` (registration-context excerpt mode, obligation label), `three-stage-scan` (one more proposer in MAP, new rungs below proof).
|
||||
- Depends: `reachability-flow-model` (entry point to operation paths); `guard-dominance-regime` (a guard dominates the path). Reachability is now on the critical path for two consumers.
|
||||
|
||||
## Constraints
|
||||
|
||||
|
||||
@@ -152,9 +152,9 @@ tests/
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `src/openultrasast/cli.py` — `_check_obligations` after `_search_variants` in MAP (standard/deep); manifest `obligations` block; `mechanisms export` writes obligation rows too.
|
||||
- `src/openultrasast/pairs.py` — `ExpectedFinding.obligation: str | None`; `_detections`-side additive rule: for a row with `obligation`, a finding with tag `obligation:<kind>` inside the labeled function counts.
|
||||
- `src/openultrasast/semantic/mechanisms.py` — no schema change; `corpus_mechanisms()` unchanged; a helper `obligation_mechanisms(records)` selects `shape["family"] == "obligation"` (added in `obligations/shapes.py`, not in the store module).
|
||||
- `src/openultrasast/cli.py`: `_check_obligations` after `_search_variants` in MAP (standard/deep); manifest `obligations` block; `mechanisms export` writes obligation rows too.
|
||||
- `src/openultrasast/pairs.py`: `ExpectedFinding.obligation: str | None`; `_detections`-side additive rule: for a row with `obligation`, a finding with tag `obligation:<kind>` inside the labeled function counts.
|
||||
- `src/openultrasast/semantic/mechanisms.py`: no schema change; `corpus_mechanisms()` unchanged; a helper `obligation_mechanisms(records)` selects `shape["family"] == "obligation"` (added in `obligations/shapes.py`, not in the store module).
|
||||
- `src/openultrasast/reports.py`, `rank.py`, `config.py`, `improve/*`, `semantic/seed.py`, `semantic/loo.py`, `benchmarks/pairs/harvest.py`, `benchmarks/pairs/mechanisms.toml` as listed above.
|
||||
|
||||
## System Flows
|
||||
@@ -177,7 +177,7 @@ flowchart LR
|
||||
I --> J[StaticFinding suspicion; tags obligation:*, discharger:*, obligation_evidence:*]
|
||||
```
|
||||
|
||||
Gating: the checker emits a finding only when (1) the operation is reached — from an entry point along a `PathRecord`, or, without paths, when the operation sits in a handler that is itself an entry point — and (2) no discharger of the required kind dominates, and (3) the obligation is justified by a sibling set (`consistency_violation`), a policy clause (`declared_policy_violation`) or, in degraded mode without either, by the operation fact alone (`function_local`). A declared public route suppresses path-guard obligations for that route only.
|
||||
Gating: the checker emits a finding only when (1) the operation is reached (from an entry point along a `PathRecord`, or, without paths, when the operation sits in a handler that is itself an entry point) and (2) no discharger of the required kind dominates, and (3) the obligation is justified by a sibling set (`consistency_violation`), a policy clause (`declared_policy_violation`) or, in degraded mode without either, by the operation fact alone (`function_local`). A declared public route suppresses path-guard obligations for that route only.
|
||||
|
||||
### Teaching and measuring
|
||||
|
||||
|
||||
@@ -8,9 +8,9 @@ AppSec engineers and maintainers of OpenUltraSAST scan web code, increasingly vi
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: obligated operations and discharger kinds as closed facts data; obligation shapes learned from trusted pairs; consistency detection across sibling handlers; an optional declared policy file with a closed schema; path-aware violation evaluation with a function-local degraded mode; obligation evidence rungs, report wording and worth-fixing weighting; a corpus recipe shape that keeps the handler's registration context; leave-one-out per obligation kind; an `obligations` improve lever; offline tests.
|
||||
- **Out of scope**: flow taint, walker or sink facts; the call graph, path records and dominance computation (owned by `reachability-flow-model` and `guard-dominance-regime`); model-authored policy or shapes; sandbox proof of absence bugs; any merge gate; vendoring unlicensed code.
|
||||
- **Adjacent expectations**: the entry-point mapper keeps providing routes with an access level and their decorators; the reachability spec provides entry-point-to-operation paths and the guard-dominance spec provides "a guard dominates the path"; the mechanism store, its deterministic ids and the lever contract from `corpus-seeded-mechanisms` are reused unchanged; the pair scorer accepts one additive label field; `local` remains the only slice that can fail CI.
|
||||
- In scope: obligated operations and discharger kinds as closed facts data; obligation shapes learned from trusted pairs; consistency detection across sibling handlers; an optional declared policy file with a closed schema; path-aware violation evaluation with a function-local degraded mode; obligation evidence rungs, report wording and worth-fixing weighting; a corpus recipe shape that keeps the handler's registration context; leave-one-out per obligation kind; an `obligations` improve lever; offline tests.
|
||||
- Out of scope: flow taint, walker or sink facts; the call graph, path records and dominance computation (owned by `reachability-flow-model` and `guard-dominance-regime`); model-authored policy or shapes; sandbox proof of absence bugs; any merge gate; vendoring unlicensed code.
|
||||
- Adjacent expectations: the entry-point mapper keeps providing routes with an access level and their decorators; the reachability spec provides entry-point-to-operation paths and the guard-dominance spec provides "a guard dominates the path"; the mechanism store, its deterministic ids and the lever contract from `corpus-seeded-mechanisms` are reused unchanged; the pair scorer accepts one additive label field; `local` remains the only slice that can fail CI.
|
||||
|
||||
## Requirements
|
||||
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
## Summary
|
||||
|
||||
- **Feature**: `authorization-obligations`
|
||||
- **Discovery Scope**: Extension of the semantic layer (facts, shapes, store, MAP proposers), the entry-point mapper, reports, the pair corpus, leave-one-out and the improve lever; complex integration because it consumes two unimplemented upstream specs.
|
||||
- **Key Findings**:
|
||||
- Feature: `authorization-obligations`
|
||||
- Discovery Scope: Extension of the semantic layer (facts, shapes, store, MAP proposers), the entry-point mapper, reports, the pair corpus, leave-one-out and the improve lever; complex integration because it consumes two unimplemented upstream specs.
|
||||
- Key Findings:
|
||||
- The absence pairs are real and structured. `vampi-books-get-by-title` has identical code on both sides except a constraint `user=user` bound from `token_validator(request.headers.get('Authorization'))['sub']`; the vulnerable branch queries `Book` by title alone. The difference is the provenance of a constraint, not a flow. 15 of 29 `agent-vfc` rows and 6 of the 19 non-teaching `vibe-py` pairs carry absence labels (`missing_auth_guard`, `permissive_default`, `identity_from_request_body`, `secret_in_client`, `validation_strength`).
|
||||
- The entry-point mapper already classifies every route as `public`, `authenticated` or `role-restricted` from its decorators (`mapping._python_route_access`), records the decorators as `access_evidence`, and lists `conditions` (`if` tests that look like checks). That is the sibling detector's upstream for path guards and needs no change.
|
||||
- `FileIR` binds carry `value_text`, `names` and `call_name`; `variants.source_kind_of_name` already traces a name through binds to a fact source, a parameter or a container read. Provenance of a constraint's value (authenticated context versus request input versus constant) is the same trace with one more source class.
|
||||
@@ -17,29 +17,29 @@
|
||||
|
||||
### What the absence pairs actually differ by
|
||||
|
||||
- **Context**: Decide whether absence bugs can be a shape family at all.
|
||||
- **Sources Consulted**: `benchmarks/pairs/vibe-py` excerpts for `missing_auth_guard` and `permissive_default` rows; `benchmarks/pairs/agent-vfc/recipes.toml` mechanisms; `mechanisms.toml` definitions.
|
||||
- **Findings**: three recurring differences: (a) a constraint added to a data access whose value comes from the authenticated context; (b) a guard call or decorator added before the operation (early return on a failed token check, `login_required`); (c) a literal changed from permissive to restrictive (`CORS(app)` → origins list, `DEBUG=True` → `False`, RLS enabled). A fourth, rarer: identity read from the body replaced by identity from the token (same as (a) seen from the other side).
|
||||
- **Implications**: discharger kinds `identity_constraint`, `path_guard`, `ownership_check`, `non_permissive_value`, `validated_input`; the value provenance is the discriminating feature for (a) and (d).
|
||||
- Context: Decide whether absence bugs can be a shape family at all.
|
||||
- Sources Consulted: `benchmarks/pairs/vibe-py` excerpts for `missing_auth_guard` and `permissive_default` rows; `benchmarks/pairs/agent-vfc/recipes.toml` mechanisms; `mechanisms.toml` definitions.
|
||||
- Findings: three recurring differences: (a) a constraint added to a data access whose value comes from the authenticated context; (b) a guard call or decorator added before the operation (early return on a failed token check, `login_required`); (c) a literal changed from permissive to restrictive (`CORS(app)` → origins list, `DEBUG=True` → `False`, RLS enabled). A fourth, rarer: identity read from the body replaced by identity from the token (same as (a) seen from the other side).
|
||||
- Implications: discharger kinds `identity_constraint`, `path_guard`, `ownership_check`, `non_permissive_value`, `validated_input`; the value provenance is the discriminating feature for (a) and (d).
|
||||
|
||||
### Where the policy comes from
|
||||
|
||||
- **Context**: Absence is relative to intent; the tool must not invent intent.
|
||||
- **Sources Consulted**: `mapping.analyze_entry_points` (routes, access level, evidence), Real-Vuln repositories (many handlers per resource), Clearwing's sibling-comparison practice.
|
||||
- **Findings**: in every Real-Vuln web app the handlers on one resource share a router and a model; the vulnerable handler is the odd one out. Route access classification already exists. A declared policy is the only way to say "this route is meant to be public"; the hunter can only guess it.
|
||||
- **Implications**: consistency first (no declaration needed, measurable now), declared policy second (stronger rung), hunter last (records intent, never raises).
|
||||
- Context: Absence is relative to intent; the tool must not invent intent.
|
||||
- Sources Consulted: `mapping.analyze_entry_points` (routes, access level, evidence), Real-Vuln repositories (many handlers per resource), Clearwing's sibling-comparison practice.
|
||||
- Findings: in every Real-Vuln web app the handlers on one resource share a router and a model; the vulnerable handler is the odd one out. Route access classification already exists. A declared policy is the only way to say "this route is meant to be public"; the hunter can only guess it.
|
||||
- Implications: consistency first (no declaration needed, measurable now), declared policy second (stronger rung), hunter last (records intent, never raises).
|
||||
|
||||
### Dominance before guard-dominance-regime exists
|
||||
|
||||
- **Context**: Req 5 needs "a discharger dominates the path"; the spec that owns dominance is not written.
|
||||
- **Findings**: within one handler, the existing `FunctionIR` order of binds and calls gives statement order; a guard call followed by an early return before the operation line is dominance in the common case; decorators dominate the whole handler; middleware attached to the router dominates every route under it. Across calls, `PathRecord.hops` gives the sequence of nodes; a discharger on any hop before the operation node dominates the path when the hop's function returns early on failure.
|
||||
- **Implications**: define `Dominance` as a narrow protocol (`dominates(witness, operation) -> bool`) with a built-in `OrderDominance` implementation (statement order in the handler, decorators, router middleware, hop order) that `guard-dominance-regime` may replace with nesting-aware dominance. The degraded mode (`function_local`) is `OrderDominance` without path records.
|
||||
- Context: Req 5 needs "a discharger dominates the path"; the spec that owns dominance is not written.
|
||||
- Findings: within one handler, the existing `FunctionIR` order of binds and calls gives statement order; a guard call followed by an early return before the operation line is dominance in the common case; decorators dominate the whole handler; middleware attached to the router dominates every route under it. Across calls, `PathRecord.hops` gives the sequence of nodes; a discharger on any hop before the operation node dominates the path when the hop's function returns early on failure.
|
||||
- Implications: define `Dominance` as a narrow protocol (`dominates(witness, operation) -> bool`) with a built-in `OrderDominance` implementation (statement order in the handler, decorators, router middleware, hop order) that `guard-dominance-regime` may replace with nesting-aware dominance. The degraded mode (`function_local`) is `OrderDominance` without path records.
|
||||
|
||||
### Store and lever reuse
|
||||
|
||||
- **Context**: Avoid a second store and a second lever machinery.
|
||||
- **Findings**: `Mechanism.shape` is a dict validated where used; `corpus_mechanism_id` hashes `shape.key()`; `validate_mechanism` checks guard and source kinds against `variants` constants. A second family needs its own closed kinds check and a `family` field in the shape dict; the tombstone, snapshot and profile clause need nothing.
|
||||
- **Implications**: `ObligationShape.to_dict()` carries `family = "obligation"`; the lever validator dispatches on `family`; `search_tree` ignores obligation rows (they are not sink shapes) and the obligation checker ignores sink rows.
|
||||
- Context: Avoid a second store and a second lever machinery.
|
||||
- Findings: `Mechanism.shape` is a dict validated where used; `corpus_mechanism_id` hashes `shape.key()`; `validate_mechanism` checks guard and source kinds against `variants` constants. A second family needs its own closed kinds check and a `family` field in the shape dict; the tombstone, snapshot and profile clause need nothing.
|
||||
- Implications: `ObligationShape.to_dict()` carries `family = "obligation"`; the lever validator dispatches on `family`; `search_tree` ignores obligation rows (they are not sink shapes) and the obligation checker ignores sink rows.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
@@ -54,23 +54,23 @@
|
||||
|
||||
### Decision: Obligation findings are `suspicion` with a data label, not new ladder rungs
|
||||
|
||||
- **Context**: Req 6.1 names three labels; the ladder is closed and owned elsewhere.
|
||||
- **Selected Approach**: `StaticFinding.evidence_level = "suspicion"`, tags `obligation:<kind>`, `discharger:<kind>`, `obligation_evidence:<label>`; reports render the label. Dispositions and `ALLOWED_EVIDENCE` unchanged.
|
||||
- **Trade-offs**: SARIF consumers see `suspicion` plus properties; honest and non-invasive.
|
||||
- Context: Req 6.1 names three labels; the ladder is closed and owned elsewhere.
|
||||
- Selected Approach: `StaticFinding.evidence_level = "suspicion"`, tags `obligation:<kind>`, `discharger:<kind>`, `obligation_evidence:<label>`; reports render the label. Dispositions and `ALLOWED_EVIDENCE` unchanged.
|
||||
- Trade-offs: SARIF consumers see `suspicion` plus properties; honest and non-invasive.
|
||||
|
||||
### Decision: Dominance is a protocol with an order-based default
|
||||
|
||||
- **Context**: `guard-dominance-regime` is unwritten; this spec cannot wait for it to be useful.
|
||||
- **Selected Approach**: `Dominance` protocol; `OrderDominance` default (decorator, router middleware, statement order before the operation, hop order along a path). The guard spec replaces it without touching callers.
|
||||
- **Trade-offs**: order-based dominance misses guards inside conditionals; recorded as the known coarseness and measured on the fixed twins.
|
||||
- Context: `guard-dominance-regime` is unwritten; this spec cannot wait for it to be useful.
|
||||
- Selected Approach: `Dominance` protocol; `OrderDominance` default (decorator, router middleware, statement order before the operation, hop order along a path). The guard spec replaces it without touching callers.
|
||||
- Trade-offs: order-based dominance misses guards inside conditionals; recorded as the known coarseness and measured on the fixed twins.
|
||||
|
||||
### Decision: Policy is never written by the tool
|
||||
|
||||
- **Selected Approach**: `declared policy` is a hand-maintained file with a closed schema; the lever may only *propose* a clause, journaled as pending until a human accepts; scans never read pending clauses.
|
||||
- Selected Approach: `declared policy` is a hand-maintained file with a closed schema; the lever may only *propose* a clause, journaled as pending until a human accepts; scans never read pending clauses.
|
||||
|
||||
### Decision: Resource identity for sibling grouping
|
||||
|
||||
- **Selected Approach**: a handler's resource is the trailing segment of its route path with parameters stripped, joined with the operation's target token (model, table, collection name from the operation fact match); router or module is the second key. Coarse but deterministic and text-free.
|
||||
- Selected Approach: a handler's resource is the trailing segment of its route path with parameters stripped, joined with the operation's target token (model, table, collection name from the operation fact match); router or module is the second key. Coarse but deterministic and text-free.
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Problem
|
||||
|
||||
`learning-harness` built the measurement layer honestly — a closed family taxonomy, a class-aware pair
|
||||
`learning-harness` built an honest measurement layer with a closed family taxonomy, a class-aware pair
|
||||
scorer, per-family detector configurations, a split enforced by construction, verifier-gated reporting,
|
||||
and a journal with attribution. What it did **not** build is a search. `run_learning_round` is a
|
||||
hand-rolled hill-climber: one proposal, one minibatch, one holdout check, a threshold rule, accept or
|
||||
@@ -28,26 +28,26 @@ Round zero on vibe-py, `deepseek-v4-flash`, K = 5, $0.69, 23/30 pair-correct:
|
||||
|
||||
Four defects follow from the loop's shape, not from its bugs:
|
||||
|
||||
1. **The acceptance rule cannot discriminate.** `decide` compares raw deltas: any regression on train or
|
||||
1. The acceptance rule cannot discriminate. `decide` compares raw deltas: any regression on train or
|
||||
holdout rejects, a tie rejects. The measured floor says 11 of 20 injection pairs disagree with
|
||||
*themselves* across five runs, so a delta over a minibatch of 8 is mostly noise. The spec built
|
||||
`sign_test` and `reliable_change` for exactly this and the rule calls neither.
|
||||
2. **The holdout is an input to the decision.** `decide` reads `holdout_delta`. Req 5.4 keeps the
|
||||
2. The holdout is an input to the decision. `decide` reads `holdout_delta`. Req 5.4 keeps the
|
||||
*proposer* blind to the holdout, and then the *selector* reads it every round. Repeated rounds
|
||||
selecting on holdout performance is search-time overfitting of the test set — the same leak class
|
||||
selecting on holdout performance is search-time overfitting of the test set, the same leak class
|
||||
this project has already measured once, one level up.
|
||||
3. **Noise is fought with repetition instead of diversity.** K = 5 spends five rollouts per pair per
|
||||
3. Noise is fought with repetition instead of diversity. K = 5 spends five rollouts per pair per
|
||||
side estimating variance. The published alternative spends that budget on more candidates and lets a
|
||||
frontier keep whatever wins somewhere.
|
||||
4. **The proposer is told a score, not what went wrong.** `FailureFacts` carries outcomes, run strings
|
||||
4. The proposer is told a score, not what went wrong. `FailureFacts` carries outcomes, run strings
|
||||
and counts. The single largest lever in the published work is textual feedback from the *evaluation
|
||||
trace* — and our evaluator produces exactly the right material (which finding landed on which line,
|
||||
trace*, and our evaluator produces exactly the right material (which finding landed on which line,
|
||||
whether it fell inside the labeled span, whether the fixed side leaked, which family was claimed)
|
||||
and then throws all of it away.
|
||||
|
||||
## What others built
|
||||
|
||||
**GEPA — Reflective Prompt Evolution (Agrawal et al., arXiv:2507.19457, ICLR 2026 oral).** Algorithm 1,
|
||||
**GEPA, Reflective Prompt Evolution (Agrawal et al., arXiv:2507.19457, ICLR 2026 oral).** Algorithm 1,
|
||||
read from the paper rather than from a summary:
|
||||
|
||||
- Split `D_train` into `D_feedback` and `D_pareto`. Initialise the candidate pool `P = [Φ]` and score
|
||||
@@ -56,7 +56,7 @@ read from the paper rather than from a summary:
|
||||
minibatch drawn from `D_feedback`; collect scores, traces and **textual feedback** via a feedback
|
||||
function `μ_f`; have a reflection LM rewrite that module's prompt; re-run the mutated system on the
|
||||
*same* minibatch.
|
||||
- **Accept if and only if the minibatch average improved.** Only then is the candidate scored on all of
|
||||
- Accept if and only if the minibatch average improved. Only then is the candidate scored on all of
|
||||
`D_pareto` and added to the pool with its parent recorded.
|
||||
- Return the candidate with the best average on `D_pareto`.
|
||||
|
||||
@@ -65,7 +65,7 @@ candidates that achieve it; drop strictly dominated ones; sample a survivor with
|
||||
proportional to the number of instances it leads on.
|
||||
|
||||
Three things about that are directly relevant to us. The acceptance gate is **cheap and noisy on
|
||||
purpose** — a single minibatch comparison — because the frontier, not the gate, is what protects
|
||||
purpose** (a single minibatch comparison) because the frontier, not the gate, is what protects
|
||||
against a bad accept: anything that wins on one instance is retained. Selection is **stochastic and
|
||||
diversity-preserving**, which is what stops the search collapsing onto one lineage. And the expensive
|
||||
full evaluation is spent only on candidates that already passed the cheap gate.
|
||||
@@ -76,20 +76,20 @@ GRPO's 24,000. Sample efficiency is the claim that matters for us, because our r
|
||||
|
||||
**DSPy's optimizer interface.** `MIPROv2` proposes instruction and demonstration candidates and searches
|
||||
them with Bayesian optimisation (Optuna TPE), scoring on minibatches and running a full validation pass
|
||||
on the incumbent every `minibatch_full_eval_steps` — an explicit two-fidelity design for a noisy metric.
|
||||
on the incumbent every `minibatch_full_eval_steps`, an explicit two-fidelity design for a noisy metric.
|
||||
`dspy.GEPA` takes `metric`, `reflection_lm`, a budget (`auto` / `max_full_evals` / `max_metric_calls`),
|
||||
`reflection_minibatch_size`, `candidate_selection_strategy` (`pareto` | `current_best`) and
|
||||
`track_stats`, and its metric returns `Prediction(score, feedback)` rather than a float.
|
||||
|
||||
**SIMBA — Stochastic Introspective Mini-Batch Ascent (`dspy.SIMBA`).** Two of its ideas fit our corpus
|
||||
**SIMBA, Stochastic Introspective Mini-Batch Ascent (`dspy.SIMBA`).** Two of its ideas fit our corpus
|
||||
better than GEPA's do. It runs `num_candidates` (6 by default) program variants per example, buckets the
|
||||
trajectories, and ranks examples by **max-to-min score gap** — deliberately spending the budget on the
|
||||
trajectories, and ranks examples by **max-to-min score gap**, deliberately spending the budget on the
|
||||
instances where variants disagree most. On our corpus that is not a nuisance, it is the training signal:
|
||||
the 11 injection pairs that disagree with themselves are precisely the pairs where a rule change can flip
|
||||
an outcome, and uniform minibatch sampling would spend most of its budget on the 9 that never move. It
|
||||
also alternates two strategies rather than round-robining blindly — `append_a_demo` (a successful
|
||||
also alternates two strategies rather than round-robining blindly, `append_a_demo` (a successful
|
||||
trajectory becomes a demonstration) and `append_a_rule` (the model introspects on the bucket and writes a
|
||||
rule) — which is exactly the split our levers already have between `counterexamples`/`hard_negatives` and
|
||||
rule), which is exactly the split our levers already have between `counterexamples`/`hard_negatives` and
|
||||
`checklist`/`prompt`. Candidates are sampled by softmax over historical scores at temperature 0.2, a
|
||||
cheaper third option beside GEPA's `pareto` and `current_best`.
|
||||
|
||||
@@ -97,16 +97,16 @@ cheaper third option beside GEPA's `pareto` and `current_best`.
|
||||
produce. It shows the reflector the current instruction, then "inputs, outputs and feedback" for the
|
||||
minibatch, and asks it to infer the task, extract "all niche and domain specific factual information ...
|
||||
as a lot of it may not be available to the assistant in the future", and name any generalizable strategy.
|
||||
That is a specification for `μ_f`, not just a prompt: feedback has to carry the domain facts a detector
|
||||
cannot re-derive — which sink, which span, which side leaked — rather than a verdict.
|
||||
That is a prompt and a specification for `μ_f`: feedback has to carry the domain facts a detector
|
||||
cannot re-derive (which sink, which span, which side leaked) rather than a verdict.
|
||||
|
||||
**GEPA's merge** is gated strictly: two candidates are crossed only when they share a common ancestor,
|
||||
optimised *disjoint* sets of prompts, are both Pareto-optimal, and both beat the ancestor's aggregate.
|
||||
Worth having once we have per-lever candidates, and worth not having before then.
|
||||
|
||||
**The standalone `gepa` library** (MIT) exposes `gepa.optimize(seed_candidate, trainset, valset,
|
||||
task_lm, reflection_lm, max_metric_calls)` over a `GEPAAdapter` protocol of three methods —
|
||||
`evaluate`, `make_reflective_dataset`, `propose_new_texts` — with a seed candidate that is just a dict
|
||||
task_lm, reflection_lm, max_metric_calls)` over a `GEPAAdapter` protocol of three methods
|
||||
(`evaluate`, `make_reflective_dataset`, `propose_new_texts`), with a seed candidate that is just a dict
|
||||
of component name to text. That adapter shape is the interface we want whether or not we take the
|
||||
dependency: it is the smallest surface that separates *what a detector is* from *how it is searched*.
|
||||
|
||||
@@ -114,26 +114,26 @@ dependency: it is the smallest surface that separates *what a detector is* from
|
||||
|
||||
Restate the detector as an optimisable program and put a published optimiser behind it.
|
||||
|
||||
- **A detector is a candidate**: a dict of named text components (`prompt`, `checklist`,
|
||||
- A detector is a candidate: a dict of named text components (`prompt`, `checklist`,
|
||||
`hard_negatives`, `counterexamples`) plus its frozen non-text settings (tools, budgets). That is
|
||||
already what `FamilyConfig` holds; it becomes the seed candidate.
|
||||
- **The metric returns a score and a reason.** `score_pair_family` gains a feedback string built from
|
||||
- The metric returns a score and a reason. `score_pair_family` gains a feedback string built from
|
||||
what the evaluation already knows. This is the highest-value change in the spec and it is independent
|
||||
of which optimiser sits on top.
|
||||
- **`D_pareto` is carved out of the train split.** The holdout leaves the acceptance rule entirely and
|
||||
- `D_pareto` is carved out of the train split. The holdout leaves the acceptance rule entirely and
|
||||
is scored once, for reporting, when the search is over. This is a requirements change to Req 9.5 and
|
||||
the most important honesty consequence of the redesign.
|
||||
- **An adapter protocol** (`evaluate`, `make_reflective_dataset`, `propose_new_texts`) with two backends:
|
||||
- An adapter protocol (`evaluate`, `make_reflective_dataset`, `propose_new_texts`) with two backends:
|
||||
our own implementation of Algorithms 1 and 2 by default, so the core install keeps
|
||||
`dependencies = []`, and `gepa` behind an optional extra so its improvements arrive for free.
|
||||
- **The Pareto pool replaces the accept/revert tree.** `Archive` already records per-pair winners; it
|
||||
- The Pareto pool replaces the accept/revert tree. `Archive` already records per-pair winners; it
|
||||
becomes the frontier, and the journal's parent records become the ancestry GEPA needs.
|
||||
|
||||
## Where the noise actually is, and why the architecture may be wrong
|
||||
|
||||
The maintainer's objection, and I think it is correct: the harness treats the detector as a black box
|
||||
emitting a noisy scalar, and everything follows from that — K runs, majority vote, a measured floor, a
|
||||
regression budget, a sign test. That is the stance you take when you cannot see inside the thing you are
|
||||
emitting a noisy scalar, and everything follows from that (K runs, majority vote, a measured floor, a
|
||||
regression budget, a sign test). That is the stance you take when you cannot see inside the thing you are
|
||||
measuring. We can see inside it.
|
||||
|
||||
The variance is not in the judgement. `temperature: 0` is sent and thinking is disabled, and 11 of 20
|
||||
@@ -147,8 +147,8 @@ than the symptom.
|
||||
|
||||
**Constraints as predicates, enforced at generation time** (`dspy.Assert`/`Suggest`, arXiv:2312.13382).
|
||||
A constraint is a boolean function of the output; on failure the past output *and the error* are injected
|
||||
into a retry. We already have the predicates — is the reported line inside a parsed function, does the
|
||||
named sink occur at that line, is the claimed family in the taxonomy, did the run cite a tool result —
|
||||
into a retry. We already have the predicates (is the reported line inside a parsed function, does the
|
||||
named sink occur at that line, is the claimed family in the taxonomy, did the run cite a tool result),
|
||||
and we apply every one of them **after the fact, at scoring time, to decide whether to count a finding**.
|
||||
Applying them at generation time to decide whether to *accept* one is free, needs no dependency, and is
|
||||
the single highest-value change available. The paper reports near-total intrinsic constraint satisfaction
|
||||
@@ -156,10 +156,10 @@ and 5–15% downstream gains.
|
||||
|
||||
**Typed decomposition instead of one free-running loop.** Our detector is a single ReAct loop that decides
|
||||
which files to read, what to grep, when to stop and what to report; every one of those is a variance
|
||||
source. DSPy's thesis is that each step gets a signature — a typed input/output contract — and a little
|
||||
source. DSPy's thesis is that each step gets a signature (a typed input/output contract), and a little
|
||||
control flow composes them. Ours would be roughly: locate candidate sink sites; trace whether an untrusted
|
||||
value reaches each; check for a guard between source and sink; judge. **Several of those steps need no
|
||||
model at all** — the CST, the entry-point mapper, `flows` and `obligations` already answer them
|
||||
model at all**. The CST, the entry-point mapper, `flows` and `obligations` already answer them
|
||||
deterministically, and we are currently paying a model to re-derive facts we hold.
|
||||
|
||||
**Best-of-N against a checkable reward, instead of a majority vote** (`dspy.Refine`, `BestOfN`). Both
|
||||
@@ -170,7 +170,7 @@ have checkable rewards for free. The same budget, used as evidence rather than a
|
||||
**Why this matters more for our models, not less.** GEPA and SIMBA are demonstrated with a strong
|
||||
reflection model over a weaker task model, and their gains come from evolving instructions. A cheap model
|
||||
fails on an under-specified task and does well on a narrow typed one. So the cheaper the detector model,
|
||||
the more of the return sits in decomposition and constraints and the less in prompt evolution — which is
|
||||
the more of the return sits in decomposition and constraints and the less in prompt evolution, which is
|
||||
the opposite of the order this spec originally proposed.
|
||||
|
||||
### The experiment, run (2026-09-06, `benchmarks/measurements/2026-09-06-trajectory-divergence.json`)
|
||||
@@ -178,7 +178,7 @@ the opposite of the order this spec originally proposed.
|
||||
Twelve injection pairs, both sides, K = 5, temperature 0, thinking disabled, recording the tool-call
|
||||
sequence of every run. $0.23.
|
||||
|
||||
**The intended statistic could not be computed. Across 120 runs there was not one repeated trajectory** —
|
||||
**The intended statistic could not be computed. Across 120 runs there was not one repeated trajectory**. There were
|
||||
4.67 distinct trajectories per vulnerable side and 4.92 per fixed side, out of 5, and 78 distinct tool-*name*
|
||||
sequences ignoring arguments entirely. The detector is not sampling one measurement five times. It is running
|
||||
five different investigations and voting on the results.
|
||||
@@ -193,21 +193,21 @@ Asked as a gradient instead, the relationship is monotone and clean:
|
||||
|
||||
Three more facts from the same run:
|
||||
|
||||
- **The variance is a search problem, and the search is `grep_repo`.** 555 grep calls against 181 `read_file`,
|
||||
- The variance is a search problem, and the search is `grep_repo`. 555 grep calls against 181 `read_file`,
|
||||
71 `entry_points`, 43 `find_refs`, 13 `read_definition`. The model spends its budget guessing patterns to
|
||||
locate a sink — while the CST, the entry-point mapper, `flows` and `obligations` already know where the
|
||||
locate a sink, while the CST, the entry-point mapper, `flows` and `obligations` already know where the
|
||||
sinks are.
|
||||
- **All of the instability is on the vulnerable side.** Every one of the twelve pairs produced exactly one
|
||||
- All of the instability is on the vulnerable side. Every one of the twelve pairs produced exactly one
|
||||
distinct finding set on the fixed side. The detector is reliably silent on the fix and unreliably vocal on
|
||||
the bug, so the noise is entirely in detection and none of it in false positives.
|
||||
- **Findings converge far more than trajectories do.** 78 distinct trajectories, but one to three distinct
|
||||
- Findings converge far more than trajectories do. 78 distinct trajectories, but one to three distinct
|
||||
finding sets per pair. Many different investigations reach the same answer; six of twelve pairs are stable
|
||||
despite never once repeating a trajectory.
|
||||
|
||||
**What this does to the noise architecture.** The floor is not measuring the detector's uncertainty about a
|
||||
question. It is measuring how often an unconstrained search happens to reach the sink. A majority vote over
|
||||
five runs is therefore not variance reduction — it is an unweighted ensemble of five arbitrary search
|
||||
procedures — and a regression budget derived from it is a budget for the search's luck. K runs, the floor,
|
||||
five runs is therefore an unweighted ensemble of five arbitrary search
|
||||
procedures rather than variance reduction, and a regression budget derived from it is a budget for the search's luck. K runs, the floor,
|
||||
the budget and the sign test are all statistical treatment for a structural problem, and the 0% and 4.2%
|
||||
rows of that table say what happens when the structure is fixed instead.
|
||||
|
||||
@@ -224,45 +224,45 @@ answer "is our own instability structural?" in twenty minutes is the thing that
|
||||
|
||||
**Was answering the wrong question.**
|
||||
|
||||
- **Req 3.5, K ≥ 3 runs per pair.** The premise is variance reduction over a repeated measurement. There is
|
||||
- Req 3.5, K ≥ 3 runs per pair. The premise is variance reduction over a repeated measurement. There is
|
||||
no repeated measurement: 120 runs, 120 trajectories. K runs is an *ensemble of distinct search
|
||||
procedures*, and a majority vote over it is an unweighted ensemble, not an estimate. It should either be
|
||||
renamed and published as an ensemble — which changes what every family number means — or dropped once the
|
||||
renamed and published as an ensemble, which changes what every family number means, or dropped once the
|
||||
search is constrained.
|
||||
- **Req 8.2, the noise floor as the regression budget.** The floor measures how often an unconstrained
|
||||
- Req 8.2, the noise floor as the regression budget. The floor measures how often an unconstrained
|
||||
search happens to reach the sink. Deriving a regression budget from it means the worse the search wanders,
|
||||
the more collateral damage a round is licensed to do. That is backwards, and it is why injection's budget
|
||||
is 11 of 20.
|
||||
- **Req 9.5, the acceptance rule.** Already flagged for reading the holdout every round. Now also: it
|
||||
- Req 9.5, the acceptance rule. Already flagged for reading the holdout every round. Now also: it
|
||||
compares deltas between two ensemble votes, and calls the difference an improvement.
|
||||
- **The sign test and `reliable_change`.** Added this session in good faith; they test whether an ensemble's
|
||||
- The sign test and `reliable_change`. Added this session in good faith; they test whether an ensemble's
|
||||
vote moved, which is not the question either.
|
||||
|
||||
**What the data adds that neither spec anticipated.**
|
||||
|
||||
- **The instability is entirely one-sided.** All twelve pairs produced exactly one finding set on the fixed
|
||||
- The instability is entirely one-sided. All twelve pairs produced exactly one finding set on the fixed
|
||||
side. Specificity is solid and recall is the noisy half, so the leak ceiling and the false-positive
|
||||
machinery are sound and it is the recall numbers that carry the caveat.
|
||||
- **`grep_repo` is the variance engine** — 555 calls against 181 `read_file` and 71 `entry_points`. The
|
||||
- `grep_repo` is the variance engine. It accounts for 555 calls against 181 `read_file` and 71 `entry_points`. The
|
||||
single highest-leverage change in either spec is not a prompt, a lever or an optimiser: **stop asking the
|
||||
model to locate the sink.** The CST, the entry-point mapper, `flows` and `obligations` already know. Give
|
||||
the detector the candidate sites and ask it the question only a model can answer — does untrusted input
|
||||
the detector the candidate sites and ask it the question only a model can answer. Does untrusted input
|
||||
reach this one, and is there a guard.
|
||||
- **The loop is partly self-correcting**, so this should be a large win rather than a marginal one: 78
|
||||
- The loop is partly self-correcting, so this should be a large win rather than a marginal one: 78
|
||||
distinct trajectories collapse to one to three finding sets per pair, and half the pairs are already
|
||||
stable despite never repeating an investigation.
|
||||
|
||||
**Limits of the experiment, stated so the conclusion is not over-read.** Twelve pairs, one family, one
|
||||
slice, one model. The high-similarity bucket holds only two run pairs, so its 0% is decoration; the
|
||||
defensible number is the mid bucket's 4.2% over 24 run pairs against 30.9% over 94. And the direction of
|
||||
causation is inferred, not established — similar trajectories may be a symptom of easy pairs rather than a
|
||||
causation is inferred, not established. Similar trajectories may be a symptom of easy pairs rather than a
|
||||
cause of agreement. The way to settle that is the constrained detector itself: fix the investigation, and
|
||||
if the floor does not collapse, this reading was wrong.
|
||||
|
||||
## The constraint that outranks the algorithm
|
||||
|
||||
Both algorithms assume a dataset. GEPA splits `D_train` into `D_feedback` and `D_pareto`; SIMBA's default
|
||||
batch size is 32. **Our injection family has 12 train teachers** — fewer than one SIMBA batch, and after a
|
||||
batch size is 32. **Our injection family has 12 train teachers** (fewer than one SIMBA batch), and after a
|
||||
GEPA split roughly six instances on each side. Every other family has one to three.
|
||||
|
||||
So the honest statement of what this spec can and cannot buy: GEPA and SIMBA are sample-efficient in
|
||||
@@ -279,42 +279,42 @@ selection, the budget accounting, and the artifacts that make a run reproducible
|
||||
|
||||
Out, and deliberately: the taxonomy, the classifier, the pair scorer's *outcome* rules, the verifier
|
||||
boundary, the write roots, redaction, the three gates. Those are the measurement, and a search whose
|
||||
objective is not trustworthy is worse than no search. Nothing here relaxes them — in particular the
|
||||
objective is not trustworthy is worse than no search. Nothing here relaxes them. In particular, the
|
||||
proposer stays blind to the holdout, verifier code and gold labels stay outside every write root, and
|
||||
no pattern or policy is written by a model.
|
||||
|
||||
## Decisions for the maintainer
|
||||
|
||||
0. **Is the optimiser even the next thing to build?** Proposed no, and this reverses the order above.
|
||||
0. Is the optimiser even the next thing to build? Proposed no, and this reverses the order above.
|
||||
Constraints at generation time, typed decomposition and best-of-N attack the variance at its source; an
|
||||
optimiser only tunes text around a program whose freedom is the problem. Proposed sequence: measure where
|
||||
the variance is, constrain generation, decompose the loop, then optimise — and at a trainset of twelve the
|
||||
the variance is, constrain generation, decompose the loop, then optimise, and at a trainset of twelve the
|
||||
optimiser that fits is `BootstrapFewShot` (collect traces that scored well, install them as demonstrations
|
||||
in a lever we already have) rather than an evolutionary search with nothing to search over.
|
||||
|
||||
0b. **Adopt DSPy itself?** Proposed no. Our detector is not a DSPy program — it is an agentic tool loop over
|
||||
0b. Adopt DSPy itself? Proposed no. Our detector is not a DSPy program. It is an agentic tool loop over
|
||||
a repository with curated tools, redaction on every prompt, path clamping, a "no findings unless a tool
|
||||
was called" rule and a final-answer turn, several of which are security properties rather than
|
||||
conveniences. Handing that loop to `dspy.ReAct` and the calls to litellm would put them behind someone
|
||||
else's abstraction, and DSPy could only ever be an optional extra under `dependencies = []`, which means
|
||||
two code paths for the thing that *is* the product. What DSPy would genuinely buy — signatures, adapters,
|
||||
multi-module composition — we barely need at one module per family. The optimisers are available without
|
||||
two code paths for the thing that *is* the product. What DSPy would buy (signatures, adapters,
|
||||
multi-module composition), we barely need at one module per family. The optimisers are available without
|
||||
the framework: `gepa` ships standalone under MIT, and SIMBA's two ideas are a hundred lines of logic.
|
||||
Proposed instead: take the interface shape and the algorithms, not the framework.
|
||||
|
||||
1. **Reimplement Algorithms 1 and 2, take `gepa` as an extra, or both?** Both is proposed: ours by
|
||||
1. Reimplement Algorithms 1 and 2, take `gepa` as an extra, or both? Both is proposed: ours by
|
||||
default (zero-dependency, and the security properties are ours to own), `gepa` behind an extra for
|
||||
comparison. The cost is one adapter and one conformance test that both backends must pass.
|
||||
2. **Drop the holdout from the acceptance rule?** Proposed yes, and it amends approved Req 9.5. The
|
||||
2. Drop the holdout from the acceptance rule? Proposed yes, and it amends approved Req 9.5. The
|
||||
holdout becomes what its name says.
|
||||
3. **Sample instances by disagreement (SIMBA) or uniformly (GEPA)?** Proposed SIMBA's ranking. Our measured
|
||||
3. Sample instances by disagreement (SIMBA) or uniformly (GEPA)? Proposed SIMBA's ranking. Our measured
|
||||
floor makes it the better fit, and it turns the noise floor from an obstacle into an instance sampler.
|
||||
|
||||
4. **K = 1 inside the search, K ≥ 3 only for published numbers?** Proposed yes. That is roughly a 5×
|
||||
4. K = 1 inside the search, K ≥ 3 only for published numbers? Proposed yes. That is roughly a 5×
|
||||
budget reclaim spent on more candidates, which is where the sample efficiency comes from.
|
||||
5. **Does the reflection model differ from the detector model?** The published work uses a stronger
|
||||
5. Does the reflection model differ from the detector model? The published work uses a stronger
|
||||
reflector than the task model. We have `deepseek-v4-pro` configured as the judge; proposed to reuse it.
|
||||
6. **Is `access_control` still the family we care most about?** It has one train teacher. Until agent-vfc
|
||||
6. Is `access_control` still the family we care most about? It has one train teacher. Until agent-vfc
|
||||
rows are reviewed above tier `title`, the optimiser cannot work on it, whatever its algorithm.
|
||||
|
||||
## Open questions
|
||||
@@ -322,7 +322,7 @@ no pattern or policy is written by a model.
|
||||
- Does `gepa` run against an OpenAI-shaped base URL that is not OpenAI (our DeepSeek endpoint), and what
|
||||
does it pull in? Both must be checked before the extra is offered, under the no-undeclared-dependency rule.
|
||||
- Our "module" granularity: is a module a family, or a lever within a family? Round-robin over levers is
|
||||
the closer analogue, but the levers are not independent — a checklist edit and a prompt edit interact.
|
||||
the closer analogue, but the levers are not independent. A checklist edit and a prompt edit interact.
|
||||
- The feedback function's failure mode is a reflection model that writes plausible prose about a trace it
|
||||
misread. What check makes a feedback string answerable rather than persuasive?
|
||||
- GEPA's acceptance is a single noisy comparison. Our floor says a single comparison is close to a coin
|
||||
|
||||
@@ -29,10 +29,10 @@ already reaches, while the **semantic IR caps it at 96.6%** with a median of ten
|
||||
|
||||
### This Spec Owns
|
||||
|
||||
- `learning/candidates.py` — what a candidate is, how the IR enumerates one per family shape, and the ceiling report.
|
||||
- `learning/judgments.py` — the typed answer, and the toolless call that produces it.
|
||||
- `learning/predicates.py` — the checks that constrain a claim at generation time and score it afterwards, defined once.
|
||||
- `learning/slices.py` — deciding versus development slices, the overfitting gap, and the cross-slice rejection rule.
|
||||
- `learning/candidates.py`, what a candidate is, how the IR enumerates one per family shape, and the ceiling report.
|
||||
- `learning/judgments.py`, the typed answer, and the toolless call that produces it.
|
||||
- `learning/predicates.py`, the checks that constrain a claim at generation time and score it afterwards, defined once.
|
||||
- `learning/slices.py`, deciding versus development slices, the overfitting gap, and the cross-slice rejection rule.
|
||||
- `run_constrained_detector` in `learning/detectors.py`, beside the existing `run_family_detector`.
|
||||
- `ousast learning candidates`, and a detector selector on `score` and `baseline`.
|
||||
|
||||
@@ -113,16 +113,16 @@ src/openultrasast/learning/
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `src/openultrasast/learning/detectors.py` — `run_constrained_detector(root, region, config, *, client, model,
|
||||
- `src/openultrasast/learning/detectors.py`, `run_constrained_detector(root, region, config, *, client, model,
|
||||
taxonomy, ranges)`; `FamilyConfig` gains nothing, because a constrained detector is configured by the same
|
||||
prompt/checklist/limits and reads no tools.
|
||||
- `src/openultrasast/learning/scoring.py` — `_hits`, `_others` and `_fabricated` delegate to `predicates`; the
|
||||
- `src/openultrasast/learning/scoring.py`, `_hits`, `_others` and `_fabricated` delegate to `predicates`; the
|
||||
family metrics gain nothing. A test asserts the recorded baseline numbers are reproduced exactly.
|
||||
- `src/openultrasast/learning/publish.py` — the overfitting gap and the per-slice table (Req 3.1, 3.4).
|
||||
- `src/openultrasast/config.py` — `[learning] detector = "hunter" | "constrained"`, default `hunter` until the
|
||||
- `src/openultrasast/learning/publish.py`, the overfitting gap and the per-slice table (Req 3.1, 3.4).
|
||||
- `src/openultrasast/config.py`, `[learning] detector = "hunter" | "constrained"`, default `hunter` until the
|
||||
comparison says otherwise.
|
||||
- `src/openultrasast/cli.py` — `learning candidates`; `--detector` on `score` and `baseline`.
|
||||
- `pyproject.toml` — unchanged.
|
||||
- `src/openultrasast/cli.py`, `learning candidates`; `--detector` on `score` and `baseline`.
|
||||
- `pyproject.toml`, unchanged.
|
||||
|
||||
## System Flows
|
||||
|
||||
@@ -282,13 +282,13 @@ def teachable(slice_name: str, cases) -> bool # False while every row of a
|
||||
|
||||
## Migration and Phases
|
||||
|
||||
1. **Candidates and the ceiling, offline.** `candidates.py`, `learning candidates`, the per-slice ceiling artifact
|
||||
1. Candidates and the ceiling, offline. `candidates.py`, `learning candidates`, the per-slice ceiling artifact
|
||||
including the real-world slices (Req 3.5). No model call, so the phase is free and its result can still cancel
|
||||
the design — if the ceiling on a deciding slice is below that slice's current detection rate, stop here.
|
||||
2. **Predicates extracted.** `predicates.py`, `scoring.py` delegating, the baseline-reproduction test.
|
||||
3. **Judgment and the constrained detector.** `judgments.py`, `run_constrained_detector`, the determinism test.
|
||||
4. **Slices and reporting.** `slices.py`, the overfitting gap, `publish` per slice.
|
||||
5. **The comparison.** Constrained against unconstrained on every slice, the re-measured trajectory count and noise
|
||||
the design. If the ceiling on a deciding slice is below that slice's current detection rate, stop here.
|
||||
2. Predicates extracted. `predicates.py`, `scoring.py` delegating, the baseline-reproduction test.
|
||||
3. Judgment and the constrained detector. `judgments.py`, `run_constrained_detector`, the determinism test.
|
||||
4. Slices and reporting. `slices.py`, the overfitting gap, `publish` per slice.
|
||||
5. The comparison. Constrained against unconstrained on every slice, the re-measured trajectory count and noise
|
||||
floor, and the named `learning-harness` amendments if the floor collapses.
|
||||
|
||||
Phase 1 is deliberately able to kill the design before any money is spent on it.
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
## Introduction
|
||||
|
||||
The detector's instability is structural, not statistical. Measured on 2026-09-06 over twelve injection pairs at
|
||||
temperature 0 with thinking disabled, **120 runs produced 120 distinct tool-call trajectories** — not one repeat —
|
||||
temperature 0 with thinking disabled, **120 runs produced 120 distinct tool-call trajectories**, not one repeat,
|
||||
and outcome disagreement tracks trajectory divergence (30.9% at low similarity against 4.2% at mid). The model
|
||||
spends its budget searching: 555 `grep_repo` calls against 71 `entry_points`, guessing where sinks are while the
|
||||
semantic IR already knows. So `learning-harness`'s K runs are an ensemble of distinct search procedures rather than
|
||||
@@ -20,22 +20,22 @@ ruleset caps recall at 12.1%, below the 64.9% the unconstrained loop already rea
|
||||
at 96.6% with a median of ten sites per labeled function.
|
||||
|
||||
Baseline to beat, `deepseek-v4-flash`, K = 5, 57 scorable web pairs: **detection 64.9%, leak 12.3%, pair-correct
|
||||
52.6%** — and 83.3% detection on hand-written vibe-py against 44.4% on agent-written agent-vfc, which is the gap
|
||||
52.6%**, and 83.3% detection on hand-written vibe-py against 44.4% on agent-written agent-vfc, which is the gap
|
||||
this feature must close rather than widen. Context and measurements: `brief.md`,
|
||||
`benchmarks/measurements/2026-09-06-trajectory-divergence.json`, `2026-09-06-candidate-ceiling.json`.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: candidate enumeration from the semantic IR, per family shape; the typed, toolless judgment step;
|
||||
- In scope: candidate enumeration from the semantic IR, per family shape; the typed, toolless judgment step;
|
||||
constraint predicates enforced at generation time with a bounded retry; the evidence that decides whether the
|
||||
K-run and noise-floor machinery still earns its place; per-slice reporting that keeps real-world code in front.
|
||||
- **Out of scope**: the family taxonomy, the classifier, the pair scorer's outcome rules, the split, the verifier
|
||||
boundary and write roots, redaction, the three gates, and the publication machinery — all of which stand and are
|
||||
- Out of scope: the family taxonomy, the classifier, the pair scorer's outcome rules, the split, the verifier
|
||||
boundary and write roots, redaction, the three gates, and the publication machinery, all of which stand and are
|
||||
what made this feature's measurements possible. Also out: prompt optimisation, reflective evolution and any
|
||||
optimiser (deferred to a later spec, on this feature's evidence); finding bugs in JavaScript interpreters, WebGL
|
||||
renderers or browser engines; memory-safety fuzzing for C and C++; fine-tuning model weights; adopting DSPy as a
|
||||
dependency.
|
||||
- **Adjacent expectations**: `learning-harness` keeps owning the taxonomy, the scorer, the split and publication;
|
||||
- Adjacent expectations: `learning-harness` keeps owning the taxonomy, the scorer, the split and publication;
|
||||
its Req 3.5 (K runs), 8.2 (noise floor) and 9.5 (acceptance) are amended by this feature only on evidence it
|
||||
produces, never on argument. The pair corpus keeps owning labels, tiers and splits. The unconstrained tool hunter
|
||||
stays as the generalist and as the baseline this feature is measured against.
|
||||
@@ -49,8 +49,8 @@ model stops searching and the limit of what it can possibly find is a number rat
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Constrained Detector shall enumerate candidate sites for a region from the semantic IR — call sites and
|
||||
bindings — and shall not use the pattern ruleset as its candidate generator.
|
||||
1. The Constrained Detector shall enumerate candidate sites for a region from the semantic IR (call sites and
|
||||
bindings), and shall not use the pattern ruleset as its candidate generator.
|
||||
2. The Constrained Detector shall shape candidates by family: a call site for injection, path, deserialization,
|
||||
untrusted destination and prototype; an operation on a protected resource with the guards in scope for access
|
||||
control; a configuration assignment or literal for permissive configuration and secrets.
|
||||
@@ -90,13 +90,13 @@ teaching examples, so that the tool is not tuned for cases nobody has.
|
||||
|
||||
1. The Constrained Detector shall report every number per slice and shall never publish an aggregate across slices
|
||||
without the per-slice numbers beside it.
|
||||
2. The Constrained Detector shall designate the slices harvested from real projects — agent-vfc, vfc-js, vfc and
|
||||
github — as the deciding slices, and the hand-written vibe-py slice as a development slice that cannot on its
|
||||
2. The Constrained Detector shall designate the slices harvested from real projects (agent-vfc, vfc-js, vfc and
|
||||
github) as the deciding slices, and the hand-written vibe-py slice as a development slice that cannot on its
|
||||
own justify accepting a change.
|
||||
3. If a change improves the development slice and regresses any deciding slice, then the Constrained Detector shall
|
||||
reject it, and the rejection shall name the slice that regressed.
|
||||
4. The Constrained Detector shall publish the **overfitting gap** — development-slice detection minus deciding-slice
|
||||
detection — beside every headline number, and shall state that the baseline gap is 83.3% against 44.4%.
|
||||
4. The Constrained Detector shall publish the **overfitting gap** (development-slice detection minus deciding-slice
|
||||
detection) beside every headline number, and shall state that the baseline gap is 83.3% against 44.4%.
|
||||
5. The Constrained Detector shall measure and publish the candidate ceiling separately for the real-world slices,
|
||||
because the functions there are larger than the teaching corpus's and the enumerator's behaviour on them is not
|
||||
implied by its behaviour on vibe-py.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Brief: contributor-scan — make the model layer a tool someone runs
|
||||
# Brief: contributor-scan, make the model layer a tool someone runs
|
||||
|
||||
**Status:** brief only, unapproved. Successor to `model-grounded-detection`, which is complete (16/16 tasks)
|
||||
and is not reopened by this.
|
||||
@@ -22,7 +22,7 @@ is wired. The spec is complete and the feature is unshipped, both at once.
|
||||
|
||||
> "a real world tool that helps developers when they want to contribute"
|
||||
|
||||
That reframes the work. A contributor does not have a labelled function and a known weakness class — they
|
||||
That reframes the work. A contributor does not have a labelled function and a known weakness class; they
|
||||
have a repository they are about to touch and a question about what is risky in it. Everything measured so
|
||||
far is function-level pairs: a labelled function, a declared family, a known target. None of it exercises the
|
||||
questions a whole repository asks.
|
||||
@@ -33,11 +33,11 @@ questions a whole repository asks.
|
||||
Without this nothing else matters.
|
||||
2. **Whole-repo scanning.** Region selection from entry points instead of a labelled function; ranking across
|
||||
thousands of candidates rather than one; a budget policy (the enumerator caps at 8 candidates per region,
|
||||
which is meaningless at repo scale); and the performance envelope — a CPG build on a 100k-line tree, not a
|
||||
which is meaningless at repo scale); and the performance envelope: a CPG build on a 100k-line tree, not a
|
||||
40-line excerpt. **This has never been run.**
|
||||
3. **A known-vulnerable repository as the target.** The 176 real-world CVE pairs already in the corpus are
|
||||
single-function excerpts and do not exercise any of the above. Running against a real checkout — curl,
|
||||
libpng, a WordPress plugin — is the only way to find what breaks at scale.
|
||||
single-function excerpts and do not exercise any of the above. Running against a real checkout (curl,
|
||||
libpng, a WordPress plugin) is the only way to find what breaks at scale.
|
||||
4. **The memory decision.** 173 of those 176 pairs are `memory` family, routed to a deferred execution tier
|
||||
that is a stub. Either adopt Clearwing for them or scope C memory-safety out explicitly. Right now it is
|
||||
neither, and a sixth of the corpus sits idle.
|
||||
@@ -49,12 +49,12 @@ questions a whole repository asks.
|
||||
These were delivered during `model-grounded-detection` without being in any spec. They belong here so the
|
||||
record is not scattered:
|
||||
|
||||
- **The pipeline itself** (`model/pipeline.py`) — Req 8 existed, task 3.2 built `judge()`, and nothing ever
|
||||
- The pipeline itself (`model/pipeline.py`): Req 8 existed, task 3.2 built `judge()`, and nothing ever
|
||||
said to connect an enumerator to it. Built, tested, measured, unspecified.
|
||||
- **Ops** — a single image carrying tool and engine, a compose file with read-only target and tmpfs scratch,
|
||||
- Ops: a single image carrying tool and engine, a compose file with read-only target and tmpfs scratch,
|
||||
bash/zsh/PowerShell wrappers that make the container invisible, and a pyinfra deploy for host installs.
|
||||
The Docker path is written and syntax-validated but **the image has never been built**.
|
||||
- **Corpus** — the `owasp` slice (40 validated xss pairs; `output_encoding` went from 1 pair to 41) and
|
||||
- Corpus: the `owasp` slice (40 validated xss pairs; `output_encoding` went from 1 pair to 41) and
|
||||
authored sink/sanitizer facts for Java and JavaScript.
|
||||
|
||||
## Known limits to carry forward
|
||||
@@ -63,7 +63,7 @@ record is not scattered:
|
||||
`debug=True` is dangerous where `httponly=True` is safe and the model cannot tell them apart.
|
||||
- Interprocedural resolution is the most repeated miss cause across every experiment: OWASP's helper classes,
|
||||
`angular-http-server`'s `resolveUrl`. Joern supports it; our queries may not be configured for it.
|
||||
- Five bugs in `model-grounded-detection` shared one shape — a taint-specific assumption applied to every
|
||||
- Five bugs in `model-grounded-detection` shared one shape: a taint-specific assumption applied to every
|
||||
family, each found by a measurement rather than a check. Any new family-specific path should be audited for
|
||||
siblings before it is measured, not after.
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Task 2.19 — measured input retention and census completeness
|
||||
# Task 2.19, measured input retention and census completeness
|
||||
|
||||
The maintainer accepted the proposed contributor-scan follow-up after the pre-push
|
||||
NO-GO report. The Kiro root-cause-first workflow identified a concrete shared-boundary
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Design Document: contributor-scan
|
||||
|
||||
## Integration amendment — 2026-09-12 (draft, unapproved)
|
||||
## Integration amendment, 2026-09-12 (draft, unapproved)
|
||||
|
||||
The maintainer has clarified the product as an actionable pre-push safety net across supported
|
||||
ecosystems. The proposed successor `../pre-push-safety-net/design.md` consumes this driver/backend
|
||||
@@ -9,7 +9,7 @@ of both targets and graphs. Existing approvals below apply to the prior design,
|
||||
|
||||
Proposed shared seams needing requirements/design review before implementation:
|
||||
|
||||
- Accept an absolute deadline across graph build, overlays, retries and queries, not only region/call caps.
|
||||
- Accept an absolute deadline across graph build, overlays, retries and queries as well as region/call caps.
|
||||
- Accept an optional validated reusable graph lease while preserving ordinary fresh-build scans;
|
||||
backend remains authoritative for artifact identity, census, compatibility and cleanup.
|
||||
- Return exact selected/deferred question IDs and context boundaries from ranker decisions;
|
||||
@@ -45,13 +45,13 @@ for someone who does not have Joern.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- New arbiters or families — the three forms exist and are measured.
|
||||
- New arbiters or families; the three forms exist and are measured.
|
||||
- Any optimiser, prompt search, or K-run averaging. The predecessor deleted that machinery; this does not
|
||||
reintroduce it by another name.
|
||||
- Building a sandbox. Req 10 of the predecessor stands: adopt, never build.
|
||||
- Changing the gates, the taxonomy, or the fact-table schema.
|
||||
|
||||
## The fact tables are an ontology, and it is general-purpose — 2026-09-21
|
||||
## The fact tables are an ontology, and it is general-purpose, 2026-09-21
|
||||
|
||||
The `[[source]]`, `[[sink]]`, `[[sanitizer]]`, `[[operation]]`, `[[discharger]]` and `[[layout]]` rows
|
||||
are not a pattern list. They are an ontology of program elements and their security relations: what
|
||||
@@ -87,18 +87,18 @@ repository is in front of the maintainer, which is what task 7.1 exists to make
|
||||
|
||||
### This Spec Owns
|
||||
|
||||
- `model/regions.py` — entry points and file fallbacks to regions, with the family routing for each.
|
||||
- `model/scan.py` — the repository-level driver: rank, budget, dispatch to `model/pipeline`, collect.
|
||||
- `model/regions.py`: entry points and file fallbacks to regions, with the family routing for each.
|
||||
- `model/scan.py`: the repository-level driver: rank, budget, dispatch to `model/pipeline`, collect.
|
||||
- The `Stage.MAP` wiring in `cli.py` and the scan's model-layer payload and report section.
|
||||
- `benchmarks/repos/` — pinned known-vulnerable checkouts and their baselines.
|
||||
- `benchmarks/repos/`: pinned known-vulnerable checkouts and their baselines.
|
||||
- The decisions in Req 7 and their record.
|
||||
|
||||
### Out of Boundary
|
||||
|
||||
- `model/pipeline`, `taint`, `dominance`, `config_value`, `ladder`, `specs`, `candidates` — built, measured,
|
||||
- `model/pipeline`, `taint`, `dominance`, `config_value`, `ladder`, `specs`, `candidates`: built, measured,
|
||||
used unchanged. If this feature needs to change one, that is a signal the change belongs in its own commit
|
||||
with its own measurement.
|
||||
- `mapping.analyze_entry_points`, the overlay, the obligations checker, the gates — untouched.
|
||||
- `mapping.analyze_entry_points`, the overlay, the obligations checker, the gates: untouched.
|
||||
|
||||
### Allowed Dependencies
|
||||
|
||||
@@ -117,8 +117,8 @@ repository is in front of the maintainer, which is what task 7.1 exists to make
|
||||
### Existing Architecture Analysis
|
||||
|
||||
`ousast scan` runs `policy_load → ruleset_load → preprocess → entry_point_mapping → rank → calibrate →
|
||||
STATIC → MAP → REGRESS → report`. Inside MAP it already builds a complexity map, runs the overlay, and — the
|
||||
closest analogue to what we are adding — runs `_check_obligations(root, targets, entries, facts, settings,
|
||||
STATIC → MAP → REGRESS → report`. Inside MAP it already builds a complexity map, runs the overlay, and (the
|
||||
closest analogue to what we are adding) runs `_check_obligations(root, targets, entries, facts, settings,
|
||||
runtime, hunter_model)`, which takes entry points, produces findings, and merges them by re-ranking.
|
||||
|
||||
The model layer slots in exactly there, with the same shape and the same failure discipline. `entry_points`
|
||||
@@ -167,10 +167,10 @@ benchmarks/repos/
|
||||
|
||||
### Changed
|
||||
|
||||
- `cli.py` — a `model` stage inside MAP, mirroring `_check_obligations`; its payload into the manifest and a
|
||||
- `cli.py`: a `model` stage inside MAP, mirroring `_check_obligations`; its payload into the manifest and a
|
||||
report section.
|
||||
- `model/report.py` — a per-rung summary block for a scan (it currently reports per-slice for corpus runs).
|
||||
- `config.py` — a `ModelConfig`-style block: enabled, budget, max regions. Zero-dep, defaults conservative.
|
||||
- `model/report.py`: a per-rung summary block for a scan (it currently reports per-slice for corpus runs).
|
||||
- `config.py`: a `ModelConfig`-style block: enabled, budget, max regions. Zero-dep, defaults conservative.
|
||||
|
||||
### Unchanged and load-bearing
|
||||
|
||||
@@ -194,8 +194,8 @@ class ScanRegion:
|
||||
def regions_for(entries, targets, *, taxonomy, language_of) -> tuple[ScanRegion, ...]
|
||||
```
|
||||
|
||||
Ranking uses what `EntryPointRecord` already carries — an externally-reachable entry point at a trust
|
||||
boundary outranks an internal one — rather than inventing a new signal.
|
||||
Ranking uses what `EntryPointRecord` already carries (an externally-reachable entry point at a trust
|
||||
boundary outranks an internal one) rather than inventing a new signal.
|
||||
|
||||
### model/scan.py
|
||||
|
||||
@@ -264,10 +264,10 @@ each; everything past exhaustion counted into `regions_unjudged`.
|
||||
1. **Wire it (Req 1, 2).** Regions, the MAP stage, findings with rungs reaching a report. Gate-protected.
|
||||
Nothing else matters until a user can see a rung.
|
||||
2. **Scale it (Req 3, 4).** Budget and ranking, then the first run against a pinned known-vulnerable
|
||||
checkout — the measurement that has never been taken and can reshape the rest.
|
||||
checkout, the measurement that has never been taken and can reshape the rest.
|
||||
3. **Ship it (Req 5, 6).** Contributor output; build and verify the image that is currently only defined.
|
||||
4. **Close the deferred (Req 7).** The memory decision; retire `evidence_level`.
|
||||
5. **Grow the corpus (Req 9).** PHP/WordPress first — the families the model arbitrates best. C CVEs only
|
||||
5. **Grow the corpus (Req 9).** PHP/WordPress first: the families the model arbitrates best. C CVEs only
|
||||
after 7.1.
|
||||
6. **Regression baselines (Req 10).**
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Release remediation research — 2026-09-14
|
||||
# Release remediation research, 2026-09-14
|
||||
|
||||
Status: investigation under the maintainer's instruction to continue towards release.
|
||||
This record does not approve rollout, replace the existing design, or mark a production
|
||||
|
||||
@@ -12,7 +12,7 @@ hunter's 76.7% / 12.3%, at a fifth of the model calls. All 16 of its tasks are c
|
||||
from a measurement harness, not from the tool. This feature closes that gap and answers the question the
|
||||
maintainer actually posed: *a real world tool that helps developers when they want to contribute.*
|
||||
|
||||
That framing changes the work. A contributor has no labelled function and no declared weakness class — they
|
||||
That framing changes the work. A contributor has no labelled function and no declared weakness class; they
|
||||
have a repository they are about to touch. Everything measured so far is function-level pairs, which
|
||||
exercise none of the questions a whole repository asks: which regions to look at, how to rank thousands of
|
||||
candidates, what to spend, and whether a CPG build on a 100k-line tree finishes at all.
|
||||
@@ -22,13 +22,13 @@ Context: `brief.md`, `.kiro/specs/model-grounded-detection/`, and the committed
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: wiring `model/pipeline` into `ousast scan`; region selection from entry points; ranking and
|
||||
- In scope: wiring `model/pipeline` into `ousast scan`; region selection from entry points; ranking and
|
||||
budget at repository scale; the performance envelope; running against known-vulnerable checkouts;
|
||||
contributor-facing output; the packaging already built (image, compose, shell wrappers, pyinfra); the
|
||||
deferred memory-family decision; retiring `evidence_level`.
|
||||
- **Out of scope**: new arbiters or families (the three forms exist); prompt optimisation of any kind;
|
||||
building a sandbox (Req 10 of the predecessor stands — adopt, never build); changing the gates.
|
||||
- **Carried forward, not re-litigated**: the evidence ladder, the taxonomy, the fact tables, and the finding
|
||||
- Out of scope: new arbiters or families (the three forms exist); prompt optimisation of any kind;
|
||||
building a sandbox (Req 10 of the predecessor stands: adopt, never build); changing the gates.
|
||||
- Carried forward, not re-litigated: the evidence ladder, the taxonomy, the fact tables, and the finding
|
||||
that the model supplies precision while the LLM supplies recall.
|
||||
|
||||
## Requirements
|
||||
@@ -45,7 +45,7 @@ built is something I can run rather than something a harness demonstrates.
|
||||
2. The system shall report model findings alongside the existing pattern and overlay findings, without
|
||||
removing either.
|
||||
3. When no CPG engine is available, the scan shall behave exactly as it does today and record
|
||||
`cpg_unavailable` — the model layer is additive, never a precondition.
|
||||
`cpg_unavailable`; the model layer is additive, never a precondition.
|
||||
4. The detection, map and local pair gates shall remain byte-identical to the committed baseline.
|
||||
5. The system shall expose the model layer's per-rung counts in the scan's own output, so a user can see how
|
||||
much of the result the model arbitrated rather than inferring it.
|
||||
@@ -109,7 +109,7 @@ in the corpus exercises it.
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Every finding shall carry its rung, its witness, and the file and line it concerns.
|
||||
2. The system shall state plainly what a `suspicion` means — that the model could not confirm it — so a
|
||||
2. The system shall state plainly what a `suspicion` means (that the model could not confirm it) so a
|
||||
reader does not treat it as established.
|
||||
3. The system shall write findings to the writable output location, never into the repository under analysis.
|
||||
4. The system shall summarise a scan in a form readable without opening a JSON file.
|
||||
@@ -121,7 +121,7 @@ in the corpus exercises it.
|
||||
throughout, and no criterion here forbade it.)*
|
||||
6. Where a family's abstraction cannot decide a weakness class, the system shall state that limit in the
|
||||
report, and no scan shall present an absence of findings as evidence of absence. *(Added 2026-09-09.
|
||||
`c/memory` scored zero true positives across 89k lines of C for a structural reason — it models string
|
||||
`c/memory` scored zero true positives across 89k lines of C for a structural reason: it models string
|
||||
functions and libpng overflows through arithmetic. A silent scan reading as a clean bill of health is
|
||||
worse than noise because it is quiet, and it is the one failure mode the rung ladder does not guard.)*
|
||||
|
||||
@@ -148,7 +148,7 @@ silently caps what the corpus can tell us.
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The system shall either reach the `memory` family through an adopted execution tier, or record explicitly
|
||||
that C memory-safety is out of scope — 173 of the 176 real-world CVE pairs currently sit idle behind a
|
||||
that C memory-safety is out of scope; 173 of the 176 real-world CVE pairs currently sit idle behind a
|
||||
stub.
|
||||
2. The system shall retire `evidence_level` in favour of `rung`, or record why both are kept.
|
||||
3. Each decision shall be recorded with its reason where a later reader will find it.
|
||||
@@ -167,7 +167,7 @@ this one, because each exists due to a specific past error.
|
||||
4. The core install shall remain zero-dependency, with the CPG engine, the LLM endpoint and any execution
|
||||
tier optional and degrading to a recorded reason.
|
||||
5. Any new family-specific code path shall be checked for sibling paths that need the same change, before it
|
||||
is measured — five bugs in the predecessor shared exactly that shape.
|
||||
is measured; five bugs in the predecessor shared exactly that shape.
|
||||
|
||||
### Requirement 9: Grow the corpus where the engine can use it
|
||||
|
||||
@@ -177,14 +177,14 @@ what developers actually contribute to rather than against 30 pairs of one slice
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The system shall add a **PHP/WordPress** slice covering plugin and core weaknesses. WordPress plugin CVEs
|
||||
are overwhelmingly injection, XSS and authorization bypass — the families the model layer already
|
||||
arbitrates well — and `php2cpg` works once a PHP interpreter is present, which the shipped image now
|
||||
are overwhelmingly injection, XSS and authorization bypass (the families the model layer already
|
||||
arbitrates well) and `php2cpg` works once a PHP interpreter is present, which the shipped image now
|
||||
provides.
|
||||
2. The system shall add **real-world C library CVEs** beyond the existing curl and OpenSSL rows — libpng and
|
||||
2. The system shall add **real-world C library CVEs** beyond the existing curl and OpenSSL rows: libpng and
|
||||
the image libraries, OpenSSH, libpam, and vulnerable kernel modules were named. These are gated on Req 7.1:
|
||||
until the memory-family decision is closed they would join 173 pairs already sitting idle, so the decision
|
||||
comes first and the harvest follows it.
|
||||
3. The system shall add further **vibe-code** pairs — agent-written and human-written web application bugs —
|
||||
3. The system shall add further **vibe-code** pairs (agent-written and human-written web application bugs)
|
||||
because that slice is where the tool's stated audience actually works, and it is currently 35 pairs.
|
||||
4. Before any slice is vendored at scale, the system shall verify on a sample that the model can **read** its
|
||||
pairs, and shall vendor only the categories that a measurement shows it can arbitrate. The OWASP harvest
|
||||
@@ -194,7 +194,7 @@ what developers actually contribute to rather than against 30 pairs of one slice
|
||||
contiguous key-shaped literal.
|
||||
6. Each new slice shall report as its own row and shall never be blended into a headline, and a synthetic
|
||||
slice shall say in its catalog that it measures breadth rather than real-world performance.
|
||||
7. The system shall record, per slice, how many pairs the model can arbitrate — so corpus growth is measured
|
||||
7. The system shall record, per slice, how many pairs the model can arbitrate, so corpus growth is measured
|
||||
by what it teaches rather than by how many rows it adds.
|
||||
|
||||
### Requirement 10: Regression tuning over a large codebase
|
||||
@@ -206,7 +206,7 @@ tables and queries can be checked for regressions that pair scoring cannot see.
|
||||
|
||||
1. The system shall support a repeatable scan over a pinned checkout of a large repository, recorded as a
|
||||
committed baseline.
|
||||
2. When the fact tables or queries change, the system shall report the delta against that baseline —
|
||||
2. When the fact tables or queries change, the system shall report the delta against that baseline:
|
||||
findings gained, findings lost, and rung movements.
|
||||
3. The system shall treat an unexplained loss of a previously-entailed finding as a regression to be
|
||||
investigated, not a number to be accepted.
|
||||
@@ -214,7 +214,7 @@ tables and queries can be checked for regressions that pair scoring cannot see.
|
||||
|
||||
---
|
||||
|
||||
## Amendment record — group 2 (approved 2026-09-09)
|
||||
## Amendment record, group 2 (approved 2026-09-09)
|
||||
|
||||
Group 2 was specified as the phase that can reshape the rest, and it did. Three amendments were drafted from
|
||||
what it measured and **approved by the maintainer on 2026-09-09**; they are folded into the numbered
|
||||
@@ -227,7 +227,7 @@ requirements above and marked there with the evidence that produced them.
|
||||
| C — a checkout that can demonstrate detection | Req 4.6 | libpng satisfies 4.1 and cannot demonstrate detection |
|
||||
|
||||
**A is only half implemented, deliberately.** Task 2.11 made pattern matches proportionate. The same argument
|
||||
applies to the model layer's own suspicion band — 38 of vampi's 60 findings — which is an enumerated
|
||||
applies to the model layer's own suspicion band (38 of vampi's 60 findings), which is an enumerated
|
||||
candidate the graph could not decide and an LLM affirmed: a pattern match with an opinion attached, not a
|
||||
reasoned claim. Task 2.17 carries that half.
|
||||
|
||||
@@ -238,5 +238,5 @@ reasoned claim. Task 2.17 carries that half.
|
||||
(2.6), where the leak fell 42.9% → 0.0% with pair-correct unchanged.
|
||||
* **Req 3.3's honesty counters were the load-bearing part.** `regions_unjudged` is what made a budget that
|
||||
stopped free work visible at all.
|
||||
* **Req 11.2's byte-identical gates held through every one of nine defect fixes** — which says the same thing
|
||||
* **Req 11.2's byte-identical gates held through every one of nine defect fixes**, which says the same thing
|
||||
from the other side: the corpus could not have caught any of them.
|
||||
|
||||
@@ -18,8 +18,8 @@ Derive a mechanism record from a pair by parsing the vulnerable excerpt with the
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: mechanism export from pairs; shape normalization; variant search as a proposer in MAP; leave-one-out measurement command and payload; `mechanisms` improve lever with validator rules; store hygiene (tier, provenance, dedupe, version); offline tests.
|
||||
- **Out**: LLM-authored shapes; changing dispositions or evidence rungs; embedding retrieval changes (existing prove-order reuse stays); inter-file flow (reachability spec); any merge gate; vendoring more pairs (corpus spec).
|
||||
- In: mechanism export from pairs; shape normalization; variant search as a proposer in MAP; leave-one-out measurement command and payload; `mechanisms` improve lever with validator rules; store hygiene (tier, provenance, dedupe, version); offline tests.
|
||||
- Out: LLM-authored shapes; changing dispositions or evidence rungs; embedding retrieval changes (existing prove-order reuse stays); inter-file flow (reachability spec); any merge gate; vendoring more pairs (corpus spec).
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
@@ -35,13 +35,13 @@ Derive a mechanism record from a pair by parsing the vulnerable excerpt with the
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `PairCase`/`load_pair_catalog` with `review_tier`; `FileIR`; `MechanismStore`; `adjudicate`; `run_improvement` gate with per-profile holdout.
|
||||
- **Downstream**: `reachability-flow-model` attaches paths to variant findings; `guard-dominance-regime` consumes the guard shapes as sanitizer candidates; reports show mechanism ids on findings.
|
||||
- Upstream: `PairCase`/`load_pair_catalog` with `review_tier`; `FileIR`; `MechanismStore`; `adjudicate`; `run_improvement` gate with per-profile holdout.
|
||||
- Downstream: `reachability-flow-model` attaches paths to variant findings; `guard-dominance-regime` consumes the guard shapes as sanitizer candidates; reports show mechanism ids on findings.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: `propose-adjudicate-prove` mechanism memory (additive fields, second write path), `three-stage-scan` MAP stage (one more proposer), `harnessx-self-improving-rulesets` improve loop (one more bounded lever).
|
||||
- **Depends**: `pair-corpus-honesty` Req 9 (tiers) for which pairs may seed.
|
||||
- Extends: `propose-adjudicate-prove` mechanism memory (additive fields, second write path), `three-stage-scan` MAP stage (one more proposer), `harnessx-self-improving-rulesets` improve loop (one more bounded lever).
|
||||
- Depends: `pair-corpus-honesty` Req 9 (tiers) for which pairs may seed.
|
||||
|
||||
## Constraints
|
||||
|
||||
|
||||
@@ -107,9 +107,9 @@ tests/
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `semantic/overlay.py` — `OverlayRecord.mechanism_id: str | None = None` (additive).
|
||||
- `regress/candidate.py` — exposes `hotspot_from_finding`; variant findings enter the ranked candidate slice as zero-score hotspots from `cli.py` (after every scored promotion, inside the cap, through the safety check).
|
||||
- `config.py` — `[variants] enabled=true, max_mechanisms=500`.
|
||||
- `semantic/overlay.py`: `OverlayRecord.mechanism_id: str | None = None` (additive).
|
||||
- `regress/candidate.py`: exposes `hotspot_from_finding`; variant findings enter the ranked candidate slice as zero-score hotspots from `cli.py` (after every scored promotion, inside the cap, through the safety check).
|
||||
- `config.py`: `[variants] enabled=true, max_mechanisms=500`.
|
||||
|
||||
## System Flows
|
||||
|
||||
|
||||
@@ -8,15 +8,15 @@ OpenUltraSAST's labeled pair corpus measures the detector but does not improve i
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: mechanism export from pairs; shape normalization per language; variant search proposer in MAP; leave-one-out evaluation; `mechanisms` improve lever; store schema additions; reports showing mechanism ids; offline tests.
|
||||
- **Out of scope**: taint, facts, walker, dispositions, evidence rungs; embedding retrieval changes; inter-file flow; LLM-authored shapes; catalogs and scorer changes beyond reading `PairCase`; any merge gate.
|
||||
- **Adjacent expectations**: `Mechanism` gains additive fields only; `order_promotions` keeps working on old records; the improve loop's existing clauses and the per-profile holdout clause apply unchanged; `local` remains the only slice that can fail CI.
|
||||
- In scope: mechanism export from pairs; shape normalization per language; variant search proposer in MAP; leave-one-out evaluation; `mechanisms` improve lever; store schema additions; reports showing mechanism ids; offline tests.
|
||||
- Out of scope: taint, facts, walker, dispositions, evidence rungs; embedding retrieval changes; inter-file flow; LLM-authored shapes; catalogs and scorer changes beyond reading `PairCase`; any merge gate.
|
||||
- Adjacent expectations: `Mechanism` gains additive fields only; `order_promotions` keeps working on old records; the improve loop's existing clauses and the per-profile holdout clause apply unchanged; `local` remains the only slice that can fail CI.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Mechanisms seeded from trusted pairs
|
||||
|
||||
**Objective:** As a maintainer, I want each reviewed pair to become a mechanism record, so that the corpus is a library the detector can search, not only a ruler.
|
||||
**Objective:** As a maintainer, I want each reviewed pair to become a mechanism record, so that the corpus is both a library the detector can search and a ruler.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
@@ -64,7 +64,7 @@ OpenUltraSAST's labeled pair corpus measures the detector but does not improve i
|
||||
|
||||
### Requirement 5: A recall-raising lever in the improve loop
|
||||
|
||||
**Objective:** As a maintainer, I want the improve loop to be able to admit mechanisms, so that it can raise recall and not only shadow noisy rules.
|
||||
**Objective:** As a maintainer, I want the improve loop to be able to admit mechanisms, so that it can raise recall as well as shadow noisy rules.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
## Summary
|
||||
|
||||
- **Feature**: `corpus-seeded-mechanisms`
|
||||
- **Discovery Scope**: Extension of mechanism memory, MAP stage, pair eval, and the improve loop
|
||||
- **Key Findings**:
|
||||
- Feature: `corpus-seeded-mechanisms`
|
||||
- Discovery Scope: Extension of mechanism memory, MAP stage, pair eval, and the improve loop
|
||||
- Key Findings:
|
||||
- `semantic/mechanisms.py` already has `Mechanism`, `MechanismStore` (append-only JSONL), `append_mechanism`, embedding cache, hard filters by language/CWE/tags, and `order_promotions`. It is written only by `cli._append_proven_mechanisms` after a sandbox trigger, so it is empty on every real tree.
|
||||
- Clearwing's recall lever is the variant loop: a proven mechanism becomes a structural search over the messy tree. OpenUltraSAST copied the store and the retrieval, not the loop.
|
||||
- `FileIR` call sites already carry callee name, argument texts, argument identifier lists, and constant flags; binds carry source text and names. That is enough for a shape without new parsing.
|
||||
@@ -16,30 +16,30 @@
|
||||
|
||||
### Mechanism store today
|
||||
|
||||
- **Context**: Can the existing store carry corpus-derived records?
|
||||
- **Sources Consulted**: `semantic/mechanisms.py`, `cli.py` `_append_proven_mechanisms`, `propose-adjudicate-prove` design §Mechanism.
|
||||
- **Findings**: `Mechanism` fields are summary, cwe, language, tags, keywords, what_made_it_exploitable, source_finding_id, source_repo. Identity is the pattern. JSONL is authoritative; vectors are a rebuildable cache keyed by id and model.
|
||||
- **Implications**: add `origin` (`sandbox` | `corpus`), `review_tier`, `pairs`, `shape` (structured), `guard` as additive fields with defaults so old rows load. A second writer (`append_from_pair`) beside the sandbox writer.
|
||||
- Context: Can the existing store carry corpus-derived records?
|
||||
- Sources Consulted: `semantic/mechanisms.py`, `cli.py` `_append_proven_mechanisms`, `propose-adjudicate-prove` design §Mechanism.
|
||||
- Findings: `Mechanism` fields are summary, cwe, language, tags, keywords, what_made_it_exploitable, source_finding_id, source_repo. Identity is the pattern. JSONL is authoritative; vectors are a rebuildable cache keyed by id and model.
|
||||
- Implications: add `origin` (`sandbox` | `corpus`), `review_tier`, `pairs`, `shape` (structured), `guard` as additive fields with defaults so old rows load. A second writer (`append_from_pair`) beside the sandbox writer.
|
||||
|
||||
### Shape derivation from FileIR
|
||||
|
||||
- **Context**: Avoid regex and LLM authoring.
|
||||
- **Sources Consulted**: `semantic/ir.py` (`CallSite.arg_names`, `arg_is_constant`, `Bind.names`, `FunctionIR.params`), `semantic/taint.py` (`_source_in`, parameter seeding).
|
||||
- **Findings**: for the labeled function, the sink call site is the call whose trailing name equals the label's sink or whose line equals the label's line; source positions are argument indexes whose identifiers trace by binds to a fact source, a parameter, or a container read. Guard kind comes from statements on the fixed side absent on the vulnerable side: `if` with `== NULL`/`is None`/`!x` → `null_test`; comparison with `<`, `>`, `len(` → `bounds_test`; membership in a literal collection or `startswith`/regex → `allowlist_test`; identity read from an auth context (`request.state`, `req.user`, `session`) → `auth_check`; parameterized second argument → `parameterized_call`; changed declared type → `type_change`.
|
||||
- **Implications**: `semantic/variants.py` implements `derive_shape(pair_ir_vuln, pair_ir_fixed, label)` and `match_shape(file_ir, shape)`. Guard classification is heuristic and closed; unknown → `none`.
|
||||
- Context: Avoid regex and LLM authoring.
|
||||
- Sources Consulted: `semantic/ir.py` (`CallSite.arg_names`, `arg_is_constant`, `Bind.names`, `FunctionIR.params`), `semantic/taint.py` (`_source_in`, parameter seeding).
|
||||
- Findings: for the labeled function, the sink call site is the call whose trailing name equals the label's sink or whose line equals the label's line; source positions are argument indexes whose identifiers trace by binds to a fact source, a parameter, or a container read. Guard kind comes from statements on the fixed side absent on the vulnerable side: `if` with `== NULL`/`is None`/`!x` → `null_test`; comparison with `<`, `>`, `len(` → `bounds_test`; membership in a literal collection or `startswith`/regex → `allowlist_test`; identity read from an auth context (`request.state`, `req.user`, `session`) → `auth_check`; parameterized second argument → `parameterized_call`; changed declared type → `type_change`.
|
||||
- Implications: `semantic/variants.py` implements `derive_shape(pair_ir_vuln, pair_ir_fixed, label)` and `match_shape(file_ir, shape)`. Guard classification is heuristic and closed; unknown → `none`.
|
||||
|
||||
### Where variant search sits
|
||||
|
||||
- **Context**: Propose, adjudicate, prove roles must stay separate.
|
||||
- **Sources Consulted**: `three-stage-scan` design (MAP stage, hunter suspicion), `propose-adjudicate-prove` (overlay records, coverage), `regress/candidate.py`.
|
||||
- **Findings**: MAP already runs overlay and the tool hunter; variant search is a third proposer that reads `FileIR` and the store. Findings at `suspicion`, origin `variant`; when the overlay has a flow at that call site, merge (attach mechanism id to the overlay record). Candidate eligibility follows the hunter-eligibility path being added in `reachability-flow-model` or the sev-5 forced-hotspot path today.
|
||||
- **Implications**: no disposition change; one new finding origin; manifest counters.
|
||||
- Context: Propose, adjudicate, prove roles must stay separate.
|
||||
- Sources Consulted: `three-stage-scan` design (MAP stage, hunter suspicion), `propose-adjudicate-prove` (overlay records, coverage), `regress/candidate.py`.
|
||||
- Findings: MAP already runs overlay and the tool hunter; variant search is a third proposer that reads `FileIR` and the store. Findings at `suspicion`, origin `variant`; when the overlay has a flow at that call site, merge (attach mechanism id to the overlay record). Candidate eligibility follows the hunter-eligibility path being added in `reachability-flow-model` or the sev-5 forced-hotspot path today.
|
||||
- Implications: no disposition change; one new finding origin; manifest counters.
|
||||
|
||||
### Leave-one-out
|
||||
|
||||
- **Context**: Prove that corpus growth raises recall, honestly.
|
||||
- **Findings**: pair eval materializes each side into a temp tree already; a leave-one-out runner seeds a temp store from the other trusted pairs and runs variant search on both sides, scoring with the pair scorer's detection and leak rules.
|
||||
- **Implications**: `pairs.py` gains a read-only consumer path (`evaluate_loo`) or a sibling module `semantic/loo.py` to keep the scorer's boundary; report per slice, profile, mechanism; artifact only.
|
||||
- Context: Prove that corpus growth raises recall, honestly.
|
||||
- Findings: pair eval materializes each side into a temp tree already; a leave-one-out runner seeds a temp store from the other trusted pairs and runs variant search on both sides, scoring with the pair scorer's detection and leak rules.
|
||||
- Implications: `pairs.py` gains a read-only consumer path (`evaluate_loo`) or a sibling module `semantic/loo.py` to keep the scorer's boundary; report per slice, profile, mechanism; artifact only.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
@@ -54,27 +54,27 @@
|
||||
|
||||
### Decision: Seed only from seeded and reviewed tiers
|
||||
|
||||
- **Context**: `title` pairs are unreviewed guesses; `advisory` pairs have correct fixes but unlabeled mechanisms.
|
||||
- **Selected Approach**: export reads `review_tier`; `advisory`/`title` may be scored leave-one-out as held-out targets but never seed.
|
||||
- **Trade-offs**: fewer seeds at first; honest about what the detector was taught.
|
||||
- Context: `title` pairs are unreviewed guesses; `advisory` pairs have correct fixes but unlabeled mechanisms.
|
||||
- Selected Approach: export reads `review_tier`; `advisory`/`title` may be scored leave-one-out as held-out targets but never seed.
|
||||
- Trade-offs: fewer seeds at first; honest about what the detector was taught.
|
||||
|
||||
### Decision: Variant is a proposer at suspicion
|
||||
|
||||
- **Context**: Evidence honesty.
|
||||
- **Selected Approach**: origin `variant`, `suspicion`; merge with overlay flow when present; sandbox may raise.
|
||||
- **Trade-offs**: variant-only hits do not fail CI; correct.
|
||||
- Context: Evidence honesty.
|
||||
- Selected Approach: origin `variant`, `suspicion`; merge with overlay flow when present; sandbox may raise.
|
||||
- Trade-offs: variant-only hits do not fail CI; correct.
|
||||
|
||||
### Decision: Lever admits records, not shapes
|
||||
|
||||
- **Context**: Keep the validator closed.
|
||||
- **Selected Approach**: `mechanisms` lever edits reference record ids produced by the exporter; free-form shapes are rejected.
|
||||
- Context: Keep the validator closed.
|
||||
- Selected Approach: `mechanisms` lever edits reference record ids produced by the exporter; free-form shapes are rejected.
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- Coarse shapes over-match (`exec(x)` everywhere) — source-position requirement plus fact/parameter/container check; report leak rate on fixed sides leave-one-out; the per-profile gate rejects admissions that leak.
|
||||
- Guard heuristics misclassify — closed set with `none`; reported, not used for demotion.
|
||||
- Store growth — dedupe by shape; one record per shape with pair provenance list.
|
||||
- Leave-one-out cost — one temp store per pair over tens of pairs; fine; cache parsed IR per excerpt.
|
||||
- Coarse shapes over-match (`exec(x)` everywhere): source-position requirement plus fact/parameter/container check; report leak rate on fixed sides leave-one-out; the per-profile gate rejects admissions that leak.
|
||||
- Guard heuristics misclassify: closed set with `none`; reported, not used for demotion.
|
||||
- Store growth: dedupe by shape; one record per shape with pair provenance list.
|
||||
- Leave-one-out cost: one temp store per pair over tens of pairs; fine; cache parsed IR per excerpt.
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@@ -9,13 +9,13 @@ numbers"), gated on the pool metric; it is not part of the G0 record-07 baseline
|
||||
The engine produces **no finding** on real unseen vulnerability changes. Live 300 s long-deadline replays show
|
||||
that the cause is usually not a detection-capability gap:
|
||||
|
||||
- **Clean Python injection change** (`aws/bedrock-agentcore-sdk-python`, `install_packages`, introducing, 3-file
|
||||
- Clean Python injection change (`aws/bedrock-agentcore-sdk-python`, `install_packages`, introducing, 3-file
|
||||
diff): `wall=303 s`, hit the deadline. The ranker found candidates (`static_rank=1.0`) but **`selected: []`** and
|
||||
every candidate `deferred` with `reason: deadline_exhausted`, `analysis_not_started=deadline_exhausted`,
|
||||
`change_context_incomplete`. No analyses ran. The ranked candidates were **repo-wide and unrelated to
|
||||
the push** (`auth.py` access_control/config_secrets), not the changed `install_packages`. The engine is trying to
|
||||
re-audit the whole repo and runs out of budget before analyzing the change.
|
||||
- **Noisy last_touch config_secrets change** (`jenkinsci/metrics-plugin`, 14-file diff): finished in 206 s but
|
||||
- Noisy last_touch config_secrets change (`jenkinsci/metrics-plugin`, 14-file diff): finished in 206 s but
|
||||
`comparison_unknown` + `capability_unavailable` + `eligibility_provenance_mismatch` +
|
||||
`contributor-scan ... projection_unavailable`. This has several causes. Correspondence was one (fixed separately,
|
||||
commit c39fbb8); the capability registry is Task 8.3.
|
||||
@@ -157,12 +157,12 @@ To be releasable, the net needs tier 2 and/or tier 3 to run and catch within a r
|
||||
The target is **≤60–120 s** (developer patience), hard ceiling 300 s. The analysis records each tier's budget against
|
||||
that window and identifies the measurements needed:
|
||||
|
||||
- **Engine barrier (tier 2).** Fixed floor `build(48) + evidence_fixed(~150) ≈ 198 s` precedes any dataflow answer.
|
||||
- Engine barrier (tier 2). Fixed floor `build(48) + evidence_fixed(~150) ≈ 198 s` precedes any dataflow answer.
|
||||
Measure (clean host): **E1** split the ~150 s into JVM-start / graph-load / census; **E2** does a warm session
|
||||
with the graph **preloaded at build** eliminate `evidence_fixed` on the first query (the architectural
|
||||
test)?; **E3** analysis (`reachableByFlows`) per-pair cost + warm reuse; **E4** build cost vs repo size.
|
||||
Levers in priority order: warm session + graph preload → cached census → change-scope the per-pair tail.
|
||||
- **LLM barrier (tier 3).** Unmeasured in the push path. Measure: **L1** model round-trip latency per call
|
||||
- LLM barrier (tier 3). Unmeasured in the push path. Measure: **L1** model round-trip latency per call
|
||||
(DeepSeek) and calls needed per push for the decision/proof stage; **L2** the recall lift of tier 3 over tiers
|
||||
1+2 on the pool, does the LLM catch what regex and dataflow miss, and at what $ and latency per push?
|
||||
- Produce a **barrier table** for budget ∈ {60, 120, 300, 600} s × each tier combination. Record whether it fits
|
||||
@@ -258,7 +258,7 @@ Reprioritised engine levers (corrected, anchoring is not the lever):
|
||||
([[search-with-proof]]) is that the LLM tier catches what regex+dataflow miss. The pilot supports
|
||||
testing it: **L1** model latency + calls/push, **L2** recall lift on the same 5 (and beyond). These calls cost money, so stage
|
||||
them (5–10 sizing → ~10–20 pilot → full only on lift).
|
||||
- **#2 taint source/sink coverage + region ranking** so the dataflow recognises these vuln sources and promotes the
|
||||
- #2 taint source/sink coverage + region ranking so the dataflow recognises these vuln sources and promotes the
|
||||
vuln region (banks path: sink found, source unrecognised; graphiti injection: score 0).
|
||||
- #3 change-anchoring robustness matters only after the static/LLM tiers produce vuln-family
|
||||
findings to anchor. Deprioritised until (1)/(2) yield findings.
|
||||
@@ -329,10 +329,10 @@ Two blockers to realising it in the live pipeline (both now concrete):
|
||||
mislocation, not a detection failure.**
|
||||
|
||||
The budget is approved. Keep the revised next steps staged:
|
||||
- **Validate the 5 pilot labels first** (cheap, LLM-assisted corpus audit): for each, is the real sink inside the
|
||||
- Validate the 5 pilot labels first (cheap, LLM-assisted corpus audit): for each, is the real sink inside the
|
||||
changed lines, and is it change-introduced vs pre-existing-in-base? Only sound, change-located vulns are fair
|
||||
targets. This separates true detection gaps from SZZ artifacts before spending on detection runs.
|
||||
- **Then run the agentic hunter** (search-with-proof) on the sound targets and size its per-vuln call count/cost;
|
||||
- Then run the agentic hunter (search-with-proof) on the sound targets and size its per-vuln call count/cost;
|
||||
measure catch + false-alarm (on ordinary changes) vs regex 1/5. The single-shot path is insufficient; the hunter
|
||||
is the real "#1".
|
||||
- Feed confirmed source shapes (e.g. library param → path join) back as guarded static facts (#2).
|
||||
@@ -362,7 +362,7 @@ which becomes a fact after repeated confirmation (the #2 feedback loop). No new
|
||||
user. This is now part of the "#2 guided by #1" workstream and must not be deferred. (DeepSeek
|
||||
`deepseek-flash` is the working default; GLM 5.3 "cyber" model is future-spec.)
|
||||
|
||||
## POOL-P1 LABEL AUDIT (LLM, 2026-10-12) — 47% of labels are sound
|
||||
## POOL-P1 LABEL AUDIT (LLM, 2026-10-12): 47% of labels are sound
|
||||
|
||||
The banks label-location check was extended to all **49 public** pool vulns via DeepSeek (advisory + head code +
|
||||
pool label → sound/unsound). Full artifact: `benchmarks/measurements/2026-10-12-pool-p1-label-audit/`.
|
||||
@@ -378,7 +378,7 @@ pool label → sound/unsound). Full artifact: `benchmarks/measurements/2026-10-1
|
||||
- Next: run the agentic search-with-proof hunter on the sound targets (start bedrock + graphiti), size its
|
||||
per-vuln call count/cost, measure catch + false-alarm. Then recompute record 07 on the sound subset.
|
||||
|
||||
## MODEL HUNTER on the 2 sound targets (2026-10-12) — first LLM catch the static tiers missed
|
||||
## MODEL HUNTER on the 2 sound targets (2026-10-12): first LLM catch the static tiers missed
|
||||
|
||||
Ran `ousast scan --mode deep` (DeepSeek `deepseek-flash`, model layer + tool_hunter, 60-call/8-hotspot bound) on
|
||||
bedrock + graphiti, the two sound pilot targets the static engine missed. Artifact:
|
||||
@@ -417,7 +417,7 @@ Next: deep scan (regex/engine/hunter) on the 5 fair targets → per-tier catch r
|
||||
stand up an app-shaped eval set from known-vulnerable apps (VAmPI/NodeGoat/DjanGoat/vibe-py) and recompute the
|
||||
floor. Revisit the deferred bets (advisory-with-conditions, reachability prover) once the fair-target results are available.
|
||||
|
||||
## FAIR-TARGET RESULT: detection 3/5 on app-shaped+sound (vs ~0 library-heavy) — target-fix validated (2026-10-12)
|
||||
## FAIR-TARGET RESULT: detection 3/5 on app-shaped+sound (vs ~0 library-heavy), target-fix validated (2026-10-12)
|
||||
|
||||
Deep scan on the 5 app-shaped+sound targets (`benchmarks/measurements/2026-10-12-hunter-fair-targets/`):
|
||||
- 3/5 detected: graphiti (hunter, Cypher injection), mcp-remote (hunter, url-scheme injection), dcmcp (ENGINE
|
||||
@@ -433,7 +433,7 @@ Detection appears on fair targets; release-grade BLOCK still needs (a) proof-arm
|
||||
(b) family coverage (template XSS/SSTI) + the DSPy hint loop for misses. Next: recompute a floor on an app-shaped
|
||||
sound subset / real app corpus (VAmPI, NodeGoat, DjanGoat, vibe-py), and measure the proof arm's suspicion→proven lift.
|
||||
|
||||
## PROOF-ARM MEASUREMENT on the 3 fair-target hits (2026-10-12) — push-time lift 0/3, structural
|
||||
## PROOF-ARM MEASUREMENT on the 3 fair-target hits (2026-10-12): push-time lift 0/3, structural
|
||||
|
||||
Measured whether the proof arm can elevate the 3 fair-target suspicions (graphiti, mcp-remote, dcmcp) to
|
||||
`exploit_demonstrated`.
|
||||
@@ -448,7 +448,7 @@ Measured whether the proof arm can elevate the 3 fair-target suspicions (graphit
|
||||
(`isolation_mode: userns`); the plane supplies a task-boundary dispatcher.
|
||||
- The scan/push path reaches only `suspicion`/`static_corroboration` (verification.py levels). There is no code
|
||||
path from a scan suspicion to `exploit_demonstrated`, and no caller wires the search Coordinator from the push
|
||||
flow. So proof-arm lift at push time = 0/3: not because the vulns are unreal, but because proving is a
|
||||
flow. The vulns are real, but proof-arm lift at push time = 0/3 because proving is a
|
||||
runtime differential operation with an offline build for each target that cannot run inside a pre-push hook. (Prior
|
||||
per-pair demo attempts were mostly inconclusive on this exact wiring, app startup, offline deps, oracle plumbing.)
|
||||
|
||||
@@ -462,7 +462,7 @@ suspicion:
|
||||
proving, with authored (eventually DSPy-authored) demos. Run a bounded, separate test on one tractable target
|
||||
(mcp-remote) to confirm that the arm demonstrates a real unseen vuln.
|
||||
|
||||
## BANKS FIXED-SEARCH MVP (2026-10-12) — "no wrong targets, only wrong searching" demonstrated
|
||||
## BANKS FIXED-SEARCH MVP (2026-10-12): "no wrong targets, only wrong searching" demonstrated
|
||||
|
||||
Record: `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`. Target: banks (the "library" that gave
|
||||
no engine path findings in the pilot). The target and CPG are unchanged; only the search changes.
|
||||
|
||||
+10
-10
@@ -12,7 +12,7 @@ Fixed via **option 1** (filter params to the declared signature) in `src/openult
|
||||
- `JoernBackend.query()` now drops any param the script does not declare before building the `--param` list
|
||||
(disposable route) and the in-session named-arg call (session route). This is faithful, not lossy: the
|
||||
single-request branch of `taint.sc` (line 1542) calls `rowsFor(...)` with top-level params only and lets
|
||||
`fixedOrigin`/`originAnchors`/`quotedSanitizers`/`guards`/`fieldParameterSources` default — it never could
|
||||
`fixedOrigin`/`originAnchors`/`quotedSanitizers`/`guards`/`fieldParameterSources` default; it never could
|
||||
consume them. The batch path keeps carrying them as JSON the script parses itself.
|
||||
- No-op for `config`/`dominance`: their own `request_params` emit only params their `.sc` declares (verified),
|
||||
so nothing is dropped. Only `taint` sheds the 5 extras.
|
||||
@@ -21,12 +21,12 @@ Fixed via **option 1** (filter params to the declared signature) in `src/openult
|
||||
queries now RUN (no `cpg query taint failed`); baseline = 0 findings, fixed search (annotation) = 2 scored
|
||||
`path`/`model_entailed` findings (`from_prompt_path` → `prompt_file.write_text(prompt.raw)` L45; `_scan` →
|
||||
`self._index_path.write_text(...)` L155). Regression test:
|
||||
`tests/test_cpg_backend.py::test_a_single_query_passes_only_params_the_script_declares` (fake runner, no Joern
|
||||
— asserts the 5 undeclared params never reach the command line and the query parses).
|
||||
`tests/test_cpg_backend.py::test_a_single_query_passes_only_params_the_script_declares` (fake runner, no Joern;
|
||||
asserts the 5 undeclared params never reach the command line and the query parses).
|
||||
|
||||
## Summary
|
||||
|
||||
Any call on the **single-query** taint path — `cpg.run("taint", request_params(...))` — fails on **both**
|
||||
Any call on the **single-query** taint path (`cpg.run("taint", request_params(...))`) fails on **both**
|
||||
of its sub-routes, returns `None`, and is read upstream as "no flows / clean". `request_params()` emits five
|
||||
keys that `taint.sc`'s `@main def exec(...)` does not declare, and the single-query dispatch passes every
|
||||
param straight to the script. The **batch** path (`run_batch` → `requestsFile` JSON) is unaffected, which is
|
||||
@@ -53,10 +53,10 @@ contextFilter, contextPaths, requests, requestsFile, trace`.
|
||||
`backend.query()` (`src/openultrasast/cpg/backend.py:1266`) dispatches a single query two ways, **both** of
|
||||
which choke on the undeclared keys:
|
||||
|
||||
- **Disposable / CLI route** (lines 1276–1278): appends `--param {key}={value}` for every param. joern errors
|
||||
- Disposable / CLI route (lines 1276–1278): appends `--param {key}={value}` for every param. joern errors
|
||||
`Unknown arguments: --fixedOrigin --guards --quotedSanitizers --originAnchors --fieldParameterSources`
|
||||
(and, as a knock-on, `Missing argument: --cpgFile`). Observed verbatim via `banks_debug.py` on 2026-10-12.
|
||||
- **Session route** (lines 1321–1326): defines `def ousast_taint(...)` by renaming `@main def exec(` and then
|
||||
- Session route (lines 1321–1326): defines `def ousast_taint(...)` by renaming `@main def exec(` and then
|
||||
calls `ousast_taint(key = value, ...)` with named args. The same five keys are not parameters of that
|
||||
method → the Scala named-arg call fails to compile. (By construction; the rename preserves the exact
|
||||
`exec` parameter list.)
|
||||
@@ -77,14 +77,14 @@ Unaffected:
|
||||
|
||||
- The push/scan **batch** path: `scan.py` uses `run_batch("taint", …)` (scan.py:1259, 1264, 1387). This is
|
||||
why the engine-on pilot produced results.
|
||||
- `config` and `dominance` single-queries (`config_value.py:46`, `dominance.py:54`) — their `.sc` signatures
|
||||
- `config` and `dominance` single-queries (`config_value.py:46`, `dominance.py:54`): their `.sc` signatures
|
||||
are presumed to match what their specs emit (confirm separately; not exercised here).
|
||||
|
||||
Open question — **CLOSED by call-graph trace (2026-10-12).** No shipped product path and no in-repo
|
||||
Open question: **CLOSED by call-graph trace (2026-10-12).** No shipped product path and no in-repo
|
||||
benchmark/test reached the real single-query `JoernBackend.query("taint")`; the bug was latent but unreached
|
||||
by the product. Traced:
|
||||
|
||||
- **Shipped push/scan/replay** (`cli.py:1334` → `scan_repository` → `_scan_repository_impl`) is DOUBLE
|
||||
- Shipped push/scan/replay (`cli.py:1334` → `scan_repository` → `_scan_repository_impl`) is DOUBLE
|
||||
shielded: Phase 2 fetches taint via the **batch route** (`run_batch`, always wired by `build()`), and
|
||||
Phase 3's `scan_region` (scan.py:559) runs against a **`_prefetched` CpgResult** whose `run` ignores the
|
||||
query and returns the already-fetched batch rows (scan.py:1472). So `scan_region → taint_verdict →
|
||||
@@ -105,7 +105,7 @@ The fix restores the single-query path for pair/direct use without altering push
|
||||
|
||||
## Evidence
|
||||
|
||||
- `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/` — the batch path works (11 flow rows,
|
||||
- `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`: the batch path works (11 flow rows,
|
||||
`write_text` sinks); the v1 single-query attempt returned 0/0 on `cpg query taint failed`.
|
||||
- Raw joern error captured by `banks_debug.py` (scratchpad): `Missing argument: --cpgFile; Unknown
|
||||
arguments: --fixedOrigin --guards --quotedSanitizers --originAnchors --fieldParameterSources`.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Brief: finding-feedback-loop — what happens after a finding is judged wrong
|
||||
# Brief: finding-feedback-loop, what happens after a finding is judged wrong
|
||||
|
||||
**Status:** brief only, unapproved. Successor scope lifted out of `contributor-scan`, which is mid-implementation
|
||||
and does not cover this.
|
||||
@@ -7,13 +7,13 @@ and does not cover this.
|
||||
|
||||
Two task groups were drafted into `contributor-scan/tasks.md` on 2026-09-09 without passing through
|
||||
requirements or design. That was a workflow error: they are not refinements of scan integration, they are a
|
||||
different feature — a loop that learns from findings a human rejected — and writing them as tasks skipped the
|
||||
different feature (a loop that learns from findings a human rejected) and writing them as tasks skipped the
|
||||
two phases that decide whether the feature is shaped right. They are lifted here to be specified properly.
|
||||
|
||||
## The gap this exists to close
|
||||
|
||||
Nine engine defects were found in one session against two repositories. **Three of them were the same bug in
|
||||
three different matchers**: a spec token matched as a substring rather than as a word — `resolve` inside
|
||||
three different matchers**: a spec token matched as a substring rather than as a word: `resolve` inside
|
||||
`resolveUrl` in the sanitizer matcher, `user` inside `users` in the discharger matcher, `gets` inside `fgets`
|
||||
in the sink matcher. Each was found by a measurement, fixed locally, and left no trace. The third survived
|
||||
until a 89k-line C library made it visible as 26 identical false positives.
|
||||
@@ -60,17 +60,17 @@ sound. The same machinery writing sink tables is the thing that produced the lea
|
||||
|
||||
## What the shape looks like
|
||||
|
||||
* **Per-repository, declarative, committed by the developer.** A dismissal file in the *target* repository —
|
||||
site, rung, reason, author — read by the tool and never written by it, the same rule the declared policy
|
||||
* **Per-repository, declarative, committed by the developer.** A dismissal file in the *target* repository (
|
||||
site, rung, reason, author), read by the tool and never written by it, the same rule the declared policy
|
||||
file already follows. Reviewed like code, diffable, and a dismissal without a reason is how a suppression
|
||||
list becomes a graveyard nobody can audit.
|
||||
* **Dismissals are labels, not suppressions.** With the known vulnerabilities a `benchmarks/repos/` recipe
|
||||
already declares, a dismissal file gives a per-repository evaluation set with *both* classes in it — which
|
||||
already declares, a dismissal file gives a per-repository evaluation set with *both* classes in it, which
|
||||
is what an optimiser needs and what a suppression list normally throws away.
|
||||
* **A ledger for the maintainer.** Counter-examples grouped by what they share (same token, same matcher
|
||||
clause, same witness shape), so a recurring cause is visible as a cause rather than as five separate bugs.
|
||||
* **Split-respecting adoption.** Train split only, holdout never read, the three gates byte-identical, and
|
||||
precision *and* recall reported on both sides — an adoption that quietly trades recall is worse than none.
|
||||
precision *and* recall reported on both sides; an adoption that quietly trades recall is worse than none.
|
||||
|
||||
## The acceptance test for the model's role
|
||||
|
||||
@@ -80,7 +80,7 @@ recover a cause we already know should not be trusted to propose one we do not.
|
||||
## Known limits to carry forward
|
||||
|
||||
* The pair corpus could not see any of the nine defects; whole-repository measurement is what found them, and
|
||||
this loop inherits that. Its evaluation set has to include repositories, not only pairs.
|
||||
this loop inherits that. Its evaluation set has to include repositories as well as pairs.
|
||||
* Only two repositories exist today (one Python, one C), and the C one has never produced a true positive.
|
||||
An optimiser tuned on that evidence base would be tuned on very little.
|
||||
* `contributor-scan` Req 8 already warns about the +16% overfitting gap between the slice a change was
|
||||
|
||||
@@ -10,7 +10,7 @@ anything.
|
||||
|
||||
The design turns on one asymmetry: **two of those three bugs needed no judgement at all.** Once the first was
|
||||
understood, the other two were the same fix carried across. So the automation that pays is *propagation and
|
||||
memory*, and the model is confined to the one step that genuinely needs inference — naming what a group has in
|
||||
memory*, and the model is confined to the one step that genuinely needs inference: naming what a group has in
|
||||
common.
|
||||
|
||||
### Goals
|
||||
@@ -22,7 +22,7 @@ common.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- **Automated fact-table edits.** Not deferred — prohibited. See Boundary Commitments.
|
||||
- **Automated fact-table edits.** Prohibited rather than deferred. See Boundary Commitments.
|
||||
- Suppressing findings silently. A dismissal makes a finding *labelled*, never invisible.
|
||||
- A general-purpose optimiser. The only surfaces evolved are two prompt constants.
|
||||
- Replacing the pair corpus. It is the holdout that keeps per-repository tuning honest.
|
||||
@@ -38,10 +38,10 @@ common.
|
||||
|
||||
### Out of Boundary
|
||||
|
||||
- `semantic/facts`, `semantic/obligations`, `ruleset/`, the declared policy file — **no automated writer, by
|
||||
- `semantic/facts`, `semantic/obligations`, `ruleset/`, the declared policy file: **no automated writer, by
|
||||
any path, at any time.** This is the boundary the whole design exists to respect.
|
||||
- The three gates. They are the invariant an adoption is measured against, not a thing this spec changes.
|
||||
- `model/taint.py`, `dominance.py`, `config_value.py` — arbiters are fixed here; a proposal that would change
|
||||
- `model/taint.py`, `dominance.py`, `config_value.py`: arbiters are fixed here; a proposal that would change
|
||||
one is a proposal for a human, not an edit.
|
||||
|
||||
### Allowed Dependencies
|
||||
@@ -64,7 +64,7 @@ Three pieces already exist and are reused rather than rebuilt:
|
||||
- **`benchmarks/repos/*.toml`** declares known vulnerabilities per checkout, with `in_scope` stating whether a
|
||||
family can decide one at all. That is the positive half of an evaluation set, already committed.
|
||||
- **`pairs.py`** already carries a train/holdout split and the machinery that respects it. The leak of
|
||||
2026-09-05 was in the *lever*, not the splitter — `semantic/loo.evaluate_loo` was honest throughout.
|
||||
2026-09-05 was in the *lever*, not the splitter; `semantic/loo.evaluate_loo` was honest throughout.
|
||||
- **`redaction.py`** is applied to every prompt and persisted trajectory, and applies here unchanged.
|
||||
|
||||
### Boundary Map
|
||||
@@ -87,21 +87,21 @@ benchmarks/repos/*.toml ── read ────▶ evaluation set
|
||||
|
||||
**A dismissal is a label, not a suppression.** A suppressed finding disappears and its reason rots; a
|
||||
labelled one is reported as dismissed with its reason, and a dismissal that stops matching is reported too.
|
||||
This is what turns a suppression list into the negative half of an evaluation set — the thing that makes the
|
||||
This is what turns a suppression list into the negative half of an evaluation set, the thing that makes the
|
||||
rest of the feature possible.
|
||||
|
||||
**The tool never writes the dismissal file.** Same rule as the declared policy file, for the same reason: a
|
||||
file the tool edits is a file no reviewer trusts. It is committed by a developer and reviewed like code.
|
||||
|
||||
**Grouping is deterministic; only the hypothesis is not.** Entries are grouped by shared spec token, matcher
|
||||
clause, family and witness shape — all mechanical. The model is asked one question about a group it did not
|
||||
clause, family and witness shape; all mechanical. The model is asked one question about a group it did not
|
||||
choose, and its answer goes to a queue. This keeps the inference confined to the step that needs it.
|
||||
|
||||
**Evolution targets the question, never the facts.** A prompt is how we ask; a fact table is what we claim.
|
||||
`CANDIDATE_QUESTION` and `residual_question` are already the LLM-facing surface and are the only mutable
|
||||
targets. Enforced by a test that fails if any writer touches the fact modules, not by convention.
|
||||
|
||||
**The holdout is never read.** Not "should not" — the evaluation-set API has no path to it during
|
||||
**The holdout is never read.** The evaluation-set API has no path to it during
|
||||
optimisation, and a test asserts the absence. The 2026-09-05 leak was possible because reading it was merely
|
||||
discouraged.
|
||||
|
||||
@@ -123,9 +123,9 @@ benchmarks/ledger/
|
||||
|
||||
### Changed
|
||||
|
||||
- `cli.py` — a `feedback` subcommand: `ledger`, `groups`, `propose`, `evolve`. Maintainer surface; `scan`
|
||||
- `cli.py`: a `feedback` subcommand: `ledger`, `groups`, `propose`, `evolve`. Maintainer surface; `scan`
|
||||
gains only the dismissal read.
|
||||
- `reports.py` — a dismissed finding is rendered as dismissed with its reason.
|
||||
- `reports.py`: a dismissed finding is rendered as dismissed with its reason.
|
||||
|
||||
### Unchanged and load-bearing
|
||||
|
||||
|
||||
@@ -5,22 +5,22 @@
|
||||
## Introduction
|
||||
|
||||
Nine engine defects were found in one session against two repositories. **Three were the same bug in three
|
||||
matchers** — a spec token matched as a substring rather than a word: `resolve` inside `resolveUrl` in the
|
||||
matchers**: a spec token matched as a substring rather than a word: `resolve` inside `resolveUrl` in the
|
||||
sanitizer matcher, `user` inside `users` in the discharger matcher, `gets` inside `fgets` in the sink matcher.
|
||||
Each was found by a measurement, fixed locally, and left no trace; the third survived until an 89k-line C
|
||||
library made it visible as 26 identical false positives.
|
||||
|
||||
Nothing in the system remembers that a finding was wrong. There is no artifact saying *this site, at this
|
||||
rung, is not a defect, and here is why* — so a class recurs, and each recurrence costs another measurement.
|
||||
rung, is not a defect, and here is why*, so a class recurs, and each recurrence costs another measurement.
|
||||
|
||||
Two audiences want that artifact for different reasons. A maintainer wants the *propagation* automated: two of
|
||||
the three matcher bugs needed no judgement at all, only the fix carried across. A developer wants to dismiss a
|
||||
finding in their own repository, have it stay dismissed, and have the reason reviewable beside the code — a
|
||||
finding in their own repository, have it stay dismissed, and have the reason reviewable beside the code; a
|
||||
tool that cannot be told it is wrong is one they stop running.
|
||||
|
||||
This feature is bounded by a rule the project has held from the start and a leak that shows why. An optimiser
|
||||
may evolve **the question** — the judge and residual prompts, which are how we ask. It may never author **the
|
||||
facts** — tables, rulesets, the declared policy file, which are what we claim. On 2026-09-05 the improve lever
|
||||
may evolve **the question**: the judge and residual prompts, which are how we ask. It may never author **the
|
||||
facts**: tables, rulesets, the declared policy file, which are what we claim. On 2026-09-05 the improve lever
|
||||
admitted candidate shapes because they "recover a currently missed holdout pair"; five of eleven were taught by
|
||||
the holdout pairs they then recovered, and a holdout Youden of **−0.059 was reported as +0.059**.
|
||||
|
||||
@@ -38,11 +38,11 @@ recorded and my colleagues can see why.
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The system shall read dismissals from a declarative file in the **target** repository, and shall never
|
||||
write to it — the same rule the declared policy file already follows.
|
||||
write to it, the same rule the declared policy file already follows.
|
||||
2. A dismissal shall carry the site it concerns, the rung it was dismissed at, a reason, and an author.
|
||||
3. The system shall reject a dismissal that carries no reason, because a suppression list without reasons is
|
||||
a graveyard nobody can audit.
|
||||
4. The system shall report a dismissed finding as dismissed, with its reason, rather than omitting it — so a
|
||||
4. The system shall report a dismissed finding as dismissed, with its reason, rather than omitting it, so a
|
||||
stale dismissal is visible rather than silent.
|
||||
5. When a dismissal no longer matches any finding, the system shall say so, because a dismissal that has
|
||||
stopped applying is either a fixed bug or a moved target and both are worth knowing.
|
||||
@@ -54,20 +54,20 @@ cause is visible as a cause rather than as five separate bugs.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a finding is contradicted — by a dismissal, by the judge with coverage, or by a declared contract —
|
||||
1. When a finding is contradicted (by a dismissal, by the judge with coverage, or by a declared contract)
|
||||
the system shall record the site, the rung, the witness, the spec tokens that produced it, and the commit.
|
||||
2. Each ledger entry shall be runnable as a regression case: this site, at this commit, shall not be reported
|
||||
at that rung.
|
||||
3. The ledger's regression cases shall run offline in CI, so that re-introducing a fixed cause fails a test
|
||||
rather than a repository scan four weeks later.
|
||||
4. The system shall group ledger entries by what they share — spec token, matcher clause, family, witness
|
||||
shape — and report the groups by size.
|
||||
4. The system shall group ledger entries by what they share (spec token, matcher clause, family, witness
|
||||
shape) and report the groups by size.
|
||||
5. The ledger shall be committed evidence, not a local artifact.
|
||||
|
||||
### Requirement 3: A model may propose a cause, and only propose it
|
||||
|
||||
**Objective:** As a maintainer, I want the model to do the part that needs judgement — naming what a group of
|
||||
counter-examples has in common — without it deciding anything.
|
||||
**Objective:** As a maintainer, I want the model to do the part that needs judgement (naming what a group of
|
||||
counter-examples has in common) without it deciding anything.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
@@ -132,7 +132,7 @@ quietly loses recall is worse than none.
|
||||
|
||||
1. The evaluation set shall include whole repositories, not pairs alone: the pair corpus could not see any of
|
||||
the nine defects that motivate this feature, and the gates stayed byte-identical through every fix.
|
||||
2. The system shall state how many repositories its evidence base contains, because two — one of which has
|
||||
never produced a true positive — is very little to tune against.
|
||||
2. The system shall state how many repositories its evidence base contains, because two (one of which has
|
||||
never produced a true positive) is very little to tune against.
|
||||
3. Where an adoption is measured on one repository only, the system shall label it as such rather than
|
||||
reporting it as a general improvement.
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
# Brief: flow-aware-ranking — spend the expensive query where the cheap evidence is strongest
|
||||
# Brief: flow-aware-ranking, spend the expensive query where the cheap evidence is strongest
|
||||
|
||||
> **Status 2026-10-02:** dormant. Brief only and unapproved; its 2026-09-12 amendment placed ranking inside the pre-push safety net, where the ranker now lives.
|
||||
|
||||
**Status:** brief only, unapproved.
|
||||
|
||||
## Current amendment — 2026-09-12
|
||||
## Current amendment, 2026-09-12
|
||||
|
||||
This section supersedes forward-looking interpretations below; the iteration history is retained.
|
||||
The product goal is a scalable, actionable pre-push security safety net as AI accelerates code changes.
|
||||
@@ -23,7 +23,7 @@ need explicit summaries or unresolved coverage, never automatic vendor reintrodu
|
||||
- Latest recorded phase-2 budgets: PMPro **82**, WP Statistics **91**, MW WP Form **54**, VAmPI **16**.
|
||||
The `<50` gate remains unmet. The evidence passes recorded no unanswered pairs. These are stored
|
||||
ranking measurements, not fresh detection/precision results or a pre-push latency demonstration.
|
||||
- **Equal-score ties are not the whole remaining problem.** At fixed scores, perfect tie-breaking
|
||||
- Equal-score ties are not the whole remaining problem. At fixed scores, perfect tie-breaking
|
||||
still needs at least **68** regions for PMPro and **86** for WP Statistics (67/85 strictly
|
||||
higher-scored same-tier regions precede the targets). An LLM restricted to equal-score ties
|
||||
cannot pass `<50`. Within-tier reordering across scores or a new feature is a distinct experiment.
|
||||
@@ -95,7 +95,7 @@ Three regions from one plugin, with what the ranker sees today and what is actua
|
||||
| `pmpro_rest_api_get_membership_level` | **0.80** | none — no SQL sink at all |
|
||||
|
||||
A region with no sink outranks two holding raw concatenated SQL. The discriminating signal is in the sink
|
||||
call's own source text — interpolated versus bound — and nothing in the ranker reads it.
|
||||
call's own source text (interpolated versus bound), and nothing in the ranker reads it.
|
||||
|
||||
One detail decides whether a keyword rule suffices, and it does not: `esc_sql` IS in scope for the CVE's
|
||||
function, applied to a different statement. "A sanitizer is present, rank it lower" demotes precisely the
|
||||
@@ -122,8 +122,8 @@ finding**, it decides what is looked at, and `regions_truncated` already reports
|
||||
Each phase ships only if it beats the previous one **on a repository the change was not developed against**.
|
||||
A phase that cannot show that is not an improvement, it is a story.
|
||||
|
||||
- **Phase 0 — the oracle and the baseline.** The evidence extractor (no model) and a scoring harness over the
|
||||
pinned CVEs, reporting each one's rank position. Gate: reproduces today's numbers — CVE-2023-23488 at
|
||||
- Phase 0, the oracle and the baseline. The evidence extractor (no model) and a scoring harness over the
|
||||
pinned CVEs, reporting each one's rank position. Gate: reproduces today's numbers, CVE-2023-23488 at
|
||||
position 390 of 4,463, 83.8% of regions in one tier. Without a baseline nothing later is measurable.
|
||||
|
||||
**Done 2026-09-10.** `benchmarks/ranking/position.py`, committed as
|
||||
@@ -138,36 +138,36 @@ A phase that cannot show that is not an improvement, it is a story.
|
||||
|
||||
**And this corrects the cross-assessment above.** "Ranking's job is concentration, not inclusion" was
|
||||
measured on `pmpro` alone, where the CVE is inside the budget. On the other two plugins the CVE sits at
|
||||
512 and 503 — **outside a 500-region budget** — so the current ranker never examines it at all, and the
|
||||
512 and 503, **outside a 500-region budget**, so the current ranker never examines it at all, and the
|
||||
slices found those CVEs only because a slice has too few regions to exclude anything. Both jobs are
|
||||
real: inclusion on two repositories, concentration on all three. The single metric `budget_at_recall`
|
||||
covers both, which is why it is the one that matters.
|
||||
|
||||
The instrument is `evidenceOnly` mode on the taint query, recorded on every scan as `ModelScanResult.tiers`
|
||||
and `tier_counts`, and acted on by nothing — so every later phase is measured against a record made by
|
||||
and `tier_counts`, and acted on by nothing, so every later phase is measured against a record made by
|
||||
scans that did not know they were being measured.
|
||||
|
||||
**Measured cost of the instrument**, 637-file graph, the scan's own 500-region request set:
|
||||
|
||||
2,500 pairs in 98.9 s including JVM load — about 40 ms per pair
|
||||
2,500 pairs in 98.9 s including JVM load, about 40 ms per pair
|
||||
tier 0 (no sink of that family in reach): 2,161 of 2,500 = 86%
|
||||
|
||||
Two attempts before that ran past an hour each without producing a payload, and the cause was mine both
|
||||
times: repository-wide walks — the sink scan, the reachable-method BFS, `familyInRepo` — repeated per
|
||||
times: repository-wide walks (the sink scan, the reachable-method BFS, `familyInRepo`) repeated per
|
||||
request. Memoising them across the batch is what made "milliseconds per pair" true rather than claimed.
|
||||
|
||||
**And the same memoisation reaches the dataflow mode.** The identical 250-request set that cost 2,026.3 s
|
||||
(8,105 ms per request) with both joins on now costs **287.0 s — 1,148 ms per request — for the same 5,779
|
||||
(8,105 ms per request) with both joins on now costs **287.0 s (1,148 ms per request) for the same 5,779
|
||||
rows.** Most of what contributor-scan 5.13 measured as "the base taint query" was the per-request scans,
|
||||
not the dataflow. That correction is recorded there.
|
||||
- **Phase 1 — tier 0 only, no model.** Exact pruning. Gate: **53% fewer taint requests issued, measured** —
|
||||
- Phase 1, tier 0 only, no model. Exact pruning. Gate: **53% fewer taint requests issued, measured**,
|
||||
and nothing more. 1,177 requests at 6.63 s is still 2.2 hours, so this phase does NOT make the scan finish;
|
||||
it halves the problem. Claiming completion here would send someone looking for a bug where there is none.
|
||||
|
||||
**Done 2026-09-10, gate exceeded, scan still does not finish — and the reason corrects phase 0's cost
|
||||
**Done 2026-09-10, gate exceeded, scan still does not finish, and the reason corrects phase 0's cost
|
||||
figure.** On the 637-file plugin: `tier_counts = {0: 2161, 1: 60, 2: 20, 3: 211, 4: 48}`, **2,161 of 2,500
|
||||
requests pruned (86%)**, evidence pass 94.5 s. The remaining **339 requests still hit the 2,400 s ceiling**
|
||||
— more than 7 s each.
|
||||
requests pruned (86%)**, evidence pass 94.5 s. The remaining **339 requests still hit the 2,400 s ceiling**,
|
||||
more than 7 s each.
|
||||
|
||||
**The scan finishes, 2026-09-11.** Not because of anything in this spec: contributor-scan 5.13 found
|
||||
that every query batch had been recomputing Joern's dataflow overlay at load since PHP builds went
|
||||
@@ -188,12 +188,12 @@ A phase that cannot show that is not an improvement, it is a story.
|
||||
which 88% were tier 0 and returned instantly; the ~30 real requests in it cost about 9.5 s each. Pruning
|
||||
removes exactly the cheap questions and keeps the expensive ones, so it cannot make those faster. The
|
||||
honest per-request cost of a taint question that *can* have an answer is **7–10 s at `callDepth = 3`**,
|
||||
and that is the number phase 2 and 5.13 must work against — not 1.1 s, and not 6.6 s either.
|
||||
- **Phase 2 — heuristic tiers, still no model.** The vector and tiers defined below, ordered within a tier
|
||||
by a published weighted sum. Gate: `budget_at_recall` — the smallest K holding every pinned CVE — **below
|
||||
and that is the number phase 2 and 5.13 must work against, not 1.1 s, and not 6.6 s either.
|
||||
- Phase 2, heuristic tiers, still no model. The vector and tiers defined below, ordered within a tier
|
||||
by a published weighted sum. Gate: `budget_at_recall` (the smallest K holding every pinned CVE) **below
|
||||
50 on every pinned repository**, and `pmpro`'s taint query completing at that K. This is the phase that
|
||||
makes a scan finish, because the job is concentration: the CVE is already inside 500, the question is
|
||||
whether it is inside 50. **If this passes, the model is a refinement, not the plan** — and knowing that is
|
||||
whether it is inside 50. **If this passes, the model is a refinement, not the plan**, and knowing that is
|
||||
worth more than assuming it.
|
||||
**Implemented 2026-09-11, measurement in flight.** `Evidence.score` is the published weighted sum, and
|
||||
every weight is in one table in `model/evidence.py`: per open sink +1.0 (capped at 5), per cleansed or
|
||||
@@ -304,17 +304,17 @@ A phase that cannot show that is not an improvement, it is a story.
|
||||
facts and a weight table can do; the ties are phase 3's. After that, within-tier order is phase 3's -- a model reading the code the ties cannot --
|
||||
or one more exact path fact (`distance`), and the gate stays unmet until one of them moves it.
|
||||
|
||||
- **Phase 3 — the LLM ranker.** Same evidence, ordered by a model. Gate: beats phase 2 leave-one-repository-out.
|
||||
- Phase 3, the LLM ranker. Same evidence, ordered by a model. Gate: beats phase 2 leave-one-repository-out.
|
||||
If it does not, phase 2 ships and phase 3 does not.
|
||||
- **Phase 4 — break the closed loop.** An exploration slice (~10% of budget sampled outside the top-K) and
|
||||
- Phase 4, break the closed loop. An exploration slice (~10% of budget sampled outside the top-K) and
|
||||
full labelling on repositories small enough to arbitrate every region. Gate: a measured estimate of the
|
||||
selection bias in phase 3's numbers.
|
||||
- **Phase 5 — prompt optimisation (GEPA / SIMBA / DSPy).** Only after 3 and 4. An optimiser run against a
|
||||
- Phase 5, prompt optimisation (GEPA / SIMBA / DSPy). Only after 3 and 4. An optimiser run against a
|
||||
censored metric optimises the censorship.
|
||||
|
||||
## What the prompt has to carry
|
||||
|
||||
The question put to the model is **not** "is this vulnerable" — that is adjudication, it is the arbiter's job,
|
||||
The question put to the model is **not** "is this vulnerable". That is adjudication, the arbiter's job,
|
||||
and it is the one thing a ranker must never be asked. It is:
|
||||
|
||||
> Given a fixed budget of expensive dataflow queries, which of these sites are worth spending it on?
|
||||
@@ -326,14 +326,14 @@ reachable from a request handler.
|
||||
|
||||
## Constraints, from measurement and from prior scars
|
||||
|
||||
- **Cost.** One model call per region is 4,463 calls a scan. Batch over a prefilter, or it does not ship.
|
||||
- **The loop is closed.** A region the ranker excludes is never arbitrated, never labelled, so the ranker
|
||||
- Cost. One model call per region is 4,463 calls a scan. Batch over a prefilter, or it does not ship.
|
||||
- The loop is closed. A region the ranker excludes is never arbitrated, never labelled, so the ranker
|
||||
never learns it was wrong. Every run confirms the ordering it already had.
|
||||
- **N projects.** Four pinned CVEs is not a training set. This project already has the closed-loop
|
||||
- N projects. Four pinned CVEs is not a training set. This project already has the closed-loop
|
||||
train-on-test scar; the reported figure must come from a repository the optimiser never saw.
|
||||
- **Determinism.** Two runs of one repository must agree or a baseline diff is meaningless. The ranking is
|
||||
- Determinism. Two runs of one repository must agree or a baseline diff is meaningless. The ranking is
|
||||
recorded in the manifest.
|
||||
- **Degradation.** With no model configured, phases 0–2 still rank, and the report says which ranker ordered
|
||||
- Degradation. With no model configured, phases 0–2 still rank, and the report says which ranker ordered
|
||||
the scan.
|
||||
|
||||
## Cross-assessment against the measurements
|
||||
@@ -342,7 +342,7 @@ Read against what was actually measured today, the plan above has two errors and
|
||||
here, corrected, rather than silently rewritten, because the corrections are themselves results.
|
||||
|
||||
**Phase 1's gate is wrong.** It says pruning makes `pmpro`'s taint query complete. It does not. Pruning
|
||||
removes 53% of requests, and 1,177 × 6.63 s is still **2.2 hours** — nowhere near the 300 s default or the
|
||||
removes 53% of requests, and 1,177 × 6.63 s is still **2.2 hours**, nowhere near the 300 s default or the
|
||||
2,400 s that was tried. Pruning halves the problem; it does not solve it. The honest gate is "53% fewer
|
||||
requests issued, measured", and completion belongs to phase 2. Stating otherwise would have sent someone
|
||||
looking for a bug in phase 1 when phase 1 had done exactly what it can.
|
||||
@@ -356,8 +356,8 @@ the numbers it had; the numbers were the instrument's.
|
||||
**Ranking's job is concentration, not inclusion.** The brief implies the CVE is missed because the ranker
|
||||
excluded it. It did not: CVE-2023-23488's region sits at position 390 of 4,463, inside the 500-region budget,
|
||||
and was examined. It goes unreported because the budget is spent on 500 regions when 50 would do, and 500
|
||||
is unaffordable at 6.63 s per family per region. So the ranker's success metric is not "is the CVE in the
|
||||
budget" — it already is — but **"how small can the budget be while the CVE stays in it"**. That is a
|
||||
is unaffordable at 6.63 s per family per region. So the ranker's success metric is
|
||||
**"how small can the budget be while the CVE stays in it"**; the CVE is already in the budget. That is a
|
||||
different objective, and it is the one that makes a scan finish.
|
||||
|
||||
**The gap: nothing above says what the evidence IS.** "Sink shape" and "cleansing on the call" are named,
|
||||
@@ -366,24 +366,24 @@ check for language leakage. The rest of this brief defines it.
|
||||
|
||||
Two things the measurements confirm and the plan should lean on harder:
|
||||
|
||||
- **The joins are cheap and must survive any ranker.** 22% of query cost, two of three CVEs. A ranker that
|
||||
- The joins are cheap and must survive any ranker. 22% of query cost, two of three CVEs. A ranker that
|
||||
demotes field- or hook-carried regions because they show no direct source is a ranker that loses those
|
||||
CVEs. The evidence vector below therefore carries the join's own stage-one result as a field.
|
||||
- **84% of regions in one tier means any separation is progress**, and the separation does not need to be
|
||||
- 84% of regions in one tier means any separation is progress, and the separation does not need to be
|
||||
clever to be large. A tier 0 that is merely exact already reorders the budget more than the current ranker
|
||||
ever has.
|
||||
|
||||
## The evidence vector, defined
|
||||
|
||||
The unit is a **(region, family) pair**. For each, a fixed vector of fields is computed. Every field is
|
||||
defined in terms of **fact-table categories** — source, sink, sanitizer, dispatch, access — never in terms of
|
||||
defined in terms of **fact-table categories** (source, sink, sanitizer, dispatch, access), never in terms of
|
||||
a language's syntax, which is what makes the definition language-agnostic: any language with a fact table
|
||||
gets the same vector, and a heuristic that mentions `$wpdb` or `wp_ajax_` has by construction left the
|
||||
vector and must be rejected in review.
|
||||
|
||||
**Where it is computed matters as much as what it is.** The graph is already built for the arbiter, at 76 s
|
||||
for 637 files. Extracting these fields is a CPG query **without dataflow** — call nodes, their arguments,
|
||||
their enclosing methods, the call graph one hop out — which is milliseconds per call rather than 6.63 s per
|
||||
for 637 files. Extracting these fields is a CPG query **without dataflow** (call nodes, their arguments,
|
||||
their enclosing methods, the call graph one hop out), which is milliseconds per call rather than 6.63 s per
|
||||
request. The extractor is therefore a sibling of `taint.sc` that never calls `reachableByFlows`:
|
||||
|
||||
> **Tiering is the CPG query without dataflow. Arbitration is the CPG query with it. Same graph, same
|
||||
@@ -393,7 +393,7 @@ That is the whole architecture in one line, and it is why the evidence is langua
|
||||
merely language-neutral-in-intent: the CPG normalises syntax across frontends, so "the argument of this call
|
||||
is a literal" means the same thing in PHP, Python and C.
|
||||
|
||||
### Sink evidence — from `[[sink]]` facts of family F
|
||||
### Sink evidence, from `[[sink]]` facts of family F
|
||||
|
||||
| field | definition | exactness |
|
||||
|---|---|---|
|
||||
@@ -401,7 +401,7 @@ is a literal" means the same thing in PHP, Python and C.
|
||||
| `shape` | per sink call, one of: **`literal`** (the vulnerable-position argument is a constant), **`bound`** (the call is a declared-safe form — a `parameterized` sanitizer fact wraps it, or its arity/shape matches the bound form the taint query already recognises), **`computed`** (the argument is an expression: concatenation, interpolation, a variable, a call result) | exact for `literal` and `bound`; `computed` is the residue |
|
||||
| `cleansed_on_call` | per sink call, whether a sanitizer of F is an **ancestor of the vulnerable-position argument in the AST** — not merely present in the same function | exact, and this is the field a text scan gets wrong: `esc_sql` in scope on a different statement is `false` here |
|
||||
|
||||
### Source evidence — from `[[source]]` facts and the entry-point contract
|
||||
### Source evidence, from `[[source]]` facts and the entry-point contract
|
||||
|
||||
| field | definition | exactness |
|
||||
|---|---|---|
|
||||
@@ -411,7 +411,7 @@ is a literal" means the same thing in PHP, Python and C.
|
||||
| `access` | the declared access level, **with provenance**: `declared` (a registration or route contract says so) or `inferred` (nothing found) — because "public" meaning "no decorator found" is the false positive the project has already paid for | exact |
|
||||
| `carried` | the region reads a **field** whose assignment is fed by a source anywhere in the file, or a **dispatch** value whose registered producer is — i.e. stage one of the two-stage join, which is already computed and memoised, reused here as evidence | exact where computed; absent (not false) where the join was not run |
|
||||
|
||||
### Path evidence — cheap structural proxies for what dataflow would establish
|
||||
### Path evidence, cheap structural proxies for what dataflow would establish
|
||||
|
||||
| field | definition |
|
||||
|---|---|
|
||||
@@ -438,14 +438,14 @@ is a literal" means the same thing in PHP, Python and C.
|
||||
|
||||
Two properties are load-bearing and must hold in every implementation:
|
||||
|
||||
1. **Only tier 0 excludes; every other tier orders.** So the only place exactness is *required* is tier 0,
|
||||
1. Only tier 0 excludes; every other tier orders. So the only place exactness is *required* is tier 0,
|
||||
and tier 0's predicate has a proof: a family with no sink in reach cannot produce a flow, in any language,
|
||||
under any dataflow model. Above tier 0, being wrong costs position, and `regions_truncated` reports what
|
||||
position cost.
|
||||
2. **Tier 0's scope must match the arbiter's.** At `callDepth = 0` the scope is the region; at depth k it is
|
||||
2. Tier 0's scope must match the arbiter's. At `callDepth = 0` the scope is the region; at depth k it is
|
||||
the region plus its reachable methods. A tier 0 computed on the file when the arbiter will look k hops
|
||||
out is not exact and must not be called so. The repository-wide form (`family_in_repo == false`) is exact
|
||||
at every depth and is the cheap first cut — it alone removes all 487 `deserialization` requests on `pmpro`.
|
||||
at every depth and is the cheap first cut. It alone removes all 487 `deserialization` requests on `pmpro`.
|
||||
|
||||
**Within a tier**, order by a score. In phase 2 that score is a fixed weighted sum over the vector, published
|
||||
in the manifest so two runs agree. In phase 3 it is the model's ordering **of the same vector**, which is
|
||||
@@ -456,23 +456,23 @@ what makes the two comparable: same input, different ordering function, one held
|
||||
The vector above is exact wherever exactness was cheap. What remains genuinely uncertain is narrower than
|
||||
"ranking":
|
||||
|
||||
- **Ambiguous cleansing.** `cleansed_on_call` is exact when the sanitizer wraps the argument directly. It is
|
||||
`false` — correctly, but unhelpfully — when cleansing happens two statements earlier into a variable that is
|
||||
- Ambiguous cleansing. `cleansed_on_call` is exact when the sanitizer wraps the argument directly. It is
|
||||
`false` (correctly, but unhelpfully) when cleansing happens two statements earlier into a variable that is
|
||||
then interpolated. Deciding whether *that* cleansing reaches *this* call is reading the code, and it is the
|
||||
single largest source of tier-2/tier-3 misplacement.
|
||||
- **Unfamiliar shapes.** A sink call whose form the `bound` rule does not recognise, in a framework with no
|
||||
- Unfamiliar shapes. A sink call whose form the `bound` rule does not recognise, in a framework with no
|
||||
facts yet. The model can say "this looks bound" as an *ordering* opinion; it cannot add a `bound` fact,
|
||||
because a fact is policy.
|
||||
- **Within-tier priority** when the fixed score ties, which on a real repository it will, constantly.
|
||||
- Within-tier priority when the fixed score ties, which on a real repository it will, constantly.
|
||||
|
||||
And the boundaries:
|
||||
|
||||
- **The model never computes tier 0.** Exclusion is exact or it is a silent miss, and a model's "no sink
|
||||
- The model never computes tier 0. Exclusion is exact or it is a silent miss, and a model's "no sink
|
||||
here" is not a proof.
|
||||
- **The model never sees the question "is this vulnerable".** It sees "which of these is worth an expensive
|
||||
- The model never sees the question "is this vulnerable". It sees "which of these is worth an expensive
|
||||
query", with the vector and the sink call's source text. Adjudication is the arbiter's, and keeping it
|
||||
there is what keeps the gates byte-identical.
|
||||
- **The model's input is the vector plus text, never the arbiter's verdict** on the same run — that is the
|
||||
- The model's input is the vector plus text, never the arbiter's verdict on the same run. That is the
|
||||
closed loop feeding on itself.
|
||||
|
||||
## Metrics, so every phase can fail
|
||||
@@ -498,8 +498,8 @@ answer is visible:
|
||||
1. ~~Where does the evidence extractor live?~~ **It is a CPG query without dataflow, a sibling of `taint.sc`
|
||||
in `cpg/queries/`.** Not the mapper: the mapper reads text and would have to re-derive per language what
|
||||
the graph already normalises. The vector is language-agnostic *because* the graph is.
|
||||
2. ~~Per file or per reachable set?~~ **Both, in that order.** `family_in_repo` first — exact at any depth,
|
||||
free, a fifth of the budget on `pmpro` — then the reachable set at the arbiter's own `callDepth`. Never the
|
||||
2. ~~Per file or per reachable set?~~ **Both, in that order.** `family_in_repo` first (exact at any depth,
|
||||
free, a fifth of the budget on `pmpro`), then the reachable set at the arbiter's own `callDepth`. Never the
|
||||
file alone above depth zero, because that is not exact and must not be called tier 0.
|
||||
3. How many pinned repositories before leave-one-out means anything? Four is not enough. This is the
|
||||
corpus ask, and it is the same ask contributor-scan 5.8 made: plugins chosen by installed base, CVEs read
|
||||
@@ -507,7 +507,7 @@ answer is visible:
|
||||
4. Does the arbiter's verdict feed back as a label? **Only through the exploration slice.** A verdict on a
|
||||
region the ranker chose is a label the policy selected, and training on it closes the loop. A verdict on
|
||||
an exploration region is the only unbiased label the scan produces, and it is the one worth keeping.
|
||||
5. **New:** the weighted sum in phase 2 has weights. Where do they come from before any model exists?
|
||||
5. New: the weighted sum in phase 2 has weights. Where do they come from before any model exists?
|
||||
Proposed: set by hand from the tier semantics, published in the manifest, and treated as a baseline for
|
||||
phase 3 to beat — not tuned, because tuning four weights on four CVEs is the overfit the plan exists to
|
||||
phase 3 to beat, not tuned, because tuning four weights on four CVEs is the overfit the plan exists to
|
||||
avoid.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Brief: ir-consolidation — stop maintaining an IR we did not need, and adopt one we do
|
||||
# Brief: ir-consolidation, stop maintaining an IR we did not need, and adopt one we do
|
||||
|
||||
> **Status 2026-10-02:** dormant. Brief only and unapproved. The subtraction it asks for still needs a measured recall comparison before the flat IR goes.
|
||||
|
||||
@@ -12,16 +12,16 @@
|
||||
Two different questions live inside that, and they have opposite answers. Keeping them apart is most of the
|
||||
value of this brief.
|
||||
|
||||
## Question 1 — the source IR we already replaced and never retired
|
||||
## Question 1, the source IR we already replaced and never retired
|
||||
|
||||
`semantic/` is **3,218 lines, 16% of the codebase**: a hand-written IR (`ir.py`, `cst.py`) and a flat taint
|
||||
engine (`taint.py`), plus the overlay that runs them. `model-grounded-detection` measured that IR against
|
||||
Joern's CPG and the result was decisive — **2.2% entailment against 41.3%** — which is why the CPG was
|
||||
Joern's CPG and the result was decisive, **2.2% entailment against 41.3%**, which is why the CPG was
|
||||
adopted for the model layer. The flat IR was never removed, and still runs as a proposer in every scan.
|
||||
|
||||
So the outgrowth is not something we are about to add. It is something already carried.
|
||||
|
||||
**Vine and P-Code do not help here.** Both are *binary* IRs — Vine lifts executables (BitBlaze, UC Berkeley,
|
||||
**Vine and P-Code do not help here.** Both are *binary* IRs. Vine lifts executables (BitBlaze, UC Berkeley,
|
||||
GPL-2.0, last significant activity 2009), and P-Code lifts machine code. Our source languages are Python,
|
||||
PHP, JavaScript and Java, and the source IR we should adopt is the one we already have. The work is
|
||||
subtraction, not substitution:
|
||||
@@ -32,16 +32,16 @@ subtraction, not substitution:
|
||||
|
||||
A candidate ~1,000-line deletion with a number attached, which is the only kind worth doing.
|
||||
|
||||
## Question 2 — the abstraction we are missing, where an existing IR is exactly right
|
||||
## Question 2, the abstraction we are missing, where an existing IR is exactly right
|
||||
|
||||
`c/memory` has produced **zero true positives** and three distinct classes of false one. The reason is
|
||||
structural and now stated in the report (Req 5.6): it models unbounded string CALLS, while real C overflows
|
||||
are *arithmetic* — an index, a pointer offset, a `memcpy` whose size is computed. Every one of libpng's is,
|
||||
are *arithmetic* (an index, a pointer offset, a `memcpy` whose size is computed). Every one of libpng's is,
|
||||
and its library code never calls a modelled string API at all.
|
||||
|
||||
"Can this expression exceed this allocation" is a value question, not a reachability one. **Rolf Rolles'
|
||||
GhidraPAL carries a three-valued-logic abstract interpreter over Ghidra P-Code** — each bit definite 0,
|
||||
definite 1, or unknown, with abstract transformers for `INT_ADD`, `INT_MULT`, `INT_AND`, the comparisons —
|
||||
GhidraPAL carries a three-valued-logic abstract interpreter over Ghidra P-Code**, with each bit definite 0,
|
||||
definite 1, or unknown, and with abstract transformers for `INT_ADD`, `INT_MULT`, `INT_AND`, the comparisons,
|
||||
validated against concrete execution in its own tests. That is precisely the shape of analysis the C family
|
||||
lacks, and it exists.
|
||||
|
||||
@@ -55,13 +55,13 @@ lacks, and it exists.
|
||||
|
||||
**angr is the strongest candidate and Vine is not one.** Three things decide it:
|
||||
|
||||
* **It is Python.** This codebase is Python with `dependencies = []` in the core, and every heavyweight
|
||||
* It is Python. This codebase is Python with `dependencies = []` in the core, and every heavyweight
|
||||
capability so far has been reached across a subprocess boundary rather than imported. angr fits that
|
||||
pattern as an extra, the way `harnessx` and the tree-sitter grammars already do.
|
||||
* **Value-set analysis is the abstraction we are missing**, by name. "Can this index exceed this allocation"
|
||||
* Value-set analysis is the abstraction we are missing, by name. "Can this index exceed this allocation"
|
||||
is a VSA question, and VSA is abstract interpretation rather than SMT -- which is the choice
|
||||
`model-grounded-detection` already made and gave its reasons for.
|
||||
* **BSD-2-Clause**, against Vine's GPL-2.0. This project has a standing rule about never vendoring code
|
||||
* BSD-2-Clause, against Vine's GPL-2.0. This project has a standing rule about never vendoring code
|
||||
without a stated licence, and a GPL dependency is a decision about the whole tool rather than about one
|
||||
analysis.
|
||||
|
||||
@@ -71,19 +71,19 @@ thing to read before writing any abstract domain here, whether the platform unde
|
||||
|
||||
It is also the honest answer to a decision this project already took. `model-grounded-detection` chose
|
||||
abstract interpretation over SMT, and chose to build the model from code because execution cannot be
|
||||
assumed. A bit-level abstract interpreter over P-Code is *still abstract interpretation* and still static —
|
||||
what changes is the artifact: it needs a **built** library rather than its source.
|
||||
assumed. A bit-level abstract interpreter over P-Code is *still abstract interpretation* and still static.
|
||||
What changes is the artifact: it needs a **built** library rather than its source.
|
||||
|
||||
## What this brief must decide
|
||||
|
||||
1. Whether the source-IR subtraction is safe, measured rather than assumed.
|
||||
2. Whether the C arithmetic class is worth a second pipeline at all, or whether `c/memory` stays scoped to
|
||||
string misuse and says so — which is where task 2.16 left it, deliberately.
|
||||
string misuse and says so, which is where task 2.16 left it, deliberately.
|
||||
3. If it is worth it: **angr** first, on language, licence and the fact that value-set analysis is the
|
||||
named abstraction -- with GhidraPAL read as the worked example of a bit-precise domain rather than as a
|
||||
platform. Vine is dormant since 2009 and is not a candidate.
|
||||
4. What a second pipeline costs in the one currency this project has been strict about — **not lines, but
|
||||
the number of things that can silently return nothing.** This session found three separate quiet failures
|
||||
4. What a second pipeline costs in the one currency this project has been strict about,
|
||||
**the number of things that can silently return nothing rather than lines.** This session found three separate quiet failures
|
||||
in one PHP scan. A binary pipeline adds a build step, a lifter, and a domain, each of which can produce
|
||||
an empty result that looks like a clean one.
|
||||
|
||||
|
||||
@@ -14,13 +14,13 @@ experiments.
|
||||
|
||||
Six facts from the repository and the recorded results shape the design:
|
||||
|
||||
- **The deployment target is a repository nobody trained on.** Maintainer: "the tool will never run against the
|
||||
- The deployment target is a repository nobody trained on. Maintainer: "the tool will never run against the
|
||||
test cases it's being trained with. it's purpose is a repo safety net." (Req 7). The primary unit is a candidate
|
||||
in changed code (`ousast pre-push`, base..head), and whole-repository `scan` is the secondary mode. Every
|
||||
reported number therefore comes from folds grouped by repository across all corpora. Features are
|
||||
repository-agnostic by an allow-list. Fit to a user's own repository is an opt-in layer outside the reported
|
||||
numbers.
|
||||
- **Labels are scarce, and full plane signals exist on 43 rows.** The only recorded plane run with
|
||||
- Labels are scarce, and full plane signals exist on 43 rows. The only recorded plane run with
|
||||
verify/agree/tie-break is `validation-46`: 15 cases, 43 candidates after triage, 19 of them with
|
||||
`site_match` true, all from population v2
|
||||
(`~/ousast-results/plane/validation-46/*-final/agreed.json`). The pair corpus is larger (427 pairs and 452
|
||||
@@ -30,19 +30,19 @@ Six facts from the repository and the recorded results shape the design:
|
||||
classifier over verify signals cannot be evaluated today. The design starts from the signals every labelled row
|
||||
can have, harvests the rest at a stated cost, and accepts that data will stay small: the engine learns by
|
||||
memory and compilation, not by fitting parameters.
|
||||
- **The core install is PyYAML only** (`pyproject.toml:11`). numpy, scipy and scikit-learn are not installed in
|
||||
- The core install is PyYAML only (`pyproject.toml:11`). numpy, scipy and scikit-learn are not installed in
|
||||
`.venv`. Retrieval (a Gower distance and a cosine over at most 40 vectors) and calibration (two-parameter Platt)
|
||||
stay pure Python; the model calls go through the plane's metered client (section 4).
|
||||
- **`pre-push` already has the decision seam.** `admit_candidates` (`push/policy.py:879`) applies one evidence
|
||||
- `pre-push` already has the decision seam. `admit_candidates` (`push/policy.py:879`) applies one evidence
|
||||
bar to `CandidateDelta`s with `novelty` (`push/policy.py:61-74`). Its capability registry is empty by design
|
||||
(`push/policy.py:802-823`: "No shipped declaration is enabled"), so `pre-push` blocks nothing today. The
|
||||
engine's operating point replaces the capability-eligibility reasons, and the structural reasons stay.
|
||||
- **The memory store is the system of record and has the needed shape.** Rows are keyed by repository and pin,
|
||||
- The memory store is the system of record and has the needed shape. Rows are keyed by repository and pin,
|
||||
with a closed `KINDS` tuple (`plane/memory.py:52`). There are two backends (FileStore, S3Store) behind one
|
||||
interface (`plane/memory.py:176-291`), content-addressed blobs (`put_facts`, `plane/memory.py:249-255`), and a
|
||||
train-on-test `Guard` (`improve/memory.py:95-133`). Features, labels, decisions, model artifacts and experiment
|
||||
outcomes become new row kinds and blob prefixes. No second store is added.
|
||||
- **Hand-written framework knowledge is embedded in the detectors** (Req 8). Examples are WordPress `$wpdb`,
|
||||
- Hand-written framework knowledge is embedded in the detectors (Req 8). Examples are WordPress `$wpdb`,
|
||||
hooks and `maybe_unserialize` in `ruleset/semantic/php.toml` and `ruleset/php/rules.toml`, and Flask and
|
||||
Django names in `ruleset/semantic/python.toml`. Section 2 tags each entry. Priors become optional features
|
||||
that are off by default. Framework roles are inferred per repository instead.
|
||||
@@ -548,7 +548,7 @@ If the pool cannot satisfy a minimum, take what exists and record `balance: "sho
|
||||
**The evaluation boundary (Req 3.4, 7.1).** `learn/retrieve.py: eligible(example, target, fold)` is the only way
|
||||
an example enters a prompt, for retrieved examples and for the compiled demonstrations alike:
|
||||
|
||||
- `example.group` is not in the fold's evaluation groups (all of them, not just the target's), and not the
|
||||
- `example.group` is not in the fold's evaluation groups (every evaluation group, including the target's), and not the
|
||||
target's group after the CVE/alias merge (`Groups`, `learn/labels.py:182`);
|
||||
- leave-one-source-out: `example.source` is not the held-out source;
|
||||
- leave-one-framework-out on F: `F not in example.frameworks`; a compiled demonstration tagged F is swapped for its
|
||||
@@ -669,23 +669,23 @@ million. Token sizes are estimates to be replaced by the smoke run's measured `u
|
||||
| candidate (80-line excerpt ~1,100, signals and roles ~300) | ~1,400 | no |
|
||||
| output (JSON, rationale <= 3 sentences) | ~200 | -- |
|
||||
|
||||
- **Per candidate, `k = 1`**: 2,750 x 0.014 + 5,950 x 0.44 + 200 x 1.32 per million = **$0.0029**.
|
||||
- **Each extra sample** (same prompt, fully cached): 8,700 x 0.014 + 200 x 1.32 per million = **$0.0004**. So
|
||||
- Per candidate, `k = 1`: 2,750 x 0.014 + 5,950 x 0.44 + 200 x 1.32 per million = **$0.0029**.
|
||||
- Each extra sample (same prompt, fully cached): 8,700 x 0.014 + 200 x 1.32 per million = **$0.0004**. So
|
||||
`k = 5` is **$0.0045** with cache hits, and $0.0146 if the cache does not hit (the smoke run decides the default).
|
||||
- **Embedding**: $0.000024 per candidate; the whole store about $0.10 once.
|
||||
- **Per compile**: bootstrap 150 x $0.0029 = $0.44; proposals ~$0.01; scoring 600 x $0.0029 = $1.74; total
|
||||
- Embedding: $0.000024 per candidate; the whole store about $0.10 once.
|
||||
- Per compile: bootstrap 150 x $0.0029 = $0.44; proposals ~$0.01; scoring 600 x $0.0029 = $1.74; total
|
||||
**~$2.2, ceiling $3.00**.
|
||||
- **Per full evaluation** of one program, evaluated families only (475 verified rows outside memory,
|
||||
- Per full evaluation of one program, evaluated families only (475 verified rows outside memory,
|
||||
config_secrets and unknown) plus a 200-unit benign-delta sample for false BLOCKs:
|
||||
- outer folds: 675 candidates, **$2.0 at k = 1, $3.0 at k = 5** (cached);
|
||||
- leave-one-source-out: 475 more, **$1.4 / $2.1**;
|
||||
- leave-one-framework-out: $0 today (no framework at the floor);
|
||||
- total **~$3.4 (k = 1) to $5.2 (k = 5)**, ceiling $7.
|
||||
- **Learning curves** (section 7): 6 memory-size points + 3 source-count points on a fixed 200-candidate subset at
|
||||
- Learning curves (section 7): 6 memory-size points + 3 source-count points on a fixed 200-candidate subset at
|
||||
`k = 1`: 9 x 200 x $0.0029 = **~$5.2**, ceiling $7.
|
||||
- **A program-variant A/B** on 300 paired candidates: arm A replays cached responses from the evaluation ($0);
|
||||
- A program-variant A/B on 300 paired candidates: arm A replays cached responses from the evaluation ($0);
|
||||
arm B costs **$0.9 (k = 1) to $1.35 (k = 5)**.
|
||||
- **Deployment**: `pre-push` with ~5 candidates is ~$0.02 per push at `k = 5`; a plane Run case with ~30 candidates
|
||||
- Deployment: `pre-push` with ~5 candidates is ~$0.02 per push at `k = 5`; a plane Run case with ~30 candidates
|
||||
~$0.14.
|
||||
|
||||
**Against the balance.** DeepSeek was ~$19.6 before the harvest; the harvest's verify passes are ~$6 expected and
|
||||
@@ -807,10 +807,10 @@ flagged in any pass as 17 for the plane vs 19 for the reference, of 20. The plan
|
||||
(`plane/tasks/verify.py:1-9, 164`, `CALLERS_CAP = 8` at :34), and the comparison is confounded (other changes,
|
||||
stochastic passes, v2 is where it was seen).
|
||||
|
||||
- **Arms.** A is `verify` as today. B is `verify` with `OUSAST_VERIFY_CALLERS=off`, a new switch (about 3 lines)
|
||||
- Arms. A is `verify` as today. B is `verify` with `OUSAST_VERIFY_CALLERS=off`, a new switch (about 3 lines)
|
||||
that makes `callers_line` return empty. Same model (`deepseek-flash`), `PER_HUNT = 6`, `MAX_STEPS = 6`, same
|
||||
candidates, same `facts.json`, same runner image digest, two passes (a, b) per arm.
|
||||
- **Units: the development corpus, not v2** (v2 generated the hypothesis):
|
||||
- Units: the development corpus, not v2 (v2 generated the hypothesis):
|
||||
- population v1, 11 cases, whose engine scans are recorded under `~/ousast-results/independent-v1/scans/` (44
|
||||
scans);
|
||||
- pmpro, MediaWiki, WP Statistics (dev-php);
|
||||
@@ -821,15 +821,15 @@ stochastic passes, v2 is where it was seen).
|
||||
model-free and frozen in the units file: per pin, the fix-changed functions, plus the top engine- and
|
||||
quick-alert functions, up to 8 candidates per pin, on the vulnerable and the fixed pin. Only units where arm A's
|
||||
callers line is non-empty are *treated*. The primary analysis is restricted to them, and the rest are reported.
|
||||
- **Metric.** Primary: the per-positive-unit rate of "flagged in any pass" (the 17-vs-19 metric), with positives
|
||||
- Metric. Primary: the per-positive-unit rate of "flagged in any pass" (the 17-vs-19 metric), with positives
|
||||
from the label builder (fix-range functions on the vulnerable pin). Secondary: the flag rate on fixed-pin
|
||||
negatives (false flags), and usd per candidate.
|
||||
- **Sample size and MDE.** About 55 positive units across 41 repositories (about 28 dev + 27 pointer pairs). With
|
||||
- Sample size and MDE. About 55 positive units across 41 repositories (about 28 dev + 27 pointer pairs). With
|
||||
a discordance rate of about 0.15, alpha 0.05 and power 0.8, the paired MDE is `(1.96 + 0.84) * sqrt(0.15) /
|
||||
sqrt(55)`, about **0.15**. The observed gap of 0.10 needs about 115 positive units, so an inconclusive result is
|
||||
the likely and honest outcome. It is registered as such in advance, and a follow-up needs new units, not a
|
||||
continuation.
|
||||
- **Budget.** Hunts per pass per arm: 14 dev repositories x 2 pins x up to 3 files (84) + 27 pairs x 2 sides (54)
|
||||
- Budget. Hunts per pass per arm: 14 dev repositories x 2 pins x up to 3 files (84) + 27 pairs x 2 sides (54)
|
||||
= 138. Four arm-passes give 552 hunts x $0.014, **about $7.7 expected**. The ceiling is **$15 total, $7.5 per
|
||||
arm**, enforced by the task budgets (`plane/budget.py`). Candidate generation reuses the recorded v1 engine
|
||||
scans. For dev-php it runs the engine once per pin on the host, one container at a time, from a frozen source
|
||||
@@ -1111,30 +1111,30 @@ Each commit is gated on the full suite's own exit code, `ruff` and `mypy`. None
|
||||
commit 9 adopts a program, and without an adopted program every path is today's. Steps that spend model money
|
||||
start only after the maintainer confirms their budget against the balance read just before.
|
||||
|
||||
1. **Schema and features** (1.1-1.3, 7.2) -- done.
|
||||
2. **Framework tags** (8.2, 8.4) -- done.
|
||||
3. **Roles** (8.1) -- done.
|
||||
4. **Labels** (2.1-2.3, 7.1) -- done.
|
||||
5. **Harvest** (1.1, data) -- in progress: verify a/b on the verify-family pairs and the development cases (about
|
||||
1. Schema and features (1.1-1.3, 7.2) -- done.
|
||||
2. Framework tags (8.2, 8.4) -- done.
|
||||
3. Roles (8.1) -- done.
|
||||
4. Labels (2.1-2.3, 7.1) -- done.
|
||||
5. Harvest (1.1, data) -- in progress: verify a/b on the verify-family pairs and the development cases (about
|
||||
$6, ceiling $10), model roles on the candidate files (ceiling $10), engine features on the host.
|
||||
6. **Program, memory, compile, first evaluation** (3.1-3.6, 5.1-5.2, 7.3-7.4, 8.3): excerpts and the example
|
||||
6. Program, memory, compile, first evaluation (3.1-3.6, 5.1-5.2, 7.3-7.4, 8.3): excerpts and the example
|
||||
store, metered embeddings and the cache, retrieval with the boundary test, the program and response cache, folds,
|
||||
compile, cross-fitted calibration; then the paid slice (smoke, embed, compile, outer evaluation, ~$5, ceiling $7)
|
||||
and, after a second go-ahead, leave-one-source-out and the memory-size curves (~$7, ceiling $10), committed under
|
||||
`benchmarks/measurements/<date>-decision-engine-v1/`.
|
||||
7. **Experiments** (4.1-4.4): `learn/experiments.py`, `register`, `experiment-outcome`, the Run annotation checks,
|
||||
7. Experiments (4.1-4.4): `learn/experiments.py`, `register`, `experiment-outcome`, the Run annotation checks,
|
||||
`OUSAST_VERIFY_CALLERS`, exp-001 registered, run and recorded, and the program-variant experiment machinery.
|
||||
8. **Integration** (6.1-6.3, 7.3, 7.5): the `embed` and `decide` tasks with the seeded memory, the `scan` stage,
|
||||
8. Integration (6.1-6.3, 7.3, 7.5): the `embed` and `decide` tasks with the seeded memory, the `scan` stage,
|
||||
`pre-push` admission with the engine, report fields, the fallback line, and the opt-in adaptation layer.
|
||||
9. **Adopt** (3.3, 8.2): `ousast learn adopt`, the packaged program with the licence filter, the priors-on/off and
|
||||
9. Adopt (3.3, 8.2): `ousast learn adopt`, the packaged program with the licence filter, the priors-on/off and
|
||||
`k` experiments with recorded decisions.
|
||||
10. **v3 check** (5.3): `prediction-v3.json` is committed first, then v3 is run by its owner under protocol v3, and
|
||||
10. v3 check (5.3): `prediction-v3.json` is committed first, then v3 is run by its owner under protocol v3, and
|
||||
`result-v3-engine.json` is committed. Nothing between the two commits may touch the program, memory snapshot,
|
||||
features, priors or thresholds.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Label scarcity decides what can be evaluated.** The status of each family today:
|
||||
- Label scarcity decides what can be evaluated. The status of each family today:
|
||||
- memory (5 repository groups, 173/173 rows, almost all openssl and curl), config_secrets (5) and prototype are
|
||||
**insufficient data** (`labels.json` `verified_pin_labels`). Users get today's behaviour there, labelled as
|
||||
such, and no money is spent evaluating them.
|
||||
@@ -1143,16 +1143,16 @@ start only after the maintainer confirms their budget against the balance read j
|
||||
- The plane profile has 43 rows from one population until the harvest Run.
|
||||
- BLOCK at 95% precision is likely **not reachable** for most families on this data. The design reports "BLOCK
|
||||
not offered" instead of lowering the target.
|
||||
- **Weak signals, strong code.** The pair baseline has quick-rule labelled recall 0.24 and Youden -0.013 overall
|
||||
- Weak signals, strong code. The pair baseline has quick-rule labelled recall 0.24 and Youden -0.013 overall
|
||||
(`pairs.json` `overall`); the v2 engine recall is 0/17 (`results-v2.json`). The program reads the code, so it
|
||||
can beat the signals, but the verify history says the model over-flags (v1 1/12, v2 0/17 adjudicated true). A
|
||||
calibrated classifier that mostly says "advisory" is honest; recall at ADVISORY 0.90 may need a threshold with
|
||||
poor precision. That is reported, not tuned away on outer folds.
|
||||
- **LLM nondeterminism.** The same prompt can answer differently, and temperature 0 is not a provider guarantee.
|
||||
- LLM nondeterminism. The same prompt can answer differently, and temperature 0 is not a provider guarantee.
|
||||
Guards: every response cached and replayable (numbers reproduce at $0); sample 1 at temperature 0; the vote mean
|
||||
over `k` samples as the score; the A/B unit is paired and both arms' caches are recorded; a re-ask agreement rate
|
||||
on a fixed 30-candidate canary set is reported with each evaluation.
|
||||
- **Label leakage into prompts.**
|
||||
- Label leakage into prompts.
|
||||
- Through rationales: memory examples carry no rationale; demonstration rationales are produced without the label
|
||||
and scanned for identities and advisory ids; the candidate's prompt never holds `EXCLUDED_FIELDS` (test).
|
||||
- Through code: fixed-side excerpts can carry fix comments ("prevent XSS", a CVE id). Advisory ids and security
|
||||
@@ -1160,11 +1160,11 @@ start only after the maintainer confirms their budget against the balance read j
|
||||
the change under test, never the later fix.
|
||||
- Through memorisation: the chat model may have seen public CVE fixes in pre-training. Not removable; the v3
|
||||
check (unpublished fixes are rarer there) and per-source numbers expose it, and the report names it.
|
||||
- **Retrieval leakage.** Same repository under another name or corpus: group normalisation, CVE merge, the
|
||||
- Retrieval leakage. Same repository under another name or corpus: group normalisation, CVE merge, the
|
||||
near-duplicate drop at cosine >= 0.98 in evaluation, and `test_boundary`. Demonstrations from the compile split
|
||||
are, by construction, from repositories never evaluated. The seed for plane Runs filters on the host, where the
|
||||
labels are; the sandbox never receives an example of its own case's group.
|
||||
- **Label-tracking signals (maintainer decision 2026-10-01: "Neutralise both for v1").** The harvest found two
|
||||
- Label-tracking signals (maintainer decision 2026-10-01: "Neutralise both for v1"). The harvest found two
|
||||
signals that follow the pair label instead of the code: verify candidates on fixed sides named functions the side
|
||||
no longer declares (now never produced: a side's candidates are only the functions it declares, the rest counted
|
||||
`absent_on_side`), and the engine's "read the file, asked no question" state, which a fixed side reaches by losing
|
||||
@@ -1173,40 +1173,40 @@ start only after the maintainer confirms their budget against the balance read j
|
||||
the stored record keeps it. The engine returns only through a pre-registered A/B evaluated on assumed-benign and
|
||||
benign-control negatives, where it cannot track a fix. `ousast learn audit-leaks` reports how well every signal
|
||||
alone separates the sides of a pair and flags any above 0.20 (counts under `benchmarks/measurements/`).
|
||||
- **Earlier leakage paths still apply.** Hand rules and framework priors written after seeing a training repository
|
||||
- Earlier leakage paths still apply. Hand rules and framework priors written after seeing a training repository
|
||||
(the `$wpdb` rule from PMPro, the hook source from WP Statistics, `taint.sc:911-949`): priors off by default,
|
||||
leave-one-framework-out, best-effort `taught_by = [repo]`. The mechanism exporter's train-on-test leak
|
||||
(`closed-loop-train-on-test-leak.md`): no exporter output is a feature. Verify and tie-break fitted on v2: the v2
|
||||
fold is design-informed. Thresholds and instructions chosen on reported folds: cross-fitting and the compile
|
||||
split. v3: fail-closed sources, never opened. User dismissals: refused by both builders.
|
||||
- **Cost.** Model spend is now per candidate and per evaluation, not one-off. Measured (section 4.6) the first
|
||||
- Cost. Model spend is now per candidate and per evaluation, not one-off. Measured (section 4.6) the first
|
||||
slice fits today's balance; everything after it needs a top-up. Guards: `MeteredClient` ceilings on every task
|
||||
and command, a smoke check that measures real tokens and cache hits before any paid run, cached replay for
|
||||
re-analysis, unevaluable families skipped, `k = 1` if the prefix cache does not hit. Deployment cost (~$0.02 per
|
||||
push) is a user-facing change: `pre-push` without a key falls back, and the report states the spend.
|
||||
- **Model and version drift.** `deepseek-flash` is an alias whose weights can change, and so can OpenRouter's
|
||||
- Model and version drift. `deepseek-flash` is an alias whose weights can change, and so can OpenRouter's
|
||||
embedding route. The artifact records model ids, the Model parameters digest and the response `model` field;
|
||||
before adoption and before the v3 prediction the canary set is re-asked, and agreement below 0.9 with the
|
||||
cached answers, or an embedding dimension change, invalidates the evaluation (re-embed, re-evaluate) rather than
|
||||
being averaged in. Prices are data in the Model manifest; a price change changes only cost reports.
|
||||
- **Latency.** A model call per candidate is slower than a dot product. The 30 s `pre-push` deadline is met by
|
||||
- Latency. A model call per candidate is slower than a dot product. The 30 s `pre-push` deadline is met by
|
||||
concurrency and the `undecided: deadline` state; a push with many candidates gets partial decisions, stated.
|
||||
- **Packaging third-party code.** Demonstrations and memory hold excerpts of corpus repositories. Only
|
||||
- Packaging third-party code. Demonstrations and memory hold excerpts of corpus repositories. Only
|
||||
permissively licensed excerpts may ship in the package; the store is never shipped. A user without a store gets
|
||||
the demonstrations-only program, whose own curve point (memory 0%) and calibration are what they are told.
|
||||
- **Framework inference is itself a model.** Model roles cost money and can be wrong. `scan` without `[models]`
|
||||
- Framework inference is itself a model. Model roles cost money and can be wrong. `scan` without `[models]`
|
||||
gets only wrapper inference, whose reach on framework-heavy code without dependency source is poor. The report
|
||||
says which role source ran. Both are measured by the roles-on/off experiment. Nothing assumes they help.
|
||||
- **Host time on the 7 GB host.** Engine features remain the time cost (7-14 hours serial for the pair corpus,
|
||||
- Host time on the 7 GB host. Engine features remain the time cost (7-14 hours serial for the pair corpus,
|
||||
dev-php scans of 200-2,000 s each), run once per engine image from a frozen source export, one container at
|
||||
`--memory 3g` at a time, never beside the kind cluster's verify pods, cached by (content sha, image id). Model
|
||||
calls add wall time but no memory pressure; excerpts and embeddings are small (~8 KB per example with a 1,536-float
|
||||
vector, ~45 MB for 5,600 examples).
|
||||
- **Experiments will often be inconclusive.** With 41 development repositories the MDE is about 15 points.
|
||||
- Experiments will often be inconclusive. With 41 development repositories the MDE is about 15 points.
|
||||
Pre-registering that and recording "inconclusive" is the intended behaviour. The failure mode to avoid is
|
||||
extending an experiment after looking.
|
||||
|
||||
- **First paid slice, 2026-10-01 (tasks 6.2, 6.6; injection only).** Four instrument defects surfaced before any
|
||||
- First paid slice, 2026-10-01 (tasks 6.2, 6.6; injection only). Four instrument defects surfaced before any
|
||||
number was believed, each fixed with a test:
|
||||
- *The memory build joined no pair.* The harvest stores a pair side's record under its unit pin (the blob sha1 of
|
||||
the side's excerpt), not under `label_pin`; `ousast learn memory build --units <harvest units.json>` joins them.
|
||||
@@ -1227,7 +1227,7 @@ start only after the maintainer confirms their budget against the balance read j
|
||||
0.90 has precision 0.54 (prevalence 0.53); the majority `vulnerable` verdict alone has precision 0.86, recall 0.43.
|
||||
Canary re-ask agreement at temperature 0 was 0.83, below the 0.9 the adoption check requires.
|
||||
|
||||
- **Slice 2, 2026-10-02 (six families, after the advisory-fix harvest).** Two instrument defects stopped runs before any
|
||||
- Slice 2, 2026-10-02 (six families, after the advisory-fix harvest). Two instrument defects stopped runs before any
|
||||
number: the memory build's side reader knew only the first pair-kind source (all 194 advisory sides unread), and
|
||||
compiled demonstrations skipped the near-duplicate rule that retrieval and the pre-call assertion apply (the
|
||||
assertion stopped the untrusted_destination evaluation). Both fixed with tests. A confound to read with the new
|
||||
|
||||
@@ -25,29 +25,29 @@ money.
|
||||
|
||||
## Requirements (proposed)
|
||||
|
||||
1. **Mining.** From each fix pair, extract what the fixed function adds on the vulnerable flow: new calls,
|
||||
1. Mining. From each fix pair, extract what the fixed function adds on the vulnerable flow: new calls,
|
||||
new conditions guarding the sink, new or changed sink arguments. Record each as a candidate fact with its
|
||||
language, family, form (sanitizer call, guard predicate, safe argument, sink restriction) and the pairs
|
||||
that propose it. Mining is deterministic (tree-sitter diff of the two function bodies). No model calls.
|
||||
2. **Generalisation is explicit.** A candidate is a pattern, not a repository name: a callee name, an
|
||||
2. Generalisation is explicit. A candidate is a pattern, not a repository name: a callee name, an
|
||||
argument key, a predicate shape. Candidates naming project-specific identifiers are rejected (Req 8: no
|
||||
application-specific trees).
|
||||
3. **Admission by held-out separation.** Folds are by repository, as in the decision engine. A candidate is
|
||||
3. Admission by held-out separation. Folds are by repository, as in the decision engine. A candidate is
|
||||
admitted for a fold only if, on that fold's held-out pairs, adding it to the fact tables:
|
||||
- increases pairs where the vulnerable side keeps a path and the fixed side loses it or becomes
|
||||
sanitized;
|
||||
- and does not remove any held-out vulnerable-side path (no recall loss).
|
||||
A candidate proposed by fewer than 2 distinct repositories in the training folds is not tried.
|
||||
4. **No train-on-test.** Mining and admission never read held-out pairs; the closed-loop leak recorded in
|
||||
4. No train-on-test. Mining and admission never read held-out pairs; the closed-loop leak recorded in
|
||||
memory (`closed-loop-train-on-test-leak`) is the failure this rule exists to prevent. Population v3 is
|
||||
never read.
|
||||
5. **Measurement by the trace runner.** Each admission round re-runs the engine on the affected pins only
|
||||
5. Measurement by the trace runner. Each admission round re-runs the engine on the affected pins only
|
||||
(the runner's resume and selection already support this; on k3s the worker Deployment runs them in
|
||||
parallel). The outcome is the pair-asymmetry table per family, before and after, on held-out folds.
|
||||
6. **Gate before production.** An admitted fact enters `src/openultrasast/ruleset/semantic/*.toml` only after
|
||||
6. Gate before production. An admitted fact enters `src/openultrasast/ruleset/semantic/*.toml` only after
|
||||
the detection gates (`gate`, `map_gate`, `pair_gate`) show no regression, and only as a registered
|
||||
experiment whose decision rule was fixed before the run.
|
||||
7. **Coverage second.** After discharge facts, the same loop proposes source and sink additions from pairs
|
||||
7. Coverage second. After discharge facts, the same loop proposes source and sink additions from pairs
|
||||
where neither side has a path (91 of 105 today), admitted under the same held-out rule.
|
||||
|
||||
## How it relates to what exists
|
||||
|
||||
@@ -9,13 +9,13 @@ Maintainer direction and context: `brief.md`.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: a feature record per candidate built from every instrument and the memory store; a labelled
|
||||
- In scope: a feature record per candidate built from every instrument and the memory store; a labelled
|
||||
dataset with provenance; training, calibration and versioning of the decision engine; A/B experiments as plane
|
||||
Runs; learning curves and leave-one-population-out estimates; the engine's use in `ousast scan`, `ousast
|
||||
pre-push` and plane Runs; a final one-time check on frozen population v3.
|
||||
- **Out of scope**: new instruments (they plug in as signal producers later); deep or large models; any change to
|
||||
- Out of scope: new instruments (they plug in as signal producers later); deep or large models; any change to
|
||||
population v3 or its protocol; publishing results (licences still gate that).
|
||||
- **Adjacent expectations**: instruments keep working standalone; the memory store (harnessx-removal Req 6) is
|
||||
- Adjacent expectations: instruments keep working standalone; the memory store (harnessx-removal Req 6) is
|
||||
the system of record; the plane attributes every experiment's token spend per task.
|
||||
|
||||
## Requirements
|
||||
@@ -121,7 +121,7 @@ rule hits.
|
||||
reported unless the engine's decision changes.
|
||||
4. *(Amended 2026-10-03, maintainer decision; refines the BLOCK rule of 7.3 and 7.4. Tested by exp-004.)* BLOCK
|
||||
precision is demonstrated pooled across families, not per family:
|
||||
- **Normalisation**, declared before any run: each family's raw score is put on one scale by a per-family Platt
|
||||
- Normalisation, declared before any run: each family's raw score is put on one scale by a per-family Platt
|
||||
map (`learn.calibrate.fit_map` with its floors) fitted on training folds only. A family the training folds
|
||||
cannot map does not block on that fold.
|
||||
- **t_block**: one threshold on the normalised score for all families, chosen on training folds as the lowest
|
||||
|
||||
@@ -62,8 +62,8 @@ Principle for the roadmap: **classify first, verify before reporting, improve on
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: items 1–8; roadmap rewrite from one command; a closure decision on `authorization-obligations` 4.3, 5 and 6.
|
||||
- **Out**: new static shape families; hand-written facts beyond what a round proposes and validates; changing the evidence ladder or dispositions; any merge gate on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights.
|
||||
- In: items 1–8; roadmap rewrite from one command; a closure decision on `authorization-obligations` 4.3, 5 and 6.
|
||||
- Out: new static shape families; hand-written facts beyond what a round proposes and validates; changing the evidence ladder or dispositions; any merge gate on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights.
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
@@ -80,14 +80,14 @@ Principle for the roadmap: **classify first, verify before reporting, improve on
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `pair-corpus-honesty` (labels, tiers, splits, `known_limit`), `corpus-seeded-mechanisms` (exporter, LOO, lever), `tool_hunter` and `provider/openrouter`, `harness_ext`, HarnessX 0.1.0 (`meta_harness`, `rl`, `tracing.journal`), `tree-sitter-overlay-extra`, the sandbox and regress recipes.
|
||||
- **Downstream**: every roadmap number; `harnessx-self-improving-rulesets` (its evolve step is this spec); `authorization-obligations` 4.3/5/6; scan reports (findings carry class, detector and verifier).
|
||||
- Upstream: `pair-corpus-honesty` (labels, tiers, splits, `known_limit`), `corpus-seeded-mechanisms` (exporter, LOO, lever), `tool_hunter` and `provider/openrouter`, `harness_ext`, HarnessX 0.1.0 (`meta_harness`, `rl`, `tracing.journal`), `tree-sitter-overlay-extra`, the sandbox and regress recipes.
|
||||
- Downstream: every roadmap number; `harnessx-self-improving-rulesets` (its evolve step is this spec); `authorization-obligations` 4.3/5/6; scan reports (findings carry class, detector and verifier).
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: `pair-corpus-honesty` Req 1 (scorer) and Req 9/10 (row classification); `corpus-seeded-mechanisms` Req 4/5 (split by construction); `three-stage-scan` MAP (per-class hunters as proposers); `propose-adjudicate-prove` (verifiers keyed by class); `harnessx-self-improving-rulesets` (the evolve loop lands here).
|
||||
- **Folds in**: the retired `closed-loop-integrity` brief.
|
||||
- **Re-homes**: `authorization-obligations` 4.3 and the corpus half of 6.1; holds 5.
|
||||
- Extends: `pair-corpus-honesty` Req 1 (scorer) and Req 9/10 (row classification); `corpus-seeded-mechanisms` Req 4/5 (split by construction); `three-stage-scan` MAP (per-class hunters as proposers); `propose-adjudicate-prove` (verifiers keyed by class); `harnessx-self-improving-rulesets` (the evolve loop lands here).
|
||||
- Folds in: the retired `closed-loop-integrity` brief.
|
||||
- Re-homes: `authorization-obligations` 4.3 and the corpus half of 6.1; holds 5.
|
||||
|
||||
## Constraints
|
||||
|
||||
|
||||
@@ -150,20 +150,20 @@ tests/test_learning_families.py, test_learning_classify.py, test_learning_scorin
|
||||
```
|
||||
|
||||
### Modified Files
|
||||
- `src/openultrasast/pairs.py` — `PairCase.unscorable: str | None` computed at load (identical twin via normalized body hash, `known_limit` passthrough); duplicate report across pairs before split; `split_by_repository` helper honoring existing `split` fields and reporting straddles; `_evaluate_hunter_pair` delegates matching and leaks to `learning.scoring` through an injected callable so `pairs` does not import `learning`; `build_pair_signals` gains a `split` filter.
|
||||
- `src/openultrasast/benchmark.py` — `ExpectedFinding.family: str | None = None` (additive, stored as an opaque string; the Classifier validates loaded labels against the taxonomy so `pairs` and `benchmark` never import `learning`).
|
||||
- `src/openultrasast/tool_hunter.py` — `run_tool_hunter` gains `system_prompt`, `context_files`, `tools`, `max_chars` and `tags` parameters with today's values as defaults; finding parsing accepts an optional `family` field; the reply loop replays `reasoning_content`.
|
||||
- `src/openultrasast/hunter_tools.py` — four new tools beside `read_file`, `grep_repo`, `find_refs`: `read_definition(symbol)`, `entry_points(path)`, `flows(path, function)`, `obligations(path, function)`, each clamped to the repository and returning identifiers and line spans only; `HUNTER_TOOLS` schemas extended.
|
||||
- `src/openultrasast/provider/openrouter.py` — `OpenRouterChatClient` gains `extra_body` (thinking, response_format, logprobs) and retains the `reasoning_content` field on assistant messages; replaying it across tool turns is the hunter loop's job (`tool_hunter`), not the client's. No behavior change for existing callers.
|
||||
- `src/openultrasast/config.py` — `[models]` gains `chat_base_url`, `chat_api_key_env`, `judge`; `[learning]` section (`families_path`, `configs_dir`, `k_runs = 5`, `round_cost_cap_usd`, `minibatch`, `cutoff_date`); `load_dotenv` unchanged, tests inject endpoints explicitly.
|
||||
- `src/openultrasast/semantic/seed.py`, `semantic/loo.py`, `improve/evolve.py` — teacher selection through `learning.split.teachers`; refusals recorded; `evaluate_loo` teaches from train only.
|
||||
- `src/openultrasast/regress/candidate.py`, `regress/verdict.py` — `run_regression` accepts an optional `oracle: Callable[[SandboxResult], bool]`; `verdict_from_result` uses the oracle when given, exit code otherwise.
|
||||
- `src/openultrasast/cli.py` — `ousast learning classify|score|baseline|round|publish`; `pairs --hunter` uses the family scorer and `--k-runs`; `[models].hunter` resolution through `learning.endpoint`.
|
||||
- `pyproject.toml` — `tree-sitter-typescript>=0.23` in the semantic extra; `src/openultrasast/semantic/extra.py` maps `typescript`/`tsx`.
|
||||
- `benchmarks/pairs/vibe-py/catalog.toml` and `recipes.toml` — the five identical twins gain `known_limit = "identical_twin"`; absence rows gain `family` and `mode = "handler_context"` (re-harvest is a maintainer step).
|
||||
- `src/openultrasast/pairs.py`: `PairCase.unscorable: str | None` computed at load (identical twin via normalized body hash, `known_limit` passthrough); duplicate report across pairs before split; `split_by_repository` helper honoring existing `split` fields and reporting straddles; `_evaluate_hunter_pair` delegates matching and leaks to `learning.scoring` through an injected callable so `pairs` does not import `learning`; `build_pair_signals` gains a `split` filter.
|
||||
- `src/openultrasast/benchmark.py`: `ExpectedFinding.family: str | None = None` (additive, stored as an opaque string; the Classifier validates loaded labels against the taxonomy so `pairs` and `benchmark` never import `learning`).
|
||||
- `src/openultrasast/tool_hunter.py`: `run_tool_hunter` gains `system_prompt`, `context_files`, `tools`, `max_chars` and `tags` parameters with today's values as defaults; finding parsing accepts an optional `family` field; the reply loop replays `reasoning_content`.
|
||||
- `src/openultrasast/hunter_tools.py`: four new tools beside `read_file`, `grep_repo`, `find_refs`: `read_definition(symbol)`, `entry_points(path)`, `flows(path, function)`, `obligations(path, function)`, each clamped to the repository and returning identifiers and line spans only; `HUNTER_TOOLS` schemas extended.
|
||||
- `src/openultrasast/provider/openrouter.py`: `OpenRouterChatClient` gains `extra_body` (thinking, response_format, logprobs) and retains the `reasoning_content` field on assistant messages; replaying it across tool turns is the hunter loop's job (`tool_hunter`), not the client's. No behavior change for existing callers.
|
||||
- `src/openultrasast/config.py`: `[models]` gains `chat_base_url`, `chat_api_key_env`, `judge`; `[learning]` section (`families_path`, `configs_dir`, `k_runs = 5`, `round_cost_cap_usd`, `minibatch`, `cutoff_date`); `load_dotenv` unchanged, tests inject endpoints explicitly.
|
||||
- `src/openultrasast/semantic/seed.py`, `semantic/loo.py`, `improve/evolve.py`: teacher selection through `learning.split.teachers`; refusals recorded; `evaluate_loo` teaches from train only.
|
||||
- `src/openultrasast/regress/candidate.py`, `regress/verdict.py`: `run_regression` accepts an optional `oracle: Callable[[SandboxResult], bool]`; `verdict_from_result` uses the oracle when given, exit code otherwise.
|
||||
- `src/openultrasast/cli.py`: `ousast learning classify|score|baseline|round|publish`; `pairs --hunter` uses the family scorer and `--k-runs`; `[models].hunter` resolution through `learning.endpoint`.
|
||||
- `pyproject.toml`: `tree-sitter-typescript>=0.23` in the semantic extra; `src/openultrasast/semantic/extra.py` maps `typescript`/`tsx`.
|
||||
- `benchmarks/pairs/vibe-py/catalog.toml` and `recipes.toml`: the five identical twins gain `known_limit = "identical_twin"`; absence rows gain `family` and `mode = "handler_context"` (re-harvest is a maintainer step).
|
||||
- **Cross-file registration context (amended during task 4.1).** Some frameworks keep the route table out of the handler module: connexion registers by `operationId` in an OpenAPI document, an Express application registers in a router file. A single-file excerpt of such a handler can never contain its registration, so Req 10.2 is unreachable for those rows without a second file. A recipe may therefore name `context = ["<repo-relative path>"]`; `benchmarks/pairs/harvest.py` fetches each document at the same two commits, extracts only the stanza that names the handler (`extract_registration_context`), and writes it under `<name>/context/<side>/<relpath>` with its own provenance header and the same redaction. The catalog carries `[[pair.context]]` rows, `PairCase.context_files` holds `(relpath, vuln, fixed)`, and `_materialize_side` copies them beside the excerpt at their repository-relative paths so the entry-point mapper sees the same layout the project has. `src/openultrasast/mapping.py` gains `_route_manifest_entry_points`, which reads `operationId` and the operation's `security` block out of a YAML or JSON document and attributes the record to the module the operation id points at. Trap pairs whose fixed side is a different, correctly guarded handler carry `fix_function` so the fixed side gets *its* registration rather than the vulnerable side's.
|
||||
- `benchmarks/pairs/mechanisms.toml` — unchanged; families reference mechanism ids.
|
||||
- `.kiro/steering/roadmap.md` — rewritten by `publish` from artifacts.
|
||||
- `benchmarks/pairs/mechanisms.toml`: unchanged; families reference mechanism ids.
|
||||
- `.kiro/steering/roadmap.md`: rewritten by `publish` from artifacts.
|
||||
|
||||
## System Flows
|
||||
|
||||
|
||||
@@ -11,9 +11,9 @@ OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: the class taxonomy and its classifier; the class-aware pair scorer and unscorable-row accounting; split enforcement for every learning path; per-family detector configurations and their routing; class-keyed verifiers and reporting rungs; the round-based evolve loop with attribution; the corpus repairs the loop needs; publishing every number with its denominator and noise floor.
|
||||
- **Out of scope**: finding bugs in JavaScript interpreters, WebGL renderers or browser engines; memory-safety fuzzing or crash oracles for C and C++ (the vfc slice stays a measured, non-gating slice under one coarse family); new hand-written static shape families; changes to the evidence ladder or dispositions; merge gates on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights.
|
||||
- **Adjacent expectations**: the pair corpus (`pair-corpus-honesty`) keeps owning labels, tiers, splits and `known_limit`; `corpus-seeded-mechanisms` keeps its exporter, leave-one-out and lever, which this feature subordinates to the split and the class attribution; `authorization-obligations` keeps its checker, whose corpus tasks 4.3 and 6.1 move here and whose lever task 5 becomes the access-control family's first round; the local detection, map and pair gates stay byte-identical; HarnessX and tree-sitter grammars remain optional extras and every step that needs them degrades to a recorded reason without them.
|
||||
- In scope: the class taxonomy and its classifier; the class-aware pair scorer and unscorable-row accounting; split enforcement for every learning path; per-family detector configurations and their routing; class-keyed verifiers and reporting rungs; the round-based evolve loop with attribution; the corpus repairs the loop needs; publishing every number with its denominator and noise floor.
|
||||
- Out of scope: finding bugs in JavaScript interpreters, WebGL renderers or browser engines; memory-safety fuzzing or crash oracles for C and C++ (the vfc slice stays a measured, non-gating slice under one coarse family); new hand-written static shape families; changes to the evidence ladder or dispositions; merge gates on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights.
|
||||
- Adjacent expectations: the pair corpus (`pair-corpus-honesty`) keeps owning labels, tiers, splits and `known_limit`; `corpus-seeded-mechanisms` keeps its exporter, leave-one-out and lever, which this feature subordinates to the split and the class attribution; `authorization-obligations` keeps its checker, whose corpus tasks 4.3 and 6.1 move here and whose lever task 5 becomes the access-control family's first round; the local detection, map and pair gates stay byte-identical; HarnessX and tree-sitter grammars remain optional extras and every step that needs them degrades to a recorded reason without them.
|
||||
|
||||
## Requirements
|
||||
|
||||
@@ -87,7 +87,7 @@ OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The
|
||||
1. The Learning Harness shall hold one detector configuration per family consisting of a prompt under a stated length cap, a curated tool set, a step budget, a cost budget, an evidence checklist and a set of hard negatives and counterexamples.
|
||||
2. While a round evolves one family, the Learning Harness shall keep every other family's configuration unchanged.
|
||||
3. When a code region is classified, the Learning Harness shall run the detector of every admitted family plus a generalist detector, and shall record on each finding the family, the detector configuration version and the verifier outcome.
|
||||
4. The Learning Harness shall give a detector the whole file of a region and the files its curated tools resolve, not only an excerpt around a hotspot.
|
||||
4. The Learning Harness shall give a detector the whole file of a region and the files its curated tools resolve instead of limiting input to an excerpt around a hotspot.
|
||||
5. The Learning Harness shall expose to a detector only curated tools that read, search and resolve symbols within the scanned repository, and no general shell.
|
||||
6. The Learning Harness shall run detectors, the classifier's model tier and the evolve proposer on a configurable chat endpoint that is independent of the embedding endpoint, with a cheap model as the default and a stronger model reserved for the second judge and for review-queue disagreements.
|
||||
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
## Summary
|
||||
|
||||
- **Feature**: `learning-harness`
|
||||
- **Discovery Scope**: Complex Integration (extension of pairs, tool hunter, provider, regress, improve; adoption of HarnessX meta-harness; corpus repairs)
|
||||
- **Key Findings**:
|
||||
- Feature: `learning-harness`
|
||||
- Discovery Scope: Complex Integration (extension of pairs, tool hunter, provider, regress, improve; adoption of HarnessX meta-harness; corpus repairs)
|
||||
- Key Findings:
|
||||
- The tool hunter already beats the overlay 12/14 vs 5/14 on the class-aware, scorable vibe-py holdout; the published 0/17 was the text-token scorer.
|
||||
- Every learning path ignores the split (`evaluate_profiles`, `evaluate_mechanism_profiles`, `propose_mechanism_edits`, `evaluate_loo`, `build_pair_signals`, `export_mechanisms`); one teacher rule fixes all of them.
|
||||
- The sandbox verdict is exit-code only and snippets are compile-only templates; a canary oracle is an additive callback that respects the existing safety bans.
|
||||
@@ -15,13 +15,13 @@
|
||||
## Research Log
|
||||
|
||||
### Codebase seams (2026-09-06, Explore subagent over `src/openultrasast`)
|
||||
- **Context**: design an extension without re-reading every module.
|
||||
- **Findings**: `run_tool_hunter(root, hotspots, *, client, model, max_steps)` with three curated tools and `suspicion` findings tagged `tool-hunter`; `ChatClient.complete(*, model, messages, tools, timeout_seconds)`; `OpenRouterChatClient.complete_chat` always sends `temperature: 0` and drops `reasoning_content`; `_hunter_matches_expected` is text-token based and every fixed-side finding leaks under `fix_policy = silent`; `run_regression(..., sandbox, sandbox_limits, images)` with `verdict_from_result` on exit code; `check_snippet_safety` bans sockets, curl and workspace writes; `MechanismStore` append-only JSONL with tombstones; `improve.run_round` journal has no hypothesis, attribution or cost fields and its regression check is per provenance profile; `load_dotenv` no-ops under pytest; `pyproject` semantic extra lacks TypeScript; `pair_gate` scores only vendored `local` pairs.
|
||||
- **Implications**: the design adds parameters and additive fields instead of replacing modules; the family scorer is injected into the hunter path so `pairs` never imports `learning`; the oracle is an optional callback on `run_regression`; endpoint resolution takes an explicit override for tests.
|
||||
- Context: design an extension without re-reading every module.
|
||||
- Findings: `run_tool_hunter(root, hotspots, *, client, model, max_steps)` with three curated tools and `suspicion` findings tagged `tool-hunter`; `ChatClient.complete(*, model, messages, tools, timeout_seconds)`; `OpenRouterChatClient.complete_chat` always sends `temperature: 0` and drops `reasoning_content`; `_hunter_matches_expected` is text-token based and every fixed-side finding leaks under `fix_policy = silent`; `run_regression(..., sandbox, sandbox_limits, images)` with `verdict_from_result` on exit code; `check_snippet_safety` bans sockets, curl and workspace writes; `MechanismStore` append-only JSONL with tombstones; `improve.run_round` journal has no hypothesis, attribution or cost fields and its regression check is per provenance profile; `load_dotenv` no-ops under pytest; `pyproject` semantic extra lacks TypeScript; `pair_gate` scores only vendored `local` pairs.
|
||||
- Implications: the design adds parameters and additive fields instead of replacing modules; the family scorer is injected into the hunter path so `pairs` never imports `learning`; the oracle is an optional callback on `run_regression`; endpoint resolution takes an explicit override for tests.
|
||||
|
||||
### DeepSeek platform API (2026-09-06, api-docs.deepseek.com)
|
||||
- **Findings**: base `https://api.deepseek.com` (no `/v1` in current docs), Anthropic-shape endpoint at `/anthropic`; models `deepseek-v4-flash`, `deepseek-v4-pro`, `deepseek-v4-flash-vision-exp`; 1M context, 384K output; thinking on by default (`thinking: {"type": "disabled"}` to turn off; `temperature` ignored while thinking); tools with `tool_choice`, `reasoning_content` must be replayed on tool turns; `response_format json_object` only, occasional empty content; `logprobs`/`top_logprobs` documented, unverified in thinking mode; no `seed`, `n`, `logit_bias`; flash 0.44/1.32 USD per M in/out peak, cache hit 0.014, pro 1.32/3.96, off-peak half; concurrency 2500 flash / 500 pro; no embeddings endpoint.
|
||||
- **Implications**: detectors and classifier run with thinking disabled and temperature 0; determinism is best-effort and the K-run noise floor absorbs it; the classifier's model tier uses `logprobs` averaged and falls to `unknown`; embeddings stay on OpenRouter; cost accounting reads DeepSeek's cache-hit usage fields.
|
||||
- Findings: base `https://api.deepseek.com` (no `/v1` in current docs), Anthropic-shape endpoint at `/anthropic`; models `deepseek-v4-flash`, `deepseek-v4-pro`, `deepseek-v4-flash-vision-exp`; 1M context, 384K output; thinking on by default (`thinking: {"type": "disabled"}` to turn off; `temperature` ignored while thinking); tools with `tool_choice`, `reasoning_content` must be replayed on tool turns; `response_format json_object` only, occasional empty content; `logprobs`/`top_logprobs` documented, unverified in thinking mode; no `seed`, `n`, `logit_bias`; flash 0.44/1.32 USD per M in/out peak, cache hit 0.014, pro 1.32/3.96, off-peak half; concurrency 2500 flash / 500 pro; no embeddings endpoint.
|
||||
- Implications: detectors and classifier run with thinking disabled and temperature 0; determinism is best-effort and the K-run noise floor absorbs it; the classifier's model tier uses `logprobs` averaged and falls to `unknown`; embeddings stay on OpenRouter; cost accounting reads DeepSeek's cache-hit usage fields.
|
||||
|
||||
### Measurement that triggered the spec (2026-09-06)
|
||||
- See `brief.md` Problem section and `benchmarks/measurements/2026-09-06-*`.
|
||||
@@ -38,37 +38,37 @@
|
||||
## Design Decisions
|
||||
|
||||
### Decision: families are the only routing and scoring key
|
||||
- **Context**: 1.4, 3.6; text tokens and CWE ids were the proxy.
|
||||
- **Alternatives**: fine CWE routing; mechanism ids as key.
|
||||
- **Selected**: ten coarse families with CWE and mechanism as attributes; hierarchical relation for partial credit.
|
||||
- **Rationale**: fine CWE from code is unsolved (≤14.7% top-1) and coarse families are what worked in AIxCC and MoEVD.
|
||||
- **Trade-offs**: less granular reporting; compensated by attributes on findings.
|
||||
- Context: 1.4, 3.6; text tokens and CWE ids were the proxy.
|
||||
- Alternatives: fine CWE routing; mechanism ids as key.
|
||||
- Selected: ten coarse families with CWE and mechanism as attributes; hierarchical relation for partial credit.
|
||||
- Rationale: fine CWE from code is unsolved (≤14.7% top-1) and coarse families are what worked in AIxCC and MoEVD.
|
||||
- Trade-offs: less granular reporting; compensated by attributes on findings.
|
||||
|
||||
### Decision: the scorer is injected into the hunter path, not imported by `pairs`
|
||||
- **Context**: dependency direction; `pairs` is owned by `pair-corpus-honesty`.
|
||||
- **Selected**: `_evaluate_hunter_pair` accepts a matcher/leak callable; the CLI wires `learning.scoring`.
|
||||
- **Trade-offs**: one more parameter; keeps `pairs` free of `learning` and the overlay path unchanged.
|
||||
- Context: dependency direction; `pairs` is owned by `pair-corpus-honesty`.
|
||||
- Selected: `_evaluate_hunter_pair` accepts a matcher/leak callable; the CLI wires `learning.scoring`.
|
||||
- Trade-offs: one more parameter; keeps `pairs` free of `learning` and the overlay path unchanged.
|
||||
|
||||
### Decision: canary oracle as an optional callback on `run_regression`
|
||||
- **Context**: 7.2; exit code cannot express "the vulnerability fired".
|
||||
- **Alternatives**: a parallel verifier runner; parsing stderr.
|
||||
- **Selected**: `oracle: Callable[[SandboxResult], bool] | None`; token in stdout.
|
||||
- **Trade-offs**: minimal change to regress; templates per family live in `learning/verifiers/`.
|
||||
- Context: 7.2; exit code cannot express "the vulnerability fired".
|
||||
- Alternatives: a parallel verifier runner; parsing stderr.
|
||||
- Selected: `oracle: Callable[[SandboxResult], bool] | None`; token in stdout.
|
||||
- Trade-offs: minimal change to regress; templates per family live in `learning/verifiers/`.
|
||||
|
||||
### Decision: HarnessX proposer behind the seam, scripted proposer first
|
||||
- **Context**: 9.x with core `dependencies = []`.
|
||||
- **Selected**: `Proposer` protocol; `HarnessXProposer` composes `MetaAgent.evolve` with `allowed_write_roots` = the family directory; `ScriptedProposer` for tests and phase 4 start.
|
||||
- **Trade-offs**: two proposers; the loop is testable offline and the extra stays optional.
|
||||
- Context: 9.x with core `dependencies = []`.
|
||||
- Selected: `Proposer` protocol; `HarnessXProposer` composes `MetaAgent.evolve` with `allowed_write_roots` = the family directory; `ScriptedProposer` for tests and phase 4 start.
|
||||
- Trade-offs: two proposers; the loop is testable offline and the extra stays optional.
|
||||
|
||||
### Decision: noise floor from K = 5 baseline runs is the regression budget
|
||||
- **Context**: 8.2, 9.5; no public numeric negative-flip budget exists.
|
||||
- **Selected**: per-family negative-flip rate between seed runs; `budget_flips = ceil(rate × scorable)`.
|
||||
- **Trade-offs**: a noisy family gets a loose budget; reported beside the number so it is visible.
|
||||
- Context: 8.2, 9.5; no public numeric negative-flip budget exists.
|
||||
- Selected: per-family negative-flip rate between seed runs; `budget_flips = ceil(rate × scorable)`.
|
||||
- Trade-offs: a noisy family gets a loose budget; reported beside the number so it is visible.
|
||||
|
||||
### Decision: DeepSeek adapter wraps the existing OpenRouter client
|
||||
- **Context**: 6.6; both clients share one base URL today.
|
||||
- **Selected**: `ChatEndpoint` resolution order DeepSeek → OpenRouter → scripted; adapter adds `extra_body` and `reasoning_content` replay; embeddings untouched.
|
||||
- **Trade-offs**: one adapter class; no new dependency.
|
||||
- Context: 6.6; both clients share one base URL today.
|
||||
- Selected: `ChatEndpoint` resolution order DeepSeek → OpenRouter → scripted; adapter adds `extra_body` and `reasoning_content` replay; embeddings untouched.
|
||||
- Trade-offs: one adapter class; no new dependency.
|
||||
|
||||
## Risks & Mitigations
|
||||
- Router error routes a region to the wrong family → multi-label routing plus the generalist; router measured separately against random and oracle.
|
||||
|
||||
@@ -8,8 +8,8 @@ skills. Related: `proof-arm-opencode` (the proof model), `pre-push-safety-net`
|
||||
## Motivation
|
||||
|
||||
The pre-push hook is the gate. Developers and their coding agents want the same
|
||||
security feedback **earlier**, while the code is still being written, not only at
|
||||
push. And in an agent-driven workflow, a mandatory security check is how a human
|
||||
security feedback **earlier**, while the code is still being written, before
|
||||
push. In an agent-driven workflow, a mandatory security check is how a human
|
||||
oversees agents: the agent's output has to clear it before it ships.
|
||||
|
||||
Proposal: expose the safety net through an MCP agent and a small set of skills so
|
||||
@@ -19,20 +19,20 @@ proposes a push.
|
||||
|
||||
## Shape
|
||||
|
||||
- **Transport:** extend the existing narrow MCP server (`ousast mcp`, stdio) that
|
||||
- Transport: extend the existing narrow MCP server (`ousast mcp`, stdio) that
|
||||
OpenCode and other agents already speak. No new network surface; same
|
||||
no-secret, bounded contract.
|
||||
- **What the agent offers:**
|
||||
- What the agent offers:
|
||||
1. Scan the current diff with the static arm (fast, no model, no cluster) and
|
||||
return findings scoped to the change, with location and a concrete repair
|
||||
direction, not a confidence label.
|
||||
2. Triage a finding: explain it in plain terms and whether it is reachable.
|
||||
3. Prove on demand: hand the top candidate to the proof arm when a runnable
|
||||
harness exists, and return the demonstrated proof-of-concept.
|
||||
- **Skills:** the existing `openultrasast-scan` and `openultrasast-triage`
|
||||
- Skills: the existing `openultrasast-scan` and `openultrasast-triage`
|
||||
product skills, plus shift-left additions (scan-the-diff, explain-a-finding,
|
||||
suggest-a-fix-direction) that a coding agent invokes mid-task.
|
||||
- **Oversight mode:** a coding agent can be configured to call the scan as a
|
||||
- Oversight mode: a coding agent can be configured to call the scan as a
|
||||
required step before it finalizes a change, so security review is automatic
|
||||
rather than remembered.
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
> deterministically from code**; execution is a later, conditional tier. The correction is integrated below.
|
||||
|
||||
**Status:** course-setting. Supersedes `learning-harness` and `constrained-detector`. Not a requirements
|
||||
document — a decision about what to build, what to delete, and what to adopt, written after the
|
||||
document, a decision about what to build, what to delete, and what to adopt, written after the
|
||||
learning-harness line produced numbers its own measurements say to discard.
|
||||
|
||||
## 1. The premise we got wrong
|
||||
@@ -17,19 +17,19 @@ averaged (K runs), floored (per-family noise budget), gated (sign test), and slo
|
||||
Every mechanism in `learning-harness` follows from that assumption. This session measured all of it and it
|
||||
does not hold:
|
||||
|
||||
- Five evolve rounds, $2.17, **zero decisions distinguishable from chance** — because the acceptance rule
|
||||
- Five evolve rounds, $2.17, **zero decisions distinguishable from chance**, because the acceptance rule
|
||||
compares two ensemble votes over a detector that never runs the same investigation twice.
|
||||
- 120 runs of one detector produced **120 distinct trajectories**. The variance is the loop's freedom to
|
||||
search, not the model's uncertainty about an answer.
|
||||
- The noise floor a regression budget is derived from measures *how often an unconstrained search happens
|
||||
to reach the sink* — so the worse the search wanders, the more collateral damage a round is licensed to do.
|
||||
to reach the sink*, so the worse the search wanders, the more collateral damage a round is licensed to do.
|
||||
|
||||
The assumption is also empirically false as a claim about LLMs, and the evidence is now overwhelming:
|
||||
|
||||
- **Google Big Sleep** (Nov 2024): the first LLM-discovered zero-day in widely-used real-world software — a
|
||||
stack buffer underflow in SQLite — found by *variant analysis*, a sandboxed test case, and an
|
||||
- Google Big Sleep (Nov 2024): the first LLM-discovered zero-day in widely-used real-world software, a
|
||||
stack buffer underflow in SQLite, found by *variant analysis*, a sandboxed test case, and an
|
||||
auto-generated root-cause report. Not pattern-matching; reproduction.
|
||||
- **DARPA AIxCC finals** (DEF CON 33, Aug 2025): seven autonomous systems processed 54M lines of code,
|
||||
- DARPA AIxCC finals (DEF CON 33, Aug 2025): seven autonomous systems processed 54M lines of code,
|
||||
patched 43 of 54 synthetic vulnerabilities and **uncovered 18 previously-unknown real 0-days**. Buttercup
|
||||
(Trail of Bits, 2nd place) found **28 vulnerabilities across 20 CWEs at 90% accuracy, $181 per point,
|
||||
using non-reasoning LLMs only**, ~$152 per task, 45 minutes each.
|
||||
@@ -47,11 +47,11 @@ Lazarus-AI, MIT, an open reproduction of Anthropic's Glasswing) share one shape:
|
||||
> **The LLM is a judgement layer on top of a deterministic substrate that arbitrates its claims. Truth comes
|
||||
> from that substrate, not from a score against a labeled corpus.**
|
||||
|
||||
In the memory-safety systems above the substrate is *execution* — a crash is the arbiter. But execution is
|
||||
In the memory-safety systems above the substrate is *execution*. A crash is the arbiter. But execution is
|
||||
only the arbiter you reach for when the target builds in isolation and the bug crashes, which is the C world,
|
||||
not ours. The general form of the substrate is a **program model built from the code**: a deterministic
|
||||
abstraction rich enough to *check* a claim — does the source actually reach the sink through resolved
|
||||
dataflow; is there a guard on the path or isn't there; is the entry point attacker-reachable. Execution is one
|
||||
abstraction rich enough to *check* a claim (does the source actually reach the sink through resolved
|
||||
dataflow; is there a guard on the path or isn't there; is the entry point attacker-reachable). Execution is one
|
||||
way to arbitrate; a model is the way that works for a SAST project and for bugs that never crash.
|
||||
|
||||
Clearwing's evidence ladder is the whole thesis in one type:
|
||||
@@ -81,24 +81,24 @@ real. **We built K-runs and noise floors because we had no ground truth. They ge
|
||||
This is not a total loss, and the course is not "throw everything away." Two results here are real,
|
||||
measured, and *absent from Clearwing*:
|
||||
|
||||
1. **IR-driven candidate enumeration with a published ceiling.** We measured that the pattern ruleset caps
|
||||
1. IR-driven candidate enumeration with a published ceiling. We measured that the pattern ruleset caps
|
||||
recall at 12.1% while call-sites-and-bindings from the semantic IR reach **96.6%** across the web slices,
|
||||
a median of ten sites per function. Clearwing's candidate feeder is C-first (taint tables for
|
||||
`memcpy`-shaped bugs); its `web_framework` specialist is a *single unconstrained prompt* with no
|
||||
equivalent candidate enumerator for logic/web classes.
|
||||
|
||||
2. **Absence-bug detection.** Our `obligations` work — an access-control bug is the *absence* of a guard
|
||||
around an operation its siblings protect — has no analogue in Clearwing, whose ladder is built around a
|
||||
2. Absence-bug detection. Our `obligations` work (an access-control bug is the *absence* of a guard
|
||||
around an operation its siblings protect) has no analogue in Clearwing, whose ladder is built around a
|
||||
*crash*. An IDOR or a missing authorization check never crashes a sandbox. Clearwing's execution
|
||||
ground-truth model does not reach these classes at all, and they are exactly the "dangerous but
|
||||
low-hanging" web bugs this project was chartered to catch.
|
||||
|
||||
So the gap Clearwing leaves open is **the web/logic taxonomy without a crash oracle** — which is where our
|
||||
So the gap Clearwing leaves open is **the web/logic taxonomy without a crash oracle**, which is where our
|
||||
IR candidate enumerator, our family taxonomy, and our obligation checker are genuinely ahead.
|
||||
|
||||
## 4b. The model layer, and the evidence ladder it produces
|
||||
|
||||
The arbiter is a program model constructed deterministically from the code — the thing an LLM claim is
|
||||
The arbiter is a program model constructed deterministically from the code, the thing an LLM claim is
|
||||
*checked against*, so that a finding's truth does not rest on the LLM that proposed it. We already hold the
|
||||
first rungs of it; they were buried under a noise layer instead of being made the arbiter.
|
||||
|
||||
@@ -113,16 +113,16 @@ The evidence ladder is the model layer made legible. Each rung is a stronger, st
|
||||
|
||||
Two properties are the whole point:
|
||||
|
||||
- **`model_corroborated` and above are deterministic.** Run the checker twice over the same model and it
|
||||
returns the same verdict. That removes the run-to-run noise the same way a crash would — no K-runs, no
|
||||
floor, no sign test — *without needing a runnable target*. The noise this session measured was the LLM
|
||||
- `model_corroborated` and above are deterministic. Run the checker twice over the same model and it
|
||||
returns the same verdict. That removes the run-to-run noise the same way a crash would (no K-runs, no
|
||||
floor, no sign test), *without needing a runnable target*. The noise this session measured was the LLM
|
||||
free-searching; a model check does not search.
|
||||
- **The LLM's job shrinks to the residual the model cannot decide** — is this string genuinely
|
||||
attacker-controlled, is this guard semantically sufficient — and even that answer is checked against the
|
||||
- The LLM's job shrinks to the residual the model cannot decide (is this string genuinely
|
||||
attacker-controlled, is this guard semantically sufficient), and even that answer is checked against the
|
||||
model rather than averaged over runs. The LLM stops being the oracle and becomes a proposer the model
|
||||
audits.
|
||||
|
||||
**Measured, 2026-09-08 — the entailment ceiling** (`benchmarks/measurements/2026-09-08-model-entailment-ceiling.json`,
|
||||
**Measured, 2026-09-08, the entailment ceiling** (`benchmarks/measurements/2026-09-08-model-entailment-ceiling.json`,
|
||||
offline, no LLM):
|
||||
|
||||
| rung | web/logic taxonomy | for comparison |
|
||||
@@ -134,17 +134,17 @@ offline, no LLM):
|
||||
|
||||
The gap between 96.6% (a site exists) and 2.2% (the model *decides* the vulnerability) is the gap between
|
||||
enumerating and arbitrating, and it is the whole finding. The model cannot be a *pure* arbiter at v1. But it
|
||||
is not a wall — the measurement diagnoses depth, not impossibility: one-hop source resolution (real taint sits
|
||||
is not a wall. The measurement diagnoses depth, not impossibility: one-hop source resolution (real taint sits
|
||||
1–3 binds upstream: `q = "…" + id; execute(q)`), a closed literal sink table (misses project wrappers and the
|
||||
path/xss/prototype sinks), and the function-naming gap that eats 20/50 injection pairs as `no_function`.
|
||||
Deepening the taint to reuse the candidate enumerator's depth-4 chain already moved injection 24% → 48%.
|
||||
|
||||
So the architecture the numbers support is a **layered arbiter**, not a pure one:
|
||||
|
||||
- where the model **entails** — report `model_entailed`, deterministic, the LLM not needed;
|
||||
- where the model **corroborates a taint path** (~35% and rising) — the LLM judges, but its claim is *checked*
|
||||
- where the model **entails**, report `model_entailed`, deterministic, the LLM not needed;
|
||||
- where the model **corroborates a taint path** (~35% and rising), the LLM judges, but its claim is *checked*
|
||||
against the resolved path, so run-to-run noise is removed on this band;
|
||||
- where the model can only **enumerate** (the rest, up to 96.6%) — honest `suspicion`, the LLM proposes with
|
||||
- where the model can only **enumerate** (the rest, up to 96.6%), honest `suspicion`, the LLM proposes with
|
||||
no deterministic backstop, and this is exactly the band where the deferred execution / differential tier
|
||||
earns its place.
|
||||
|
||||
@@ -152,32 +152,32 @@ Pure `model_entailed`-as-arbiter is the north star. The v1 is the layered arbite
|
||||
named by the measurement: fix function naming, build per-family sink/sanitizer tables, and lift the flow
|
||||
model onto the multi-hop binding chain we already have.
|
||||
|
||||
This is the classical SAST arbiter — dataflow / abstract interpretation, the CodeQL and Infer lineage —
|
||||
This is the classical SAST arbiter (dataflow / abstract interpretation, the CodeQL and Infer lineage),
|
||||
with the LLM supplying the semantic judgement the model is weakest at (aliasing intent, whether a value is
|
||||
really untrusted, whether a guard really covers the case). It does not need soundness. A partial model that
|
||||
can confirm *some* claims and honestly returns "could not confirm — stays suspicion" for the rest is already
|
||||
can confirm *some* claims and honestly returns "could not confirm, stays suspicion" for the rest is already
|
||||
the independent arbiter we never had.
|
||||
|
||||
## 4c. IL or AST, build or adopt — the substrate decision
|
||||
## 4c. IL or AST, build or adopt, the substrate decision
|
||||
|
||||
**IL, and adopt one.** Both halves are forced by the entailment measurement.
|
||||
|
||||
*Why an IL, not the AST we have.* Our `semantic/ir.py` is not an IL — it is a lossy AST projection: per-function
|
||||
*Why an IL, not the AST we have.* Our `semantic/ir.py` is a lossy AST projection rather than an IL: per-function
|
||||
flat lists of calls and binds, no control-flow graph, no basic blocks, no data-dependence edges. That is
|
||||
*why* entailment sits at 2.2%. Real abstract interpretation is a fixpoint over a **CFG**: guard *dominance*
|
||||
(is the check on every path to the operation) is a dominator-tree question, and path-sensitive taint (a
|
||||
sanitizer on the taint path specifically) needs the branch structure. A flat bind list cannot express either;
|
||||
it can only over-approximate taint flow-insensitively by name, which is the one-hop ceiling we measured. The
|
||||
model layer wants the classical IL that merges AST + CFG + PDG + call graph into one graph — a **Code Property
|
||||
Graph (CPG)** — over which taint is a reachability query (source → sink minus sanitizer) and absence is a
|
||||
model layer wants the classical IL that merges AST + CFG + PDG + call graph into one graph (a **Code Property
|
||||
Graph (CPG)**), over which taint is a reachability query (source → sink minus sanitizer) and absence is a
|
||||
dominance query.
|
||||
|
||||
*Why adopt, not build.* Building a multi-language CFG/PDG/taint engine on top of tree-sitter is precisely
|
||||
reinventing CodeQL, Semgrep and Joern — a multi-year effort, and the maximal case of the failure mode this
|
||||
reinventing CodeQL, Semgrep and Joern, a multi-year effort, and the maximal case of the failure mode this
|
||||
session logged eighteen times. **Joern** is the adoptable answer: Apache-2.0, a CPG engine for exactly our
|
||||
corpus (C/C++/Java/JavaScript/Python/Kotlin/binary), a built-in taint-propagation engine, embeddable, and
|
||||
there is already a published CPG-plus-LLM bridge (Codebadger, an MCP server over Joern) and an arXiv line on
|
||||
CPGs as the substrate for LLM program analysis. Joern *is* "a model built up from code" as a graph — the exact
|
||||
CPGs as the substrate for LLM program analysis. Joern *is* "a model built up from code" as a graph, the exact
|
||||
thing the maintainer described. The alternatives: CodeQL is more powerful but least adoptable (restrictive
|
||||
license, query-only, GitHub-owned); Semgrep's taint engine is lighter and we already have integration
|
||||
precedent through Clearwing's sidecar, but interprocedural taint is behind the paid Pro engine.
|
||||
@@ -192,14 +192,14 @@ source ──▶ Joern: compile to CPG (one IL: AST + CFG + PDG + call graph, la
|
||||
```
|
||||
|
||||
The CPG is language-agnostic once its frontend exists, so the taint and dominance logic is written *once*, not
|
||||
per language — which is the whole reason our per-language tree-sitter approach kept hitting naming and grammar
|
||||
per language, which is the whole reason our per-language tree-sitter approach kept hitting naming and grammar
|
||||
gaps.
|
||||
|
||||
*What stays ours, and is the contribution.* Joern is a workbench, not an opinionated web/logic detector with
|
||||
an LLM judge. What we own and port onto it: the closed **family taxonomy**; the **source / sink / sanitizer /
|
||||
guard models** for the web/logic CWEs — our obligation facts and closed guard vocabulary become CPG queries
|
||||
and taint specifications; and the **LLM-adjudication-checked-against-the-CPG** orchestration. That security
|
||||
modelling — expressed as queries over an adopted CPG — is where our work is genuinely ahead of both Joern
|
||||
guard models** for the web/logic CWEs (our obligation facts and closed guard vocabulary become CPG queries
|
||||
and taint specifications); and the **LLM-adjudication-checked-against-the-CPG** orchestration. That security
|
||||
modelling, expressed as queries over an adopted CPG, is where our work is ahead of both Joern
|
||||
(no opinion on these classes) and Clearwing (crash-oracle, no arbiter for bugs that do not crash).
|
||||
|
||||
*The one real cost.* Joern is JVM/Scala; embedding it in a Python tool means its server mode or the MCP bridge,
|
||||
@@ -212,38 +212,38 @@ noise architecture; keep execution as a later, conditional tier.**
|
||||
|
||||
Concretely, superseding both prior specs:
|
||||
|
||||
1. **Delete, do not fix.** The K-run machinery, the per-family noise floor, the regression budget, the sign
|
||||
1. Delete, do not fix. The K-run machinery, the per-family noise floor, the regression budget, the sign
|
||||
test, the evolve loop, the acceptance rule, the mechanism-lever optimizer. All of it existed to average an
|
||||
LLM oracle we could not otherwise trust. A deterministic model arbiter removes the need for every piece.
|
||||
`learning-harness` Req 3.5, 8.2, 9.5 and the evolve tasks are retired, not amended.
|
||||
|
||||
2. **Adopt a CPG engine (Joern) as the arbiter substrate; port our security models onto it.** We do not lift
|
||||
2. Adopt a CPG engine (Joern) as the arbiter substrate; port our security models onto it. We do not lift
|
||||
our flat IR to a CFG ourselves (§4c). The taxonomy, the source/sink/sanitizer/guard models, and the
|
||||
obligation/absence logic move from prompt context to **CPG taint specifications and dominance queries** —
|
||||
obligation/absence logic move from prompt context to **CPG taint specifications and dominance queries**,
|
||||
deterministic `model_corroborated` and `model_entailed` verdicts over an engine that already has the CFG
|
||||
and PDG our IR lacks. Our tree-sitter IR retains one job: cheap candidate enumeration for the `suspicion`
|
||||
band where the CPG has no verdict.
|
||||
|
||||
3. **The LLM proposes; the model disposes.** One bounded, typed question per candidate — the constrained
|
||||
detector we already started — but the answer is not scored against a corpus, it is *checked against the
|
||||
model*. A claim the model corroborates advances; a claim it contradicts is dropped with the contradiction
|
||||
3. The LLM proposes; the model disposes. One bounded, typed question per candidate (the constrained
|
||||
detector we already started), with the answer *checked against the
|
||||
model* rather than scored against a corpus. A claim the model corroborates advances; a claim it contradicts is dropped with the contradiction
|
||||
recorded; a claim it can neither confirm nor deny stays `suspicion` and is reported as such. No K-runs, no
|
||||
evolve loop.
|
||||
|
||||
4. **Corpus is the model's calibration set, not the LLM's training signal.** Keep the vulnerable-parent /
|
||||
fix-commit pairs — but their role changes. The pair is ground truth for *the model's precision*: the fix
|
||||
4. Corpus is the model's calibration set, not the LLM's training signal. Keep the vulnerable-parent /
|
||||
fix-commit pairs, but their role changes. The pair is ground truth for *the model's precision*: the fix
|
||||
is what the model should be able to see (a guard added, a sanitizer inserted, a bound checked), and a pair
|
||||
the model cannot distinguish is a gap in the model, named and measured, not a detector miss to average.
|
||||
Harvest more pairs at repository scale from CVE history, as Clearwing does; truth is the fix diff read by
|
||||
the model, not a label we hand-assign.
|
||||
|
||||
5. **Execution is a deferred, conditional tier.** For the classes a static model genuinely cannot arbitrate —
|
||||
memory-safety-in-C, where reachability and exploitability turn on runtime layout — a sandbox crash oracle
|
||||
5. Execution is a deferred, conditional tier. For the classes a static model genuinely cannot arbitrate
|
||||
(memory-safety-in-C, where reachability and exploitability turn on runtime layout), a sandbox crash oracle
|
||||
is the right arbiter, and Clearwing already is that system. Adopt it *there*, later, only when a class
|
||||
needs it. It is the `execution_confirmed` rung, not the foundation.
|
||||
|
||||
6. **Reporting keeps the honesty disciplines that stand:** per-slice numbers with the real-world slice
|
||||
deciding, the overfitting gap published, and the evidence rung on every finding — now meaning a model
|
||||
6. Reporting keeps the honesty disciplines that stand: per-slice numbers with the real-world slice
|
||||
deciding, the overfitting gap published, and the evidence rung on every finding, now meaning a model
|
||||
verdict, not a sandbox result.
|
||||
|
||||
## 6. Build vs. adopt, revised
|
||||
@@ -252,14 +252,14 @@ The earlier draft of this brief over-rotated to "adopt Clearwing wholesale," bec
|
||||
foregrounds execution. The maintainer's correction re-centres it, and the honest split is cleaner than either
|
||||
extreme:
|
||||
|
||||
- **The model layer is adopt-plus-contribute, not build.** The arbiter *engine* is Joern's CPG (Apache-2.0,
|
||||
our languages, taint + dominance built in); the *contribution* is our security modelling on top of it — the
|
||||
- The model layer is adopt-plus-contribute, not build. The arbiter *engine* is Joern's CPG (Apache-2.0,
|
||||
our languages, taint + dominance built in); the *contribution* is our security modelling on top of it, the
|
||||
taxonomy and the source/sink/sanitizer/guard specs for the web/logic classes, plus the LLM adjudication
|
||||
checked against the CPG. The entailment measurement (2.2% on our flat IR) is what rules out building the
|
||||
engine ourselves: that number is the distance between our AST projection and a real CPG, and closing it by
|
||||
hand is reimplementing Joern.
|
||||
- **The execution tier is Clearwing's to adopt**, later, for the memory-safety classes where it is strong and
|
||||
we are weak — via the fork the maintainer already maintains. We do not reimplement its sandbox lifecycle,
|
||||
- The execution tier is Clearwing's to adopt, later, for the memory-safety classes where it is strong and
|
||||
we are weak, via the fork the maintainer already maintains. We do not reimplement its sandbox lifecycle,
|
||||
container pooling, sanitizer images or PoC stability; those are exactly the parts this session proved we get
|
||||
wrong.
|
||||
|
||||
@@ -268,24 +268,24 @@ arbiter we cannot, when a class demands it.
|
||||
|
||||
## Open questions for the requirements phase
|
||||
|
||||
- **How far up the ladder can a partial model reach without soundness?** `model_corroborated` needs only a
|
||||
- How far up the ladder can a partial model reach without soundness? `model_corroborated` needs only a
|
||||
consistency check; `model_entailed` needs a real interprocedural dataflow/guard-dominance result. What
|
||||
fraction of our corpus can the model *entail* today, and where does it honestly fall back to `suspicion`?
|
||||
This is the first measurement to run — the model analogue of the 96.6% candidate ceiling — and it can end
|
||||
This is the first measurement to run (the model analogue of the 96.6% candidate ceiling), and it can end
|
||||
or reshape the design before any LLM cost, exactly as the ceiling did.
|
||||
- **What is the model's arbiter for an absence bug?** A missing guard has no dataflow path to confirm. The
|
||||
- What is the model's arbiter for an absence bug? A missing guard has no dataflow path to confirm. The
|
||||
obligation/sibling-dominance model is the candidate arbiter; does it distinguish the vulnerable-parent from
|
||||
the fixed twin across the corpus on its own, deterministically?
|
||||
- **Where does the LLM's residual judgement resist checking?** Some claims — "this value is attacker-
|
||||
controlled by convention", "this guard is semantically insufficient" — the model cannot audit. How large is
|
||||
- Where does the LLM's residual judgement resist checking? Some claims ("this value is attacker-
|
||||
controlled by convention", "this guard is semantically insufficient") the model cannot audit. How large is
|
||||
that residual, and is `suspicion` an acceptable home for it or does it need the execution tier sooner?
|
||||
- **Which classes cross into execution, and when.** Memory-safety-in-C is the clear case for the deferred
|
||||
- Which classes cross into execution, and when. Memory-safety-in-C is the clear case for the deferred
|
||||
tier. Is anything in the web/logic taxonomy genuinely un-arbitrable by a model, forcing the differential
|
||||
request oracle earlier than "later, if needed"?
|
||||
- **What of the existing corpus survives** as the model's calibration set, given the twin/straddle/tier
|
||||
- What of the existing corpus survives as the model's calibration set, given the twin/straddle/tier
|
||||
defects the learning-harness reviews surfaced?
|
||||
|
||||
Sources: [Big Sleep / SQLite zero-day](https://thehackernews.com/2024/11/googles-ai-tool-big-sleep-finds-zero.html) ·
|
||||
[AIxCC finals — Buttercup, 28 vulns / 20 CWEs / non-reasoning LLMs](https://blog.trailofbits.com/2025/08/09/trail-of-bits-buttercup-wins-2nd-place-in-aixcc-challenge/) ·
|
||||
[AIxCC results — 18 real 0-days, 43/54 patched](https://semiengineering.com/aixcc-2025-what-it-means-for-device-security/) ·
|
||||
[AIxCC finals, Buttercup, 28 vulns / 20 CWEs / non-reasoning LLMs](https://blog.trailofbits.com/2025/08/09/trail-of-bits-buttercup-wins-2nd-place-in-aixcc-challenge/) ·
|
||||
[AIxCC results, 18 real 0-days, 43/54 patched](https://semiengineering.com/aixcc-2025-what-it-means-for-device-security/) ·
|
||||
[Clearwing](https://github.com/Lazarus-AI/clearwing) · local: `/home/mc/Source/clearwing`, fork `norandom/deepwing`.
|
||||
|
||||
@@ -4,14 +4,14 @@
|
||||
|
||||
The detector stops being a noisy oracle and becomes a proposer that a deterministic program model audits. The
|
||||
model is a **Code Property Graph** (Joern, Apache-2.0): one graph merging AST, CFG, PDG and call graph, over
|
||||
which taint reachability and guard dominance produce verdicts on the evidence ladder — `suspicion`,
|
||||
which taint reachability and guard dominance produce verdicts on the evidence ladder, `suspicion`,
|
||||
`model_corroborated`, `model_entailed`, `execution_confirmed`. The LLM is asked one bounded question per
|
||||
candidate; its answer is checked against the CPG, not averaged over runs.
|
||||
|
||||
Two disciplines shape every decision below. **Subtraction first:** the noise architecture and the outgrown
|
||||
test surface are removed and audited before the CPG arbiter is adopted, because the session that produced this
|
||||
spec logged eighteen defects building on an overgrown surface. **Adopt, don't build:** the CFG/PDG our flat IR
|
||||
lacks comes from Joern, not from a hand-built engine — the 2.2% entailment ceiling is the measured distance
|
||||
lacks comes from Joern, not from a hand-built engine. The 2.2% entailment ceiling is the measured distance
|
||||
between our AST projection and a real CPG, and closing it by hand is reimplementing Joern.
|
||||
|
||||
### Goals
|
||||
@@ -19,12 +19,12 @@ between our AST projection and a real CPG, and closing it by hand is reimplement
|
||||
- Remove the noise architecture and reduce the test suite with it, gate-protected and reversible.
|
||||
- A deterministic model verdict on the band the model covers; honest `suspicion` on the rest.
|
||||
- The CPG engine, the LLM endpoint and any execution tier are optional; the core stays zero-dependency.
|
||||
- The security modelling — taxonomy, source/sink/sanitizer/guard specs — is ours, expressed as CPG queries.
|
||||
- The security modelling (taxonomy, source/sink/sanitizer/guard specs) is ours, expressed as CPG queries.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- Building a control-flow or dataflow engine (adopted, Req 4). Prompt optimisation, evolve loops, acceptance
|
||||
rules (deleted, Req 8.5). The execution tier itself (deferred to Clearwing, Req 10). Soundness — the
|
||||
rules (deleted, Req 8.5). The execution tier itself (deferred to Clearwing, Req 10). Soundness, the
|
||||
north-star `model_entailed`-as-pure-arbiter is not v1.
|
||||
|
||||
## Boundary Commitments
|
||||
@@ -32,18 +32,18 @@ between our AST projection and a real CPG, and closing it by hand is reimplement
|
||||
### This Spec Owns
|
||||
|
||||
- The subtraction: which modules are removed, the dangling-import check, the module-audit manifest.
|
||||
- `cpg/` — the Joern seam (`CpgBackend`, the subprocess/query implementation, the null backend).
|
||||
- `model/` — `Verdict`/`Rung`, taint reachability, guard dominance, constant abstraction; the security specs
|
||||
- `cpg/`, the Joern seam (`CpgBackend`, the subprocess/query implementation, the null backend).
|
||||
- `model/`, `Verdict`/`Rung`, taint reachability, guard dominance, constant abstraction; the security specs
|
||||
ported from the deleted modules' vocabularies.
|
||||
- The proposer/judge orchestration where the LLM's claim is checked against the model.
|
||||
- The corpus-as-calibration harness and its model-gap report.
|
||||
|
||||
### Out of Boundary
|
||||
|
||||
- The family **taxonomy** (`learning/families.py`) — retained unchanged, re-homed under `model/`.
|
||||
- The corpus loader, per-slice split and gates — retained; the gates stay byte-identical.
|
||||
- Redaction, the LLM endpoint client (`learning/endpoint.py`) — retained.
|
||||
- Clearwing's sandbox lifecycle — never reimplemented; the execution tier adopts it.
|
||||
- The family **taxonomy** (`learning/families.py`), retained unchanged, re-homed under `model/`.
|
||||
- The corpus loader, per-slice split and gates, retained; the gates stay byte-identical.
|
||||
- Redaction, the LLM endpoint client (`learning/endpoint.py`), retained.
|
||||
- Clearwing's sandbox lifecycle, never reimplemented; the execution tier adopts it.
|
||||
|
||||
### Allowed Dependencies
|
||||
|
||||
@@ -63,7 +63,7 @@ between our AST projection and a real CPG, and closing it by hand is reimplement
|
||||
### Existing Architecture Analysis
|
||||
|
||||
The `learning/` package (4,091 lines, 16 modules) implements the noise architecture and is **absent from the
|
||||
gate path** — the gates import only from `pairs`, `benchmark`, `findings`, `mapping`, `preprocess`, `rank`,
|
||||
gate path**. The gates import only from `pairs`, `benchmark`, `findings`, `mapping`, `preprocess`, `rank`,
|
||||
`complexity.map`. So the subtraction is clean: removing `rounds`, `acceptance`, `proposer`, `journal`, the
|
||||
noise machinery in `scoring`, and the mechanism loop cannot move a gate. What survives and re-homes: the
|
||||
taxonomy (`families.py`), the candidate enumerator (`candidates.py`, the `suspicion`-band feeder), the LLM
|
||||
@@ -140,7 +140,7 @@ src/openultrasast/model/
|
||||
- `learning/candidates.py` → `model/candidates.py` (the suspicion-band enumerator, Req 8.4).
|
||||
- `learning/endpoint.py` → `model/endpoint.py` (the judge client).
|
||||
- `semantic/obligations/` retained; its operation/discharge facts feed `model/dominance.py`.
|
||||
- `pairs.py`, the gates, redaction — unchanged.
|
||||
- `pairs.py`, the gates, redaction, unchanged.
|
||||
|
||||
### The audit manifest (Req 3)
|
||||
|
||||
@@ -288,21 +288,21 @@ is dominated by no discharger and its siblings are, else `CORROBORATED`/`None`.
|
||||
|
||||
## Migration and Phases
|
||||
|
||||
1. **Subtract and audit (Req 1–3).** Remove the noise architecture; port the guard/sink vocabularies into
|
||||
1. Subtract and audit (Req 1–3). Remove the noise architecture; port the guard/sink vocabularies into
|
||||
`model/specs.py` before deleting their modules; run the import-graph check and the gate-baseline; reduce
|
||||
the tests in the same change; commit the audit manifest. Gate-protected and reversible; no CPG yet.
|
||||
2. **Joern spike on one family (Req 4, 6.1).** The `CpgBackend` seam and the injection `TaintSpec`, proven
|
||||
2. Joern spike on one family (Req 4, 6.1). The `CpgBackend` seam and the injection `TaintSpec`, proven
|
||||
end to end on the injection slice: build a CPG, run the taint query, produce entailed/corroborated
|
||||
verdicts, assert determinism. This phase can stop the design if Joern cannot deliver a deterministic
|
||||
verdict at acceptable cost — the analogue of the candidate-ceiling gate.
|
||||
3. **The ladder and the judge (Req 5, 8).** `Verdict`/`Rung`, and `judge.py` checking the LLM's residual
|
||||
verdict at acceptable cost, the analogue of the candidate-ceiling gate.
|
||||
3. The ladder and the judge (Req 5, 8). `Verdict`/`Rung`, and `judge.py` checking the LLM's residual
|
||||
answer against the verdict. No K-runs, nothing to average.
|
||||
4. **Absence and the rest of the taxonomy (Req 6.2, 7).** Dominance queries from the obligation facts; the
|
||||
4. Absence and the rest of the taxonomy (Req 6.2, 7). Dominance queries from the obligation facts; the
|
||||
remaining `TaintSpec`s. Regenerate the entailment ceiling (Req 6.5) and watch it rise as depth improves.
|
||||
5. **Corpus calibration and reporting (Req 9, 11).** The vulnerable-parent/fix differential, the model-gap
|
||||
5. Corpus calibration and reporting (Req 9, 11). The vulnerable-parent/fix differential, the model-gap
|
||||
report, per-slice + overfitting gap.
|
||||
6. **Execution tier stub (Req 10).** The `execution_confirmed` rung and the Clearwing adoption seam, built
|
||||
only when a class needs it — memory/C first, and not for web/logic v1.
|
||||
6. Execution tier stub (Req 10). The `execution_confirmed` rung and the Clearwing adoption seam, built
|
||||
only when a class needs it, memory/C first, and not for web/logic v1.
|
||||
|
||||
Phase 1 removes ~5,000 lines before a line of Joern is written. Phase 2 can end the design before the
|
||||
integration cost is paid, exactly as the candidate ceiling could have in the prior spec.
|
||||
|
||||
@@ -6,12 +6,12 @@
|
||||
|
||||
This feature supersedes `learning-harness` and `constrained-detector`. It replaces the assumption that the LLM
|
||||
is a noisy oracle to be averaged, floored and gated with an architecture in which a **deterministic program
|
||||
model arbitrates the LLM's claims**. The arbiter is a Code Property Graph (CPG) — an intermediate language
|
||||
merging AST, control-flow graph, program-dependence graph and call graph — over which taint reachability and
|
||||
model arbitrates the LLM's claims**. The arbiter is a Code Property Graph (CPG), an intermediate language
|
||||
merging AST, control-flow graph, program-dependence graph and call graph, over which taint reachability and
|
||||
guard dominance produce verdicts the LLM's assertion is no longer needed for. The measurements that force this
|
||||
design are committed: the candidate ceiling (96.6% of labeled functions have an enumerable site), and the
|
||||
entailment ceiling (a deterministic model *decides* only 2.2% today on our flat AST-projection IR, ~24-35%
|
||||
with deeper taint — the distance to a real CPG). Context: `brief.md`,
|
||||
with deeper taint, the distance to a real CPG). Context: `brief.md`,
|
||||
`benchmarks/measurements/2026-09-06-trajectory-divergence.json`, `2026-09-06-candidate-ceiling-all-slices.json`,
|
||||
`2026-09-08-model-entailment-ceiling.json`.
|
||||
|
||||
@@ -22,15 +22,15 @@ overgrown, superseded surface is how this session logged eighteen defects.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: removing the noise architecture and the superseded arbiter; reducing the test suite to
|
||||
- In scope: removing the noise architecture and the superseded arbiter; reducing the test suite to
|
||||
surviving behavior; auditing the whole tree for orphaned modules; adopting a CPG engine (Joern) as the
|
||||
model-layer substrate behind an optional extra; the evidence ladder as the verdict type; taint + dominance
|
||||
verdicts; porting our taxonomy and source/sink/sanitizer/guard models to the CPG; the LLM as a proposer
|
||||
checked against the model; the corpus as the model's calibration set; per-slice reporting.
|
||||
- **Out of scope**: building a dataflow/CFG engine by hand (the CPG is adopted, §Req 4); the execution tier
|
||||
- Out of scope: building a dataflow/CFG engine by hand (the CPG is adopted, §Req 4); the execution tier
|
||||
itself (deferred to Clearwing, `execution_confirmed` rung only, Req 10); prompt optimisation, reflective
|
||||
evolution, any optimiser (deleted, not deferred); fine-tuning weights.
|
||||
- **North star, not v1**: pure `model_entailed`-as-arbiter. The entailment ceiling says the model decides a
|
||||
- North star, not v1: pure `model_entailed`-as-arbiter. The entailment ceiling says the model decides a
|
||||
minority today; v1 is the *layered* arbiter (entail where it can, corroborate the taint band, honest
|
||||
suspicion for the rest). This is stated so the requirements are not read as promising a soundness the
|
||||
measurement denies.
|
||||
@@ -45,12 +45,12 @@ so that nothing is maintained, tested, or built upon that this feature's measure
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The system shall remove the round loop, the per-family noise floor, the regression budget, the acceptance
|
||||
rule, the sign test, the evolve journal and the meta-agent proposer — concretely `learning/rounds.py`,
|
||||
rule, the sign test, the evolve journal and the meta-agent proposer, concretely `learning/rounds.py`,
|
||||
`learning/acceptance.py`, `learning/proposer.py`, `learning/journal.py`, and the K-run / floor / sign-test
|
||||
machinery in `learning/scoring.py`.
|
||||
2. The system shall remove the mechanism-optimisation loop — the mechanism store and lever in
|
||||
2. The system shall remove the mechanism-optimisation loop (the mechanism store and lever in
|
||||
`improve/evolve.py`, and the shape-search machinery in `semantic/mechanisms.py` and the optimisation
|
||||
surface of `semantic/variants.py` — while preserving the *security models* those modules carried (the
|
||||
surface of `semantic/variants.py`), while preserving the *security models* those modules carried (the
|
||||
guard vocabulary and source/sink classification) for porting under Req 7.
|
||||
2b. The system shall remove the per-family evolvable detector configuration and its canary/verifier surface
|
||||
(`learning/detectors.py` config-evolution parts, `learning/canaries.py`, `learning/verifiers.py`,
|
||||
@@ -70,7 +70,7 @@ ratchet that pins deleted mechanisms and the suite is once again a description o
|
||||
|
||||
1. When a mechanism is removed under Req 1, the tests that existed only to pin its behavior shall be removed
|
||||
with it, in the same change, not left skipped or xfailed.
|
||||
2. Every surviving test shall map to a surviving requirement — of this spec or of a still-shipping feature —
|
||||
2. Every surviving test shall map to a surviving requirement (of this spec or of a still-shipping feature)
|
||||
and a test that maps to nothing shall be removed.
|
||||
3. The reduced suite shall be smaller than the current 832 tests, and the change shall report the count before
|
||||
and after with the requirement each removed test cluster served.
|
||||
@@ -78,7 +78,7 @@ ratchet that pins deleted mechanisms and the suite is once again a description o
|
||||
determinism test (Req 6.4), the per-slice honesty tests, and the two ceiling measurements' regeneration
|
||||
tests.
|
||||
5. A test that asserts a shape "does nothing" or greps `--help`, or that passes with its capability absent,
|
||||
shall be removed rather than kept — the session logged several; none survive.
|
||||
shall be removed rather than kept. The session logged several; none survive.
|
||||
|
||||
### Requirement 3: Audit the whole tree for outgrowth
|
||||
|
||||
@@ -89,7 +89,7 @@ found and removed rather than carried indefinitely.
|
||||
|
||||
1. The system shall classify every source module as one of: load-bearing for the model-grounded architecture;
|
||||
a retained standalone capability with its own users; or orphaned.
|
||||
2. If a module is orphaned — no importer, no CLI entry, no gate dependency, no surviving requirement — then the
|
||||
2. If a module is orphaned (no importer, no CLI entry, no gate dependency, no surviving requirement), then the
|
||||
system shall remove it and record the classification.
|
||||
3. The system shall name, for the audit, the older-phase subsystems whose status is not obvious
|
||||
(`fusion.py`, `mcp.py`, `skills.py`, `harness_ext.py`, `hunter_harness.py`) and give each an explicit
|
||||
@@ -122,7 +122,7 @@ model result is never confused with an LLM suspicion.
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The system shall assign every finding one rung: `suspicion` (a site was enumerated / the LLM flagged it),
|
||||
`model_corroborated` (the LLM's claim is consistent with the CPG — the source it names reaches the sink
|
||||
`model_corroborated` (the LLM's claim is consistent with the CPG, the source it names reaches the sink
|
||||
through resolved dataflow, or a guard its siblings carry is absent), `model_entailed` (the CPG alone
|
||||
establishes it, independent of trusting the LLM), or `execution_confirmed` (a sandbox reproduced it).
|
||||
2. The system shall establish `model_corroborated` and `model_entailed` from the CPG deterministically, with
|
||||
@@ -145,21 +145,21 @@ project measured is removed on the band the model covers.
|
||||
4. When the system runs the model layer twice over the same CPG, the sequence of verdicts shall be identical,
|
||||
and a test shall assert it.
|
||||
5. The system shall report, per family and per slice, the fraction of the corpus it can `model_entail` and
|
||||
`model_corroborate` — the regeneration of the committed entailment ceiling — and shall name where it falls
|
||||
`model_corroborate` (the regeneration of the committed entailment ceiling), and shall name where it falls
|
||||
back to `suspicion`.
|
||||
|
||||
### Requirement 7: Port the taxonomy and security models onto the CPG
|
||||
|
||||
**Objective:** As a maintainer, I want our security knowledge — the classes we target and how their bugs look
|
||||
— expressed as queries over the adopted CPG, so that the contribution is the modelling, not the engine.
|
||||
**Objective:** As a maintainer, I want our security knowledge (the classes we target and how their bugs look)
|
||||
expressed as queries over the adopted CPG, so that the contribution is the modelling, not the engine.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The system shall retain the closed family taxonomy as the routing key, unchanged in meaning.
|
||||
2. The system shall express the source, sink and sanitizer models for the web/logic CWEs as CPG taint
|
||||
specifications, seeded from the models the deleted modules carried.
|
||||
3. The system shall express the obligation / absence model — an operation on a protected resource, and the
|
||||
guards that discharge it — as CPG dominance queries, as the arbiter for the classes that never crash.
|
||||
3. The system shall express the obligation / absence model (an operation on a protected resource, and the
|
||||
guards that discharge it) as CPG dominance queries, as the arbiter for the classes that never crash.
|
||||
4. The system shall keep the verifier boundary: the models and gold labels stay outside anything an LLM or a
|
||||
future optimiser may write.
|
||||
|
||||
@@ -177,7 +177,7 @@ with its claim checked against the model, so that there is no oracle to average.
|
||||
`model_corroborated`.
|
||||
4. When the model can neither confirm nor contradict, the system shall report the claim at `suspicion`, and
|
||||
this band is where a future execution tier earns its place.
|
||||
5. The system shall contain no K-run averaging, no noise floor, no acceptance rule, no evolve loop — the model
|
||||
5. The system shall contain no K-run averaging, no noise floor, no acceptance rule, no evolve loop. The model
|
||||
check replaces all of them.
|
||||
|
||||
### Requirement 9: The corpus is the model's calibration set
|
||||
@@ -188,7 +188,7 @@ vote, so that a gap is a gap in the model, named and fixable.
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The system shall use the vulnerable-parent / fix-commit pair as ground truth for whether the model can
|
||||
distinguish the two — the fix is what the model should be able to see.
|
||||
distinguish the two. The fix is what the model should be able to see.
|
||||
2. If the model cannot distinguish a pair's two sides, then the system shall record it as a named model gap
|
||||
with the reason, not as a detector miss to be averaged away.
|
||||
3. The system shall keep the per-slice split with the real-world slices deciding, and shall not present the
|
||||
@@ -204,7 +204,7 @@ arbitrate, without building or requiring it at v1.
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The system shall treat `execution_confirmed` as an optional rung reached only for classes where the model
|
||||
layer cannot decide — memory-safety in C first.
|
||||
layer cannot decide, memory-safety in C first.
|
||||
2. When execution confirmation is used, the system shall adopt it from Clearwing (the existing fork), not
|
||||
reimplement the sandbox lifecycle, container pooling, sanitizer images or PoC stability.
|
||||
3. The system shall not require the execution tier for any web/logic class at v1.
|
||||
|
||||
@@ -25,8 +25,8 @@ Fix the scorer and labels first (no engine change). Anchor harvest at the declar
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: scorer fidelity; label schema; harvest anchor and modes; corpus hygiene for sast; three new slices; dataset pointers; provenance fingerprint; per-profile and per-mechanism dashboard; per-profile improve-loop gate; offline tests.
|
||||
- **Out**: any change to taint, facts, CST walker, or dispositions (engine specs); tolerating tree-sitter ERROR nodes (engine); BaxBench generation and exploit-labelled slices (needs deep mode; later); profile-specific fact tables; making any dashboard a merge gate; vendoring SecBench.js test files (no license).
|
||||
- In: scorer fidelity; label schema; harvest anchor and modes; corpus hygiene for sast; three new slices; dataset pointers; provenance fingerprint; per-profile and per-mechanism dashboard; per-profile improve-loop gate; offline tests.
|
||||
- Out: any change to taint, facts, CST walker, or dispositions (engine specs); tolerating tree-sitter ERROR nodes (engine); BaxBench generation and exploit-labelled slices (needs deep mode; later); profile-specific fact tables; making any dashboard a merge gate; vendoring SecBench.js test files (no license).
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
@@ -42,14 +42,14 @@ Fix the scorer and labels first (no engine change). Anchor harvest at the declar
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: pair catalog schema; overlay eval path; `build_pair_signals`; `run_improvement` accept gate; preprocess tags.
|
||||
- **Downstream**: `overlay-ir-completeness` and `guard-dominance-regime` measure against these slices and vocabularies; later `baxgen` slice reuses provenance and mechanism fields.
|
||||
- Upstream: pair catalog schema; overlay eval path; `build_pair_signals`; `run_improvement` accept gate; preprocess tags.
|
||||
- Downstream: `overlay-ir-completeness` and `guard-dominance-regime` measure against these slices and vocabularies; later `baxgen` slice reuses provenance and mechanism fields.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Supersedes**: `real-world-vfc-slice` (corpus/scorer plane). Its seed pairs, harvest script, and dual scoreboard are kept and extended.
|
||||
- **Extends**: roadmap direct candidate "record inventory vs overlay Youden side-by-side in sast JSON".
|
||||
- **Adjacent**: `propose-adjudicate-prove` (one additive field on `OverlayRecord`), `harnessx-self-improving-rulesets` (accept gate grows a per-profile clause, ledger format unchanged).
|
||||
- Supersedes: `real-world-vfc-slice` (corpus/scorer plane). Its seed pairs, harvest script, and dual scoreboard are kept and extended.
|
||||
- Extends: roadmap direct candidate "record inventory vs overlay Youden side-by-side in sast JSON".
|
||||
- Adjacent: `propose-adjudicate-prove` (one additive field on `OverlayRecord`), `harnessx-self-improving-rulesets` (accept gate grows a per-profile clause, ledger format unchanged).
|
||||
|
||||
## Constraints
|
||||
|
||||
|
||||
@@ -172,11 +172,11 @@ tests/
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `src/openultrasast/pairs.py` — see above; `OVERLAY_SLICES` grows; `PairCase` gains `provenance`, `mechanism` (on expected), `split`, `known_limit`.
|
||||
- `src/openultrasast/semantic/overlay.py` — fill `function` from the `FunctionIR` whose range contains the record line; default `None`.
|
||||
- `src/openultrasast/improve/evolve.py` — `_Eval` computes `per_profile` on holdout; new reject reason `profile_regression:<profile>`.
|
||||
- `src/openultrasast/preprocess.py`, `reports.py`, `cli.py` — wiring only.
|
||||
- `benchmarks/pairs/README.md`, `.kiro/steering/roadmap.md` — slice list and spec order.
|
||||
- `src/openultrasast/pairs.py`: see above; `OVERLAY_SLICES` grows; `PairCase` gains `provenance`, `mechanism` (on expected), `split`, `known_limit`.
|
||||
- `src/openultrasast/semantic/overlay.py`: fill `function` from the `FunctionIR` whose range contains the record line; default `None`.
|
||||
- `src/openultrasast/improve/evolve.py`: `_Eval` computes `per_profile` on holdout; new reject reason `profile_regression:<profile>`.
|
||||
- `src/openultrasast/preprocess.py`, `reports.py`, `cli.py`: wiring only.
|
||||
- `benchmarks/pairs/README.md`, `.kiro/steering/roadmap.md`: slice list and spec order.
|
||||
|
||||
## System Flows
|
||||
|
||||
|
||||
@@ -8,9 +8,9 @@ Engineers evolving OpenUltraSAST's overlay cannot trust the pair scoreboard: cov
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: pair scorer fidelity; catalog label schema (function, mechanism, provenance, known limits, held-out split); harvest anchoring and extraction modes; sast corpus hygiene; `vibe-py`, `vfc-js`, `agent-vfc` slices; dataset pointers; deterministic provenance fingerprint at scan time; per-profile and per-mechanism dashboard; per-profile acceptance in the improve loop; offline tests.
|
||||
- **Out of scope**: taint, facts, CST walker, dispositions, or tolerance of parser ERROR nodes (engine specs); BaxBench generation slices; profile-specific fact tables or thresholds; any new merge gate; vendoring files without a stated license; LLM-based classification.
|
||||
- **Adjacent expectations**: `local` stays the 100% pair gate; the stage-1 smoke gate stays inventory-only; `OverlayRecord` may gain one optional additive field; the improve loop's existing accept gate (recall floor, FP ceiling, score, no matched regression) stays and grows one clause; ledger file formats do not change.
|
||||
- In scope: pair scorer fidelity; catalog label schema (function, mechanism, provenance, known limits, held-out split); harvest anchoring and extraction modes; sast corpus hygiene; `vibe-py`, `vfc-js`, `agent-vfc` slices; dataset pointers; deterministic provenance fingerprint at scan time; per-profile and per-mechanism dashboard; per-profile acceptance in the improve loop; offline tests.
|
||||
- Out of scope: taint, facts, CST walker, dispositions, or tolerance of parser ERROR nodes (engine specs); BaxBench generation slices; profile-specific fact tables or thresholds; any new merge gate; vendoring files without a stated license; LLM-based classification.
|
||||
- Adjacent expectations: `local` stays the 100% pair gate; the stage-1 smoke gate stays inventory-only; `OverlayRecord` may gain one optional additive field; the improve loop's existing accept gate (recall floor, FP ceiling, score, no matched regression) stays and grows one clause; ledger file formats do not change.
|
||||
|
||||
## Requirements
|
||||
|
||||
@@ -23,7 +23,7 @@ Engineers evolving OpenUltraSAST's overlay cannot trust the pair scoreboard: cov
|
||||
1. When an overlay slice pair is scored, the system shall count a `coverage` record that carries a source-to-sink flow as a detection on the vulnerable side and as a leak on the fixed side, symmetrically with `promote`.
|
||||
2. When an expected row names a function, the system shall match a promotion or coverage record only if the record lies inside that function's line range.
|
||||
3. When an expected row names neither a rule id nor a sink nor a function, the system shall not count a CWE-only match as a detection and shall flag the row as `weak_label` in the payload.
|
||||
4. When any overlay slice is evaluated, the system shall report inventory and overlay metrics side by side for that slice, not only for `vfc`.
|
||||
4. When any overlay slice is evaluated, the system shall report inventory and overlay metrics side by side for that slice as well as for `vfc`.
|
||||
5. When a pair is evaluated, the system shall report how many files on each side were `parse_failed` and how many proposals were `unadjudicated`, so parser loss is visible on the dashboard.
|
||||
6. The system shall keep `local` as the only slice whose result can fail CI.
|
||||
7. Where a hunter model is configured, the system shall report a third scorer, `hunter`, that runs the tool hunter on each isolated pair function and scores it with the same pair rules, so the LLM plane is measured on the same corpus; absence of a model shall skip the scorer with a recorded degradation.
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
## Summary
|
||||
|
||||
- **Feature**: `pair-corpus-honesty`
|
||||
- **Discovery Scope**: Extension of the pair catalog, scorer, and harvest; supersedes `real-world-vfc-slice`
|
||||
- **Key Findings**:
|
||||
- Feature: `pair-corpus-honesty`
|
||||
- Discovery Scope: Extension of the pair catalog, scorer, and harvest; supersedes `real-world-vfc-slice`
|
||||
- Key Findings:
|
||||
- The overlay scorer ignores `coverage`. The Java sqli vuln file already produces a coverage record on `prepareCall` from a servlet source; the scorer reports MISS.
|
||||
- The overlay column equals the inventory column on all 176 vfc pairs because C facts have no memory sink, so every proposal is `flow_incomplete` and the scorer falls back to inventory. The dashboard cannot show engine progress until the scorer reports parser and adjudication loss explicitly.
|
||||
- Harvest anchors on the first textual occurrence of the function name. For curl that is the doc comment mention (`Curl_follow()`), so 25 pairs start mid-comment and fail to parse.
|
||||
@@ -18,23 +18,23 @@
|
||||
|
||||
### Scorer blind spots
|
||||
|
||||
- **Context**: Why the sast overlay number did not move after the tree-sitter extra.
|
||||
- **Sources Consulted**: `src/openultrasast/pairs.py` (`_evaluate_overlay_pair`, `_overlay_matches_expected`, `_overlay_leaks`), `semantic/overlay.py`, direct runs of `adjudicate` on the sast files.
|
||||
- **Findings**: only `promote` is counted; `coverage` is excluded on both sides. `_overlay_matches_expected` accepts a CWE-only match. `ExpectedFinding` already has `function` and `line`, and `benchmark._matches_expected` honors them, but the overlay path does not. `OverlayRecord` has no function field; `FunctionIR` has line ranges.
|
||||
- **Implications**: count coverage symmetrically; add optional `function` to `OverlayRecord`; require function or sink or rule id for a match when present; flag CWE-only rows.
|
||||
- Context: Why the sast overlay number did not move after the tree-sitter extra.
|
||||
- Sources Consulted: `src/openultrasast/pairs.py` (`_evaluate_overlay_pair`, `_overlay_matches_expected`, `_overlay_leaks`), `semantic/overlay.py`, direct runs of `adjudicate` on the sast files.
|
||||
- Findings: only `promote` is counted; `coverage` is excluded on both sides. `_overlay_matches_expected` accepts a CWE-only match. `ExpectedFinding` already has `function` and `line`, and `benchmark._matches_expected` honors them, but the overlay path does not. `OverlayRecord` has no function field; `FunctionIR` has line ranges.
|
||||
- Implications: count coverage symmetrically; add optional `function` to `OverlayRecord`; require function or sink or rule id for a match when present; flag CWE-only rows.
|
||||
|
||||
### Harvest anchor defect
|
||||
|
||||
- **Context**: 50 vfc files `parse_failed`.
|
||||
- **Sources Consulted**: `benchmarks/pairs/vfc/harvest.py` `extract_function`; tree-sitter ERROR nodes on `curl-cve-2022-27774/vuln.c` and `curl-cve-2023-38545-do_socks5/vuln.c`.
|
||||
- **Findings**: the name search accepts a match followed by `(` even inside a comment, and the brace search then finds the real body. A second, smaller class is `#if`-split `else if` chains, which tree-sitter cannot attach; that is an engine tolerance question, not harvest.
|
||||
- **Implications**: skip comment and string regions before searching; anchor on a line that starts a declarator; add line-range and hunk modes so Python and JS/TS are covered; report parse loss on the dashboard so the engine spec can measure its tolerance work.
|
||||
- Context: 50 vfc files `parse_failed`.
|
||||
- Sources Consulted: `benchmarks/pairs/vfc/harvest.py` `extract_function`; tree-sitter ERROR nodes on `curl-cve-2022-27774/vuln.c` and `curl-cve-2023-38545-do_socks5/vuln.c`.
|
||||
- Findings: the name search accepts a match followed by `(` even inside a comment, and the brace search then finds the real body. A second, smaller class is `#if`-split `else if` chains, which tree-sitter cannot attach; that is an engine tolerance question, not harvest.
|
||||
- Implications: skip comment and string regions before searching; anchor on a line that starts a declarator; add line-range and hunk modes so Python and JS/TS are covered; report parse loss on the dashboard so the engine spec can measure its tolerance work.
|
||||
|
||||
### Candidate corpora
|
||||
|
||||
- **Context**: User asked for vibe-coded and Node/TypeScript/Python web corpora.
|
||||
- **Sources Consulted**: Real-Vuln-Benchmark repo and paper; SecBench.js repo and ICSE 2023 paper; AIDev dataset card and paper; BaxBench site, repo, and paper; SecurityEval, CodeSecEval, CWEval; CyberSecEval benchmarks README; VibeApps paper; TS-VulBench page; SecureVibeBench repo; GitHub commit search via `gh api search/commits`.
|
||||
- **Findings**:
|
||||
- Context: User asked for vibe-coded and Node/TypeScript/Python web corpora.
|
||||
- Sources Consulted: Real-Vuln-Benchmark repo and paper; SecBench.js repo and ICSE 2023 paper; AIDev dataset card and paper; BaxBench site, repo, and paper; SecurityEval, CodeSecEval, CWEval; CyberSecEval benchmarks README; VibeApps paper; TS-VulBench page; SecureVibeBench repo; GitHub commit search via `gh api search/commits`.
|
||||
- Findings:
|
||||
| Corpus | Languages | Labels | Fixed twin | License | Verdict |
|
||||
|---|---|---|---|---|---|
|
||||
| Real-Vuln-Benchmark | Python web | file, line range, function, CWEs, authorship, FP traps | traps | Apache-2.0 | `vibe-py` |
|
||||
@@ -46,14 +46,14 @@
|
||||
| VibeApps/VibeVulns | TS, JS, React | not released | no | n/a | method only |
|
||||
| TS-VulBench | TS synthetic | paywalled | no | unknown | skip |
|
||||
| SecureVibeBench | C/C++ | agent tasks | n/a | MIT | skip for web |
|
||||
- **Implications**: three slices now; BaxBench after deep mode is stable; CyberSecEval never as recall ground truth.
|
||||
- Implications: three slices now; BaxBench after deep mode is stable; CyberSecEval never as recall ground truth.
|
||||
|
||||
### Classification and branching
|
||||
|
||||
- **Context**: User expects tuning to split into branches per corpus class.
|
||||
- **Sources Consulted**: `improve/evolve.py` accept gate; `ruleset/store.py` ledger; `calibration.py` scope; `preprocess.detect_tags`; `build_pair_signals`.
|
||||
- **Findings**: the accept gate already ANDs four clauses and reverts byte-for-byte. Pair signals carry pair name but no profile. Ledger is one file per target. Preprocess tags are deterministic and per file.
|
||||
- **Implications**: classify on two axes (provenance for gating, mechanism for reporting). Branch the acceptance decision, not the ledger format: reject a change if any profile's holdout regresses. Profile-specific facts and thresholds are deferred to the engine specs where those data live.
|
||||
- Context: User expects tuning to split into branches per corpus class.
|
||||
- Sources Consulted: `improve/evolve.py` accept gate; `ruleset/store.py` ledger; `calibration.py` scope; `preprocess.detect_tags`; `build_pair_signals`.
|
||||
- Findings: the accept gate already ANDs four clauses and reverts byte-for-byte. Pair signals carry pair name but no profile. Ledger is one file per target. Preprocess tags are deterministic and per file.
|
||||
- Implications: classify on two axes (provenance for gating, mechanism for reporting). Branch the acceptance decision, not the ledger format: reject a change if any profile's holdout regresses. Profile-specific facts and thresholds are deferred to the engine specs where those data live.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
@@ -68,48 +68,48 @@
|
||||
|
||||
### Decision: Count coverage-with-flow as detection and as leak
|
||||
|
||||
- **Context**: Sink aliases will always land as coverage.
|
||||
- **Alternatives Considered**: promote coverage to `promote` in the engine; count coverage only on the vulnerable side.
|
||||
- **Selected Approach**: scorer treats `coverage` with a non-empty source the same as `promote` on both sides.
|
||||
- **Rationale**: `propose-adjudicate-prove` already states coverage enters the same state machine. Asymmetry would inflate recall.
|
||||
- **Trade-offs**: some pairs move from MISS to LEAK; that is the honest reading.
|
||||
- Context: Sink aliases will always land as coverage.
|
||||
- Alternatives Considered: promote coverage to `promote` in the engine; count coverage only on the vulnerable side.
|
||||
- Selected Approach: scorer treats `coverage` with a non-empty source the same as `promote` on both sides.
|
||||
- Rationale: `propose-adjudicate-prove` already states coverage enters the same state machine. Asymmetry would inflate recall.
|
||||
- Trade-offs: some pairs move from MISS to LEAK; that is the honest reading.
|
||||
|
||||
### Decision: Function-scoped matching with weak-label flagging
|
||||
|
||||
- **Context**: CWE-only matching over-counts.
|
||||
- **Selected Approach**: expected rows carry `function`; match requires the record inside that function's lines; rows with no function, sink, or rule id are `weak_label` and never match by CWE alone.
|
||||
- **Trade-offs**: vfc recall may drop on first run; the drop is real.
|
||||
- Context: CWE-only matching over-counts.
|
||||
- Selected Approach: expected rows carry `function`; match requires the record inside that function's lines; rows with no function, sink, or rule id are `weak_label` and never match by CWE alone.
|
||||
- Trade-offs: vfc recall may drop on first run; the drop is real.
|
||||
|
||||
### Decision: Provenance is deterministic and closed
|
||||
|
||||
- **Context**: No LLM in classification.
|
||||
- **Selected Approach**: signals from `.claude/`, `CLAUDE.md`, `AGENTS.md`, `.cursor/`, `.github/copilot-instructions.md`, Lovable meta tag, "Generated with Claude Code", and co-author trailers for Claude, Codex, Copilot, Cursor, Devin, Jules when a git log is readable. Threshold for `agent` on a git tree: at least 85% of commits or lines in the last N commits carry a signal, following the VibeApps method; otherwise `mixed` when any signal exists; `human` when none; `synthetic` only from catalog declaration.
|
||||
- **Trade-offs**: repos that scrub trailers read as human. Acceptable; the label is a profile, not a verdict.
|
||||
- Context: No LLM in classification.
|
||||
- Selected Approach: signals from `.claude/`, `CLAUDE.md`, `AGENTS.md`, `.cursor/`, `.github/copilot-instructions.md`, Lovable meta tag, "Generated with Claude Code", and co-author trailers for Claude, Codex, Copilot, Cursor, Devin, Jules when a git log is readable. Threshold for `agent` on a git tree: at least 85% of commits or lines in the last N commits carry a signal, following the VibeApps method; otherwise `mixed` when any signal exists; `human` when none; `synthetic` only from catalog declaration.
|
||||
- Trade-offs: repos that scrub trailers read as human. Acceptable; the label is a profile, not a verdict.
|
||||
|
||||
### Decision: Branch the accept decision, not the data
|
||||
|
||||
- **Context**: Overfitting risk with per-slice tuning.
|
||||
- **Selected Approach**: `run_improvement` evaluates holdout pairs per profile; any profile regression beyond tolerance rejects. Profiles capped at four. Minimum holdout size reported.
|
||||
- **Follow-up**: profile-keyed facts land with the engine specs, gated by this clause.
|
||||
- Context: Overfitting risk with per-slice tuning.
|
||||
- Selected Approach: `run_improvement` evaluates holdout pairs per profile; any profile regression beyond tolerance rejects. Profiles capped at four. Minimum holdout size reported.
|
||||
- Follow-up: profile-keyed facts land with the engine specs, gated by this clause.
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- Recall numbers drop after function-scoped matching and weak-label flags — expected; record before and after in the roadmap.
|
||||
- SecBench.js has no license — never vendor its files; harvest from upstream package repos and record each package license.
|
||||
- Agent-authored commits may be false labels (a "fix" that was not a vuln) — human review required; keep `agent-vfc` small.
|
||||
- Real-Vuln-Benchmark apps are educational and CTF style — say so in the slice README; do not treat `vibe-py` as production.
|
||||
- Held-out halves are small on day one — report minimum-size warnings; do not gate on a profile under the minimum.
|
||||
- Recall numbers drop after function-scoped matching and weak-label flags; expected; record before and after in the roadmap.
|
||||
- SecBench.js has no license; never vendor its files; harvest from upstream package repos and record each package license.
|
||||
- Agent-authored commits may be false labels (a "fix" that was not a vuln); human review required; keep `agent-vfc` small.
|
||||
- Real-Vuln-Benchmark apps are educational and CTF style; say so in the slice README; do not treat `vibe-py` as production.
|
||||
- Held-out halves are small on day one; report minimum-size warnings; do not gate on a profile under the minimum.
|
||||
|
||||
## References
|
||||
|
||||
- [Real-Vuln-Benchmark](https://github.com/kolega-ai/Real-Vuln-Benchmark) — Apache-2.0, manifest with pinned SHAs, ground-truth schema
|
||||
- [RealVuln paper](https://arxiv.org/abs/2604.13764) — 26 human plus 40 LLM-generated Python repos, FP traps
|
||||
- [SecBench.js](https://github.com/cristianstaicu/SecBench.js) — 600 npm vulns, `fixCommit`, `sinkLocation`
|
||||
- [Real-Vuln-Benchmark](https://github.com/kolega-ai/Real-Vuln-Benchmark): Apache-2.0, manifest with pinned SHAs, ground-truth schema
|
||||
- [RealVuln paper](https://arxiv.org/abs/2604.13764): 26 human plus 40 LLM-generated Python repos, FP traps
|
||||
- [SecBench.js](https://github.com/cristianstaicu/SecBench.js): 600 npm vulns, `fixCommit`, `sinkLocation`
|
||||
- [SecBench.js paper](https://software-lab.org/publications/icse2023_SecBenchJS.pdf)
|
||||
- [AIDev dataset](https://huggingface.co/datasets/hao-li/AIDev) — agent-authored PRs, CC-BY-4.0 index
|
||||
- [BaxBench](https://github.com/logic-star-ai/baxbench) — MIT, scenarios and exploits, generation by CLI
|
||||
- [AIDev dataset](https://huggingface.co/datasets/hao-li/AIDev): agent-authored PRs, CC-BY-4.0 index
|
||||
- [BaxBench](https://github.com/logic-star-ai/baxbench): MIT, scenarios and exploits, generation by CLI
|
||||
- [SecurityEval](https://github.com/s2e-lab/SecurityEval), [CodeSecEval](https://arxiv.org/pdf/2407.02395), [CWEval](https://github.com/Co1lin/CWEval)
|
||||
- [CyberSecEval](https://github.com/meta-llama/PurpleLlama/tree/main/CybersecurityBenchmarks) — labels defined by ICD rules
|
||||
- [VibeApps paper](https://arxiv.org/html/2606.23130v1) — identification method
|
||||
- `.kiro/specs/real-world-vfc-slice/` — superseded corpus spec
|
||||
- Memory note `overlay-gaps-diagnosis-2026-09` — measured state and engine gaps
|
||||
- [CyberSecEval](https://github.com/meta-llama/PurpleLlama/tree/main/CybersecurityBenchmarks): labels defined by ICD rules
|
||||
- [VibeApps paper](https://arxiv.org/html/2606.23130v1): identification method
|
||||
- `.kiro/specs/real-world-vfc-slice/`: superseded corpus spec
|
||||
- Memory note `overlay-gaps-diagnosis-2026-09`: measured state and engine gaps
|
||||
|
||||
@@ -206,25 +206,25 @@ URL; the CLI mints them and polls the store. The URLs travel in the start body,
|
||||
"inputs": {"CANDIDATES": "<presigned GET runs/<run>/<producer>/<artifact>>"}}}
|
||||
```
|
||||
|
||||
- **Why the start body.** A presigned URL is a bearer capability, so the credentials-only-in-the-start-request
|
||||
- Why the start body. A presigned URL is a bearer capability, so the credentials-only-in-the-start-request
|
||||
rule applies; env would bake it into the ActorTemplate and hand it to the golden boot; and the body has no
|
||||
32 KB ceiling (Substrate rejects env values over 32768 characters, `reconciler.py:141`). `StartGate.offer`
|
||||
(`runner.py:250-275`) validates `delivery` the way it validates `credentials`: `https` URLs only, hostnames,
|
||||
no echo. Expiry = `OUSAST_TASK_TIMEOUT` (7200 s) + 600 s; a re-start after a re-resume re-mints.
|
||||
- **Why presigned rather than a scoped delivery credential.** A per-task S3 key needs a per-task policy on RustFS,
|
||||
- Why presigned rather than a scoped delivery credential. A per-task S3 key needs a per-task policy on RustFS,
|
||||
gives the sandbox `ListBucket` and reads beyond its inputs, and would travel in the same body. Presigned URLs
|
||||
name exactly the objects a task may write and read, which is what the receiver enforces today
|
||||
(`Receiver.input_file`, `egress.py:230-247`).
|
||||
- **Why not an in-cluster receiver Service.** It is a server to run, holds state in a process, and is reachable
|
||||
- Why not an in-cluster receiver Service. It is a server to run, holds state in a process, and is reachable
|
||||
only from a pod; that is the gap remote mode has. **Rejected.**
|
||||
- **Objects.** `runs/<run>/<task>/output.tar` (the runner's PUT); after the CLI extracts it, the declared
|
||||
- Objects. `runs/<run>/<task>/output.tar` (the runner's PUT); after the CLI extracts it, the declared
|
||||
`outputs[]` are re-put as `runs/<run>/<task>/<artifact>` so a consumer's presigned GET names one artifact, and
|
||||
`runs/<run>/<task>/done.json` carries the summary's status. The 30-day `runs/` expiry already applies; memory
|
||||
rows are ingested from the local extraction by `remember` as today (f: annotations and `memory_key` seeding
|
||||
unchanged, `generate.py:_chain_inputs`, `memory.seed`).
|
||||
- **Completion.** `_await`'s `delivered()` becomes `Delivery.delivered(task)`: a HEAD on `output.tar`, polled at
|
||||
- Completion. `_await`'s `delivered()` becomes `Delivery.delivered(task)`: a HEAD on `output.tar`, polled at
|
||||
`OUSAST_POLL_SECONDS` (3 s). The runner's completion marker logic (`runner.py`, `_run_once`) is unchanged.
|
||||
- **Inputs no task produces** (`candidates.json`, `functions.json`, `case.json`) may also be store objects
|
||||
- Inputs no task produces (`candidates.json`, `functions.json`, `case.json`) may also be store objects
|
||||
declared as inputs, which lifts the ~20 KB inline-files limit (`ops/ax/README.md:117`); Workspace `files`
|
||||
stay for the smoke Task and small cases.
|
||||
|
||||
@@ -251,7 +251,7 @@ shim and `/workspace`), so one *contract* in two images: `ousast-runner` (slim,
|
||||
2.8 GB and every golden snapshot carry it; the engine runs for a minority of cases); Docker-in-pod (no socket on
|
||||
containerd; `Dockerfile:1-7` already argues against it).
|
||||
|
||||
- **Task.** Template `plane/tasks/engine.yaml`: `command: ["engine"]`, env `OUSAST_WORKSPACE_DIR`,
|
||||
- Task. Template `plane/tasks/engine.yaml`: `command: ["engine"]`, env `OUSAST_WORKSPACE_DIR`,
|
||||
`OUSAST_FIXED_DIR`, `OUSAST_INPUT_CASE`, pins and `OUSAST_FAMILIES`, bound to the same two Workspaces the
|
||||
`alerts` task binds (`generate.py:342-353`). The module lifts `benchmarks/push/finding_dump.py`'s scan into
|
||||
`openultrasast.cpg.dump` (the image ships `src/`, not `benchmarks/`) and reuses `engine_alerts.engine_rows`,
|
||||
@@ -259,7 +259,7 @@ containerd; `Dockerfile:1-7` already argues against it).
|
||||
`alerts.jsonl` rows (`rule_id: engine:<family>`, `source: "engine"`) and `summary.json` (coverage with the
|
||||
engine's languages, files and bytes read, questions, seconds, degradations per pin) as `alerts-engine`, with
|
||||
`host: false`. The instrument rule stays: no `read N files` line or `questions == 0` fails the task loudly.
|
||||
- **Sizing from the record** (`benchmarks/measurements/2026-09-30-php-engine-alerts/record.json`): container
|
||||
- Sizing from the record (`benchmarks/measurements/2026-09-30-php-engine-alerts/record.json`): container
|
||||
peak 2.129 GiB under a 3 GiB limit, 789 s for both pins of froxlor, deadline 1800 s per pin. Task resources:
|
||||
requests 3 Gi / 1 CPU, limits 4 Gi / 2 CPU; `engine-pool` workers 4 Gi / 2 CPU, replicas 1 on kind (the 7 GB
|
||||
host holds it only with `ousast-pool` at one worker), production replicas in `k3s.toml`. Task timeout: two pins
|
||||
@@ -314,23 +314,23 @@ runbook: a Run with `--workers N` beyond the pool's replicas queues in ax as Sus
|
||||
|
||||
`plane/scan.py`, reusing `generate.py`'s render path with a new case source:
|
||||
|
||||
1. **Workspace generation** (`generate.scan_cases(url, base, head, cache)`): a shallow clone into the case cache
|
||||
1. Workspace generation (`generate.scan_cases(url, base, head, cache)`): a shallow clone into the case cache
|
||||
(`runner.find_cached_checkout` layout), two pinned Workspaces `<repo_dir>-<head7>` and `<repo_dir>-<base7>`
|
||||
(`workspaces.py` shape, pins in `openultrasast.io/git-commits`), repository key by `memory.repo_key`.
|
||||
2. **Candidates from changed functions**: `generate.fix_ranges(repo, base, head, side="new")` gives the head-side
|
||||
2. Candidates from changed functions: `generate.fix_ranges(repo, base, head, side="new")` gives the head-side
|
||||
hunks (the `push/` delta primitive `_diff_records` is the same `git diff -U0`), `repo_facts.enclosing` names
|
||||
the function per hunk line, filtered to `repo_facts.product_files`. Quick rules on the changed files
|
||||
(`tasks.alerts.scan`) add alerted functions; `roles` runs once per repository when memory has no `roles` rows
|
||||
for it; `repo-facts` supplies callers. `candidates.json` and `functions.json` are store inputs (section 3).
|
||||
3. **Run composition**: per changed-function set `facts -> va, vb -> agree -> vc -> final -> features ->
|
||||
3. Run composition: per changed-function set `facts -> va, vb -> agree -> vc -> final -> features ->
|
||||
remember`, plus `engine` when a changed file's language is engine-covered and not quick-covered, or PHP
|
||||
(`ENGINE_ALONGSIDE_QUICK`). Budgets from `generate.verify_budget` (0.022 USD per candidate, headroom 1.5);
|
||||
`--ceiling` caps the sum. `--all` (whole repository, secondary) takes every product function.
|
||||
4. **Results** land in memory under `repos/<host>__<owner>__<name>/<head>.jsonl` through `remember` and
|
||||
4. Results land in memory under `repos/<host>__<owner>__<name>/<head>.jsonl` through `remember` and
|
||||
`ousast plane remember` (rows already carry `repo` and `pin`, `memory.py:77`); `runs/<run>/` holds the raw
|
||||
artifacts for 30 days. The report prints the attribution table and the plane's agreement (`final`); the
|
||||
decision engine's verdict only when a model is adopted (learned-decision-engine Req 6 unchanged).
|
||||
5. **Latency split (e).** `ousast pre-push` stays the blocking hook: quick rules and the delta engine inside
|
||||
5. Latency split (e). `ousast pre-push` stays the blocking hook: quick rules and the delta engine inside
|
||||
`OUSAST_PUSH_DEADLINE` (30 s, `ops/README.md:322`). `ousast plane scan --exec remote --no-wait` submits and
|
||||
prints the Run name in the hook's report; `--wait SECONDS` blocks for CI. `ousast plane status <run>` reports
|
||||
later. The wrapper `ops/pre-push` gains an opt-in `OUSAST_PLANE_SCAN=1`; it never blocks on the plane.
|
||||
@@ -378,7 +378,7 @@ template names `imagePullSecret` from the profile.
|
||||
- `tests/test_plane_engine_task.py`: fixture record -> rows and summary equal `engine_alerts.produce`'s shape;
|
||||
the instrument failures.
|
||||
- `tests/test_plane_scan.py`: candidates from a two-commit fixture repository; Run composition; budgets.
|
||||
- **Only live on kind:** placement on `engine-pool`; Substrate accepting the SA token for `egress-policy`;
|
||||
- Only live on kind: placement on `engine-pool`; Substrate accepting the SA token for `egress-policy`;
|
||||
the gateway passing TLS to the store host; GHCR pulls; the HPA (k3s only). These are the kind proofs above.
|
||||
|
||||
## Data Models
|
||||
@@ -411,16 +411,16 @@ from `StartGate.offer`. The engine task fails loudly on an unread input, as `rea
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Substrate authorization for the CI token.** `kubectl-ate` sends a ServiceAccount token; which audience and
|
||||
1. Substrate authorization for the CI token. `kubectl-ate` sends a ServiceAccount token; which audience and
|
||||
which Substrate-side grant let `ousast-ci` create `egress-policy` in `default` is not in the checkouts read
|
||||
here. Proven or refuted in increment 3.
|
||||
2. **Engine placement.** Whether Substrate places a 3 Gi actor only on workers that fit it (section 4). The
|
||||
2. Engine placement. Whether Substrate places a 3 Gi actor only on workers that fit it (section 4). The
|
||||
fallback is named.
|
||||
3. **gVisor tarball reachability on k3s** (`gs://gvisor/...` in the SandboxConfig): mirror or egress needed.
|
||||
4. **Store endpoint hostname** for any future in-cluster RustFS; today's external endpoint is fine.
|
||||
5. **Two images under one contract** versus the maintainer's "one runner image for all tasks": the design
|
||||
3. gVisor tarball reachability on k3s (`gs://gvisor/...` in the SandboxConfig): mirror or egress needed.
|
||||
4. Store endpoint hostname for any future in-cluster RustFS; today's external endpoint is fine.
|
||||
5. Two images under one contract versus the maintainer's "one runner image for all tasks": the design
|
||||
reads it as one contract; confirm.
|
||||
6. **GHCR visibility** (public packages need no pull secret) and whether Substrate's and ax's `ko` images go to
|
||||
6. GHCR visibility (public packages need no pull secret) and whether Substrate's and ax's `ko` images go to
|
||||
the same registry namespace.
|
||||
7. **Reconciler size**: remote mode and `Delivery` must stay under the 500-line test; if they do not, the test's
|
||||
7. Reconciler size: remote mode and `Delivery` must stay under the 500-line test; if they do not, the test's
|
||||
bound moves with a reason, not silently.
|
||||
|
||||
@@ -9,24 +9,24 @@ production entry point (a delta Run for a push) is added. Context: `brief.md`, `
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: two profiles of one code base, `kind` (local, kept) and `k3s` (production); a remote execution mode of the CLI; artifacts delivered to the store; configurable
|
||||
- In scope: two profiles of one code base, `kind` (local, kept) and `k3s` (production); a remote execution mode of the CLI; artifacts delivered to the store; configurable
|
||||
cluster context, registry and image pins; our own snapshot and memory buckets; the Joern engine as a plane task
|
||||
on a larger worker pool; worker pool sizing and autoscaling; a delta Run command and Workspaces from URL +
|
||||
commit; a deployment runbook with a proof on kind and a checklist for the target cluster.
|
||||
- **Out of scope**: provisioning or operating the target cluster; modifying Agent Substrate or ax; any change to
|
||||
- Out of scope: provisioning or operating the target cluster; modifying Agent Substrate or ax; any change to
|
||||
detection, scoring, populations or the decision engine; replacing the `ax` CLI with a bespoke client beyond what
|
||||
remote submission needs.
|
||||
- **Adjacent expectations**: `ai-service-plane` requirements (one Model + budget per task, attribution,
|
||||
- Adjacent expectations: `ai-service-plane` requirements (one Model + budget per task, attribution,
|
||||
credentials only in the start request, egress policies, verify-at-startup memory store) keep holding;
|
||||
`harnessx-removal` Req 6 (memory on the S3 store) keeps holding; nothing is deleted before its replacement
|
||||
passes its test (the `harnessx-removal` sequencing rule).
|
||||
|
||||
## Target Environment (maintainer, 2026-10-02)
|
||||
|
||||
- **Cluster: k3s.** Consequences: containerd (no Docker socket; the engine's host-Docker path does not exist there),
|
||||
- Cluster: k3s. Consequences: containerd (no Docker socket; the engine's host-Docker path does not exist there),
|
||||
gVisor needs `runsc` and a `RuntimeClass` installed on the nodes Substrate's workers run on, k3s ships Traefik
|
||||
and local-path storage, and single-node k3s is close enough to kind that the kind proof transfers.
|
||||
- **Images and source: GitHub.** Images are published to GitHub Container Registry (`ghcr.io/norandom/...`),
|
||||
- Images and source: GitHub. Images are published to GitHub Container Registry (`ghcr.io/norandom/...`),
|
||||
built by GitHub Actions from tagged commits, digest-pinned in the manifests; private packages need an
|
||||
`imagePullSecret` on the worker ServiceAccounts and the CI principal. Substrate's and ax's own images are built with
|
||||
`ko` from their checkouts and pushed to the same registry.
|
||||
|
||||
@@ -69,7 +69,7 @@ family semantics remain with their existing specs rather than being reimplemente
|
||||
- No promise of all-language coverage; capabilities are admitted per language/framework/family.
|
||||
- Initial work must establish feasibility and noise before investing in an LLM ranker.
|
||||
|
||||
## Cross-language amendment — 2026-09-12
|
||||
## Cross-language amendment, 2026-09-12
|
||||
|
||||
Keep PMPro as the primary worked optimization example, and require a JavaScript/Express/Node API
|
||||
transfer experiment before claiming the core improvement generalizes. Abstract evidence, ranking,
|
||||
|
||||
@@ -24,7 +24,7 @@ an irrelevant old defect, or an unactionable witness can still produce noise.
|
||||
- Do not build a second ranker/scope heuristic, an incremental CPG engine, a daemon, an automatic fixer,
|
||||
or a mandatory model service. Do not scan intermediate pushed history in this feature.
|
||||
|
||||
## What this hook is, and is not — 2026-09-21
|
||||
## What this hook is, and is not, 2026-09-21
|
||||
|
||||
A general-purpose pre-commit hook, not an application-specific one. It may be measured on WordPress
|
||||
plugins and Node applications, and it may not become a tool that only works on them. The practical
|
||||
@@ -63,8 +63,8 @@ The fourth and fifth cost the declared v0.1 target its entire admissibility: a W
|
||||
PHP plus admin JavaScript, so the product could detect defects on its own target and never admit
|
||||
one. Neither passing subject could reveal it, because both hold one language.
|
||||
|
||||
**In review:** every new degradation reason must state the scope it reaches — one question, one file,
|
||||
one partition, or the whole read — and a reason that demotes an outcome must justify reaching that
|
||||
**In review:** every new degradation reason must state the scope it reaches (one question, one file,
|
||||
one partition, or the whole read), and a reason that demotes an outcome must justify reaching that
|
||||
far. A gap that costs coverage without threatening correctness is a declared exclusion: it stays
|
||||
published and it does not demote.
|
||||
|
||||
@@ -462,7 +462,7 @@ decision in the terminal; do not fabricate a successful cache entry or hang whil
|
||||
separate from a future C/C++ positive control. See `evaluation.md` for selection and labeling gates.
|
||||
|
||||
Measure the accepted experimental acceptance profile in requirements on declared hardware and fixed workloads.
|
||||
Warm means prior reusable work from the preceding state, not only repeating the identical tip.
|
||||
Warm means prior reusable work from the preceding state. Repeating the identical tip alone is insufficient.
|
||||
Report identical-tip hits separately from a one-function change and a dependency/configuration
|
||||
change. Include first installs, cold graphs, growing repository size, multi-ref pushes and incomplete
|
||||
outcomes. Freeze the applicable population before ranker selection; deferred questions count against
|
||||
|
||||
@@ -68,7 +68,7 @@ real capability; task 8.3 still owns actual eligibility. Task 4.2 brings progres
|
||||
|
||||
The agent service rejected the task 4.3 dispatch with `agent thread limit reached`.
|
||||
The [kiro-impl skill](../../../.agents/skills/kiro-impl/SKILL.md) explicitly provides:
|
||||
“If multi-agent is not available, fall back to manual mode execution for all tasks.”
|
||||
"If multi-agent is not available, fall back to manual mode execution for all tasks."
|
||||
Implementation, review and completion checks therefore ran in the main context.
|
||||
This is a manual review record, not an independent-review claim.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Implementation verification: group 5
|
||||
|
||||
Group 5 was authorized by the maintainer's “approved” after group 4 completion.
|
||||
Group 5 was authorized by the maintainer's "approved" after group 4 completion.
|
||||
Tasks 5.1–5.2 are implemented and verified (16/27 total); the measured feasibility decision is NO-GO.
|
||||
Real capability eligibility and rollout readiness remain unproven.
|
||||
|
||||
@@ -8,7 +8,7 @@ Real capability eligibility and rollout readiness remain unproven.
|
||||
|
||||
Fresh implementer dispatch for 5.1 returned `agent thread limit reached`.
|
||||
The [Kiro implementation skill](../../../.agents/skills/kiro-impl/SKILL.md) explicitly
|
||||
states: “If multi-agent is not available, fall back to manual mode execution for all tasks.”
|
||||
states: "If multi-agent is not available, fall back to manual mode execution for all tasks."
|
||||
Its manual review rule permits review in the main context when a fresh reviewer is unavailable.
|
||||
This run therefore uses manual implementation and adversarial review, not independent review.
|
||||
|
||||
|
||||
@@ -300,15 +300,15 @@ and its [bundle](../../../benchmarks/measurements/2026-09-16-nodegoat-m1b-explai
|
||||
Three evaluation findings were rendered on the vulnerable revision, one per eval site. For
|
||||
line 32:
|
||||
|
||||
- **Location** app/routes/contributions.js:32, family injection, novelty new with
|
||||
- Location: app/routes/contributions.js:32, family injection, novelty new with
|
||||
`operation_absent_from_comparable_base`.
|
||||
- **Witness** `eval(req.body.preTax) -> const preTax = eval(req.body.preTax)`.
|
||||
- **Change attribution** `base:app/routes/contributions.js:32-34`.
|
||||
- **Provenance** `file_scope`: the question named no function, the engine reported `<lambda>2`.
|
||||
- **Consequence** "A request field reaches const preTax = eval(req.body.preTax), where it is
|
||||
- Witness: `eval(req.body.preTax) -> const preTax = eval(req.body.preTax)`.
|
||||
- Change attribution: `base:app/routes/contributions.js:32-34`.
|
||||
- Provenance: `file_scope`: the question named no function, the engine reported `<lambda>2`.
|
||||
- Consequence: "A request field reaches const preTax = eval(req.body.preTax), where it is
|
||||
evaluated as JavaScript, so an attacker controlling that field can execute arbitrary code in
|
||||
the server process."
|
||||
- **Repair** "Replace the evaluation at const preTax = eval(req.body.preTax) with a numeric
|
||||
- Repair: "Replace the evaluation at const preTax = eval(req.body.preTax) with a numeric
|
||||
conversion such as Number(...) or parseInt(...), then validate the result before use."
|
||||
|
||||
Both sentences come from the declaration's templates through the grounded-template path, not from
|
||||
|
||||
@@ -64,9 +64,9 @@ code the precision estimate will move, against a bar of 95% with zero false aler
|
||||
benign sides. Expect the first M4 evaluation to fail and budget for a second. That is the strongest
|
||||
argument for decision 2.
|
||||
|
||||
## M1 — One real security change works end to end
|
||||
## M1: One real security change works end to end
|
||||
|
||||
Status: **NEXT — not verified**. Dependencies: existing immutable replay and scan machinery.
|
||||
Status: **NEXT, not verified**. Dependencies: existing immutable replay and scan machinery.
|
||||
M1 is split into M1a (diagnose with vetoes recorded, not applied) and M1b (repair only
|
||||
the vetoes M1a showed to matter). M1 is verified only when M1b passes.
|
||||
|
||||
@@ -80,7 +80,7 @@ a global completion veto, line ambiguity inherited by every question, and query
|
||||
differing from witness identity. Requiring all four repaired before the first visible
|
||||
result is a full production pass, not one detection.
|
||||
|
||||
### M1a — Show the detection with every veto recorded
|
||||
### M1a: Show the detection with every veto recorded
|
||||
|
||||
Status: **MET 2026-09-16**, inside its one-week box. Evidence:
|
||||
[M1a record](m1a-recorded-vetoes.md) and
|
||||
@@ -112,9 +112,9 @@ source and census receipts. **Kill criterion:** if the fixed twin is not quiet u
|
||||
recorded-veto run, M1 has a detector problem, and M1b does not start until that is
|
||||
diagnosed and recorded.
|
||||
|
||||
### M1b — Repair only the vetoes M1a showed to matter
|
||||
### M1b: Repair only the vetoes M1a showed to matter
|
||||
|
||||
Status: **NEXT — not verified**. M1a supplied the veto list: per-family change context,
|
||||
Status: **NEXT, not verified**. M1a supplied the veto list: per-family change context,
|
||||
question-owned vendor relevance, operation correspondence for an operation on a changed line,
|
||||
file-wide correspondence ambiguity scope, and query-to-operation provenance. The owner
|
||||
diagnosis expected the first and the last two; operation correspondence is the addition M1a
|
||||
@@ -144,7 +144,7 @@ Owners: contributor-scan for query/operation/context evidence; pre-push-safety-n
|
||||
comparison, admission, the recorded-veto flag and reporting. Requirements: 1.1–1.4,
|
||||
2.1–2.4, 3.1–3.5, 6.3, 7.1–7.3.
|
||||
|
||||
## M2 — The same useful analysis fits the push budget
|
||||
## M2: The same useful analysis fits the push budget
|
||||
|
||||
Status: **MEASURED AND UNMET, 2026-09-17.** Two attacks on engine startup were measured and
|
||||
neither reaches the gate. A transaction-owned session is implemented and proven equivalent for
|
||||
@@ -179,7 +179,7 @@ completion at least 95%, under the predeclared profile. A fast census alone does
|
||||
Owner: contributor-scan runtime, with pre-push-safety-net deadline/cache integration.
|
||||
Requirements: 4.1–4.4, 6.2–6.5, 7.3.
|
||||
|
||||
## M3 — The shared design transfers across languages
|
||||
## M3: The shared design transfers across languages
|
||||
|
||||
Status: **PARTIAL, 2026-09-17. Not met.** Python is established and PHP is blocked by runtime.
|
||||
|
||||
@@ -218,7 +218,7 @@ remain a separate unsupported abstraction; adding a new engine is not a prerequi
|
||||
Owners: contributor-scan shared evidence and framework adapters; pre-push-safety-net evaluation.
|
||||
Requirements: 6.2–6.4, 8.1–8.4.
|
||||
|
||||
## M4 — Independent examples establish useful detection and low noise
|
||||
## M4: Independent examples establish useful detection and low noise
|
||||
|
||||
Status: **PENDING M3**. Population preparation may proceed earlier, but evaluation follows
|
||||
the frozen candidate. Reserve untouched cases before examining their scan results.
|
||||
@@ -241,7 +241,7 @@ do not require every possible language or family to become eligible for a narrow
|
||||
Owner: pre-push-safety-net task 8.3, consuming verified contributor-scan semantics.
|
||||
Requirements: 6.1–6.5, 8.1–8.4 and the accepted experimental acceptance profile.
|
||||
|
||||
## M5 — Enable a narrow advisory rollout
|
||||
## M5: Enable a narrow advisory rollout
|
||||
|
||||
Status: **PENDING M4**.
|
||||
|
||||
|
||||
@@ -196,6 +196,6 @@ failures without lowering the bar after inspecting holdout results.
|
||||
- >= 95% observed actionable-alert precision with counts and confidence interval; zero known false
|
||||
alerts on the fixed/benign regression suite. Small samples do not establish broad precision.
|
||||
- >= 90% recall on the predeclared supported security-change suite; unsupported classes remain in
|
||||
coverage reporting, not reclassified after a miss. Publish per-capability results, not only an aggregate.
|
||||
coverage reporting, not reclassified after a miss. Publish per-capability results; an aggregate alone is insufficient.
|
||||
- No capability is admitted with zero positive controls, zero reviewed emitted alerts, or no untouched
|
||||
evaluation repository. Failed profiles remain experimental and silent in the normal hook.
|
||||
|
||||
@@ -89,7 +89,7 @@ feasibility gate. [Official CPG slicing documentation](https://docs.joern.io/cpg
|
||||
2. **Separate claim from interruption.** Existing rungs describe analysis evidence. Hook admission
|
||||
additionally requires evaluated capability, change attribution and actionability. This meets the
|
||||
user's low-noise requirement without pretending an LLM confidence score proves a bug.
|
||||
3. **Compare evidence, not just finding lists.** A missing base finding can mean truncated analysis.
|
||||
3. **Compare evidence as well as finding lists.** A missing base finding can mean truncated analysis.
|
||||
Targeted comparable base checks are required for novelty, including removed-guard cases.
|
||||
4. **Adopt Git and the existing engine.** Build only the push transaction, cache and admission adapters.
|
||||
Do not add an IR, database or service merely to describe future scale.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Pre-push safety net status — v1.2.0-alpha.1
|
||||
# Pre-push safety net status, v1.2.0-alpha.1
|
||||
|
||||
Updated 2026-09-16. Phase: **implementation**. Requirements, design and tasks are
|
||||
approved. **28/29 executable tasks complete; task 8.3 remains blocked.**
|
||||
@@ -141,7 +141,7 @@ Four of seven declared families establish findings on the two development subjec
|
||||
injection, access_control, config_secrets and untrusted_destination. Output encoding, path,
|
||||
prototype and deserialization have established nothing anywhere. Every question completed on
|
||||
both subjects, 299 of 299 on NodeGoat and 35 of 35 on VAmPI, so the silences are answers
|
||||
rather than failures — but three of them are diagnosed gaps rather than clean code:
|
||||
rather than failures, but three of them are diagnosed gaps rather than clean code:
|
||||
NodeGoat's documented SSRF is invisible because `needle.get` is not in the sink vocabulary,
|
||||
its XSS is invisible because output encoding declares only DOM sinks, and its two
|
||||
access-control defects raise no obligation because the modelled data operations live in DAO
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Packaged acceptance validation — group 8
|
||||
# Packaged acceptance validation, group 8
|
||||
|
||||
## Validation Report
|
||||
- DECISION: NO-GO
|
||||
- FEATURE_COMPLETE: false
|
||||
- BLOCKED_TASKS: 8.3 — independent reviewed populations and verified VAmPI transitive evidence are missing; useful completion and runtime gates fail.
|
||||
- BLOCKED_TASKS: 8.3, independent reviewed populations and verified VAmPI transitive evidence are missing; useful completion and runtime gates fail.
|
||||
- OWNERSHIP: UPSTREAM for engine/census/context prerequisites; LOCAL for evaluation population and capability qualification.
|
||||
- UPSTREAM_SPEC: contributor-scan; see `pre-push-runtime-followup.md` there.
|
||||
|
||||
|
||||
@@ -7,11 +7,11 @@ Status: brief / not yet specced. Proposed 2026-10-07 by the maintainer. Builds o
|
||||
|
||||
The search arm (directed, Cairn-inspired) works. The **proof-authoring** last mile is the drag:
|
||||
|
||||
1. **Safety-classifier refusals.** Authoring exploit demos repeatedly tripped a safety classifier
|
||||
1. Safety-classifier refusals. Authoring exploit demos repeatedly tripped a safety classifier
|
||||
this session when exploit payloads or step-by-step attack prose passed through the model or a
|
||||
tool prompt. The mitigation so far (keep payloads on disk, treat demos as opaque) is a workaround,
|
||||
not a fix, and it limits how directly the proof model can reason about the exploit.
|
||||
2. **Nondeterministic wiring.** The agent authors a demo but intermittently mis-wires the app to
|
||||
2. Nondeterministic wiring. The agent authors a demo but intermittently mis-wires the app to
|
||||
the oracle fixture (served directory for path, database URL for SQL), so a demonstrable pair
|
||||
comes back inconclusive until a retry. Each miss has cost a paid run.
|
||||
|
||||
@@ -21,20 +21,20 @@ proof-authoring policy with DSPy from **traces logged in OpenObserve**.
|
||||
|
||||
## Shape
|
||||
|
||||
- **Search model:** DeepSeek (current). Good at the directed fact/intent search.
|
||||
- **Proof model:** GLM 5.3 (Anthropic-recommended) and/or an abliterated model via
|
||||
- Search model: DeepSeek (current). Good at the directed fact/intent search.
|
||||
- Proof model: GLM 5.3 (Anthropic-recommended) and/or an abliterated model via
|
||||
`ABLITERATION_BASE_URL` (`abliterated large v2`). The abliterated model authors exploit **demos**
|
||||
without the refusals that block a safety-trained model on this defensive task.
|
||||
- **Proof driver:** OpenCode MCP (github.com/AlaeddineMessadi/opencode-mcp) as the agentic
|
||||
- Proof driver: OpenCode MCP (github.com/AlaeddineMessadi/opencode-mcp) as the agentic
|
||||
demo-author, running inside the gVisor verification task so the exploit-authoring is sandboxed.
|
||||
- **Learning:** DSPy compiles/optimizes the proof-authoring program (demo schema, wiring, steps)
|
||||
- Learning: DSPy compiles/optimizes the proof-authoring program (demo schema, wiring, steps)
|
||||
from past traces, instead of the per-family hand-guidance we keep adding (which is now hitting
|
||||
the prompt character budget — a signal the hand-rule approach is near its limit). Matches the
|
||||
the prompt character budget, a signal the hand-rule approach is near its limit). Matches the
|
||||
maintainer's "learned decisions, not hand-tuning" direction.
|
||||
- **Tracing:** proof runs (reason/explore/author/verify, timings, outcomes, error classes) log to
|
||||
- Tracing: proof runs (reason/explore/author/verify, timings, outcomes, error classes) log to
|
||||
**OpenObserve AI** at `OPENOBSERVE_URL` (org `OPENOBSERVE_ORG`, stream `OPENOBSERVE_TRACES_STREAM`).
|
||||
DSPy consumes these traces as its training signal.
|
||||
- **Credentials:** stored in `.env` like the existing keys, referenced by name only:
|
||||
- Credentials: stored in `.env` like the existing keys, referenced by name only:
|
||||
`ABLITERATION_API_KEY`, `ABLITERATION_BASE_URL`, `ABLITERATION_MODEL`, `OPENOBSERVE_URL`,
|
||||
`OPENOBSERVE_ORG`, `OPENOBSERVE_TRACES_STREAM`, `OPENOBSERVE_TOKEN`. No secret value ever enters a
|
||||
tracked file, a prompt, a log stream, or a commit.
|
||||
@@ -83,8 +83,8 @@ in a **gVisor-isolated** corpus, to demonstrate a regression before push. Guardr
|
||||
|
||||
## OpenCode MCP as the endpoint for both PoC and fix
|
||||
|
||||
Maintainer note (2026-10-07): OpenCode MCP is a good server endpoint for the PoC **and** the fix,
|
||||
not only proof authoring. The same agentic endpoint can:
|
||||
Maintainer note (2026-10-07): OpenCode MCP is a good server endpoint for both the PoC **and** the fix.
|
||||
The same agentic endpoint can:
|
||||
|
||||
1. Author the PoC (the demonstrated proof) for a caught finding.
|
||||
2. Write a candidate fix against the finding's repair direction and the failing PoC.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Problem
|
||||
|
||||
OpenUltraSAST already has three scan stages (`quick` inventory, `standard` map, `deep` regression), but **only the first stage actually decides**. Pattern rules propose sinks; the map ranks files; the sandbox can run recipes. Nothing in the middle answers “does this sink receive attacker-controlled data, or is it a Juliet-style good twin / dead assignment / hardcoded command?”
|
||||
OpenUltraSAST already has three scan stages (`quick` inventory, `standard` map, `deep` regression), but **only the first stage actually decides**. Pattern rules propose sinks; the map ranks files; the sandbox can run recipes. Nothing in the middle answers "does this sink receive attacker-controlled data, or is it a Juliet-style good twin / dead assignment / hardcoded command?"
|
||||
|
||||
Calibration corpora (cheat-sheet fixtures, OWASP Benchmark true/false files, Juliet `bad`/`good`, CVE pairs) are **nice**: one CWE per file, named sinks, labeled counterparts. Real working trees are not: mixed languages, generated code, unparseable files, framework wrappers, comments, one-hop assignments (`sql = f"…{x}"; cur.execute(sql)`), and no ground-truth labels.
|
||||
|
||||
@@ -12,17 +12,17 @@ Measured honesty on the labeled SAST slice is about **+9% Youden**. Widening reg
|
||||
|
||||
Keep three roles, never collapse them:
|
||||
|
||||
1. **Propose** — pattern rules + optional SARIF. Stage 1. Does not decide worth-fixing.
|
||||
2. **Adjudicate** — tree-sitter IR (multi-language) + taint facts; optional Joern CPG sidecar; Python stdlib parse only as last-resort fallback. Promote, demote, or **unadjudicated**.
|
||||
3. **Budget** — ranker + **mechanism memory** (JSONL + OpenRouter embeddings, hard CWE/language/tag filters) **before** prove. Not a verdict. Not TF-IDF. Not raw file-slice embeddings.
|
||||
4. **Prove** — sandbox only on promoted, reachable, budgeted candidates.
|
||||
1. **Propose**, pattern rules + optional SARIF. Stage 1. Does not decide worth-fixing.
|
||||
2. **Adjudicate**, tree-sitter IR (multi-language) + taint facts; optional Joern CPG sidecar; Python stdlib parse only as last-resort fallback. Promote, demote, or **unadjudicated**.
|
||||
3. **Budget**, ranker + **mechanism memory** (JSONL + OpenRouter embeddings, hard CWE/language/tag filters) **before** prove. It is a budget rather than a verdict; it uses neither TF-IDF nor raw file-slice embeddings.
|
||||
4. **Prove**, sandbox only on promoted, reachable, budgeted candidates.
|
||||
|
||||
The primary in-scope target is a **messy working tree the operator is writing**, not a labeled benchmark. Known-vuln corpora remain calibration scoreboards (smoke 90/10, pair Youden, GitHub silent-on-fix). They do not define success on unlabeled code.
|
||||
|
||||
## Scope
|
||||
|
||||
- Semantic overlay as MAP-stage adjudication; **tree-sitter is the prio engine** (CLI/API, fail-closed). Joern optional. Embeddings tune prove order.
|
||||
- Explicit `unadjudicated` when parse or flow fails — do not treat silence as safe.
|
||||
- Explicit `unadjudicated` when parse or flow fails. Do not treat silence as safe.
|
||||
- Sandbox prove only after promote.
|
||||
- Pair eval scores the overlay on the sast slice; smoke gate stays inventory-only.
|
||||
|
||||
|
||||
@@ -10,10 +10,10 @@
|
||||
|
||||
### Goals
|
||||
|
||||
- Map every current scan module onto propose, adjudicate, budget, or prove — rank/embed is not a verdict.
|
||||
- Map every current scan module onto propose, adjudicate, budget, or prove (rank/embed is not a verdict).
|
||||
- Treat messy unlabeled trees as the default; labeled corpora as calibration.
|
||||
- Fail closed: cannot parse or cannot follow flow → `unadjudicated`, not demote.
|
||||
- **Prioritize the semantic engine now**: tree-sitter IR for in-scope languages; Joern optional CPG sidecar; stdlib Python parse only as last resort.
|
||||
- Prioritize the semantic engine now: tree-sitter IR for in-scope languages; Joern optional CPG sidecar; stdlib Python parse only as last resort.
|
||||
- Tunable embeddings + existing ranker order **prove** candidates.
|
||||
- SAST pair slice scores the overlay; smoke gate stays inventory-only.
|
||||
|
||||
@@ -51,11 +51,11 @@
|
||||
- Preprocess, mapping, rank (`heuristic_rank` 0.5/0.2/0.3), `quick_scan_findings`, complexity map, `ousast index` / json-local vectorstore, tool hunter, sandbox runner, pair eval, reports.
|
||||
- Host binaries when present: `tree-sitter`, `joern` / `joern-parse` (probed, never required for inventory).
|
||||
- Optional embedding model from existing `[embeddings]` config.
|
||||
- Clearwing as an **oracle** for tree-sitter callgraph + taint-pattern tables + mechanism retrieval — not a dependency.
|
||||
- Clearwing as an **oracle** for tree-sitter callgraph + taint-pattern tables + mechanism retrieval, not a dependency.
|
||||
|
||||
### Revalidation Triggers
|
||||
|
||||
- Adding a fourth disposition that implies “safe.”
|
||||
- Adding a fourth disposition that implies "safe."
|
||||
- Demoting when flow is incomplete.
|
||||
- Stamping overlay above `static_corroboration`.
|
||||
- Selecting sandbox candidates from demoted or unadjudicated sets.
|
||||
@@ -84,7 +84,7 @@ Concrete mapping of **what exists today** onto the three roles:
|
||||
| Prove | REGRESS / `deep` | `regress` + sandbox | Recipes on ranked findings | Recipes on **promoted then budgeted** findings |
|
||||
| Calibrate | CI | `gate.py` / `pairs` | Cheat-sheet vs Youden | Inventory vs overlay, unchanged split |
|
||||
|
||||
**Selected pattern.** Propose → adjudicate → **budget** → prove. Clearwing’s tree-sitter callgraph + taint tables + mechanism memory are the oracle, not a library import. Embeddings never demote.
|
||||
**Selected pattern.** Propose → adjudicate → **budget** → prove. Clearwing's tree-sitter callgraph + taint tables + mechanism memory are the oracle, not a library import. Embeddings never demote.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
@@ -161,13 +161,13 @@ src/openultrasast/ruleset/semantic/
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `stages.py` / `cli.py` — overlay in MAP; prove-budget before regress.
|
||||
- `findings.py` — inventory unchanged.
|
||||
- `regress/candidate.py` — `promote` only, then budget order.
|
||||
- `rank.py` / `index.py` / `vectorstore.py` — reuse; do not invent a second store.
|
||||
- `reports.py` — disposition columns.
|
||||
- `pairs.py` — sast slice uses overlay.
|
||||
- `mcp.py` — still no `deep`.
|
||||
- `stages.py` / `cli.py`, overlay in MAP; prove-budget before regress.
|
||||
- `findings.py`, inventory unchanged.
|
||||
- `regress/candidate.py`, `promote` only, then budget order.
|
||||
- `rank.py` / `index.py` / `vectorstore.py`, reuse; do not invent a second store.
|
||||
- `reports.py`, disposition columns.
|
||||
- `pairs.py`, sast slice uses overlay.
|
||||
- `mcp.py`, still no `deep`.
|
||||
|
||||
## System Flows
|
||||
|
||||
@@ -301,11 +301,11 @@ Identity is the *pattern*, not a file slice:
|
||||
| what_made_it_exploitable | why a near-miss became a bug |
|
||||
| source_finding_id, source_repo | provenance only |
|
||||
|
||||
Persistence: append-only JSONL (portable). Recall is **not** Clearwing’s TF-IDF ladder.
|
||||
Persistence: append-only JSONL (portable). Recall is **not** Clearwing's TF-IDF ladder.
|
||||
|
||||
1. **Filter** — language, CWE, tags (exact overlap). This is a hard prefilter, not a ranker.
|
||||
2. **Rank** — OpenRouter `/embeddings` (`OpenRouterEmbeddingClient`, existing `[embeddings].model`) over `summary + what_made_it_exploitable + tags`. Cosine against the promotion’s overlay text. Cache vectors in json-local `mechanisms` namespace, keyed by mechanism id + model id.
|
||||
3. **Degrade** — no key / no model / embed failure → heuristic ranker only, degradation recorded. Never bag-of-words cosine.
|
||||
1. **Filter**, language, CWE, tags (exact overlap). This is a hard prefilter, not a ranker.
|
||||
2. **Rank**, OpenRouter `/embeddings` (`OpenRouterEmbeddingClient`, existing `[embeddings].model`) over `summary + what_made_it_exploitable + tags`. Cosine against the promotion's overlay text. Cache vectors in json-local `mechanisms` namespace, keyed by mechanism id + model id.
|
||||
3. **Degrade**, no key / no model / embed failure → heuristic ranker only, degradation recorded. Never bag-of-words cosine.
|
||||
|
||||
Query for a promotion: embed the overlay rationale (sources, sink, CWE). Result is a **boost to prove order**, never a demote.
|
||||
|
||||
@@ -327,7 +327,7 @@ Repo-code chunks (`ousast index` 80-line slices) stay for hunter/verifier retrie
|
||||
|
||||
- Unit: parse fail → unadjudicated; constant `exec("ls")` → demote; `eval(request.args)` → promote; `sql = f"…{x}"; execute(sql)` → coverage or promote.
|
||||
- Pair: `--slice sast` uses overlay; `juliet-c-cwe134-printf` stays PASS; `owasp-python-deser` stays PASS; Juliet `goodG2B` exec/strcpy must **demote or unadjudicated**, not inventory-leak counted as pair fail for overlay.
|
||||
- Unlabeled: tmp tree with a syntax-error file next to a clean `eval(request.args)` — error file unadjudicated, clean file promoted; no expected.toml required.
|
||||
- Unlabeled: tmp tree with a syntax-error file next to a clean `eval(request.args)`. Error file unadjudicated, clean file promoted; no expected.toml required.
|
||||
- Gate: `python -m openultrasast.gate` still inventory-only.
|
||||
- Deep: candidate list excludes demoted.
|
||||
|
||||
@@ -346,4 +346,4 @@ Repo-code chunks (`ousast index` 80-line slices) stay for hunter/verifier retrie
|
||||
|
||||
- Default `standard` runs overlay when facts load; if facts missing, degrade as Requirement 7.5.
|
||||
- Existing `quick` CI unchanged.
|
||||
- Pair gate local 100% remains inventory-or-overlay-equivalent on fixture counterparts (those files are still “nice”; unlabeled tests are additional).
|
||||
- Pair gate local 100% remains inventory-or-overlay-equivalent on fixture counterparts (those files are still "nice"; unlabeled tests are additional).
|
||||
|
||||
@@ -4,15 +4,15 @@
|
||||
|
||||
## Introduction
|
||||
|
||||
OpenUltraSAST users scan **working trees they are writing**: mixed files, incomplete edits, generated code, framework wrappers, and no labels. Today the harness still *decides* with pattern inventory. The complexity map ranks; the sandbox can run; neither adjudicates “this sink is fed by a source” versus “this sink is hardcoded, sanitized, or unreadable.”
|
||||
OpenUltraSAST users scan **working trees they are writing**: mixed files, incomplete edits, generated code, framework wrappers, and no labels. Today the harness still *decides* with pattern inventory. The complexity map ranks; the sandbox can run; neither adjudicates "this sink is fed by a source" versus "this sink is hardcoded, sanitized, or unreadable."
|
||||
|
||||
This feature splits the scan into three operator-visible roles that must not collapse: **regex proposes, semantics adjudicates, sandbox proves.** Labeled suites (cheat-sheet fixtures, OWASP Benchmark, Juliet, CVE pairs) remain **calibration**. They are not the definition of in-scope code.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: Role split on `quick` / `standard` / `deep`; overlay dispositions; tree-sitter (and optional Joern) as the semantic engine; tunable retrieval/rank **before prove**; `unadjudicated` on messy files; sandbox only after promote; calibration vs unlabeled working trees.
|
||||
- **Out of scope**: Deleting pattern rules; making an LLM the adjudicator; rewriting CWE policy or project-score formula; requiring Joern or an embedding model for `quick`; auto-generated flow facts from the improve loop; treating benchmark Youden as a merge gate that can unblock a smoke-gate breach. Embedding rank is a **budget**, not a fourth verdict.
|
||||
- **Adjacent expectations**: Stage plan, complexity map, Docker sandbox, pair catalog, and 90/10 detection gate from `three-stage-scan` keep their contracts. This spec overlays adjudication on that pipeline; it does not replace stage accounting, sandbox flags, or `LANGUAGE_MANIFESTS`.
|
||||
- In scope: Role split on `quick` / `standard` / `deep`; overlay dispositions; tree-sitter (and optional Joern) as the semantic engine; tunable retrieval/rank **before prove**; `unadjudicated` on messy files; sandbox only after promote; calibration vs unlabeled working trees.
|
||||
- Out of scope: Deleting pattern rules; making an LLM the adjudicator; rewriting CWE policy or project-score formula; requiring Joern or an embedding model for `quick`; auto-generated flow facts from the improve loop; treating benchmark Youden as a merge gate that can unblock a smoke-gate breach. Embedding rank is a **budget**, not a fourth verdict.
|
||||
- Adjacent expectations: Stage plan, complexity map, Docker sandbox, pair catalog, and 90/10 detection gate from `three-stage-scan` keep their contracts. This spec overlays adjudication on that pipeline; it does not replace stage accounting, sandbox flags, or `LANGUAGE_MANIFESTS`.
|
||||
|
||||
## Requirements
|
||||
|
||||
@@ -36,9 +36,9 @@ This feature splits the scan into three operator-visible roles that must not col
|
||||
|
||||
1. When a scanned file is incomplete, generated, minified, or otherwise not interpretable as structured source, the system shall leave matching inventory proposals **unadjudicated** and shall record that reason on the finding.
|
||||
2. When no available semantic engine can follow data from a source to a sink for that file (parse failure, missing grammar, cross-file call, dynamic dispatch, or missing callee), the system shall leave the proposal unadjudicated rather than demote it.
|
||||
3. When OpenUltraSAST scans a working tree with no benchmark manifest and no expected-finding labels, the system shall still emit proposals, dispositions, and (in `deep`) proofs without requiring a counterpart “fixed” tree.
|
||||
3. When OpenUltraSAST scans a working tree with no benchmark manifest and no expected-finding labels, the system shall still emit proposals, dispositions, and (in `deep`) proofs without requiring a counterpart "fixed" tree.
|
||||
4. If the operator reads a report for an unadjudicated proposal, then the system shall present it as unresolved inventory, not as a verified vulnerability and not as a cleared finding.
|
||||
5. The system shall not require a file to resemble an OWASP Benchmark or Juliet test case in order to produce a proposal.
|
||||
5. The system shall not require a file to resemble an OWASP Benchmark or Juliet test case to produce a proposal.
|
||||
|
||||
### Requirement 3: Adjudication promote, demote, coverage
|
||||
|
||||
@@ -46,8 +46,8 @@ This feature splits the scan into three operator-visible roles that must not col
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a proposal’s sink is reachable from an identified untrusted source without an identified sanitizer in the adjudicated region, the system shall **promote** the proposal.
|
||||
2. When a proposal’s sink is fed only by a constant, a sanitizer, or an identified non-attacker source in the adjudicated region, the system shall **demote** the proposal and shall not list it as a promoted finding.
|
||||
1. When a proposal's sink is reachable from an identified untrusted source without an identified sanitizer in the adjudicated region, the system shall **promote** the proposal.
|
||||
2. When a proposal's sink is fed only by a constant, a sanitizer, or an identified non-attacker source in the adjudicated region, the system shall **demote** the proposal and shall not list it as a promoted finding.
|
||||
3. When adjudication finds an untrusted source flowing to a dangerous sink that inventory did not propose, the system shall emit a **coverage** finding at corroboration-or-lower evidence, tagged as overlay-originated.
|
||||
4. When both a source-to-sink path and a sanitizer appear to apply, the system shall not promote unless the sanitizer does not dominate that path.
|
||||
5. The system shall persist promote, demote, unadjudicated, and coverage outcomes in the scan artifacts with the proposal identifier they refer to.
|
||||
@@ -62,7 +62,7 @@ This feature splits the scan into three operator-visible roles that must not col
|
||||
2. When the pair-efficiency evaluation runs on the SAST slice, the system shall score **adjudicated** outcomes (promote on the vulnerable side, demote or silent on the false or good side), not raw inventory hits.
|
||||
3. When overlay changes raise SAST Youden and also cause the stage-1 detection gate to fail, the system shall treat the detection-gate failure as blocking.
|
||||
4. The system shall keep labeled pair fixtures and SAST-slice cases out of the stage-1 cheat-sheet corpus map.
|
||||
5. The system shall not require a labeled counterpart tree in order to scan an operator working tree.
|
||||
5. The system shall not require a labeled counterpart tree to scan an operator working tree.
|
||||
|
||||
### Requirement 5: Evidence and models stay honest
|
||||
|
||||
@@ -74,7 +74,7 @@ This feature splits the scan into three operator-visible roles that must not col
|
||||
2. When the tool hunter emits a finding, the system shall keep that finding at `suspicion` unless an independent corroborating stage has already raised it.
|
||||
3. If an overlay or hunter result claims an evidence level above corroboration without a sandbox or verifier step that earned it, then the system shall reject that claim.
|
||||
4. The system shall not use a language model as the adjudication engine that decides promote or demote.
|
||||
5. Where CWE policy applies, the system shall resolve severity from the central CWE policy, not from the overlay or the proposing rule’s private severity string.
|
||||
5. Where CWE policy applies, the system shall resolve severity from the central CWE policy, not from the overlay or the proposing rule's private severity string.
|
||||
|
||||
### Requirement 6: Sandbox proves only promoted work
|
||||
|
||||
@@ -106,7 +106,7 @@ This feature splits the scan into three operator-visible roles that must not col
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a tree-sitter grammar for the file’s language is available, the system shall use that parse as the primary intra-file IR for adjudication.
|
||||
1. When a tree-sitter grammar for the file's language is available, the system shall use that parse as the primary intra-file IR for adjudication.
|
||||
2. When tree-sitter is unavailable for that language, the system shall try a stdlib parse only for Python and shall otherwise leave the proposal unadjudicated with language_unsupported or parse_failed.
|
||||
3. Where Joern is installed and the language is one Joern can export, the system shall be allowed to attach Joern flow facts as **additional** corroboration; Joern absence shall not fail `quick` or `standard`.
|
||||
4. If Joern and tree-sitter disagree or Joern returns incomplete paths, then the system shall not demote on that disagreement; it shall promote only on a complete dominating path or leave the item unadjudicated.
|
||||
@@ -124,6 +124,6 @@ This feature splits the scan into three operator-visible roles that must not col
|
||||
3. The system shall not demote or drop a promotion solely because mechanism similarity is low.
|
||||
4. When the embedding index is missing or stale, the system shall rebuild it from the mechanism log via OpenRouter embeddings; if that rebuild fails, the system shall record the degradation and continue on heuristic order.
|
||||
5. The system shall keep hard filters operator-tunable (language, CWE, tags) without changing CWE severity policy.
|
||||
6. When a finding is sandbox-proven triggerable (or otherwise verified at corroboration-or-higher with a prove step), the system shall append an abstract mechanism record to the mechanism log: summary, CWE, language, tags, what made it exploitable — without file:line as the identity.
|
||||
6. When a finding is sandbox-proven triggerable (or otherwise verified at corroboration-or-higher with a prove step), the system shall append an abstract mechanism record to the mechanism log: summary, CWE, language, tags, what made it exploitable, without file:line as the identity.
|
||||
7. The system shall treat the append-only mechanism log as the source of truth; any vector index shall be a rebuildable cache of that log.
|
||||
8. The system shall not persist mechanisms from inventory-only, demoted, or unadjudicated proposals.
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
## Summary
|
||||
|
||||
- **Feature**: `propose-adjudicate-prove`
|
||||
- **Discovery Scope**: Extension of `three-stage-scan` (MAP grows an overlay; STATIC and REGRESS stay)
|
||||
- **Key Findings**:
|
||||
- Feature: `propose-adjudicate-prove`
|
||||
- Discovery Scope: Extension of `three-stage-scan` (MAP grows an overlay; STATIC and REGRESS stay)
|
||||
- Key Findings:
|
||||
- Inventory already decides; MAP ranks; REGRESS can run. Nothing adjudicates flow.
|
||||
- Labeled SAST slice Youden is about +9%. Leaks are OWASP-false / Juliet `goodG2B` that still contain the sink. Misses are one-hop assignments and sink aliases.
|
||||
- Clearwing already does the split we want: tree-sitter callgraph + pattern taint tables, then ranker (surface/influence/reachability 0.5/0.2/0.3), then mechanism-memory embeddings. OpenUltraSAST already copied the rank formula and has `json-local` vectors; they are not wired as prove-budget.
|
||||
@@ -15,55 +15,55 @@
|
||||
|
||||
### Labeled vs unlabeled code
|
||||
|
||||
- **Context**: Operator: known-vuln corpora are “nice”; written code will not be.
|
||||
- **Sources Consulted**: In-tree `LANGUAGE_MANIFESTS`, `benchmarks/pairs/`, OWASP Benchmark true/false files, Juliet `bad`/`goodG2B`/`goodB2G`, CWE-Bench-Java catalog.
|
||||
- **Findings**:
|
||||
- Context: Operator: known-vuln corpora are "nice"; written code will not be.
|
||||
- Sources Consulted: In-tree `LANGUAGE_MANIFESTS`, `benchmarks/pairs/`, OWASP Benchmark true/false files, Juliet `bad`/`goodG2B`/`goodB2G`, CWE-Bench-Java catalog.
|
||||
- Findings:
|
||||
- Cheat-sheets: one CWE, named sink, often same-line source.
|
||||
- OWASP: true and false files both often contain `exec` / `eval`; the difference is source constancy or infeasible branches.
|
||||
- Juliet `goodG2B`: hardcoded source, **same sink**.
|
||||
- Juliet `goodB2G`: same source, **sanitized sink** (`printf("%s")`).
|
||||
- Working trees: syntax errors, generated bundles, wrappers (`helpers.db_sqlite.execute`), no labels.
|
||||
- **Implications**: Overlay scored on sast slice; unlabeled scans must not pretend demote. Unadjudicated is a first-class disposition.
|
||||
- Implications: Overlay scored on sast slice; unlabeled scans must not pretend demote. Unadjudicated is a first-class disposition.
|
||||
|
||||
### Engine choice
|
||||
|
||||
- **Context**: Operator prefers tree-sitter + Joern look, embeddings before prove, still v1-focused.
|
||||
- **Sources Consulted**: `clearwing/sourcehunt/callgraph.py`, `taint.py`, `ranker.py`, `mechanism_memory.py`; `openultrasast/rank.py`, `vectorstore.py`, `index.py`.
|
||||
- **Findings**:
|
||||
- Context: Operator prefers tree-sitter + Joern look, embeddings before prove, still v1-focused.
|
||||
- Sources Consulted: `clearwing/sourcehunt/callgraph.py`, `taint.py`, `ranker.py`, `mechanism_memory.py`; `openultrasast/rank.py`, `vectorstore.py`, `index.py`.
|
||||
- Findings:
|
||||
- Clearwing tree-sitter is optional: missing grammar disables callgraph, does not crash.
|
||||
- Clearwing taint is intra-procedural, pattern tables for sources/sinks — same shape as our fact TOML.
|
||||
- Clearwing taint is intra-procedural, pattern tables for sources/sinks, same shape as our fact TOML.
|
||||
- Clearwing ranker fills reachability from the callgraph; taint_paths boost surface. Embeddings retrieve *mechanisms*, they do not decide taint.
|
||||
- Our `heuristic_rank` already uses surface·0.5 + influence·0.2 + reachability·0.3. Vector store bakeoff selected `json-local`. `VALID_NAMESPACES` already includes `mechanisms` but nothing writes Clearwing-style records into it.
|
||||
|
||||
### Clearwing vector-store reference (real-world bugs)
|
||||
|
||||
- **Context**: Operator: Clearwing’s store is effective on real bugs, not textbook files.
|
||||
- **Sources Consulted**: `clearwing/sourcehunt/mechanism_memory.py`, `data/memory/semantic_memory.py`, `docs/architecture.md` steps 7–8; `openultrasast/index.py` namespaces.
|
||||
- **Findings**:
|
||||
- Store **mechanisms**, not findings and not 80-line file windows. Example: “length field trusted before alloc; 16-bit value widened to size_t.”
|
||||
- Context: Operator: Clearwing's store is effective on real bugs, not textbook files.
|
||||
- Sources Consulted: `clearwing/sourcehunt/mechanism_memory.py`, `data/memory/semantic_memory.py`, `docs/architecture.md` steps 7–8; `openultrasast/index.py` namespaces.
|
||||
- Findings:
|
||||
- Store **mechanisms**, not findings and not 80-line file windows. Example: "length field trusted before alloc; 16-bit value widened to size_t."
|
||||
- JSONL is the source of truth (`~/.clearwing/sourcehunt/mechanisms.jsonl`). Chroma/sqlite embeddings are ephemeral caches rebuilt from JSONL.
|
||||
- Write path is **verified findings only** (after adversarial verify / crash). Suspicion never seeds the store.
|
||||
- Clearwing’s JSONL + verified-write + metadata filters are the reference. Its TF-IDF default is **not**: bag-of-words will not match paraphrased real bugs (`length field trusted before alloc` vs `user-controlled size_t in malloc`).
|
||||
- Clearwing's JSONL + verified-write + metadata filters are the reference. Its TF-IDF default is **not**: bag-of-words will not match paraphrased real bugs (`length field trusted before alloc` vs `user-controlled size_t in malloc`).
|
||||
- OpenUltraSAST already has `OpenRouterEmbeddingClient` and json-local cosine. Use that as the only semantic ranker; CWE/language/tags are hard filters. No TF-IDF, no Chroma, no local MiniLM.
|
||||
- Query is language + file tags + optional free text; same-language boost +0.5. Top-N injects hunter context and variant-grep seeds — it does not decide taint.
|
||||
- Query is language + file tags + optional free text; same-language boost +0.5. Top-N injects hunter context and variant-grep seeds. It does not decide taint.
|
||||
- Variant loop then greps the *messy* tree for structural matches of that mechanism. That is how it leaves Juliet behind.
|
||||
- **Implications**: Copy JSONL + verified-write + metadata filters. Rank with OpenRouter embeddings only. Do not copy TF-IDF. Do not embed 80-line file windows as the prove store. Seed from sandbox-proven mechanisms, not OWASP `BenchmarkTest00007`.
|
||||
- Implications: Copy JSONL + verified-write + metadata filters. Rank with OpenRouter embeddings only. Do not copy TF-IDF. Do not embed 80-line file windows as the prove store. Seed from sandbox-proven mechanisms, not OWASP `BenchmarkTest00007`.
|
||||
- Joern CLI (`joern-parse` / scripts) is the only practical CPG without a Scala runtime in-process.
|
||||
- **Implications**: Probe tree-sitter like Docker. Joern sidecar cannot demote. Embeddings order prove only.
|
||||
- Implications: Probe tree-sitter like Docker. Joern sidecar cannot demote. Embeddings order prove only.
|
||||
|
||||
### Decision: Python intra-file first
|
||||
|
||||
- **Context**: Zero-dep core vs multi-language trees.
|
||||
- **Selected**: **Superseded.** Tree-sitter is the prio engine; stdlib `ast` is Python last resort; `quick` stays zero-dep.
|
||||
- **Rationale**: Operator working trees are mixed-language; Python-only adjudication would leave JS/C/Java as perpetual unadjudicated.
|
||||
- **Trade-offs**: `standard` needs a grammar or CLI on PATH; missing tools → unadjudicated, not skip the scan.
|
||||
- Context: Zero-dep core vs multi-language trees.
|
||||
- Selected: **Superseded.** Tree-sitter is the prio engine; stdlib `ast` is Python last resort; `quick` stays zero-dep.
|
||||
- Rationale: Operator working trees are mixed-language; Python-only adjudication would leave JS/C/Java as perpetual unadjudicated.
|
||||
- Trade-offs: `standard` needs a grammar or CLI on PATH; missing tools → unadjudicated, not skip the scan.
|
||||
|
||||
### Evidence ladder
|
||||
|
||||
- **Context**: Three-stage-scan forbids hunter JSON as corroboration.
|
||||
- **Sources Consulted**: `verification.EvidenceLevel`, `tool_hunter` suspicion rule, `worth_fixing` gate.
|
||||
- **Findings**: Overlay can justify promote at `static_corroboration`. It cannot skip to crash or exploit. Worth-fixing stays sandbox + reachability.
|
||||
- **Implications**: Disposition is not an evidence rung. Do not add a seventh ladder value.
|
||||
- Context: Three-stage-scan forbids hunter JSON as corroboration.
|
||||
- Sources Consulted: `verification.EvidenceLevel`, `tool_hunter` suspicion rule, `worth_fixing` gate.
|
||||
- Findings: Overlay can justify promote at `static_corroboration`. It cannot skip to crash or exploit. Worth-fixing stays sandbox + reachability.
|
||||
- Implications: Disposition is not an evidence rung. Do not add a seventh ladder value.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
@@ -78,28 +78,28 @@
|
||||
|
||||
### Decision: Unadjudicated is not demote
|
||||
|
||||
- **Context**: Messy files and cross-file flow.
|
||||
- **Alternatives**: Silent drop; treat as safe; force demote.
|
||||
- **Selected**: Explicit `unadjudicated` with reason.
|
||||
- **Rationale**: Operator writing code must not read silence as “clean.”
|
||||
- **Trade-offs**: Reports stay noisy until languages and inter-file land.
|
||||
- **Follow-up**: Count unadjudicated rate on unlabeled trees as an honesty metric.
|
||||
- Context: Messy files and cross-file flow.
|
||||
- Alternatives: Silent drop; treat as safe; force demote.
|
||||
- Selected: Explicit `unadjudicated` with reason.
|
||||
- Rationale: Operator writing code must not read silence as "clean."
|
||||
- Trade-offs: Reports stay noisy until languages and inter-file land.
|
||||
- Follow-up: Count unadjudicated rate on unlabeled trees as an honesty metric.
|
||||
|
||||
### Decision: Score overlay on sast slice, inventory on smoke gate
|
||||
|
||||
- **Context**: +9% Youden vs 93% cheat-sheet recall.
|
||||
- **Selected**: Two gates stay separate; overlay Youden cannot unblock smoke failure.
|
||||
- **Rationale**: Same as detection-gate vs score in the improve loop.
|
||||
- Context: +9% Youden vs 93% cheat-sheet recall.
|
||||
- Selected: Two gates stay separate; overlay Youden cannot unblock smoke failure.
|
||||
- Rationale: Same as detection-gate vs score in the improve loop.
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- False demote on real bugs (infeasible-branch OWASP tricks, wrappers) — never demote when flow is incomplete; only demote when the engine *has* a dominating constant or sanitizer.
|
||||
- Youden chase via extra sinks — each coverage rule needs a false/good counterpart that must stay demoted or silent.
|
||||
- Parse failures on operator WIP — unadjudicated, `quick` still works.
|
||||
- False demote on real bugs (infeasible-branch OWASP tricks, wrappers). Never demote when flow is incomplete; only demote when the engine *has* a dominating constant or sanitizer.
|
||||
- Youden chase via extra sinks. Each coverage rule needs a false/good counterpart that must stay demoted or silent.
|
||||
- Parse failures on operator WIP, unadjudicated, `quick` still works.
|
||||
|
||||
## References
|
||||
|
||||
- `.kiro/specs/three-stage-scan/design.md` — stage plan, sandbox, hunter suspicion
|
||||
- `benchmarks/pairs/sast/README.md` — OWASP/Juliet slice
|
||||
- `.kiro/specs/three-stage-scan/design.md`, stage plan, sandbox, hunter suspicion
|
||||
- `benchmarks/pairs/sast/README.md`, OWASP/Juliet slice
|
||||
- NIST Juliet `goodG2B` vs `goodB2G` templates
|
||||
- OWASP Benchmark expectedresults true/false protocol
|
||||
|
||||
@@ -24,8 +24,8 @@ Build a name-resolved call graph from the `FunctionIR` records tree-sitter alrea
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: call graph, summaries, path records, reachability attachment, rank and map integration, hunter on paths, hunter-to-candidate path, fact oracle with cache, `facts` lever with validator, fix-point ranking and report section, measurement on split-sink, vibe-py, vfc slices, offline tests.
|
||||
- **Out**: guard dominance semantics (`guard-dominance-regime` consumes path records), CST walker fixes (`overlay-ir-completeness`), a whole-program points-to or symbolic engine, concolic execution, executing target code outside the sandbox, new dispositions, LLM as parser, any merge gate change.
|
||||
- In: call graph, summaries, path records, reachability attachment, rank and map integration, hunter on paths, hunter-to-candidate path, fact oracle with cache, `facts` lever with validator, fix-point ranking and report section, measurement on split-sink, vibe-py, vfc slices, offline tests.
|
||||
- Out: guard dominance semantics (`guard-dominance-regime` consumes path records), CST walker fixes (`overlay-ir-completeness`), a whole-program points-to or symbolic engine, concolic execution, executing target code outside the sandbox, new dispositions, LLM as parser, any merge gate change.
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
@@ -42,14 +42,14 @@ Build a name-resolved call graph from the `FunctionIR` records tree-sitter alrea
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `FileIR`/`FunctionIR`, `analyze_entry_points`, `analyze_ir`, `load_facts`, `Hotspot`, `run_tool_hunter`, `select_candidates`, `run_improvement`, pair holdout metrics per profile.
|
||||
- **Downstream**: `guard-dominance-regime` attaches guards to path records; later web-authorization detection is a path whose hops include no auth guard; reports and MCP read fix points.
|
||||
- Upstream: `FileIR`/`FunctionIR`, `analyze_entry_points`, `analyze_ir`, `load_facts`, `Hotspot`, `run_tool_hunter`, `select_candidates`, `run_improvement`, pair holdout metrics per profile.
|
||||
- Downstream: `guard-dominance-regime` attaches guards to path records; later web-authorization detection is a path whose hops include no auth guard; reports and MCP read fix points.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: `three-stage-scan` (hunter tools, candidate selection, worth-fixing input), `propose-adjudicate-prove` (facts consumed unchanged; one new fact source), `harnessx-self-improving-rulesets` (one new lever under the same gate).
|
||||
- **Depends**: `pair-corpus-honesty` for per-profile holdout metrics and the hunter scorer.
|
||||
- **Adjacent**: `overlay-ir-completeness` improves the records this spec walks; neither blocks the other.
|
||||
- Extends: `three-stage-scan` (hunter tools, candidate selection, worth-fixing input), `propose-adjudicate-prove` (facts consumed unchanged; one new fact source), `harnessx-self-improving-rulesets` (one new lever under the same gate).
|
||||
- Depends: `pair-corpus-honesty` for per-profile holdout metrics and the hunter scorer.
|
||||
- Adjacent: `overlay-ir-completeness` improves the records this spec walks; neither blocks the other.
|
||||
|
||||
## Constraints
|
||||
|
||||
|
||||
@@ -146,10 +146,10 @@ tests/
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `semantic/cst.py`, `semantic/ir.py` — collect import statements into `FileIR.imports`; no other IR change.
|
||||
- `mapping.py` — new function; `EntryPointRecord` unchanged.
|
||||
- `stage_processors.py` — MAP stage runs graph, summaries, paths before overlay; overlay unchanged.
|
||||
- `map_gate.py` — path assertion per language family.
|
||||
- `semantic/cst.py`, `semantic/ir.py`, collect import statements into `FileIR.imports`; no other IR change.
|
||||
- `mapping.py`, new function; `EntryPointRecord` unchanged.
|
||||
- `stage_processors.py`, MAP stage runs graph, summaries, paths before overlay; overlay unchanged.
|
||||
- `map_gate.py`, path assertion per language family.
|
||||
|
||||
## System Flows
|
||||
|
||||
|
||||
@@ -8,9 +8,9 @@ OpenUltraSAST cannot yet steer effort where risk is reduced because it has no mo
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: call graph, taint summaries, path records, reachability attachment to findings, rank, complexity map and worth-fixing; hunter path context; hunter findings as sandbox candidates; fact oracle and candidate-fact cache; `facts` improve lever; fix-point ranking and report section; offline tests and measurement on existing slices.
|
||||
- **Out of scope**: guard semantics; CST walker fixes; whole-program alias analysis; symbolic or concolic execution; new dispositions or evidence rungs; LLM as parser; any merge gate change; profile-specific fact tables beyond the gate defined by `pair-corpus-honesty`.
|
||||
- **Adjacent expectations**: `FileIR` shape is frozen except additive fields; `load_facts` merges candidate facts below hand-written facts; the improve loop's existing accept gate and per-profile holdout clause apply unchanged to the new lever; sandbox safety checks apply to hunter snippets as today.
|
||||
- In scope: call graph, taint summaries, path records, reachability attachment to findings, rank, complexity map and worth-fixing; hunter path context; hunter findings as sandbox candidates; fact oracle and candidate-fact cache; `facts` improve lever; fix-point ranking and report section; offline tests and measurement on existing slices.
|
||||
- Out of scope: guard semantics; CST walker fixes; whole-program alias analysis; symbolic or concolic execution; new dispositions or evidence rungs; LLM as parser; any merge gate change; profile-specific fact tables beyond the gate defined by `pair-corpus-honesty`.
|
||||
- Adjacent expectations: `FileIR` shape is frozen except additive fields; `load_facts` merges candidate facts below hand-written facts; the improve loop's existing accept gate and per-profile holdout clause apply unchanged to the new lever; sandbox safety checks apply to hunter snippets as today.
|
||||
|
||||
## Requirements
|
||||
|
||||
@@ -79,7 +79,7 @@ OpenUltraSAST cannot yet steer effort where risk is reduced because it has no mo
|
||||
|
||||
### Requirement 6: Facts as an improve-loop lever
|
||||
|
||||
**Objective:** As a maintainer, I want the improve loop to be able to accept or reject facts, so that the evolutionary machinery can move detection and not only rule status.
|
||||
**Objective:** As a maintainer, I want the improve loop to be able to accept or reject facts, so that the evolutionary machinery can move both detection and rule status.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
## Summary
|
||||
|
||||
- **Feature**: `reachability-flow-model`
|
||||
- **Discovery Scope**: Complex Integration across semantic, mapping, rank, complexity, hunter, regress, improve
|
||||
- **Key Findings**:
|
||||
- Feature: `reachability-flow-model`
|
||||
- Discovery Scope: Complex Integration across semantic, mapping, rank, complexity, hunter, regress, improve
|
||||
- Key Findings:
|
||||
- `FunctionIR` already carries every call site per function for Python, JS, TS, C, and Java when the extra is present. A name-resolved call graph is a few hundred lines over data that exists.
|
||||
- Reachability today is "line inside an entry-point handler" (`mapping.attach_reachability_hints`); `rank` bumps an integer from tags and hints; `worth_fixing` needs `reachable` or `inferred-file-surface`. No call edge exists anywhere.
|
||||
- Sandbox candidates come only from hotspots filtered to overlay-promoted inventory ids (`cli.py:404`, `prove_filter.filter_promoted_hotspots`); a hunter snippet is used only for an already promoted hotspot (`candidate.py:199`). The LLM plane cannot create a candidate.
|
||||
@@ -18,38 +18,38 @@
|
||||
|
||||
### Where a call graph can come from
|
||||
|
||||
- **Context**: Avoid a new parser or a whole-program engine.
|
||||
- **Sources Consulted**: `semantic/ir.py`, `semantic/cst.py`, `mapping.py`, `preprocess.py`, `hunter_tools.find_refs`.
|
||||
- **Findings**: `FileIR.functions[*].calls[*].name` is the callee text. Python stdlib parse yields the same records. Imports are not in the IR; the walker can add an `imports` tuple per file cheaply. `find_refs` is a word-boundary regex over files and is the only cross-file lookup today.
|
||||
- **Implications**: `semantic/callgraph.py` consumes `FileIR` plus an additive `imports` field. Resolution order: same-file definition, imported module definition, any definition with that name (ambiguous edges). Method calls resolve by trailing name. C resolves by bare name across the tree.
|
||||
- Context: Avoid a new parser or a whole-program engine.
|
||||
- Sources Consulted: `semantic/ir.py`, `semantic/cst.py`, `mapping.py`, `preprocess.py`, `hunter_tools.find_refs`.
|
||||
- Findings: `FileIR.functions[*].calls[*].name` is the callee text. Python stdlib parse yields the same records. Imports are not in the IR; the walker can add an `imports` tuple per file cheaply. `find_refs` is a word-boundary regex over files and is the only cross-file lookup today.
|
||||
- Implications: `semantic/callgraph.py` consumes `FileIR` plus an additive `imports` field. Resolution order: same-file definition, imported module definition, any definition with that name (ambiguous edges). Method calls resolve by trailing name. C resolves by bare name across the tree.
|
||||
|
||||
### Summaries versus whole-program taint
|
||||
|
||||
- **Context**: `propose-adjudicate-prove` deferred inter-file taint as too large and risky for false demotes.
|
||||
- **Sources Consulted**: `semantic/taint.py` (`_analyze_function` seeds parameters as tainted, tracks names, emits `TaintPath` and `Demotion`), overlay invariants.
|
||||
- **Findings**: Because parameters are already seeded tainted, a per-function run already answers "which parameter reaches which sink." Adding "which parameter reaches the return" is a small extension. Composition is then a graph walk with a depth bound. Nothing needs alias analysis to produce a useful path; anything unresolved stays `flow_incomplete`.
|
||||
- **Implications**: Summaries are a pure function of `FunctionIR` plus facts, cacheable by file hash. Composition never demotes; it only adds paths or marks incompleteness.
|
||||
- Context: `propose-adjudicate-prove` deferred inter-file taint as too large and risky for false demotes.
|
||||
- Sources Consulted: `semantic/taint.py` (`_analyze_function` seeds parameters as tainted, tracks names, emits `TaintPath` and `Demotion`), overlay invariants.
|
||||
- Findings: Because parameters are already seeded tainted, a per-function run already answers "which parameter reaches which sink." Adding "which parameter reaches the return" is a small extension. Composition is then a graph walk with a depth bound. Nothing needs alias analysis to produce a useful path; anything unresolved stays `flow_incomplete`.
|
||||
- Implications: Summaries are a pure function of `FunctionIR` plus facts, cacheable by file hash. Composition never demotes; it only adds paths or marks incompleteness.
|
||||
|
||||
### Hunter to candidate
|
||||
|
||||
- **Context**: The LLM plane is the recall lever the field data supports, but it is walled off from prove.
|
||||
- **Sources Consulted**: `tool_hunter.py`, `regress/candidate.py`, `three-stage-scan` design (hunter snippets run only in the sandbox after safety check), `verification.py`.
|
||||
- **Findings**: The safety check and sandbox are origin-agnostic. The wall is `filter_promoted_hotspots` and the candidate selector's reliance on inventory ids. A hunter finding with a locatable sink line can be given a synthetic hotspot the same way `_hotspot_from_forced_finding` already does for severity-5 inventory.
|
||||
- **Implications**: Add eligibility for hunter findings that name a sink line resolvable to a path record or an overlay sink. Evidence stays suspicion until sandbox trigger or path agreement.
|
||||
- Context: The LLM plane is the recall lever the field data supports, but it is walled off from prove.
|
||||
- Sources Consulted: `tool_hunter.py`, `regress/candidate.py`, `three-stage-scan` design (hunter snippets run only in the sandbox after safety check), `verification.py`.
|
||||
- Findings: The safety check and sandbox are origin-agnostic. The wall is `filter_promoted_hotspots` and the candidate selector's reliance on inventory ids. A hunter finding with a locatable sink line can be given a synthetic hotspot the same way `_hotspot_from_forced_finding` already does for severity-5 inventory.
|
||||
- Implications: Add eligibility for hunter findings that name a sink line resolvable to a path record or an overlay sink. Evidence stays suspicion until sandbox trigger or path agreement.
|
||||
|
||||
### Facts from a model, safely
|
||||
|
||||
- **Context**: The engine lacks callee knowledge: nullable returns (`tls1_lookup_sigalg`), bounded arguments, transformer set/get (`ConfigParser`), macro out-params (`n2s`).
|
||||
- **Sources Consulted**: `semantic/facts.py` loader, `improve/validator.py`, `improve/journal.py`, `calibration.py` ledger patterns, `redaction.py`.
|
||||
- **Findings**: Facts are versioned TOML loaded by `load_facts`. The loader can merge a second directory at lower precedence. The improve loop has a strict validator and byte-for-byte revert. Redaction exists for prompts.
|
||||
- **Implications**: Model answers are candidate facts in `.openultrasast/calibration/candidate_facts.toml`; accepted facts in `accepted_facts.toml`; loader precedence hand-written > accepted > none. The `facts` lever moves rows between candidate and accepted under the same gate. No pattern text is ever model-authored; question kinds are closed.
|
||||
- Context: The engine lacks callee knowledge: nullable returns (`tls1_lookup_sigalg`), bounded arguments, transformer set/get (`ConfigParser`), macro out-params (`n2s`).
|
||||
- Sources Consulted: `semantic/facts.py` loader, `improve/validator.py`, `improve/journal.py`, `calibration.py` ledger patterns, `redaction.py`.
|
||||
- Findings: Facts are versioned TOML loaded by `load_facts`. The loader can merge a second directory at lower precedence. The improve loop has a strict validator and byte-for-byte revert. Redaction exists for prompts.
|
||||
- Implications: Model answers are candidate facts in `.openultrasast/calibration/candidate_facts.toml`; accepted facts in `accepted_facts.toml`; loader precedence hand-written > accepted > none. The `facts` lever moves rows between candidate and accepted under the same gate. No pattern text is ever model-authored; question kinds are closed.
|
||||
|
||||
### Fix points
|
||||
|
||||
- **Context**: The principle is risk reduced per fix, not findings per file.
|
||||
- **Sources Consulted**: `complexity/map.py`, `complexity/ledger.py`, `regress/verdict.py`, `scoring/project_score.py`.
|
||||
- **Findings**: The path graph is small (functions, not statements). Dominators over a DAG of entry-to-sink paths are cheap. Severity is available per CWE from policy.
|
||||
- **Implications**: `risk/fixpoints.py` computes dominator sets per sink path and aggregates risk removed. It is a report and artifact, not a gate; it cannot hide severity-5 reachable findings.
|
||||
- Context: The principle is risk reduced per fix, not findings per file.
|
||||
- Sources Consulted: `complexity/map.py`, `complexity/ledger.py`, `regress/verdict.py`, `scoring/project_score.py`.
|
||||
- Findings: The path graph is small (functions, not statements). Dominators over a DAG of entry-to-sink paths are cheap. Severity is available per CWE from policy.
|
||||
- Implications: `risk/fixpoints.py` computes dominator sets per sink path and aggregates risk removed. It is a report and artifact, not a gate; it cannot hide severity-5 reachable findings.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
@@ -65,49 +65,49 @@
|
||||
|
||||
### Decision: Over-approximate edges, never infer safety
|
||||
|
||||
- **Context**: Missing edges silently hide risk; false edges cost hunter budget.
|
||||
- **Selected Approach**: Ambiguous names produce edges to every definition, flagged. Unresolved calls produce an unresolved edge. Distance is infinite only within the depth bound and is reported as `unknown`.
|
||||
- **Rationale**: Consistent with the fail-closed rule in the overlay.
|
||||
- **Trade-offs**: More hunter work on large trees; bounded by existing hotspot caps.
|
||||
- Context: Missing edges silently hide risk; false edges cost hunter budget.
|
||||
- Selected Approach: Ambiguous names produce edges to every definition, flagged. Unresolved calls produce an unresolved edge. Distance is infinite only within the depth bound and is reported as `unknown`.
|
||||
- Rationale: Consistent with the fail-closed rule in the overlay.
|
||||
- Trade-offs: More hunter work on large trees; bounded by existing hotspot caps.
|
||||
|
||||
### Decision: Summaries are pure and cached
|
||||
|
||||
- **Context**: Cost on large repos.
|
||||
- **Selected Approach**: Summary keyed by file hash plus facts version; recomputed only when either changes.
|
||||
- **Trade-offs**: Cache invalidation on facts version bump; acceptable.
|
||||
- Context: Cost on large repos.
|
||||
- Selected Approach: Summary keyed by file hash plus facts version; recomputed only when either changes.
|
||||
- Trade-offs: Cache invalidation on facts version bump; acceptable.
|
||||
|
||||
### Decision: Evidence rises on checks, not on origin
|
||||
|
||||
- **Context**: Regex stamped corroboration, hunter stamped suspicion.
|
||||
- **Selected Approach**: This spec does not change existing stamps. It adds the checks that raise hunter evidence: composed path agreement or sandbox trigger. A follow-up may lower regex-only hits to inventory; recorded as a revalidation trigger for `propose-adjudicate-prove`.
|
||||
- **Rationale**: Keep this spec additive; changing existing stamps changes gate baselines.
|
||||
- Context: Regex stamped corroboration, hunter stamped suspicion.
|
||||
- Selected Approach: This spec does not change existing stamps. It adds the checks that raise hunter evidence: composed path agreement or sandbox trigger. A follow-up may lower regex-only hits to inventory; recorded as a revalidation trigger for `propose-adjudicate-prove`.
|
||||
- Rationale: Keep this spec additive; changing existing stamps changes gate baselines.
|
||||
|
||||
### Decision: Closed question kinds, gated acceptance
|
||||
|
||||
- **Context**: Auto-facts were excluded for good reason: unbounded model-authored patterns.
|
||||
- **Selected Approach**: Five question kinds, structured answers, candidate cache with provenance, acceptance only via the `facts` lever under the per-profile holdout gate.
|
||||
- **Trade-offs**: Slower than trusting answers directly; the gate is the point.
|
||||
- Context: Auto-facts were excluded for good reason: unbounded model-authored patterns.
|
||||
- Selected Approach: Five question kinds, structured answers, candidate cache with provenance, acceptance only via the `facts` lever under the per-profile holdout gate.
|
||||
- Trade-offs: Slower than trusting answers directly; the gate is the point.
|
||||
|
||||
### Decision: Fix points are dominators, reported not gated
|
||||
|
||||
- **Context**: Do not add a seventh evidence rung or a new CI condition.
|
||||
- **Selected Approach**: Compute and report; the severity-5 reachable invariant from `three-stage-scan` stays.
|
||||
- Context: Do not add a seventh evidence rung or a new CI condition.
|
||||
- Selected Approach: Compute and report; the severity-5 reachable invariant from `three-stage-scan` stays.
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- Dynamic dispatch and callbacks in JS and Python produce unresolved edges — report unresolved counts; hunter `find_refs` remains available as a fallback.
|
||||
- Graph blow-up on large trees — depth and fan-out bounds in config; hotspot caps unchanged.
|
||||
- Model answers wrong — never used unless accepted by the gate; provenance recorded; hand facts win.
|
||||
- Hunter candidates increase sandbox load — existing `max_candidates` cap; hunter candidates ranked after overlay promotions unless a path agrees.
|
||||
- Fix-point estimates misread as guarantees — label as estimates; show witness paths.
|
||||
- Dynamic dispatch and callbacks in JS and Python produce unresolved edges. Report unresolved counts; hunter `find_refs` remains available as a fallback.
|
||||
- Graph blow-up on large trees, depth and fan-out bounds in config; hotspot caps unchanged.
|
||||
- Model answers can be wrong. They are never used unless accepted by the gate; provenance recorded; hand facts win.
|
||||
- Hunter candidates increase sandbox load. The existing `max_candidates` cap applies; hunter candidates ranked after overlay promotions unless a path agrees.
|
||||
- Fix-point estimates misread as guarantees. Label as estimates; show witness paths.
|
||||
|
||||
## References
|
||||
|
||||
- Clearwing sourcehunt callgraph, taint, ranker (reference architecture; not a dependency)
|
||||
- [RealVuln](https://arxiv.org/abs/2604.13764) — rule-based versus LLM scanners on real Python web code
|
||||
- [CVE-Bench](https://arxiv.org/abs/2503.17332) — agent exploitation ceiling on real web CVEs
|
||||
- `.kiro/specs/three-stage-scan/design.md` — hunter tools, candidate selection, worth-fixing
|
||||
- `.kiro/specs/propose-adjudicate-prove/design.md` — facts as data, dispositions, evidence honesty
|
||||
- `.kiro/specs/harnessx-self-improving-rulesets/design.md` — bounded levers, accept gate, revert
|
||||
- `.kiro/specs/pair-corpus-honesty/` — per-profile holdout gate, hunter scorer
|
||||
- [RealVuln](https://arxiv.org/abs/2604.13764), rule-based versus LLM scanners on real Python web code
|
||||
- [CVE-Bench](https://arxiv.org/abs/2503.17332), agent exploitation ceiling on real web CVEs
|
||||
- `.kiro/specs/three-stage-scan/design.md`, hunter tools, candidate selection, worth-fixing
|
||||
- `.kiro/specs/propose-adjudicate-prove/design.md`, facts as data, dispositions, evidence honesty
|
||||
- `.kiro/specs/harnessx-self-improving-rulesets/design.md`, bounded levers, accept gate, revert
|
||||
- `.kiro/specs/pair-corpus-honesty/`, per-profile holdout gate, hunter scorer
|
||||
- Memory note `overlay-gaps-diagnosis-2026-09`
|
||||
|
||||
@@ -2,11 +2,11 @@
|
||||
|
||||
## Problem
|
||||
|
||||
Operators care whether the harness fires on a real buggy snapshot and stays quiet after the patch — Firefox, Chromium, OpenSSL — not whether OWASP BenchmarkTest00074 is a TP. The labeled SAST slice is 11 textbook files. Overlay Youden there is **worse** than inventory (+9.09% → 0%). We cannot tell if the tool got better on messy product code.
|
||||
Operators care whether the harness fires on a real buggy snapshot and stays quiet after the patch (Firefox, Chromium, OpenSSL), not whether OWASP BenchmarkTest00074 is a TP. The labeled SAST slice is 11 textbook files. Overlay Youden there is **worse** than inventory (+9.09% → 0%). We cannot tell if the tool got better on messy product code.
|
||||
|
||||
## Current State
|
||||
|
||||
Slices today: `local` (CI 100%), `github` (6 vendored functions: SVEN, hutool, node-serialize), `sast` (OWASP/Juliet). `datasets.toml` already points at CVEfixes, MoreFixes, CWE-Bench-Java, Vul4J. Nothing is harvested from gecko-dev / Firefox. Whole mozilla-central is not in-tree (and should not be). FixFox (Zenodo) is embargoed until 2030. Firefox security commit messages are often scrubbed of “security” words, so grep-the-log is a bad miner.
|
||||
Slices today: `local` (CI 100%), `github` (6 vendored functions: SVEN, hutool, node-serialize), `sast` (OWASP/Juliet). `datasets.toml` already points at CVEfixes, MoreFixes, CWE-Bench-Java, Vul4J. Nothing is harvested from gecko-dev / Firefox. Whole mozilla-central is not in-tree (and should not be). FixFox (Zenodo) is embargoed until 2030. Firefox security commit messages are often scrubbed of "security" words, so grep-the-log is a bad miner.
|
||||
|
||||
Measured on the current 11-pair **sast** slice (2026-09-04):
|
||||
|
||||
@@ -27,8 +27,8 @@ Do not clone Firefox. Filter public VFC databases for `gecko-dev` / `mozilla-fir
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: harvest script or documented procedure; catalog slice; provenance headers; overlay-vs-inventory scoreboard on that slice; dataset.toml pointers for Firefox/Chromium.
|
||||
- **Out**: Vendoring mozilla-central; live Bugzilla scraping; making Youden a merge gate; buffer-size semantics; tree-sitter packaging (owned by `tree-sitter-overlay-extra`).
|
||||
- In: harvest script or documented procedure; catalog slice; provenance headers; overlay-vs-inventory scoreboard on that slice; dataset.toml pointers for Firefox/Chromium.
|
||||
- Out: Vendoring mozilla-central; live Bugzilla scraping; making Youden a merge gate; buffer-size semantics; tree-sitter packaging (owned by `tree-sitter-overlay-extra`).
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
@@ -41,17 +41,17 @@ Do not clone Firefox. Filter public VFC databases for `gecko-dev` / `mozilla-fir
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `pairs.py` catalog format; `propose-adjudicate-prove` overlay scoring
|
||||
- **Downstream**: improve-loop signals from real misses; later memory-safety spec can reuse the same slice
|
||||
- Upstream: `pairs.py` catalog format; `propose-adjudicate-prove` overlay scoring
|
||||
- Downstream: improve-loop signals from real misses; later memory-safety spec can reuse the same slice
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: pair catalog only (new slice). Does not change smoke gate.
|
||||
- **Adjacent**: `tree-sitter-overlay-extra` (C/Java overlay on this slice once the extra exists)
|
||||
- Extends: pair catalog only (new slice). Does not change smoke gate.
|
||||
- Adjacent: `tree-sitter-overlay-extra` (C/Java overlay on this slice once the extra exists)
|
||||
|
||||
## Constraints
|
||||
|
||||
Isolated functions, same `relpath` vuln/fixed. License in the header. No 16 GB dumps in git. Firefox pairs only when a public git commit pair exists — MFSA text without a commit is not a pair.
|
||||
Isolated functions, same `relpath` vuln/fixed. License in the header. No 16 GB dumps in git. Firefox pairs only when a public git commit pair exists; MFSA text without a commit is not a pair.
|
||||
|
||||
## Closure (2026-09-04)
|
||||
|
||||
|
||||
@@ -132,12 +132,12 @@ benchmarks/pairs/
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `src/openultrasast/pairs.py` — concatenate vfc catalog; overlay path for `vfc`; `scorers` on the eval result; `DEFAULT_VFC_CATALOG`.
|
||||
- `src/openultrasast/cli.py` — `--slice` choice `vfc`.
|
||||
- `src/openultrasast/pair_gate.py` — comment that vfc is honesty-only; fail condition stays local.
|
||||
- `tests/test_pair_corpus.py` — vfc load, provenance, CLI, not in `LANGUAGE_MANIFESTS`, pair_gate still passes.
|
||||
- `benchmarks/pairs/datasets.toml` — project pointers.
|
||||
- `benchmarks/pairs/README.md` — slice list and harvest pointer.
|
||||
- `src/openultrasast/pairs.py`: concatenate vfc catalog; overlay path for `vfc`; `scorers` on the eval result; `DEFAULT_VFC_CATALOG`.
|
||||
- `src/openultrasast/cli.py`: `--slice` choice `vfc`.
|
||||
- `src/openultrasast/pair_gate.py`: comment that vfc is honesty-only; fail condition stays local.
|
||||
- `tests/test_pair_corpus.py`: vfc load, provenance, CLI, not in `LANGUAGE_MANIFESTS`, pair_gate still passes.
|
||||
- `benchmarks/pairs/datasets.toml`: project pointers.
|
||||
- `benchmarks/pairs/README.md`: slice list and harvest pointer.
|
||||
|
||||
## System Flows
|
||||
|
||||
@@ -203,8 +203,8 @@ Harvest (maintainer, not CI): recipe → fetch parent blob and fix blob → extr
|
||||
- Runnable set stays small; recipes may list more than CI runs.
|
||||
|
||||
**Dependencies**
|
||||
- Inbound: pair loader — concatenate file (P0)
|
||||
- Outbound: evaluate_pair — cases (P0)
|
||||
- Inbound: pair loader: concatenate file (P0)
|
||||
- Outbound: evaluate_pair: cases (P0)
|
||||
|
||||
**Contracts**: State [x]
|
||||
|
||||
|
||||
@@ -8,9 +8,9 @@ AppSec engineers cannot tell whether OpenUltraSAST improved on real product code
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: A new pair slice of reviewed isolated function pairs from OpenSSL, Firefox (gecko-dev / mozilla-central / mozilla-firefox/firefox), and Chromium; provenance on every vendored file; overlay-versus-inventory scoreboard on that slice; a harvest procedure that does not clone megarepos; offline runnable catalog; testability of catalog load and slice evaluation.
|
||||
- **Out of scope**: Cloning mozilla-central or Chromium into the repository or CI; FixFox (embargoed until 2030); live Bugzilla scraping; making overlay or vfc Youden a merge or smoke gate; buffer-size / CWE-121 semantics; inter-file taint; parser packaging; changing pattern inventory rules so the seed pairs pass.
|
||||
- **Adjacent expectations**: Existing pair evaluation still scores `local` as the 100% CI gate and `github` / `sast` as honesty dashboards. Overlay dispositions, taint, and the stage-1 detection gate stay owned by their current specs. Dataset pointers may list Firefox/Chromium/OpenSSL sources; the dumps themselves are not vendored.
|
||||
- In scope: A new pair slice of reviewed isolated function pairs from OpenSSL, Firefox (gecko-dev / mozilla-central / mozilla-firefox/firefox), and Chromium; provenance on every vendored file; overlay-versus-inventory scoreboard on that slice; a harvest procedure that does not clone megarepos; offline runnable catalog; testability of catalog load and slice evaluation.
|
||||
- Out of scope: Cloning mozilla-central or Chromium into the repository or CI; FixFox (embargoed until 2030); live Bugzilla scraping; making overlay or vfc Youden a merge or smoke gate; buffer-size / CWE-121 semantics; inter-file taint; parser packaging; changing pattern inventory rules so the seed pairs pass.
|
||||
- Adjacent expectations: Existing pair evaluation still scores `local` as the 100% CI gate and `github` / `sast` as honesty dashboards. Overlay dispositions, taint, and the stage-1 detection gate stay owned by their current specs. Dataset pointers may list Firefox/Chromium/OpenSSL sources; the dumps themselves are not vendored.
|
||||
|
||||
## Requirements
|
||||
|
||||
|
||||
@@ -2,49 +2,49 @@
|
||||
|
||||
## Summary
|
||||
|
||||
- **Feature**: `real-world-vfc-slice`
|
||||
- **Discovery Scope**: Extension of the existing pair catalog and pair eval CLI
|
||||
- **Key Findings**:
|
||||
- Feature: `real-world-vfc-slice`
|
||||
- Discovery Scope: Extension of the existing pair catalog and pair eval CLI
|
||||
- Key Findings:
|
||||
- Pair eval already concatenates `benchmarks/pairs/catalog.toml` with `sast/catalog.toml`. A third file `vfc/catalog.toml` is the smallest seam.
|
||||
- CLI `--slice` choices are a closed set (`all`, `local`, `github`, `sast`). `vfc` must be added there and in tests that assert slice names.
|
||||
- `sast` already scores overlay with inventory fallback when nothing is promoted or demoted. `vfc` should use that path as the primary scorer and add a second inventory-only figure for the honesty dashboard.
|
||||
- `pair_gate` evaluates the full catalog but fails CI only on `local` 100%. Adding vfc pairs does not change the fail condition if names stay out of `LANGUAGE_MANIFESTS`.
|
||||
- Seed commits with public VCS snapshots exist: OpenSSL Heartbleed (`tls1_process_heartbeat`), Firefox libmar (`mar_insert_item`, CVE-2020-15667), Chromium FileReader (`ArrayBufferResult`, CVE-2019-5786). All three keep a sink-shaped call after the fix, so overlay/inventory may LEAK — that is the dashboard, not a 100% gate.
|
||||
- Seed commits with public VCS snapshots exist: OpenSSL Heartbleed (`tls1_process_heartbeat`), Firefox libmar (`mar_insert_item`, CVE-2020-15667), Chromium FileReader (`ArrayBufferResult`, CVE-2019-5786). All three keep a sink-shaped call after the fix, so overlay/inventory may LEAK; that is the dashboard, not a 100% gate.
|
||||
- mozilla-central / gecko-dev must not be cloned. Phabricator and GitHub raw blobs are enough to extract one function. FixFox stays embargoed.
|
||||
|
||||
## Research Log
|
||||
|
||||
### Existing pair catalog and scoring
|
||||
|
||||
- **Context**: Brownfield extension; must not invent a second eval engine.
|
||||
- **Sources Consulted**: `src/openultrasast/pairs.py`, `cli.py`, `pair_gate.py`, `tests/test_pair_corpus.py`, `benchmarks/pairs/catalog.toml`, `benchmarks/pairs/sast/catalog.toml`, `benchmarks/pairs/README.md`
|
||||
- **Findings**:
|
||||
- Context: Brownfield extension; must not invent a second eval engine.
|
||||
- Sources Consulted: `src/openultrasast/pairs.py`, `cli.py`, `pair_gate.py`, `tests/test_pair_corpus.py`, `benchmarks/pairs/catalog.toml`, `benchmarks/pairs/sast/catalog.toml`, `benchmarks/pairs/README.md`
|
||||
- Findings:
|
||||
- `PairCase.slice` is a free string in the catalog loader; the closed set lives only in argparse and tests.
|
||||
- Overlay pair eval is gated on `case.slice == "sast"`.
|
||||
- GitHub pairs already use provenance comments plus catalog fields (`repo`, `commit`, `commit_url`, `cve`, `license`).
|
||||
- Pair names must not appear in `LANGUAGE_MANIFESTS`.
|
||||
- **Implications**: Add `vfc` as a concatenated catalog; extend the overlay branch to `sast` **or** `vfc`; keep the gate local-only.
|
||||
- Implications: Add `vfc` as a concatenated catalog; extend the overlay branch to `sast` **or** `vfc`; keep the gate local-only.
|
||||
|
||||
### Harvestable public commits (OpenSSL, Firefox, Chromium)
|
||||
|
||||
- **Context**: User asked for those three projects and a labeled corpus that can actually be tested.
|
||||
- **Sources Consulted**:
|
||||
- Context: User asked for those three projects and a labeled corpus that can actually be tested.
|
||||
- Sources Consulted:
|
||||
- OpenSSL `96db9023b881d7cd9f379b0c154650d6c108e9a3` (parent `0d7717fc9c83dafab8153cbd5e2180e6e04cc802`), `ssl/t1_lib.c` `tls1_process_heartbeat`, CVE-2014-0160
|
||||
- Firefox hg `b79b6cc78248eea7fda10bfb76aa273c19c9fa65` (parent `0c0f777161a9499dd149853ff62d356f75d16c2a`), `modules/libmar/src/mar_read.c` `mar_insert_item`, CVE-2020-15667 / Bug 1653371; Phabricator raw diff
|
||||
- Chromium `ba9748e78ec7e9c0d594e7edf7b2c07ea2a90449` (parent `9f3fccdba567f30ea39b08f57d8d5b49cd832b4a`), `FileReaderLoader::ArrayBufferResult`, CVE-2019-5786
|
||||
- Rejected: CVE-2019-11730 (SOP/file URI policy, not a sink pair); FixFox (embargoed to 2030); MFSA text without a commit
|
||||
- **Findings**:
|
||||
- Findings:
|
||||
- Heartbleed fix adds a record-length check; `memcpy(bp, pl, payload)` remains. Inventory `c-unsafe-memory-copy` will fire on both sides (LEAK) until CWE-121 semantics exist.
|
||||
- Firefox fix is `int namelen` → `uint32_t namelen`; `memcpy(item->name, name, namelen + 1)` remains. Same LEAK class.
|
||||
- Chromium fix copies the ArrayBuffer on partial results (UAF). No memcpy/eval sink; expected MISS on inventory and overlay. Still a labeled pair.
|
||||
- GitHub raw and Phabricator `?diff=1` return the function text without cloning.
|
||||
- **Implications**: Seed three reviewed pairs. Tests assert presence and that the slice **runs**, not that pair_correct is 100%. Memory-safety pairs are allowed as known leaks (Req 6.3).
|
||||
- Implications: Seed three reviewed pairs. Tests assert presence and that the slice **runs**, not that pair_correct is 100%. Memory-safety pairs are allowed as known leaks (Req 6.3).
|
||||
|
||||
### Dataset pointers versus vendored dumps
|
||||
|
||||
- **Context**: `datasets.toml` already lists CVEfixes, MoreFixes, SVEN, CWE-Bench-Java. Roadmap asked for Firefox/Chromium/OpenSSL pointers.
|
||||
- **Findings**: Pointers are metadata only. CI must stay offline on vendored excerpts.
|
||||
- **Implications**: Add dataset rows for OpenSSL, gecko-dev / mozilla-firefox/firefox, and Chromium. Do not fetch dumps in tests.
|
||||
- Context: `datasets.toml` already lists CVEfixes, MoreFixes, SVEN, CWE-Bench-Java. Roadmap asked for Firefox/Chromium/OpenSSL pointers.
|
||||
- Findings: Pointers are metadata only. CI must stay offline on vendored excerpts.
|
||||
- Implications: Add dataset rows for OpenSSL, gecko-dev / mozilla-firefox/firefox, and Chromium. Do not fetch dumps in tests.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
@@ -59,48 +59,48 @@
|
||||
|
||||
### Decision: Concatenated `vfc` catalog, overlay primary, inventory sidecar
|
||||
|
||||
- **Context**: Need a labeled corpus that is offline, isolated, and honest about overlay versus regex.
|
||||
- **Alternatives Considered**:
|
||||
1. Inventory-only like `github` — misses the overlay question the user is evolving.
|
||||
2. Overlay-only like `sast` without a sidecar — cannot see whether overlay helped or hurt.
|
||||
3. Dual scoreboard with overlay primary and inventory sidecar — **selected**.
|
||||
- **Selected Approach**: `slice = "vfc"` uses the overlay pair path (inventory fallback when unadjudicated). `PairEvalResult.scorers["vfc"]` holds both `overlay` and `inventory` metrics. Pair gate still fails only on `local`.
|
||||
- **Rationale**: Reuses `sast` overlay machinery; satisfies Req 4 without making Youden a merge gate.
|
||||
- **Trade-offs**: Pair gate runtime grows by three small overlay scans. Acceptable.
|
||||
- **Follow-up**: Do not retune memcpy rules to force these pairs to PASS.
|
||||
- Context: Need a labeled corpus that is offline, isolated, and honest about overlay versus regex.
|
||||
- Alternatives Considered:
|
||||
1. Inventory-only like `github`: misses the overlay question the user is evolving.
|
||||
2. Overlay-only like `sast` without a sidecar: cannot see whether overlay helped or hurt.
|
||||
3. Dual scoreboard with overlay primary and inventory sidecar: **selected**.
|
||||
- Selected Approach: `slice = "vfc"` uses the overlay pair path (inventory fallback when unadjudicated). `PairEvalResult.scorers["vfc"]` holds both `overlay` and `inventory` metrics. Pair gate still fails only on `local`.
|
||||
- Rationale: Reuses `sast` overlay machinery; satisfies Req 4 without making Youden a merge gate.
|
||||
- Trade-offs: Pair gate runtime grows by three small overlay scans. Acceptable.
|
||||
- Follow-up: Do not retune memcpy rules to force these pairs to PASS.
|
||||
|
||||
### Decision: Maintainer harvest script, vendored seed, no CI fetch
|
||||
|
||||
- **Context**: Grow the catalog later without cloning mozilla-central.
|
||||
- **Selected Approach**: `benchmarks/pairs/vfc/harvest.py` plus `recipes.toml`. CI never runs harvest. Seed files are committed excerpts with provenance headers.
|
||||
- **Rationale**: Req 5 and Req 7 (offline testability).
|
||||
- **Trade-offs**: Harvest needs network when a maintainer grows the set; tests mock extraction locally.
|
||||
- Context: Grow the catalog later without cloning mozilla-central.
|
||||
- Selected Approach: `benchmarks/pairs/vfc/harvest.py` plus `recipes.toml`. CI never runs harvest. Seed files are committed excerpts with provenance headers.
|
||||
- Rationale: Req 5 and Req 7 (offline testability).
|
||||
- Trade-offs: Harvest needs network when a maintainer grows the set; tests mock extraction locally.
|
||||
|
||||
### Decision: Public hg counts as a version-control commit
|
||||
|
||||
- **Context**: Firefox canonical history is hg.mozilla.org; gecko-dev git SHAs are not the hg SHAs; GitHub 422s on hg identifiers.
|
||||
- **Selected Approach**: Record the hg revision and Phabricator/hg URL. Dataset pointers still list gecko-dev and mozilla-firefox/firefox.
|
||||
- **Rationale**: Brief forbids MFSA-without-commit, not hg. The MAR fix is a public VCS snapshot.
|
||||
- **Follow-up**: If a git SHA is later mapped, add it as an extra catalog field without changing the excerpt.
|
||||
- Context: Firefox canonical history is hg.mozilla.org; gecko-dev git SHAs are not the hg SHAs; GitHub 422s on hg identifiers.
|
||||
- Selected Approach: Record the hg revision and Phabricator/hg URL. Dataset pointers still list gecko-dev and mozilla-firefox/firefox.
|
||||
- Rationale: Brief forbids MFSA-without-commit, not hg. The MAR fix is a public VCS snapshot.
|
||||
- Follow-up: If a git SHA is later mapped, add it as an extra catalog field without changing the excerpt.
|
||||
|
||||
## Synthesis
|
||||
|
||||
- **Generalization**: `vfc` is another honesty slice, not a new product. Overlay-versus-inventory is a report field, not a new engine.
|
||||
- **Build vs adopt**: Adopt pair catalog schema, overlay eval, provenance comment style. Build only harvest extract + dual scoreboard payload.
|
||||
- **Simplification**: No Joern, no megarepo, no 90% gate, no rule changes to chase Youden on three memory/UAF pairs.
|
||||
- Generalization: `vfc` is another honesty slice, not a new product. Overlay-versus-inventory is a report field, not a new engine.
|
||||
- Build vs adopt: Adopt pair catalog schema, overlay eval, provenance comment style. Build only harvest extract + dual scoreboard payload.
|
||||
- Simplification: No Joern, no megarepo, no 90% gate, no rule changes to chase Youden on three memory/UAF pairs.
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- Overlay/inventory LEAK on Heartbleed and MAR memcpy-after-fix — document as known; tests must not assert PASS.
|
||||
- Chromium UAF is a MISS — still required so the corpus is not only memcpy.
|
||||
- Overlay/inventory LEAK on Heartbleed and MAR memcpy-after-fix; document as known; tests must not assert PASS.
|
||||
- Chromium UAF is a MISS; still required so the corpus includes more than memcpy.
|
||||
- `pair_gate` currently scans the full catalog; three extra pairs are small. If the catalog grows, gate should keep evaluating local-only for the fail condition (already true) and may later skip overlay slices for speed.
|
||||
- License: OpenSSL (OpenSSL License / SSLeay), Firefox MPL-2.0, Chromium BSD-3-Clause — headers must state them.
|
||||
- License: OpenSSL (OpenSSL License / SSLeay), Firefox MPL-2.0, Chromium BSD-3-Clause; headers must state them.
|
||||
- gecko-dev is archived; harvest should accept hg.mozilla.org and github.com/chromium/chromium / openssl/openssl as hosts.
|
||||
|
||||
## References
|
||||
|
||||
- [OpenSSL Heartbleed fix](https://github.com/openssl/openssl/commit/96db9023b881d7cd9f379b0c154650d6c108e9a3) — CVE-2014-0160
|
||||
- [Firefox MAR heap overflow](https://bugzilla.mozilla.org/show_bug.cgi?id=1653371) — CVE-2020-15667, hg `b79b6cc78248`
|
||||
- [Chromium FileReader UAF](https://github.com/chromium/chromium/commit/ba9748e78ec7e9c0d594e7edf7b2c07ea2a90449) — CVE-2019-5786
|
||||
- [OpenSSL Heartbleed fix](https://github.com/openssl/openssl/commit/96db9023b881d7cd9f379b0c154650d6c108e9a3): CVE-2014-0160
|
||||
- [Firefox MAR heap overflow](https://bugzilla.mozilla.org/show_bug.cgi?id=1653371): CVE-2020-15667, hg `b79b6cc78248`
|
||||
- [Chromium FileReader UAF](https://github.com/chromium/chromium/commit/ba9748e78ec7e9c0d594e7edf7b2c07ea2a90449): CVE-2019-5786
|
||||
- `.kiro/specs/real-world-vfc-slice/brief.md`
|
||||
- `.kiro/steering/roadmap.md`
|
||||
|
||||
@@ -1,32 +1,32 @@
|
||||
# Search-with-proof — handoff (2026-10-06)
|
||||
# Search-with-proof, handoff (2026-10-06)
|
||||
|
||||
State at handoff, for resuming on the search problem and abstract interpretation.
|
||||
|
||||
## RESOLVED 2026-10-07 — the one open question is CLOSED: first cluster demonstration achieved
|
||||
## RESOLVED 2026-10-07, the one open question is CLOSED: first cluster demonstration achieved
|
||||
|
||||
The "first `demonstrated`" question below is answered. search-with-proof produced its first autonomous
|
||||
`demonstrated` finding end to end on kube-ax, and a blind negative confirmed no false proof.
|
||||
|
||||
- **Pair:** hanxi/xiaomusic (pilot pair 13), family `path`, vulnerability class **prefix-check bypass**
|
||||
- Pair: hanxi/xiaomusic (pilot pair 13), family `path`, vulnerability class **prefix-check bypass**
|
||||
(`startswith(root)` without a trailing separator; the fix adds `os.sep`). Image `e2c8b460`.
|
||||
- **Vulnerable side:** `demonstrated` — owned effect in 3/3 affected runs, 0/3 safe runs. $0.37, 55 calls.
|
||||
- **Fixed side (blind negative, Req 6.3):** `inconclusive`, **0 false proofs**. $0.56, 85 calls.
|
||||
- **Pair total:** $1.22. Record: `benchmarks/measurements/2026-10-07-search-first-cluster-demonstration/`.
|
||||
- Vulnerable side: `demonstrated`, owned effect in 3/3 affected runs, 0/3 safe runs. $0.37, 55 calls.
|
||||
- Fixed side (blind negative, Req 6.3): `inconclusive`, **0 false proofs**. $0.56, 85 calls.
|
||||
- Pair total: $1.22. Record: `benchmarks/measurements/2026-10-07-search-first-cluster-demonstration/`.
|
||||
|
||||
**The original pair-6 both-sides-false was a TWO-part diagnosis, both now fixed:** the demo did not reach the
|
||||
observable (demo/agent), and — the decisive one — the path oracle only modelled **classic parent traversal** while
|
||||
observable (demo/agent), and (the decisive one) the path oracle only modelled **classic parent traversal** while
|
||||
many real path CVEs (xiaomusic included) are **prefix-bypass**, which reaches a sibling that string-extends the served
|
||||
dir, not the parent. The oracle now also plants a prefix-sibling canary.
|
||||
|
||||
**Fix chain that enabled the demonstration (each merged to main and validated by a paid run):**
|
||||
1. Parity — agent is told the observable's location (path oracle reached parity with command/ssrf).
|
||||
2. Verify-time-only — the observable is injected at verify time and absent during exploration; stop the explore-time
|
||||
1. Parity. The agent is told the observable's location (path oracle reached parity with command/ssrf).
|
||||
2. Verify-time-only. The observable is injected at verify time and absent during exploration; stop the explore-time
|
||||
reachability trap that burned budget without authoring a demo.
|
||||
3. Fair-share pilot budget — each search gets a fair share of `--ceiling-usd` with rollover, so one expensive search
|
||||
3. Fair-share pilot budget. Each search gets a fair share of `--ceiling-usd` with rollover, so one expensive search
|
||||
no longer starves the rest.
|
||||
4. Served-root guidance — the demo MUST set the app's served directory to the `${served_root}` placeholder.
|
||||
5. Oracle prefix-bypass coverage — plant a sibling canary for the `startswith`-without-separator class.
|
||||
6. Control-signal feedback — the keystone control (does the app serve `${served_root}`?) is fed back to the agent as a
|
||||
4. Served-root guidance. The demo MUST set the app's served directory to the `${served_root}` placeholder.
|
||||
5. Oracle prefix-bypass coverage. Plant a sibling canary for the `startswith`-without-separator class.
|
||||
6. Control-signal feedback. The keystone control (does the app serve `${served_root}`?) is fed back to the agent as a
|
||||
`served-root` hint so it self-corrects its wiring. This was the decisive last step.
|
||||
|
||||
**Keystone (honest measurement), merged earlier:** the verifier runs a benign positive control; a broken or mis-wired
|
||||
@@ -38,7 +38,7 @@ dependency build-break (`filed`→`mime@4`) is pinned.
|
||||
|
||||
---
|
||||
|
||||
## The one open question (HISTORICAL — resolved above)
|
||||
## The one open question (HISTORICAL, resolved above)
|
||||
|
||||
The full chain runs end to end on kube-ax. Every execution layer is cleared. The **first `demonstrated` has not
|
||||
happened yet**, and the remaining gap is narrow and reproducible at **$0 model cost** via `demo_replay`.
|
||||
@@ -59,7 +59,7 @@ Both-sides-false means it is **not the harness** any more. It is one of:
|
||||
capture in `oracles.py` `PathOracle.observe`).
|
||||
- Re-run with `python -m benchmarks.search.demo_replay --both --demo <stored demo.json> --family path --lane ax
|
||||
--image <digest> ...`. Stored demos live under `benchmarks/search/private/runs/*/<pair>/demo/demo.json` (gitignored).
|
||||
- **Classifier note:** keep exploit payloads on disk. Hand Codex file-path tasks that treat demo contents as opaque;
|
||||
- Classifier note: keep exploit payloads on disk. Hand Codex file-path tasks that treat demo contents as opaque;
|
||||
print outcomes / phase timings / error *classes*, never request strings, env values, or step bodies. See memory
|
||||
`search-with-proof-direction` and the classifier false-positive note. Code changes never tripped it; echoing
|
||||
payloads or step-by-step attack prose did.
|
||||
@@ -95,9 +95,9 @@ image (pip 3.13 vs venv 3.12 → use `sys.executable`) → checkout root on sys.
|
||||
- `/workspace` ~3 GiB free per node until the volume move; scratch guard 2 GiB/task.
|
||||
- Pilot spend to date ~$11 across 9 paid runs; the decisive replay was $0.
|
||||
|
||||
## Decision model vs search — current read
|
||||
## Decision model vs search (current read)
|
||||
|
||||
- Decision/classification: measured, **promising as triage** (within-pair AUC 0.762; trained logistic ties at 0.721),
|
||||
**capped as a blocker** (cannot prove 95% precision). Reframed as triage (`learned-decision-engine` Req 6.5).
|
||||
- Search-with-proof: **not yet demonstrated**, but no observation so far argues against the idea — every failure was
|
||||
- Search-with-proof: **not yet demonstrated**, but no observation so far argues against the idea. Every failure was
|
||||
our execution, now cleared. The pair-6 demo/oracle question is the first verdict on the method itself.
|
||||
|
||||
@@ -16,13 +16,13 @@ each confirm one fact; the loop ends when the goal is demonstrably met.
|
||||
|
||||
Three things follow for OpenUltraSAST:
|
||||
|
||||
1. **An objective success condition.** A finding is demonstrated when a test or proof of concept fails on the
|
||||
1. An objective success condition. A finding is demonstrated when a test or proof of concept fails on the
|
||||
vulnerable code and passes on the fixed code (or, for a push, fails on head and passes on base where applicable).
|
||||
A demonstrated finding is right by construction; it needs no calibration and no training data, so it generalises
|
||||
to unseen repositories by construction.
|
||||
2. **Exploration with tools.** Agents that read other files, follow callers, build and run the code, instead of a
|
||||
2. Exploration with tools. Agents that read other files, follow callers, build and run the code, instead of a
|
||||
single-shot judgement of one excerpt.
|
||||
3. **Search state per repository and push.** Facts and intents accumulate on a board instead of every question
|
||||
3. Search state per repository and push. Facts and intents accumulate on a board instead of every question
|
||||
starting from zero.
|
||||
|
||||
## Shape
|
||||
|
||||
@@ -73,7 +73,7 @@ Other files already modified/untracked at session start remain uncommitted.
|
||||
|
||||
- Requested suite, with `-rs` for skip reasons:
|
||||
`/home/mc/Source/OpenUltraSAST/.venv/bin/pytest -q tests/test_search_*.py tests/test_ax_batch.py tests/test_plane_literals.py -rs`
|
||||
— **181 passed, 3 skipped, exit 0**, 24.94 seconds.
|
||||
Result: **181 passed, 3 skipped, exit 0**, 24.94 seconds.
|
||||
- Final executor/storage regression and module audit tests after tightening stderr
|
||||
truncation: **34 passed, exit 0**, including all **6 module audit tests**.
|
||||
- Shared dispatcher consumer `tests/test_unseen_calibrate.py`: **10 passed, exit 0**.
|
||||
|
||||
@@ -9,12 +9,12 @@ kube-ax cluster. BLOCK becomes "demonstrated", not "scored above a threshold". C
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: the board (facts, intents, hints) in the memory store; reason and explore steps as AX Tasks; the
|
||||
- In scope: the board (facts, intents, hints) in the memory store; reason and explore steps as AX Tasks; the
|
||||
demonstration contract and its verification; triage-to-search budgeting; a pilot on known fix pairs; a G1 arm on the
|
||||
unseen pool; model access through an operator-managed scoped key.
|
||||
- **Out of scope**: copying Cairn code (AGPL-3.0); network actions against anything but the code under analysis;
|
||||
- Out of scope: copying Cairn code (AGPL-3.0); network actions against anything but the code under analysis;
|
||||
changing the existing quick rules, engine or classifier (they are inputs here).
|
||||
- **Adjacent expectations**: `unseen-repo-evaluation` scores this arm; the AX batch dispatcher (`benchmarks/ax/batch.py`)
|
||||
- Adjacent expectations: `unseen-repo-evaluation` scores this arm; the AX batch dispatcher (`benchmarks/ax/batch.py`)
|
||||
and the kube-ax constraints (2 heavy lanes, `ousast-engine-` names, egress to public HTTPS) apply;
|
||||
`learned-decision-engine` Req 6.4 BLOCK rule is superseded for this arm by "demonstrated".
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@ directed state-space **search**, then a DAST-style **proof (verification)**.
|
||||
between the vulnerable and fixed revisions: the owned effect appears on the vulnerable side and
|
||||
not on the fixed side (3/3 vs 0/3). Only this differential earns `demonstrated`.
|
||||
|
||||
## Trust boundary — what runs where
|
||||
## Trust boundary (what runs where)
|
||||
|
||||
The brain runs on the VM; the agent's tool executions and the DAST proof run as isolated tasks in
|
||||
the kube-ax cluster. Secrets never enter the cluster.
|
||||
|
||||
@@ -19,9 +19,9 @@ AppSec engineers and DevOps teams running OpenUltraSAST in CI get a regex static
|
||||
|
||||
A scan answers three operator questions, in order:
|
||||
|
||||
1. **What exists?** A static inventory of files, surfaces, and pattern/SARIF hits — honest about being inventory, not verified bugs.
|
||||
2. **Where are problems most likely, and how do I tune tests?** A complexity map of hotspots plus test-gap hints a DevOps team can act on.
|
||||
3. **Is this worth fixing?** Isolated regression against the top hotspots. Triggerable and reachable → worth a CI fail. Not triggerable → do not fail the build.
|
||||
1. What exists? A static inventory of files, surfaces, and pattern/SARIF hits, honest about being inventory, not verified bugs.
|
||||
2. Where are problems most likely, and how do I tune tests? A complexity map of hotspots plus test-gap hints a DevOps team can act on.
|
||||
3. Is this worth fixing? Isolated regression against the top hotspots. Triggerable and reachable → worth a CI fail. Not triggerable → do not fail the build.
|
||||
|
||||
Stages 2–3 use a tool-calling hunter (read / grep / callers) and a Docker-isolated runner. Evaluation uses split-sink and mutation corpora, not cheat sheets. Missing Docker or a model degrades visibly; stage 1 still runs.
|
||||
|
||||
@@ -37,12 +37,12 @@ Do not require the HarnessX extra for the LLM path. Use the existing stdlib Open
|
||||
|
||||
Do not add a Python Docker SDK. Drive the Docker CLI via subprocess with the threat-model flags (no network, read-only source, tmpfs scratch, memory/pids/timeout, cap-drop). Core `dependencies = []` stays.
|
||||
|
||||
Complexity map is a new artifact built from existing ranking, entry points, finding density, nesting/size, and test-file presence — not a rewrite of `RankingScore`. Stage 3 writes a worth-fixing verdict that feeds back into a complexity ledger (evolutionary), without editing detection pattern text.
|
||||
Complexity map is a new artifact built from existing ranking, entry points, finding density, nesting/size, and test-file presence, not a rewrite of `RankingScore`. Stage 3 writes a worth-fixing verdict that feeds back into a complexity ledger (evolutionary), without editing detection pattern text.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: three-stage scan contract; complexity map + test-tuning hints; Docker-isolated regression and worth-fixing verdicts; tool-calling hunter on hotspots; split-sink/mutation evaluation; CI mapping (PR = stage 1, nightly = stage 2/3); visible degradation; evolutionary feedback from regress → map.
|
||||
- **Out**: rewriting CWE policy or project-score formula; auto-authoring regex patterns; full fuzzing campaigns; patch generation/application; network pentest; importing Clearwing.
|
||||
- In: three-stage scan contract; complexity map + test-tuning hints; Docker-isolated regression and worth-fixing verdicts; tool-calling hunter on hotspots; split-sink/mutation evaluation; CI mapping (PR = stage 1, nightly = stage 2/3); visible degradation; evolutionary feedback from regress → map.
|
||||
- Out: rewriting CWE policy or project-score formula; auto-authoring regex patterns; full fuzzing campaigns; patch generation/application; network pentest; importing Clearwing.
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
@@ -61,13 +61,13 @@ Complexity map is a new artifact built from existing ranking, entry points, find
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: existing preprocess, rank, mapping, findings, verification, config, CLI modes, OpenRouter client, threat-model sandbox limits.
|
||||
- **Downstream**: CI fail-on `worth-fixing`; later fuzz/patch specs; MCP scan status.
|
||||
- Upstream: existing preprocess, rank, mapping, findings, verification, config, CLI modes, OpenRouter client, threat-model sandbox limits.
|
||||
- Downstream: CI fail-on `worth-fixing`; later fuzz/patch specs; MCP scan status.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: `openrouter-sast-harness` gates `standard_security_harness` and `sandboxed_dynamic_harness` (still unsatisfied).
|
||||
- **Adjacent**: `harnessx-self-improving-rulesets` owns CWE policy, score, shadow, optional HarnessX extra. This spec must not reopen those.
|
||||
- Extends: `openrouter-sast-harness` gates `standard_security_harness` and `sandboxed_dynamic_harness` (still unsatisfied).
|
||||
- Adjacent: `harnessx-self-improving-rulesets` owns CWE policy, score, shadow, optional HarnessX extra. This spec must not reopen those.
|
||||
|
||||
## Constraints
|
||||
|
||||
|
||||
@@ -32,13 +32,13 @@
|
||||
|
||||
### This Spec Owns
|
||||
|
||||
- **Stage plan**: which stages a mode runs, skip/degrade accounting, `stages_requested` / `stages_completed` in the manifest.
|
||||
- **Complexity map**: `Hotspot` records, signals, test-gap hints, `complexity_map.json`.
|
||||
- **Complexity ledger**: overlay of hotspot scores from regression verdicts.
|
||||
- **Sandbox**: capability probe and Docker CLI runner that never mounts the host Docker socket.
|
||||
- **Regression / worth-fixing**: candidate selection, language recipes, verdicts, `--fail-on worth-fixing`.
|
||||
- **Tool hunter**: repo-bound read / grep / callers loop for stages 2–3, evidence honesty.
|
||||
- **Split-sink evaluation**: corpora and the "beats pattern-hit ranking" gate.
|
||||
- Stage plan: which stages a mode runs, skip/degrade accounting, `stages_requested` / `stages_completed` in the manifest.
|
||||
- Complexity map: `Hotspot` records, signals, test-gap hints, `complexity_map.json`.
|
||||
- Complexity ledger: overlay of hotspot scores from regression verdicts.
|
||||
- Sandbox: capability probe and Docker CLI runner that never mounts the host Docker socket.
|
||||
- Regression / worth-fixing: candidate selection, language recipes, verdicts, `--fail-on worth-fixing`.
|
||||
- Tool hunter: repo-bound read / grep / callers loop for stages 2–3, evidence honesty.
|
||||
- Split-sink evaluation: corpora and the "beats pattern-hit ranking" gate.
|
||||
|
||||
### Out of Boundary
|
||||
|
||||
@@ -167,14 +167,14 @@ benchmarks/manifests/split-sink-*.toml
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `cli.py` — run stage 2/3; stop exiting on `deep`; `--fail-on worth-fixing`; write new artifacts.
|
||||
- `reports.py` — `scan_exit_code` grows `worth-fixing`; report sections for inventory, map, and worth-fixing.
|
||||
- `config.py` — `ComplexityConfig` (top_k, max_hunter_hotspots) and `RegressConfig` (max_candidates, image pins); **consume** existing `SandboxConfig` (do not add a second sandbox TOML).
|
||||
- `hunter.py` — schedule remains; execution delegates to `tool_hunter` when a model is set, else inventory-only.
|
||||
- `verification.py` / reports — inventory vs worth-fixing labeling; hunter findings stay `suspicion`.
|
||||
- `gate.py` — stage-1 smoke gate unchanged; add split-sink map gate.
|
||||
- `reports.py` — map and verdict sections in markdown/manifest.
|
||||
- `.github/workflows/ci.yml` — PR stays quick; optional nightly deep when Docker is available.
|
||||
- `cli.py`, run stage 2/3; stop exiting on `deep`; `--fail-on worth-fixing`; write new artifacts.
|
||||
- `reports.py`, `scan_exit_code` grows `worth-fixing`; report sections for inventory, map, and worth-fixing.
|
||||
- `config.py`, `ComplexityConfig` (top_k, max_hunter_hotspots) and `RegressConfig` (max_candidates, image pins); **consume** existing `SandboxConfig` (do not add a second sandbox TOML).
|
||||
- `hunter.py`, schedule remains; execution delegates to `tool_hunter` when a model is set, else inventory-only.
|
||||
- `verification.py` / reports, inventory vs worth-fixing labeling; hunter findings stay `suspicion`.
|
||||
- `gate.py`, stage-1 smoke gate unchanged; add split-sink map gate.
|
||||
- `reports.py`, map and verdict sections in markdown/manifest.
|
||||
- `.github/workflows/ci.yml`, PR stays quick; optional nightly deep when Docker is available.
|
||||
|
||||
## System Flows
|
||||
|
||||
@@ -468,7 +468,7 @@ Degradations reuse the existing list: `{ "stage": "regress", "reason": "sandbox_
|
||||
|
||||
- Mode→stage mapping and skip accounting.
|
||||
- Path clamp rejects `../`.
|
||||
- Map order on a fixture where regex hits a trivial helper and the real bug is a split-sink parser — parser ranks higher.
|
||||
- Map order on a fixture where regex hits a trivial helper and the real bug is a split-sink parser, parser ranks higher.
|
||||
- Covered hotspot yields `gap=covered` and no test kind.
|
||||
- Safety check rejects docker.sock and network snippets.
|
||||
- Ledger boost/demote; sev-5 reachable remains a candidate.
|
||||
|
||||
@@ -13,9 +13,9 @@ The work is a follow-up to the HarnessX ruleset-governance spec, not a replaceme
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: a three-stage scan contract mapped onto existing scan depths; a static inventory that reports what exists without claiming verification; a complexity map of hotspots with test-tuning hints; isolated regression that produces a worth-fixing verdict; a tool-using hunter confined to stages 2–3; evaluation against split-sink and mutation corpora; visible degradation when the sandbox or a model is unavailable; evolutionary feedback from regression verdicts back into the complexity map.
|
||||
- **Out of scope**: rewriting CWE-to-severity policy or the project-score formula; authoring or rewriting detection pattern text; full fuzzing campaigns; generating or applying patches; network pentesting; importing Clearwing.
|
||||
- **Adjacent expectations**: stage 1 continues to run in the zero-dependency default install with no model and no sandbox. CWE policy, scoring, traces, and SARIF remain owned by the existing governance spec. The operator supplies an isolated-sandbox runtime (Docker CLI) for stage 3 and an LLM credential for the tool-using hunter; neither is a packaged library dependency of the core install.
|
||||
- In scope: a three-stage scan contract mapped onto existing scan depths; a static inventory that reports what exists without claiming verification; a complexity map of hotspots with test-tuning hints; isolated regression that produces a worth-fixing verdict; a tool-using hunter confined to stages 2–3; evaluation against split-sink and mutation corpora; visible degradation when the sandbox or a model is unavailable; evolutionary feedback from regression verdicts back into the complexity map.
|
||||
- Out of scope: rewriting CWE-to-severity policy or the project-score formula; authoring or rewriting detection pattern text; full fuzzing campaigns; generating or applying patches; network pentesting; importing Clearwing.
|
||||
- Adjacent expectations: stage 1 continues to run in the zero-dependency default install with no model and no sandbox. CWE policy, scoring, traces, and SARIF remain owned by the existing governance spec. The operator supplies an isolated-sandbox runtime (Docker CLI) for stage 3 and an LLM credential for the tool-using hunter; neither is a packaged library dependency of the core install.
|
||||
|
||||
## Requirements
|
||||
|
||||
|
||||
@@ -5,9 +5,9 @@
|
||||
---
|
||||
|
||||
## Summary
|
||||
- **Feature**: `three-stage-scan`
|
||||
- **Discovery Scope**: Extension of the existing scan harness (brownfield), with one new runtime prerequisite (Docker CLI for stage 3) and one new evaluation contract (split-sink / mutation corpora).
|
||||
- **Key Findings**:
|
||||
- Feature: `three-stage-scan`
|
||||
- Discovery Scope: Extension of the existing scan harness (brownfield), with one new runtime prerequisite (Docker CLI for stage 3) and one new evaluation contract (split-sink / mutation corpora).
|
||||
- Key Findings:
|
||||
- Stage 1 already exists as `quick` mode: preprocess, mapping, rank, `quick_scan_findings`, structural verify, score, SARIF. Do not rebuild it.
|
||||
- Stage 2 is not a hunter dump and is not `RankingScore`. Ranking is per-file attention. A complexity map is per-file-and-function hotspot attention plus test-gap hints. Existing signals (tags, loc, reachability hints, finding density, `has_fuzz_entry_point`) are enough for a heuristic map; the tool-using hunter refines it.
|
||||
- Stage 3 has config and a threat-model table, but `cli.py` exits on `--mode deep`. No container is ever started. The first execution of untrusted code must be Docker CLI with the documented isolation flags, not a Python SDK and not host `subprocess` of target code.
|
||||
@@ -17,69 +17,69 @@
|
||||
## Research Log
|
||||
|
||||
### Existing scan pipeline and mode mapping
|
||||
- **Context**: Whether to add `--stage` or reuse `--mode quick|standard|deep`.
|
||||
- **Sources Consulted**: `src/openultrasast/cli.py` (`_run_scan`, mode choices), `README.md` maturity legend, `.kiro/specs/openrouter-sast-harness/requirements.md` (hunter pool, docker sandbox, dynamic analysis), `.kiro/specs/harnessx-self-improving-rulesets/design.md`.
|
||||
- **Findings**:
|
||||
- Context: Whether to add `--stage` or reuse `--mode quick|standard|deep`.
|
||||
- Sources Consulted: `src/openultrasast/cli.py` (`_run_scan`, mode choices), `README.md` maturity legend, `.kiro/specs/openrouter-sast-harness/requirements.md` (hunter pool, docker sandbox, dynamic analysis), `.kiro/specs/harnessx-self-improving-rulesets/design.md`.
|
||||
- Findings:
|
||||
- `quick` is implemented and deterministic. `standard` schedules a hunter pool that calls `quick_scan_findings` unless HarnessX extra + hunter model are set. `deep` raises `SystemExit`.
|
||||
- OpenRouter spec already defined: quick = static + bounded LLM review; standard = sandboxed tool-calling hunters; deep = harness/fuzz/dynamic. Implementation never reached that.
|
||||
- Existing CI and README teach `--mode`. A parallel `--stage` flag would fork the contract.
|
||||
- **Implications**: Map modes onto stages 1 / 1+2 / 1+2+3. Record `stages_requested` and `stages_completed` in the manifest so the three questions are explicit even if the flag names stay.
|
||||
- Implications: Map modes onto stages 1 / 1+2 / 1+2+3. Record `stages_requested` and `stages_completed` in the manifest so the three questions are explicit even if the flag names stay.
|
||||
|
||||
### Ranking vs complexity map
|
||||
- **Context**: `rank.py` already scores files. Is stage 2 just ranking?
|
||||
- **Sources Consulted**: `src/openultrasast/rank.py` (`heuristic_rank`, `composite_priority` = surface 0.5 + influence 0.2 + reachability 0.3), `preprocess.py` (`FileTarget` tags, loc, `has_fuzz_entry_point`), `mapping.py` (`analyze_entry_points`).
|
||||
- **Findings**:
|
||||
- Context: `rank.py` already scores files. Is stage 2 just ranking?
|
||||
- Sources Consulted: `src/openultrasast/rank.py` (`heuristic_rank`, `composite_priority` = surface 0.5 + influence 0.2 + reachability 0.3), `preprocess.py` (`FileTarget` tags, loc, `has_fuzz_entry_point`), `mapping.py` (`analyze_entry_points`).
|
||||
- Findings:
|
||||
- Ranking is file-level and exists to budget the hunter. It does not name functions, test gaps, or "tune your tests."
|
||||
- Tags already encode likely problem classes (`memory_unsafe`, `parser`, `network_entry`, `syscall_entry`). Loc ≥ 500 boosts influence. No nesting/branch signal. No test-file presence.
|
||||
- A hotspot that stage 1 missed (split-sink) can still rank high if the file is a reachable parser. That is the point of stage 2.
|
||||
- **Implications**: Keep `RankingScore`. Add `ComplexityMap` / `Hotspot` with function identity when recoverable, test-gap hints, and a ledger overlay. Ranking still feeds hunter budgets; the map feeds humans and stage 3.
|
||||
- Implications: Keep `RankingScore`. Add `ComplexityMap` / `Hotspot` with function identity when recoverable, test-gap hints, and a ledger overlay. Ranking still feeds hunter budgets; the map feeds humans and stage 3.
|
||||
|
||||
### Hunter: regex, HarnessX dump, or tools
|
||||
- **Context**: Previous spec adopted HarnessX "at the hunter loop" but the default hunter is regex and the HarnessX path dumps source into a prompt.
|
||||
- **Sources Consulted**: `hunter.py` (`_run_hunter_task` → `quick_scan_findings`), `hunter_harness.py` (`_describe`, `_extract_findings`), `provider/openrouter.py` (`OpenRouterChatClient.complete_json`, stdlib `urllib`), `harness_ext.py` (optional extra).
|
||||
- **Findings**:
|
||||
- Context: Previous spec adopted HarnessX "at the hunter loop" but the default hunter is regex and the HarnessX path dumps source into a prompt.
|
||||
- Sources Consulted: `hunter.py` (`_run_hunter_task` → `quick_scan_findings`), `hunter_harness.py` (`_describe`, `_extract_findings`), `provider/openrouter.py` (`OpenRouterChatClient.complete_json`, stdlib `urllib`), `harness_ext.py` (optional extra).
|
||||
- Findings:
|
||||
- OpenRouter client is already zero-extra (stdlib). It is used for optional ranking, not for hunting.
|
||||
- HarnessX extra is SHA-pinned, heavy, absent in CI. Requiring it repeats the last spec's failure mode.
|
||||
- Clearwing hunters use `read_source_file`, `grep_source`, `find_callers` with path clamping. That is the evolutionary piece, not MetaAgent regex-status flips.
|
||||
- **Implications**: Stage 2/3 hunter is a bounded tool loop over the OpenRouter client (or HarnessX if already wired). Tools are stdlib (read, Python grep, caller search from mapping + text). Path clamp at repo root. Hunter output evidence = `suspicion`.
|
||||
- Implications: Stage 2/3 hunter is a bounded tool loop over the OpenRouter client (or HarnessX if already wired). Tools are stdlib (read, Python grep, caller search from mapping + text). Path clamp at repo root. Hunter output evidence = `suspicion`.
|
||||
|
||||
### Docker sandbox: CLI vs SDK vs skip
|
||||
- **Context**: Threat model already lists sandbox limits. Need a runner that does not invert zero-dep posture.
|
||||
- **Sources Consulted**: `docs/threat-model.md` (sandbox table), `config.py` `SandboxConfig`, Clearwing `hunter_sandbox.py` (image build + spawn, Docker SDK), Docker isolation guidance (`docker run --network none --read-only --tmpfs --memory --pids-limit --cap-drop ALL --security-opt no-new-privileges`), initrunner docker-sandbox docs (CLI `docker run --rm --init`, optional `runsc`).
|
||||
- **Findings**:
|
||||
- Context: Threat model already lists sandbox limits. Need a runner that does not invert zero-dep posture.
|
||||
- Sources Consulted: `docs/threat-model.md` (sandbox table), `config.py` `SandboxConfig`, Clearwing `hunter_sandbox.py` (image build + spawn, Docker SDK), Docker isolation guidance (`docker run --network none --read-only --tmpfs --memory --pids-limit --cap-drop ALL --security-opt no-new-privileges`), initrunner docker-sandbox docs (CLI `docker run --rm --init`, optional `runsc`).
|
||||
- Findings:
|
||||
- Python `docker` SDK would be a new core or extra dependency. The operator already has the Docker CLI if they can run containers.
|
||||
- Subprocess `docker run --rm --init` with bind-mount of the repo read-only and a tmpfs/scratch write area matches the threat model and keeps `dependencies = []`.
|
||||
- Unit tests must not require a daemon: inject a `SandboxRunner` protocol, ship a fake. Integration tests skip when `docker info` fails.
|
||||
- Full sanitizer-image builds (Clearwing) are out of scope for this spec. Stage 3 runs language-specific commands on a small pinned image per language family, or `inconclusive` if the recipe is missing.
|
||||
- **Implications**: `SandboxProbe` + `DockerCliRunner`. No SDK. gVisor/`runsc` is optional `security-opt` when `docker info` reports it; not required.
|
||||
- Implications: `SandboxProbe` + `DockerCliRunner`. No SDK. gVisor/`runsc` is optional `security-opt` when `docker info` reports it; not required.
|
||||
|
||||
### Evaluation corpora
|
||||
- **Context**: User rejected cheat-sheet fixtures. Current gate is ≥90% recall on those fixtures.
|
||||
- **Sources Consulted**: `benchmarks/fixtures/python-vulnerable/app.py` (line 77 split-sink comment), `javascript-vulnerable/src/app.js` (prototype pollution comment), `tests/test_detection_benchmarks.py`, `gate.py`.
|
||||
- **Findings**:
|
||||
- Context: User rejected cheat-sheet fixtures. Current gate is ≥90% recall on those fixtures.
|
||||
- Sources Consulted: `benchmarks/fixtures/python-vulnerable/app.py` (line 77 split-sink comment), `javascript-vulnerable/src/app.js` (prototype pollution comment), `tests/test_detection_benchmarks.py`, `gate.py`.
|
||||
- Findings:
|
||||
- Stage 1 smoke fixtures are still useful as unit tests. They must not be the stage 2/3 merge gate.
|
||||
- A split-sink corpus is small and owned: source and sink on different lines; mutations through wrappers/variables. Success = map ranks the true function in the top band; stage 3 triggers at least one stage-1 miss when Docker is present.
|
||||
- Comparing map ranking to "sort by pattern-hit count" is an anti-cheat-sheet test: if regex density is the map, the gate fails.
|
||||
- **Implications**: New `benchmarks/fixtures/split-sink-*` and a stage-2 ranking gate. Keep existing `openultrasast.gate` for stage 1 only.
|
||||
- Implications: New `benchmarks/fixtures/split-sink-*` and a stage-2 ranking gate. Keep existing `openultrasast.gate` for stage 1 only.
|
||||
|
||||
### Seams that already exist and must not be confused
|
||||
- **Context**: Brownfield traps for a spec named "map" that also adds Docker.
|
||||
- **Sources Consulted**: `skills.py` (`STAGES = ("map", "hunt", "verify", "fix")`), `mcp.py:68` (rejects `deep`), `reports.py:115` (`scan_exit_code`), `mapping.py` (no Java/Groovy AST), `config.py` (`SandboxConfig` parsed, never consumed by `_run_scan`), `rank.py` (`rank_targets_with_model` unused by CLI).
|
||||
- **Findings**:
|
||||
- Context: Brownfield traps for a spec named "map" that also adds Docker.
|
||||
- Sources Consulted: `skills.py` (`STAGES = ("map", "hunt", "verify", "fix")`), `mcp.py:68` (rejects `deep`), `reports.py:115` (`scan_exit_code`), `mapping.py` (no Java/Groovy AST), `config.py` (`SandboxConfig` parsed, never consumed by `_run_scan`), `rank.py` (`rank_targets_with_model` unused by CLI).
|
||||
- Findings:
|
||||
- Trail of Bits skill routing already uses a stage named `map` (mapping-discipline skills: Semgrep/CodeQL/sharp-edge). That is not the complexity map. `stages.py` must not reuse `skills.STAGES`.
|
||||
- MCP scan tools reject `deep` and forbid docker/shell. This spec does not widen the MCP surface; deep is CLI/CI only.
|
||||
- `--fail-on` is implemented in `scan_exit_code`, not only argparse. `worth-fixing` must be added there.
|
||||
- `--fail-on` is implemented in `scan_exit_code` as well as argparse. `worth-fixing` must be added there.
|
||||
- Java/Groovy entry points are tag-only (`line=None` → `inferred-file-surface`). Hotspot `function_name` will often be empty on Java; the map must still rank the file.
|
||||
- `[sandbox]`, `[dynamic]`, `[evidence]`, and `[models].ranker` are loaded into resolved config and then ignored. Stage 3 consumes `SandboxConfig`; it does not invent a parallel sandbox TOML.
|
||||
- **Implications**: Naming, MCP, fail-on, and Java identity are integration constraints, not new products.
|
||||
- Implications: Naming, MCP, fail-on, and Java identity are integration constraints, not new products.
|
||||
|
||||
### Evidence ladder honesty
|
||||
- **Context**: HarnessX hunter stamped LLM JSON as `static_corroboration`. Original spec forbids reporting LLM-only claims as verified.
|
||||
- **Sources Consulted**: `verification.py` (`verify_finding` accepts when reachability is not `unknown`), `hunter_harness.py` evidence_level assignment, openrouter requirements Evidence Ladder / Verification.
|
||||
- **Findings**:
|
||||
- Context: HarnessX hunter stamped LLM JSON as `static_corroboration`. Original spec forbids reporting LLM-only claims as verified.
|
||||
- Sources Consulted: `verification.py` (`verify_finding` accepts when reachability is not `unknown`), `hunter_harness.py` evidence_level assignment, openrouter requirements Evidence Ladder / Verification.
|
||||
- Findings:
|
||||
- Pattern hits as `static_corroboration` is consistent with the ladder. LLM hits as `suspicion` is the missing rule.
|
||||
- `--fail-on verified` today means reachable + static corroboration. That remains the cheap PR gate. `--fail-on worth-fixing` is the rugged gate.
|
||||
- **Implications**: Do not change CWE policy. Change hunter evidence. Add fail-on value. Report sections: Inventory, Complexity map, Worth fixing.
|
||||
- Implications: Do not change CWE policy. Change hunter evidence. Add fail-on value. Report sections: Inventory, Complexity map, Worth fixing.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
@@ -93,64 +93,64 @@
|
||||
## Design Decisions
|
||||
|
||||
### Decision D1: Modes map to stages
|
||||
- **Context**: Need a three-question scan without breaking CI that already passes `--mode`.
|
||||
- **Selected Approach**: `quick`→stage 1, `standard`→1+2, `deep`→1+2+3. Manifest records both.
|
||||
- **Rationale**: Evolutionary adoption of the CLI the project already shipped.
|
||||
- **Trade-offs**: Operators must learn that `standard` now means "map" not "regex hunter pool." README must say so.
|
||||
- Context: Need a three-question scan without breaking CI that already passes `--mode`.
|
||||
- Selected Approach: `quick`→stage 1, `standard`→1+2, `deep`→1+2+3. Manifest records both.
|
||||
- Rationale: Evolutionary adoption of the CLI the project already shipped.
|
||||
- Trade-offs: Operators must learn that `standard` now means "map" not "regex hunter pool." README must say so.
|
||||
|
||||
### Decision D2: Docker CLI subprocess, no SDK
|
||||
- **Context**: Zero-dep core vs isolated execution.
|
||||
- **Selected Approach**: `docker` binary via `subprocess`; probe with `docker info`; fake runner in unit tests.
|
||||
- **Rationale**: Operator runtime, not a Python dependency. Matches threat-model flags.
|
||||
- **Trade-offs**: Windows/macOS Docker Desktop variance; integration tests skip without a daemon.
|
||||
- Context: Zero-dep core vs isolated execution.
|
||||
- Selected Approach: `docker` binary via `subprocess`; probe with `docker info`; fake runner in unit tests.
|
||||
- Rationale: Operator runtime, not a Python dependency. Matches threat-model flags.
|
||||
- Trade-offs: Windows/macOS Docker Desktop variance; integration tests skip without a daemon.
|
||||
|
||||
### Decision D3: OpenRouter client for the tool hunter; HarnessX optional
|
||||
- **Context**: Last spec hid the LLM behind an extra CI does not install.
|
||||
- **Selected Approach**: Tool loop calls `OpenRouterChatClient` (or any `RankerClient`-shaped chat client) when `[models].hunter` is set. If HarnessX extra is present, `HxScanOrchestrator` may host the same tools; it is not required.
|
||||
- **Rationale**: Evolutionary use of code already in tree. Stdlib HTTP.
|
||||
- **Trade-offs**: Two possible runtimes until HarnessX is either used or ignored; tests mock the client.
|
||||
- Context: Last spec hid the LLM behind an extra CI does not install.
|
||||
- Selected Approach: Tool loop calls `OpenRouterChatClient` (or any `RankerClient`-shaped chat client) when `[models].hunter` is set. If HarnessX extra is present, `HxScanOrchestrator` may host the same tools; it is not required.
|
||||
- Rationale: Evolutionary use of code already in tree. Stdlib HTTP.
|
||||
- Trade-offs: Two possible runtimes until HarnessX is either used or ignored; tests mock the client.
|
||||
|
||||
### Decision D4: Complexity map is a new artifact
|
||||
- **Context**: Ranking already exists.
|
||||
- **Selected Approach**: `ComplexityMap` with `Hotspot` records. Ranking remains the hunter budget. Map is what humans and stage 3 consume.
|
||||
- **Rationale**: Different question (where to look / how to test vs which file to hunt first).
|
||||
- Context: Ranking already exists.
|
||||
- Selected Approach: `ComplexityMap` with `Hotspot` records. Ranking remains the hunter budget. Map is what humans and stage 3 consume.
|
||||
- Rationale: Different question (where to look / how to test vs which file to hunt first).
|
||||
|
||||
### Decision D5: Worth-fixing is a gate, not an evidence level
|
||||
- **Context**: Evidence ladder is already six rungs. Do not invent a seventh for CI.
|
||||
- **Selected Approach**: Verdicts `triggerable|not_triggerable|already_covered|inconclusive` → `worth_fixing|not_worth_fixing`. Evidence levels unchanged. Hunter stays at `suspicion`.
|
||||
- **Rationale**: CI owners need a fail-on that means "we proved it," without collapsing the ladder.
|
||||
- Context: Evidence ladder is already six rungs. Do not invent a seventh for CI.
|
||||
- Selected Approach: Verdicts `triggerable|not_triggerable|already_covered|inconclusive` → `worth_fixing|not_worth_fixing`. Evidence levels unchanged. Hunter stays at `suspicion`.
|
||||
- Rationale: CI owners need a fail-on that means "we proved it," without collapsing the ladder.
|
||||
|
||||
### Decision D6: Split-sink / mutation gate for stages 2–3
|
||||
- **Context**: 90% recall on cheat sheets is circular.
|
||||
- **Selected Approach**: New corpus + "beats pattern-hit ranking" test. Stage 1 smoke gate stays.
|
||||
- **Rationale**: Directly answers the user's evaluation complaint.
|
||||
- Context: 90% recall on cheat sheets is circular.
|
||||
- Selected Approach: New corpus + "beats pattern-hit ranking" test. Stage 1 smoke gate stays.
|
||||
- Rationale: Directly answers the user's evaluation complaint.
|
||||
|
||||
### Decision D7: Ledger overlays hotspot scores, not patterns
|
||||
- **Context**: Evolutionary feedback without repeating "loop must not edit patterns."
|
||||
- **Selected Approach**: `.openultrasast/calibration/complexity_ledger.json` keyed by `repo_fingerprint + path + function`. Triggerable boosts, not_triggerable demotes. Cannot hide reachable severity-5 inventory from worth-fixing consideration.
|
||||
- **Rationale**: Same overlay pattern as `rule_policy.json`, different object.
|
||||
- Context: Evolutionary feedback without repeating "loop must not edit patterns."
|
||||
- Selected Approach: `.openultrasast/calibration/complexity_ledger.json` keyed by `repo_fingerprint + path + function`. Triggerable boosts, not_triggerable demotes. Cannot hide reachable severity-5 inventory from worth-fixing consideration.
|
||||
- Rationale: Same overlay pattern as `rule_policy.json`, different object.
|
||||
|
||||
### Decision D8: Language recipes, not a universal executor
|
||||
- **Context**: Running untrusted Python vs C vs Java is different.
|
||||
- **Selected Approach**: Small recipe registry (Python and JavaScript required for MVP; C compile+run when a compiler exists in the image; Java `inconclusive` until a recipe ships). Missing recipe → `inconclusive`, not a crash.
|
||||
- **Rationale**: Simplification. Do not build Clearwing's sanitizer image matrix in this spec.
|
||||
- Context: Running untrusted Python vs C vs Java is different.
|
||||
- Selected Approach: Small recipe registry (Python and JavaScript required for MVP; C compile+run when a compiler exists in the image; Java `inconclusive` until a recipe ships). Missing recipe → `inconclusive`, not a crash.
|
||||
- Rationale: Simplification. Do not build Clearwing's sanitizer image matrix in this spec.
|
||||
|
||||
## Synthesis
|
||||
|
||||
- **Generalization**: Inventory → attention → proof is one pipeline with a `StagePlan`. Map, hints, and regress are processors on that plan, not three products.
|
||||
- **Build vs adopt**: Adopt Docker CLI and the in-tree OpenRouter client. Build the map, ledger, recipes, and split-sink corpus. Do not adopt Clearwing, docker SDK, or a cyclomatic-complexity library (nesting depth from indentation/braces is enough).
|
||||
- **Simplification**: No `--stage` flag, no HarnessX requirement, no fuzz campaign, no patch oracle, no Python Docker extra.
|
||||
- Generalization: Inventory → attention → proof is one pipeline with a `StagePlan`. Map, hints, and regress are processors on that plan, not three products.
|
||||
- Build vs adopt: Adopt Docker CLI and the in-tree OpenRouter client. Build the map, ledger, recipes, and split-sink corpus. Do not adopt Clearwing, docker SDK, or a cyclomatic-complexity library (nesting depth from indentation/braces is enough).
|
||||
- Simplification: No `--stage` flag, no HarnessX requirement, no fuzz campaign, no patch oracle, no Python Docker extra.
|
||||
|
||||
## Risks & Mitigations
|
||||
- Docker-in-Docker / rootless CI — probe, skip stage 3, degrade; do not fail stage 1.
|
||||
- Hunter cost on large repos — hotspot cap + existing `[harnessx] max_cost_usd` / hardening `max_findings` / new `max_regress_candidates`.
|
||||
- Untrusted snippet execution — structural deny (network, writes outside scratch, curl, docker socket mount). Never mount `/var/run/docker.sock`.
|
||||
- Stage 2 still heuristic without a model — allowed; gate is ranking quality on split-sink, which heuristics can pass if tags+reachability work.
|
||||
- Breaking `deep` from "exits" to "runs" — intended; document in README.
|
||||
- Docker-in-Docker / rootless CI, probe, skip stage 3, degrade; do not fail stage 1.
|
||||
- Hunter cost on large repos, hotspot cap + existing `[harnessx] max_cost_usd` / hardening `max_findings` / new `max_regress_candidates`.
|
||||
- Untrusted snippet execution, structural deny (network, writes outside scratch, curl, docker socket mount). Never mount `/var/run/docker.sock`.
|
||||
- Stage 2 still heuristic without a model, allowed; gate is ranking quality on split-sink, which heuristics can pass if tags+reachability work.
|
||||
- Breaking `deep` from "exits" to "runs", intended; document in README.
|
||||
|
||||
## References
|
||||
- OpenUltraSAST threat model — `docs/threat-model.md`
|
||||
- OpenRouter SAST harness requirements — Docker sandbox, hunter pool, evidence ladder
|
||||
- HarnessX self-improving rulesets — adjacent governance; do not reopen
|
||||
- Docker run isolation flags — `--network none`, `--read-only`, `--cap-drop ALL`, `--security-opt no-new-privileges`
|
||||
- Clearwing sourcehunt hunter tools and sandbox — implementation oracle, not a dependency
|
||||
- OpenUltraSAST threat model, `docs/threat-model.md`
|
||||
- OpenRouter SAST harness requirements, Docker sandbox, hunter pool, evidence ladder
|
||||
- HarnessX self-improving rulesets, adjacent governance; do not reopen
|
||||
- Docker run isolation flags, `--network none`, `--read-only`, `--cap-drop ALL`, `--security-opt no-new-privileges`
|
||||
- Clearwing sourcehunt hunter tools and sandbox, implementation oracle, not a dependency
|
||||
|
||||
@@ -6,7 +6,7 @@ AppSec scanning mixed working trees gets `language_unsupported` on JS/C/Java, an
|
||||
|
||||
## Current State
|
||||
|
||||
`parse_tree_sitter_cli` returns `None` even on a successful parse. Python falls back to stdlib `ast`. JS/C/Java stay unadjudicated. `quick` is correctly zero-dep. SAST overlay Youden on the 11-pair textbook slice is 0% vs +9% inventory — not because regex got worse, but because Python overlay fail-closes on ConfigParser-style heap and C/Java never parse.
|
||||
`parse_tree_sitter_cli` returns `None` even on a successful parse. Python falls back to stdlib `ast`. JS/C/Java stay unadjudicated. `quick` is correctly zero-dep. SAST overlay Youden on the 11-pair textbook slice is 0% vs +9% inventory, not because regex got worse, but because Python overlay fail-closes on ConfigParser-style heap and C/Java never parse.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
@@ -18,8 +18,8 @@ Add optional extra `openultrasast[semantic]` (`tree-sitter` + official language
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: extra, probe that prefers the Python API when importable, CST → FileIR, tests with and without the extra.
|
||||
- **Out**: Joern queries, inter-file taint, CWE-121 buffer semantics, new pair corpora (owned by `real-world-vfc-slice`).
|
||||
- In: extra, probe that prefers the Python API when importable, CST → FileIR, tests with and without the extra.
|
||||
- Out: Joern queries, inter-file taint, CWE-121 buffer semantics, new pair corpora (owned by `real-world-vfc-slice`).
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
@@ -32,13 +32,13 @@ Add optional extra `openultrasast[semantic]` (`tree-sitter` + official language
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `propose-adjudicate-prove` overlay records and taint
|
||||
- **Downstream**: `real-world-vfc-slice` overlay scoring on C/JS/Java pairs
|
||||
- Upstream: `propose-adjudicate-prove` overlay records and taint
|
||||
- Downstream: `real-world-vfc-slice` overlay scoring on C/JS/Java pairs
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: none (new spec). Does not reopen `propose-adjudicate-prove` tasks.
|
||||
- **Adjacent**: `three-stage-scan` modes; `quick` must remain zero-dep.
|
||||
- Extends: none (new spec). Does not reopen `propose-adjudicate-prove` tasks.
|
||||
- Adjacent: `three-stage-scan` modes; `quick` must remain zero-dep.
|
||||
|
||||
## Constraints
|
||||
|
||||
|
||||
@@ -116,13 +116,13 @@ tests/test_semantic_cst.py
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `pyproject.toml` — extra `semantic`; pytest marker; mypy overrides.
|
||||
- `src/openultrasast/semantic/ir.py` — dispatch; delete `parse_tree_sitter_cli` as success path.
|
||||
- `src/openultrasast/semantic/engines.py` — `tree_sitter_available` must not mean CLI-only; extra/grammar helpers live in `extra.py`.
|
||||
- `src/openultrasast/semantic/__init__.py` — export `has_semantic_extra` only if needed; do not import cst at package load.
|
||||
- `tests/test_semantic_overlay.py` — extra-free JS stays `language_unsupported` without monkeypatching a fake IR as “tree-sitter success”.
|
||||
- `.github/workflows/ci.yml` — second job with `--extra semantic` running `@pytest.mark.semantic`.
|
||||
- `dagger/ci.py` — keep extra-free unless the same extra-on command is cheap to add; default is GHA second job only.
|
||||
- `pyproject.toml`: extra `semantic`; pytest marker; mypy overrides.
|
||||
- `src/openultrasast/semantic/ir.py`: dispatch; delete `parse_tree_sitter_cli` as success path.
|
||||
- `src/openultrasast/semantic/engines.py`: `tree_sitter_available` must not mean CLI-only; extra/grammar helpers live in `extra.py`.
|
||||
- `src/openultrasast/semantic/__init__.py`: export `has_semantic_extra` only if needed; do not import cst at package load.
|
||||
- `tests/test_semantic_overlay.py`: extra-free JS stays `language_unsupported` without monkeypatching a fake IR as "tree-sitter success".
|
||||
- `.github/workflows/ci.yml`: second job with `--extra semantic` running `@pytest.mark.semantic`.
|
||||
- `dagger/ci.py`: keep extra-free unless the same extra-on command is cheap to add; default is GHA second job only.
|
||||
|
||||
Do not modify `taint.py`, `overlay.py`, `facts.py`, `gate.py`, `pairs.py`.
|
||||
|
||||
|
||||
@@ -8,9 +8,9 @@ AppSec engineers scanning mixed working trees need overlay adjudication on Pytho
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: Optional extra install; overlay using a structured parse for Python, JavaScript, C/C++, and Java when that extra provides a grammar; fail-closed `unadjudicated` when the extra or grammar is missing; `quick` unchanged and extra-free; existing overlay dispositions unchanged.
|
||||
- **Out of scope**: New pair corpora or Firefox/VFC harvest (`real-world-vfc-slice`); Joern CPG queries; inter-file or whole-program taint; buffer-size / CWE-121 semantics; language-model parse or taint; replacing pattern inventory.
|
||||
- **Adjacent expectations**: `propose-adjudicate-prove` still owns dispositions, facts, taint rules, and prove-budget. `three-stage-scan` still owns `quick` / `standard` / `deep` and the sandbox. This spec only supplies the structured parse those stages already asked for.
|
||||
- In scope: Optional extra install; overlay using a structured parse for Python, JavaScript, C/C++, and Java when that extra provides a grammar; fail-closed `unadjudicated` when the extra or grammar is missing; `quick` unchanged and extra-free; existing overlay dispositions unchanged.
|
||||
- Out of scope: New pair corpora or Firefox/VFC harvest (`real-world-vfc-slice`); Joern CPG queries; inter-file or whole-program taint; buffer-size / CWE-121 semantics; language-model parse or taint; replacing pattern inventory.
|
||||
- Adjacent expectations: `propose-adjudicate-prove` still owns dispositions, facts, taint rules, and prove-budget. `three-stage-scan` still owns `quick` / `standard` / `deep` and the sandbox. This spec only supplies the structured parse those stages already asked for.
|
||||
|
||||
## Requirements
|
||||
|
||||
@@ -21,7 +21,7 @@ AppSec engineers scanning mixed working trees need overlay adjudication on Pytho
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When OpenUltraSAST runs a `quick` scan, the system shall emit inventory proposals without requiring the semantic extra.
|
||||
2. The system shall not require the semantic extra to be installed in order to import or run the default OpenUltraSAST command.
|
||||
2. The system shall not require the semantic extra to be installed to import or run the default OpenUltraSAST command.
|
||||
3. If the semantic extra is absent, then a `quick` scan shall still complete and shall not abort because a parser package is missing.
|
||||
|
||||
### Requirement 2: Extra enables structured parse on in-scope languages
|
||||
@@ -30,7 +30,7 @@ AppSec engineers scanning mixed working trees need overlay adjudication on Pytho
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Where the semantic extra is installed and a grammar for the file’s language is available, when OpenUltraSAST runs a `standard` or `deep` scan, the system shall use that structured parse as the primary intra-file input to overlay adjudication for Python, JavaScript, C, C++, and Java.
|
||||
1. Where the semantic extra is installed and a grammar for the file's language is available, when OpenUltraSAST runs a `standard` or `deep` scan, the system shall use that structured parse as the primary intra-file input to overlay adjudication for Python, JavaScript, C, C++, and Java.
|
||||
2. When a JavaScript file contains `eval` of a request query (or the language-equivalent untrusted source to a known sink) and the extra provides a JavaScript grammar, the system shall be able to emit a `promote` overlay disposition for that proposal.
|
||||
3. When a C or Java file contains a sink fed only by a constant or a dominating sanitizer (for example a literal format string to `printf`) and the extra provides a grammar for that language, the system shall `demote` that proposal rather than leave it `language_unsupported`.
|
||||
4. The system shall feed the structured parse into the existing overlay dispositions (`promote`, `demote`, `unadjudicated`, `coverage`) and shall not introduce a new disposition.
|
||||
@@ -42,7 +42,7 @@ AppSec engineers scanning mixed working trees need overlay adjudication on Pytho
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. If the semantic extra is not installed, then when OpenUltraSAST adjudicates a JavaScript, C, C++, or Java file, the system shall leave matching inventory proposals `unadjudicated` with `language_unsupported` or `parse_failed`.
|
||||
2. If the extra is installed but no grammar is available for that file’s language, then the system shall leave those proposals `unadjudicated` with `language_unsupported` or `parse_failed` and shall not crash the scan.
|
||||
2. If the extra is installed but no grammar is available for that file's language, then the system shall leave those proposals `unadjudicated` with `language_unsupported` or `parse_failed` and shall not crash the scan.
|
||||
3. When the extra is absent, the system shall still try a stdlib parse only for Python and shall otherwise follow the fail-closed rule in criterion 1.
|
||||
4. If a file cannot be parsed even with a grammar present (syntax error or incomplete source), then the system shall leave matching proposals `unadjudicated` with `parse_failed`.
|
||||
5. The system shall not treat a missing extra as a failed `standard` scan by itself.
|
||||
@@ -55,7 +55,7 @@ AppSec engineers scanning mixed working trees need overlay adjudication on Pytho
|
||||
|
||||
1. When the extra supplies grammars for more than one in-scope language, the system shall extract overlay parse facts (assignments, calls, constants) through one shared structured-parse path rather than a separate visitor per language.
|
||||
2. The system shall not add a new language-specific stdlib or regex AST as the product parse engine for JavaScript, C, C++, or Java.
|
||||
3. Where a host parser CLI is present but the semantic extra is not, the system shall not claim a successful structured parse for overlay unless the extra’s grammar actually produced one.
|
||||
3. Where a host parser CLI is present but the semantic extra is not, the system shall not claim a successful structured parse for overlay unless the extra's grammar actually produced one.
|
||||
|
||||
### Requirement 5: Overlay and stages stay in their existing contracts
|
||||
|
||||
|
||||
@@ -26,12 +26,12 @@ Steering `product.md` / `tech.md` / `structure.md` are absent. Context is the ov
|
||||
- Optional native/heavy deps stay off the default install; capability probe must not import at module load (`harness_ext`, Docker sandbox probe).
|
||||
- mypy `ignore_missing_imports` for extras (`harnessx.*`).
|
||||
- pytest marker already exists for `docker`; extra-enabled tests would follow that skip pattern.
|
||||
- CI `checks` job: `uv sync --group dev` then pytest — **no optional extras**.
|
||||
- CI `checks` job: `uv sync --group dev` then pytest, **no optional extras**.
|
||||
|
||||
### Integration surfaces
|
||||
|
||||
- `parse_file(path, text, language) -> FileIR` is the only seam taint/overlay need.
|
||||
- `tree_sitter_available()` is a **presence** probe, not “grammar produced IR”. Host CLI on PATH makes it True even with no PyPI wheels (this environment has `/home/linuxbrew/.../tree-sitter` and **no languages**).
|
||||
- `tree_sitter_available()` is a **presence** probe, not "grammar produced IR". Host CLI on PATH makes it True even with no PyPI wheels (this environment has `/home/linuxbrew/.../tree-sitter` and **no languages**).
|
||||
- Pair eval already falls back to inventory when a language never promotes/demotes.
|
||||
|
||||
## 2. Requirements Feasibility
|
||||
@@ -60,23 +60,23 @@ Steering `product.md` / `tech.md` / `structure.md` are absent. Context is the ov
|
||||
|
||||
Put extra pins in `pyproject.toml`, implement API parse + CST walk inside `ir.py`, tighten `tree_sitter_available()`.
|
||||
|
||||
- **Pros:** Few files; `parse_file` stays the seam.
|
||||
- **Cons:** `ir.py` is already ~277 lines (Python ast + dead CLI). A walker + language table will bloat it and mix packaging probe with extract.
|
||||
- Pros: Few files; `parse_file` stays the seam.
|
||||
- Cons: `ir.py` is already ~277 lines (Python ast + dead CLI). A walker + language table will bloat it and mix packaging probe with extract.
|
||||
|
||||
### Option B: New components only
|
||||
|
||||
New `semantic/cst.py` + `semantic/extra.py`; leave `ir.py` as Python ast; overlay calls a new entrypoint.
|
||||
|
||||
- **Pros:** Clean extra guard.
|
||||
- **Cons:** Two parse entrypoints; easy to fork `parse_file` consumers.
|
||||
- Pros: Clean extra guard.
|
||||
- Cons: Two parse entrypoints; easy to fork `parse_file` consumers.
|
||||
|
||||
### Option C: Hybrid (recommended for design)
|
||||
|
||||
- **New:** `semantic/cst.py` — one walker: tree-sitter `Node` → `Bind` / `CallSite` / `FunctionIR` using a node-kind map.
|
||||
- **New or thin:** extra guard modeled on `harness_ext` (`has_semantic_extra`, lazy grammar load, never import wheels at module load).
|
||||
- **Extend:** `pyproject.toml` `[project.optional-dependencies] semantic = [...]`; mypy override `tree_sitter*`; pytest marker `semantic`; CI extra-free job unchanged; optional second job or skip.
|
||||
- **Extend:** `parse_file` prefers extra CST when a grammar loads; else Python ast; else `language_unsupported`. **Delete or stop calling** `parse_tree_sitter_cli` as a success path (Req 4.3).
|
||||
- **Do not extend:** `taint.py`, `overlay.py`, facts, gate, pair catalog.
|
||||
- New: `semantic/cst.py`: one walker: tree-sitter `Node` → `Bind` / `CallSite` / `FunctionIR` using a node-kind map.
|
||||
- New or thin: extra guard modeled on `harness_ext` (`has_semantic_extra`, lazy grammar load, never import wheels at module load).
|
||||
- Extend: `pyproject.toml` `[project.optional-dependencies] semantic = [...]`; mypy override `tree_sitter*`; pytest marker `semantic`; CI extra-free job unchanged; optional second job or skip.
|
||||
- Extend: `parse_file` prefers extra CST when a grammar loads; else Python ast; else `language_unsupported`. **Delete or stop calling** `parse_tree_sitter_cli` as a success path (Req 4.3).
|
||||
- Do not extend: `taint.py`, `overlay.py`, facts, gate, pair catalog.
|
||||
|
||||
**Trade-offs:** Extra files, but `ir.py` shrinks (drop dead CLI), taint stays language-agnostic, extra follows an existing packaging seam.
|
||||
|
||||
@@ -89,7 +89,7 @@ New `semantic/cst.py` + `semantic/extra.py`; leave `ir.py` as Python ast; overla
|
||||
|
||||
- Prefer **Option C**.
|
||||
- Treat `FileIR` as frozen contract; do not add dispositions.
|
||||
- Decide probe semantics: `tree_sitter_available` should mean “can produce FileIR for this language”, not “CLI binary exists”.
|
||||
- Decide probe semantics: `tree_sitter_available` should mean "can produce FileIR for this language", not "CLI binary exists".
|
||||
- Carry research items 1–4 into design (ABI pins, node-kind map, cpp package, CI job vs skip).
|
||||
- Do not pull Joern, pair harvest, or CWE-121 into this spec.
|
||||
|
||||
@@ -97,9 +97,9 @@ New `semantic/cst.py` + `semantic/extra.py`; leave `ir.py` as Python ast; overla
|
||||
|
||||
## Summary (design discovery)
|
||||
|
||||
- **Feature**: tree-sitter-overlay-extra
|
||||
- **Discovery Scope**: Extension of `parse_file` / overlay IR; optional extra packaging
|
||||
- **Key Findings**:
|
||||
- Feature: tree-sitter-overlay-extra
|
||||
- Discovery Scope: Extension of `parse_file` / overlay IR; optional extra packaging
|
||||
- Key Findings:
|
||||
- Official PyPI wheels: `tree-sitter` (0.26), `tree-sitter-python`, `tree-sitter-javascript` (0.25), `tree-sitter-c` (0.24.x), `tree-sitter-cpp` (0.23.4), `tree-sitter-java`. API is `Parser(Language(mod.language()))`; `build_library` is gone.
|
||||
- `FileIR` is the only overlay seam; taint does not need language visitors.
|
||||
- Host CLI presence must not count as a successful overlay parse (Req 4.3).
|
||||
@@ -108,16 +108,16 @@ New `semantic/cst.py` + `semantic/extra.py`; leave `ir.py` as Python ast; overla
|
||||
|
||||
### PyPI ABI and load API
|
||||
|
||||
- **Context**: Pin wheels that work on Python 3.11+ without compiling grammars in CI.
|
||||
- **Sources Consulted**: https://pypi.org/project/tree-sitter/ ; https://pypi.org/project/tree-sitter-javascript/ ; https://pypi.org/project/tree-sitter-cpp/ ; py-tree-sitter 0.23 release notes (Language capsule, Parser constructor).
|
||||
- **Findings**: Grammars ship abi3 wheels. Load with `Language(tree_sitter_<lang>.language())` then `Parser(language)`. Do not use `Language.build_library`.
|
||||
- **Implications**: Extra lists core + five grammar packages. mypy ignores `tree_sitter*`.
|
||||
- Context: Pin wheels that work on Python 3.11+ without compiling grammars in CI.
|
||||
- Sources Consulted: https://pypi.org/project/tree-sitter/ ; https://pypi.org/project/tree-sitter-javascript/ ; https://pypi.org/project/tree-sitter-cpp/ ; py-tree-sitter 0.23 release notes (Language capsule, Parser constructor).
|
||||
- Findings: Grammars ship abi3 wheels. Load with `Language(tree_sitter_<lang>.language())` then `Parser(language)`. Do not use `Language.build_library`.
|
||||
- Implications: Extra lists core + five grammar packages. mypy ignores `tree_sitter*`.
|
||||
|
||||
### Probe vs grammar
|
||||
|
||||
- **Context**: `tree_sitter_available()` is true when CLI is on PATH with zero grammars.
|
||||
- **Findings**: CLI parse cannot populate `FileIR` without a walker. Extra Python API is the product parse.
|
||||
- **Implications**: Split `has_semantic_extra()` from per-language `grammar_for`. Remove CLI as a success path.
|
||||
- Context: `tree_sitter_available()` is true when CLI is on PATH with zero grammars.
|
||||
- Findings: CLI parse cannot populate `FileIR` without a walker. Extra Python API is the product parse.
|
||||
- Implications: Split `has_semantic_extra()` from per-language `grammar_for`. Remove CLI as a success path.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
@@ -131,37 +131,37 @@ New `semantic/cst.py` + `semantic/extra.py`; leave `ir.py` as Python ast; overla
|
||||
|
||||
### Decision: Adopt official grammar wheels, build one walker
|
||||
|
||||
- **Context**: Avoid per-language AST visitors (Req 4).
|
||||
- **Alternatives**: tree-sitter CLI sexp; py-tree-sitter-languages (unmaintained); hand-rolled JS/C parsers.
|
||||
- **Selected**: `openultrasast[semantic]` = `tree-sitter` plus python/javascript/c/cpp/java grammar packages. One CST walk keyed by a node-kind map.
|
||||
- **Rationale**: Wheels are MIT, maintained, match in-scope languages. Taint stays language-agnostic.
|
||||
- **Trade-offs**: Native wheels in the extra; ABI pins. Core install stays empty.
|
||||
- **Follow-up**: Verify Parser constructor against the pinned `tree-sitter` minor at implement time.
|
||||
- Context: Avoid per-language AST visitors (Req 4).
|
||||
- Alternatives: tree-sitter CLI sexp; py-tree-sitter-languages (unmaintained); hand-rolled JS/C parsers.
|
||||
- Selected: `openultrasast[semantic]` = `tree-sitter` plus python/javascript/c/cpp/java grammar packages. One CST walk keyed by a node-kind map.
|
||||
- Rationale: Wheels are MIT, maintained, match in-scope languages. Taint stays language-agnostic.
|
||||
- Trade-offs: Native wheels in the extra; ABI pins. Core install stays empty.
|
||||
- Follow-up: Verify Parser constructor against the pinned `tree-sitter` minor at implement time.
|
||||
|
||||
### Decision: Extra-free CI stays default; extra-on is a second job plus skip marker
|
||||
|
||||
- **Context**: Req 6 dual matrix.
|
||||
- **Selected**: pytest `semantic` marker skips when extra absent. GitHub Actions adds a job `uv sync --extra semantic --group dev` that runs marked tests. Default `checks` job unchanged.
|
||||
- **Rationale**: Default CI must not fail without wheels. Extra-on tests must actually run somewhere.
|
||||
- **Trade-offs**: Two pytest invocations in CI.
|
||||
- Context: Req 6 dual matrix.
|
||||
- Selected: pytest `semantic` marker skips when extra absent. GitHub Actions adds a job `uv sync --extra semantic --group dev` that runs marked tests. Default `checks` job unchanged.
|
||||
- Rationale: Default CI must not fail without wheels. Extra-on tests must actually run somewhere.
|
||||
- Trade-offs: Two pytest invocations in CI.
|
||||
|
||||
### Decision: has_error implies parse_failed
|
||||
|
||||
- **Context**: tree-sitter is error-tolerant; Req 3.4 wants fail-closed on syntax errors.
|
||||
- **Selected**: If `tree.root_node.has_error` (or ERROR children), return `FileIR(parse_ok=False, reason=parse_failed)`.
|
||||
- **Rationale**: Partial IR would invite false demote.
|
||||
- Context: tree-sitter is error-tolerant; Req 3.4 wants fail-closed on syntax errors.
|
||||
- Selected: If `tree.root_node.has_error` (or ERROR children), return `FileIR(parse_ok=False, reason=parse_failed)`.
|
||||
- Rationale: Partial IR would invite false demote.
|
||||
|
||||
## Synthesis
|
||||
|
||||
- **Generalization**: One `parse_file` dispatch; extra is a grammar source, not a second overlay.
|
||||
- **Build vs adopt**: Adopt tree-sitter wheels; build only the FileIR projection (binds/calls).
|
||||
- **Simplification**: No CLI sexp walker; no second parse API; no taint/overlay changes.
|
||||
- Generalization: One `parse_file` dispatch; extra is a grammar source, not a second overlay.
|
||||
- Build vs adopt: Adopt tree-sitter wheels; build only the FileIR projection (binds/calls).
|
||||
- Simplification: No CLI sexp walker; no second parse API; no taint/overlay changes.
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- Grammar ABI mismatch — pin extras together; extra-on CI job catches import/parse failure.
|
||||
- Node-type drift — table in cst.py; tests assert JS eval and C printf on real trees.
|
||||
- CLI still on PATH — must not set parse_ok.
|
||||
- Grammar ABI mismatch: pin extras together; extra-on CI job catches import/parse failure.
|
||||
- Node-type drift: table in cst.py; tests assert JS eval and C printf on real trees.
|
||||
- CLI still on PATH: must not set parse_ok.
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Brief: zero-with-a-reason — a finding count is meaningless without a denominator
|
||||
# Brief: zero-with-a-reason, a finding count is meaningless without a denominator
|
||||
|
||||
> **Status 2026-10-02:** dormant. Brief only and unapproved; not touched since 2026-09-10.
|
||||
|
||||
@@ -33,7 +33,7 @@ produced. There is nothing to put on a ladder.
|
||||
**Every zero must be decomposable, and every scan must be able to prove it did work.** Three mechanisms,
|
||||
independent, in order of value per line of code.
|
||||
|
||||
### 1. The funnel — make the denominator first-class
|
||||
### 1. The funnel, make the denominator first-class
|
||||
|
||||
The scan is already a chain of narrowing stages, each of which computes a count and throws it away:
|
||||
|
||||
@@ -57,10 +57,10 @@ Cheap, because every number already exists at the moment it is computed. `ModelS
|
||||
the coverage disclosure (Req 5.6, already written) is where it prints. The hard rule that gives it teeth: a
|
||||
report may not describe a repository as clean without the funnel beside it.
|
||||
|
||||
### 2. The dumb oracle — an independent lower bound on the graph
|
||||
### 2. The dumb oracle, an independent lower bound on the graph
|
||||
|
||||
The CPG is an elaborate instrument. A regex over the source text is a stupid one, and stupid instruments
|
||||
fail differently — which is the entire reason to have two.
|
||||
fail differently, which is the entire reason to have two.
|
||||
|
||||
Before querying, count the fact tables' sink names in the raw source text. That is a floor on what the graph
|
||||
must contain:
|
||||
@@ -72,13 +72,13 @@ the CPG's sink match returns 0
|
||||
```
|
||||
|
||||
This catches an empty graph, the wrong frontend, dropped files, a broken name matcher, and a missing
|
||||
overlay — bugs 2, 3, 5, 6 and 7 above, none of which it needs to know about in advance. It is the same trick
|
||||
overlay (bugs 2, 3, 5, 6 and 7 above), none of which it needs to know about in advance. It is the same trick
|
||||
the pair corpus already uses: two independent measurements of one thing must agree.
|
||||
|
||||
What it cannot do: say anything about over-reporting, or about flows. It is purely "does the graph contain
|
||||
the syntax the text contains". That narrowness is why it is trustworthy.
|
||||
|
||||
### 3. The canary — an end-to-end positive control
|
||||
### 3. The canary, an end-to-end positive control
|
||||
|
||||
One synthetic file with an unmistakable flow, one extra request in the batch:
|
||||
|
||||
@@ -91,7 +91,7 @@ ran, the file reached the graph, overlays applied, the fact tables loaded, the q
|
||||
round-tripped, the payload parsed, the arbiter ran and the ladder assigned a rung.
|
||||
|
||||
With one honest caveat that must be designed around: **adding a file to a PHP build can itself trigger the
|
||||
php2cpg race** — a two-line file added to `class.memberorder.php` is exactly what emptied that graph. So on
|
||||
php2cpg race**. A two-line file added to `class.memberorder.php` is exactly what emptied that graph. So on
|
||||
PHP the canary belongs in its own build (`ousast doctor`), where it proves the toolchain works, and the
|
||||
funnel and the oracle prove that *this* build worked. They are complementary, not redundant.
|
||||
|
||||
@@ -101,7 +101,7 @@ Every instrument failure in this project has the same shape, and it is not a coi
|
||||
unreadable and the tool reported a plausible zero.** `file_exists()` returns false, the parser writes nothing,
|
||||
the frontend exits 0, and "0 files dropped, 5,703-byte graph" reads exactly like success.
|
||||
|
||||
It has now happened four times in one day — a docker shim that did not mount `~/.cache` (twice, hours apart,
|
||||
It has now happened four times in one day. The failures were a docker shim that did not mount `~/.cache` (twice, hours apart,
|
||||
the second time after I had diagnosed the first), a measurement harness whose script path did not resolve
|
||||
from its working directory, and a container that could not follow the symlinks in the tree it was handed.
|
||||
|
||||
@@ -132,15 +132,15 @@ prevents recurrence rather than detecting it.
|
||||
|
||||
## What this is not, and must not become
|
||||
|
||||
- **Not a confidence score.** The rung ladder exists precisely so uncertainty is carried as a claim about
|
||||
- Not a confidence score. The rung ladder exists precisely so uncertainty is carried as a claim about
|
||||
evidence rather than a number nobody can act on. A "coverage percentage" that averages the funnel away
|
||||
would undo it.
|
||||
- **Not a retry loop.** Retrying a failed build until it succeeds hides bug 6's race instead of reporting it.
|
||||
- Not a retry loop. Retrying a failed build until it succeeds hides bug 6's race instead of reporting it.
|
||||
Three of five builds succeeding is a fact about the toolchain and belongs in the output.
|
||||
- **Not anomaly detection over historical finding counts.** 6.1's baselines already give the diff, and
|
||||
"findings dropped 90%" is a lagging indicator that needs a known-good prior run — which is exactly what a
|
||||
- Not anomaly detection over historical finding counts. 6.1's baselines already give the diff, and
|
||||
"findings dropped 90%" is a lagging indicator that needs a known-good prior run, which is exactly what a
|
||||
first scan of a new repository does not have.
|
||||
- **Not more code than it saves.** The funnel is plumbing that already exists, the oracle is one pass over
|
||||
- Not more code than it saves. The funnel is plumbing that already exists, the oracle is one pass over
|
||||
files the preprocessor already reads, the canary is one file and one request.
|
||||
|
||||
## Open questions for the design phase
|
||||
|
||||
Reference in New Issue
Block a user