mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
merge: legacy cleanup -- audit by real imports, dead modules removed, unread config retired, archives, dockerignore, comments
Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
@@ -0,0 +1,27 @@
|
||||
# Build context for the root Dockerfile. The image copies pyproject.toml, README.md, src/, benchmarks/ and
|
||||
# ops/frontend-retention/install.py; everything below is either never copied or is evidence the image does not
|
||||
# need. plane/Dockerfile.runner has its own plane/Dockerfile.runner.dockerignore.
|
||||
|
||||
# Local state, never part of an image (agent worktrees alone run to hundreds of MB; .env holds keys).
|
||||
.git
|
||||
.claude
|
||||
.venv
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
**/__pycache__
|
||||
**/*.pyc
|
||||
.mypy_cache
|
||||
.ruff_cache
|
||||
.pytest_cache
|
||||
.openultrasast
|
||||
reports
|
||||
*.sarif
|
||||
|
||||
# The measurement archive (about 20 MB) is evidence, not runtime input. These records stay:
|
||||
# ops/smoke_report.py reads the two 2026-09-13 ones from /app inside the image (ops/README.md).
|
||||
benchmarks/measurements/*
|
||||
!benchmarks/measurements/2026-09-13-delta-comparison-smoke.json
|
||||
!benchmarks/measurements/2026-09-13-admission-policy-smoke.json
|
||||
# A label source of `ousast learn` (src/openultrasast/learn/sources.toml).
|
||||
!benchmarks/measurements/2026-09-30-harnessx-removal-baseline/pairs.json
|
||||
@@ -1,5 +1,7 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Status 2026-10-02:** dormant. Tasks were approved but none of the 24 was started; the absence-bug case it targets is handled in practice by the dominance arbiter (`model/dominance.py`) and `semantic/obligations/`. Reopen it only with a measured gap the arbiter misses.
|
||||
|
||||
## Introduction
|
||||
|
||||
AppSec engineers and maintainers of OpenUltraSAST scan web code, increasingly vibe-coded or agent-written, in which the dominant bugs are absences rather than flows: a handler queries a user-owned record without constraining it to the caller, a privileged route has no guard on its path, an identity is read from the request body instead of the authenticated context, a security control stays at its permissive default. Today the detector, the overlay, and the variant search model only source-to-sink flows, so these pairs score zero and cannot teach a mechanism (6 of 19 non-teaching vibe-py pairs; 15 of 29 agent-vfc rows carry absence labels). This spec adds obligations: an operation that must be discharged by a guard, an identity constraint, or a safe value before it is reached. The application's own policy, recovered from the consistency of its sibling handlers and optionally declared in a versioned file, says which operations carry which obligations. Violations are reported with the missing discharger and its evidence, measured leave-one-out on the absence pairs, and admitted into the detector through the existing improve-loop gate.
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Status 2026-10-02:** superseded by `model-grounded-detection` (spec.json `superseded_by`). 4 of 20 tasks were done and the tasks were never approved; none will be. Kept as decision history; nothing in src, tests or CI points here.
|
||||
|
||||
## Introduction
|
||||
|
||||
The detector's instability is structural, not statistical. Measured on 2026-09-06 over twelve injection pairs at
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Status 2026-10-02:** complete, 17 of 17 tasks (spec.json phase corrected to `implementation-complete`). Its `[variants]` config keys were retired on 2026-10-02: the variant search they configured (`semantic/variants.py`) had already been deleted.
|
||||
|
||||
## Introduction
|
||||
|
||||
OpenUltraSAST's labeled pair corpus measures the detector but does not improve it, and its mechanism memory is empty because only sandbox-proven findings may write to it. This spec turns every trusted pair into a mechanism record, adds a structural variant search that proposes matches on any scanned tree, measures the corpus's own detection rate leave-one-out, and gives the improve loop a lever that can raise recall under the existing false-positive ceiling and per-profile holdout gate.
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
{
|
||||
"feature_name": "corpus-seeded-mechanisms",
|
||||
"created_at": "2026-09-05T06:18:31Z",
|
||||
"updated_at": "2026-09-05T16:30:00Z",
|
||||
"updated_at": "2026-10-02T00:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"phase": "implementation-complete",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Brief: flow-aware-ranking — spend the expensive query where the cheap evidence is strongest
|
||||
|
||||
> **Status 2026-10-02:** dormant. Brief only and unapproved; its 2026-09-12 amendment placed ranking inside the pre-push safety net, where the ranker now lives.
|
||||
|
||||
**Status:** brief only, unapproved.
|
||||
|
||||
## Current amendment — 2026-09-12
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Status 2026-10-02:** retired with HarnessX (see the note below). Its task 6.3 modules `stage_processors.py` and `slot_contract.py` were deleted on 2026-10-02: nothing imported them. Kept as decision history; nothing in src, tests or CI points here.
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Brief: ir-consolidation — stop maintaining an IR we did not need, and adopt one we do
|
||||
|
||||
> **Status 2026-10-02:** dormant. Brief only and unapproved. The subtraction it asks for still needs a measured recall comparison before the flat IR goes.
|
||||
|
||||
**Status:** brief only, unapproved.
|
||||
|
||||
## Why now
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Status 2026-10-02:** retired, superseded by `model-grounded-detection` (spec.json `superseded_by`). 30 of 33 tasks were done; the open three will not be. Kept as decision history; nothing in src, tests or CI points here.
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Requirements Document: model-grounded-detection
|
||||
|
||||
> **Status 2026-10-02:** complete, 16 of 16 tasks (spec.json phase corrected to `implementation-complete`). On 2026-10-02 three of its modules were deleted because nothing imported them: `model/judge.py` (Req 8; `model/pipeline.py` does the judging), `model/calibrate.py` (Req 9; the decision engine's `learn/calibrate.py` is the live calibration) and `model/execution.py`. Deleting `execution.py` closes the Req 10 execution-tier option: a future execution tier is a new design, not this seam.
|
||||
|
||||
## Introduction
|
||||
|
||||
This feature supersedes `learning-harness` and `constrained-detector`. It replaces the assumption that the LLM
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
{
|
||||
"feature_name": "model-grounded-detection",
|
||||
"created_at": "2026-09-08T00:00:00Z",
|
||||
"updated_at": "2026-09-08T10:00:00Z",
|
||||
"updated_at": "2026-10-02T00:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"phase": "implementation-complete",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Requirements: OpenRouter SAST Harness
|
||||
|
||||
> **Status 2026-10-02:** retired (see the note below). Its OpenCode skill `openultrasast-kiro-impl`, which still sent agents to this tasks.md, was deleted. Kept as decision history; nothing in src, tests or CI points here.
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"feature_name": "openrouter-sast-harness",
|
||||
"language": "en",
|
||||
"status": "retired",
|
||||
"updated_at": "2026-10-02T00:00:00Z",
|
||||
"retired_note": "2026-09-30: HarnessX was removed by harnessx-removal; this spec had no spec.json before 2026-10-02 and is kept as decision history."
|
||||
}
|
||||
@@ -1,5 +1,7 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Status 2026-10-02:** complete, 32 of 32 tasks (spec.json phase corrected to `implementation-complete`).
|
||||
|
||||
## Introduction
|
||||
|
||||
Engineers evolving OpenUltraSAST's overlay cannot trust the pair scoreboard: coverage detections are dropped, vfc labels mostly say `sink = "unknown"`, a quarter of the vfc corpus fails to parse because excerpts start mid-comment, two sast pairs are unlabelable as vendored, and there is no Node/TypeScript/Python web corpus and no agent-authored corpus. This spec makes the scoreboard honest per slice, per mechanism, and per provenance, adds three web and agent-authored slices, and lets the improve loop accept a change only when no provenance profile regresses. It supersedes `real-world-vfc-slice` for the corpus and scorer plane.
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
{
|
||||
"feature_name": "pair-corpus-honesty",
|
||||
"created_at": "2026-09-04T16:12:17Z",
|
||||
"updated_at": "2026-09-05T12:30:00Z",
|
||||
"updated_at": "2026-10-02T00:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"phase": "implementation-complete",
|
||||
"supersedes": [
|
||||
"real-world-vfc-slice"
|
||||
],
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Status 2026-10-02:** complete, 22 of 22 tasks (spec.json phase corrected to `implementation-complete`).
|
||||
|
||||
## Introduction
|
||||
|
||||
OpenUltraSAST users scan **working trees they are writing**: mixed files, incomplete edits, generated code, framework wrappers, and no labels. Today the harness still *decides* with pattern inventory. The complexity map ranks; the sandbox can run; neither adjudicates “this sink is fed by a source” versus “this sink is hardcoded, sanitized, or unreadable.”
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
{
|
||||
"feature_name": "propose-adjudicate-prove",
|
||||
"created_at": "2026-09-04T10:58:28Z",
|
||||
"updated_at": "2026-09-04T12:00:00Z",
|
||||
"updated_at": "2026-10-02T00:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"phase": "implementation-complete",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Status 2026-10-02:** dormant. 23 tasks generated, nothing approved; adopting Joern (`cpg/`, `model/taint.py`) delivered most of what it asked for. Reopen it only for a reachability gap Joern's taint does not cover.
|
||||
|
||||
## Introduction
|
||||
|
||||
OpenUltraSAST cannot yet steer effort where risk is reduced because it has no model of how an entry point reaches a sink. Reachability today means "inside a handler," taint is one hop inside one file, the LLM hunter cannot create a sandbox candidate, and the improve loop cannot add the facts that would raise recall. This spec adds a call graph and composed taint summaries over the existing `FunctionIR` records, attaches path-based reachability to findings and ranking, puts the hunter on paths with one checkable question per gap, turns model answers into gated facts, and ranks fix points by reachable risk removed. Concolic execution is out of scope; the sandbox remains the only place target code runs.
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Status 2026-10-02:** superseded by `pair-corpus-honesty` (spec.json `superseded_by`); its 12 tasks were done and the vfc pairs live on in that spec's corpus. Kept as decision history; nothing in src, tests or CI points here.
|
||||
|
||||
## Introduction
|
||||
|
||||
AppSec engineers cannot tell whether OpenUltraSAST improved on real product code. The labeled pair slices today are in-tree fixtures (`local`), small GitHub/SVEN functions (`github`), and textbook OWASP/Juliet files (`sast`). Operators care whether the harness fires on a real buggy snapshot of OpenSSL, Firefox, or Chromium and stays quiet after the patch. This spec adds a reviewed, isolated-function **vfc** slice drawn from public vulnerability-fixing commits in those projects, scored as an honesty dashboard (not a merge gate), so the tool can be evolved against labeled product code instead of cheat-sheet recall.
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Status 2026-10-02:** retired (see the note below); all 31 tasks were done before HarnessX was removed. Kept as decision history; nothing in src, tests or CI points here.
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Status 2026-10-02:** complete, 13 of 13 tasks (spec.json phase corrected from `tasks-generated` to `implementation-complete`).
|
||||
|
||||
## Introduction
|
||||
|
||||
AppSec engineers scanning mixed working trees need overlay adjudication on Python, JavaScript, C/C++, and Java, not only on files a Python stdlib parse can read. Today a `standard` or `deep` scan leaves JS/C/Java inventory as `language_unsupported` even when a host parser CLI is present, so those proposals never promote or demote. This spec makes an **optional semantic extra** the way to turn on a real structured parse for those languages, while `quick` stays usable with no extra packages and a missing extra stays fail-closed.
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
{
|
||||
"feature_name": "tree-sitter-overlay-extra",
|
||||
"created_at": "2026-09-04T14:00:00Z",
|
||||
"updated_at": "2026-09-04T15:00:00Z",
|
||||
"updated_at": "2026-10-02T00:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"phase": "implementation-complete",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Brief: zero-with-a-reason — a finding count is meaningless without a denominator
|
||||
|
||||
> **Status 2026-10-02:** dormant. Brief only and unapproved; not touched since 2026-09-10.
|
||||
|
||||
**Status:** brief only, unapproved.
|
||||
|
||||
## Why now
|
||||
|
||||
@@ -194,18 +194,26 @@ function-level pairs could not see any of them, and a repository showed all nine
|
||||
|
||||
## Where the specs stand
|
||||
|
||||
Counted 2026-10-02 from each spec's `tasks.md` checkboxes and `spec.json`.
|
||||
|
||||
| spec | phase | state |
|
||||
|---|---|---|
|
||||
| `contributor-scan` | implementation | 14 done, 15 open — **the active one** |
|
||||
| `learned-decision-engine` | implementation | 11 of 17 tasks done |
|
||||
| `pre-push-safety-net` | implementation | 67 of 79 tasks done |
|
||||
| `contributor-scan` | implementation | 27 of 51 tasks done |
|
||||
| `ai-service-plane` | implementation | 11 of 11 tasks done; the ax plane is the agentic path |
|
||||
| `harnessx-removal` | complete | 8 of 8; HarnessX removed 2026-09-30 |
|
||||
| `finding-feedback-loop` | design | requirements approved, design awaiting approval |
|
||||
| `model-grounded-detection` | complete | 16/16; built the arbiter this all rests on |
|
||||
| `authorization-obligations` | tasks approved | 24 open — superseded in practice by the dominance arbiter |
|
||||
| `constrained-detector` | tasks | 4 of 20, tasks unapproved |
|
||||
| `reachability-flow-model` | tasks generated | 23 open, nothing approved — largely delivered by adopting Joern |
|
||||
| `learning-harness` | implementation | 30 done, 3 open |
|
||||
| six others | complete | `pair-corpus-honesty`, `corpus-seeded-mechanisms`, `propose-adjudicate-prove`, `three-stage-scan`, `real-world-vfc-slice`, `harnessx-self-improving-rulesets`, `tree-sitter-overlay-extra` |
|
||||
| `model-grounded-detection` | complete | 16 of 16; built the arbiter this all rests on |
|
||||
| `authorization-obligations` | dormant | tasks approved, 0 of 24 started; superseded in practice by the dominance arbiter |
|
||||
| `reachability-flow-model` | dormant | 23 tasks generated, nothing approved; largely delivered by adopting Joern |
|
||||
| `flow-aware-ranking`, `zero-with-a-reason`, `ir-consolidation` | dormant | brief only, unapproved |
|
||||
| four others | complete | `pair-corpus-honesty`, `corpus-seeded-mechanisms`, `propose-adjudicate-prove`, `tree-sitter-overlay-extra` |
|
||||
| six others | retired or superseded | `openrouter-sast-harness`, `three-stage-scan`, `harnessx-self-improving-rulesets`, `learning-harness` (retired with HarnessX), `constrained-detector`, `real-world-vfc-slice` (superseded) |
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax (`ai-service-plane`),
|
||||
> and evolutionary proposals come from the plane memory. The HarnessX specs in the table above are history.
|
||||
|
||||
As of 2026-09-09: 97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests.
|
||||
150 source modules, 145 load-bearing and 5 standalone capabilities, 0 orphaned
|
||||
(`benchmarks/measurements/2026-09-08-module-audit.json`, regenerated 2026-10-02); 1,964 tests collected.
|
||||
|
||||
@@ -90,7 +90,7 @@ supersede this accepted sequence. Blocking deployment follows a later explicit d
|
||||
The sections below preserve prior plans and measurements. Statements about the current bottleneck,
|
||||
PHP coverage, rollout order or feature readiness are superseded by the current section above.
|
||||
|
||||
**Rewritten 2026-09-09.** The previous version (kept as `roadmap-2026-09-04.md`) described a pre-Joern world —
|
||||
**Rewritten 2026-09-09.** The previous version (kept as `.kiro/archive/steering/roadmap-2026-09-04.md`, outside steering) described a pre-Joern world —
|
||||
"adjudication is still Python-`ast` plus a tree-sitter CLI stub", "Joern-as-required deferred". Joern was
|
||||
adopted, the model layer was built, wired and measured, and two repositories have been scanned. See
|
||||
`overview.md` for what is true today; this is what happens next and in what order.
|
||||
|
||||
@@ -1,26 +0,0 @@
|
||||
---
|
||||
name: openultrasast-kiro-impl
|
||||
description: OpenUltraSAST Kiro implementation steering. Use when continuing `.kiro/specs/openrouter-sast-harness/tasks.md`, choosing the next phase, updating checkboxes, or keeping implementation aligned with the spec.
|
||||
---
|
||||
|
||||
# OpenUltraSAST Kiro Implementation
|
||||
|
||||
Use this skill when implementing the OpenUltraSAST Kiro spec from opencode.
|
||||
|
||||
## Rules
|
||||
|
||||
1. Read `.kiro/specs/openrouter-sast-harness/tasks.md` before choosing work.
|
||||
2. Implement the next unchecked task slice unless the user names a different phase.
|
||||
3. Keep each slice independently verifiable with `uv run pytest`.
|
||||
4. Update task checkboxes only after implementation and tests pass.
|
||||
5. Prefer minimal harness-compatible primitives over large framework imports.
|
||||
6. Keep OpenUltraSAST independent of Clearwing; use Clearwing only as an oracle for design patterns.
|
||||
7. Use project-local opencode skills to steer scan, triage, and fix workflows.
|
||||
|
||||
## Current Phase Order
|
||||
|
||||
- Phase 5: Harness runtime events, processors, contracts, and traces.
|
||||
- Phase 5A: Static mapping disciplines for SARIF, Semgrep, CodeQL, differential review, and sharp edges.
|
||||
- Phase 6: Evidence ladder verifier.
|
||||
- Phase 7 and 7A: Docker sandbox and dynamic evidence.
|
||||
- Phase 8 and later: Hunter pool, SARIF, deep mode, patch oracle, MCP, fusion, skill index, hardening.
|
||||
@@ -33,6 +33,18 @@ The README is reduced to user-facing material; scan internals moved to
|
||||
[docs/evaluation.md](docs/evaluation.md). Removed references to commands that no longer exist
|
||||
(`ousast mechanisms export`, `ousast pairs --loo`) and corrected the description of `deep`
|
||||
mode, which runs sandboxed regression snippets when Docker is available.
|
||||
## Unread config keys retired (2026-10-02)
|
||||
These keys loaded but nothing ever read them. They now fail with `RetiredConfigError` (the
|
||||
CLI prints it and exits 2), naming why. Remove them from your config:
|
||||
- `[models] patcher`: no patching stage calls a model.
|
||||
- `[dynamic] enabled`, `[dynamic] network_scope`: no dynamic or deep tier exists yet.
|
||||
- `[evidence] minimum_report_verified`, `minimum_exploit`, `minimum_patch`: evidence tiers
|
||||
come from the program model's ladder, not from configuration.
|
||||
- `[variants] enabled`, `[variants] max_mechanisms`: the variant search they configured was
|
||||
deleted earlier.
|
||||
**Output shape.** `resolved_config.json` no longer has the `dynamic`, `evidence` and
|
||||
`variants` sections or `models.patcher`. `harness.json` keeps its shape:
|
||||
`model_roles.patcher` is always `null`, like `model_roles.verifier`.
|
||||
|
||||
## HarnessX removed; the ax plane is the agentic path (2026-09-30)
|
||||
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
# Archived benchmark scripts
|
||||
|
||||
Reference-only scripts, moved here with `git mv` on 2026-10-02 (legacy cleanup) so their history
|
||||
follows them. Nothing in `src/` imports them and no current workflow runs them. They stay because
|
||||
committed records cite them and their numbers: to re-run a record, check out the commit the record
|
||||
names rather than running these copies against today's package.
|
||||
|
||||
The old paths still appear inside committed records and in some docstrings. [paths.md](paths.md)
|
||||
maps each old path to the new one. The records themselves are not rewritten.
|
||||
|
||||
| script | what it was | records that cite it | last recorded run |
|
||||
|---|---|---|---|
|
||||
| `verify_batched.py` | Stage-2 verifier, second design: triage per file, verify per file, report what two passes agree on. Replaced by the plane's `verify` + `agree` tasks. | `independent/results-v2-model-pipeline.json`, `independent/verifier-batched-check-2026-09-29.json`, `independent/plane-increment-2.json` (parity baseline) | `072a102` (named in `results-v2-model-pipeline.json`) |
|
||||
| `score_batched.py` | Exploratory scorer for the batched verifier: sites anchored to the candidate, matched by `evaluate.matches_v2`. | `independent/verifier-batched-check-2026-09-29.json`, `independent/plane-increment-2.json`; `tests/test_plane_scoring_parity.py` still loads it as the reference | added in `6acd354` (2026-09-29); the parity test replays it on hosts with the recorded scans |
|
||||
| `verify_sinks.py` | The first stage-2 verifier, one candidate at a time. `verify_batched.py` imports `ACCOUNT_ERRORS` and `_Recording` from it. | `independent/verifier-check-2026-09-29.json` | the known-answer check after `041d871` |
|
||||
| `model_sinks.py` | Stage-1 source-only model classification of sinks. Moved into the package as `plane/tasks/roles.py`. | `independent/selection-v2-model-sinks.json`, `measurements/2026-10-01-decision-engine-harvest/record.json`, both `2026-10-01-decision-engine-leak-audit` records | `05d5699` (named in `selection-v2-model-sinks.json`) |
|
||||
| `sink_candidates.py` | Diagnostic: vocabulary-declared sink candidates per case. Its declaration patterns live on in `plane/tasks/repo_facts.py`. | `independent/results-v2-model-pipeline.json` ("vocabulary sink-first: 2/17") | no commit recorded |
|
||||
| `hunt.py` | The v2 tool-hunter run over the population (protocol `hunter-protocol-v2.md`, including its `FAMILY_WORDS` table). Replaced by the plane's `verify` / `agree` tasks. | `independent/hunter-protocol-v2.md`, `independent/results-v2-hunter.json` (a label source in `learn/sources.toml`) | `6bfb05f`, the commit that introduced the protocol, which pins the runner |
|
||||
| `harnessx_removal_equality.py` | One-off equality proof for `harnessx-removal` (record and compare deterministic outputs). The spec is closed. `tests/test_harnessx_removal_equality.py` still checks that the comparison detects differences. | `measurements/2026-09-30-harnessx-removal-baseline/`, `measurements/2026-09-30-harnessx-removal-equality/` | baseline `ca5caf7`, candidate `b9afb66` |
|
||||
|
||||
The six `independent` scripts still read the population, `evaluate.py` and the results from
|
||||
`benchmarks/independent/`. Their `HERE` points there, so outputs land where they always did.
|
||||
`evaluate.py` and `freeze.py` are current and were not moved.
|
||||
|
||||
`verify_sinks.py --help` fails with an argparse `%` formatting error. That bug was already there
|
||||
before the move. The script still imports cleanly.
|
||||
+3
-3
@@ -5,8 +5,8 @@
|
||||
non-zero on the first difference outside the exclusion list, naming the file and key.
|
||||
|
||||
Usage:
|
||||
python benchmarks/harnessx_removal_equality.py record <out-dir> [--tree-ish HEAD] [--workdir DIR]
|
||||
python benchmarks/harnessx_removal_equality.py compare <baseline-dir> <candidate-dir> \\
|
||||
python benchmarks/archive/harnessx_removal_equality.py record <out-dir> [--tree-ish HEAD] [--workdir DIR]
|
||||
python benchmarks/archive/harnessx_removal_equality.py compare <baseline-dir> <candidate-dir> \\
|
||||
[--suite-map map.json] [--record comparison.json]
|
||||
|
||||
The instrument checks come first and fail loudly: ``harnessx`` must not be importable, ``tree_sitter`` must be,
|
||||
@@ -30,7 +30,7 @@ import xml.etree.ElementTree as ET
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
REPO = Path(__file__).resolve().parents[2] # archived under benchmarks/archive/
|
||||
|
||||
# Design section 3 "Comparison": timestamps, durations and run identifiers are excluded; nothing else is.
|
||||
EXCLUDED_KEYS = frozenset({"runtime_seconds", "scan_id", "timestamp", "started", "finished", "created_at", "generated_at"})
|
||||
@@ -10,9 +10,9 @@ Model calls go to the resolved hunter client (DeepSeek when its key is set); the
|
||||
the recorded spend reaches `--budget-usd`.
|
||||
|
||||
Usage:
|
||||
python benchmarks/independent/hunt.py --population population-v2.toml [--only ID,...] [--pins a,b]
|
||||
python benchmarks/independent/hunt.py --population population-v2.toml --score
|
||||
python benchmarks/independent/hunt.py --pilot-root <checkout> --family injection # a development case
|
||||
python benchmarks/archive/hunt.py --population population-v2.toml [--only ID,...] [--pins a,b]
|
||||
python benchmarks/archive/hunt.py --population population-v2.toml --score
|
||||
python benchmarks/archive/hunt.py --pilot-root <checkout> --family injection # a development case
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -25,7 +25,8 @@ import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
# Archived 2026-10-02 from benchmarks/independent/: HERE stays that directory (population, evaluate.py, results).
|
||||
HERE = Path(__file__).resolve().parents[1] / "independent"
|
||||
sys.path.insert(0, str(HERE))
|
||||
|
||||
import evaluate # noqa: E402
|
||||
@@ -12,7 +12,7 @@ counted as UNCLASSIFIED, never as "no sinks". Hard spend ceiling `--budget-usd`;
|
||||
|
||||
v2 is spent for tuning; this informs the selection design and qualifies nothing.
|
||||
|
||||
Usage: python benchmarks/independent/model_sinks.py [--population population-v2.toml] [--only ID,...]
|
||||
Usage: python benchmarks/archive/model_sinks.py [--population population-v2.toml] [--only ID,...]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -27,7 +27,8 @@ import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
# Archived 2026-10-02 from benchmarks/independent/: HERE stays that directory (population, evaluate.py, results).
|
||||
HERE = Path(__file__).resolve().parents[1] / "independent"
|
||||
sys.path.insert(0, str(HERE))
|
||||
|
||||
import evaluate # noqa: E402
|
||||
@@ -0,0 +1,24 @@
|
||||
# Old path to new path (2026-10-02)
|
||||
|
||||
Committed records, specs and some docstrings name these scripts by their old path. Those files are
|
||||
not rewritten. Read the old path as the new one:
|
||||
|
||||
| old path | new path |
|
||||
|---|---|
|
||||
| `benchmarks/independent/verify_batched.py` | `benchmarks/archive/verify_batched.py` |
|
||||
| `benchmarks/independent/score_batched.py` | `benchmarks/archive/score_batched.py` |
|
||||
| `benchmarks/independent/verify_sinks.py` | `benchmarks/archive/verify_sinks.py` |
|
||||
| `benchmarks/independent/model_sinks.py` | `benchmarks/archive/model_sinks.py` |
|
||||
| `benchmarks/independent/sink_candidates.py` | `benchmarks/archive/sink_candidates.py` |
|
||||
| `benchmarks/independent/hunt.py` | `benchmarks/archive/hunt.py` |
|
||||
| `benchmarks/harnessx_removal_equality.py` | `benchmarks/archive/harnessx_removal_equality.py` |
|
||||
|
||||
Files that still use the old path:
|
||||
|
||||
- Records: `independent/hunter-protocol-v2.md`, `independent/results-v2-model-pipeline.json`,
|
||||
`independent/selection-v2-model-sinks.json`, `independent/verifier-batched-check-2026-09-29.json`,
|
||||
`measurements/2026-09-30-harnessx-removal-equality/README`
|
||||
- Specs: `.kiro/specs/ai-service-plane/`, `.kiro/specs/harnessx-removal/design.md`,
|
||||
`.kiro/specs/learned-decision-engine/`
|
||||
- Docstrings: `src/openultrasast/plane/generate.py`, `plane/tasks/{verify,agree,roles,repo_facts}.py`.
|
||||
These are left for the owners of those packages.
|
||||
@@ -10,7 +10,7 @@ detected when an AGREED finding matches; unstable when only a DISPUTED one does;
|
||||
finding at the fixed pin matches the fix's new side. The precision sample is every third agreed vulnerable-pin
|
||||
finding (all if 15 or fewer), to be adjudicated by hand. v2 is spent for tuning: this qualifies nothing.
|
||||
|
||||
Usage: python benchmarks/independent/score_batched.py [--results-suffix batched] [--population population-v2.toml]
|
||||
Usage: python benchmarks/archive/score_batched.py [--results-suffix batched] [--population population-v2.toml]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -20,7 +20,8 @@ import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
# Archived 2026-10-02 from benchmarks/independent/: HERE stays that directory (population, evaluate.py, results).
|
||||
HERE = Path(__file__).resolve().parents[1] / "independent"
|
||||
sys.path.insert(0, str(HERE))
|
||||
|
||||
import evaluate # noqa: E402
|
||||
@@ -12,7 +12,7 @@ changed files -- never to choose scope.
|
||||
|
||||
v2 is spent for tuning; this informs the selection design and qualifies nothing.
|
||||
|
||||
Usage: python benchmarks/independent/sink_candidates.py [--population population-v2.toml] [--only ID,...]
|
||||
Usage: python benchmarks/archive/sink_candidates.py [--population population-v2.toml] [--only ID,...]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -24,7 +24,8 @@ import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
# Archived 2026-10-02 from benchmarks/independent/: HERE stays that directory (population, evaluate.py, results).
|
||||
HERE = Path(__file__).resolve().parents[1] / "independent"
|
||||
sys.path.insert(0, str(HERE))
|
||||
|
||||
import evaluate # noqa: E402
|
||||
@@ -22,7 +22,7 @@ Source only; the stage-1 candidates come from `model_sinks.py`. Results in `~/ou
|
||||
EXPLORATORY: v2 is spent for tuning; this qualifies nothing.
|
||||
|
||||
Usage:
|
||||
python benchmarks/independent/verify_batched.py [--only ID,...] [--subset sets.json] [--passes 2] [--budget-usd 40]
|
||||
python benchmarks/archive/verify_batched.py [--only ID,...] [--subset sets.json] [--passes 2] [--budget-usd 40]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -35,8 +35,10 @@ import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
# Archived 2026-10-02 from benchmarks/independent/: HERE stays that directory (population, evaluate.py, results).
|
||||
HERE = Path(__file__).resolve().parents[1] / "independent"
|
||||
sys.path.insert(0, str(HERE))
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent)) # archived siblings: model_sinks, verify_sinks
|
||||
|
||||
import evaluate # noqa: E402
|
||||
from model_sinks import OPERATIONS, chunks # noqa: E402
|
||||
@@ -10,7 +10,7 @@ Scope: the candidates `model_sinks.py` recorded for each case's vulnerable pin
|
||||
detected sites (`--fixed`). Findings are written in the scan-result shape `evaluate.py` reads. Hard spend
|
||||
ceiling. EXPLORATORY: v2 is spent for tuning; this qualifies nothing.
|
||||
|
||||
Usage: python benchmarks/independent/verify_sinks.py [--only ID,...] [--budget-usd 40] [--fixed]
|
||||
Usage: python benchmarks/archive/verify_sinks.py [--only ID,...] [--budget-usd 40] [--fixed]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -23,8 +23,10 @@ import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
# Archived 2026-10-02 from benchmarks/independent/: HERE stays that directory (population, evaluate.py, results).
|
||||
HERE = Path(__file__).resolve().parents[1] / "independent"
|
||||
sys.path.insert(0, str(HERE))
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent)) # archived siblings: model_sinks, verify_sinks
|
||||
|
||||
import evaluate # noqa: E402
|
||||
from model_sinks import OPERATIONS # noqa: E402
|
||||
@@ -1,8 +1,8 @@
|
||||
{
|
||||
"modules": 155,
|
||||
"modules": 150,
|
||||
"counts": {
|
||||
"load_bearing": 148,
|
||||
"standalone_capability": 7
|
||||
"load_bearing": 145,
|
||||
"standalone_capability": 5
|
||||
},
|
||||
"orphaned": [],
|
||||
"standalone": {
|
||||
@@ -10,15 +10,13 @@
|
||||
"mcp": "narrow MCP server over stdio; its own `ousast mcp` entry point and OpenCode integration",
|
||||
"push_inputs": "pre-push-safety-net task 1.2: offline input provenance validator; python -m openultrasast.push_inputs",
|
||||
"push_scoring": "pre-push-safety-net task 1.3: frozen-profile diagnostic outcome scorer; python -m openultrasast.push_scoring",
|
||||
"skills": "skill router; consumed by mcp and by the OpenCode integration, not by the scan path",
|
||||
"slot_contract": "the slot/contract protocol stage_processors implements",
|
||||
"stage_processors": "deterministic slot-contracted scan stages; pinned by the zero-dependency guard in test_gate"
|
||||
"skills": "skill router; consumed by mcp and by the OpenCode integration, not by the scan path"
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"module": "benchmark",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"gate",
|
||||
@@ -32,7 +30,7 @@
|
||||
{
|
||||
"module": "calibration",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli"
|
||||
]
|
||||
@@ -48,7 +46,7 @@
|
||||
{
|
||||
"module": "complexity",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"complexity.hints",
|
||||
@@ -64,7 +62,7 @@
|
||||
{
|
||||
"module": "complexity.hints",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"complexity",
|
||||
"complexity.map"
|
||||
@@ -73,7 +71,7 @@
|
||||
{
|
||||
"module": "complexity.ledger",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"complexity",
|
||||
@@ -84,8 +82,9 @@
|
||||
{
|
||||
"module": "complexity.map",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"complexity",
|
||||
"complexity.hints",
|
||||
"complexity.ledger",
|
||||
@@ -99,7 +98,7 @@
|
||||
{
|
||||
"module": "complexity.signals",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"complexity",
|
||||
"complexity.hints",
|
||||
@@ -109,7 +108,7 @@
|
||||
{
|
||||
"module": "config",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"harness",
|
||||
@@ -150,7 +149,6 @@
|
||||
"cpg.backend",
|
||||
"model.config_value",
|
||||
"model.dominance",
|
||||
"model.judge",
|
||||
"model.partitions",
|
||||
"model.pipeline",
|
||||
"model.scan",
|
||||
@@ -184,7 +182,6 @@
|
||||
"cpg.artifact",
|
||||
"model.config_value",
|
||||
"model.dominance",
|
||||
"model.judge",
|
||||
"model.partitions",
|
||||
"model.pipeline",
|
||||
"model.scan",
|
||||
@@ -206,7 +203,7 @@
|
||||
{
|
||||
"module": "cpg.session",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cpg.backend"
|
||||
]
|
||||
@@ -214,7 +211,7 @@
|
||||
{
|
||||
"module": "findings",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"benchmark",
|
||||
"calibration",
|
||||
@@ -239,7 +236,6 @@
|
||||
"semantic.obligations.check",
|
||||
"semantic.overlay",
|
||||
"semantic.prove_filter",
|
||||
"stage_processors",
|
||||
"tool_hunter",
|
||||
"verification"
|
||||
]
|
||||
@@ -264,16 +260,15 @@
|
||||
{
|
||||
"module": "harness",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"slot_contract"
|
||||
"cli"
|
||||
]
|
||||
},
|
||||
{
|
||||
"module": "hunter",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli"
|
||||
]
|
||||
@@ -281,7 +276,7 @@
|
||||
{
|
||||
"module": "hunter_tools",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"tool_hunter"
|
||||
]
|
||||
@@ -289,7 +284,7 @@
|
||||
{
|
||||
"module": "improve",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"improve.evolve",
|
||||
@@ -300,7 +295,7 @@
|
||||
{
|
||||
"module": "improve.evolve",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"improve",
|
||||
"plane.tasks.loop"
|
||||
@@ -309,7 +304,7 @@
|
||||
{
|
||||
"module": "improve.journal",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"improve",
|
||||
"improve.evolve",
|
||||
@@ -320,7 +315,7 @@
|
||||
{
|
||||
"module": "improve.memory",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"improve.evolve",
|
||||
@@ -330,7 +325,7 @@
|
||||
{
|
||||
"module": "improve.validator",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"improve",
|
||||
"improve.evolve",
|
||||
@@ -341,7 +336,7 @@
|
||||
{
|
||||
"module": "index",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"skills"
|
||||
@@ -350,7 +345,7 @@
|
||||
{
|
||||
"module": "learn",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"learn.calibrate",
|
||||
@@ -376,7 +371,7 @@
|
||||
{
|
||||
"module": "learn.calibrate",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"learn.evaluate"
|
||||
]
|
||||
@@ -384,7 +379,7 @@
|
||||
{
|
||||
"module": "learn.cli",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli"
|
||||
]
|
||||
@@ -392,7 +387,7 @@
|
||||
{
|
||||
"module": "learn.compile",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"learn.cli",
|
||||
"learn.evaluate"
|
||||
@@ -401,7 +396,7 @@
|
||||
{
|
||||
"module": "learn.embeddings",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"learn.cli",
|
||||
"learn.retrieve"
|
||||
@@ -410,7 +405,7 @@
|
||||
{
|
||||
"module": "learn.evaluate",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"learn.cli",
|
||||
"learn.compile"
|
||||
@@ -419,7 +414,7 @@
|
||||
{
|
||||
"module": "learn.examples",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"learn.cli",
|
||||
"learn.compile",
|
||||
@@ -432,7 +427,7 @@
|
||||
{
|
||||
"module": "learn.excerpt",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"learn.embeddings",
|
||||
"learn.examples",
|
||||
@@ -443,7 +438,7 @@
|
||||
{
|
||||
"module": "learn.features",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"learn.excerpt",
|
||||
"plane.tasks.features"
|
||||
@@ -452,7 +447,7 @@
|
||||
{
|
||||
"module": "learn.folds",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"learn.cli",
|
||||
"learn.compile",
|
||||
@@ -464,7 +459,7 @@
|
||||
{
|
||||
"module": "learn.labels",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"learn.calibrate",
|
||||
@@ -480,7 +475,7 @@
|
||||
{
|
||||
"module": "learn.leaks",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"learn.cli"
|
||||
]
|
||||
@@ -498,7 +493,7 @@
|
||||
{
|
||||
"module": "learn.retrieve",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"learn.evaluate",
|
||||
"learn.program"
|
||||
@@ -507,7 +502,7 @@
|
||||
{
|
||||
"module": "learn.roles",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"learn.features",
|
||||
"plane.tasks.features",
|
||||
@@ -517,7 +512,7 @@
|
||||
{
|
||||
"module": "learn.schema",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"learn.compile",
|
||||
"learn.evaluate",
|
||||
@@ -539,7 +534,7 @@
|
||||
{
|
||||
"module": "mapping",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"gate",
|
||||
@@ -551,8 +546,7 @@
|
||||
"pairs",
|
||||
"plane.tasks.alerts",
|
||||
"preprocess",
|
||||
"push.runner",
|
||||
"stage_processors"
|
||||
"push.runner"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -566,7 +560,7 @@
|
||||
{
|
||||
"module": "model",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"cpg.backend",
|
||||
@@ -577,8 +571,6 @@
|
||||
"model.candidates",
|
||||
"model.config_value",
|
||||
"model.dominance",
|
||||
"model.execution",
|
||||
"model.judge",
|
||||
"model.layout",
|
||||
"model.partitions",
|
||||
"model.pipeline",
|
||||
@@ -610,16 +602,10 @@
|
||||
"reason": "model-grounded-detection Req 3: this classifier; run by the maintainer, not by the scan",
|
||||
"importers": []
|
||||
},
|
||||
{
|
||||
"module": "model.calibrate",
|
||||
"classification": "load_bearing",
|
||||
"reason": "model-grounded-detection Req 9: the corpus as the model's calibration set",
|
||||
"importers": []
|
||||
},
|
||||
{
|
||||
"module": "model.candidates",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"model.scan"
|
||||
@@ -668,7 +654,7 @@
|
||||
{
|
||||
"module": "model.endpoint",
|
||||
"classification": "load_bearing",
|
||||
"reason": "model-grounded-detection Req 8: the judge client, wired by model/judge.py in group 3",
|
||||
"reason": "model-grounded-detection Req 8: the bounded model client the hunter, the plane and the CLI share",
|
||||
"importers": [
|
||||
"cli",
|
||||
"learn.cli",
|
||||
@@ -681,23 +667,11 @@
|
||||
{
|
||||
"module": "model.evidence",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"model.scan"
|
||||
]
|
||||
},
|
||||
{
|
||||
"module": "model.execution",
|
||||
"classification": "load_bearing",
|
||||
"reason": "model-grounded-detection Req 10: the deferred execution tier seam, adopted not built",
|
||||
"importers": []
|
||||
},
|
||||
{
|
||||
"module": "model.judge",
|
||||
"classification": "load_bearing",
|
||||
"reason": "model-grounded-detection Req 8: one bounded question, checked against the model",
|
||||
"importers": []
|
||||
},
|
||||
{
|
||||
"module": "model.ladder",
|
||||
"classification": "load_bearing",
|
||||
@@ -705,8 +679,6 @@
|
||||
"importers": [
|
||||
"model.config_value",
|
||||
"model.dominance",
|
||||
"model.execution",
|
||||
"model.judge",
|
||||
"model.pipeline",
|
||||
"model.scan",
|
||||
"model.taint",
|
||||
@@ -716,7 +688,7 @@
|
||||
{
|
||||
"module": "model.layout",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"model.partitions",
|
||||
"model.scan",
|
||||
@@ -728,7 +700,7 @@
|
||||
{
|
||||
"module": "model.partitions",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"model.scan"
|
||||
]
|
||||
@@ -778,7 +750,7 @@
|
||||
{
|
||||
"module": "model.shipped",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"push.runner"
|
||||
@@ -791,7 +763,6 @@
|
||||
"importers": [
|
||||
"model.config_value",
|
||||
"model.dominance",
|
||||
"model.judge",
|
||||
"model.pipeline",
|
||||
"model.regions",
|
||||
"model.report",
|
||||
@@ -805,7 +776,6 @@
|
||||
"classification": "load_bearing",
|
||||
"reason": "model-grounded-detection Req 6.1: taint reachability over the CPG, the flow-family arbiter",
|
||||
"importers": [
|
||||
"model.judge",
|
||||
"model.pipeline",
|
||||
"model.scan"
|
||||
]
|
||||
@@ -813,7 +783,7 @@
|
||||
{
|
||||
"module": "model.taxonomy",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"learn.features",
|
||||
@@ -836,7 +806,7 @@
|
||||
{
|
||||
"module": "pairs",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"improve",
|
||||
@@ -894,7 +864,7 @@
|
||||
{
|
||||
"module": "plane.doctor",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"plane.reconciler"
|
||||
]
|
||||
@@ -902,7 +872,7 @@
|
||||
{
|
||||
"module": "plane.egress",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"plane.reconciler"
|
||||
]
|
||||
@@ -911,7 +881,9 @@
|
||||
"module": "plane.engine_alerts",
|
||||
"classification": "load_bearing",
|
||||
"reason": "harnessx-removal Req 6.3: engine alerts where quick mode has no rules (PHP), on the host",
|
||||
"importers": []
|
||||
"importers": [
|
||||
"cli"
|
||||
]
|
||||
},
|
||||
{
|
||||
"module": "plane.generate",
|
||||
@@ -925,7 +897,7 @@
|
||||
{
|
||||
"module": "plane.harvest",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli"
|
||||
]
|
||||
@@ -950,6 +922,7 @@
|
||||
"classification": "load_bearing",
|
||||
"reason": "harnessx-removal Req 6.1/6.4: the memory store (file, MinIO), ingest and fact reuse; `ousast plane remember`",
|
||||
"importers": [
|
||||
"cli",
|
||||
"improve.memory",
|
||||
"learn.cli",
|
||||
"learn.compile",
|
||||
@@ -966,6 +939,8 @@
|
||||
"classification": "load_bearing",
|
||||
"reason": "ai-service-plane Req 3.1-3.5, 7.3: ax-backed Run execution, status attribution and doctor; `ousast plane`",
|
||||
"importers": [
|
||||
"cli",
|
||||
"plane.engine_alerts",
|
||||
"plane.generate",
|
||||
"plane.harvest",
|
||||
"plane.memory",
|
||||
@@ -975,7 +950,7 @@
|
||||
{
|
||||
"module": "plane.router",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"plane.reconciler",
|
||||
"plane.runner"
|
||||
@@ -1093,7 +1068,7 @@
|
||||
{
|
||||
"module": "policy",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"findings",
|
||||
@@ -1102,14 +1077,13 @@
|
||||
"improve.validator",
|
||||
"plane.tasks.loop",
|
||||
"scoring.project_score",
|
||||
"semantic.overlay",
|
||||
"stage_processors"
|
||||
"semantic.overlay"
|
||||
]
|
||||
},
|
||||
{
|
||||
"module": "policy.verycode",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"policy"
|
||||
]
|
||||
@@ -1117,7 +1091,7 @@
|
||||
{
|
||||
"module": "preprocess",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"complexity.hints",
|
||||
@@ -1146,14 +1120,13 @@
|
||||
"rank",
|
||||
"regress.candidate",
|
||||
"semantic.overlay",
|
||||
"skills",
|
||||
"stage_processors"
|
||||
"skills"
|
||||
]
|
||||
},
|
||||
{
|
||||
"module": "profiling",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli"
|
||||
]
|
||||
@@ -1161,7 +1134,7 @@
|
||||
{
|
||||
"module": "provenance",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli"
|
||||
]
|
||||
@@ -1169,7 +1142,7 @@
|
||||
{
|
||||
"module": "provider",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"learn.cli",
|
||||
"learn.compile",
|
||||
@@ -1181,7 +1154,7 @@
|
||||
{
|
||||
"module": "provider.openrouter",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"learn.cli",
|
||||
"learn.compile",
|
||||
@@ -1209,7 +1182,7 @@
|
||||
{
|
||||
"module": "push.assistance",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"push.runner"
|
||||
]
|
||||
@@ -1240,7 +1213,7 @@
|
||||
{
|
||||
"module": "push.eligibility",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"push.runner"
|
||||
]
|
||||
@@ -1268,7 +1241,7 @@
|
||||
{
|
||||
"module": "push.reuse",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"push.runner"
|
||||
]
|
||||
@@ -1276,7 +1249,7 @@
|
||||
{
|
||||
"module": "push.runner",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"push.assistance"
|
||||
@@ -1293,7 +1266,7 @@
|
||||
{
|
||||
"module": "push.vetoes",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"push.runner"
|
||||
]
|
||||
@@ -1315,7 +1288,7 @@
|
||||
{
|
||||
"module": "rank",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"calibration",
|
||||
"cli",
|
||||
@@ -1326,17 +1299,15 @@
|
||||
"improve.evolve",
|
||||
"map_gate",
|
||||
"pairs",
|
||||
"plane.tasks.alerts",
|
||||
"stage_processors"
|
||||
"plane.tasks.alerts"
|
||||
]
|
||||
},
|
||||
{
|
||||
"module": "redaction",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"harness",
|
||||
"model.judge",
|
||||
"model.pipeline",
|
||||
"push.assistance",
|
||||
"push.report",
|
||||
@@ -1347,7 +1318,7 @@
|
||||
{
|
||||
"module": "regress",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"regress.candidate"
|
||||
@@ -1356,7 +1327,7 @@
|
||||
{
|
||||
"module": "regress.candidate",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"regress"
|
||||
]
|
||||
@@ -1364,7 +1335,7 @@
|
||||
{
|
||||
"module": "regress.recipes",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"regress",
|
||||
"regress.candidate"
|
||||
@@ -1373,7 +1344,7 @@
|
||||
{
|
||||
"module": "regress.safety",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"regress",
|
||||
"regress.candidate"
|
||||
@@ -1382,7 +1353,7 @@
|
||||
{
|
||||
"module": "regress.verdict",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"regress",
|
||||
"regress.candidate"
|
||||
@@ -1391,7 +1362,7 @@
|
||||
{
|
||||
"module": "reports",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli"
|
||||
]
|
||||
@@ -1399,7 +1370,7 @@
|
||||
{
|
||||
"module": "repos",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"push_inputs"
|
||||
@@ -1408,7 +1379,7 @@
|
||||
{
|
||||
"module": "ruleset",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"findings",
|
||||
@@ -1424,14 +1395,13 @@
|
||||
"plane.tasks.features",
|
||||
"plane.tasks.loop",
|
||||
"ruleset.store",
|
||||
"semantic.facts",
|
||||
"stage_processors"
|
||||
"semantic.facts"
|
||||
]
|
||||
},
|
||||
{
|
||||
"module": "ruleset.frameworks",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"learn.labels",
|
||||
"learn.roles",
|
||||
@@ -1442,7 +1412,7 @@
|
||||
{
|
||||
"module": "ruleset.store",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"improve.validator",
|
||||
"ruleset"
|
||||
@@ -1451,7 +1421,7 @@
|
||||
{
|
||||
"module": "run",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"reports"
|
||||
@@ -1460,7 +1430,7 @@
|
||||
{
|
||||
"module": "sandbox",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"regress.candidate",
|
||||
@@ -1472,7 +1442,7 @@
|
||||
{
|
||||
"module": "sandbox.probe",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"sandbox"
|
||||
]
|
||||
@@ -1480,7 +1450,7 @@
|
||||
{
|
||||
"module": "sandbox.runner",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"regress.candidate",
|
||||
"regress.recipes",
|
||||
@@ -1491,17 +1461,16 @@
|
||||
{
|
||||
"module": "scoring",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"improve.evolve",
|
||||
"stage_processors"
|
||||
"improve.evolve"
|
||||
]
|
||||
},
|
||||
{
|
||||
"module": "scoring.project_score",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"scoring"
|
||||
]
|
||||
@@ -1509,7 +1478,7 @@
|
||||
{
|
||||
"module": "semantic",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"cpg.capability",
|
||||
@@ -1538,7 +1507,7 @@
|
||||
{
|
||||
"module": "semantic.cst",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"semantic.ir"
|
||||
]
|
||||
@@ -1546,7 +1515,7 @@
|
||||
{
|
||||
"module": "semantic.engines",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cpg.capability",
|
||||
"semantic",
|
||||
@@ -1556,7 +1525,7 @@
|
||||
{
|
||||
"module": "semantic.extra",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"semantic",
|
||||
"semantic.cst",
|
||||
@@ -1566,7 +1535,7 @@
|
||||
{
|
||||
"module": "semantic.facts",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"hunter_tools",
|
||||
@@ -1590,7 +1559,7 @@
|
||||
{
|
||||
"module": "semantic.functions",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"hunter_tools",
|
||||
"pairs",
|
||||
@@ -1601,7 +1570,7 @@
|
||||
{
|
||||
"module": "semantic.ir",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"hunter_tools",
|
||||
@@ -1619,7 +1588,7 @@
|
||||
{
|
||||
"module": "semantic.obligations",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"hunter_tools",
|
||||
@@ -1638,7 +1607,7 @@
|
||||
{
|
||||
"module": "semantic.obligations.check",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"semantic.obligations",
|
||||
@@ -1648,7 +1617,7 @@
|
||||
{
|
||||
"module": "semantic.obligations.dominance",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"hunter_tools",
|
||||
@@ -1659,7 +1628,7 @@
|
||||
{
|
||||
"module": "semantic.obligations.facts",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"model.specs",
|
||||
"pairs",
|
||||
@@ -1674,7 +1643,7 @@
|
||||
{
|
||||
"module": "semantic.obligations.intent",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli"
|
||||
]
|
||||
@@ -1682,7 +1651,7 @@
|
||||
{
|
||||
"module": "semantic.obligations.operations",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"model.candidates",
|
||||
"semantic.obligations.check",
|
||||
@@ -1693,7 +1662,7 @@
|
||||
{
|
||||
"module": "semantic.obligations.policy",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"semantic.obligations.check"
|
||||
@@ -1702,7 +1671,7 @@
|
||||
{
|
||||
"module": "semantic.obligations.shapes",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"semantic.obligations",
|
||||
"semantic.obligations.operations"
|
||||
@@ -1711,7 +1680,7 @@
|
||||
{
|
||||
"module": "semantic.obligations.siblings",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"semantic.obligations.check"
|
||||
]
|
||||
@@ -1719,7 +1688,7 @@
|
||||
{
|
||||
"module": "semantic.overlay",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"semantic",
|
||||
"semantic.prove_filter"
|
||||
@@ -1728,7 +1697,7 @@
|
||||
{
|
||||
"module": "semantic.prove_filter",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"semantic"
|
||||
]
|
||||
@@ -1736,7 +1705,7 @@
|
||||
{
|
||||
"module": "semantic.taint",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"semantic.overlay"
|
||||
]
|
||||
@@ -1749,24 +1718,10 @@
|
||||
"hunter"
|
||||
]
|
||||
},
|
||||
{
|
||||
"module": "slot_contract",
|
||||
"classification": "standalone_capability",
|
||||
"reason": "the slot/contract protocol stage_processors implements",
|
||||
"importers": [
|
||||
"stage_processors"
|
||||
]
|
||||
},
|
||||
{
|
||||
"module": "stage_processors",
|
||||
"classification": "standalone_capability",
|
||||
"reason": "deterministic slot-contracted scan stages; pinned by the zero-dependency guard in test_gate",
|
||||
"importers": []
|
||||
},
|
||||
{
|
||||
"module": "stages",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli"
|
||||
]
|
||||
@@ -1774,8 +1729,9 @@
|
||||
{
|
||||
"module": "tool_hunter",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"cli",
|
||||
"model.endpoint",
|
||||
"plane.budget",
|
||||
"plane.tasks.roles",
|
||||
@@ -1785,7 +1741,7 @@
|
||||
{
|
||||
"module": "verification",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"reason": "reachable from gate, map_gate, pair_gate, cli",
|
||||
"importers": [
|
||||
"calibration",
|
||||
"cli",
|
||||
|
||||
@@ -1 +1,3 @@
|
||||
Excluded from comparison: JSON keys: created_at, finished, generated_at, runtime_seconds, scan_id, started, timestamp; JSON keys ending in: _seconds, _ms; run directory names / run ids (<timestamp>-<8 hex>, benchmark.py create_benchmark_run, run.py create_scan_run) -> <RUN_ID>; trace/events.jsonl as a whole (timings); never copied; the export's absolute root path -> <ROOT>; the children's TMPDIR -> <TMP>; suite: only the outcome per test id is kept (junit durations dropped); env.json is a provenance record (commit, freeze hash, instrument checks), not compared
|
||||
|
||||
scan/report.sarif was removed from git on 2026-10-02. It had been committed in ca5caf7 despite .gitignore excluding *.sarif, and the candidate side (../2026-09-30-harnessx-removal-equality/candidate/scan) has none, so the committed pair was asymmetric. What a reader can recheck from git is the remaining scan files: findings.json, verification.json, fusion.json, harness.json, manifest.json and stdout.txt.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+19
-58
@@ -20,10 +20,24 @@ _PLANE_RUN = "`ousast plane run`, see ops/ax/README.md"
|
||||
_RETIRED_PANELS = (
|
||||
f"LLM fusion panels were retired; fusion is deterministic; independent LLM agreement is the plane's `agree` task ({_PLANE_RUN})"
|
||||
)
|
||||
# Keys that loaded but were never read (retired 2026-10-02, legacy cleanup): accepting them silently would let
|
||||
# a user believe they configure something.
|
||||
_UNREAD_ON = "retired 2026-10-02 because nothing ever read it"
|
||||
_UNREAD_DYNAMIC = "no dynamic or deep tier exists; deep mode will define its own keys when it is built"
|
||||
_UNREAD_EVIDENCE = "evidence tiers come from the program model's ladder (model/ladder.py), not from configuration"
|
||||
_UNREAD_VARIANTS = "the structural variant search it configured was deleted with semantic/variants.py"
|
||||
RETIRED_KEYS: dict[tuple[str, str], str] = {
|
||||
("models", "verifier"): f"LLM verification runs on the plane (`verify` + `agree`, {_PLANE_RUN})",
|
||||
("fusion", "panel_model"): _RETIRED_PANELS,
|
||||
("fusion", "decider_model"): _RETIRED_PANELS,
|
||||
("models", "verifier"): f"{_RETIRED_ON}: LLM verification runs on the plane (`verify` + `agree`, {_PLANE_RUN})",
|
||||
("fusion", "panel_model"): f"{_RETIRED_ON}: {_RETIRED_PANELS}",
|
||||
("fusion", "decider_model"): f"{_RETIRED_ON}: {_RETIRED_PANELS}",
|
||||
("models", "patcher"): f"{_UNREAD_ON}: no patching stage calls a model",
|
||||
("dynamic", "enabled"): f"{_UNREAD_ON}: {_UNREAD_DYNAMIC}",
|
||||
("dynamic", "network_scope"): f"{_UNREAD_ON}: {_UNREAD_DYNAMIC}",
|
||||
("evidence", "minimum_report_verified"): f"{_UNREAD_ON}: {_UNREAD_EVIDENCE}",
|
||||
("evidence", "minimum_exploit"): f"{_UNREAD_ON}: {_UNREAD_EVIDENCE}",
|
||||
("evidence", "minimum_patch"): f"{_UNREAD_ON}: {_UNREAD_EVIDENCE}",
|
||||
("variants", "enabled"): f"{_UNREAD_ON}: {_UNREAD_VARIANTS}",
|
||||
("variants", "max_mechanisms"): f"{_UNREAD_ON}: {_UNREAD_VARIANTS}",
|
||||
}
|
||||
RETIRED_SECTION = "harnessx" # retired 2026-09-30: loads with one warning and is otherwise ignored (Req 3.3)
|
||||
RETIRED_SECTION_WARNING = (
|
||||
@@ -38,10 +52,10 @@ class RetiredConfigError(ValueError):
|
||||
|
||||
def _check_retired(data: dict[str, object]) -> None:
|
||||
problems = []
|
||||
for (section, key), replacement in RETIRED_KEYS.items():
|
||||
for (section, key), reason in RETIRED_KEYS.items():
|
||||
table = data.get(section)
|
||||
if isinstance(table, dict) and key in table:
|
||||
problems.append(f"`[{section}] {key}` was {_RETIRED_ON}: {replacement}. Remove the key.")
|
||||
problems.append(f"`[{section}] {key}` was {reason}. Remove the key.")
|
||||
if problems:
|
||||
raise RetiredConfigError("\n".join(problems))
|
||||
if RETIRED_SECTION in data:
|
||||
@@ -70,7 +84,6 @@ def load_dotenv(path: Path | None = None, *, force: bool = False) -> None:
|
||||
class ModelConfig:
|
||||
ranker: str | None = None
|
||||
hunter: str | None = None
|
||||
patcher: str | None = None
|
||||
judge: str | None = None # second, independent judgement before anything is published as proven
|
||||
chat_base_url: str | None = None # chat endpoint; independent of the embedding endpoint
|
||||
chat_api_key_env: str | None = None # environment variable holding that endpoint's key
|
||||
@@ -96,19 +109,6 @@ class SandboxConfig:
|
||||
pids_limit: int = 512
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DynamicConfig:
|
||||
enabled: bool = False
|
||||
network_scope: tuple[str, ...] = ()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EvidenceConfig:
|
||||
minimum_report_verified: str = "static_corroboration"
|
||||
minimum_exploit: str = "crash_reproduced"
|
||||
minimum_patch: str = "root_cause_explained"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class StaticAnalysisConfig:
|
||||
sarif_paths: tuple[str, ...] = ()
|
||||
@@ -160,13 +160,6 @@ class RegressConfig:
|
||||
images: tuple[tuple[str, str], ...] = ()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class VariantsConfig:
|
||||
# corpus-seeded-mechanisms: structural variant search in MAP (standard/deep only).
|
||||
enabled: bool = True
|
||||
max_mechanisms: int = 500
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ModelLayerConfig:
|
||||
# contributor-scan: the model layer in MAP. Named for the LAYER, not the model: `ModelConfig` is already
|
||||
@@ -214,8 +207,6 @@ class ResolvedConfig:
|
||||
models: ModelConfig = ModelConfig()
|
||||
embeddings: EmbeddingConfig = EmbeddingConfig()
|
||||
sandbox: SandboxConfig = SandboxConfig()
|
||||
dynamic: DynamicConfig = DynamicConfig()
|
||||
evidence: EvidenceConfig = EvidenceConfig()
|
||||
static_analysis: StaticAnalysisConfig = StaticAnalysisConfig()
|
||||
score: ScoreConfig = ScoreConfig()
|
||||
ruleset: RulesetConfig = RulesetConfig()
|
||||
@@ -223,7 +214,6 @@ class ResolvedConfig:
|
||||
hardening: HardeningConfig = HardeningConfig()
|
||||
complexity: ComplexityConfig = ComplexityConfig()
|
||||
regress: RegressConfig = RegressConfig()
|
||||
variants: VariantsConfig = VariantsConfig()
|
||||
obligations: ObligationsConfig = ObligationsConfig()
|
||||
model: ModelLayerConfig = ModelLayerConfig()
|
||||
push: PushConfig = PushConfig()
|
||||
@@ -243,8 +233,6 @@ def load_config(config_path: Path | None = None, *, dotenv: bool = True) -> Reso
|
||||
models=_load_models(data.get("models", {})),
|
||||
embeddings=_load_embeddings(data.get("embeddings", {})),
|
||||
sandbox=_load_sandbox(data.get("sandbox", {})),
|
||||
dynamic=_load_dynamic(data.get("dynamic", {})),
|
||||
evidence=_load_evidence(data.get("evidence", {})),
|
||||
static_analysis=_load_static_analysis(data.get("static_analysis", {})),
|
||||
score=_load_score(data.get("score", {})),
|
||||
ruleset=_load_ruleset_config(data.get("ruleset", {})),
|
||||
@@ -252,7 +240,6 @@ def load_config(config_path: Path | None = None, *, dotenv: bool = True) -> Reso
|
||||
hardening=_load_hardening(data.get("hardening", {})),
|
||||
complexity=_load_complexity(data.get("complexity", {})),
|
||||
regress=_load_regress(data.get("regress", {})),
|
||||
variants=_load_variants(data.get("variants", {})),
|
||||
obligations=_load_obligations(data.get("obligations", {})),
|
||||
model=_load_model(data.get("model", {})),
|
||||
push=PushConfig.from_payload(data.get("push", {})),
|
||||
@@ -283,7 +270,6 @@ def _load_models(value: object) -> ModelConfig:
|
||||
return ModelConfig(
|
||||
ranker=_string(data.get("ranker")),
|
||||
hunter=_string(data.get("hunter")),
|
||||
patcher=_string(data.get("patcher")),
|
||||
judge=_string(data.get("judge")),
|
||||
chat_base_url=_string(data.get("chat_base_url")),
|
||||
chat_api_key_env=_string(data.get("chat_api_key_env")),
|
||||
@@ -310,23 +296,6 @@ def _load_sandbox(value: object) -> SandboxConfig:
|
||||
)
|
||||
|
||||
|
||||
def _load_dynamic(value: object) -> DynamicConfig:
|
||||
data = _section(value)
|
||||
scope = data.get("network_scope", [])
|
||||
if not isinstance(scope, list):
|
||||
scope = []
|
||||
return DynamicConfig(enabled=bool(data.get("enabled", False)), network_scope=tuple(str(item) for item in scope))
|
||||
|
||||
|
||||
def _load_evidence(value: object) -> EvidenceConfig:
|
||||
data = _section(value)
|
||||
return EvidenceConfig(
|
||||
minimum_report_verified=str(data.get("minimum_report_verified", "static_corroboration")),
|
||||
minimum_exploit=str(data.get("minimum_exploit", "crash_reproduced")),
|
||||
minimum_patch=str(data.get("minimum_patch", "root_cause_explained")),
|
||||
)
|
||||
|
||||
|
||||
def _int_value(value: object, default: int) -> int:
|
||||
if isinstance(value, bool):
|
||||
return default
|
||||
@@ -416,14 +385,6 @@ def _float_value(value: object, default: float) -> float:
|
||||
return default
|
||||
|
||||
|
||||
def _load_variants(value: object) -> VariantsConfig:
|
||||
data = _section(value)
|
||||
return VariantsConfig(
|
||||
enabled=bool(data.get("enabled", True)),
|
||||
max_mechanisms=_int_value(data.get("max_mechanisms"), 500),
|
||||
)
|
||||
|
||||
|
||||
def _load_model(value: object) -> ModelLayerConfig:
|
||||
data = _section(value)
|
||||
return ModelLayerConfig(
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
"""Fusion adjudication — OpenUltraCode-style two-panel deepening (Phase 13).
|
||||
"""Fusion adjudication — OpenUltraCode-style two-panel deepening (Phase 13 of the retired openrouter-sast-harness plan).
|
||||
|
||||
Fusion is the deepening mechanism for findings that need more reasoning than the
|
||||
ranker → hunter → verifier → mapping loop provides: critical/high severity, verifier
|
||||
disagreement, static-vs-semantic evidence conflict, findings that gate a risky fix or
|
||||
disclosure, or an explicit high-assurance request. Two panels independently review the
|
||||
bounded evidence (one steel-manning the vulnerability case, one the false-positive
|
||||
case), critique and revise, vote, and a decider issues the final disposition.
|
||||
case) and vote, and a decider issues the final disposition.
|
||||
|
||||
The panels and the decider are deterministic and auditable. Every fused finding receives
|
||||
exactly one of the five dispositions, and the decision discloses panel roles, votes and the
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Detection-quality gate — the hard feasibility constraint (Phase 4 task 8).
|
||||
"""Detection-quality gate — the hard feasibility constraint (Phase 4 task 8 of the retired openrouter-sast-harness plan).
|
||||
|
||||
Aggregate recall and false-positive rate over the benchmark corpus must clear the
|
||||
project goal: recall >= 90% AND false-positive rate < 10%. This gate is computed
|
||||
|
||||
@@ -181,7 +181,7 @@ def write_harness_config(
|
||||
"ranker": config.models.ranker,
|
||||
"hunter": config.models.hunter,
|
||||
"verifier": None, # schema kept byte for byte; LLM verification runs on the plane
|
||||
"patcher": config.models.patcher,
|
||||
"patcher": None, # schema kept; `[models] patcher` was retired 2026-10-02 (never read)
|
||||
},
|
||||
processor_versions={processor.spec.name: processor.spec.version for processor in processors},
|
||||
prompt_hashes={name: _hash_text(prompt) for name, prompt in sorted(prompts.items())},
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
"""Narrow MCP server for OpenCode integration (Phase 12).
|
||||
"""Narrow MCP server for OpenCode integration (Phase 12 of the retired openrouter-sast-harness plan).
|
||||
|
||||
Exposes only stable, project-level operations over MCP stdio (newline-delimited
|
||||
JSON-RPC 2.0), implemented with the standard library only. No tool exposes arbitrary
|
||||
shell, Docker, or internal hunter tools — those stay inside controlled harness stages.
|
||||
shell, Docker, or internal hunter tools — those stay inside the scan's own stages.
|
||||
The pure :meth:`McpServer.handle` dispatch is separated from the :func:`serve` stdio
|
||||
loop so the surface is unit-testable without a live client.
|
||||
"""
|
||||
|
||||
@@ -6,9 +6,12 @@ checkable rather than a thing someone notices two specs later.
|
||||
|
||||
A module is:
|
||||
|
||||
load_bearing reachable from a gate, the scan CLI, or a surviving requirement
|
||||
standalone_capability its own entry point and users, not on the scan path
|
||||
orphaned no importer, no entry point, no requirement — it should not be in the tree
|
||||
load_bearing reachable by imports from a gate, the CLI, or a module run with ``python -m``
|
||||
standalone_capability reachable the same way, but its own entry point and users, not on the scan path
|
||||
orphaned not reachable from any of those -- it should not be in the tree
|
||||
|
||||
Reachability decides the class. ``SPEC_OWNED`` and ``STANDALONE`` only supply the reason text: being named
|
||||
by a spec does not keep a module nobody imports alive.
|
||||
|
||||
Pure stdlib, no imports of the package it audits: it reads the import graph with ``ast`` so a broken module
|
||||
is still classifiable.
|
||||
@@ -26,7 +29,7 @@ PACKAGE = "openultrasast"
|
||||
# Entry points a module can be reachable from. The three gates are the contract; the CLI is the product.
|
||||
ROOTS = ("gate", "map_gate", "pair_gate", "cli", "__main__")
|
||||
|
||||
# Modules that stand alone: their own CLI subcommand or external protocol, deliberately off the scan path.
|
||||
# Reason text for reachable modules, by the requirement that owns them. Not a classification: see audit().
|
||||
SPEC_OWNED = {
|
||||
"push.cache": "pre-push-safety-net task 6.2: bounded artifact publication and leased eviction; runner integration in 6.4",
|
||||
"push.report": "pre-push-safety-net Req 5.1/5.2/5.5/7.3: compact output and complete artifacts; runner integration pending",
|
||||
@@ -38,16 +41,13 @@ SPEC_OWNED = {
|
||||
"push": "pre-push-safety-net Req 5.5/7.3: experimental push contract package; no hook installed",
|
||||
"push.contracts": "pre-push-safety-net Req 5.5/7.3: immutable comparisons and independent result statuses; runner pending",
|
||||
"model.specs": "model-grounded-detection Req 7.2-7.4: the security vocabularies the CPG queries are parameterised by",
|
||||
"model.endpoint": "model-grounded-detection Req 8: the judge client, wired by model/judge.py in group 3",
|
||||
"model.endpoint": "model-grounded-detection Req 8: the bounded model client the hunter, the plane and the CLI share",
|
||||
"model.audit": "model-grounded-detection Req 3: this classifier; run by the maintainer, not by the scan",
|
||||
"model.ladder": "model-grounded-detection Req 5: the evidence ladder every finding carries",
|
||||
"model.taint": "model-grounded-detection Req 6.1: taint reachability over the CPG, the flow-family arbiter",
|
||||
"model.judge": "model-grounded-detection Req 8: one bounded question, checked against the model",
|
||||
"model.dominance": "model-grounded-detection Req 6.2: guard dominance, the arbiter for bugs that never crash",
|
||||
"model.config_value": "model-grounded-detection Req 6.3: constant abstraction for the configuration families",
|
||||
"model.calibrate": "model-grounded-detection Req 9: the corpus as the model's calibration set",
|
||||
"model.report": "model-grounded-detection Req 9.3/11.1: per-slice reporting with the overfitting gap",
|
||||
"model.execution": "model-grounded-detection Req 10: the deferred execution tier seam, adopted not built",
|
||||
"model.pipeline": "model-grounded-detection Req 8: the enumerator proposes, the LLM answers, the model disposes",
|
||||
"model.regions": "contributor-scan Req 2: regions from entry points, with the families each admits",
|
||||
"model.scan": "contributor-scan Req 3: the repository driver -- one CPG, one budget, ranked spend",
|
||||
@@ -81,14 +81,13 @@ SPEC_OWNED = {
|
||||
"plane.runner": "ai-service-plane Req 4.1-4.4: the ax-task-runner entrypoint (PID 1 of the task image); delivers the output directory",
|
||||
}
|
||||
|
||||
# Reachable modules that stand alone: their own CLI subcommand or external protocol, off the scan path.
|
||||
STANDALONE = {
|
||||
"push_scoring": "pre-push-safety-net task 1.3: frozen-profile diagnostic outcome scorer; python -m openultrasast.push_scoring",
|
||||
"push_inputs": "pre-push-safety-net task 1.2: offline input provenance validator; python -m openultrasast.push_inputs",
|
||||
"mcp": "narrow MCP server over stdio; its own `ousast mcp` entry point and OpenCode integration",
|
||||
"skills": "skill router; consumed by mcp and by the OpenCode integration, not by the scan path",
|
||||
"fusion": "two-panel adjudication engine; reached from the scan's report stage and used standalone",
|
||||
"stage_processors": "deterministic slot-contracted scan stages; pinned by the zero-dependency guard in test_gate",
|
||||
"slot_contract": "the slot/contract protocol stage_processors implements",
|
||||
}
|
||||
|
||||
|
||||
@@ -140,6 +139,10 @@ def import_graph(root: Path) -> dict[str, set[str]]:
|
||||
continue
|
||||
if target:
|
||||
imported.add(target)
|
||||
# `from .plane import engine_alerts` imports the submodule `plane.engine_alerts`, not only the
|
||||
# package; names that are not modules are dropped later because they are not in the graph.
|
||||
prefix = f"{target}." if target else ""
|
||||
imported.update(f"{prefix}{alias.name}" for alias in node.names if alias.name != "*")
|
||||
elif isinstance(node, ast.Import):
|
||||
for alias in node.names:
|
||||
if alias.name.startswith(f"{PACKAGE}."):
|
||||
@@ -148,6 +151,28 @@ def import_graph(root: Path) -> dict[str, set[str]]:
|
||||
return graph
|
||||
|
||||
|
||||
def main_modules(root: Path) -> set[str]:
|
||||
"""Modules with a ``__main__`` block: run by name (``python -m``, the plane runner), so they are entry points."""
|
||||
found: set[str] = set()
|
||||
for path in sorted(root.rglob("*.py")):
|
||||
if "__pycache__" in path.parts:
|
||||
continue
|
||||
try:
|
||||
tree = ast.parse(path.read_text())
|
||||
except (OSError, SyntaxError):
|
||||
continue
|
||||
for node in tree.body:
|
||||
if (
|
||||
isinstance(node, ast.If)
|
||||
and isinstance(node.test, ast.Compare)
|
||||
and isinstance(node.test.left, ast.Name)
|
||||
and node.test.left.id == "__name__"
|
||||
and any(isinstance(c, ast.Constant) and c.value == "__main__" for c in node.test.comparators)
|
||||
):
|
||||
found.add(module_name(path, root))
|
||||
return found
|
||||
|
||||
|
||||
def _reachable(graph: Mapping[str, set[str]], roots: Iterable[str]) -> set[str]:
|
||||
seen: set[str] = set()
|
||||
stack = [root for root in roots if root in graph]
|
||||
@@ -173,24 +198,33 @@ def audit(root: Path) -> tuple[ModuleAudit, ...]:
|
||||
for candidate in (target, target.rsplit(".", 1)[0]):
|
||||
if candidate in importers and candidate != name:
|
||||
importers[candidate].add(name)
|
||||
live = _reachable(graph, ROOTS)
|
||||
run_by_name = main_modules(root)
|
||||
live = _reachable(graph, (*ROOTS, *sorted(run_by_name)))
|
||||
results: list[ModuleAudit] = []
|
||||
for name in sorted(graph):
|
||||
if not name: # the package __init__ itself
|
||||
continue
|
||||
who = tuple(sorted(importers[name]))
|
||||
if name in SPEC_OWNED:
|
||||
listed = SPEC_OWNED.get(name) or STANDALONE.get(name)
|
||||
# Reachability decides; the lists only supply reason text. A listed module nothing reaches is still
|
||||
# orphaned -- the list is what hid five dead modules behind "load_bearing" until 2026-10-02.
|
||||
if name not in live:
|
||||
reason = "no import path from an entry point or a python -m module"
|
||||
if who:
|
||||
reason += f" (imported only by unreachable {', '.join(who)})"
|
||||
if listed:
|
||||
reason += f"; listed as: {listed}"
|
||||
results.append(ModuleAudit(name, "orphaned", reason, who))
|
||||
elif name in STANDALONE:
|
||||
results.append(ModuleAudit(name, "standalone_capability", STANDALONE[name], who))
|
||||
elif name in SPEC_OWNED:
|
||||
results.append(ModuleAudit(name, "load_bearing", SPEC_OWNED[name], who))
|
||||
elif name in ROOTS:
|
||||
results.append(ModuleAudit(name, "load_bearing", "entry point", who))
|
||||
elif name in STANDALONE:
|
||||
results.append(ModuleAudit(name, "standalone_capability", STANDALONE[name], who))
|
||||
elif name in live:
|
||||
results.append(ModuleAudit(name, "load_bearing", f"reachable from {', '.join(ROOTS[:3])} or the CLI", who))
|
||||
elif who:
|
||||
results.append(ModuleAudit(name, "load_bearing", f"imported by {', '.join(who)}", who))
|
||||
elif name in run_by_name:
|
||||
results.append(ModuleAudit(name, "load_bearing", f"python -m openultrasast.{name}", who))
|
||||
else:
|
||||
results.append(ModuleAudit(name, "orphaned", "no importer, no entry point, no surviving requirement", who))
|
||||
results.append(ModuleAudit(name, "load_bearing", f"reachable from {', '.join(ROOTS[:4])}", who))
|
||||
return tuple(results)
|
||||
|
||||
|
||||
@@ -211,4 +245,12 @@ def to_dict(results: Iterable[ModuleAudit]) -> dict[str, object]:
|
||||
}
|
||||
|
||||
|
||||
__all__ = ["ModuleAudit", "SPEC_OWNED", "STANDALONE", "audit", "import_graph", "module_name", "to_dict"]
|
||||
__all__ = ["ModuleAudit", "SPEC_OWNED", "STANDALONE", "audit", "import_graph", "main_modules", "module_name", "to_dict"]
|
||||
|
||||
|
||||
if __name__ == "__main__": # python -m openultrasast.model.audit [SRC] -- the manifest's regeneration command
|
||||
import json
|
||||
import sys
|
||||
|
||||
source = Path(sys.argv[1]) if len(sys.argv) > 1 else Path(__file__).resolve().parents[1]
|
||||
print(json.dumps(to_dict(audit(source)), indent=2))
|
||||
|
||||
@@ -1,180 +0,0 @@
|
||||
"""The corpus calibrates the model, not an LLM vote (model-grounded-detection, Req 9).
|
||||
|
||||
This is the inversion the whole feature rests on. In the prior architecture a pair the detector missed was a
|
||||
*detector* failure, to be averaged away over K runs and chased with a prompt change. Here the arbiter is
|
||||
deterministic: it will miss that pair the same way every time, and the only thing that moves it is better
|
||||
modelling. So a pair the model cannot split is a **named gap in the model**, recorded with its reason, and
|
||||
the report's job is to make those gaps legible by family and by slice rather than to produce a score.
|
||||
|
||||
`covered` means the model reached a strictly higher rung on the vulnerable side than on its fix. Flagging
|
||||
both sides equally is not coverage — it distinguishes nothing, which is exactly the failure the group-2
|
||||
measurement found taint reachability making on every parameterised SQL pair.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterable, Mapping, Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from typing import TypedDict
|
||||
|
||||
|
||||
class FamilyGaps(TypedDict):
|
||||
"""Gaps for one family: how many pairs, and the tally of reasons behind them."""
|
||||
|
||||
pairs: int
|
||||
reasons: dict[str, int]
|
||||
|
||||
|
||||
# Ascending, matching `ladder.Rung`. A pair is split when the vulnerable side outranks its fix.
|
||||
_ORDER = {"none": 0, "": 0, "suspicion": 0, "model_corroborated": 1, "model_entailed": 2, "execution_confirmed": 3}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PairOutcome:
|
||||
"""What the model made of one vulnerable/fixed pair."""
|
||||
|
||||
pair: str
|
||||
family: str
|
||||
slice: str
|
||||
vuln_rung: str
|
||||
fixed_rung: str
|
||||
reason: str = ""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ModelGap:
|
||||
"""A pair the model could not split, and why. Not a miss to average — a gap to close."""
|
||||
|
||||
pair: str
|
||||
family: str
|
||||
slice: str
|
||||
reason: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HarvestRecipe:
|
||||
"""Where a pair comes from. The fix commit's parent is the vulnerable side, by construction."""
|
||||
|
||||
repo: str
|
||||
fix_commit: str
|
||||
relpath: str
|
||||
function: str = ""
|
||||
|
||||
@property
|
||||
def parent_ref(self) -> str:
|
||||
return f"{self.fix_commit}^"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CalibrationReport:
|
||||
covered: tuple[str, ...]
|
||||
gaps: tuple[ModelGap, ...]
|
||||
per_slice: Mapping[str, Mapping[str, int]]
|
||||
per_family: Mapping[str, Mapping[str, int]] = field(default_factory=dict)
|
||||
|
||||
def gaps_by_family(self) -> dict[str, FamilyGaps]:
|
||||
"""Gaps grouped by family, each with a tally of the reasons behind them."""
|
||||
out: dict[str, FamilyGaps] = {}
|
||||
for gap in self.gaps:
|
||||
block = out.setdefault(gap.family, {"pairs": 0, "reasons": {}})
|
||||
block["pairs"] += 1
|
||||
block["reasons"][gap.reason] = block["reasons"].get(gap.reason, 0) + 1
|
||||
return out
|
||||
|
||||
def to_markdown(self) -> str:
|
||||
"""Per-slice rows first; there is no headline here that is not backed by them (Req 9.3, 11.1)."""
|
||||
lines = [
|
||||
"## Model calibration",
|
||||
"",
|
||||
"Coverage is a pair the model *splits*: a strictly higher rung on",
|
||||
"the vulnerable side than on its fix. Everything else is a named gap.",
|
||||
"",
|
||||
"| slice | pairs | covered | gaps | coverage |",
|
||||
"|---|---|---|---|---|",
|
||||
]
|
||||
for slice_name, block in sorted(self.per_slice.items()):
|
||||
pairs = block["pairs"] or 1
|
||||
lines.append(f"| {slice_name} | {block['pairs']} | {block['covered']} | {block['gaps']} | {block['covered'] / pairs:.0%} |")
|
||||
by_family = self.gaps_by_family()
|
||||
if by_family:
|
||||
lines += ["", "### Gaps by family", ""]
|
||||
for family, gaps in sorted(by_family.items()):
|
||||
detail = ", ".join(f"{k} x{v}" for k, v in sorted(gaps["reasons"].items()))
|
||||
lines.append(f"- `{family}`: {gaps['pairs']} pair(s) — {detail}")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def calibrate(outcomes: Iterable[PairOutcome]) -> CalibrationReport:
|
||||
"""Split the corpus into what the model covers and what it cannot, with reasons."""
|
||||
covered: list[str] = []
|
||||
gaps: list[ModelGap] = []
|
||||
per_slice: dict[str, dict[str, int]] = {}
|
||||
per_family: dict[str, dict[str, int]] = {}
|
||||
|
||||
for outcome in outcomes:
|
||||
split = _ORDER.get(outcome.vuln_rung, 0) > _ORDER.get(outcome.fixed_rung, 0)
|
||||
for key, store in ((outcome.slice, per_slice), (outcome.family, per_family)):
|
||||
block = store.setdefault(key, {"pairs": 0, "covered": 0, "gaps": 0})
|
||||
block["pairs"] += 1
|
||||
block["covered" if split else "gaps"] += 1
|
||||
if split:
|
||||
covered.append(outcome.pair)
|
||||
else:
|
||||
gaps.append(
|
||||
ModelGap(pair=outcome.pair, family=outcome.family, slice=outcome.slice, reason=outcome.reason or _default_reason(outcome))
|
||||
)
|
||||
return CalibrationReport(
|
||||
covered=tuple(sorted(covered)),
|
||||
gaps=tuple(sorted(gaps, key=lambda g: (g.family, g.pair))),
|
||||
per_slice=per_slice,
|
||||
per_family=per_family,
|
||||
)
|
||||
|
||||
|
||||
def _default_reason(outcome: PairOutcome) -> str:
|
||||
"""Every gap carries a reason, even when the caller supplied none — a bare gap teaches nothing."""
|
||||
if _ORDER.get(outcome.vuln_rung, 0) == 0:
|
||||
return "no_verdict_on_the_vulnerable_side"
|
||||
if outcome.vuln_rung == outcome.fixed_rung:
|
||||
return f"both_sides_reached_{outcome.vuln_rung}: the model distinguished nothing"
|
||||
return "fixed_side_outranked_the_vulnerable_one"
|
||||
|
||||
|
||||
def harvest_recipe(entry: Mapping[str, object]) -> HarvestRecipe:
|
||||
"""A pair from CVE history: the fix commit and its parent, with the fix diff as the oracle (Req 9.4).
|
||||
|
||||
Nothing here is hand-labelled. The recipe names where to fetch both sides; what makes the pair *true* is
|
||||
the fix diff itself, read by the model rather than annotated by us.
|
||||
"""
|
||||
return HarvestRecipe(
|
||||
repo=str(entry.get("repo", "")),
|
||||
fix_commit=str(entry.get("fix_commit", "")),
|
||||
relpath=str(entry.get("relpath", "")),
|
||||
function=str(entry.get("function", "") or ""),
|
||||
)
|
||||
|
||||
|
||||
def outcomes_from_rows(rows: Sequence[Mapping[str, object]]) -> list[PairOutcome]:
|
||||
"""Adapt the ceiling artifact's rows, so calibration reads the measurement rather than re-running it."""
|
||||
return [
|
||||
PairOutcome(
|
||||
pair=str(row.get("pair", "")),
|
||||
family=str(row.get("family", "")),
|
||||
slice=str(row.get("slice", "")),
|
||||
vuln_rung=str(row.get("vuln_rung", "none")),
|
||||
fixed_rung=str(row.get("fixed_rung", "none")),
|
||||
reason=str(row.get("miss_cause", "") or ""),
|
||||
)
|
||||
for row in rows
|
||||
]
|
||||
|
||||
|
||||
__all__ = [
|
||||
"CalibrationReport",
|
||||
"HarvestRecipe",
|
||||
"ModelGap",
|
||||
"PairOutcome",
|
||||
"calibrate",
|
||||
"harvest_recipe",
|
||||
"outcomes_from_rows",
|
||||
]
|
||||
@@ -1,118 +0,0 @@
|
||||
"""The execution tier: deferred, adopted, and deliberately narrow (model-grounded-detection, Req 10).
|
||||
|
||||
This is the rung this project does **not** build. Clearwing already runs sandbox lifecycles, container pools,
|
||||
sanitizer images, PoC replay and stability classification, and the corrective brief that started this feature
|
||||
concluded — from this project's own defect history — that reimplementing that class of machinery is exactly
|
||||
where we go wrong. So the module is a seam: it decides *eligibility*, calls an adopted tool, and records what
|
||||
happened. A test asserts structurally that it imports no container or subprocess machinery and defines no
|
||||
build/run/compile/patch function, because the failure mode to prevent is this file quietly growing a sandbox.
|
||||
|
||||
Eligibility is narrow on purpose, and both halves matter:
|
||||
|
||||
* **Only a family a static model cannot arbitrate.** Memory safety in C turns on runtime layout, so a crash
|
||||
is the right oracle. Every web/logic family has taint reachability, guard dominance or constant
|
||||
abstraction, and Req 10.3 forbids any of them depending on this tier.
|
||||
* **Only a finding the model could not decide.** The tier escalates uncertainty; it never re-checks a verdict
|
||||
the model already reached.
|
||||
|
||||
And a failure to reproduce is *not* a refutation. A sandbox that does not trigger a bug has not shown the bug
|
||||
is absent — it has shown this attempt did not trigger it. The candidate stays exactly where the model left it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from typing import Protocol
|
||||
|
||||
from .ladder import Rung
|
||||
|
||||
# Families whose bugs a static model genuinely cannot arbitrate. Everything else is served by taint
|
||||
# reachability, guard dominance or constant abstraction, and must not wait on a sandbox.
|
||||
EXECUTION_FAMILIES = frozenset({"memory"})
|
||||
|
||||
# Rungs that mean "the model could not decide". Anything above is already arbitrated.
|
||||
UNDECIDED = frozenset({"suspicion", ""})
|
||||
|
||||
CLEARWING_ENV = "OPENULTRASAST_CLEARWING"
|
||||
_OFF = frozenset({"0", "off", "false", "no"})
|
||||
_ON = frozenset({"1", "on", "true", "yes"})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Candidate:
|
||||
"""What the execution tier is asked about: a finding the static model left undecided."""
|
||||
|
||||
finding_id: str
|
||||
family: str
|
||||
rung: str
|
||||
path: str
|
||||
line: int | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Outcome:
|
||||
"""What the tier concluded. ``rung`` is ``None`` whenever nothing was established."""
|
||||
|
||||
rung: Rung | None
|
||||
reason: str
|
||||
|
||||
|
||||
class ExecutionBackend(Protocol):
|
||||
def available(self) -> bool: ...
|
||||
|
||||
def reproduce(self, candidate: Candidate) -> bool: ...
|
||||
|
||||
|
||||
@dataclass
|
||||
class NullExecutionBackend:
|
||||
"""No adopted tool present. The normal case, and not an error."""
|
||||
|
||||
def available(self) -> bool:
|
||||
return False
|
||||
|
||||
def reproduce(self, candidate: Candidate) -> bool:
|
||||
del candidate
|
||||
return False
|
||||
|
||||
|
||||
def eligible(candidate: Candidate) -> bool:
|
||||
"""Is this candidate one the execution tier should even look at?"""
|
||||
return candidate.family in EXECUTION_FAMILIES and candidate.rung in UNDECIDED
|
||||
|
||||
|
||||
def confirm(candidate: Candidate, *, backend: ExecutionBackend | None = None) -> Outcome:
|
||||
"""Escalate one undecided candidate to the adopted execution tier, if there is one."""
|
||||
if not eligible(candidate):
|
||||
return Outcome(
|
||||
rung=None,
|
||||
reason=f"not_eligible: family {candidate.family!r} at rung {candidate.rung!r} is arbitrated statically",
|
||||
)
|
||||
engine = backend if backend is not None else resolve_execution_backend()
|
||||
if not engine.available():
|
||||
return Outcome(rung=None, reason="clearwing_unavailable: no adopted execution tier on this machine")
|
||||
if not engine.reproduce(candidate):
|
||||
# Deliberate wording: this is not a refutation. Failing to trigger a bug is not showing it is absent.
|
||||
return Outcome(rung=None, reason="not_reproduced: the attempt did not trigger it; the finding is unchanged")
|
||||
return Outcome(rung=Rung.EXECUTION_CONFIRMED, reason="reproduced in the adopted sandbox")
|
||||
|
||||
|
||||
def resolve_execution_backend() -> ExecutionBackend:
|
||||
"""The adopted tier for this machine. Absent by default; the environment flag is what the matrix drives."""
|
||||
flag = os.environ.get(CLEARWING_ENV, "").strip().lower()
|
||||
if flag in _OFF or flag not in _ON:
|
||||
return NullExecutionBackend()
|
||||
return NullExecutionBackend() # the adopted tool is wired here when a class needs it (Req 10.1)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"CLEARWING_ENV",
|
||||
"EXECUTION_FAMILIES",
|
||||
"Candidate",
|
||||
"ExecutionBackend",
|
||||
"NullExecutionBackend",
|
||||
"Outcome",
|
||||
"confirm",
|
||||
"eligible",
|
||||
"resolve_execution_backend",
|
||||
]
|
||||
@@ -1,136 +0,0 @@
|
||||
"""The LLM proposes; the model disposes (model-grounded-detection, Req 8).
|
||||
|
||||
This module is where the prior architecture's premise is inverted. There, the LLM was an oracle whose noise
|
||||
had to be averaged over K runs, floored per family, and gated by a sign test. Here it answers one bounded,
|
||||
typed question about a candidate it was *handed*, and its answer is checked against the CPG. Three outcomes,
|
||||
and they are exhaustive:
|
||||
|
||||
the model entails it report `model_entailed`, and never call the model at all
|
||||
the model contradicts it drop the claim, recording the contradiction
|
||||
the model cannot decide report `suspicion`, honestly
|
||||
|
||||
Nothing here averages, votes, retries for agreement, or maintains a budget: a deterministic arbiter removes
|
||||
the need for every one of those. The LLM is consulted only for the residual the graph cannot settle — whether
|
||||
a sanitizer on a real flow is actually sufficient — and even that answer only ever *confirms* a flow the CPG
|
||||
already found. It can never conjure one.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
from ..cpg.backend import CpgResult
|
||||
from ..redaction import redact_secrets
|
||||
from .ladder import Rung, Verdict
|
||||
from .specs import TaintSpec
|
||||
from .taint import verdict as taint_verdict
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
QUESTION = """You are judging ONE candidate a static model already found. Do not look for other issues.
|
||||
|
||||
A dataflow analysis resolved this path in {language} code:
|
||||
|
||||
source: {source}
|
||||
sink: {sink} (line {line}, function {function})
|
||||
a sanitizer appears on the path: {sanitized}
|
||||
|
||||
The graph has established that the value reaches the sink. The only open question is whether the sanitizer on
|
||||
that path actually makes it safe for the {family} weakness class.
|
||||
|
||||
Answer ONLY with JSON: {{"vulnerable": true|false, "why": "<one sentence>"}}"""
|
||||
|
||||
|
||||
def judge(
|
||||
cpg: CpgResult,
|
||||
spec: TaintSpec,
|
||||
*,
|
||||
function: str = "",
|
||||
client: Any | None = None,
|
||||
model: str = "",
|
||||
parameter_sources: bool = False,
|
||||
) -> Verdict:
|
||||
"""Arbitrate one candidate. Always returns a verdict; ``suspicion`` is a real answer, not a failure."""
|
||||
answer = taint_verdict(cpg, spec, function=function, parameter_sources=parameter_sources)
|
||||
|
||||
if answer is not None and answer.rung is Rung.ENTAILED:
|
||||
return answer # the graph decided; the model is not consulted
|
||||
|
||||
if answer is None:
|
||||
# No resolved flow. Whatever anyone claims about this site, the model saw nothing to support it.
|
||||
return Verdict(
|
||||
rung=Rung.SUSPICION,
|
||||
family=spec.family,
|
||||
contradiction="the model resolved no flow: no source reaches this sink",
|
||||
)
|
||||
|
||||
# A real flow through a sanitizer. Whether that sanitizer suffices is a semantic judgement the graph
|
||||
# cannot make, so it is the one thing the LLM is asked -- about evidence the model itself produced.
|
||||
if client is None:
|
||||
return Verdict(
|
||||
rung=Rung.SUSPICION,
|
||||
family=spec.family,
|
||||
witness=answer.witness,
|
||||
contradiction="no chat endpoint: the sanitizer's sufficiency was never judged",
|
||||
)
|
||||
|
||||
said_vulnerable = _ask(client, model, spec, answer.witness, function)
|
||||
if said_vulnerable is None:
|
||||
return Verdict(rung=Rung.SUSPICION, family=spec.family, witness=answer.witness, contradiction="the judge returned no usable answer")
|
||||
if not said_vulnerable:
|
||||
return Verdict(
|
||||
rung=Rung.SUSPICION,
|
||||
family=spec.family,
|
||||
witness=answer.witness,
|
||||
contradiction="the judge held the sanitizer on this path sufficient",
|
||||
)
|
||||
return answer # CORROBORATED: a real flow the judge agrees is not neutralised
|
||||
|
||||
|
||||
def _ask(client: Any, model: str, spec: TaintSpec, witness: str, function: str) -> bool | None:
|
||||
"""One question, one answer. No tools, no retries for agreement, no second opinion."""
|
||||
source, _, sink = witness.partition(" -> ")
|
||||
line = ""
|
||||
if "(line " in sink:
|
||||
sink, _, tail = sink.partition(" (line ")
|
||||
line = tail.split(",")[0].strip(") ")
|
||||
prompt = redact_secrets(
|
||||
QUESTION.format(
|
||||
language=spec.language,
|
||||
source=source.strip(),
|
||||
sink=sink.strip(),
|
||||
line=line or "?",
|
||||
function=function or "?",
|
||||
sanitized="yes",
|
||||
family=spec.family,
|
||||
)
|
||||
)
|
||||
try:
|
||||
response = client.complete(
|
||||
model=model,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
tools=[], # the candidate is given; the model never searches for it
|
||||
json_object=True,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 -- an endpoint failure is a degradation, never a verdict
|
||||
logger.warning("judge call failed: %s", exc)
|
||||
return None
|
||||
return _vulnerable(getattr(response, "content", None))
|
||||
|
||||
|
||||
def _vulnerable(content: object) -> bool | None:
|
||||
if not isinstance(content, str) or not content.strip():
|
||||
return None
|
||||
try:
|
||||
payload = json.loads(content)
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
if isinstance(payload, Mapping) and isinstance(payload.get("vulnerable"), bool):
|
||||
return bool(payload["vulnerable"])
|
||||
return None
|
||||
|
||||
|
||||
__all__ = ["QUESTION", "judge"]
|
||||
@@ -4,8 +4,8 @@ Everything before this module built an *arbiter* — something that judges a flo
|
||||
found on its own. Nothing proposed candidates to it, which is why the arbiter's recall was never comparable to
|
||||
a detector's: it could only report what it happened to find unaided.
|
||||
|
||||
This is the wiring the architecture was designed around, and it turns on a distinction that `judge` alone
|
||||
conflated:
|
||||
This is the wiring the architecture was designed around, and it turns on a distinction that the first judge
|
||||
(`model/judge.py`, deleted 2026-10-02 because this module replaced it) conflated:
|
||||
|
||||
the model CONTRADICTS it has coverage at this site and says no — a sanitizer lies on the path, or the
|
||||
sink is written in its declared-safe shape. The claim is dropped (Req 8.2).
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Secret redaction for traces and reports (Phase 15 hardening).
|
||||
"""Secret redaction for traces and reports (Phase 15 hardening of the retired openrouter-sast-harness plan).
|
||||
|
||||
Scan artifacts (trace events, reports) can quote source code that contains live
|
||||
credentials. This module masks well-known secret shapes before they are persisted, so
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Trail of Bits skill index + router (Phase 14).
|
||||
"""Trail of Bits skill index + router (Phase 14 of the retired openrouter-sast-harness plan).
|
||||
|
||||
Security skills are scoped expertise, not global prompt bloat. Each skill is a small
|
||||
descriptor with routing metadata (languages, tags, vulnerability classes, stages) and
|
||||
|
||||
@@ -1,88 +0,0 @@
|
||||
"""Slot-contract discipline for model-disabled deterministic stages (task 6.3).
|
||||
|
||||
The deterministic scan stages carry their work in named *slots* and run with model
|
||||
invocation disabled (``skip_model = True``). Each stage declares a read/write slot
|
||||
allow-list that is validated on every lifecycle hook, preserving the
|
||||
``ProcessorSpec.reads/writes`` provenance discipline of the sync ``HarnessRuntime``
|
||||
(the top-level driver).
|
||||
|
||||
This is the zero-dependency substrate for the deterministic pipeline.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from typing import Any, Protocol
|
||||
|
||||
from .harness import HarnessContractError
|
||||
|
||||
|
||||
class SlotContractError(HarnessContractError):
|
||||
"""Raised when a stage reads or writes a slot outside its declared allow-list."""
|
||||
|
||||
|
||||
class SlotContractMixin:
|
||||
"""A model-disabled stage that carries work in slots under a read/write allow-list."""
|
||||
|
||||
name: str = "slot-stage"
|
||||
version: str = "1"
|
||||
reads_slots: tuple[str, ...] = ()
|
||||
writes_slots: tuple[str, ...] = ()
|
||||
skip_model: bool = True # deterministic: never invokes a model
|
||||
|
||||
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||
raise NotImplementedError
|
||||
|
||||
def validate_reads(self, slots: Mapping[str, Any]) -> None:
|
||||
missing = sorted(slot for slot in self.reads_slots if slot not in slots)
|
||||
if missing:
|
||||
raise SlotContractError(f"{self.name}: missing read slots: {', '.join(missing)}")
|
||||
|
||||
def validate_writes(self, updates: Mapping[str, Any]) -> None:
|
||||
undeclared = sorted(set(updates) - set(self.writes_slots))
|
||||
if undeclared:
|
||||
raise SlotContractError(f"{self.name}: undeclared write slots: {', '.join(undeclared)}")
|
||||
|
||||
def execute(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||
"""Validate the read allow-list, run the stage, validate the write allow-list."""
|
||||
self.validate_reads(slots) # pre-hook
|
||||
updates = self.run(slots) or {}
|
||||
if not isinstance(updates, dict):
|
||||
raise SlotContractError(f"{self.name}: stage returned a non-dict update")
|
||||
self.validate_writes(updates) # post-hook
|
||||
return updates
|
||||
|
||||
|
||||
class _Emitter(Protocol):
|
||||
def emit(self, event_type: Any, stage: str | None, payload: dict[str, Any]) -> None: ...
|
||||
|
||||
|
||||
class SlotPipeline:
|
||||
"""Runs slot processors in order, validating contracts and enforcing model-disable."""
|
||||
|
||||
def __init__(self, processors: list[SlotContractMixin]) -> None:
|
||||
for proc in processors:
|
||||
if not proc.skip_model:
|
||||
raise SlotContractError(f"{proc.name}: deterministic stage must be model-disabled (skip_model=True)")
|
||||
self._processors = processors
|
||||
|
||||
@property
|
||||
def processors(self) -> list[SlotContractMixin]:
|
||||
return list(self._processors)
|
||||
|
||||
def run(self, initial_slots: Mapping[str, Any], *, runtime: _Emitter | None = None) -> dict[str, Any]:
|
||||
"""Execute every stage; with ``runtime`` set, emit stage_start/stage_end lifecycle hooks."""
|
||||
slots: dict[str, Any] = dict(initial_slots)
|
||||
for proc in self._processors:
|
||||
if runtime is not None:
|
||||
runtime.emit(
|
||||
"stage_start", proc.name, {"reads": list(proc.reads_slots), "writes": list(proc.writes_slots), "skip_model": True}
|
||||
)
|
||||
updates = proc.execute(slots)
|
||||
slots.update(updates)
|
||||
if runtime is not None:
|
||||
runtime.emit("stage_end", proc.name, {"status": "succeeded", "wrote": sorted(updates)})
|
||||
return slots
|
||||
|
||||
|
||||
__all__ = ["SlotContractError", "SlotContractMixin", "SlotPipeline"]
|
||||
@@ -1,135 +0,0 @@
|
||||
"""Deterministic scan stages as model-disabled, slot-contracted processors (task 6.3).
|
||||
|
||||
Each stage is a :class:`slot_contract.SlotContractMixin` with ``skip_model = True`` and
|
||||
a declared read/write slot allow-list. Running them through a :class:`SlotPipeline`
|
||||
reproduces the deterministic quick-scan path byte-for-byte while enforcing the slot
|
||||
contract on every lifecycle hook. The hunter and verifier stages are *not* here; this
|
||||
module is the deterministic, model-free remainder that ``test_gate`` pins.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from .findings import quick_scan_findings
|
||||
from .mapping import analyze_entry_points, attach_reachability_hints
|
||||
from .policy import load_policy
|
||||
from .preprocess import preprocess_repository
|
||||
from .rank import rank_targets
|
||||
from .ruleset import DEFAULT_RULESET_DIR, load_ruleset
|
||||
from .scoring import build_score_artifact
|
||||
from .slot_contract import SlotContractMixin, SlotPipeline
|
||||
|
||||
|
||||
class PreprocessProcessor(SlotContractMixin):
|
||||
name = "preprocess"
|
||||
reads_slots = ("root",)
|
||||
writes_slots = ("snapshot", "targets")
|
||||
|
||||
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||
snapshot, targets = preprocess_repository(slots["root"])
|
||||
return {"snapshot": snapshot, "targets": targets}
|
||||
|
||||
|
||||
class EntryPointProcessor(SlotContractMixin):
|
||||
name = "entry_point"
|
||||
reads_slots = ("root", "targets")
|
||||
writes_slots = ("targets", "entry_points")
|
||||
|
||||
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||
entry_points = analyze_entry_points(slots["root"], slots["targets"])
|
||||
targets = attach_reachability_hints(slots["targets"], entry_points)
|
||||
return {"targets": targets, "entry_points": entry_points}
|
||||
|
||||
|
||||
class RankProcessor(SlotContractMixin):
|
||||
name = "rank"
|
||||
reads_slots = ("targets",)
|
||||
writes_slots = ("rankings",)
|
||||
|
||||
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||
return {"rankings": rank_targets(slots["targets"])}
|
||||
|
||||
|
||||
class QuickFindingsProcessor(SlotContractMixin):
|
||||
name = "quick_findings"
|
||||
reads_slots = ("root", "targets", "rankings", "ruleset", "policy")
|
||||
writes_slots = ("findings",)
|
||||
|
||||
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||
findings = quick_scan_findings(slots["root"], slots["targets"], slots["rankings"], slots["ruleset"], slots["policy"])
|
||||
return {"findings": findings}
|
||||
|
||||
|
||||
class PolicyScoringProcessor(SlotContractMixin):
|
||||
name = "policy_scoring"
|
||||
reads_slots = ("findings",)
|
||||
writes_slots = ("reported_findings", "shadow_findings")
|
||||
|
||||
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||
findings = slots["findings"]
|
||||
shadow = [finding for finding in findings if finding.status == "shadow"]
|
||||
reported = [finding for finding in findings if finding.status != "shadow"]
|
||||
return {"reported_findings": reported, "shadow_findings": shadow}
|
||||
|
||||
|
||||
class ScoreProcessor(SlotContractMixin):
|
||||
name = "score"
|
||||
reads_slots = ("reported_findings", "ruleset", "policy")
|
||||
writes_slots = ("score",)
|
||||
|
||||
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||
cwe_by_rule = {rule.rule_id: rule.cwe for rule in slots["ruleset"]}
|
||||
reported = slots["reported_findings"]
|
||||
rule_cwe_by_id = {f.finding_id: cwe_by_rule.get(f.finding_id.split(":", 1)[0], "") for f in reported}
|
||||
return {"score": build_score_artifact(reported, rule_cwe_by_id, slots["policy"])}
|
||||
|
||||
|
||||
def deterministic_processors() -> list[SlotContractMixin]:
|
||||
"""The ordered deterministic stages (preprocess → entry-point → rank → findings → policy → score)."""
|
||||
return [
|
||||
PreprocessProcessor(),
|
||||
EntryPointProcessor(),
|
||||
RankProcessor(),
|
||||
QuickFindingsProcessor(),
|
||||
PolicyScoringProcessor(),
|
||||
ScoreProcessor(),
|
||||
]
|
||||
|
||||
|
||||
def build_deterministic_pipeline() -> SlotPipeline:
|
||||
return SlotPipeline(deterministic_processors())
|
||||
|
||||
|
||||
def run_quick_pipeline(
|
||||
root: Path | str,
|
||||
*,
|
||||
ruleset: Any = None,
|
||||
policy: Any = None,
|
||||
runtime: Any = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Run the deterministic quick-scan pipeline and return the final slot map.
|
||||
|
||||
Defaults to the bundled ruleset and the vendored policy, which makes the
|
||||
``reported_findings`` slot byte-identical to ``findings.quick_scan_findings`` on
|
||||
the same target.
|
||||
"""
|
||||
resolved_ruleset = ruleset if ruleset is not None else load_ruleset(DEFAULT_RULESET_DIR)
|
||||
resolved_policy = policy if policy is not None else load_policy()
|
||||
initial = {"root": Path(root), "ruleset": resolved_ruleset, "policy": resolved_policy}
|
||||
return build_deterministic_pipeline().run(initial, runtime=runtime)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"EntryPointProcessor",
|
||||
"PolicyScoringProcessor",
|
||||
"PreprocessProcessor",
|
||||
"QuickFindingsProcessor",
|
||||
"RankProcessor",
|
||||
"ScoreProcessor",
|
||||
"build_deterministic_pipeline",
|
||||
"deterministic_processors",
|
||||
"run_quick_pipeline",
|
||||
]
|
||||
+30
-6
@@ -1,3 +1,4 @@
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from dataclasses import replace
|
||||
@@ -17,9 +18,6 @@ def test_load_config_reads_toml(tmp_path: Path) -> None:
|
||||
'ranker = "openrouter/test-ranker"',
|
||||
"[sandbox]",
|
||||
"memory_mb = 1024",
|
||||
"[dynamic]",
|
||||
"enabled = true",
|
||||
'network_scope = ["127.0.0.1:8080"]',
|
||||
]
|
||||
)
|
||||
)
|
||||
@@ -29,8 +27,6 @@ def test_load_config_reads_toml(tmp_path: Path) -> None:
|
||||
assert config.models.ranker == "openrouter/test-ranker"
|
||||
assert config.embeddings.store == "json-local"
|
||||
assert config.sandbox.memory_mb == 1024
|
||||
assert config.dynamic.enabled is True
|
||||
assert config.dynamic.network_scope == ("127.0.0.1:8080",)
|
||||
|
||||
|
||||
def test_retired_agentic_section_loads_with_one_warning(tmp_path: Path, caplog: pytest.LogCaptureFixture) -> None:
|
||||
@@ -67,6 +63,31 @@ def test_retired_llm_keys_fail_naming_the_replacement(tmp_path: Path, body: str,
|
||||
assert key in message and replacement in message and "ousast plane run" in message and "Remove the key" in message
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("body", "key"),
|
||||
[
|
||||
('[models]\npatcher = "gpt-4o"\n', "`[models] patcher`"),
|
||||
("[dynamic]\nenabled = true\n", "`[dynamic] enabled`"),
|
||||
('[dynamic]\nnetwork_scope = ["127.0.0.1:8080"]\n', "`[dynamic] network_scope`"),
|
||||
('[evidence]\nminimum_report_verified = "static_corroboration"\n', "`[evidence] minimum_report_verified`"),
|
||||
('[evidence]\nminimum_exploit = "crash_reproduced"\n', "`[evidence] minimum_exploit`"),
|
||||
('[evidence]\nminimum_patch = "root_cause_explained"\n', "`[evidence] minimum_patch`"),
|
||||
("[variants]\nenabled = false\n", "`[variants] enabled`"),
|
||||
("[variants]\nmax_mechanisms = 10\n", "`[variants] max_mechanisms`"),
|
||||
],
|
||||
)
|
||||
def test_unread_keys_fail_naming_why(tmp_path: Path, body: str, key: str) -> None:
|
||||
"""Keys that loaded but were never read are retired (2026-10-02) the same way, each naming its reason."""
|
||||
config_path = tmp_path / "openultrasast.toml"
|
||||
config_path.write_text(body)
|
||||
|
||||
with pytest.raises(RetiredConfigError) as raised:
|
||||
load_config(config_path)
|
||||
|
||||
message = str(raised.value)
|
||||
assert key in message and "retired 2026-10-02 because nothing ever read it" in message and "Remove the key" in message
|
||||
|
||||
|
||||
def test_the_cli_exits_2_on_a_retired_key(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
|
||||
from openultrasast.cli import main
|
||||
|
||||
@@ -116,7 +137,10 @@ def test_write_resolved_config_creates_json(tmp_path: Path) -> None:
|
||||
write_resolved_config(load_config(None), output)
|
||||
|
||||
assert output.exists()
|
||||
assert '"minimum_report_verified": "static_corroboration"' in output.read_text()
|
||||
payload = json.loads(output.read_text())
|
||||
assert payload["sandbox"]["memory_mb"] == 2048
|
||||
# retired 2026-10-02 (never read): the unread sections are not echoed as if they configured something
|
||||
assert not {"dynamic", "evidence", "variants"} & set(payload) and "patcher" not in payload["models"]
|
||||
|
||||
|
||||
def test_complexity_and_regress_defaults_when_section_absent() -> None:
|
||||
|
||||
@@ -134,8 +134,6 @@ GOVERNANCE_SCORING_BENCHMARK_PLANES = (
|
||||
"openultrasast.ruleset.store",
|
||||
"openultrasast.improve",
|
||||
"openultrasast.improve.evolve",
|
||||
"openultrasast.slot_contract",
|
||||
"openultrasast.stage_processors",
|
||||
"openultrasast.fusion",
|
||||
"openultrasast.skills",
|
||||
"openultrasast.redaction",
|
||||
|
||||
@@ -12,7 +12,7 @@ import json
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
_SPEC = importlib.util.spec_from_file_location("hxr_equality", ROOT / "benchmarks" / "harnessx_removal_equality.py")
|
||||
_SPEC = importlib.util.spec_from_file_location("hxr_equality", ROOT / "benchmarks" / "archive" / "harnessx_removal_equality.py")
|
||||
assert _SPEC is not None and _SPEC.loader is not None
|
||||
equality = importlib.util.module_from_spec(_SPEC)
|
||||
_SPEC.loader.exec_module(equality)
|
||||
|
||||
@@ -1,97 +0,0 @@
|
||||
"""model-grounded-detection task 5.1: the corpus calibrates the MODEL, not an LLM vote (Req 9).
|
||||
|
||||
The inversion this feature rests on. Previously a pair the detector missed was a detector failure to be
|
||||
averaged away over K runs. Here a pair the model cannot split is a *named gap in the model*, with the reason
|
||||
recorded, because the model is deterministic: it will miss that pair the same way every time, and the only
|
||||
thing that changes it is better modelling.
|
||||
|
||||
`covered` therefore means the model reached a strictly higher rung on the vulnerable side than on its fix --
|
||||
not that it flagged something. Flagging both sides equally distinguishes nothing.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def _outcome(name, family="injection", slice_name="vibe-py", vuln="model_entailed", fixed="none", reason=""): # type: ignore[no-untyped-def]
|
||||
from openultrasast.model.calibrate import PairOutcome
|
||||
|
||||
return PairOutcome(pair=name, family=family, slice=slice_name, vuln_rung=vuln, fixed_rung=fixed, reason=reason)
|
||||
|
||||
|
||||
def test_a_pair_the_model_splits_is_covered() -> None:
|
||||
from openultrasast.model.calibrate import calibrate
|
||||
|
||||
report = calibrate([_outcome("a")])
|
||||
assert report.covered == ("a",)
|
||||
assert report.gaps == ()
|
||||
|
||||
|
||||
def test_a_pair_the_model_cannot_split_is_a_named_gap_not_a_miss() -> None:
|
||||
"""Req 9.2: the reason travels with the gap, because a deterministic model fails the same way every time."""
|
||||
from openultrasast.model.calibrate import calibrate
|
||||
|
||||
report = calibrate([_outcome("b", vuln="none", fixed="none", reason="sink_not_in_table")])
|
||||
assert report.gaps and report.gaps[0].pair == "b"
|
||||
assert report.gaps[0].reason == "sink_not_in_table"
|
||||
assert report.covered == ()
|
||||
|
||||
|
||||
def test_flagging_both_sides_equally_is_a_gap_not_a_success() -> None:
|
||||
"""The model saw something on the fix too, so it distinguished nothing — that is the whole metric."""
|
||||
from openultrasast.model.calibrate import calibrate
|
||||
|
||||
report = calibrate([_outcome("c", vuln="model_entailed", fixed="model_entailed")])
|
||||
assert report.covered == ()
|
||||
assert report.gaps[0].reason, "an unsplit pair must carry a reason even when none was supplied"
|
||||
|
||||
|
||||
def test_gaps_are_grouped_by_family_with_their_reasons() -> None:
|
||||
from openultrasast.model.calibrate import calibrate
|
||||
|
||||
report = calibrate(
|
||||
[
|
||||
_outcome("a", family="path", vuln="none", reason="sink_not_in_table"),
|
||||
_outcome("b", family="path", vuln="none", reason="sink_not_in_table"),
|
||||
_outcome("c", family="injection", vuln="none", reason="source_unmodelled"),
|
||||
_outcome("d", family="injection"),
|
||||
]
|
||||
)
|
||||
by_family = report.gaps_by_family()
|
||||
assert set(by_family) == {"path", "injection"}
|
||||
assert by_family["path"]["pairs"] == 2
|
||||
assert by_family["path"]["reasons"] == {"sink_not_in_table": 2}
|
||||
assert by_family["injection"]["pairs"] == 1
|
||||
|
||||
|
||||
def test_the_report_is_per_slice_and_names_the_deciding_one() -> None:
|
||||
"""Req 9.3/11.1: the real-world slice decides; a development slice must not stand in for the corpus."""
|
||||
from openultrasast.model.calibrate import calibrate
|
||||
|
||||
report = calibrate(
|
||||
[
|
||||
_outcome("a", slice_name="vibe-py"),
|
||||
_outcome("b", slice_name="vfc-js", vuln="none", reason="sink_not_in_table"),
|
||||
]
|
||||
)
|
||||
assert set(report.per_slice) == {"vibe-py", "vfc-js"}
|
||||
assert report.per_slice["vibe-py"]["covered"] == 1
|
||||
assert report.per_slice["vfc-js"]["gaps"] == 1
|
||||
|
||||
|
||||
def test_the_report_refuses_a_bare_aggregate() -> None:
|
||||
from openultrasast.model.calibrate import calibrate
|
||||
|
||||
text = calibrate([_outcome("a"), _outcome("b", slice_name="vfc-js", vuln="none")]).to_markdown()
|
||||
assert "vibe-py" in text and "vfc-js" in text, "no headline without its per-slice rows"
|
||||
assert "coverage" in text.lower()
|
||||
|
||||
|
||||
def test_a_cve_recipe_yields_the_parent_and_fix_commits() -> None:
|
||||
"""Req 9.4: the harvest seam. The fix diff is the oracle; nothing is hand-labelled."""
|
||||
from openultrasast.model.calibrate import harvest_recipe
|
||||
|
||||
recipe = harvest_recipe({"repo": "org/proj", "fix_commit": "abc123", "relpath": "app.py", "function": "run"})
|
||||
assert recipe.repo == "org/proj"
|
||||
assert recipe.fix_commit == "abc123"
|
||||
assert recipe.parent_ref == "abc123^", "the vulnerable side is the fix commit's parent, by construction"
|
||||
assert recipe.relpath == "app.py" and recipe.function == "run"
|
||||
@@ -1,116 +0,0 @@
|
||||
"""model-grounded-detection task 6.1: the execution tier is deferred, adopted, and narrow (Req 10).
|
||||
|
||||
The rung this project does NOT build. Clearwing already runs sandboxes, container pools, sanitizer images and
|
||||
PoC stability, and this session's own history says reimplementing that class of machinery is where we go
|
||||
wrong. So `execution_confirmed` exists as a rung, the seam calls an adopted tool, and the eligibility rule is
|
||||
deliberately narrow: only a finding the static model could not decide, in a family a static model genuinely
|
||||
cannot arbitrate.
|
||||
|
||||
Everything here runs with Clearwing absent, which is the normal case.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
def _finding(family="memory", rung="suspicion"): # type: ignore[no-untyped-def]
|
||||
from openultrasast.model.execution import Candidate
|
||||
|
||||
return Candidate(finding_id="f1", family=family, rung=rung, path="a.c", line=10)
|
||||
|
||||
|
||||
def test_only_a_family_a_static_model_cannot_arbitrate_is_eligible() -> None:
|
||||
"""Req 10.3: no web/logic class depends on this tier. They have taint, dominance and constant abstraction."""
|
||||
from openultrasast.model.execution import eligible
|
||||
|
||||
assert eligible(_finding(family="memory")) is True
|
||||
for family in ("injection", "access_control", "path", "config_secrets", "output_encoding"):
|
||||
assert eligible(_finding(family=family)) is False, f"{family} must be arbitrated statically"
|
||||
|
||||
|
||||
def test_a_finding_the_model_already_decided_is_not_escalated() -> None:
|
||||
"""Req 10.1: the tier is reached only where the model could not decide -- never to double-check itself."""
|
||||
from openultrasast.model.execution import eligible
|
||||
|
||||
assert eligible(_finding(rung="suspicion")) is True
|
||||
assert eligible(_finding(rung="model_entailed")) is False
|
||||
assert eligible(_finding(rung="model_corroborated")) is False
|
||||
|
||||
|
||||
def test_without_clearwing_the_rung_is_never_reached_and_the_reason_is_recorded(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Req 11.3: an absent capability degrades to a recorded reason, never a traceback and never a verdict."""
|
||||
from openultrasast.model.execution import confirm, resolve_execution_backend
|
||||
|
||||
monkeypatch.setenv("OPENULTRASAST_CLEARWING", "off")
|
||||
backend = resolve_execution_backend()
|
||||
assert backend.available() is False
|
||||
outcome = confirm(_finding(), backend=backend)
|
||||
assert outcome.rung is None, "no engine, no rung"
|
||||
assert "clearwing_unavailable" in outcome.reason
|
||||
|
||||
|
||||
def test_an_ineligible_finding_is_not_sent_to_the_sandbox_even_when_one_exists() -> None:
|
||||
from openultrasast.model.execution import confirm
|
||||
|
||||
class _Backend:
|
||||
def __init__(self) -> None:
|
||||
self.calls = 0
|
||||
|
||||
def available(self) -> bool:
|
||||
return True
|
||||
|
||||
def reproduce(self, candidate: object) -> bool:
|
||||
self.calls += 1
|
||||
return True
|
||||
|
||||
backend = _Backend()
|
||||
outcome = confirm(_finding(family="injection"), backend=backend)
|
||||
assert outcome.rung is None and backend.calls == 0
|
||||
assert "not_eligible" in outcome.reason
|
||||
|
||||
|
||||
def test_a_reproduced_candidate_reaches_execution_confirmed() -> None:
|
||||
from openultrasast.model.execution import confirm
|
||||
from openultrasast.model.ladder import Rung
|
||||
|
||||
class _Backend:
|
||||
def available(self) -> bool:
|
||||
return True
|
||||
|
||||
def reproduce(self, candidate: object) -> bool:
|
||||
return True
|
||||
|
||||
assert confirm(_finding(), backend=_Backend()).rung is Rung.EXECUTION_CONFIRMED
|
||||
|
||||
|
||||
def test_a_candidate_that_does_not_reproduce_stays_where_it_was() -> None:
|
||||
"""Not reproducing is not a refutation: the sandbox failing to trigger a bug does not prove its absence."""
|
||||
from openultrasast.model.execution import confirm
|
||||
|
||||
class _Backend:
|
||||
def available(self) -> bool:
|
||||
return True
|
||||
|
||||
def reproduce(self, candidate: object) -> bool:
|
||||
return False
|
||||
|
||||
outcome = confirm(_finding(), backend=_Backend())
|
||||
assert outcome.rung is None
|
||||
assert "not_reproduced" in outcome.reason and "refut" not in outcome.reason.lower()
|
||||
|
||||
|
||||
def test_the_seam_reimplements_no_sandbox_machinery() -> None:
|
||||
"""Req 10.2, asserted structurally. The one thing this module must not grow is a sandbox."""
|
||||
import ast
|
||||
from pathlib import Path
|
||||
|
||||
source = Path("src/openultrasast/model/execution.py").read_text()
|
||||
tree = ast.parse(source)
|
||||
imported = {n.module.split(".")[0] for n in ast.walk(tree) if isinstance(n, ast.ImportFrom) and n.module}
|
||||
imported |= {a.name.split(".")[0] for n in ast.walk(tree) if isinstance(n, ast.Import) for a in n.names}
|
||||
for banned in ("docker", "podman", "subprocess", "container", "sandbox"):
|
||||
assert banned not in imported, f"the execution tier must adopt, not build: {banned}"
|
||||
names = {n.name for n in ast.walk(tree) if isinstance(n, ast.FunctionDef)}
|
||||
for banned in ("build_image", "run_container", "compile_target", "apply_patch"):
|
||||
assert banned not in names, f"{banned} belongs to the adopted tool, not here"
|
||||
@@ -1,163 +0,0 @@
|
||||
"""model-grounded-detection task 3.2: the LLM proposes, the model disposes (Req 8).
|
||||
|
||||
The judge is where the prior architecture's whole premise is inverted. There is no K-run averaging, no noise
|
||||
floor and no acceptance rule; there is one bounded, typed question per candidate, and the answer is CHECKED
|
||||
against the CPG rather than voted on. The three outcomes are exhaustive:
|
||||
|
||||
the model entails it -> report it, and never call the model at all
|
||||
the model contradicts it -> drop the claim, and record the contradiction
|
||||
the model cannot decide -> report `suspicion`, honestly
|
||||
|
||||
Every test here runs with no network and a scripted client.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _spec(): # type: ignore[no-untyped-def]
|
||||
from openultrasast.model.specs import TaintSpec
|
||||
|
||||
return TaintSpec(family="injection", language="python", sources=("request.args",), sinks=("os.system",), sanitizers=("escape",))
|
||||
|
||||
|
||||
def _cpg(rows): # type: ignore[no-untyped-def]
|
||||
from openultrasast.cpg.backend import CpgResult
|
||||
|
||||
return CpgResult(cpg_path=Path("cpg.bin"), run=lambda q, p: rows)
|
||||
|
||||
|
||||
ENTAILING_ROW = [
|
||||
{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run", "source": "request.args['c']", "sanitized": False, "length": 3}
|
||||
]
|
||||
|
||||
|
||||
class _Client:
|
||||
"""A scripted chat client that records whether it was called at all."""
|
||||
|
||||
def __init__(self, answer: str = '{"vulnerable": true, "family": "injection"}') -> None:
|
||||
self.calls: list[object] = []
|
||||
self._answer = answer
|
||||
|
||||
def complete(self, **kwargs: object): # type: ignore[no-untyped-def]
|
||||
from openultrasast.tool_hunter import ChatResponse
|
||||
|
||||
self.calls.append(kwargs)
|
||||
return ChatResponse(content=self._answer, tool_calls=[], reasoning=None)
|
||||
|
||||
|
||||
def test_an_entailed_candidate_costs_no_model_call() -> None:
|
||||
"""The point of the arbiter: where the graph decides, the LLM is not consulted at all."""
|
||||
from openultrasast.model.judge import judge
|
||||
from openultrasast.model.ladder import Rung
|
||||
|
||||
client = _Client()
|
||||
answer = judge(_cpg(ENTAILING_ROW), _spec(), function="run", client=client, model="m")
|
||||
assert answer.rung is Rung.ENTAILED
|
||||
assert client.calls == [], "an entailed candidate must not reach the model"
|
||||
|
||||
|
||||
def test_a_claim_the_model_contradicts_is_dropped_with_its_reason() -> None:
|
||||
"""Req 8.2. The LLM says vulnerable; the CPG shows no flow reaches the sink. The graph wins."""
|
||||
from openultrasast.model.judge import judge
|
||||
from openultrasast.model.ladder import Rung
|
||||
|
||||
client = _Client('{"vulnerable": true, "family": "injection"}')
|
||||
answer = judge(_cpg([]), _spec(), function="run", client=client, model="m")
|
||||
assert answer.rung is Rung.SUSPICION
|
||||
assert answer.contradiction, "a dropped claim must say why it was dropped"
|
||||
assert "no flow" in answer.contradiction.lower() or "not reach" in answer.contradiction.lower()
|
||||
|
||||
|
||||
def test_a_claim_the_model_supports_advances_to_corroborated() -> None:
|
||||
"""Req 8.3: a sanitized path is a real flow whose sufficiency only the LLM can judge."""
|
||||
from openultrasast.model.judge import judge
|
||||
from openultrasast.model.ladder import Rung
|
||||
|
||||
rows = [{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run", "source": "request.args['c']", "sanitized": True, "length": 3}]
|
||||
client = _Client('{"vulnerable": true, "family": "injection"}')
|
||||
answer = judge(_cpg(rows), _spec(), function="run", client=client, model="m")
|
||||
assert answer.rung is Rung.CORROBORATED
|
||||
assert len(client.calls) == 1, "exactly one bounded question, never a loop"
|
||||
|
||||
|
||||
def test_the_llm_declining_leaves_a_corroborated_flow_at_suspicion() -> None:
|
||||
"""The model saw a flow but the LLM says the sanitizer suffices; the claim does not advance."""
|
||||
from openultrasast.model.judge import judge
|
||||
from openultrasast.model.ladder import Rung
|
||||
|
||||
rows = [{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run", "source": "request.args['c']", "sanitized": True, "length": 3}]
|
||||
client = _Client('{"vulnerable": false, "family": "injection"}')
|
||||
answer = judge(_cpg(rows), _spec(), function="run", client=client, model="m")
|
||||
assert answer.rung is Rung.SUSPICION
|
||||
|
||||
|
||||
def test_without_a_client_the_verdict_still_stands_on_its_own() -> None:
|
||||
"""Req 11.3: the endpoint is optional. The model layer does not need it to entail."""
|
||||
from openultrasast.model.judge import judge
|
||||
from openultrasast.model.ladder import Rung
|
||||
|
||||
assert judge(_cpg(ENTAILING_ROW), _spec(), function="run", client=None, model="").rung is Rung.ENTAILED
|
||||
sanitized = [
|
||||
{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run", "source": "request.args['c']", "sanitized": True, "length": 3}
|
||||
]
|
||||
answer = judge(_cpg(sanitized), _spec(), function="run", client=None, model="")
|
||||
assert answer.rung is Rung.SUSPICION and "endpoint" in answer.contradiction.lower()
|
||||
|
||||
|
||||
def test_the_question_is_bounded_typed_and_offers_no_tools() -> None:
|
||||
"""Req 8.1: the LLM answers one question about a site it was handed; it never searches for the site."""
|
||||
from openultrasast.model.judge import judge
|
||||
|
||||
rows = [{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run", "source": "request.args['c']", "sanitized": True, "length": 3}]
|
||||
client = _Client()
|
||||
judge(_cpg(rows), _spec(), function="run", client=client, model="m")
|
||||
call = client.calls[0]
|
||||
assert call["tools"] == [], "no tools: the candidate is given, not searched for"
|
||||
assert call["json_object"] is True, "a typed answer, not prose"
|
||||
text = " ".join(str(m.get("content", "")) for m in call["messages"]) # type: ignore[union-attr]
|
||||
assert "os.system(cmd)" in text and "request.args['c']" in text, "the question carries the model's own evidence"
|
||||
|
||||
|
||||
def test_the_prompt_is_redacted() -> None:
|
||||
"""Req 11.4: every prompt passes through redaction."""
|
||||
from openultrasast.model.judge import judge
|
||||
|
||||
rows = [
|
||||
{
|
||||
"sink": "connect(pw)",
|
||||
"sinkLine": "4",
|
||||
"sinkMethod": "run",
|
||||
"source": "AKIA" + "IOSFODNN7" + "EXAMPLE",
|
||||
"sanitized": True,
|
||||
"length": 2,
|
||||
}
|
||||
]
|
||||
client = _Client()
|
||||
judge(_cpg(rows), _spec(), function="run", client=client, model="m")
|
||||
text = " ".join(str(m.get("content", "")) for m in client.calls[0]["messages"]) # type: ignore[index,union-attr]
|
||||
assert "AKIAIOSFODNN7EXAMPLE" not in text
|
||||
|
||||
|
||||
def test_there_is_no_k_run_averaging_anywhere_in_the_judge() -> None:
|
||||
"""Req 8.5, asserted structurally: the noise architecture must not grow back."""
|
||||
import ast
|
||||
|
||||
source = Path("src/openultrasast/model/judge.py").read_text()
|
||||
tree = ast.parse(source)
|
||||
|
||||
# Check CODE, not prose: the docstring is allowed to name the machinery it replaced.
|
||||
names = (
|
||||
{node.id for node in ast.walk(tree) if isinstance(node, ast.Name)}
|
||||
| {node.attr for node in ast.walk(tree) if isinstance(node, ast.Attribute)}
|
||||
| {node.name for node in ast.walk(tree) if isinstance(node, ast.FunctionDef)}
|
||||
)
|
||||
for banned in ("k_runs", "majority", "vote", "votes", "average", "mean", "acceptance", "floor", "budget"):
|
||||
assert banned not in names, f"{banned} has no place in the judge"
|
||||
|
||||
# No loop over runs anywhere in the module.
|
||||
assert not [n for n in ast.walk(tree) if isinstance(n, ast.For | ast.While)], "the judge asks once"
|
||||
|
||||
imported = {n.module for n in ast.walk(tree) if isinstance(n, ast.ImportFrom) and n.module}
|
||||
assert not any("rounds" in m or "acceptance" in m for m in imported)
|
||||
@@ -5,6 +5,8 @@ from __future__ import annotations
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
SRC = Path("src/openultrasast")
|
||||
MANIFEST = Path("benchmarks/measurements/2026-09-08-module-audit.json")
|
||||
|
||||
@@ -34,6 +36,8 @@ def test_the_named_subsystems_each_carry_an_explicit_decision() -> None:
|
||||
assert rows[name].reason, f"{name} has no stated reason"
|
||||
# The removed agentic plane's modules are gone from the tree, not carried as rows (harnessx-removal Req 2.1).
|
||||
assert not {"harness_ext", "hunter_harness", "verify_judge"} & set(rows)
|
||||
# The five the reachability-first audit found orphaned are gone too (legacy cleanup 2026-10-02).
|
||||
assert not {"model.judge", "model.execution", "model.calibrate", "stage_processors", "slot_contract"} & set(rows)
|
||||
|
||||
|
||||
def test_the_manifest_matches_the_tree() -> None:
|
||||
@@ -45,3 +49,45 @@ def test_the_manifest_matches_the_tree() -> None:
|
||||
current = to_dict(audit(SRC))
|
||||
assert committed["counts"] == current["counts"], "the tree drifted from its committed audit"
|
||||
assert committed["orphaned"] == current["orphaned"] == []
|
||||
|
||||
|
||||
def _tree(root: Path, files: dict[str, str]) -> Path:
|
||||
for name, text in files.items():
|
||||
path = root / name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(text)
|
||||
return root
|
||||
|
||||
|
||||
def test_a_spec_listed_module_nothing_imports_is_still_orphaned(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Reachability decides; SPEC_OWNED and STANDALONE only supply the reason (the list hid five dead modules)."""
|
||||
from openultrasast.model import audit as module
|
||||
|
||||
src = _tree(tmp_path, {"cli.py": "from . import used\n", "used.py": "", "ghost.py": "", "lonely.py": ""})
|
||||
monkeypatch.setitem(module.SPEC_OWNED, "ghost", "some spec Req 1: still listed")
|
||||
monkeypatch.setitem(module.STANDALONE, "lonely", "its own entry point, supposedly")
|
||||
rows = {row.module: row for row in module.audit(src)}
|
||||
assert rows["used"].classification == "load_bearing"
|
||||
assert rows["ghost"].classification == "orphaned"
|
||||
assert "still listed" in rows["ghost"].reason
|
||||
assert rows["lonely"].classification == "orphaned"
|
||||
|
||||
|
||||
def test_submodule_imports_and_python_m_modules_count_as_reachable(tmp_path: Path) -> None:
|
||||
from openultrasast.model.audit import audit
|
||||
|
||||
src = _tree(
|
||||
tmp_path,
|
||||
{
|
||||
"cli.py": "def run():\n from .pkg import sub\n",
|
||||
"pkg/__init__.py": "",
|
||||
"pkg/sub.py": "",
|
||||
"tool.py": 'from . import helper\n\nif __name__ == "__main__":\n pass\n',
|
||||
"helper.py": "",
|
||||
"dead.py": "from . import deader\n",
|
||||
"deader.py": "",
|
||||
},
|
||||
)
|
||||
rows = {row.module: row.classification for row in audit(src)}
|
||||
assert rows["pkg.sub"] == rows["tool"] == rows["helper"] == "load_bearing"
|
||||
assert rows["dead"] == rows["deader"] == "orphaned"
|
||||
|
||||
@@ -9,7 +9,6 @@ def test_project_opencode_skills_have_required_frontmatter() -> None:
|
||||
|
||||
assert {path.parent.name for path in skill_files} == {
|
||||
"openultrasast-fix-audit",
|
||||
"openultrasast-kiro-impl",
|
||||
"openultrasast-scan",
|
||||
"openultrasast-triage",
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"""`agree` scores exactly as the reference measurement did (ai-service-plane Requirement 6).
|
||||
|
||||
The reference is `benchmarks/independent/score_batched.py` over the batched-check scans: sites anchored by
|
||||
The reference is `benchmarks/archive/score_batched.py` over the batched-check scans: sites anchored by
|
||||
`score_batched.anchored`, matched by `evaluate.matches_v2` against `evaluate.hunks(case, "old")`. The plane cannot
|
||||
ship the benchmark tree, so the generator resolves the ranges into `case.json` and `agree.matches_declared`
|
||||
reproduces the rule. These tests run both on the recorded reference outputs of the 15 validation cases and
|
||||
@@ -23,6 +23,7 @@ from openultrasast.plane.manifests import load_manifests
|
||||
from openultrasast.plane.tasks import agree
|
||||
|
||||
BENCH = Path("benchmarks/independent")
|
||||
ARCHIVE = Path("benchmarks/archive") # score_batched.py, archived 2026-10-02
|
||||
POPULATION = BENCH / "population-v2.toml"
|
||||
SCANS = Path.home() / "ousast-results" / "independent-v2-batched-check" / "scans"
|
||||
REPOS = Path.home() / ".cache" / "openultrasast" / "independent"
|
||||
@@ -33,10 +34,10 @@ pytestmark = pytest.mark.skipif(
|
||||
)
|
||||
|
||||
|
||||
def _load(name: str) -> ModuleType:
|
||||
def _load(name: str, where: Path = BENCH) -> ModuleType:
|
||||
sys.path.insert(0, str(BENCH.resolve())) # score_batched imports evaluate by name
|
||||
try:
|
||||
spec = importlib.util.spec_from_file_location(f"parity_{name}", BENCH / f"{name}.py")
|
||||
spec = importlib.util.spec_from_file_location(f"parity_{name}", where / f"{name}.py")
|
||||
assert spec is not None and spec.loader is not None
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
@@ -47,7 +48,7 @@ def _load(name: str) -> ModuleType:
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def reference() -> tuple[ModuleType, ModuleType]:
|
||||
evaluate, score_batched = _load("evaluate"), _load("score_batched")
|
||||
evaluate, score_batched = _load("evaluate"), _load("score_batched", ARCHIVE)
|
||||
evaluate.use(POPULATION.resolve())
|
||||
return evaluate, score_batched
|
||||
|
||||
|
||||
@@ -1,108 +0,0 @@
|
||||
"""Model-disabled, slot-contracted deterministic stages (Phase 3 task 6.3).
|
||||
|
||||
Proves: deterministic stages carry work in declared slots, a slot-contract violation
|
||||
is raised on undeclared slot access, the pipeline is model-disabled, the slot pipeline
|
||||
reproduces the deterministic quick-scan path byte-for-byte, and lifecycle hooks fire.
|
||||
"""
|
||||
|
||||
from collections.abc import Mapping
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from openultrasast.gate import _quick_scan
|
||||
from openultrasast.slot_contract import SlotContractError, SlotContractMixin, SlotPipeline
|
||||
from openultrasast.stage_processors import (
|
||||
build_deterministic_pipeline,
|
||||
deterministic_processors,
|
||||
run_quick_pipeline,
|
||||
)
|
||||
|
||||
|
||||
class _GoodStage(SlotContractMixin):
|
||||
name = "good"
|
||||
reads_slots = ("a",)
|
||||
writes_slots = ("b",)
|
||||
|
||||
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||
return {"b": slots["a"] + 1}
|
||||
|
||||
|
||||
class _UndeclaredWriteStage(SlotContractMixin):
|
||||
name = "bad_write"
|
||||
reads_slots = ()
|
||||
writes_slots = ("declared",)
|
||||
|
||||
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||
return {"undeclared": 1}
|
||||
|
||||
|
||||
class _MissingReadStage(SlotContractMixin):
|
||||
name = "bad_read"
|
||||
reads_slots = ("required",)
|
||||
writes_slots = ()
|
||||
|
||||
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||
return {}
|
||||
|
||||
|
||||
# ---- slot contract enforcement (1.5) ----------------------------------------
|
||||
|
||||
|
||||
def test_declared_slots_pass() -> None:
|
||||
assert _GoodStage().execute({"a": 1}) == {"b": 2}
|
||||
|
||||
|
||||
def test_undeclared_write_is_rejected() -> None:
|
||||
with pytest.raises(SlotContractError, match="undeclared write slots: undeclared"):
|
||||
_UndeclaredWriteStage().execute({})
|
||||
|
||||
|
||||
def test_missing_read_is_rejected() -> None:
|
||||
with pytest.raises(SlotContractError, match="missing read slots: required"):
|
||||
_MissingReadStage().execute({})
|
||||
|
||||
|
||||
# ---- model-disabled invariant (1.6) -----------------------------------------
|
||||
|
||||
|
||||
def test_all_deterministic_stages_are_model_disabled() -> None:
|
||||
assert all(proc.skip_model for proc in deterministic_processors())
|
||||
|
||||
|
||||
def test_pipeline_rejects_a_model_enabled_stage() -> None:
|
||||
enabled = _GoodStage()
|
||||
enabled.skip_model = False
|
||||
with pytest.raises(SlotContractError, match="model-disabled"):
|
||||
SlotPipeline([enabled])
|
||||
|
||||
|
||||
# ---- parity with the deterministic quick-scan path --------------------------
|
||||
|
||||
|
||||
def test_pipeline_findings_match_quick_scan_baseline() -> None:
|
||||
target = Path("benchmarks/fixtures/python-vulnerable")
|
||||
slots = run_quick_pipeline(target)
|
||||
assert slots["reported_findings"] == _quick_scan(target)
|
||||
assert "score" in slots # the score slot is produced too
|
||||
|
||||
|
||||
# ---- lifecycle hooks fire per stage -----------------------------------------
|
||||
|
||||
|
||||
class _Recorder:
|
||||
def __init__(self) -> None:
|
||||
self.events: list[tuple[str, str | None]] = []
|
||||
|
||||
def emit(self, event_type: Any, stage: str | None, payload: dict[str, Any]) -> None:
|
||||
self.events.append((event_type, stage))
|
||||
|
||||
|
||||
def test_pipeline_emits_lifecycle_hooks_per_stage() -> None:
|
||||
recorder = _Recorder()
|
||||
run_quick_pipeline(Path("benchmarks/fixtures/python-vulnerable"), runtime=recorder)
|
||||
names = [proc.name for proc in build_deterministic_pipeline().processors]
|
||||
for name in names:
|
||||
assert ("stage_start", name) in recorder.events
|
||||
assert ("stage_end", name) in recorder.events
|
||||
Reference in New Issue
Block a user