mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
chore(p0): retire regress, the skill router, the module audit tool and vocabulary.sc
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts). Deleted: src/openultrasast/regress/ and the deep-mode REGRESS branch (deep now records a regress_retired degradation and equals standard; proof is the search-with-proof arm's job), the sandbox docker runner and probe that only regress used, --fail-on worth-fixing, RegressConfig, the two ledger helpers only regress called; skills.py and index.py with the hunter pool's skill routing; model/audit.py; cpg/queries/vocabulary.sc with benchmarks/push/ontology_gap.py. Docs and the scan skill describe the retirement. Net: 3,895 lines deleted, 86 added. Targeted tests: 180 passed, 28 skipped; 3,331 tests collect. Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Fable 5.1
Claude Fable 5.1
parent
94e96c6885
commit
f5a904b7f2
+3
-13
@@ -16,7 +16,7 @@ The CLI is thin. `HarnessRuntime` owns the lifecycle. It writes an event trace f
|
||||
|
||||
- `quick` runs STATIC;
|
||||
- `standard` adds MAP;
|
||||
- `deep` adds REGRESS.
|
||||
- `deep` requests the REGRESS stage, which was retired on 2026-10-13: the scan records a `regress_retired` degradation and otherwise equals `standard`. Execution proof now comes from the search-with-proof arm.
|
||||
|
||||
```
|
||||
policy_load ─▶ ruleset_load ─▶ policy_check (fail loud on an ungoverned CWE)
|
||||
@@ -31,7 +31,7 @@ policy_load ─▶ ruleset_load ─▶ policy_check (fail loud on an
|
||||
│ obligations absence bugs: an obligated operation with no governing guard
|
||||
│ model one Joern CPG per repository; taint, dominance and configuration arbiters
|
||||
│ tool hunter only with `[models] hunter`
|
||||
─▶ REGRESS (deep) sandboxed regression snippets ─▶ verdicts.json, worth-fixing
|
||||
─▶ REGRESS (deep) retired 2026-10-13, recorded as `regress_retired`
|
||||
─▶ report.md, report.sarif, manifest.json
|
||||
```
|
||||
|
||||
@@ -67,17 +67,7 @@ nothing, the LLM sees the enumerator's candidates (at most `MAX_JUDGED_CANDIDATE
|
||||
Anything it affirms is reported at `suspicion`. Without a provider key, neither question is
|
||||
asked. Only the graph's entailed findings are reported then.
|
||||
|
||||
**REGRESS** (`regress/`, `sandbox/`) runs only when `docker info` succeeds. Each promoted
|
||||
candidate gets a small language recipe (Python, JavaScript, C). The recipe loads the
|
||||
candidate's source inside a container with:
|
||||
|
||||
- no network;
|
||||
- a read-only `/workspace`;
|
||||
- a non-root user;
|
||||
- the `[sandbox]` limits.
|
||||
|
||||
Before anything runs, a structural deny-list rejects snippets that mention sockets, curl, host
|
||||
networking or writes under `/workspace`.
|
||||
**REGRESS** was retired on 2026-10-13. Deep mode records a `regress_retired` degradation and otherwise equals standard. Execution proof is the search-with-proof arm's job (`search/verify.py`, bubblewrap sandbox, differential runs against owned canaries).
|
||||
|
||||
## Evidence ladder: how a false positive is eliminated
|
||||
|
||||
|
||||
@@ -542,10 +542,6 @@ reports the worker JVM's heap (`census.sc:17-31`). It is the instrument check of
|
||||
settles whether a frontend warning mattered: if the graph holds every expected file, the
|
||||
warning is ignored (`backend.py:827-829`).
|
||||
|
||||
`vocabulary.sc` is a related query that needs no dataflow. It lists every call not covered by
|
||||
the fact tables as a candidate list, "never an ontology" (`vocabulary.sc:72-74`). It feeds
|
||||
`benchmarks/push/ontology_gap.py`, not the scan.
|
||||
|
||||
## 5. Regions, ranking and budgets
|
||||
|
||||
A repository is too large to ask the graph about everything. So the scan asks about regions in a
|
||||
@@ -706,69 +702,14 @@ explains how those signals become a decision. A tool-equipped hunter was measure
|
||||
overlay on a Python holdout: 12/14 against 5/14 once scoring was class-aware. That gap is in
|
||||
`benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-*.json`.
|
||||
|
||||
## 7. REGRESS: sandboxed reproduction
|
||||
## 7. REGRESS: retired
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
hot["Promoted hotspots, top max_candidates = 5"] --> snip["Snippet: proposed by the finding, else a load-and-compile stub"]
|
||||
snip --> deny{"Structural deny-list"}
|
||||
deny -- "socket, curl, docker.sock, host network, writes under /workspace" --> rej["inconclusive: safety_rejected"]
|
||||
deny -- "clean" --> run["docker run: no network, read-only, tmpfs scratch, cap-drop ALL, uid 65534"]
|
||||
run --> exit["Exit code and output"]
|
||||
exit --> v["triggerable, not_triggerable, already_covered, inconclusive"]
|
||||
v --> worth["worth_fixing if triggerable and reachable"]
|
||||
```
|
||||
|
||||
REGRESS runs only in `deep` mode (`stages.py:17`) and only when `docker info` succeeds.
|
||||
Otherwise the stage records `sandbox_unavailable` (`sandbox/probe.py:24-36`, `cli.py:691-693`).
|
||||
|
||||
It selects the top `max_candidates = 5` hotspots by score, plus forced severity-5 reachable
|
||||
inventory (`regress/candidate.py:63-88`, `config.py:157-160`). A snippet is the one a finding
|
||||
proposed. If there is none, a default only loads the file (`candidate.py:153-174`):
|
||||
|
||||
- Python compiles it;
|
||||
- JavaScript reads it;
|
||||
- C is `int main(void)
|
||||
{ return 0; }` (`candidate.py:153-174`).
|
||||
|
||||
Recipes exist for Python, JavaScript and C on `python:3.12-alpine`, `node:22-alpine` and
|
||||
`gcc:13` (`regress/recipes.py:16-52`, `candidate.py:30-34`). Any other language is
|
||||
`inconclusive` with `missing_recipe`.
|
||||
|
||||
The deny-list is a regex check before any container starts (`regress/safety.py:63-75`). It
|
||||
rejects (`safety.py:21-60`):
|
||||
|
||||
- the Docker socket;
|
||||
- the word `socket`;
|
||||
- the word `curl`;
|
||||
- a host-network request;
|
||||
- writes under `/workspace` in Python, Node, C and shell forms.
|
||||
|
||||
The real boundary is the container (`sandbox/runner.py:113-137`): `--network none --read-only --tmpfs /scratch:rw,exec --memory
|
||||
<mb>m --pids-limit <n> --cap-drop ALL --security-opt no-new-privileges --user 65534:65534`. The
|
||||
repository is bind-mounted read-only at `/workspace`. The runner also refuses an argv containing
|
||||
`--privileged`, the Docker socket or host networking (`runner.py:182-192`). Defaults are 2048 MB,
|
||||
300 s and 512 pids (`config.py:104-109`).
|
||||
|
||||
Verdicts (`regress/verdict.py:10-26, 64-110`):
|
||||
|
||||
- a safety rejection, missing recipe or sandbox, or timeout is `inconclusive`;
|
||||
- exit 0 under a covering test is `already_covered`;
|
||||
- with an oracle: `triggerable` on `oracle_fired`, else `not_triggerable`;
|
||||
- without an oracle: a nonzero exit is `triggerable` with reason `sanitizer_abort`, `assertion`,
|
||||
`crash` or `nonzero_exit`, and exit 0 is `not_triggerable`.
|
||||
|
||||
`is_worth_fixing` requires `triggerable` and reachability `reachable` or
|
||||
`inferred-file-surface` (`verdict.py:28, 59-61`). Verdicts feed the complexity ledger and
|
||||
`rank_obligations`. `rank_obligations` caps every obligation finding strictly below the lowest
|
||||
proven finding (`cli.py:724-727`, `rank.py:243-265`).
|
||||
|
||||
**What `execution_confirmed` means today.** The rung is defined as "a sandbox reproduced it
|
||||
(deferred tier, memory/C first)" (`ladder.py:10, 33`). It is ordered above `model_entailed`. No
|
||||
code path assigns it. REGRESS produces the four verdicts above, not a rung. Nothing maps
|
||||
`triggerable` onto `Rung.EXECUTION_CONFIRMED`. The default snippets prove that a file loads, not
|
||||
that a vulnerability fires. A reproduction needs a finding-proposed snippet or an oracle. Read
|
||||
the rung as reserved.
|
||||
The sandboxed snippet stage was retired on 2026-10-13 (degradation `regress_retired`). It proved that a file
|
||||
loads, not that a vulnerability fires, and never produced a rung. Execution proof is now the search-with-proof
|
||||
arm: a finding-proposed demo run under bubblewrap, three runs on the affected revision against three on the
|
||||
safe one, owned canary oracles, `demonstrated` only on 3 of 3 against 0 of 3 (`search/verify.py`).
|
||||
`Rung.EXECUTION_CONFIRMED` is still assigned by no code path; wiring the verify outcome onto it is the next
|
||||
step in that spec.
|
||||
|
||||
## 8. What each technique misses
|
||||
|
||||
@@ -781,7 +722,6 @@ the rung as reserved.
|
||||
| Configuration | A value computed at runtime (reported `CORROBORATED`, never safe) | `config_value.py:75-86` |
|
||||
| Ranking | A pinned CVE deep in a tier: PMPro's at position 390 of 4,463 under access rank, with evidence ordering still off by default | `benchmarks/ranking/baselines/2026-09-10-access-rank.json`; `scan.py:98` |
|
||||
| The model | Anything the enumerator did not propose (a hard ceiling on recall); the residual question is bounded to 8 candidates per region; a public fix the model may have memorised is indistinguishable from reasoning, which is why labels come from ground truth and the engine block is withheld where it tracks the fix | `candidates.py:7-10`; `pipeline.py:44`; `learn/schema.py:97-109` |
|
||||
| REGRESS | Anything without a Python, JavaScript or C recipe; anything the default load-only snippet cannot trigger; no finding reaches `execution_confirmed` today | `recipes.py:16-52`; `candidate.py:153-174`; `ladder.py:33` |
|
||||
|
||||
The three representations are kept separate on purpose. A finding carries the rung and the
|
||||
degradations of the representation that produced it. So a reader can tell a flow the graph
|
||||
|
||||
+3
-6
@@ -15,7 +15,7 @@ The stages each mode requests are listed in `MODE_STAGES` in `src/openultrasast/
|
||||
|
||||
- `quick` runs STATIC.
|
||||
- `standard` adds MAP.
|
||||
- `deep` adds REGRESS.
|
||||
- `deep` requests the REGRESS stage, which was retired on 2026-10-13: the scan records a `regress_retired` degradation and otherwise equals `standard`. Execution proof now comes from the search-with-proof arm.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
@@ -29,17 +29,14 @@ flowchart TD
|
||||
verify --> mapq{"MAP requested?"}
|
||||
mapq -- "quick: no" --> out["report.md, report.sarif, manifest.json"]
|
||||
mapq -- "standard, deep" --> map["MAP: complexity map, semantic overlay, obligations, Joern model layer"]
|
||||
map --> regq{"REGRESS requested?"}
|
||||
regq -- "standard: no" --> out
|
||||
regq -- "deep" --> reg["REGRESS: snippets in a Docker sandbox, verdicts.json"]
|
||||
reg --> out
|
||||
map --> out
|
||||
```
|
||||
|
||||
| Mode | Adds | Needs | Without it |
|
||||
| --- | --- | --- | --- |
|
||||
| `quick` | pattern rules, entry-point hints, ranking, verification, score | nothing | - |
|
||||
| `standard` | MAP: complexity map, semantic overlay, authorization obligations, the Joern model layer; `fusion` (deterministic) | Joern; a provider key for the residual LLM question | `cpg_unavailable` degradation; the LLM's questions are skipped and only the graph's entailed findings are reported |
|
||||
| `deep` | REGRESS: promoted candidates loaded in a sandbox with no network, read-only source, non-root | a working `docker` | `sandbox_unavailable` degradation |
|
||||
| `deep` | nothing today: REGRESS was retired on 2026-10-13 and is recorded as `regress_retired`; execution proof belongs to the search-with-proof arm | - | - |
|
||||
|
||||
Sources: `README.md` (scan modes), `src/openultrasast/cli.py` (`run_stage` calls),
|
||||
[Architecture](architecture.md#the-scan-pipeline).
|
||||
|
||||
+7
-19
@@ -11,9 +11,9 @@ It also lists the controls that keep a scan safe to run in CI and safe to share.
|
||||
- Standard mode also parses it into a code property graph (a graph of the program's syntax,
|
||||
control flow and data flow) with Joern. It uses `joern-parse`, a parser, not a build.
|
||||
Standard mode may send code excerpts to the configured model.
|
||||
- Only `deep` mode executes target-derived code. Its REGRESS stage loads promoted candidates
|
||||
inside the Docker sandbox described below. It runs only when `docker
|
||||
info` succeeds (otherwise the stage is skipped and recorded as `sandbox_unavailable`).
|
||||
- No scan mode executes target-derived code any more: the deep-mode REGRESS stage was retired on 2026-10-13
|
||||
and is recorded as `regress_retired`. Execution happens only in the search-with-proof arm's sandbox,
|
||||
described in `.kiro/specs/search-with-proof/`.
|
||||
- **The ruleset and CWE policy are trusted, governed data.** They change
|
||||
through reviewed commits.
|
||||
- **Provider credentials are the operator's.** The core install (PyYAML only) makes no network
|
||||
@@ -78,7 +78,7 @@ malformed JSON.
|
||||
### Visible degradation & determinism
|
||||
A scan is never silently downgraded. When an optional engine is missing, the scan falls back to
|
||||
its deterministic equivalent. It records a `degradations` entry in the manifest (Joern:
|
||||
`cpg_unavailable`; Docker: `sandbox_unavailable`).
|
||||
`cpg_unavailable`; retired snippet stage: `regress_retired`).
|
||||
|
||||
Without a provider key, the model layer asks no question. It reports only what the graph
|
||||
entailed.
|
||||
@@ -92,18 +92,6 @@ internal hunter tools. No tool accepts a free-form command argument.
|
||||
|
||||
## Sandbox (deep mode)
|
||||
|
||||
The REGRESS stage (`regress/`, `sandbox/runner.py`) runs each snippet with `docker run`. It never
|
||||
mounts the host Docker socket.
|
||||
|
||||
| Control | Setting |
|
||||
|---|---|
|
||||
| network | `--network none`, always; a host-network argument is refused |
|
||||
| source | the target bind-mounted read-only at `/workspace`; root filesystem `--read-only` |
|
||||
| scratch | a tmpfs at `/scratch` for the snippet |
|
||||
| user | non-root `65534:65534`, `--cap-drop ALL`, `no-new-privileges` |
|
||||
| memory | `[sandbox] memory_mb`, default 2048 MB |
|
||||
| pids | `[sandbox] pids_limit`, default 512 |
|
||||
| timeout | `[sandbox] timeout_seconds`, default 300 s |
|
||||
|
||||
A structural deny-list (`regress/safety.py`) checks each snippet before anything runs. It rejects
|
||||
a snippet that mentions sockets, curl, host networking or writes under `/workspace`.
|
||||
Retired on 2026-10-13 with the REGRESS stage. The search-with-proof arm runs demos under bubblewrap with
|
||||
seccomp, no network unless the oracle needs a namespace, and differential runs against owned canaries;
|
||||
see `src/openultrasast/search/_sandbox.py` and `.kiro/specs/search-with-proof/`.
|
||||
|
||||
Reference in New Issue
Block a user