mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-19 04:59:28 +00:00
docs: plain-English pass over the active specs
Ran the humanizer rules over 79 prose files in the active specs (brief, design, requirements, research and narrative docs). Removed em dashes, significance inflation, forced rule-of-three, bold-label list items and filler. Skipped tasks.md checklists and the retired specs (harnessx-*, openrouter-sast-harness, unseen-repo-evaluation). Every number, path, command, hash and identifier was preserved: checked each file against HEAD for drift in numbers, inline code and code fences, zero drift across all 79 files. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
b08f372022
commit
39a1873e5b
@@ -9,13 +9,13 @@ numbers"), gated on the pool metric; it is not part of the G0 record-07 baseline
|
||||
The engine produces **no finding** on real unseen vulnerability changes. Live 300 s long-deadline replays show
|
||||
that the cause is usually not a detection-capability gap:
|
||||
|
||||
- **Clean Python injection change** (`aws/bedrock-agentcore-sdk-python`, `install_packages`, introducing, 3-file
|
||||
- Clean Python injection change (`aws/bedrock-agentcore-sdk-python`, `install_packages`, introducing, 3-file
|
||||
diff): `wall=303 s`, hit the deadline. The ranker found candidates (`static_rank=1.0`) but **`selected: []`** and
|
||||
every candidate `deferred` with `reason: deadline_exhausted`, `analysis_not_started=deadline_exhausted`,
|
||||
`change_context_incomplete`. No analyses ran. The ranked candidates were **repo-wide and unrelated to
|
||||
the push** (`auth.py` access_control/config_secrets), not the changed `install_packages`. The engine is trying to
|
||||
re-audit the whole repo and runs out of budget before analyzing the change.
|
||||
- **Noisy last_touch config_secrets change** (`jenkinsci/metrics-plugin`, 14-file diff): finished in 206 s but
|
||||
- Noisy last_touch config_secrets change (`jenkinsci/metrics-plugin`, 14-file diff): finished in 206 s but
|
||||
`comparison_unknown` + `capability_unavailable` + `eligibility_provenance_mismatch` +
|
||||
`contributor-scan ... projection_unavailable`. This has several causes. Correspondence was one (fixed separately,
|
||||
commit c39fbb8); the capability registry is Task 8.3.
|
||||
@@ -157,12 +157,12 @@ To be releasable, the net needs tier 2 and/or tier 3 to run and catch within a r
|
||||
The target is **≤60–120 s** (developer patience), hard ceiling 300 s. The analysis records each tier's budget against
|
||||
that window and identifies the measurements needed:
|
||||
|
||||
- **Engine barrier (tier 2).** Fixed floor `build(48) + evidence_fixed(~150) ≈ 198 s` precedes any dataflow answer.
|
||||
- Engine barrier (tier 2). Fixed floor `build(48) + evidence_fixed(~150) ≈ 198 s` precedes any dataflow answer.
|
||||
Measure (clean host): **E1** split the ~150 s into JVM-start / graph-load / census; **E2** does a warm session
|
||||
with the graph **preloaded at build** eliminate `evidence_fixed` on the first query (the architectural
|
||||
test)?; **E3** analysis (`reachableByFlows`) per-pair cost + warm reuse; **E4** build cost vs repo size.
|
||||
Levers in priority order: warm session + graph preload → cached census → change-scope the per-pair tail.
|
||||
- **LLM barrier (tier 3).** Unmeasured in the push path. Measure: **L1** model round-trip latency per call
|
||||
- LLM barrier (tier 3). Unmeasured in the push path. Measure: **L1** model round-trip latency per call
|
||||
(DeepSeek) and calls needed per push for the decision/proof stage; **L2** the recall lift of tier 3 over tiers
|
||||
1+2 on the pool, does the LLM catch what regex and dataflow miss, and at what $ and latency per push?
|
||||
- Produce a **barrier table** for budget ∈ {60, 120, 300, 600} s × each tier combination. Record whether it fits
|
||||
@@ -258,7 +258,7 @@ Reprioritised engine levers (corrected, anchoring is not the lever):
|
||||
([[search-with-proof]]) is that the LLM tier catches what regex+dataflow miss. The pilot supports
|
||||
testing it: **L1** model latency + calls/push, **L2** recall lift on the same 5 (and beyond). These calls cost money, so stage
|
||||
them (5–10 sizing → ~10–20 pilot → full only on lift).
|
||||
- **#2 taint source/sink coverage + region ranking** so the dataflow recognises these vuln sources and promotes the
|
||||
- #2 taint source/sink coverage + region ranking so the dataflow recognises these vuln sources and promotes the
|
||||
vuln region (banks path: sink found, source unrecognised; graphiti injection: score 0).
|
||||
- #3 change-anchoring robustness matters only after the static/LLM tiers produce vuln-family
|
||||
findings to anchor. Deprioritised until (1)/(2) yield findings.
|
||||
@@ -329,10 +329,10 @@ Two blockers to realising it in the live pipeline (both now concrete):
|
||||
mislocation, not a detection failure.**
|
||||
|
||||
The budget is approved. Keep the revised next steps staged:
|
||||
- **Validate the 5 pilot labels first** (cheap, LLM-assisted corpus audit): for each, is the real sink inside the
|
||||
- Validate the 5 pilot labels first (cheap, LLM-assisted corpus audit): for each, is the real sink inside the
|
||||
changed lines, and is it change-introduced vs pre-existing-in-base? Only sound, change-located vulns are fair
|
||||
targets. This separates true detection gaps from SZZ artifacts before spending on detection runs.
|
||||
- **Then run the agentic hunter** (search-with-proof) on the sound targets and size its per-vuln call count/cost;
|
||||
- Then run the agentic hunter (search-with-proof) on the sound targets and size its per-vuln call count/cost;
|
||||
measure catch + false-alarm (on ordinary changes) vs regex 1/5. The single-shot path is insufficient; the hunter
|
||||
is the real "#1".
|
||||
- Feed confirmed source shapes (e.g. library param → path join) back as guarded static facts (#2).
|
||||
@@ -362,7 +362,7 @@ which becomes a fact after repeated confirmation (the #2 feedback loop). No new
|
||||
user. This is now part of the "#2 guided by #1" workstream and must not be deferred. (DeepSeek
|
||||
`deepseek-flash` is the working default; GLM 5.3 "cyber" model is future-spec.)
|
||||
|
||||
## POOL-P1 LABEL AUDIT (LLM, 2026-10-12) — 47% of labels are sound
|
||||
## POOL-P1 LABEL AUDIT (LLM, 2026-10-12): 47% of labels are sound
|
||||
|
||||
The banks label-location check was extended to all **49 public** pool vulns via DeepSeek (advisory + head code +
|
||||
pool label → sound/unsound). Full artifact: `benchmarks/measurements/2026-10-12-pool-p1-label-audit/`.
|
||||
@@ -378,7 +378,7 @@ pool label → sound/unsound). Full artifact: `benchmarks/measurements/2026-10-1
|
||||
- Next: run the agentic search-with-proof hunter on the sound targets (start bedrock + graphiti), size its
|
||||
per-vuln call count/cost, measure catch + false-alarm. Then recompute record 07 on the sound subset.
|
||||
|
||||
## MODEL HUNTER on the 2 sound targets (2026-10-12) — first LLM catch the static tiers missed
|
||||
## MODEL HUNTER on the 2 sound targets (2026-10-12): first LLM catch the static tiers missed
|
||||
|
||||
Ran `ousast scan --mode deep` (DeepSeek `deepseek-flash`, model layer + tool_hunter, 60-call/8-hotspot bound) on
|
||||
bedrock + graphiti, the two sound pilot targets the static engine missed. Artifact:
|
||||
@@ -417,7 +417,7 @@ Next: deep scan (regex/engine/hunter) on the 5 fair targets → per-tier catch r
|
||||
stand up an app-shaped eval set from known-vulnerable apps (VAmPI/NodeGoat/DjanGoat/vibe-py) and recompute the
|
||||
floor. Revisit the deferred bets (advisory-with-conditions, reachability prover) once the fair-target results are available.
|
||||
|
||||
## FAIR-TARGET RESULT: detection 3/5 on app-shaped+sound (vs ~0 library-heavy) — target-fix validated (2026-10-12)
|
||||
## FAIR-TARGET RESULT: detection 3/5 on app-shaped+sound (vs ~0 library-heavy), target-fix validated (2026-10-12)
|
||||
|
||||
Deep scan on the 5 app-shaped+sound targets (`benchmarks/measurements/2026-10-12-hunter-fair-targets/`):
|
||||
- 3/5 detected: graphiti (hunter, Cypher injection), mcp-remote (hunter, url-scheme injection), dcmcp (ENGINE
|
||||
@@ -433,7 +433,7 @@ Detection appears on fair targets; release-grade BLOCK still needs (a) proof-arm
|
||||
(b) family coverage (template XSS/SSTI) + the DSPy hint loop for misses. Next: recompute a floor on an app-shaped
|
||||
sound subset / real app corpus (VAmPI, NodeGoat, DjanGoat, vibe-py), and measure the proof arm's suspicion→proven lift.
|
||||
|
||||
## PROOF-ARM MEASUREMENT on the 3 fair-target hits (2026-10-12) — push-time lift 0/3, structural
|
||||
## PROOF-ARM MEASUREMENT on the 3 fair-target hits (2026-10-12): push-time lift 0/3, structural
|
||||
|
||||
Measured whether the proof arm can elevate the 3 fair-target suspicions (graphiti, mcp-remote, dcmcp) to
|
||||
`exploit_demonstrated`.
|
||||
@@ -448,7 +448,7 @@ Measured whether the proof arm can elevate the 3 fair-target suspicions (graphit
|
||||
(`isolation_mode: userns`); the plane supplies a task-boundary dispatcher.
|
||||
- The scan/push path reaches only `suspicion`/`static_corroboration` (verification.py levels). There is no code
|
||||
path from a scan suspicion to `exploit_demonstrated`, and no caller wires the search Coordinator from the push
|
||||
flow. So proof-arm lift at push time = 0/3: not because the vulns are unreal, but because proving is a
|
||||
flow. The vulns are real, but proof-arm lift at push time = 0/3 because proving is a
|
||||
runtime differential operation with an offline build for each target that cannot run inside a pre-push hook. (Prior
|
||||
per-pair demo attempts were mostly inconclusive on this exact wiring, app startup, offline deps, oracle plumbing.)
|
||||
|
||||
@@ -462,7 +462,7 @@ suspicion:
|
||||
proving, with authored (eventually DSPy-authored) demos. Run a bounded, separate test on one tractable target
|
||||
(mcp-remote) to confirm that the arm demonstrates a real unseen vuln.
|
||||
|
||||
## BANKS FIXED-SEARCH MVP (2026-10-12) — "no wrong targets, only wrong searching" demonstrated
|
||||
## BANKS FIXED-SEARCH MVP (2026-10-12): "no wrong targets, only wrong searching" demonstrated
|
||||
|
||||
Record: `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`. Target: banks (the "library" that gave
|
||||
no engine path findings in the pilot). The target and CPG are unchanged; only the search changes.
|
||||
|
||||
+10
-10
@@ -12,7 +12,7 @@ Fixed via **option 1** (filter params to the declared signature) in `src/openult
|
||||
- `JoernBackend.query()` now drops any param the script does not declare before building the `--param` list
|
||||
(disposable route) and the in-session named-arg call (session route). This is faithful, not lossy: the
|
||||
single-request branch of `taint.sc` (line 1542) calls `rowsFor(...)` with top-level params only and lets
|
||||
`fixedOrigin`/`originAnchors`/`quotedSanitizers`/`guards`/`fieldParameterSources` default — it never could
|
||||
`fixedOrigin`/`originAnchors`/`quotedSanitizers`/`guards`/`fieldParameterSources` default; it never could
|
||||
consume them. The batch path keeps carrying them as JSON the script parses itself.
|
||||
- No-op for `config`/`dominance`: their own `request_params` emit only params their `.sc` declares (verified),
|
||||
so nothing is dropped. Only `taint` sheds the 5 extras.
|
||||
@@ -21,12 +21,12 @@ Fixed via **option 1** (filter params to the declared signature) in `src/openult
|
||||
queries now RUN (no `cpg query taint failed`); baseline = 0 findings, fixed search (annotation) = 2 scored
|
||||
`path`/`model_entailed` findings (`from_prompt_path` → `prompt_file.write_text(prompt.raw)` L45; `_scan` →
|
||||
`self._index_path.write_text(...)` L155). Regression test:
|
||||
`tests/test_cpg_backend.py::test_a_single_query_passes_only_params_the_script_declares` (fake runner, no Joern
|
||||
— asserts the 5 undeclared params never reach the command line and the query parses).
|
||||
`tests/test_cpg_backend.py::test_a_single_query_passes_only_params_the_script_declares` (fake runner, no Joern;
|
||||
asserts the 5 undeclared params never reach the command line and the query parses).
|
||||
|
||||
## Summary
|
||||
|
||||
Any call on the **single-query** taint path — `cpg.run("taint", request_params(...))` — fails on **both**
|
||||
Any call on the **single-query** taint path (`cpg.run("taint", request_params(...))`) fails on **both**
|
||||
of its sub-routes, returns `None`, and is read upstream as "no flows / clean". `request_params()` emits five
|
||||
keys that `taint.sc`'s `@main def exec(...)` does not declare, and the single-query dispatch passes every
|
||||
param straight to the script. The **batch** path (`run_batch` → `requestsFile` JSON) is unaffected, which is
|
||||
@@ -53,10 +53,10 @@ contextFilter, contextPaths, requests, requestsFile, trace`.
|
||||
`backend.query()` (`src/openultrasast/cpg/backend.py:1266`) dispatches a single query two ways, **both** of
|
||||
which choke on the undeclared keys:
|
||||
|
||||
- **Disposable / CLI route** (lines 1276–1278): appends `--param {key}={value}` for every param. joern errors
|
||||
- Disposable / CLI route (lines 1276–1278): appends `--param {key}={value}` for every param. joern errors
|
||||
`Unknown arguments: --fixedOrigin --guards --quotedSanitizers --originAnchors --fieldParameterSources`
|
||||
(and, as a knock-on, `Missing argument: --cpgFile`). Observed verbatim via `banks_debug.py` on 2026-10-12.
|
||||
- **Session route** (lines 1321–1326): defines `def ousast_taint(...)` by renaming `@main def exec(` and then
|
||||
- Session route (lines 1321–1326): defines `def ousast_taint(...)` by renaming `@main def exec(` and then
|
||||
calls `ousast_taint(key = value, ...)` with named args. The same five keys are not parameters of that
|
||||
method → the Scala named-arg call fails to compile. (By construction; the rename preserves the exact
|
||||
`exec` parameter list.)
|
||||
@@ -77,14 +77,14 @@ Unaffected:
|
||||
|
||||
- The push/scan **batch** path: `scan.py` uses `run_batch("taint", …)` (scan.py:1259, 1264, 1387). This is
|
||||
why the engine-on pilot produced results.
|
||||
- `config` and `dominance` single-queries (`config_value.py:46`, `dominance.py:54`) — their `.sc` signatures
|
||||
- `config` and `dominance` single-queries (`config_value.py:46`, `dominance.py:54`): their `.sc` signatures
|
||||
are presumed to match what their specs emit (confirm separately; not exercised here).
|
||||
|
||||
Open question — **CLOSED by call-graph trace (2026-10-12).** No shipped product path and no in-repo
|
||||
Open question: **CLOSED by call-graph trace (2026-10-12).** No shipped product path and no in-repo
|
||||
benchmark/test reached the real single-query `JoernBackend.query("taint")`; the bug was latent but unreached
|
||||
by the product. Traced:
|
||||
|
||||
- **Shipped push/scan/replay** (`cli.py:1334` → `scan_repository` → `_scan_repository_impl`) is DOUBLE
|
||||
- Shipped push/scan/replay (`cli.py:1334` → `scan_repository` → `_scan_repository_impl`) is DOUBLE
|
||||
shielded: Phase 2 fetches taint via the **batch route** (`run_batch`, always wired by `build()`), and
|
||||
Phase 3's `scan_region` (scan.py:559) runs against a **`_prefetched` CpgResult** whose `run` ignores the
|
||||
query and returns the already-fetched batch rows (scan.py:1472). So `scan_region → taint_verdict →
|
||||
@@ -105,7 +105,7 @@ The fix restores the single-query path for pair/direct use without altering push
|
||||
|
||||
## Evidence
|
||||
|
||||
- `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/` — the batch path works (11 flow rows,
|
||||
- `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`: the batch path works (11 flow rows,
|
||||
`write_text` sinks); the v1 single-query attempt returned 0/0 on `cpg query taint failed`.
|
||||
- Raw joern error captured by `banks_debug.py` (scratchpad): `Missing argument: --cpgFile; Unknown
|
||||
arguments: --fixedOrigin --guards --quotedSanitizers --originAnchors --fieldParameterSources`.
|
||||
|
||||
Reference in New Issue
Block a user