mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
docs(g2): open the change-scoped engine-analysis workstream (brief + roadmap)
Track the engine-detection lever found while driving G0. Live diagnosis on the frozen pool: the engine's "no finding" on unseen changes is usually analysis-never-runs -- it ranks repo-wide candidates and defers them all (deadline_exhausted) before analyzing the pushed change (bedrock injection, 303s, selected=[], ranked sites unrelated to the change). Lever: seed analysis from the change's declarations + dataflow neighbourhood (rank.py/mapping.py) so it runs within budget. - .kiro/specs/engine-change-scoped-analysis/brief.md: problem (measured), hypothesis, anchors, and checkable increments (0 confirm build-vs-analysis cost split -> 1 locate selection/budget -> 2 seed from change -> 3 soundness + tests -> 4 live re-test -> 5 measure delta on the engine arm). - roadmap.md G2: active workstreams (change-scoped analysis; the done-but- necessary-not-sufficient line-correspondence fix), for progress tracking. Honest scope: G2, gated on the pool metric; not part of the G0 quick-rule record-07. Coverage also needs the capability registry (Task 8.3); arm-scale measurement needs the kube lane. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
c39fbb8457
commit
9f312177ce
@@ -0,0 +1,61 @@
|
|||||||
|
# Engine change-scoped analysis (G2 workstream)
|
||||||
|
|
||||||
|
Status: **proposed / in progress** (opened 2026-10-11). Owner: engine. Tracks a concrete lever for the
|
||||||
|
engine-detection gap found while driving the G0 baseline. This is G2 ("improve what moves the unseen-repo
|
||||||
|
numbers"), gated on the pool metric; it is NOT part of the G0 record-07 baseline (that runs on the quick-rule tier).
|
||||||
|
|
||||||
|
## Problem (measured, not assumed)
|
||||||
|
|
||||||
|
On real unseen vulnerability changes the engine produces **no finding** — but the diagnosis (live, 300 s long-deadline
|
||||||
|
replays) shows it is usually **not a detection-capability gap**:
|
||||||
|
|
||||||
|
- **Clean Python injection change** (`aws/bedrock-agentcore-sdk-python`, `install_packages`, introducing, 3-file
|
||||||
|
diff): `wall=303 s`, hit the deadline. The ranker found candidates (`static_rank=1.0`) but **`selected: []`** and
|
||||||
|
every candidate `deferred` with `reason: deadline_exhausted`, `analysis_not_started=deadline_exhausted`,
|
||||||
|
`change_context_incomplete`. **Zero analyses ran.** And the ranked candidates were **repo-wide and unrelated to
|
||||||
|
the push** (`auth.py` access_control/config_secrets), not the changed `install_packages`. The engine is trying to
|
||||||
|
re-audit the whole repo and runs out of budget before analyzing the change.
|
||||||
|
- **Noisy last_touch config_secrets change** (`jenkinsci/metrics-plugin`, 14-file diff): finished in 206 s but
|
||||||
|
`comparison_unknown` + `capability_unavailable` + `eligibility_provenance_mismatch` +
|
||||||
|
`contributor-scan ... projection_unavailable`. Multi-causal; correspondence was one part (fixed separately,
|
||||||
|
commit c39fbb8), capability registry is Task 8.3.
|
||||||
|
|
||||||
|
## Hypothesis
|
||||||
|
|
||||||
|
A pre-push hook should vet **the code this push changed**, not re-scan the whole repository. Seeding/scoping the
|
||||||
|
engine's analysis to the **changed declarations and their dataflow neighborhood** (the `change_sites` are already in
|
||||||
|
the replay spec) should let analysis **actually run within budget** instead of being deferred — which is where
|
||||||
|
detection is being lost, not in Joern's capability.
|
||||||
|
|
||||||
|
## Anchors (confirm in increment 1)
|
||||||
|
|
||||||
|
- `src/openultrasast/rank.py` — the static-evidence ranker (`ranking_mode="evidence"`, `existing-static-evidence-v1`).
|
||||||
|
- `src/openultrasast/mapping.py` — builds the `selected`/`deferred` scope + the budget allocation that defers on
|
||||||
|
`deadline_exhausted`.
|
||||||
|
- `src/openultrasast/push/policy.py`, `push/runner.py` — consume the scope; change_sites arrive via the push input.
|
||||||
|
- `benchmarks/unseen/replay_entry.py` — carries `changed_paths` / `declaration_paths` in the spec.
|
||||||
|
|
||||||
|
## Increments (progress tracking)
|
||||||
|
|
||||||
|
- [ ] **0. Confirm the cost split.** Instrument one replay (bedrock) to measure CPG-build vs rank vs analysis time
|
||||||
|
within the 300 s budget. If the **CPG build** alone exhausts it, scoping the analysis won't help (needs a build-time
|
||||||
|
or budget fix instead); if **ranking the whole repo** is the overspend (candidates got ranked, so the build likely
|
||||||
|
finished), change-scoping is the right lever. Decides the whole approach — do this first (instrument rule).
|
||||||
|
- [ ] **1. Locate the exact selection/budget code** in `rank.py`/`mapping.py` that marks candidates
|
||||||
|
`selected` vs `deferred:deadline_exhausted`.
|
||||||
|
- [ ] **2. Seed analysis from the change.** Prioritize (or restrict first-pass to) the changed declarations plus
|
||||||
|
their call/data neighborhood over repo-wide static rank, within budget.
|
||||||
|
- [ ] **3. Soundness.** Keep cross-file dataflow INTO the changed function (don't scope to literal changed lines);
|
||||||
|
add tests; keep existing push tests green.
|
||||||
|
- [ ] **4. Live re-test.** Re-run the bedrock change: does analysis now RUN and produce a finding on
|
||||||
|
`install_packages` (vs `deferred`)? Re-run metrics-plugin.
|
||||||
|
- [ ] **5. Measure the delta.** Engine-arm sample on the pool (catch/false-alarm + engine-coverage rate) — needs the
|
||||||
|
kube lane (blocked on the AX completion path) for scale.
|
||||||
|
|
||||||
|
## Dependencies / notes
|
||||||
|
|
||||||
|
- Coverage `complete_within_scope` also needs the capability registry (Task 8.3) regardless of this work; the win
|
||||||
|
here is findings moving from `comparison_unknown`/deferred to a decided novelty → advisory engine finding.
|
||||||
|
- Implement directly and verify on the LIVE instrument (a fake-client/unit pass is not evidence the live engine
|
||||||
|
changed); the kube-lane and store endpoints are operator-owned infra, not assumptions to make here.
|
||||||
|
</content>
|
||||||
@@ -97,6 +97,19 @@ Candidates, each admitted only if it improves the pool metric on a fresh slice:
|
|||||||
(approved plan, now one arm, not the spine), engine coverage and TypeScript, decision-engine prompt and retrieval
|
(approved plan, now one arm, not the spine), engine coverage and TypeScript, decision-engine prompt and retrieval
|
||||||
changes, more labelled pairs from more distinct repositories.
|
changes, more labelled pairs from more distinct repositories.
|
||||||
|
|
||||||
|
Active G2 workstreams (2026-10-11, opened while driving G0; progress tracked in their briefs):
|
||||||
|
|
||||||
|
- **Change-scoped engine analysis** (`.kiro/specs/engine-change-scoped-analysis/brief.md`). Live diagnosis on the
|
||||||
|
frozen pool showed the engine's "no finding" on unseen changes is usually *analysis-never-runs*: the engine ranks
|
||||||
|
repo-wide candidates and `deferred: deadline_exhausted` before analyzing anything (even at 300 s), and the ranked
|
||||||
|
sites are unrelated to the pushed change. Lever: seed analysis from the change's declarations + dataflow
|
||||||
|
neighbourhood (`rank.py`/`mapping.py`) so it runs within budget. Increment 0 (confirm CPG-build-vs-analysis cost
|
||||||
|
split) gates the rest. Scale measurement needs the kube lane.
|
||||||
|
- **Line-correspondence (done, necessary-not-sufficient).** `snapshot.py` discarded base↔head anchors on
|
||||||
|
non-globally-unique lines → `comparison_unknown` → suppressed findings; fixed to run-aware anchoring
|
||||||
|
(commit c39fbb8, 163 tests). Removes one cause; coverage still needs change-scoped analysis + the capability
|
||||||
|
registry (Task 8.3).
|
||||||
|
|
||||||
**G3. Decide BLOCK per family** on the pool's false-alarm rate and the pooled precision bound (Req 6.4).
|
**G3. Decide BLOCK per family** on the pool's false-alarm rate and the pooled precision bound (Req 6.4).
|
||||||
|
|
||||||
**G4. Detection on push for any repository** (`plane-on-kubernetes` P4 work: `ousast plane scan --base --head`,
|
**G4. Detection on push for any repository** (`plane-on-kubernetes` P4 work: `ousast plane scan --base --head`,
|
||||||
|
|||||||
Reference in New Issue
Block a user