mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
docs(pre-push-safety-net): record approved design and implementation plan
This commit is contained in:
@@ -1,5 +1,17 @@
|
||||
# Overview — what this is, and what is actually true of it
|
||||
|
||||
**Current direction, 2026-09-12:** an actionable pre-push security safety net for AI-accelerated
|
||||
development across supported ecosystems. See `safety-net.md` and the proposed `pre-push-safety-net`
|
||||
spec. The ranker remains the scope mechanism; third-party vendor code stays outside targets and
|
||||
graphs. Advisory must meet the same actionability bar as blocking.
|
||||
|
||||
**Historical scoreboard below:** it was written September 9. Later commits record a completing
|
||||
whole-PMPro scan and Joern 4.0.625 ranking measurements (82/91/54 on the three plugins), while
|
||||
default installation pins remain 4.0.623. The pre-push workflow, changed-code latency and actionable
|
||||
precision are not demonstrated by those results. In particular, the manually selected two-file
|
||||
slice below is a useful experiment, not evidence that an automatic push scope would select the
|
||||
same context. Current scope/status corrections live in `flow-aware-ranking/brief.md` and the roadmap.
|
||||
|
||||
**Written 2026-09-09.** Fourteen specs have accumulated and the roadmap still described a pre-Joern world, so
|
||||
this is the synthesis above them: what the system does, what has been measured, and what is not true yet.
|
||||
Every number here comes from a committed artifact in `benchmarks/measurements/`.
|
||||
|
||||
@@ -1,5 +1,72 @@
|
||||
# Roadmap
|
||||
|
||||
## Current direction and next review — 2026-09-12
|
||||
|
||||
The product is an **actionable pre-push security safety net for AI-accelerated development**, across
|
||||
supported languages/frameworks. WordPress is one regression workload. Advisory must be useful and
|
||||
low noise; moving speculative SAST output into a nonblocking hook does not meet the goal. See
|
||||
`safety-net.md` for current steering. The earlier narrative below is retained as dated history and
|
||||
does not override this direction or the latest measurements.
|
||||
|
||||
### Approach decision
|
||||
|
||||
Keep the existing ranker as the scope mechanism. Supply it with pushed-change and affected
|
||||
first-party context. Third-party vendor code remains outside both targets and graphs. Reuse
|
||||
compatible graph/results artifacts and apply a total deadline; treat unknown external semantics
|
||||
as a coverage boundary. Require change attribution, witness and repair direction before a finding
|
||||
earns normal hook visibility. A second heuristic scope selector and an assumed incremental Joern
|
||||
engine are out of scope.
|
||||
|
||||
### Current evidence and limitations
|
||||
|
||||
- Whole-PMPro analysis has completed with its known CVE; the earlier "does not finish" diagnosis
|
||||
is superseded. The recorded 72-minute scan is not a pre-push performance result.
|
||||
- The latest Joern 4.0.625 + layout ranking budgets are PMPro 82, WP Statistics 91, MW WP Form 54,
|
||||
VAmPI 16. `<50` remains unmet; VAmPI's transitive SQLI is not measured by this position metric.
|
||||
- Equal-score tie-breaking alone cannot bring PMPro or WP Statistics under 50. The existing
|
||||
ranker is useful, but its measured ordering is not enabled by the ordinary CLI caller.
|
||||
- Docker/Compose/ops still default to 4.0.623. Pin and provenance alignment precede deployment claims.
|
||||
- `contributor-scan` has 24/40 checked tasks; historical checkboxes are not rewritten by this review.
|
||||
`flow-aware-ranking` is still an unapproved brief despite implementation history.
|
||||
|
||||
## Existing Spec Updates
|
||||
|
||||
- [ ] `contributor-scan` — review the draft driver/backend integration amendment; reconcile packaging
|
||||
pins, family-specific sanitizer task 5.6, exact scope reporting and repository baseline honesty.
|
||||
Preserve previously approved work; the new amendment is not implicitly approved.
|
||||
- [ ] `flow-aware-ranking` — review the amended experiment contract: ranker owns scope, vendor
|
||||
boundary preserved, missing targets stay unresolved, actual detection and cost accompany rank,
|
||||
untouched evaluation and exploration precede learned-ranking claims.
|
||||
- [ ] `zero-with-a-reason` — reconcile the broader funnel proposal with the hook's consumed coverage
|
||||
contract; it is adjacent work, not a requirement to finish every diagnostics mechanism first.
|
||||
|
||||
## Specs (dependency order)
|
||||
|
||||
- [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact
|
||||
reuse, total deadline, change attribution, actionability admission and compact hook behavior.
|
||||
Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated
|
||||
(8 groups, 27 executable subtasks), independently reviewed and approved; group 1 implementation started.
|
||||
|
||||
### Next concrete step
|
||||
|
||||
The maintainer approved `pre-push-safety-net/tasks.md` and starting group 1. Requirements and design
|
||||
were accepted when the maintainer requested the next Kiro phase; the 27-subtask plan passed independent
|
||||
sanity review. Complete group 1 (reproducible runtime, readable PMPro/Node inputs, honest scoring
|
||||
and shared contracts). The first complete experiment is 5.2, before persistent caching and hook
|
||||
installation work. Keep PMPro as the primary core-optimization example, then freeze the core and
|
||||
prove a changed security flow and its fixed twin on a JavaScript/Express Node API workload through
|
||||
the same ranker, with a real deadline and vendor-free graph. Include benign pushes and retain Python
|
||||
regressions. `pre-push-safety-net/evaluation.md` proposes NodeGoat for integration, a separate untouched
|
||||
Node API repository for generalization, and libpng for C envelope/abstraction-limit evidence. Shared
|
||||
abstract evidence and decisions are mandatory; framework facts/adaptations are reported separately.
|
||||
Measure realistic changed-code cache behavior before more ranker tuning. Broader runtime and
|
||||
capability admission follow that result; an LLM ranker is optional.
|
||||
|
||||
## Historical roadmap narrative — September 9–11
|
||||
|
||||
The sections below preserve prior plans and measurements. Statements about the current bottleneck,
|
||||
PHP coverage, rollout order or feature readiness are superseded by the current section above.
|
||||
|
||||
**Rewritten 2026-09-09.** The previous version (kept as `roadmap-2026-09-04.md`) described a pre-Joern world —
|
||||
"adjudication is still Python-`ast` plus a tree-sitter CLI stub", "Joern-as-required deferred". Joern was
|
||||
adopted, the model layer was built, wired and measured, and two repositories have been scanned. See
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
# Product direction: a security safety net before push
|
||||
|
||||
Updated 2026-09-12 from the maintainer's explicit direction. This direction supersedes the
|
||||
WordPress-first release framing in older steering; historical measurements remain historical.
|
||||
|
||||
## Purpose
|
||||
|
||||
AI accelerates code production. OpenUltraSAST should help developers catch concrete security
|
||||
regressions before pushing, with enough evidence to understand and repair them. It is also a
|
||||
teaching system for understanding what abstract analysis can establish and where it stops.
|
||||
|
||||
The maintainer's acceptance criterion is an actionable safety net, not a code nag. Advisory mode
|
||||
does not lower the evidence bar. A nonblocking stream of speculative findings still fails the product.
|
||||
|
||||
## Product principles
|
||||
|
||||
- Make new or worsened security defects visible; do not repeatedly present unrelated backlog.
|
||||
- Every normal hook alert needs a supported security claim, a connection to the proposed change,
|
||||
an exact location, and a concrete repair direction. A confidence label alone is insufficient.
|
||||
- Keep suspicion and uncalibrated detector output in opt-in diagnostic artifacts.
|
||||
- Separate finding disposition from coverage. An incomplete scan gets a concise operational notice,
|
||||
not a fabricated vulnerability or a clean verdict. Silence is bounded by the recorded scope.
|
||||
- The ranker is the scope mechanism: it selects and orders the regions/questions receiving analysis.
|
||||
Change information supplies its context; there is no second heuristic scope selector in the hook.
|
||||
Ranking never establishes a defect or earns permission to alert.
|
||||
- Third-party vendor code is excluded from both analysis targets and graphs. This is an explicit
|
||||
product boundary. Flows requiring its internals need declared summaries or remain unresolved;
|
||||
context expansion must not silently reintroduce vendor sources.
|
||||
- Preserve vulnerability-family semantics: escaping for one context does not discharge another.
|
||||
- Reuse results only when code, context, analysis semantics, and scope still match.
|
||||
- A push deadline covers all work, including snapshot preparation, graph construction, retries,
|
||||
queries, optional model calls, and reporting. A region count is not a runtime bound.
|
||||
|
||||
## Scope and evidence
|
||||
|
||||
PMPro remains the primary worked example for core optimization. JavaScript/Node APIs, especially
|
||||
Express routing and middleware, are a required transfer workload alongside Python; they must not
|
||||
be deferred until a PHP-specific implementation has hardened. Polyglot repositories and common
|
||||
injection, path, authorization, and configuration scenarios belong in the evaluation according to
|
||||
demonstrated capabilities. Parsing a language does not imply support for its security properties.
|
||||
C/C++ supplies an additional shared-machinery and abstraction-limit workload; C bounds/arithmetic
|
||||
reasoning remains a separate abstraction question.
|
||||
|
||||
Language-agnostic analysis is an architectural requirement: frontends and versioned framework facts
|
||||
map concrete code to shared sources, sinks, carried values, guards/obligations, configuration values
|
||||
and evidence. Core ranking, scope selection and arbitration consume those contracts. No PMPro paths,
|
||||
WordPress API names, Express method names or language-specific score boosts belong in core ranking.
|
||||
Equivalent normalized evidence must produce equivalent core tier/score decisions. This is a contract
|
||||
to verify, not an assumption that every frontend already produces equally complete graphs.
|
||||
|
||||
Optimize on PMPro, freeze the core, and measure transfer to Node/Express before further tuning.
|
||||
Report changes needed in framework facts separately from core changes. Once an example informs a
|
||||
change it becomes development/regression data; keep a separate untouched repository for evaluation.
|
||||
|
||||
Measure precision and actionability of alerts, recall on labeled security changes, fixed-side
|
||||
silence, benign-push interruption rate, completed coverage, and cold/warm latency. Report sample
|
||||
counts and uncertainty. More findings or a better rank position is not product success.
|
||||
|
||||
Keep the repeatedly inspected plugins as regression workloads. Evaluate new ranking or alert
|
||||
policies on untouched repositories and benign changes as well. A policy that suppresses all alerts
|
||||
does not pass: useful recall and coverage are required alongside low noise.
|
||||
|
||||
## Specification boundaries
|
||||
|
||||
- `pre-push-safety-net`: immutable pushed snapshots, ranker integration, reusable results, total budget,
|
||||
change attribution, actionability admission, and concise hook behavior.
|
||||
- `contributor-scan`: reusable scan driver/backend, family correctness, packaging, repository validation.
|
||||
- `flow-aware-ranking`: reusable scheduling of questions and honest ranking experiments.
|
||||
- `zero-with-a-reason`: broader pipeline diagnostics; the hook consumes concrete coverage information.
|
||||
- `finding-feedback-loop`: review ledger, dismissals, and later feedback-driven evolution.
|
||||
|
||||
The hook does not require an LLM ranker or an optimizer. Start with the existing detector, earn
|
||||
alert admission per capability, and add scheduling sophistication only when it improves the measured
|
||||
developer outcome. Do not rebuild a custom IR, add a second engine, or assume incremental Joern
|
||||
updates as a shortcut to an unmeasured latency promise.
|
||||
|
||||
## Approval state
|
||||
|
||||
The product direction above is user-requested. The amended requirements, design and initial
|
||||
evaluation profile in `pre-push-safety-net` were approved for task generation on 2026-09-12
|
||||
("ok, next kiro phase"). Task approval remains pending. This is not authorization to implement
|
||||
or a claim that hook performance or capability admission has been demonstrated.
|
||||
Reference in New Issue
Block a user