docs(pre-push-safety-net): record approved design and implementation plan

This commit is contained in:
Marius Ciepluch
2026-09-12 18:27:14 +02:00
parent a8069acb41
commit 7e3b3bbc8b
15 changed files with 1398 additions and 3 deletions
+12
View File
@@ -1,5 +1,17 @@
# Overview — what this is, and what is actually true of it
**Current direction, 2026-09-12:** an actionable pre-push security safety net for AI-accelerated
development across supported ecosystems. See `safety-net.md` and the proposed `pre-push-safety-net`
spec. The ranker remains the scope mechanism; third-party vendor code stays outside targets and
graphs. Advisory must meet the same actionability bar as blocking.
**Historical scoreboard below:** it was written September 9. Later commits record a completing
whole-PMPro scan and Joern 4.0.625 ranking measurements (82/91/54 on the three plugins), while
default installation pins remain 4.0.623. The pre-push workflow, changed-code latency and actionable
precision are not demonstrated by those results. In particular, the manually selected two-file
slice below is a useful experiment, not evidence that an automatic push scope would select the
same context. Current scope/status corrections live in `flow-aware-ranking/brief.md` and the roadmap.
**Written 2026-09-09.** Fourteen specs have accumulated and the roadmap still described a pre-Joern world, so
this is the synthesis above them: what the system does, what has been measured, and what is not true yet.
Every number here comes from a committed artifact in `benchmarks/measurements/`.
+67
View File
@@ -1,5 +1,72 @@
# Roadmap
## Current direction and next review — 2026-09-12
The product is an **actionable pre-push security safety net for AI-accelerated development**, across
supported languages/frameworks. WordPress is one regression workload. Advisory must be useful and
low noise; moving speculative SAST output into a nonblocking hook does not meet the goal. See
`safety-net.md` for current steering. The earlier narrative below is retained as dated history and
does not override this direction or the latest measurements.
### Approach decision
Keep the existing ranker as the scope mechanism. Supply it with pushed-change and affected
first-party context. Third-party vendor code remains outside both targets and graphs. Reuse
compatible graph/results artifacts and apply a total deadline; treat unknown external semantics
as a coverage boundary. Require change attribution, witness and repair direction before a finding
earns normal hook visibility. A second heuristic scope selector and an assumed incremental Joern
engine are out of scope.
### Current evidence and limitations
- Whole-PMPro analysis has completed with its known CVE; the earlier "does not finish" diagnosis
is superseded. The recorded 72-minute scan is not a pre-push performance result.
- The latest Joern 4.0.625 + layout ranking budgets are PMPro 82, WP Statistics 91, MW WP Form 54,
VAmPI 16. `<50` remains unmet; VAmPI's transitive SQLI is not measured by this position metric.
- Equal-score tie-breaking alone cannot bring PMPro or WP Statistics under 50. The existing
ranker is useful, but its measured ordering is not enabled by the ordinary CLI caller.
- Docker/Compose/ops still default to 4.0.623. Pin and provenance alignment precede deployment claims.
- `contributor-scan` has 24/40 checked tasks; historical checkboxes are not rewritten by this review.
`flow-aware-ranking` is still an unapproved brief despite implementation history.
## Existing Spec Updates
- [ ] `contributor-scan` — review the draft driver/backend integration amendment; reconcile packaging
pins, family-specific sanitizer task 5.6, exact scope reporting and repository baseline honesty.
Preserve previously approved work; the new amendment is not implicitly approved.
- [ ] `flow-aware-ranking` — review the amended experiment contract: ranker owns scope, vendor
boundary preserved, missing targets stay unresolved, actual detection and cost accompany rank,
untouched evaluation and exploration precede learned-ranking claims.
- [ ] `zero-with-a-reason` — reconcile the broader funnel proposal with the hook's consumed coverage
contract; it is adjacent work, not a requirement to finish every diagnostics mechanism first.
## Specs (dependency order)
- [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact
reuse, total deadline, change attribution, actionability admission and compact hook behavior.
Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated
(8 groups, 27 executable subtasks), independently reviewed and approved; group 1 implementation started.
### Next concrete step
The maintainer approved `pre-push-safety-net/tasks.md` and starting group 1. Requirements and design
were accepted when the maintainer requested the next Kiro phase; the 27-subtask plan passed independent
sanity review. Complete group 1 (reproducible runtime, readable PMPro/Node inputs, honest scoring
and shared contracts). The first complete experiment is 5.2, before persistent caching and hook
installation work. Keep PMPro as the primary core-optimization example, then freeze the core and
prove a changed security flow and its fixed twin on a JavaScript/Express Node API workload through
the same ranker, with a real deadline and vendor-free graph. Include benign pushes and retain Python
regressions. `pre-push-safety-net/evaluation.md` proposes NodeGoat for integration, a separate untouched
Node API repository for generalization, and libpng for C envelope/abstraction-limit evidence. Shared
abstract evidence and decisions are mandatory; framework facts/adaptations are reported separately.
Measure realistic changed-code cache behavior before more ranker tuning. Broader runtime and
capability admission follow that result; an LLM ranker is optional.
## Historical roadmap narrative — September 9–11
The sections below preserve prior plans and measurements. Statements about the current bottleneck,
PHP coverage, rollout order or feature readiness are superseded by the current section above.
**Rewritten 2026-09-09.** The previous version (kept as `roadmap-2026-09-04.md`) described a pre-Joern world —
"adjudication is still Python-`ast` plus a tree-sitter CLI stub", "Joern-as-required deferred". Joern was
adopted, the model layer was built, wired and measured, and two repositories have been scanned. See
+82
View File
@@ -0,0 +1,82 @@
# Product direction: a security safety net before push
Updated 2026-09-12 from the maintainer's explicit direction. This direction supersedes the
WordPress-first release framing in older steering; historical measurements remain historical.
## Purpose
AI accelerates code production. OpenUltraSAST should help developers catch concrete security
regressions before pushing, with enough evidence to understand and repair them. It is also a
teaching system for understanding what abstract analysis can establish and where it stops.
The maintainer's acceptance criterion is an actionable safety net, not a code nag. Advisory mode
does not lower the evidence bar. A nonblocking stream of speculative findings still fails the product.
## Product principles
- Make new or worsened security defects visible; do not repeatedly present unrelated backlog.
- Every normal hook alert needs a supported security claim, a connection to the proposed change,
an exact location, and a concrete repair direction. A confidence label alone is insufficient.
- Keep suspicion and uncalibrated detector output in opt-in diagnostic artifacts.
- Separate finding disposition from coverage. An incomplete scan gets a concise operational notice,
not a fabricated vulnerability or a clean verdict. Silence is bounded by the recorded scope.
- The ranker is the scope mechanism: it selects and orders the regions/questions receiving analysis.
Change information supplies its context; there is no second heuristic scope selector in the hook.
Ranking never establishes a defect or earns permission to alert.
- Third-party vendor code is excluded from both analysis targets and graphs. This is an explicit
product boundary. Flows requiring its internals need declared summaries or remain unresolved;
context expansion must not silently reintroduce vendor sources.
- Preserve vulnerability-family semantics: escaping for one context does not discharge another.
- Reuse results only when code, context, analysis semantics, and scope still match.
- A push deadline covers all work, including snapshot preparation, graph construction, retries,
queries, optional model calls, and reporting. A region count is not a runtime bound.
## Scope and evidence
PMPro remains the primary worked example for core optimization. JavaScript/Node APIs, especially
Express routing and middleware, are a required transfer workload alongside Python; they must not
be deferred until a PHP-specific implementation has hardened. Polyglot repositories and common
injection, path, authorization, and configuration scenarios belong in the evaluation according to
demonstrated capabilities. Parsing a language does not imply support for its security properties.
C/C++ supplies an additional shared-machinery and abstraction-limit workload; C bounds/arithmetic
reasoning remains a separate abstraction question.
Language-agnostic analysis is an architectural requirement: frontends and versioned framework facts
map concrete code to shared sources, sinks, carried values, guards/obligations, configuration values
and evidence. Core ranking, scope selection and arbitration consume those contracts. No PMPro paths,
WordPress API names, Express method names or language-specific score boosts belong in core ranking.
Equivalent normalized evidence must produce equivalent core tier/score decisions. This is a contract
to verify, not an assumption that every frontend already produces equally complete graphs.
Optimize on PMPro, freeze the core, and measure transfer to Node/Express before further tuning.
Report changes needed in framework facts separately from core changes. Once an example informs a
change it becomes development/regression data; keep a separate untouched repository for evaluation.
Measure precision and actionability of alerts, recall on labeled security changes, fixed-side
silence, benign-push interruption rate, completed coverage, and cold/warm latency. Report sample
counts and uncertainty. More findings or a better rank position is not product success.
Keep the repeatedly inspected plugins as regression workloads. Evaluate new ranking or alert
policies on untouched repositories and benign changes as well. A policy that suppresses all alerts
does not pass: useful recall and coverage are required alongside low noise.
## Specification boundaries
- `pre-push-safety-net`: immutable pushed snapshots, ranker integration, reusable results, total budget,
change attribution, actionability admission, and concise hook behavior.
- `contributor-scan`: reusable scan driver/backend, family correctness, packaging, repository validation.
- `flow-aware-ranking`: reusable scheduling of questions and honest ranking experiments.
- `zero-with-a-reason`: broader pipeline diagnostics; the hook consumes concrete coverage information.
- `finding-feedback-loop`: review ledger, dismissals, and later feedback-driven evolution.
The hook does not require an LLM ranker or an optimizer. Start with the existing detector, earn
alert admission per capability, and add scheduling sophistication only when it improves the measured
developer outcome. Do not rebuild a custom IR, add a second engine, or assume incremental Joern
updates as a shortcut to an unmeasured latency promise.
## Approval state
The product direction above is user-requested. The amended requirements, design and initial
evaluation profile in `pre-push-safety-net` were approved for task generation on 2026-09-12
("ok, next kiro phase"). Task approval remains pending. This is not authorization to implement
or a claim that hook performance or capability admission has been demonstrated.